Heading into Rails TOOOOOO-TOOOOOO-TOOOOOO!

When I played little league baseball we had announcers for the game, one of which was some nice old guy in a ramshackle shelter behind home plate that was the “booth” – and every time there’d be two outs, two strikes and two balls – he’d croon “TOOOOO-TOOOOO-TOOOO”. Which of course, has little to do with this post other than version numbers.

Rails 2.2 was just released – so of course, I’m taking the most critical application we have (well, maybe the second critical, and that’s really only because other apps depend on it, it’s not critical in terms of features) – and upgrading it first, because, well, that’s how system administrators roll.

The release notes are great. Seriously. Really great. So great I shouldn’t even be writing this post. Which I am anyway – because here are some of the highlights of what I had to change to make my app work.

NoMethodError for Association Methods

Getting exceptions when you go to Rails 2.2.2 that don’t say anything more than NoMethodError when you know good and damn well the method that it’s saying no method on exists? Yeah, me too. And I bet it’s a method in an associated model. And if it is, you probably should be ashamed of yourself.

Rails 2.2 now enforces privacy on private methods called through associations. So in my case, I had two issues, 1) I was calling “update” on an associated model in some code I blindly copied and pasted a long time ago, and 2) I have a few of my own SQL queries that I’m not entirely sure how to do using Rails associations and named scopes, and I was cheating by calling self.connection.sanitize_limit to take advantage of Rails’ own function for cleaning up provided “LIMIT” params. And sanitize_limit, like the instance method update is private.

Update Rails Footnotes

If you use Rails footnotes in development mode – you’ll want to update for this change for Rails 2.2 compatibility.

Aside… Piston 1.9.5

A great way to stay current with Rails plugins is to use Piston – which has a new 1.9.5 release. You can build your own.

“quoted_table_name” and Has Many Polymorphs

If you start getting some error about undefined method `quoted_table_name' and you use has_many_polymorphs – you’ll want this change.

I don’t use the plugin, I use the gem. So I built my own has_many_polymorphs 2.12.1 gem – by doing:

gem install echoe
git clone git://github.com/fauna/has_many_polymorphs.git
cd has_many_polymorphs


edit CHANGELOG with a “v2.12.1 line” (e.g. v2.12.1. Cloned GitHub project and rebuilt gem for our nefarious purposes.)


rake manifest
rake package
rake install


add_joins! and Has Many Polymorphs (or anything else for that matter)

HMP includes a ‘tagged_with’ method for finding collections ‘tagged_with’ a set of tags. I use a heavily modified version of that. The method supports custom scopes, in theory. (well, probably more than theory, I’ve just never tried it). While, I don’t have any scopes on models that call my functions – I still had some of the private ActiveRecord method calls in mine – particularly add_joins!.

Well, this change changed the params for the method to make sure that the scoped joins were merged, and not overwritten – which changes calls into it. If you are calling it with your own options use options[:joins]. My code doesn’t use the scopes in combination with my tagged_with method, so I just pulled them.

And…. thankfully that’s it

Other than cleaning up deprecations like number_with_precision now preferring (number, :precision => myprecision) instead of (number, myprecision) – and ActiveRecord::Base.verification_timeout no longer valid, and mb_chars being preferred over chars – we seem to be good to go for Rails 2.2. I still need to test some crons, but I imagine that our app will go production as 2.2 shortly.

TOOOOOO-TOOOOOO-TOOOOOO!

A Story of a Bug

It started, as it should, with the belief that it was my own bug.

I’ve been working for the last few days to generate daily summaries of the activity flowing through our tools. It’s nothing earth shattering, but it’s been a stepping stone to understand a little bit more about the Rails framework – and gave me the chance to begin experimenting with the Google Visualization API. Toward the latter part of the week, there was something a little odd with the “total valid” numbers with the daily account creations – I had made data changes to make sure I had some idea when accounts were vouched for and when they had been retired – so I naturally assumed it was something I did. I even went back and modified the model to make it more consitent with it’s peer models. And kept running the script that produced the daily stats in the Einstein-esque insanity of the doing the same thing twice and expecting different results. After about a dozen combinations of DATE(date_column) comparisons – I went to google, because I knew by then I was either going crazy or this was a legitimate “it’s not my problem” bug.

Which led me to this mysql bug. Reported July 19, 2007. Apparently introduced in MySQL 5.0.42 (May 2007) when the DATE comparison changed from strings to ints. Fixed within two days as part of 5.0.48 (Released August 2007).

But guess which mysql package Red Hat EL 5 (well, RHEL5 update 2) provides? – right, in between. It’s MySQL 5.0.45. And the forthcoming RHEL5 update 3 release doesn’t update MySQL either.

Development and System Administration is a weird, weird world. I use RHEL, not for support (I’m not even sure we have support with the University contract), but to have some degree of patch level stability that’s slightly longer than the fedora releases (and at the time I went to RHEL, Fedora was still dealing with it’s Red Hat transition) – but that stability comes with the price of things like this. I already use my own Ruby to get beyond the base install, but configure, make, make install for one piece of core software is a little different than dealing with it (or MySQL-supplied RPMs) for other software.

I’m glad the open source world gives me that choice, but open source + my labor + thousands of moving parts does give provide the reality that even when a bug is fixed two days later, in the open, patchable by everyone – that sometimes you can find yourself over a year later modifying your own DATE queries so that they don’t include nulls.

So that’s the overall summary of the post I guess – part of it to go into google that if you are getting odd MySQL DATE function results on MySQL 5.0.45 on Red Hat Enterprise Linux (RHEL 5) – it’s a bug. And it’s fixed. But not included in RHEL 5. And if you aren’t getting odd results with DATE comparisons – you probably don’t know that you are.

And maybe one part as a lament to that inevitable ongoing intersection of thousands of moving parts in every environment, not the least of which ours. And you trade off replacing mysql on multiple servers and just turn nulls into zeros (which then breaks your signup form that desparately needs an overhaul) – well, because it seemed to make sense at the time.

And people say you don’t use probability after college.

Creating your own Gem Server, redux

One of the most popular landing pages for rambleon.org (which isn’t saying much) – is a post I put together in April, 2007 about running your own gem server.

Unfortunately it’s outdated. Set let me revisit it for those landing on the old page and looking to put together their own gem server. The nice thing is, gem has (almost) all the tools you need.

Setting up your server

1. You need a web server. (yes I know you can run gem_server, but get a real one). You are on your own for that one. You also need a rubygems install on that box. You are also on your own for bootstrapping rubygems on that box and any other ones.
2. Decide where you will put your gems (say in a “mycoolgems” directory off the docroot for your webserver)
3. $ mkdir [docroot]/mycoolgems/gems
4. Copy your .gem files that you want to host to $ [docroot]/mycoolgems/gems
5. The gem suite of commands includes a generate_index command to generate a yaml-based index of your gems, and other supporting files. See gem help generate_index for more information

Pointing your systems to your own server

This used to be a complete PITA that involved rebuilding the sources gem. No more! After you install rubygems – just make sure to do a:

gem sources --remove http://gems.rubyforge.org

and a

gem sources --add http://yourwaycool.gem.source

That’s All Folks (probably)

Voila! You just managed to point your server to your own gem server! Install away.

It’s good to keep one box pointed to http://gems.rubyforge.org – and take advantage of the new “gem outdated” command to keep track of changes in your installed gems that have been deployed to rubyforge.

Your own gem server and Ruby 1.8.5

A while back, I wrote about creating your own gem server – which basically consisted of custom building a sources gem. Mainly because gem sources --remove at the time wouldn’t let you remove rubyforge from the sources list.

Thankfully, gem sources --remove now works great. And with RubyGems 0.9.5 – they’ve included a new gem generate_index command that will generate the indexes for you. So now – if you want to create your own gem server, you can run the web server software of your choice and use the gem generate_indixe -d [docroot]/mycoolgems command to build the indexes.

(If you do update the rubygems-update gem on your “client” systems, make sure to do a gem sources --add http://yourgemserver and a gem sources --remove http://gems.rubyforge.org from your sources. Or continue building your own sources gem.)

Unfortunately, through no fault of the RubyGems devs, this might not work on a system running Ruby 1.8.5 – like, say, Red Hat Enterprise Linux, version 5. The fault lies in Ruby itself. gem generate_index builds its index in /tmp – and uses FileUtils to move it out of /tmp and into the place you specified as the target directory (or ‘.’). If your servers are like mine, /tmp and your web server directories are on different partitions, and unfortunately FileUtils in Ruby 1.8.5 has a bug moving directories across partitions.

The bug was fixed, I think with this revision in March 2007 – and it’s in Ruby 1.8.6 (at least as of p110/111) – but it’s not in Red Hat’s 1.8.5 packages – and I’m not sure it qualifies as a security bug, so I doubt it will get backported.

Anyway, I went the configure/make/make install route (package purists, feel free to cringe, but I don’t want to make RPM’s and this is core enough for me to stay on top of). Your mileage may vary.

Creating Your Own Gem Server

Update – August 1, 2008

This page remains one of my most active pages for Google Searches. A number of folks want to run their own gem server, and for good reason.

Unfortunately it’s outdated and has been for a while, and I’ve been too lazy to update it.

I also hate for you to leave with outdated information – so please check out the updated information about the commands that are now built into gem and available for facilitating running your own gem distribution site.

So before I automate things, I needed a way to point to my own gem server. Which thankfully, it turns out, that it’s not all that hard to do so (it’s harder than it has to be, and man oh man is the gem cache a pain in the rear). This all assumes that you already had rubygems installed on the boxes that you are moving over to point to your own server. (This also assumes that you use a Linux/Unix server, I’m sure all this works on a Windows server, but I haven’t tested one and honestly don’t care).

Setting up your server

1. You need a web server. (yes I know you can run gem_server, but get a real one). You are on your own for that one. You also need a rubygems install on that box. You are also on your own for bootstrapping rubygems on that box and any other ones.
2. Decide where you will put your gems (say in a “mycoolgems” directory off the docroot for your webserver)
3. $ mkdir [docroot]/mycoolgems/gems
4. Copy your .gem files that you want to host to $ [docroot]/mycoolgems/gems
5. Rubygems installed a "index_gem_repository.rb" ruby script in your path. (well probably in your path, it’s in /usr/bin on my systems). You want to run this to generate a yaml-based index of your gems. (appropriately named “yaml” – and a compressed yaml.Z) e.g. index_gem_repository.rb -d [docroot]/mycoolgems

Pointing your systems to your own server

Setting up the server is the easy part. The harder part is pointing all your boxes to your own server. And only your own server. You’d think that one neat thing about the 0.9.2 rubygems release is that it includes a “gem sources” command to theoretically add and remove gem sources that your boxes would look at, but you’d be wrong. Because you can’t get rid of the base source of http://gems.rubyforge.org without modifying the sources gem or the sources distribution on your own box. You can theoretically modify [lib/ruby]/gems/1.8/gems/sources-0.0.1/lib/sources.rb and change:

module Gem
  @sources = ["http://gems.rubyforge.org"]
  def self.sources
    @sources
  end
end

to


@sources = ["http://yourserver.yourdomain"]


However, that customization is likely going to get blown away the next time you update rubygems with a gem update --system – because the sources gem is built by the rubygems update. So what do you do? Build your own gem update.

1. Download the RubyGems source
2. Edit pkgs/sources/lib/sources.rb to point to your own server
3. Rebuild the gem by issuing a rake package – which will build the rubygems update gem with your source changes (in pkg/ – in my case pkg/rubygems-update-0.9.2.gem)
4. Copy this gem to your gem server’s gems directories (and rebuild the yaml index as appropriate)
5. On your other servers – clear out [lib/ruby]/gems/1.8/cache
6. Remove the [lib/ruby]/gems/1.8/source_cache
7. Run gem update --system --source http://yourserver.yourdomain

That’s All Folks (probably)

Voila! You just managed to point your server to your own gem server! Install away.

It’s good to keep one box pointed to http://gems.rubyforge.org – and take advantage of the new “gem outdated” command to keep track of changes in your installed gems that have been deployed to rubyforge.

[Updated: April 19, 2007 Thanks to Jim Weirich for pointing out that the index script should actually be index_gem_repository.rb and NOT generate_yaml_index.rb. generate_yaml_index.rb is a holdover from earlier rubygems versions]

Don’t Do That

So… maybe you are coding up your totally way rad awesome application in Rails – and you are thinking to yourself.

“Self, I really would like to set my own created_at and updated_at timestamps. Look – there’s even a way to do that in the Rails documentation”


class Feed < ActiveRecord::Base
self.record_timestamps = false
# ...
end


At this point you need to back away from the keyboard. Quickly. If you don’t, pretty soon, somewhere in your application – you are going to run into this error:

Mysql::Error: Column 'created_at' cannot be null

Or ALL KINDS OF OTHER FUN SIDE EFFECTS (FUN is actually a euphemism here for various four letter words)

See, record_timestamps is a class variable for ActiveRecord::Base created with the rails :cattr_accessor – maybe the self.record_timestamps should have tipped us all off – maybe not (there’s also a class_inheritable_accessor – I’m not sure where all that gets used though)

Even experienced developers not all that fluent in ruby minutiae (I think class variables count as minutiae) cut-and-paste first and figure out how it works second (yeah, don’t do that either).

So – anyway – once you change record_timestamps once – you change it for all descendants of ActiveRecord::Base

There’s a bit of discussion this on a separate, but related problem at Evan Weaver’s blog (pay special attention to that threading issue for those playing along with the home game). And of course, your friendly neighborhood reminder of what happens with class variables at Nic Williams’ blog (I recommend reading that twice and breaking out the home game version of irb)

So the moral of the story? self.record_timestamps – Don’t Do That.

p.s. Production of this blog entry was made possible through various grants and assessments, and with some moans, groans, sighs, and “what tha–?” from my colleagues James Robinson and Aaron Hundley (doesn’t have a blog, he needs to get with the program)

p.p.s edited to change “you chance it for all descendants…” to “you change it for all descendants” – I think the first one is quite apropos however

p.s. it’s never really about a single post

so I went into snarky overdrive with my vendor dependencies post. it wasn’t quite as funny as last year when I went off on the people that can’t unsubscribe from lists either. Nor was it as funny as some other commentary on Rails I’ve snarkily made

I really do love the err.the.blog guys – – they have the best rails blog – bar none – that I’ve ever found – I even have their toolbox post burned into my retina I think.

They really know their stuff. Rails needs devs like this – and they do a great service educating other folks about the framework.

But I don’t agree (obviously) with packaging up all the dependencies with an application. I get all the reasons for doing so. It just sets a really dangerous precedent for the people that are going to take it as the gospel and never think about the ramifications of what packaging up everything with your application means. (like simple things – remind me to tell the inode story sometime. 20 capistrano delivered copies of edge rails might not kill your storage but it dang sure can eat some inodes)

But hell, you can’t really trust the system administrators to get it right either about not breaking dozens of rails apps that they don’t have a clue about. I, um, er, have known some sysadmins to do that (more than once even).

p.p.s Oh, man, I forgot about the sponsorship link. This post sponsored by the Static Linking Historical Society. And support also comes from Microsoft Corporation. Proud facilitators of DLL Hell for all the static linkers that decided to go dynamic, but distributed their own libraries.

Good Grief People, stop with the local gems

From Err The Blog: Vendor Everything

For hosted environments? sure.

But if you are responsible for the application AND the server? (or your shop is?)

No.

Not just no. But HELL No. And I’d really like to write “HELL No” in an <h1> but I’m going to avoid that for the sake of sanity

I’ve yet to figure out why the rails community has this inbred desire to cause harm to their reputation in organizations that aren’t pure dev shops. I’m not even talking about the enterprise, I’m talking about small business, non-profits, companies they contract with, academic shops…

I don’t disagree with Chris’s reason here for using vendor for WayCoolFunkyGemThatYouThinkIsTheBeesKnees (WCFGTYTITBK) and not being “That Person” for breaking the build (and your peeps) is laudable. But really – I don’t buy it. If you are small Rails shop and you plan on using test/spec or any other WCFGTYTITBK – for goodness sake you communicate that with the rest of your team (hello? IM? email? even that ringy thing on the hip or desk we all hate to use?)

If you think that someone else’s code is so great that it ought to be in your application – well then it ought to be in everyone’s install too. Go get up and install it for them (take the train if you can’t fly there) That’s what good developers do. They have sane development environments set up and they are completely proficient at “gem install blah” – and makes them completely aware that a brand-new third-party dependency just showed up in the application. I dare say that “gem install blah” is a lot less intrusive than “why in the sam heck did 1000 lines of crap just show up in vendor – I evaluated that third-party code last month and it was crap then and is crap now”

Local copies of every gem is madness – especially gems that are core to your application (and would break builds) It creates situations where the whole team (and often the people that run the servers and are ultimately responsible for the application) is not fully aware of the dependency needs of the application. Let me repeat that again – EVERY DEVELOPER ON A SMALL TEAM SHOULD KNOW EXACTLY WHAT AN APPLICATION DEPENDS ON, WHAT VERSION, AND SHOULD TASK THEMSELVES WITH CHECKING UP ON THOSE VERSIONS.

One app? not a big deal either way. 5 or 6 apps running in the same environment? It’s a Big deal. (of course it’s probably a complete architectural failure to have your 5 person team working on 5 or 6 apps at the same time – but that’s another post)

We had pinned rails in our applications – at least until the “Upgrade Your Rails NOW NOW NOW” event – and going through multiple applications on multiple staging servers and multiple versions was a complete pain in the ass. Okay, so that’s a little hyperbolic – but it was more trouble than it needed to be. You upgrade the server – when you control the server and your application – and you know that that the dependencies are handled.

I had these arguments a few months ago with a developer that was contracting with us – and it was like imposing a little (certainly not anything like some waterfall corporate development shop) structure on the process (“Hey – tell us exactly why you are using edge rails so that we all understand the issues”) was like we were impeding progress (“no, we are trying to make sure we understand what you’ve done when you get bored with us”). I know that new software introductions are disruptive. But that’s what developers (and I’m counting myself here for the sake of that sentence) do. Things break, we tell others, and we fix them. (and some of my other colleagues think WE are the lack of planning ones – you have no idea)

While every application should have a definite lifecycle – you know, and I know, and everyone else knows that in many, many, many environments apps get written, and they live well beyond the developers, the systems people, and everyone else that every had any responsibility for it – and local copies of everything creates a maze of having to upgrade the third-party dependencies all over the place when some script kiddie decides to take advantage of that 2-year old failure to sanity check POST.

Rails developers have to start figuring out that someone beyond them is going to be responsible for inheriting what they’ve done – and they have to start thinking more seriously about dependencies, third-party code, add-ons, and the lifecycle of what they do. It’s like two-bytes for the year value all over again. Seriously people, no amount of “unit tests,” “syntactic sugar,” and vendor kung-foo will ever trump communication and documentation (I don’t mean constantly out-of-date systems analyst documentation – I mean documentation about decisions and why something was done, or why it was added, etc.)

Rails 1.2 Route Changes Are a Pain in the Arse

So, the rails routing changes that have apparently happened in Rails 1.2? Yeah – not a fan.

Thankfully, last night, the night before we plan a production transition to Rails 1.2, we got spidered by crawlers looking for old, vulnerable copies of phpMyAdmin – they hit a systems testing server.

I don’t think I’ve ever said “thankfully” with regard to a spider ever. This is a first.

That generated well over a hundred emails – where Rails oops with:

A ActionController::RoutingError occurred in application#index:

  no route found to match "/whatever" with {:method=>:get}

We were able to find some information about using a catchall route at the bottom of routes.rb:

map.connect '*path', :controller => 'application', :action => 'show404', :requirements => { :path => /.*/ }

With a happy little trees show404 method to show a 404.rhtml and return a 404.

This really isn’t the annoying part, I like the routes idea, I like how easy it is to fix this.

What I don’t like is that between Rails 1.1.6 (or even the “edge rails” for 1.2) and 1.2.x – the routing changed such that requests to URLs that are not handled with a default route (e.g. :controller/:action/:id) result, not in a nice, happy, proper “404 – Not Found” but the “500 Internal Error”

I’m sure that this can all devolve into semantics, but if the URL isn’t handled – return not found, don’t crash.

Upgrading your Rails 1.1 application to Rails 1.2

So, I’m in the process of getting our servers upgraded to Rails 1.2.

( We don’t freeze rails, because contrary to the cult of Rails, freezing is not the way to go in small, controlled shops with multiple applications. If say, you have to upgrade your Rails, NOW NOW NOW – it’s much better to upgrade the system, and not every copy of your application everywhere, but I digress)

That process, of course, involves making sure our applications work with Rails 1.2 and eating my own dog food – this is what a cursory run through of the application that I’m responsible for had me running into:

Deprecations

The biggest issue – which really isn’t an issue, which was cool – seems to be the deprecations coming in Rails 2.0. But for anyone playing along with the home game – deprecations are errors. (maybe I should start a “Cult of the Clean Log”).

The deprecations I had:

• multiple references to @request, @params, @session…
• references to @flash, including a really odd error that resulted from the fact that I had a partial named “_flash.rhtml”. Thankfully I wasn’t the only person that ran into that
• a lot of start_form_tag and end_form_tag references. And while I to this day get a little squicked out by phrases like “syntactic sugar/vinegar” – I do very, very much like the block form for <% form_tag(:action => :blah) do %> ... <% end %>

Aggressive Unloading

In the famous words of Keith Jackson:

“Whoa, Nellie!”

So we are probably doing something incredibly stupid and against every known convention of Ruby and Rails – but, hey, it works. We have an class for application configuration that we use a class variable to store persistent configuration information (with a default set of values, merged with values loaded from a configuration file where appropriate). Prior to now, we’ve happily loaded this up on application startup in environment.rb to load the configuration – and even in development mode, the class stayed loaded throughout the lifetime of the mongrel process.

Well, apparently in development mode in Rails 1.2 – the automagic dependency management has significantly changed (ob. ref. Jonathan Weiss and the RoR weblog. That’s cool, it’s probably what it should be, and what I’m doing in this application probably isn’t “the right thing.” But the AppConfig class would unload, and be reloaded and because the load_config method was only called in environment.rb, it was Oops’ing all over the place. My hack was to load the config within the body of the class, so it would do it when the class was loaded (loaded might not be the right word here). If I continue with this AppConfig thing – what it probably should be doing on accessing the configtable is to have checks that when it’s nil – reload the config. I’ll solve that one later, thankfully for now, it works again.

If any Rails person that actually reads this has any kind of visceral – “why the heck do you guys do that?!?” – reaction – and know a better way of doing this (the whole persistent configuration-defined-at-run-time-not-in-code-or-the-db problem) please do share.

A HUGE “thanks!” to the Rails team for including Dependencies.log_activity = true as part of dependencies.rb. That helped a lot – and helped provide a glimpse into the automagic dependency management of Rails too.