Ruby 1.9.3

Hooray – Ruby 1.9.3 is out.

So inquiring minds (at least my inquiring mind) wanted to know – does it really help with the rake issues related to the ruby require code?

Yes.

$ rvm get head$ rvm install 1.9.3$ rvm use 1.9.3@exrails31 --create$ gem install bundler$ cd project; bundle install

Results of a time rails console with immediate exit with a patched 1.9.2 (it’s a little slower than my previous post due to additional gems we’ve put in our project):

rubydev:learn (ruby-1.9.2@exrails31)$ time rails consoleexitLoading development environment (Rails 3.1.1)>> exit    real	0m13.684suser	0m12.257ssys	0m1.452s

In Ruby 1.9.3:

rubydev:learn (ruby-1.9.3@exrails31)$ time rails consoleexit/Users/jayoung/.rvm/gems/ruby-1.9.3-p0@exrails31/gems/activesupport-3.1.1/lib/active_support/dependencies.rb:240:in `block in require': iconv will be deprecated in the future, use String#encode instead.Loading development environment (Rails 3.1.1)>> exit    real	0m10.803suser	0m10.020ssys	0m0.777s

Here’s a mention of the deprecation warning. – but I’m not sure it’s reported yet (I honestly have a little trouble following the rails issues).

I’m personally still trying to get my app started – going to have to install my own rails_config for a few days until a new gem is released, so I’m not all sure what else might be a problem in 1.9.3 yet. But that will all work out, and I’ll take the three seconds!

[Update: everything’s working in our current under-development app with my own rails_config, and I’m not sure the iconv error is a rails issue as I trace through things]

[Update 2: The deprecation warning is coming from css_parser which is required by inline-style which we have forked in order to provide specific functionality we apparently need to reference an external mobile stylesheet but still convert inline styles.]

[Update 3: if you are landing on this post from search for the iconv error – you’ve got something requiring it in your app – see this stack overflow post]

Speeding up Rake and Rails in Ruby 1.9.2

Our team at work is currently in the beginning stages of a new project, that we started on Ruby 1.9.2 and Rails 3.1.  Most of our previous work was all Ruby 1.8.7 (Ruby Enterprise Edition) and Rails 2, though I have worked on a couple of REE 1.8.7 + Rails 3.0 apps.

I’ve also jumped with both feet (and not in the fun, fancy, parkour sense, in the 37-year old, flat arched, off a perfectly good one-story building onto concrete sense) into testing, and tenuously into TDD – and I think my first comment was “man, these tests are hella slow.”.  Only it’s not the tests that are slow (yet) – it’s rake.  That was my second comment – “what the hell is wrong with rake?”  My co-workers even looked at me a little funny when I brought it up.

As we were getting started last month, Rails 3.1 was still in RC status – and I would spend time browsing the Rails issues keeping track of the progress – and I remember reading this issue about Rails 3.1 boot performance – and reading the comment about the “require issue”.   That led me via Google to this stack overflow post and this rubyonrails core post – which I basically chalked up “core problem, it’ll be fixed one day”

Well that “one day” is now, because I can’t take it anymore.

This is the rails console startup in REE-1.8.7/Rails 2.3.14

rubydev:darmok (ree-1.8.7@extension_production)$ time script/consoleLoading development environment (Rails 2.3.14)>> exit    real	0m5.820suser	0m4.847ssys	0m0.787s

This is the rails console startup in 1.9.2/Rails 3.1

rubydev:knappsack (ruby-1.9.2@exrails31)$ time rails consoleLoading development environment (Rails 3.1.0)>> exit    real	0m18.766suser	0m17.020ssys	0m1.444s

Just a flat out “rake –tasks” takes over 9 seconds in 1.9.2/Rails 3.1!

So thankfully, google, the internet, and other irritated developers to the rescue. Thanks to this blog post from Xavier Shay (via: Peter Cooper at Ruby Inside, that I missed in my feeds the first time because “1.9.3” is not on my radar) – and especially this gist from Todd Fisher – and I have a glimmer of relief.

The quick instructions for the above:

1) grab the patch from the gist

curl https://raw.github.com/gist/1008945/4edd1e1dcc1f0db52d4816843a9d1e6b60661122/ruby-1.9.2p290.patch > load.patch

2) patch ruby

$ rvm cleanup all; rvm install 1.9.2 --patch ./load.patch -n patched

3) copy my gemset(s) over

rvm gemset copy 1.9.2@exrails31 1.9.2-p290-patched@exrails31

4) change my .rvmrc for the project…and voila!

rubydev:knappsack (ruby-1.9.2@exrails31)$ time rails consoleLoading development environment (Rails 3.1.0)>> exit    real	0m9.722suser	0m8.607ssys	0m1.081s

rake –tasks is now 6 seconds instead of 9, and test initialization is correspondingly faster as well.

It’s not REE 1.8.7/Rails 2 – but I’ll take any speedup I can get.

Refactoring my Rails devserver script

So with the last post – after I found out (while writing the post) that with the changes to Passenger 3 standalone – it (understandably) doesn’t support a debugger option – I needed a backup, like say, thin.

So, I refactored rewrote my script

This one will now run your choice of server (where your choice is passenger, thin, or mongrel, but it could be extended to anything. Since passenger doesn’t support –debug/–debugger – it will use one of the others.

I pulled the “–daemonize” option – because that doesn’t make a whole lot of sense for a devserver environment – and I haven’t used it yet since I’ve switched away from my Cocoa-based launcher. If you want another terminal for tailing the development log (passenger will pipe the development log to stdout for you), or running memcached, just open one.

One thing I’ve wanted to do for a while is get away from using “getoptlong” and over to something else for command-line option parsing. There’s where trollop comes in. I like trollop (and love the built-in –help, which I never write with getoptlong) – the only thing I don’t like very much is that because its “opt” method takes a block as the third parameter – so I can’t do something like opt(:setting,’description’,:default => @config[:setting]) – because that @config gets passed as part of a block and gets evaluated inside trollop’s library when it’s creating dynamic methods for the options. This would be fine if I had a set of hardcoded defaults that I wanted to override, but I want to read those from a config file as well as get overrides from the command line. That’s the reason for all the local variable assignments for the defaults. I’m sure there’s a better way of handling that, I’m just not sure what.

Update: 2011/01/07 The script has become its own gem

Creating a Rails Development Environment on Mac OS X

I’ve been writing Ruby on Rails apps for several years now, and both at my day job, and my contract work, I’ve been utilizing a modified version of the Locomotive Project from Ryan Raaum. I rebranded it as “MiCodaLoco” for our use, and modified it to be a little more modular for it’s startup environment, moving the MacPorts distribution it used to a central folder and out of the “bundles” it used for each Ruby and RubyGems environments.

It worked out quite nicely for several years. But it was a complete pain to rebuild ruby and rails versions, because I never took the extra steps to automate the bundle building process. I always had grand plans to create a “debug” button as well to launch mongrel (or thin, about the only change I’ve made in the last year has been to start thin instead of mongrel for one of our apps that ran into a problem with Rails 2.3.8 and/or Rack changes and cookies and mongrel) in debugging mode for ruby-debug.

I had poked my way around the Objective-C app, and I understood where to make modifications I needed, but I didn’t really understand how the app was put together (and parts of it were pre-Interface Builder and had code created interface that I still don’t totally grok, even after a week long iPhone training that had enough Obj-C in it to make me dangerous).

Best-laid plans being what they are, I never did update it, and it was time to move on. With a Rails 3 and possibly a Ruby 1.9 update in the next six months to year for all these apps – I wanted to spend more time on that than learning Cocoa. So I needed something new.

Enter Homebrew, RVM, and Passenger standalone.

Homebrew

So, I’ve been a Fink, then DarwinPorts MacPorts user forever, and would recommend it to anyone to get open source packages (like wget, or wireshark) that don’t come already with OS X. But sometimes suggesting it to your colleagues just to install something to facilitate something else doesn’t always go over well.

Enter Homebrew. Because Homebrew uses a lot of of what’s already there in OS X instead of installing it’s own versions, it’s faster to get the packages you need (with the risk that something is broken with an OS update, but I haven’t seen something like that in a while with 10.5/10.6 on Intel). And in our case, we only need the mysql client libraries for the mysql gem, and the imagemagick libraries for the rmagick. And wget. Because everyone needs wget.

It’s easy to get started, just follow the instructions (The system admin in me recommends you look at the executed ruby script first – but I know you won’t).

Oh, you need Xcode. You do have Xcode installed right? Even the designers need it for the iPhone emulator. Trust me.

In our case we needed to

brew install mysql
  • which installs mysql server, but we use MAMP for that, so it’s just for the libraries (in theory, you probably can just link the mysql gem against the MAMP libraries if you use MAMP. But I don’t want some MAMP upgrade where I put it in a different path wreak havoc on that). We also needed to

    brew install imagemagick

. And if you are testing your caching code and use memcache,

brew install memcached

You might need something else installed, but we haven’t as of yet.

RVM

RVM is everything I ever wanted doing multiple Locomotive/MiCodaLoco bundles to be, and then some, with the added bonus that somebody else does it.

RVM lets you run multiple rubies and multiple gemsets per ruby installation, making it really convenient to not only switch between them for testing and dev purposes, but also just to have an effective development environment.

RVM’s documentation is very comprehensive, though it can be a little tricky to get started. Here’s what we needed (after installing RVM):

rvm install ree                           # I use ruby enterprise edition - version 1.8.7 on our, ree is a shortcut string for this version - see "rvm strings" for morervm gemset create busterleague # creates a gemset for a person project of minervm --default ree@busterleague  # sets my default ruby to ree and the default gemset to busterleaguervm use ree@busterleague        # sets up the session to use ree and the busterleague gemsetgem install rails --version=2.3.10  # still at rails 2gem install passenger               # passenger v3gem install ...                           # rest of gems

As a convenience – because you like will end up shifting between rubies – or at least gemsets – create some bash aliases to make that easier (Locomotive/MiCodaLoco had a “open terminal” command for each project that automatically put one into the right bundle environment and the working directory for the project, these aliases replace that as well).

Here’s mine for my work, contract, and personal projects (I put my working directories in a “dev” subdirectory to my home directory)

alias rubydevprompt='PS1="rubydev:W ($(~/.rvm/bin/rvm-prompt i v g))$ "'alias smallwonder='rubydevprompt && rvm use ree@trixietracker && cd ~/dev/tt/smallwonder'alias darmok='rubydevprompt && rvm use ree@extension_production && cd ~/dev/extension/darmok'alias dega='rubydevprompt && rvm use ree@extension_production&& cd ~/dev/extension/dega'alias busterleague='rubydevprompt && rvm use ree@busterleague && cd ~/dev/personal/dmbdraft'

I can then switch between them at ease, add more to do things like start passenger (more on that below), or to use development gemsets, like rails 3 (and even add git commands to make sure I’m in the right branch (ala rails 3)

rvm-prompt has a nice documentation page

Passenger

Finally, I use Passenger in standalone mode running on an arbitrary port to deliver the dev versions of our apps (actually I use passenger everywhere, currently running under Apache, it’s made my system admin life much easier than other rails environments).  After installing passenger, just run the “passenger start” command, and it will self-install and configure a version of nginx that it uses to do this.

As a convenience, I’ve created a ruby script that starts passenger with a set of default parameters – which can be overridden by a yaml configuration file – or from the command line. I called this “script/devserver” to distinguish it from the rails webrick “script/server” command.

You can see a version of that devserver script at github

What I probably need to do is go back and have a debugging option that would start webrick, mongrel, or thin in order to use it with ruby-debug (since Passenger 3 isn’t really setup to support ruby-debug)

[Update] You probably want this link to the devserver scripthere’s why.

Heading into Rails TOOOOOO-TOOOOOO-TOOOOOO!

When I played little league baseball we had announcers for the game, one of which was some nice old guy in a ramshackle shelter behind home plate that was the “booth” – and every time there’d be two outs, two strikes and two balls – he’d croon “TOOOOO-TOOOOO-TOOOO”. Which of course, has little to do with this post other than version numbers.

Rails 2.2 was just released – so of course, I’m taking the most critical application we have (well, maybe the second critical, and that’s really only because other apps depend on it, it’s not critical in terms of features) – and upgrading it first, because, well, that’s how system administrators roll.

The release notes are great. Seriously. Really great. So great I shouldn’t even be writing this post. Which I am anyway – because here are some of the highlights of what I had to change to make my app work.

NoMethodError for Association Methods

Getting exceptions when you go to Rails 2.2.2 that don’t say anything more than NoMethodError when you know good and damn well the method that it’s saying no method on exists? Yeah, me too. And I bet it’s a method in an associated model. And if it is, you probably should be ashamed of yourself.

Rails 2.2 now enforces privacy on private methods called through associations. So in my case, I had two issues, 1) I was calling “update” on an associated model in some code I blindly copied and pasted a long time ago, and 2) I have a few of my own SQL queries that I’m not entirely sure how to do using Rails associations and named scopes, and I was cheating by calling self.connection.sanitize_limit to take advantage of Rails’ own function for cleaning up provided “LIMIT” params. And sanitize_limit, like the instance method update is private.

Update Rails Footnotes

If you use Rails footnotes in development mode – you’ll want to update for this change for Rails 2.2 compatibility.

Aside… Piston 1.9.5

A great way to stay current with Rails plugins is to use Piston – which has a new 1.9.5 release. You can build your own.

“quoted_table_name” and Has Many Polymorphs

If you start getting some error about undefined method quoted_table_name’` and you use has_many_polymorphs – you’ll want this change.

I don’t use the plugin, I use the gem. So I built my own has_many_polymorphs 2.12.1 gem – by doing: ` gem install echoe git clone git://github.com/fauna/has_many_polymorphs.git cd has_many_polymorphs `

edit CHANGELOG with a “v2.12.1 line” (e.g. v2.12.1. Cloned GitHub project and rebuilt gem for our nefarious purposes.)

rake manifest rake package rake install

add_joins! and Has Many Polymorphs (or anything else for that matter)

HMP includes a ‘tagged_with’ method for finding collections ‘tagged_with’ a set of tags. I use a heavily modified version of that. The method supports custom scopes, in theory. (well, probably more than theory, I’ve just never tried it). While, I don’t have any scopes on models that call my functions – I still had some of the private ActiveRecord method calls in mine – particularly add_joins!.

Well, this change changed the params for the method to make sure that the scoped joins were merged, and not overwritten – which changes calls into it. If you are calling it with your own options use options[:joins]. My code doesn’t use the scopes in combination with my tagged_with method, so I just pulled them.

And…. thankfully that’s it

Other than cleaning up deprecations like number_with_precision now preferring (number, :precision => myprecision) instead of (number, myprecision) – and ActiveRecord::Base.verification_timeout no longer valid, and mb_chars being preferred over chars – we seem to be good to go for Rails 2.2. I still need to test some crons, but I imagine that our app will go production as 2.2 shortly.

TOOOOOO-TOOOOOO-TOOOOOO!

A Story of a Bug

It started, as it should, with the belief that it was my own bug.

I’ve been working for the last few days to generate daily summaries of the activity flowing through our tools. It’s nothing earth shattering, but it’s been a stepping stone to understand a little bit more about the Rails framework – and gave me the chance to begin experimenting with the Google Visualization API. Toward the latter part of the week, there was something a little odd with the “total valid” numbers with the daily account creations – I had made data changes to make sure I had some idea when accounts were vouched for and when they had been retired – so I naturally assumed it was something I did. I even went back and modified the model to make it more consitent with it’s peer models. And kept running the script that produced the daily stats in the Einstein-esque insanity of the doing the same thing twice and expecting different results. After about a dozen combinations of DATE(date_column) comparisons – I went to google, because I knew by then I was either going crazy or this was a legitimate “it’s not my problem” bug.

Which led me to this mysql bug. Reported July 19, 2007. Apparently introduced in MySQL 5.0.42 (May 2007) when the DATE comparison changed from strings to ints. Fixed within two days as part of 5.0.48 (Released August 2007).

But guess which mysql package Red Hat EL 5 (well, RHEL5 update 2) provides? – right, in between. It’s MySQL 5.0.45. And the forthcoming RHEL5 update 3 release doesn’t update MySQL either.

Development and System Administration is a weird, weird world. I use RHEL, not for support (I’m not even sure we have support with the University contract), but to have some degree of patch level stability that’s slightly longer than the fedora releases (and at the time I went to RHEL, Fedora was still dealing with it’s Red Hat transition) – but that stability comes with the price of things like this. I already use my own Ruby to get beyond the base install, but configure, make, make install for one piece of core software is a little different than dealing with it (or MySQL-supplied RPMs) for other software.

I’m glad the open source world gives me that choice, but open source + my labor + thousands of moving parts does give provide the reality that even when a bug is fixed two days later, in the open, patchable by everyone – that sometimes you can find yourself over a year later modifying your own DATE queries so that they don’t include nulls.

So that’s the overall summary of the post I guess – part of it to go into google that if you are getting odd MySQL DATE function results on MySQL 5.0.45 on Red Hat Enterprise Linux (RHEL 5) – it’s a bug. And it’s fixed. But not included in RHEL 5. And if you aren’t getting odd results with DATE comparisons – you probably don’t know that you are.

And maybe one part as a lament to that inevitable ongoing intersection of thousands of moving parts in every environment, not the least of which ours. And you trade off replacing mysql on multiple servers and just turn nulls into zeros (which then breaks your signup form that desparately needs an overhaul) – well, because it seemed to make sense at the time.

And people say you don’t use probability after college.

Creating your own Gem Server, redux

One of the most popular landing pages for rambleon.org (which isn’t saying much) – is a post I put together in April, 2007 about running your own gem server.

Unfortunately it’s outdated. Set let me revisit it for those landing on the old page and looking to put together their own gem server. The nice thing is, gem has (almost) all the tools you need.

Setting up your server

  1. You need a web server. (yes I know you can run gem_server, but get a real one). You are on your own for that one. You also need a rubygems install on that box. You are also on your own for bootstrapping rubygems on that box and any other ones.
  2. Decide where you will put your gems (say in a “mycoolgems” directory off the docroot for your webserver)
  3. $ mkdir [docroot]/mycoolgems/gems
  4. Copy your .gem files that you want to host to $ [docroot]/mycoolgems/gems
  5. The gem suite of commands includes a generate_index command to generate a yaml-based index of your gems, and other supporting files. See gem help generate_index for more information

Pointing your systems to your own server

This used to be a complete PITA that involved rebuilding the sources gem. No more! After you install rubygems – just make sure to do a:

gem sources --remove http://gems.rubyforge.org

and a

gem sources --add http://yourwaycool.gem.source

That’s All Folks (probably)

Voila! You just managed to point your server to your own gem server! Install away.

It’s good to keep one box pointed to http://gems.rubyforge.org – and take advantage of the new “gem outdated” command to keep track of changes in your installed gems that have been deployed to rubyforge.

Your own gem server and Ruby 1.8.5

A while back, I wrote about creating your own gem server – which basically consisted of custom building a sources gem. Mainly because gem sources --remove at the time wouldn’t let you remove rubyforge from the sources list.

Thankfully, gem sources --remove now works great. And with RubyGems 0.9.5 – they’ve included a new gem generate_index command that will generate the indexes for you. So now – if you want to create your own gem server, you can run the web server software of your choice and use the gem generate_indixe -d [docroot]/mycoolgems command to build the indexes.

(If you do update the rubygems-update gem on your “client” systems, make sure to do a gem sources --add http://yourgemserver and a gem sources --remove http://gems.rubyforge.org from your sources. Or continue building your own sources gem.)

Unfortunately, through no fault of the RubyGems devs, this might not work on a system running Ruby 1.8.5 – like, say, Red Hat Enterprise Linux, version 5. The fault lies in Ruby itself. gem generate_index builds its index in /tmp – and uses FileUtils to move it out of /tmp and into the place you specified as the target directory (or ‘.’). If your servers are like mine, /tmp and your web server directories are on different partitions, and unfortunately FileUtils in Ruby 1.8.5 has a bug moving directories across partitions.

The bug was fixed, I think with this revision in March 2007 – and it’s in Ruby 1.8.6 (at least as of p110/111) – but it’s not in Red Hat’s 1.8.5 packages – and I’m not sure it qualifies as a security bug, so I doubt it will get backported.

Anyway, I went the configure/make/make install route (package purists, feel free to cringe, but I don’t want to make RPM’s and this is core enough for me to stay on top of). Your mileage may vary.

Don’t Do That

So… maybe you are coding up your totally way rad awesome application in Rails – and you are thinking to yourself.

“Self, I really would like to set my own created_at and updated_at timestamps. Look – there’s even a way to do that in the Rails documentation

class Feed < ActiveRecord::Base self.record_timestamps = false # ... end

At this point you need to back away from the keyboard. Quickly. If you don’t, pretty soon, somewhere in your application – you are going to run into this error:

Mysql::Error: Column 'created_at' cannot be null

Or ALL KINDS OF OTHER FUN SIDE EFFECTS (FUN is actually a euphemism here for various four letter words)

See, record_timestamps is a class variable for ActiveRecord::Base created with the rails :cattr_accessor – maybe the self.record_timestamps should have tipped us all off – maybe not (there’s also a class_inheritable_accessor – I’m not sure where all that gets used though)

Even experienced developers not all that fluent in ruby minutiae (I think class variables count as minutiae) cut-and-paste first and figure out how it works second (yeah, don’t do that either).

So – anyway – once you change record_timestamps once – you change it for all descendants of ActiveRecord::Base

There’s a bit of discussion this on a separate, but related problem at Evan Weaver’s blog (pay special attention to that threading issue for those playing along with the home game). And of course, your friendly neighborhood reminder of what happens with class variables at Nic Williams’ blog (I recommend reading that twice and breaking out the home game version of irb)

So the moral of the story? self.record_timestamps – Don’t Do That.

p.s. Production of this blog entry was made possible through various grants and assessments, and with some moans, groans, sighs, and “what tha–?” from my colleagues James Robinson and Aaron Hundley (doesn’t have a blog, he needs to get with the program) 🙂

p.p.s edited to change “you chance it for all descendants…” to “you change it for all descendants” – I think the first one is quite apropos however

p.s. it’s never really about a single post

so I went into snarky overdrive with my vendor dependencies post. it wasn’t quite as funny as last year when I went off on the people that can’t unsubscribe from lists either. Nor was it as funny as some other commentary on Rails I’ve snarkily made

I really do love the err.the.blog guys – – they have the best rails blog – bar none – that I’ve ever found – I even have their toolbox post burned into my retina I think.

They really know their stuff. Rails needs devs like this – and they do a great service educating other folks about the framework.

But I don’t agree (obviously) with packaging up all the dependencies with an application. I get all the reasons for doing so. It just sets a really dangerous precedent for the people that are going to take it as the gospel and never think about the ramifications of what packaging up everything with your application means. (like simple things – remind me to tell the inode story sometime. 20 capistrano delivered copies of edge rails might not kill your storage but it dang sure can eat some inodes)

But hell, you can’t really trust the system administrators to get it right either about not breaking dozens of rails apps that they don’t have a clue about. I, um, er, have known some sysadmins to do that (more than once even).

p.p.s Oh, man, I forgot about the sponsorship link. This post sponsored by the Static Linking Historical Society. And support also comes from Microsoft Corporation. Proud facilitators of DLL Hell for all the static linkers that decided to go dynamic, but distributed their own libraries.