Refactoring my Rails devserver script

So with the last post – after I found out (while writing the post) that with the changes to Passenger 3 standalone – it (understandably) doesn’t support a debugger option – I needed a backup, like say, thin.

So, I refactored rewrote my script

This one will now run your choice of server (where your choice is passenger, thin, or mongrel, but it could be extended to anything. Since passenger doesn’t support –debug/–debugger – it will use one of the others.

I pulled the “–daemonize” option – because that doesn’t make a whole lot of sense for a devserver environment – and I haven’t used it yet since I’ve switched away from my Cocoa-based launcher. If you want another terminal for tailing the development log (passenger will pipe the development log to stdout for you), or running memcached, just open one.

One thing I’ve wanted to do for a while is get away from using “getoptlong” and over to something else for command-line option parsing. There’s where trollop comes in. I like trollop (and love the built-in –help, which I never write with getoptlong) – the only thing I don’t like very much is that because its “opt” method takes a block as the third parameter – so I can’t do something like opt(:setting,’description’,:default => @config[:setting]) – because that @config gets passed as part of a block and gets evaluated inside trollop’s library when it’s creating dynamic methods for the options. This would be fine if I had a set of hardcoded defaults that I wanted to override, but I want to read those from a config file as well as get overrides from the command line. That’s the reason for all the local variable assignments for the defaults. I’m sure there’s a better way of handling that, I’m just not sure what.

Update: 2011/01/07 The script has become its own gem

Creating a Rails Development Environment on Mac OS X

I’ve been writing Ruby on Rails apps for several years now, and both at my day job, and my contract work, I’ve been utilizing a modified version of the Locomotive Project from Ryan Raaum. I rebranded it as “MiCodaLoco” for our use, and modified it to be a little more modular for it’s startup environment, moving the MacPorts distribution it used to a central folder and out of the “bundles” it used for each Ruby and RubyGems environments.

It worked out quite nicely for several years. But it was a complete pain to rebuild ruby and rails versions, because I never took the extra steps to automate the bundle building process. I always had grand plans to create a “debug” button as well to launch mongrel (or thin, about the only change I’ve made in the last year has been to start thin instead of mongrel for one of our apps that ran into a problem with Rails 2.3.8 and/or Rack changes and cookies and mongrel) in debugging mode for ruby-debug.

I had poked my way around the Objective-C app, and I understood where to make modifications I needed, but I didn’t really understand how the app was put together (and parts of it were pre-Interface Builder and had code created interface that I still don’t totally grok, even after a week long iPhone training that had enough Obj-C in it to make me dangerous).

Best-laid plans being what they are, I never did update it, and it was time to move on. With a Rails 3 and possibly a Ruby 1.9 update in the next six months to year for all these apps – I wanted to spend more time on that than learning Cocoa. So I needed something new.

Enter Homebrew, RVM, and Passenger standalone.

Homebrew

So, I’ve been a Fink, then DarwinPorts MacPorts user forever, and would recommend it to anyone to get open source packages (like wget, or wireshark) that don’t come already with OS X. But sometimes suggesting it to your colleagues just to install something to facilitate something else doesn’t always go over well.

Enter Homebrew. Because Homebrew uses a lot of of what’s already there in OS X instead of installing it’s own versions, it’s faster to get the packages you need (with the risk that something is broken with an OS update, but I haven’t seen something like that in a while with 10.5/10.6 on Intel). And in our case, we only need the mysql client libraries for the mysql gem, and the imagemagick libraries for the rmagick. And wget. Because everyone needs wget.

It’s easy to get started, just follow the instructions (The system admin in me recommends you look at the executed ruby script first – but I know you won’t).

Oh, you need Xcode. You do have Xcode installed right? Even the designers need it for the iPhone emulator. Trust me.

In our case we needed to

brew install mysql
  • which installs mysql server, but we use MAMP for that, so it’s just for the libraries (in theory, you probably can just link the mysql gem against the MAMP libraries if you use MAMP. But I don’t want some MAMP upgrade where I put it in a different path wreak havoc on that). We also needed to

    brew install imagemagick

. And if you are testing your caching code and use memcache,

brew install memcached

You might need something else installed, but we haven’t as of yet.

RVM

RVM is everything I ever wanted doing multiple Locomotive/MiCodaLoco bundles to be, and then some, with the added bonus that somebody else does it.

RVM lets you run multiple rubies and multiple gemsets per ruby installation, making it really convenient to not only switch between them for testing and dev purposes, but also just to have an effective development environment.

RVM’s documentation is very comprehensive, though it can be a little tricky to get started. Here’s what we needed (after installing RVM):

rvm install ree                           # I use ruby enterprise edition - version 1.8.7 on our, ree is a shortcut string for this version - see "rvm strings" for morervm gemset create busterleague # creates a gemset for a person project of minervm --default ree@busterleague  # sets my default ruby to ree and the default gemset to busterleaguervm use ree@busterleague        # sets up the session to use ree and the busterleague gemsetgem install rails --version=2.3.10  # still at rails 2gem install passenger               # passenger v3gem install ...                           # rest of gems

As a convenience – because you like will end up shifting between rubies – or at least gemsets – create some bash aliases to make that easier (Locomotive/MiCodaLoco had a “open terminal” command for each project that automatically put one into the right bundle environment and the working directory for the project, these aliases replace that as well).

Here’s mine for my work, contract, and personal projects (I put my working directories in a “dev” subdirectory to my home directory)

alias rubydevprompt='PS1="rubydev:W ($(~/.rvm/bin/rvm-prompt i v g))$ "'alias smallwonder='rubydevprompt && rvm use ree@trixietracker && cd ~/dev/tt/smallwonder'alias darmok='rubydevprompt && rvm use ree@extension_production && cd ~/dev/extension/darmok'alias dega='rubydevprompt && rvm use ree@extension_production&& cd ~/dev/extension/dega'alias busterleague='rubydevprompt && rvm use ree@busterleague && cd ~/dev/personal/dmbdraft'

I can then switch between them at ease, add more to do things like start passenger (more on that below), or to use development gemsets, like rails 3 (and even add git commands to make sure I’m in the right branch (ala rails 3)

rvm-prompt has a nice documentation page

Passenger

Finally, I use Passenger in standalone mode running on an arbitrary port to deliver the dev versions of our apps (actually I use passenger everywhere, currently running under Apache, it’s made my system admin life much easier than other rails environments).  After installing passenger, just run the “passenger start” command, and it will self-install and configure a version of nginx that it uses to do this.

As a convenience, I’ve created a ruby script that starts passenger with a set of default parameters – which can be overridden by a yaml configuration file – or from the command line. I called this “script/devserver” to distinguish it from the rails webrick “script/server” command.

You can see a version of that devserver script at github

What I probably need to do is go back and have a debugging option that would start webrick, mongrel, or thin in order to use it with ruby-debug (since Passenger 3 isn’t really setup to support ruby-debug)

[Update] You probably want this link to the devserver scripthere’s why.

How Project Names Happen

So, being a developer and the systems manager hath its privileges. I get to wade through a convoluted process to come up with names of projects – and I get to apply them. Because, well, I can.

I’ve already rambled on more than once in the blog about desktop and server names, but I’ve never really talked much about coding projects.

Really, mainly that’s because the names are usually boring. And also that I/we thankfully don’t start many coding projects. Starting an application/project is not something to be taken lightly – and well, we already have what is likely too many projects anyway.

However, today I am taking the baby steps into a new rails project. Currently – our “Identity” project is serving as a hub of activity tracking. For a whole host of reasons, I’m thinking of separating the actual data collection and storage and “generation” of that data into a separate application (but leaving some of the views into that data within the Identity tool).

So, what to name it? We’ve named projects after Greek Words, the protagonist from a movie about a talking plant with psychological issues, and boring things like “project function.”

But this needed different. Our activity views are part of the idea of [freeranging][5] or [“work wide open”][6]. And after spending a few minutes in Google and Wikipedia trying to come up with names that were related to “free-range” and getting no further than pasture management strategies – I needed a new tack.

Our activity views are both a reporting/monitoring mechanism – but they are also an experiment in [workstreaming][7].

Hmmm…. streams. The [first free-associated “stream” idea that I had][8] was clearly NOT going to do.

So what’s the next most famous “streaming” idea? That’s right…

Dr. Egon Spengler: There’s something very important I forgot to tell you. Dr. Peter Venkman: What? Dr. Egon Spengler: Don’t cross the streams. Dr. Peter Venkman: Why? Dr. Egon Spengler: It would be bad. Dr. Peter Venkman: I’m a little fuzzy on the whole “good/bad” thing here. What do you mean, “bad”? Dr. Egon Spengler: Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light. Dr. Ray Stantz: Total protonic reversal! Dr. Peter Venkman: That’s bad. Okay. All right, important safety tip. Thanks, Egon.

Hence, the new project for activity tracking, Google visualization, json data feeds, etc.

[Protonic Reversal][9]

And except for the ones brought to you by the stork, and the tooth fairy – is how project names happen. I mean, really, who you going to call?

(okay, sorry, couldn’t resist)

[5]:
[6]: http://rambleon.org/2008/10/18/work-wide-open/ [7]: http://blog.k1v1n.com/2008/11/freerange-tools-workstreaming.html [8]: http://en.wikipedia.org/wiki/Islands_in_the_Stream [9]: http://justcode.extension.org/projects/show/activity

It’s about the details

My default feed yesterday switch back to RSS 2 from Atom (and a huge thanks to Sam for pointing this out) because of a upgrade to WordPress 2.3.1.

It was another reminder of how things fail because of a cascade of missed details.

The immediate detail, was that I completely forgot my own quick hack to WordPress 2.3 to force the Atom default. That’s the danger in making “one-off” changes to software that you use. Quick one-offs don’t become part of one’s standard workflow, they are too easily missed later, and upgrades will usually wipe out one’s changes. The old axiom applies here – “just because you can, doesn’t mean you do, because it will bite you in the rear later”

If I’m going to make changes like this, I probably should do what I do with MediaWiki at work, and pull things from Subversion and merge them into my own trunk. But that’s just flat out silly for a few lines of changed code.

The second mistake is that I should have – at the least – taken Mark Pilgrim’s advice and put together the two liner plugin to remove the rss2 action.

That’s just complete laziness on my part. I had it working and I didn’t bother it again – I wanted to spend more time investigating how the whole “actions” thing works in WordPress, and had grand dreams of extending that idea into a plugin that could be configured from WordPress itself. Then I decided I’d rather play with the dog.

There’s this continuum of solving problems that starts with hacks and goes to solutions. And the problem is, solutions take a lot of work. So the happy medium is somewhere in between. My change was hack that bit me in the rear later. Mark’s change is something that will likely work across upgrades. And that’s what I should have done right then, and I absolutely knew better than to continue with my own podunk modification. There’s another axiom there “just because it works, doesn’t mean it’s right, especially if isn’t going to work later”

So, the default feed format changing is completely my fault. If I care, and I do, I have to make sure that the things I really care about keep working. There’s an ongoing maintenance cost to my commitment.

But the third detail in this, not that it excuses my mistakes in not being diligent about my details, is that this really should be part of the core of WordPress. It really doesn’t make sense that the Atom feeds are there now, but the software is so bent on making RSS2 the default and not letting folks change it easily. There’s really no excuse going forward, especially given the patches available.

(and yes, it works just fine, as my own patch -p0 and plugin drop-in can attest to – that I’ll have to do again for 2.3.2 – Mark’s plugin is there as a backup this time 🙂

I’ve been on the WordPress dev’s case about this a long time. Always in this blog, and I’ve never put code where my mouth is. I should have put up or shut up. Sam and Mark, and dozens of others thankfully have, with WordPress, and in dozens of other places that matter.

But even if I should have shut up already – there is something core that I’ll repeat here. I know that the users don’t care. I know that it doesn’t make a dang bit of difference to Aunt Millie whether things are RSS 2.0 or Atom. But RSS 2.0 is the IE 6 of data interchange. WordPress’s continued traction on making it easier to allow the software’s users to make the Atom feed the default is like continuing to excuse IE 6. Sure, we have to put up with it, but we don’t have to keep making it hard to choose other things.

As developers, we have the ultimate responsibility to make sure that we are laying the foundation for how we ship data around – now, and 5 years from now. I sure hope that the WordPress core devs run with the patch this time. It really does matter. You’ll still have a successful business model without it, but don’t make us continue routing around the damage.

[ed. note – I read this again after Sam added his second paragraph about the patch and my own post was a little unclear, so it was edited slightly for my actual intent. While I don’t think that RSS2 should be the default – I get not making the change right now, just stop making it hard to make a choice with the feed format]

Peeling the Onion

And no, I don’t mean The Onion – which would have been far more entertaining.

Through Joi Ito’s blog I have recently become aware of the phrase “Yak Shaving.” Joi wrote about it in 2005, here’s WikiPedia’s take – and here’s an etymology from Alexandra Samuel.

When I first read Joi’s blog. I took Yak Shaving to mean a pointless activity (Joi writes in a bit more layered fashion than most folks). It’s part that of course (read Alexandra’s post). But it’s more about good problem solving. Especially when you go and read the entry in the Jargon File

Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you’re working on.

One of the things that I’ve spent my entire career doing is looking for the root causes of problems. Yes, just like every other system administrator/developer, there are times that I’ll defer the problem to another day (to this day, I’m still avoiding a mime types/magic file problem with Red Hat and PHP and MediaWiki that I’ve spent too much time on already). But I recognized a long time ago that digging into something, rather than stopping when the problem was mitigated was going to be much better for everyone. I spent a lot of long nights doing this early on, and still do occasionally – and I’m thankful for some early examples from mentors that encouraged this. It’s made me a much, much better troubleshooter over the years for doing so.

The latest peeling the onion activity came last thursday. I arrived at work, with every intent of taking the example OpenID rails server in the ruby-openid distribution and beginning to pull it in our internal authentication tool. Doing that is very much a “Yak Shaving” activity. There’s some other more pressing problems, but doing OpenID in the internal application solves part of 2 or 3 other problems.

Well that fell by the wayside by mid-morning. We have a preview version of our public website. Most days I’m actually surprised that it works given our Rube-Goldberg esque technical framework for taking mediawiki pages and getting them to the public web site. But it’s been a real benefit internally to have the preview site. Making it happen also made the public application more flexible too.

Well, mid-morning thursday, there was a report that content wasn’t updating on the preview site. At first it was thought this might be a by-product of the previous day’s activity – pulling out a rewrite rule that was eating encoded “?” characters (i.e. %3F) in MediaWiki page titles and causing 404’s by the time those URL links made it into a Flash application and a Rails application. In the process of fixing that, we actually fixed another problem where the source URL for each page in our atom update feed was wrong.

Making that URL correct was what broke things the next day. It turns out that Problem #1 was that the process that pulled in the atom feed keyed in on the URL as the unique identifier for the content (as a fake unique identifier actually, it wasn’t enforced by MySQL). Since the URLs changed when they were fixed – voila! duplicate content – and of course the Find was doing a Find first – and pulling the original, un-updated article.

There was a whole lot of scratching our heads (okay, there was a lot of cursing) about that unique key. The URLs in question are “internal” and pretty much guaranteed to change. Keying off that certainly wasn’t great design. I guess it goes back to that original issue – it solved the problem for a day, but no one gave any future thought as to what it would impact.

So we needed to key off the page title. Well, the page titles weren’t unique either. Which was also a head scratching/cursing problem. MediaWiki page titles are unique in a namespace, and our find functions assume they’ll be unique as imported, but that uniqueness was not enforced.

Well, MySQL folks can guess what happened next. We’ve never actually ever dealt with the collation issues with our MySQL server (there’s a lot we haven’t dealt with with our MySQL server – but that’s another post for another day).

For good or for bad, I really didn’t understand why our collation was set to “latin1_swedish_ci” – and thought that I had made a major mistake setting up the server defaults in the first place, that no dev ever caught when thinking about their schemas. I was pretty relieved to find out that that was just the default for MySQL in the first place.

James’ absolutely groaner of a quote?

Well at least we didn’t bork it up

Well, MediaWiki titles are case sensitive, and it made sense for that column to be case sensitive too – so in went the migration. This actually gave the additional benefit that searches for articles titles would actually be accurate now (even though we have some content that differs only in case that needs to be fixed).

execute "DROP INDEX title_idx ON wiki_articles" execute "alter table wiki_articles change title title text character set latin1 collate latin1_general_cs null default null" execute "alter table wiki_articles add unique title_idx (title(255))"

(p.s. select somecolumn,count(*) as n from table group by somecolumn having n > 1 is a wonderful tool to stick in the tool belt)

After all this is done, we had to import the content again. It’s about 25MB of an atom file – 5,000+ pages of content dating back to last september. Our standard processes of trying to pull this data in with a HTTP GET takes too long to run with the HTTP timeouts in the libraries we use – so a long time ago I modified our code to read the data from a file when needed.

Well, when a contractor modified the code to stop using the FeedTools library and just do some simplified parsing for our internal purposes they took out the “read from file” functionality and didn’t replace it. Which generated some more head scratching and cursing. So that had to go back in to get all the content back into and corrected.

A simple problem of content not being updated highlighted 4 different problems: wrong key for the content, no unique enforcement for any keys, wrong collation on the column, and data import from files missing.

We could have stopped early on by just killing the duplicated content with the wrong URL, updating it, and reimporting the latest changes. But we didn’t. The application we fixed didn’t matter for public use – but our fixes prevented some future problems.

I guess we shaved a few yaks that day. And proved yet again how important it is to get to the root of problems. And how painful it is later when you have to go back in behind yourself and others because it wasn’t done originally.

Designing Content for the Web

Shelly Powers resurrected a post a few days ago on javascript “widgets” (without much thought you could extend this to any blob of javascript doing http requests and fun little rendering things with local and server data without refreshing the page).

It’s a long piece, that I think boils down to “web designers should use them responsibly”

I’m going to extend this in another way. Shelly said something in particular that I want to point out:

The same person who wrote the comment about widgets also mentioned how browsers load top to bottom, left to right. It’s been a long established policy to make your web pages more accessible to screen readers by putting the changing material first in the page layout, and then the more static elements. In a weblog, this means putting the content first, and then the sidebars. The bars may appear to the left, but in the actual physical layout design, they should physically exist after the content, so that the content is always first.

You can’t design content for the web unless you know how the web works. I mean at least at a high level – about the fact that browser software uses HTTP to request data, web server(s) return that data, and the browser software is responsible for beginning to parse that data as soon as it arrives back from the web server(s), and follow its instructions – either make more requests, start rendering text and graphics in the data, or start running code embedded in the data in whatever code parser/compiler the browser supports. And add on top of that how other software (xml clients/”feed readers”, embedded browsers, screen readers, the code that browsers execute, other web servers even) also will request your data and follow its instructions.

Well, I mean you really can design content for the web without knowing any of this – but if and only if you trust the guidance of the people that do.

[Updated: The more I read and re-read my original post, the more I realized that it was one big long “duh” So I just decided to edit the thing and get to the heart of the point I wanted to make.]

wp-atom and wpLicense

Thanks to a couple of line changes in the wpLicense plugin – namely after line 384 – adding:

add_action('atom10_head', 'cc_atom_head');

and because of what is likely an output buffering issue somewhere – I had to change line 84 from:

link rel="license" type="text/html" href="'.licenseUri().'"

to:

link rel="license" type="text/html" href="'.get_option('cc_content_license_uri').'"

(yes, I need to debug this – it’s kind of a hack)

I now have the wpLicense plugin working with Benjamin Smedberg’s atom 1.0 plugin

All this so that Conversations with Plastic Dinosaurs is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 license.

Quote of the Day

From me in email:

run-of-the-mill developers fix things great developers fix things and share those fixes with their colleagues

I don’t do this as often as I should, but by damn I’m going to get better at it. There’s absolutely no excuse for making a fix, and not being aware that your fix might be useful for other tools that your colleagues work on.

Wow They Did It

This isn’t quite on the scale of Linden Labs opening the sim code (actually, I’d be far more in favor of them packaging it up so that you could run your own sim) – but I’ll be damned – they open-sourced the client

That will certainly reverberate across the 2nd life and other 3D virtual world blogs today.