Let’s Do Lunch

I was pulling some of the activity data out of one of our applications – and looking to do some “live activity” and “activity today” views. Our data is stored with UTC timestamps, but we almost exclusively a U.S. shop – and for that matter, most likely a U.S. Eastern shop – so I needed to pull the working curve back into the U.S. time zone.

I got a little curious about “working hours” and decided to run a working hour query (relative to EST) – and rather amused to see the ~20% off morning/afternoon peak during the noon hour:

Yep, U.S. Eastern Time, working hour, non freerange centric. I was smug, “I can’t wait to see mine” I said to myself.

Uh. Um. Ok, well – my job really doesn’t involve editing a lot of content in the tools we track edits in. Maybe my subversion commits are different.


Google reader? (reads are in red)

Google search?

Ok, well, the latter two are a little different. Guess I’m still pretty much “traditional work hour bound” when it comes to production – although consumption is spread out into the evenings relatively evenly (there are search peaks that correspond to high edit/coding times, which I’d expect).

And clearly, I’m dead to the world midnight to 7am or so.

I love data.


When I played little league baseball we had announcers for the game, one of which was some nice old guy in a ramshackle shelter behind home plate that was the “booth” – and every time there’d be two outs, two strikes and two balls – he’d croon “TOOOOO-TOOOOO-TOOOO”. Which of course, has little to do with this post other than version numbers.

Rails 2.2 was just released – so of course, I’m taking the most critical application we have (well, maybe the second critical, and that’s really only because other apps depend on it, it’s not critical in terms of features) – and upgrading it first, because, well, that’s how system administrators roll.

The release notes are great. Seriously. Really great. So great I shouldn’t even be writing this post. Which I am anyway – because here are some of the highlights of what I had to change to make my app work.

NoMethodError for Association Methods

Getting exceptions when you go to Rails 2.2.2 that don’t say anything more than NoMethodError when you know good and damn well the method that it’s saying no method on exists? Yeah, me too. And I bet it’s a method in an associated model. And if it is, you probably should be ashamed of yourself.

Rails 2.2 now enforces privacy on private methods called through associations. So in my case, I had two issues, 1) I was calling “update” on an associated model in some code I blindly copied and pasted a long time ago, and 2) I have a few of my own SQL queries that I’m not entirely sure how to do using Rails associations and named scopes, and I was cheating by calling self.connection.sanitize_limit to take advantage of Rails’ own function for cleaning up provided “LIMIT” params. And sanitize_limit, like the instance method update is private.

Update Rails Footnotes

If you use Rails footnotes in development mode – you’ll want to update for this change for Rails 2.2 compatibility.

Aside… Piston 1.9.5

A great way to stay current with Rails plugins is to use Piston – which has a new 1.9.5 release. You can build your own.

“quoted_table_name” and Has Many Polymorphs

If you start getting some error about undefined method quoted_table_name’` and you use has_many_polymorphs – you’ll want this change.

I don’t use the plugin, I use the gem. So I built my own has_many_polymorphs 2.12.1 gem – by doing: ` gem install echoe git clone git://github.com/fauna/has_many_polymorphs.git cd has_many_polymorphs `

edit CHANGELOG with a “v2.12.1 line” (e.g. v2.12.1. Cloned GitHub project and rebuilt gem for our nefarious purposes.)

rake manifest rake package rake install

add_joins! and Has Many Polymorphs (or anything else for that matter)

HMP includes a ‘tagged_with’ method for finding collections ‘tagged_with’ a set of tags. I use a heavily modified version of that. The method supports custom scopes, in theory. (well, probably more than theory, I’ve just never tried it). While, I don’t have any scopes on models that call my functions – I still had some of the private ActiveRecord method calls in mine – particularly add_joins!.

Well, this change changed the params for the method to make sure that the scoped joins were merged, and not overwritten – which changes calls into it. If you are calling it with your own options use options[:joins]. My code doesn’t use the scopes in combination with my tagged_with method, so I just pulled them.

And…. thankfully that’s it

Other than cleaning up deprecations like number_with_precision now preferring (number, :precision => myprecision) instead of (number, myprecision) – and ActiveRecord::Base.verification_timeout no longer valid, and mb_chars being preferred over chars – we seem to be good to go for Rails 2.2. I still need to test some crons, but I imagine that our app will go production as 2.2 shortly.


Champagne Dreams and Caviar Wishes, NOT

(yes, this was a real dream, this isn’t some wacky metaphor post or anything, a real, crazy, take-as-you-will dream)

So, the dream started with incredulity. My incredulity. A lot like real life.

It all started when my boss came up to me to let me know and give me the heads up that a pair of visiting faculty/researchers were upset about the compiler on our servers. And wanted to replace with their own. Only it was only the binary – not the source code.

“WHAT?!?” was followed by “NO WAY IN HELL”

And just like real life in “Eye-Tee”, my bluster was followed by the sinking feeling that no matter how absurd the request that somehow, someway, I was going to have to do this. And that pissed me off more.

Somehow then – and it gets a bit fuzzy here as dreams and the recollections of them are wont to do – I find myself watching a press conference given by the researchers, both of whom had some combination of Irish-European accents. They were pressing their case for the compiler, noting that it was the binary version because they had lost the compile flags. Then one of then started crying. Apparently their deceased mother had come up with the compile flags, and when she passed, they only were left with the binary.

I was not moved.

Fast forward to the confrontation. They were insistent, taken aback that a mere lowly system administrator would dare deny their request – all the while I’m trying to explain that (a) there was no way I was replacing core system software with some unknown binary and (b) it probably wasn’t going to work anyway. Some compiled binary from some unknown version of linux? No way it was going to run, and no way it was going to keep running through OS and libc upgrades. They ignored the former and didn’t understand the latter. We started getting spectators – sitting on walls around this kind of courtyard were in. Well, a courtyard inside a warehouse. You know the kind, steel roof, lots of dust, doodads and thingamajigs and whatchamacallits littering the floor.

I tried explaining it to them “It’s like taking a program written for OS X and running it on Windows” I cried. The spectators got it. The researchers didn’t.

I got angrier. Here I was having a one-sided conversation about having to upend my entire infrastructure for… wait, I didn’t even know why they wanted this compiler there in the first place.

One of the spectators – who was wearing a pair of headphones – well, half a headphone like some dance-club DJ, proposed parsing the binary’s header to see if the compiler flags could be reverse-engineered.

The researchers became more insistent. I got even angrier, looked at the DJ, who shook his head.

Then I dropped the F-bomb. (hey, a lot like real life – I told you this was vivid). The spectators were amused. The researchers were not.

I remember later, coming back up to them – at a picnic table under a park shelter. Offended/angry they didn’t go anywhere, but weren’t real happy to see me either. Maybe, like I knew that I had to do it, they knew that unless I did, it wasn’t going to get done.

I apologized, a sincere one, but with a “but” attached. It just wasn’t going to work. So maybe they could tell me what the compiler flags did, and why the current compiler didn’t work. The lead researcher relaxed, told me to call him “Erich” – and went beyond my question a step further and told me about the goals of the project – and what their current compiler did. (See, I told you it was a dream) – and I started thinking about how to come up with a way of getting the source and flags for their compiler and modifying it so we could get multiple ones to communicate in parallel with each other.

After that, all I remember is that I had this really cool, incredible shoes that had some kind of rubber on the outsole – but the entire insole was this silicone material that had no friction coefficient, sorta like those shoes with the wheels on them that the kids have now – only I was gliding – along the ground, through all these labs, trying to get back to wherever I was going thinking about totally awesome the shoes were, and how best to market them.

And then I woke up.

I really have to stop coding right before I go to bed.

Because Meat Loaf and Twitter are like Peas and Carrots

This week, a colleague from Auburn, and a colleague from Iowa State twittered our mini list of Geek versions of Meat Loaf songs, starting with “2 out of 3 can’t be accurately represented in binary – but close enough ain’t bad” and followed by:

  • “Paradise by the LCD backlight”
  • “NAT out of Hell” (like a NAT out of hell… you’ll be gone when I go to logon)
  • “Skype is a Lemon and I want my money back” (a little Alanis irony that one)
  • “For Crying Outloud [Groupwise], You Know I Hate You” (an Auburn favorite)
  • “I’d do anything for my email (but I won’t do that)”
  • “You took the urls right out of my tweet”

This is why I twitter. are you twittering? you should be.

Ten Things You Should Be Saying At Work

Earlier this week James shared this article from publishing CEO Michael Hyatt about the “10 things you’d love to say at work but can’t”

But you know, in my job, I find it more prevalent that there’s an awful lot of things at work that we should be saying – but don’t. And I don’t mean those “I love the work you do” express-the-unspoken-gratitude-and-appreciation-for-your-colleagues-more-than-you-do things . I mean, real, legitimate definitive discourse about things we do, and the things we don’t do.

Here are some of mine – and as you might imagine if you have perused prior postings heretofore mine come mainly the the form of questions.

  1. Who, exactly, is “they”?
  2. What do you think [word phrase] means in this situation?
  3. What data do we have that backs up that assertion?
  4. Thanks for that data, can you tell me how it was generated?
  5. How does this help us get to where we want to be?
  6. I don’t think that this has worked like we thought it would. Let’s try something different.
  7. I don’t know.
  8. Yes.
  9. No.
  10. Why?

That’s my 10 off the top – what are yours?

A Story of a Bug

It started, as it should, with the belief that it was my own bug.

I’ve been working for the last few days to generate daily summaries of the activity flowing through our tools. It’s nothing earth shattering, but it’s been a stepping stone to understand a little bit more about the Rails framework – and gave me the chance to begin experimenting with the Google Visualization API. Toward the latter part of the week, there was something a little odd with the “total valid” numbers with the daily account creations – I had made data changes to make sure I had some idea when accounts were vouched for and when they had been retired – so I naturally assumed it was something I did. I even went back and modified the model to make it more consitent with it’s peer models. And kept running the script that produced the daily stats in the Einstein-esque insanity of the doing the same thing twice and expecting different results. After about a dozen combinations of DATE(date_column) comparisons – I went to google, because I knew by then I was either going crazy or this was a legitimate “it’s not my problem” bug.

Which led me to this mysql bug. Reported July 19, 2007. Apparently introduced in MySQL 5.0.42 (May 2007) when the DATE comparison changed from strings to ints. Fixed within two days as part of 5.0.48 (Released August 2007).

But guess which mysql package Red Hat EL 5 (well, RHEL5 update 2) provides? – right, in between. It’s MySQL 5.0.45. And the forthcoming RHEL5 update 3 release doesn’t update MySQL either.

Development and System Administration is a weird, weird world. I use RHEL, not for support (I’m not even sure we have support with the University contract), but to have some degree of patch level stability that’s slightly longer than the fedora releases (and at the time I went to RHEL, Fedora was still dealing with it’s Red Hat transition) – but that stability comes with the price of things like this. I already use my own Ruby to get beyond the base install, but configure, make, make install for one piece of core software is a little different than dealing with it (or MySQL-supplied RPMs) for other software.

I’m glad the open source world gives me that choice, but open source + my labor + thousands of moving parts does give provide the reality that even when a bug is fixed two days later, in the open, patchable by everyone – that sometimes you can find yourself over a year later modifying your own DATE queries so that they don’t include nulls.

So that’s the overall summary of the post I guess – part of it to go into google that if you are getting odd MySQL DATE function results on MySQL 5.0.45 on Red Hat Enterprise Linux (RHEL 5) – it’s a bug. And it’s fixed. But not included in RHEL 5. And if you aren’t getting odd results with DATE comparisons – you probably don’t know that you are.

And maybe one part as a lament to that inevitable ongoing intersection of thousands of moving parts in every environment, not the least of which ours. And you trade off replacing mysql on multiple servers and just turn nulls into zeros (which then breaks your signup form that desparately needs an overhaul) – well, because it seemed to make sense at the time.

And people say you don’t use probability after college.

The second greatest joy

While this blog and most especially that even more geekily weird one involving plastic dinosaurs continue to languish in appropriate obscurity – it is with great mirth that I watch the stats on my humble flickr account. From time to time I think of leaving it to go to smugmug or another service… but you just can’t beat landing on page one for a Google Search for “dust cloth” with images like this:

Improvised Dust Cloth _Improvised Dust Cloth: I had to dust the cabinets today. Never let an engineer get the bright idea to do this. _

There’s real funny, and there’s just funny, and questionably funny, and lastly my own funny, and my own crazy jokes are joy – at least to me, and that’s what it’s all about.

November 4, 2008

President-elect Obama:

To all those who have wondered if America’s beacon still burns as bright – tonight we proved once more that the true strength of our nation comes not from the might of our arms or the scale of our wealth, but from the enduring power of our ideals: democracy, liberty, opportunity, and unyielding hope.

That’s my America.