Ramble On

So, what do you do when you need to quickly shell script out the running of day-based reports for 29 different vhosts for every day from March 1st until the present?

You cheat.

You can’t really run a for loop across every month and every day in the month, because every month doesn’t have the same number of days, and it looks really stupid to have a blank report for April 31st. In a scripting language with more date arithmetic, I’d probably start with March 1st, and keep adding a day until date == the day I was running the reports, but it’s been way too long since I’ve written anything of substance in perl or php - and well, I haven’t sat down to learn Ruby yet - so all I have is the shell. And I’m not very good at that. But for looping something is certainly better than copy and past for 4 months times <=31days times 29 vhosts.

So, yeah, back to the cheating.

My for loop looked like:

for i in `seq -f %02g 3 7`;
do
    for j in `seq -f %02g 1 31`
    do
        dateme="$year/$i/$j"
        month=`date --date=$dateme +%m`
        day=`date --date=$dateme +%d`

“date” is smart to treat April 31st and May 1st as the same day, and I get away with this because it’s just static data that overwrites the previous output (the report is ran and output to a $year/$month/$day directory. So I’m just running the May 1st and July 1st report twice. A fair tradeoff for not having to build the date comparison rules into the shell script myself.

I am not sure I want to know what prompted this:

to show up in the front of the manual for the Motorola Razr v3m

It was because of the ants.

The ants returned to the kitchen today, I guess exploring to escape the sweltering furnace that was Raleigh, North Carolina today. But they came, noticed by the brown-eyed girl about to leave to go to the wedding shower this weekend.

I despise the ants. No, I absolutely loathe the little critters once they enter my abode. Outside they may have their life, and more than once, I have hiked up in the mountains of North Carolina, sat in a grassy meadow taking in the wonder of creation, and watched the ants go about their business. Try not to disturb them or their homes.

But once they enter mine, it’s a death sentence. I will immediately go to the store to go get whatever rancid poisons I can find, mostly likely because I emptied the last bottle of toxicity the last time they appeared.

So I go to Target. Pick up whatever bottle of poison that indicates things like “MAXX!” and “KILLS THEM DEAD!”

and then I make the mistake of entering the software aisle. I’m a Macintosh user. I’m immune to the Target and the Best Buy and the software aisle of everywhere but the Apple Store. But it’s fun to look at what kind of crap is being foisted on the Windows Weenies of the World.

That’s when I spied it. The Win/Mac CDROM version of Cars. I don’t really play computer games. I can’t really win any of them. I own enigmo, and iconquer, and I played the remake of Choplifter like it was going out of style for like two days. But eventually I just give up on them.

But I had downloaded the demo for Cars and thought it was cute, made for 10 year olds, and playable by 30 years olds, about as simple as Pong. Right up my alley. And the brown-eyed girl is out of town, and I’m set for a weekend of work and boredom Why not waste hours on a 10 year old’s game while waiting for the ants to die?

So, seduced by the $15 Target exclusive pricing, I buy the game, kill the ants, and dutifully place the game in the drive.

It spins, whirs, chunks. the system.log starts spitting out:

kernel[0]: SAM Multimedia: READ or WRITE failed, ASC = 0x11, ASCQ = 0x05
kernel[0]: SAM Multimedia: READ or WRITE failed, ASC = 0x11, ASCQ = 0x05
kernel[0]: SAM Multimedia: READ or WRITE failed, ASC = 0x11, ASCQ = 0x05
kernel[0]: SAM Multimedia: READ or WRITE failed, ASC = 0x11, ASCQ = 0x05
.
.
.

ad nauseum.

And the disk refuses to eject.

I fire up disk utility.

kernel[0]: disk2s1s2: I/O error.
kernel[0]: IOATAController device blocking bus.
kernel[0]: IOATAController device blocking bus.
kernel[0]: IOATAController device blocking bus.
.
.
.

The only hope is to reboot, and hold the mouse button down.

Then, oh then, I get the bright idea to use the brown-eyed girl’s laptop. It won’t run on intel macs, but maybe, maybe the laptop will read the disk.

No luck. Same error, I reboot the laptop, and it chunks, whirs, and refuses to eject the cdrom after what seems to be decades of seconds. Oh my, the game has killed the girl’s treasured laptop. Finish of having to hang my shame at the genius bar dance in my head. Why didn’t I use the work laptop?

I turn it off, connect it in firewire target mode, same deal, until it finally, finally, gives up the ghost of the cd.

And to think, it was Dreamworks that did Antz

So the CD’s are in the trash, and I’m never, ever, buying a pixar game again.

( and what kind of stupid operating system chokes on an unreadable CD? )

So, it’s getting toward the end of the month.

And I’m thinking about the webstats we run. And the webstats are produced today based on yesterday’s data.

And I think to myself, “Self, what happens on the first of the month?”

And Self responds “you stupid idiot, you produce output directories based on today for yesterday”

Sigh. Why can’t I wait until after the bugs manifest themselves and people complain to fix them like 90% of the rest of the IT companies in the world?

p.s. using `date –date=yesterday` works great in shell scripts

Happy SysAdmin Day

Maybe like the coordinated flush, we should do a coordinated server crash. Non-production of course.

So, when your perl script that came with your mod-log-sql distribution that dumps your database out in combined log format managed to put the remote_user value (which is usually only valid for a handful of internal sites anyway) in the wrong log field meaning that your multiple hour process that created your historical awstats process that ran all night last night produced data that looks like:

address user - [date]

instead of

address - user [date]

what do you do? You fix the perl script, and run the whole damn process over again.

Sigh.

p.s. yay again for screen

Thank goodness for screen and for me remembering to use it.

I managed to lock my macintosh up hard trying to copy 20GB of Data from one drive to another. really hard. Like no mouse and keyboard response hard. With my shell open to the server processing my gimongous log file data set already immortalized in song today.

That would have completely stunk without being run in screen, as such, it was pop open the laptop, shell into the log server and a quick screen -d -r later, I’m happily watching basically nothing other than a line kicked out to the console for every 8192 lines processed.

but what a sweet uninterrupted nothing it is.

Apologies to the whole cast of Rent

Two Million Seventy-Six Thousand
Two Hundred Log Lines
Two Million Seventy-Six Thousand
Log Lines - This Bites
Two Million Seventy-Six Thousand
Two Hundred Log Lines
How do you measure - measure a Web Site?
In Jpegs - In Post/Gets
In Doctypes - In Script Kiddie Hits
In 404s - In 200s
In Checking perms and rights

How about 19 hours of processing?
How about 19 hours of processing?
How about 19 hours of processing?
Too damn many log lines

Finally using awstats to process the last several months of log files. 2,076,227 log lines for a few dozen low-traffic (thankfully) websites from around March to prior to yesterday. Even with my own private DNS server running on the same box as the update script, it’s about 30 lines a second. I think it should be done in 19 hours

A modern day allegory.

[cue ominous music, and sweeping panoramas of man's finest engineering projects through the ages]

So last week, I lost a drive in what was my primary database server, and as I went to replace the drives that day, I forgot my phillips head screwdriver.

The tiniest mistake that less than a week later, would result in multiple wasted hours of time.

[cue thunderclaps, and grainy film of the tacoma narrows bridge, 1950's rocket tests, and other engineering failures]

The server room with the failed drives is just a few miles as the crow flies from my office, but it’s a royal pain in the arse with traffic to get there some days, over one of the roads-that-most-needs-paving-in-Raleigh (hillsborough street for those keeping score). And then there’s the parking issue. I would be digressing, but I’m not. For my cautionary tale involves my complete avoidance of this.

So I go to replace the drives, carrying the spare drives, and the Dell drive carriers for my server, minus screwdriver. I get there, and have no way of attaching the drives to the carriers, something pretty much necessary in order to hot swap the drives.

But wait, (Lo!) there’s a cordless drill there with phillips bit, left by another group, and I figure that it wouldn’t be a big deal to use the cordless drill with phillips bit in order to attach my carrier(s) - it’s just 8 little screws for two drives! And of course I don’t want to have to drive all the way back just for a screwdriver.

Well, one of the spares turns out to be bad, which I find out after trying to make it a hot spare so I can rebuild the Raid. But one of them works fine, so it goes in the server.

Fast forward a few business days. I’ve got a replacement for the replacement. I start to unscrew the carrier from the failed replacement. And of course, you know what comes next, I can’t get the carrier off, the drill applied too much torque, and while trying, I strip the screw enough that there’s no way it’s coming out manually. eXtension doesn’t have a cordless drill.

So I think “I’ll go back to the server room and see if the drill is still there” - I go, it’s still there! But unfortunately, you guessed it, it’s out of power.

So the only recourse I have is to drive all the way home, get my own cordless drill, unscrew the carrier, and carry back the carrier to place on the good replacement drive.

Net time lost? Oh, about 4 hours. All because I forgot a screwdriver, and (importantly now, pay attention) - under the first time pressure, made an ill-formed decision to use the drill in the first place.

So what’s the moral of the story?

* do not forget the screwdriver
* do not make a ill-thought out decision under duress to avoid doing the work you know you really should do
* given (a) and (b) - you are going to pay dearly for it in the end, redoing your system, or your code, or losing a few screws

Today’s Electric Company moment brought to you by the letter “J” and the number “3″

See, I bought my Fiance an iPod Nano for our our 2 year “date-aversary” she’s got an old iMac G3 that we haven’t upgraded yet, so the short term plan was to just use my desktop and ask Apple about merging iTunes Libraries as we get married. I’ve got 300+ songs from the iTMS, she’s got 90+.

To: Apple Support
Date: Saturday, July 8th

Hello,

My fiance and I are getting married in September, each of us are Macintosh users bringing our music collections with us to the family. I’d like to get the accounts merged in some way so the music collection can be merged, played on each computer. We’ve decided to merge them under my account if possible. How do we re-authorize her computer (and her purchased music) under my account?

The response was nice enough, lots of information about the number of computers you can authorize and information deauthorizing / authorizing a computer (which of course I can find in the KB). But the opening sentence says it:

To: Me
Date: Sunday, July 9th

Dear Jason,

I apologize but it is not possible to join two or more accounts together.

Interesting.

Weyahl, there’s ways around that. Maybe a little murkily questionable vis-a-vis the DMCA, but certainly within the bounds of fair use.

Hey look - it’s a link blog entry!

* Real Men of Genius - blogger person style I love a good parody. (via Jeremy Zawodny
* Saturday in the Park in Beloit On July 1st, the community of Beloit, Wisconsin came together on the banks of the Rock River to recreate George Seurat’s “Sunday Afternoon on the Island of LaGrande Jatte” — “Saturday in the Park with Friends” Cool!
* Social Networking Essays linked by Guy Kawasaki. From a quote in one of the links - an essay by Fred Stutzman at UNC-CH There’s gold in them hills, people. Now go get it. In related news, I’m looking forward to making some time to install Lyceum. (Kawasaki link via Chuq)
* MySQL Performance blog. I need to spend some serious time just learning a modicum of things about tweaking our MySQL performance. (Also via Chuq).
* Top 50 Scientist Blogs (via Steve Rubel via Chuq). This is exactly why we need Extension Specialist and Agent blogs (likely via Lyceum) We just have to figure out our internal battles about common look and feel and branding vs. letting our users have some freedom of expression.
* Data Leakage and crappy tele-apps from Sprint. Sigh.
* Damn Interesting. WHAT A GREAT BLOG! This is what keeping up with too much information is about. Damn Interesting indeed.
* LiveJournal does Jabber. (via Peter. It’s happening folks!. After the server transitions are done. I’m spending serious quality time with XMPP and our Extension System Chat Server.
* Yet another reason to license that software Looks like that logo flashes when you author using a trial version of Apple’s Shake. So basically from what we can ascertain, Sony (or its ad agency) is too cheap to buy their people the full version of Shake, which goes for $500 list on Apple’s site. Tisk, tisk! Bwahahahahahahahaha.

I was reminded of this post in 2003 by Brent Simmons (NetNewsWire’s developer) about good bug reports. It’s also echoed in this note from Joel Spolsky.

I’m going to add to Brent’s just a bit, but this should be shared far and wide. Especially among the IT staff. IT staff are the absolute worst (pot, kettle, black) when it comes to bug reports. Worse even than faculty ;-). Smart people seem to turn their brains off with problem reports. Which they never would tolerate in the own discipline or support area.

Effective bug reports should include:

1. What you did. This should include as much detail as possible. And for web applications - please include the URL where the problem occurred and your operating system and browser/client software!
2. What you expected.
3. What actually happened.

I just thought that the total on my grocery store receipt while getting a few things we needed for the next few days was rather appropriate for July 4th