Omni needs a urug

I subscribe to the Omni Mouth for a number of reasons, one I really, really like OmniGraffle, and two, the blog might be the absolute best example of a small company blog I’ve ever seen. The writers of it have style, pizzaz, and it’s just fun. Really fun. Although, pictures of star-nosed moles really don’t belong in my news list.

(they also have full feeds, so they get it).

But I have to admit that not only did I groan when I saw “System Guru Opening” in the titles list in my unread Google Reader items, and I went so far as to give it a “Say it ain’t so Joe” (which is really odd, because I don’t think any of them are named Joe, but I digress), when I saw that the job was at OmniGroup

Omni, guys, gals, pals, writers of great software. For good grief – don’t hire anyone at all that calls themselves a “guru”

People that call themselves gurus, aren’t. Ever.

However, for any system administration brethren that do Macintosh administration – you very likely want to check the job out. It sounds like a great job and a great place to work.(*)

Additionally, they get serious mad props for being up front about throwing your word-formatted resumes away.

But really, I know they say to put “guru” in the subject line – but for the sake of the whole profession, please don’t call yourself that. If they call you that – well you can’t help that, but you should shirk off and mumble incoherently. Or grunt. But do that after you are hired. I really don’t recommend mumbling incoherently in the interview. Been there, done that. I haven’t grunted though. Guffawed yes. Grunted no.

(* I have zero association with OmniGroup and wouldn’t know any of them if I ever saw them. I just really like their software and their blog).

Question of the Day

So, if a work created by an officer or employee of the United States Government that’s part of the person’s official duties (see also Cornell Law) – is Public Domain

Then why on earth isn’t it the same for officers and employees of states, and state and federal funded institutions that are created as part of that person’s official duties?

Or at the very least, under the auspices of a derivable, share-alike license?

Inquiring minds really want to know.

sysadmins vs. the world

You know, it wasn’t until I started this job and had more interaction with folks that didn’t have a University background that I came to really understand that there was any kind of us vs. them perception between developers and system administrators.

In the University, it’s typically “support staff” vs. “systems staff” – and “systems staff” are typically system administrators/pseudo-developers – at least on the academic side. There are very few pure developers in the academic side of Universities that I’ve found that only create code and aren’t responsible for deployment, and systems setup. (and then, those system admins doing development have never really done the kind of development that I was aware was going on in software shops).

Anyway, that’s not really what this post is about. I’m just reminded of that perception of “the systems administrator” by a great exchange with a colleague today while getting some background on some errors that I was seeing on one of our servers from earlier in the week. He gave me a pretty funny quote that I think bears preserving for posteriety:

Actually, I know it will happen again too… but that’s just not something I feel comfortable openly admitting to the system administrator. It’s like saying to the parole board, “Well, I don’t want to kill anybody again, but you never know… it there’s always the chance it could happen.” 😉

To which I had to respond:

Not a bad analogy. But I’m your defense lawyer – not the parole board 🙂 It’s my job to keep you out of jail – or at least reduce the sentence 🙂

But in hindsight, I’m really not sure that calling myself a defense lawyer is really doing anything for the system administrators/managers of the world.

Anatomy of a screw up

So, because you’ll learn far more from screwing up, then getting it always right – here’s the latest screw up from yours truly. Documented for all the world to see 🙂

So, I have been making slow progress the last few weeks in updating our account registration application to be a bit more normalized, and collect a few data elements that we weren’t previously collecting. This took surprisingly far longer than it should have taken.

(it actually turns out to be a bigger pain in the ass to implement a selection form with a defined set of options – and include “other field” and have that other field create new, user defined option and have that live properly across submits, than I thought, but I digress)

A lot of the work went into doing things like changing object names from “University” to “Institution” because well, we are associating folks with groups that aren’t Universities, and maintaining the University names makes the app semantically incorrect. Of course, that then creates a fairly healthy rename nightmare, that search and replace doesn’t really fix. I managed to handle that okay.

And I managed to handle the infamous Rails “nil-error-in-views” problem. For those that aren’t familiar with this, inevitably you will run into a situation where you are going to print out something like user.county – where .county is the automagic accessor created for the county field in your user database. Well, when you start normalizing county to actually be a reference to an entry in the county db, and not a copied string – you have to start doing things like user.county.name – where .county is the automagic accessor created to get at the county object that associated through a belongs_to: or similar – and then name is the accessor for the name column in the county db for the county associated with the user.

The dreaded nil error is that you have to make sure that “county” is an instantiated object. If the user doesn’t have a county – when you go to access user.county.name – county might be nil and Rails goes “Ooops!” I’m sure there’s a highly elegant idiomatic way to solve this, but if statements normally work for me. As long as I use them.

I managed to check most of all the nil places in my views. Where I missed it though is in the XML response to a third-party authentication request. Our other rails applications and our wiki environments proxy-authenticate via a POST to our account application and get back success/failure and profile data in a xml stub block.

I sort of tested this. I tested one application against it, and watched it authenticate just fine. However, I never checked the local log. As a system administrator first, I should know better. always, always, always check the logs. But I didn’t. I would have know the thing was oopsing then.

Well maybe. Test screw-up #2 was the fact that in order to test the new data elements – I actually put myself in a county. So for my test account, there was no oops – county existed and wasn’t nil. For a good portion of any of the users that would be migrated, they were going to be missing a county. But noooooooo, I didn’t test that.

So armed with what looked to be a highly functioning application – I deployed. And I tested all the authenticating apps right after deployment. With a valid county. “Works for me!” “Whoo!”

Five minutes later – I get an IM, from the director. Who couldn’t login.

I look at the logs that are in-app. hmmmm…. seems to login in just fine. logins from the rails apps work for folks. I hadn’t yet seen the application errors emails that we email ourselves when our Rails apps go Ooops! (yay for exception notifiable) (mistake #3) I go and look at exactly what the auth code is doing from the wikis and it dawns on me at that moment that I left the IF statements out of the .rxml I knew it was the dreaded nil

I can’t remember at that moment if I said “Oh shit!” or not. But I did change my IM status to “Yes I know. Yes I’m working on it”

Then I see the app error emails and see that they confirmed the nil. And 5 minutes after that, James pipes up and says “just ignore the passwords”

(the passwords? yeah, apparently there’s a second bug that was highlighted, our auth code had a debugging feature left over that included the necessary POST params in the querystring. When the wiki auth occurs, it was passing those in addition to the POST. So when the exception mails were sent, while the POST passwords were filtered, the query string really can’t be. Whooops. At that point I definitely said “Oh shit”)

Mistake #4 was ignoring the logs on that one for the last several weeks, which now my web access logs have all kinds of passwords in them – thankfully that’s restricted access to just me (and they are going to get a gigantic search and replace soon).

Well, anyway. Being system administrator has its privileges. So I managed to fix the problem, create a test environment to test the fix, deploy the fix (twice), touch base with Rafe about fixing the discovered wiki authplugin feature – who gets it checked in PDQ, and I got that deployed too.

Report, to fix, to test, to deployment, to fix side effect bug, to the emails to the staff, to recommendations for the group that triggered the exception notifications to change their passwords – 40 minutes. That is definitely the silver lining in this. It’s really hard to be more responsive and faster than that.

The summary?

  • check the logs
  • watch your rails nils
  • check the logs
  • test with missing data
  • test from everywhere
  • check the logs
  • clean up debugging code
  • check the logs
  • know your code so dang well, that you can fix it faster than butter on the morning toast

The healthcare mess

Unless you’ve been under a rock the last few days, you probably know that the President’s State of the Union address had a proposal dealing with insurance and healthcare – basically trying to use the tax system as an incentive to generate “lower cost” insurance plans. It was on NPR, it’s been the forefront of a lot of news coverage, Barack Obama is stumping about it, as is Hillary Clinton.

Healthcare is incredibly complex. I’ve spent much of my adult life to this point avoiding anything having to do with it. I have the most basic plan that the University provides, which has the university paying all my premium. My wife stayed on her own plan because the premiums were cheaper. I do most of what I can to avoid the doctor. So I don’t even usually know the values and the limits of my own coverage. Any time I get close to it, I get frustrated. I can’t imagine what it’s like for those that are dealing with the system far more than I am.

There seem to be “bad guys” and “good guys” in all the stakeholder groups – from patients to the doctors, to the medical bureaucrats that serve the doctors, to the pharmaceutical and medical equipment industry, to the insurance industry, to the government. While I have had and continue to have my various biases against those stakeholders, it’s impossible to paint the groups with a broad brush. I can’t even begin to fathom how you unravel the gigantic interwoven mess that our system has become. There are no simple solutions. I’m rather wary of the administration’s plan because it seems largely to benefit the insurance industry and manipulating an already convoluted tax code seems to create more hassle – but maybe it starts debate, I don’t know.

Traditionally my sympathies are with the doctors. I feel like they get caught between the bureaucrats in the front office, crazy and ridiculous malpractice situations, insurance bureaucrats in both the government and private industry, and the relentless pimping from the pharmaceuticals. My sympathies are still there, though the reality is more complex than my simplified sentences could address.

But one thing I’ve seen recently I think highlights the core of the mess that is healthcare. And that is that somewhere between the doctors, the insurers, the government, the lawyers, patients with hangnails, and the lavish marketing of the med supplies corp, the reality of the costs of care have completely gone out the window. I’m not talking about hidden costs of care. I’m talking about the reality of the value printed on the page. It seems completely arbitrary.

I have had some tests recently, in one of the situations the lab sent me a bill in the multiple hundreds of dollars. They for some reason didn’t have my insurance information, and didn’t file a claim, so I got the bill and had to follow up. They did eventually file the claim.

And here’s the arbitrary. My insurer apparently negotiates pricing schedules with the healthcare vendors (labs, doctor’s offices, etc.) Most of you are probably saying “duh” because you know that already. And the claim comes back, and there’s a whole column of the “Amount you do not have to pay” And the vendors apparently agree to that. This isn’t some trivial 5-10% difference, we are talking 50%-80% price differences on bills that are multiple hundreds of dollars.

And that’s the problem I have with this whole mess. If that bill came directly to me, and I didn’t have coverage, And I didn’t feel comfortable negotiating (guess what, I don’t) and I had zero knowledge that there was even such a thing as pricing tiers – I’d be paying hundreds, thousands, maybe tens of thousands over price of what others are paying and wouldn’t be the wiser that person/group A gets price A and person/group B gets price B, based almost solely on some kind of back and forth price haggling.

Maybe those of you in sales or business are also going “duh” – that’s the way the service market works. But in my mind, that’s corrupt. There’s a price that something costs. There’s a reasonable overhead to that price. There’s a risk aspect (those that pay their bills in full and on time deserve better rates), there are expenses, there is capital, there is the right to enjoy a good life for the professionals providing the service – all in all, there’s a fair market price, a price that’s supposed to be largely constant from person to person to person from a given provider.

But what’s happening now is that it’s not fair. It seems highly discriminatory And for those that are underinsured or uninsured, and uninformed – those often least able to pay, they are getting screwed in this (or they are creating a add-on cost burden because they don’t end up paying, and drive up the costs for the next tier up)

I’d honestly love to treat insurance as I think it should be treated – as something reserved for catastrophe. My stomachaches and sprains would come completely out of a high-deductible pocket. But when the paper price difference “without insurance” is so much greater than that “with insurance” – before any money is actually changing hands – something is very, very, very wrong with our system.

That’s what has to get fixed.

Usenet is the new, er, the old, er, the new, Facebook

Now that Google has released an updated groups.google.com view – I decided to check it out last night. It also looks like that have a far more complete record of Usenet postings than I remember having them last.

For all the warnings about Facebook and privacy to keep the Youth of America from posting stupid pictures of themselves in Facebook or Myspace or whatever – all I can say is “been there, done that” – only in Usenet.

I wrote some awfully embarrassing stuff. Sheesh. Not that I had half a brain when I was in college (not that I have half a brain now) – but wow.

One, I apparently was actually pulling for UNC Chapel Hill in 1993 while posting to alt.fan.jimmy-buffet. Embarrassing.

Two, in protest of some telecommunications bill – where it was rumored that the bill was going to outlaw all discourse on abortion (it wasn’t – but this was pre-Thomas-on-the-web, and I was just reading half-baked media reports, not the actual bill). So in order to practice civil disobedience – I launched into some diatribe about abortion. While I certainly was vociferously in favor of free speech and the right to debate publicly any and all ideas (and thankfully still am that way) – and that’s really at the core what my post was about – I was also at the time a pseudo-brainwashed evangelical right-winger (which thankfully I’m not any longer). Highly embarrassing. (and stupid). There’s a whole lot in life that can’t be boiled down to binary positions and I cringe ever time I think or read my own past binary thoughts on that and similar issues.

They still don’t have the post where I bitched out Timo Salmi in defense of me illegally using Turbo Pascal because I was a poor college student that couldn’t afford the software (actually I had a legal copy of TP, but not wordperfect I don’t think). So maybe there are still some mysteries buried out there.

Ah, usenet. Ah. college.

Atom only

The continued lack of Atom 1.0 in wordpress reminded me that my feeds were most likely shifted back to RSS2 due to the politics of the wordpress leaders – and recent upgrades and me not paying attention.

Thanks to this plugin for making it easier.

If any of the estimated 6.5 readers of the blog have their feed readers broken – get a new feed reader.

p.s. See also Shelley’s post – with the appropriate quote:

There’s nothing like syndication to bring out the schoolyard in supposedly reasonable adults. I don’t think even the “vi versus emacs” wars compare.

Internet Jerry Springer. I’ve said it multiple times.

My, my those deprecation warnings are annoying

Sometimes the interpreted language equivalent of “compiler warnings” get really quite annoying (this might explain why subconsciously I have always been incredibly pedantic about compiler warnings).

Anyway, with advent of RubyGems 0.9.0, the “require gem” command is deprecated.

And as of 0.9.1 of RubyGems (to which you might want to update because of a security hole) – you know get lovely little warnings scattered all over the place

Warning: require_gem is obsolete. Use gem instead.

And of course things like the mongrel_rails command include “require_gem” – as does feed_tools – which of course is kicked off by cron jobs in our environment, and is now happily filling my root mail with the warning messages.

Sigh.