Week Notes – Updates, Updates, Updates

In which I don’t talk about Kuberneetus like the last three posts to avoid writing up things about kubernetes – I just avoid writing up things about kubernetes, well mostly.

Simon Willison is one of the developers that I’ve admired for years, and have followed his work off and mostly on every since I came across his work on pingback and xml-rpc back when I did PHP (way, way back in the early aught’s). I’ve got a stack of stars about a mile long in my feed reader starring all his work with sqlite tools and especially Datasette – which is something that I haven’t been able to integrate into my day job(s) – and I keep wanting to clear the decks to make time for in the hobby space.

A few weeks ago, Simon did another thing that I am going to try to emulate – he’s started writing weeknotes (he links to more information here )

It’s just a great idea, if only for myself, to be able to look back and see some progression (or regression) in the work that I do.

I didn’t start a post of notes last week, so this first one of mine is going to be a bit stilted most likely, but we’ll see how it goes.

I hold a lot of stress/tension in my neck, which comes up and haunts me sometimes, I ended up getting a fairly severe muscle cramp (what I would have called/do call “pulling a muscle” before I learned better) – enough that it knocked me out of the commute into work. I was able to work from home that day though, and worked on a Bash script. That might not be the best idea for a cure for a muscle cramp ūüėČ

One of our WordPress multisite platforms is available for students and organizations and has been self-serve for years, and has built up somewhere north of 15,000 idle blogs/sites. My colleagues that do the actual WordPress work (I am responsible for the web platforms themselves) are cleaning those out, and worked on a process to identify and flag those blogs as deleted (WordPress has a deleted flag that will 404 the site, but leave all the tables and content). We have a Jenkins platform already in place that’s running some scripts (typically all Bash) that deploy the site and do things like run the wp-cli tool to rewrite urls when needed. Since that’s the model, I went with it, and ended up writing a Jenkins-executed Bash script to use wp-cli to query the database for all the blogs flagged as deleted, dump them to a file and loop over them at 2,000 an hour to remove.

In between Google Searches and Bash Heredoc games, I upgraded my personal iMac to Catalina. I think it would have been straightforward if I had better than 6.0Mbps DSL and a slow 2TB spinning disk in the iMac, it was pretty rough around the edges for sure. But it eventually settled out.

I upgraded my work Macbook to Catalina the next day without much event other than all the Windows Vista-esque permission prompts for Documents, Downloads and notification access.

Sometime last month, we had some accounts get created in our GitLab platform(s) with the wrong UID – so I spent some time trying to debug where that might have gone off the rails. We implemented signed/encrypted SAML – so I can’t get the attributes that were returned from Shibboleth in the logs easily. While OneLogin has a SAML decryptor tool – it’s online only, and well, no private keys are getting pasted there, so I looked into what it would take to write a (non-Java, preferably python) tool to do it locally – with some leads, but no end-success other than to realize the labor effort probably isn’t worth it, and I just need to spend some time with a non-encrypted dev box to work out where things went wrong with the uid value.

In kubernetes news, Rancher 2.3 was released – so both my homelab cluster and my Kuberneetus/Kuberneetle√ľs cluster were updated. And it just worked (which is definitely more than I can say for every OpenShift upgrade I’ve watched, or the ones I attempted last fall in the homelab)

Friday I ended up needing to take a leave day and go out of town for a bit – so there wasn’t anything work or hobby wise that came up then, but on the way back, I did on a spur of the moment suddenly decide that it was time to upgrade my iPhone – so I swung by the Apple store, and worked through that, where again the curse of 6.0Mbps DSL in trying to download a 3.2GB iOS update, and some untold gigabytes of app re-downloads stretched that whole process over 5 hours. But the low-light photo options are pretty amazing.

So, yeah a bit stilted – but maybe that’s appropriate for update week.

I’ve got better notes to start this week on – and it’s likely going to be all about “JupyterHub” this week, which honestly I’m fairly excited about.

Kuberneetus, Kuberneetus, Kuberneetus

Well, clearly, writing about writing doesn’t make writing happen. So maybe if I just write a third blog post where I utter the word “kuberneetus” – Wilford Brimley’s ghost will appear out of nowhere and write up my blog post for me.

Well, close.

In my last post, I wrote that the fundamental reason that I have a kubernetes cluster running this blog was that: “I needed to port an application that I developed from the ground up – and understand the end-to-end experience as both developer, cluster support, and sysadmin.”

So I set out to take an app I have developed over the last several years and that was rebuilt this past year that helps some friends of mine manage a simulation baseball league (and then for good measure the previous version, which conducts the yearly draft ) and deploy it to the kubernetes cluster.

My secondary goal was when I did a “git push” I wanted the application to automatically update.

And that’s what I have, and I’ll be exploring in a series of posts over the next few weeks (months, years? undead lifetimes?) – but here’s the summary statement:

I’ve worked out a GitLab CI/CD (emphasis on the CD and not the CI) automated (using Ansible) deployment of a Ruby on Rails application with persistent storage to a Rancher-based kubernetes cluster. And it’s all (besides the passwords) openly available (and open source for that matter).

And while it’s worth exploring, and there are some useful ideas for how to go about containerizing an application and deploying it, it’s definitely not a route that I’d actually recommend. It was useful for me to get it working as a first step, and it was a useful learning exercise. But it’s a lot of custom (and for that, it might be a good exercise for legacy monolithic applications and setups that actually need custom).

I had already developed a working docker-based local development process for the application in the last year, replacing a rvm and puma-dev based process that I had in place (well using pow.cx) for years – based on work that I did for my previous job way back in 2012 (and that maybe deserves its own post in this series). And that I think is where things break down. I was able to do this because I had already invested in docker for that purpose – and my deployment automation builds on years of Ansible knowledge.

There are much better alternatives emerging. From OpenFaaS to Rio (or any of a whole other set of PaaS frameworks being built on kubernetes) – there’s a much better future ahead.

But first, ku ber neetle √ľs. It’s showtime!

Ready for my close up, Mr. DeMille

So, apparently when I called this blog a 10x blog, I really wasn’t joking.

Just like a 10x engineer, the blog disappears for a month, over-engineers the hell out of something without telling anyone, comes back and can’t explain what it did, and expects an award or something.

I’m going going to blame… yeah that’s right… you know it:

Ku Ber Neet Us

So, a month later; $500 of hardware for the home lab and $95 a month in hosting charges; and at least three complete reinstalls of two separate Rancher-based clusters – this blog (and other things) are back running in an over-engineered infrastructure, maybe again because I can because it’s there.

Really the fundamental reason I did all this was that I realized in order to understand the kubernetes ecosystem more, and really to support it better in my current day job (a part of which is helping to run Red Hat’s OpenShift product for UNC Chapel Hill) – I needed to port an application that I developed from the ground up – and understand the end-to-end experience as both developer, cluster support, and sysadmin.

WordPress is one thing, custom code is another. So the cluster is really for those “other things”.

It’s interesting how a whole bunch of us in this business are on this same path of simultaneous discovery (and following along from others that have made this mistake/taken this path previously) As I have been doing this this past month, Christine Dodrill wrote up a far better technical and process description of her experience.

I’m not really sure I agree with her twitter respondents that Kubernetes is a cult. It is complex. And I think lot of that complexity is really unnecessary.

But despite the complexity, and despite even the overkill nature of an entire control plane and monitoring stack to manage a wordpress blog and a rails application, trading the overhead of the additional resources to run those has given me a chance to experiment (I’m also running jupyterhub as well). And do so in a way that I can build up and tear down things a lot faster than I could cobbling all this together with shell scripts (or ansible, e.g. shell scripts on steroids) on my own.

So now that the other things are running, that’s next. How I got started, and how I went about accomplishing building a local development and CI/CD auto-deployment workflow for my rails application – and maybe it won’t take me a month to write it.

My Rancher Cluster, ready for its closeup

This blog is now a 10x blog

So, as you might completely expect from a “blog” in 2019 – but in what seems an aberration for one called “RambleOn” – it’s been almost a year since I’ve written anything here.

To make up for it, like the proud lineage of 10x engineers that have come before it (obligatory reference: this twitter thread – and more importantly its replies, go ahead and gawk, I’ll be here when you get back) – this blog is now a 10x blog.

That is to say it’s costing me 10x to run it now (actually, really, 13x) and it’s been quite the adventure (read: pain in the ass) setting it up, and well maybe I learned something from it, in the same way that you learn things by grabbing an electric fence surrounding a cow pasture. Which I did at least three times as a child. Because you know once wasn’t enough.

(This really explains a lot about how I became a sysadmin/devops/developer/full stack yak shaver – also recursive delete )

This blog was running WordPress on a $5 a month Ubuntu VPS that ran perfectly fine ‚Äď and it’s now running on ‚Äď you guessed it…

Help me out Wilford Brimley:

Wilford Brimley, as America’s favorite grandpa, pronouncing “Kubernetes” really should end, once and for all, any debate about whether “kubectl” is pronounced “kube-cuddle”

Yes, that’s right “Kubernetes” – $65 monthly for infrastructure components – that aren’t even at the recommended configuration for high availability – to run a $5 VPS blog(*)

( * and some other things )

It really is pretty much as bad as it sounds. Though when you add the part about “and some other things.” it simultaneously gets a whole lot worse. And a whole lot better. Maybe somewhat like bluegrass Gangnam Style

Outside the hyperbole: Kubernetes is one phenomenal collection of software components. I’m serious about how it’s worse, but I’m super serious about how it’s better. I think I’ll let an expert say it best:

Why would I do this? Well, because maybe finally after almost a year (yes, I know, a year, there’s a story or three coming – one even involves an ER visit) I can almost, but not quite, understand what I’m doing enough to actually run the blog (and other things) using kubernetes, and really, the important part, fix it when it inevitably breaks.

Like knowing that the electric fence hurts, but wondering if you can grab it differently and whether it hurts differently that way.

So basically, because I can. And that is pretty phenomenal too, and worth sharing. And if I can help others with that fence, that’s really the most important part.

So that’s maybe the teaser to leave this at – a random pronouncement after almost year of silence that the blog is back and some vague promise to write more in it.

Which, really, sounds like blogging and Rambleon both.

Welcome to 2019 my 2003 live journal self!

Colophon: In case you’re wondering – this is all hosted at Digital Ocean. It’s a 3GB/1vCPU control host that’s running Rancher and an NFS server. It’s managing a three node Rancher-installed cluster: a 2GB/1vCPU etcd/control plane node and two 4GB/2vCPU worker nodes. All are Ubuntu 18.04 nodes, mainly because there are some Digital Ocean block storage things I originally wanted to do that wouldn’t work on RancherOS. I’ll save the rest of the sordid details to the future posts.

Signs That Might Be Omens

The last time¬†I posted here, I was 42 days away from attending the O’Reilly¬† Open Source Conference.

It turns out, that as I found myself moved to post here again, that I began writing this last Monday, which was 42 days since the start of OSCON.

Life has such an interesting symmetry to it.

It was also the day that I submitted my resignation to NC State University, where I’ve been, with two short interruptions, for 27 years as a student and a staff member.

As far as a review of OSCON itself, it deserves a longer review than I’m prepared to give it – and I’m not sure that quite does it justice, pro or con.

All of the break out sessions that I attended were pretty good. The tutorials were incredibly well put together, if blindingly fast. Mostly I got exposed to things that I need to go back later and dig into.

And I learned that whether or not containers are the future for all production services, they are absolutely the present and future of setting up environments for training sessions and labs and demonstrations.

The keynotes didn’t inspire me. Some of the people giving them certainly did.¬† Particularly Nikky Mill, Jerome Hardaway, and Suz Hinton

The vendor keynotes were the same old drivel they’ve always been.

But something kind of remarkable happened outside of the keynotes, sessions and events.

The grant that funds my work at NC State is now beginning its eighth and final year.  While it may (likely) return in the Agriculture Bill for a new round, and a competitive application, as I mentioned in the last post the non-profit organization that that oversees our application has changed focus.  Less in-depth technology work in support of Extension programs nationwide, and more process improvement and professional development in design thinking.

So I’ve been working on ramping down the technological projects I built or helped build over the years, and keeping a very selective eye out for what might be next.

There’s a huge number of factors that find their way into my job decisions now.¬† The same things that have always been there: the technology is important; the culture is important, the meaning of the work is incredibly important; the feeling that my work is valued and that I can make a contribution and help others learn and grow and in the process learn and grow myself is non-negotiable.

But there are other intangibles: I’ve been full remote for over 10 years.¬† I work really well, if a little too long, remotely. How important is remote to me? Could my ~5Mbps DSL only-low-latency-service-I-get internet service even support remote work? It does now because I designed my infrastructure to allow it to be easily managed over reliable, low-latency, but low bit-rate connections, but what about one I didn’t design?

And retirement. My retirement is a pension plan that now returns a significant amount per year at retirement for every year I stay in the plan. I can’t transfer it outside of state institutions. I can’t get the state’s contributions and roll those over elsewhere. I certainly can freeze what I have now and start new, but to come close to the yearly increase in value, any new plan has to involve a significant contribution amount. Any 401K match or even an unheard of full-employer contribution isn’t going to do it.

And there’s a very large pool, employer-paid premium health care. I thankfully am healthy enough that I don’t need it right now. But when I have needed it, just for minor things, it alleviates a significant amount of stress and worry. (Do not get me started on the abysmal state of healthcare portability and expense).

Any opportunity needs to be the right one. The right culture, technologies that I believe in and enjoy (there’s a lot of those). The work really has to mean something. I want my work to help others and to have value and to continue to learn. The work needs to be flexible, if not remote. And then there’s that retirement and healthcare thing.

I’m really incredibly thankful to be in the field I’m in. There’s a shortage of employees and there’s a lot of work available. Most of it is still not the right opportunity, but that leaves a lot that is.

But as that job board at OSCON filled up over the days I realized something, that I’ve known for a long time, I’ve even stated it to myself and others, but it came back stronger than it ever had.

I want to do leading-edge technology work in a top-tier Public University environment.

I still want the same exact thing I wrote last time I tried to make a job change. That is, to Do, Learn Рbut within higher education computing.

Higher education computing is so fascinating, and it varies tremendously by division and department, academic vs. administrative vs. research.¬† It varies dramatically in scale and scope.¬† No other environment that I’ve ever had external exposure to comes anywhere near the generalization and the opportunity (if not the requirement) for individuals and small teams to do so many different things (and still need to master a few). I think it’s fashionable these days to call this “T-shaped skills” – only it’s not only collaborative breadth, it’s functional breadth.

I would have never had the opportunity to do as many things as I’ve done, from computing support to development to networking to hardware to servers to being a business analyst applying my work to multiple knowledge domains, anywhere else.

And in its promise, if not always the practice, I believe fundamentally in the teaching, research and outreach mission of the University.

There are a lot of problems in higher education (I’d be happy to enumerate all of them) and higher education computing, but the ability to make an impact with technology, both quickly, and more importantly, layered over months and years and decades, is really unmatched. The rapid pace of technology and academic (undergraduate) turnover is balanced by a long lived and deliberate culture.¬† Affecting change can be long, slow, and hard, but time also allows for pretty amazing systems to emerge bit-by-bit and brick-by-brick.

Higher education computing (not just research) shaped what would become the web on top of its internet, it gave us the distributed filesystem. It produced the fundamental authentication technologies and concepts that underpin every platform. It honed Linux.  There are thousands more technologies, large and small that emerged from higher education computing.

Most of all, despite itself perhaps (it might not again today), it gave us open source, and open source completely changed the world of computing.

I want to be a part of that. That still exists in pockets in higher education computing at every institution, a provision of time and the space and the problems that interest me most.¬†The University is still the best place – or at least my best place –¬† to have the chance to do and to learn.

My realization at OSCON was that at next conference I had the opportunity to go to, I wanted my badge  to read University of ____ or ____ University below my name, and not a technology company or a retailer or a medical institution, or any of a thousand other great places.

On August 1, 2018 I had an interview for the first time in four years.

I’ve been doing some form of what I do now professionally for 26 years, I’ve dealt with tens of thousands of computing problems and situations, I’ve given presentations, great, meh, and downright bombed awful. I’ve been on the stage with the Governor.

And I still get nervous as hell talking about it.

This was a University job. And from every bit of University web searching, and technology searching, and blog posts and twitters, it was a job that I felt like I could be a good fit for, and that I would enjoy.  It was a pay cut and a commute. But it was interesting and meaningful work, and a peaceful commute (maybe even actual public transportation!), and if you are going to be on site for any job, as walkable and pretty an environment as you can find.

It’s also the type of technology work that I will never ever be able to explain to my parents, which is usually the best technology level I enjoy the most (sorry y’all! I’ll fix those browser popups soon!)

I drove around for a bit, because I’m a sysadmin and I get to the things with plenty of time to spare, because anything can go wrong.¬† Then I started worrying about a speeding ticket. Because I’m a sysadmin and anything can go wrong.

I really enjoyed the interview. I was clearly nervous, I couldn’t remember what Ansible galaxy roles I had used. I couldn’t remember what all changed between Apache 2.2 on Ubuntu 14.04 and Apache 2.4 on Ubuntu 16.04. I’m sure I couldn’t remember a bunch of things I now don’t remember I couldn’t remember.

But I was me, fully me.¬† Rambly, knowledgeable, and completely honest about what I did and didn’t know, fully hopeful and realistic both at what University computing is about.

And more, the people I had the opportunity to meet seemed to be completely okay with me being me. And that was pretty awesome.

I asked a few questions to understand the team dynamics as best you can in that setting. That team, and the culture that’s been built there seems pretty fantastic. The team values their work, and they most certainly value each other, and they are most definitely making an impact.

I don’t know if it’s one of those “don’t look too eager”¬†faux paus to say, and I said as much. But you know, to hell with conventional wisdom. When you know something is right, you say it.¬† We got to the end of the time:

“I really want to work with all of you, and I hope I have that opportunity”

On August 23rd (Universities take¬†forever to process hires, and this time not one, but two were involved). I found out officially that I’d get that opportunity. I accepted on the spot.

I’m going to be working on who knows what all, but to start, it’s going to be a lot of OpenShift. I‚Äôm joining the UNC Middleware team within their central IT group as a Senior Solutions Engineer – which is by far my favorite official title that I’ve ever had.

Beginning September 24th, my badge at any future conference that I have the chance to attend, or speak at, or support, for as long as they might have me is going to read:

University of North Carolina at Chapel Hill

And I couldn’t be more excited.

42 Days To Go

Three years ago the non-profit organization that also manages the USDA grant that funds my position at NC State University hired a new director. And as is often the case with new leadership, and honestly needed to happen in this case, the organization developed a new focus and set of priorities in guiding and coaching individuals and teams to take projects from concept to implementation – an interesting combination of design thinking and a strategy that Adobe uses called “Kickbox“.

My part of the project that focused on custom developing and hosting innovative technologies is no longer a core priority, though also as is often the case in grants and Higher Education, and to my employment benefit, the work that was part of the ongoing grant still remained, and a portion of my work now is to help transition that to off-the-shelf alternatives or to find a soft landing for the custom platforms.

With that new focus came new needs and new costs, enough that travel funding (which is not part of the grant) needed to be curtailed. The unfortunate part of that, however, is that effectively ended any travel for employee professional development, particularly professional development that supported earlier development efforts.

This past week, though, I realized just how much I’ve missed that PD opportunity these last few years. It started when I was reminded of how incredibly awesome the [now defunct] O’Reilly Solid conference was – where in 2014 I had the opportunity to touch the Boston Dynamics “Big Dog” (and more importantly explore a whole bunch of IoT and sensors and open hardware topics).

That all came to a head this past weekend, where after reading again just how incredibly inspiring it was to attend the O’Reilly Open Source Conference in 2011

I joked to my partner that if I won the NC Lottery over the weekend that “I’m going to OSCON!”

And she replied “why don’t you just go?”

And I sputtered a bit, and did the “but… but… BUT…” thing not unlike a¬†Westworld android that has gone off its loop.

And then it dawned on me (it took a few hours and some Jameson’s) – “why don’t I?”

I’ve got a bunch of free airline points from some credit cards… I’ve got a little savings for the conference cost… And an AirBNB is a lot cheaper than a conference hotel… The NC legislature is giving University employees a bonus week of leave… There are few better places to be than Portland in July.

Why don’t I?¬† What was the lesson from 2011?¬†¬†Doing something changes how we see it.

So 42¬† (42!) days from now, for the first time in three years, I’m going beyond the online PD I got used to doing again, and I’m investing in myself.

I’m going to OSCON!

Summer Solstice

Truman keeping watch on the river, Summer Solstice 2017

It’s been a while ‚Äď over 3 years ‚Äď since I last officially rambled on at “rambleon.org” and after some machination that deserve its own post, the summer solstice finds me posting here again, with old posts – and even posts at what was my “home” site at littleriverview.org mixed in.

I can’t guarantee that time won’t find me again posting again six months from now, at the next winter solstice. But it seems time again to return a bit to all the things that made rambleon, well, rambly, nerdy, funny, and maybe even sometimes poignant

In the meantime, there’s lot of history here, and a whole lot of broken links, and weird formatting that has creeped in my wordpress-to-octopress-to-jekyll-to-wordpress transition, so consider it to be web wabi-sabi and take with that the joy therein.

Winter Solstice

The last rays of light of the Solstice Sun just a few hundred yards down the road from the LittleRiverView.

And we spin, another day, another dawn,
another year, another song.
Into winter we surrender,
Deep into it’s splendor.
In mirth we warm our bones, we dream these seeds we’ll sow.

– Elephant Revival, “Season Song

The last time I wrote words in this space, I wrote of a young lady that I encountered on a flight to Indianapolis, and the wish I ended the set of words with then was as much for me as it was for her, that our sense of wonder would not abate whatever the reality was to come.

That reality for me was release and pain, deeply intertwined: the official separation of a nine-year marriage, and the subsequent divorce. The separation and loss for both of pets that stayed with the other. Hard thoughts, hard feelings, hard grace, hard love.

It is an emptiness that hasn’t gone away. ¬†I’m not sure it will ever quite go away.

And I’m not sure it should.

But in release, a reminder that life, that like the Universe, presents infinite opportunities to  dream, and to learn, to grow, and hopefully to be. A life that continually expands and reminds that there are thoughts/feelings/grace/love that have/do/will expand my capacity to fill in new spaces around that emptiness.

And this past year has been filled those expanded spaces.

With music.

With observation.

With exploration.

With energy.

With love.

With, to borrow from Kipling, both Triumph and Disaster, and I am learning, slowly, daily, to treat those two imposters just the same.

Life, reality, dreams, all that it has to hold, has shown itself to be an incredibly beautiful and amazing thing, and I wouldn’t trade one moment of the last year, or the nine years prior, or the last forty-three¬†years of my time on this earth for anything. ¬†Not one moment of the pain. For that provides the contrast to better see the joy that the last year held and all the years before.

Today is the Winter Solstice. A pause in the Sun’s declination as it appears to return again to focus on the Earth’s Northern Hemisphere, a key moment in the seasons on this Earth. And maybe that pause in the Sun’s declination provides the opportunity¬†to both reflect and the opportunity to return to this space after this time of quiet.

I’ve seen a lot of musical groups this year. More, I think, then in all the years prior. ¬†The words at the beginning of the this post are from a band that I had the chance to share¬†this year with a dear friend that¬†has been and is a deeply integral connection with this year’s music, observation, exploration, energy, and especially love.

No more appropriate words could mark the dual ending/beginning that this Solstice represents.

Here’s to¬†warmth and mirth, but especially¬†new seeds, new words, new¬†wonder¬†found each day and each dawn as the Earth turns and tilts the LittleRiverView back towards the Sun.