What do cartoons have to do with build systems?

You know who this guy is? Probably not, he's Rube Goldberg.

I'm surprised by how few engineers know his work. Rube Goldberg was a cartoonist who lived from 1883-1970, he's famous for drawing cartoons of ridiculous and inconceivably complex machines. His work was important during a time in which the world was becoming increasingly mechanized and automated providing a sort of cultural "steam vent" - a way for people to poke fun at machines and industry. I'd embed his work here, but none of it is public domain, so see for yourself or search Google Images. (Be warned, you can spend hours looking at these cartoons.)

I learned about Rube Goldberg from an Engineering professor who, at the time, said, "Rube Goldberg is the most important thing you'll learn over the next four years". Back then, we all thought he was joking, but it turns out that he wasn't. In fact, I wish more people, especially "build engineers" had some exposure to these cartoons. If they had, they'd take a step back and realize that there has to be a better way.

"accomplishing by complex means what seemingly could be done simply"

The title of this section is the Webster's New Dictionary's definition of "Rube Goldberg". Does this describe your internal software distribution process? I fear it might. I've seen some builds and deployment procedures that convince me that people are actively trying to create new Rube Goldberg machines with SCM tools and build scripts. Almost every month I see some new variation of what I eventually characterize as a "Crazy Rube Goldberg" approach to releasing software. Systems that barely work, and that no one dares to change for fear of breaking some crazy contraption.

Goldberg's cartoons and the build's inspired by them share common characteristics. The complexity of his cartoons are almost always defined by the complex ways in which some object is transported from one "stage" of the cartoon to the next and the comic "indirection" involved in every process.

A cartoon example, a man slips and falls, stepping on a switch which then activates a mechanical hand that throws a bag of sand on to a scale which then triggers a mechanism to fire an arrow at a balloon. A build example, Jeff runs a build on his laptop because that's the only place release builds work properly, he then checks the WAR into Subversion and then sends a series of emails to several other people. A deployment engineer then takes the output of Jeff's build and runs it through a custom, somewhat confusing Perl script to customize a series of configuration variables.

Common Characteristics

The common characteristics between Rube Goldberg cartoons and Rube Goldberg builds are:

• Unnecessary Indirection

  • Cartoon: Instead of just creating a machine to pop a balloon, a cartoon creates a contrived process that involves unnecessary (and often impractical) indirection involving pulleys. Goldberg's cartons often find inventive ways to convert potential energy to kinetic energy in some fantastic way you wouldn't have thought of.
  • Awful Build: Instead of using a technology to keep track of binary artifacts, a Rube Goldberg build overloads SCM or (worse) email to send multimegabyte binaries around. Instead of using a technology with built in support for automatic email notification, your build relies on some ad-hoc system of people remember to send just the right individuals a set of process-related emails.

• Reliance on the Unlikely

  • Cartoon: A bag of sand thrown across a room lands exactly on a bulls eye of a target triggering yet another unlikely series of events, continue, repeat...
  • Awful Build: A complex build that takes more than an hour to complete has a random dependency on a web service that provides an series of configuration variables used to configure a binary for production. If the web service happens to be unavailable, or if it returns invalid results, the entire build process must be started from scratch.

• Reliance on Animals (or Humans)

  • Cartoon: One of the comic elements almost always present in a Goldberg cartoon is the presence of an animal of some sort. A bird who, when hit with an arrow squawks loud enough to scare a dog into running and subsequently lifting a weight via a pulley system.
  • Awful Build: Awful build systems frequently require direct human intervention to the point where the build doesn't work unless the same individual is always "strapped" into the build in just the right way. "After that compilation step completes, Jack usually logs in and modifies the build number before he runs the deployment script".

• Comedy

  • Cartoons: Rube Goldberg's cartoons are comedic because they involve the impractical or the impossible. Readers find them amusing because they are plausible, yet inadvisable ways to implement a process.
  • Rube Goldberg-style builds are comedic because they involve the impractical or the impossible. Developers find them amusing because they are plausible, yet inadvisable ways to implement a process.

Build/Deploy systems are Comedy?

If they work, they can be. It's quite common for the build/deploy lifecycle at a corporation to involve a series of steps that no one fully comprehends. Maybe there's a crazy 4000 line Ant script or maybe there's a multi-headed Hydra of bash magic that happens to produce a working system. In the last few months I've talked to people with build systems that work, but no one on staff understands how. When you ask about these release procedures people often chuckle a bit - "Oh, right, our release process.... well let's say it's a little confusing...".

"Why does the build do it like this?" is often met by, "I don't know, but I'm too scared to change this thing right now, the contractor left us with this and it works... for now." In other words, many of these builds involve a series of unknown pulleys and unexpected indirections that resemble a cartoon.

In the next part of this series, we're going to talk about how Nexus can be used to untangle a build into something more reasonable. Something that doesn't involve pulleys, magic, and a series of unlikely events.

Note: Rube Goldberg doesn't do it for you? Then just watch the OK Go video "This Too Shall Pass" to get a modern idea of what I'm talking about. If you haven't seen that, you should. In fact, if you haven't seen this video yet, stop reading our boring Nexus blog and take a break, it's worth it.