Meet BSD: Questions with Leigh Heyman

Leigh Heyman is the Lead System Administrator for Blue State Digital. He is one of the longest tenured BSD employees, ensuring reliable performance, security and sustainable growth of our servers since 2005. Prior to starting at BSD, Leigh was a system administrator at MIT’s Computer Science and Artificial Intelligence Laboratory.

1. Can you explain what a systems administrator does to someone who has minimal technology skills?

A system administrator (or "sysadmin") in many senses is a jack-of-all-trades with computer systems. The way I describe my work in layman's terms is that I'm part engineer, part mechanic. Here at BSD our main responsibility is managing all the servers that host our clients' websites and run the software we build for our clients. I realize that's a bit vague, but the point is that we cover so many aspects, from installing new servers to troubleshooting performance problems to keeping the network secure from outside attackers that there's no way to encapsulate in just a few words. When things are running smoothly we spend our time looking for better ways to manage the system, or finding out how we can save money, or use less electricity or simply make the software run faster. We're the guys who get paged at 3am when there's a malfunction. Although we're not always the people who can fix the problem, we're often the first to know about it; unfortunately we're also usually the folks the rest of the company looks to for explanations when things don't get fixed fast enough.

By the way, it's "system" administrator, not "systems." It always irks me when people call it "systems administration." If I saw myself as managing many small systems, instead of one large one (where all the small systems work together), I wouldn't be doing my job very well. So the correct term is simply "system administrator."

2. What were some of the challenges you faced maintaining the servers during the 2008 campaign? Was it like anything that you had ever done before?

I think I speak for most of the tech team when I say that the 2008 presidential campaign was unlike anything any of us had ever done before. One of the main challenges, from the very beginning, was that we had almost no frame of reference for simply estimating resource usage. Sure we had the numbers from the 2004 and 2006 elections, but we knew from the start that those would only give us a little help. So it was very complicated just trying to figure out things like how many servers we would need, how much bandwidth we might use, how much disk space and so on. Of course very few of our original estimates were accurate so we had many situations where we were scrambling to add capacity.

Another big challenge was that we were using a lot of software in ways that the designers had never quite anticipated and under the kinds of circumstances that put us way outside the normal parameters. This often meant there was very little documentation or guidance available, either from our vendors or on the web, on how to configure things for our needs. As a result we spent a lot of time fumbling around in the dark trying to figure things out on our own. One positive outcome though, was that this forced us into in-depth studies of those tools and as a result all of us on the tech team have built a level of expertise we never would have gained otherwise.

3. Do you remember specific moments where there were traffic spikes in the '08 campaign? Any interesting stories about that?

It's kind of funny, but there were so many moments where we'd "top out," where we'd get a huge spike and be really impressed with ourselves at serving, say, 1000 hits per second without crashing. Only to find ourselves joking, just a few weeks later, "hey remember back in the 'old days' when we thought 1000 per second was high?? Haha... that was funny wasn't it? Yeah... good times, good times."

The truth is that our traffic spike "records" corresponded pretty closely to the significant moments of the campaign. In other words, whatever event was "the most important moment of the campaign... no really we mean it this time" always corresponded to a new traffic record. For example, "Super Tuesday" was supposed to be the "turning point" in the primaries; new traffic record. Then "oops, just kidding, ok, March 4th will REALLY be the turning point," another new record, and so on. By the end of the campaign our baseline traffic, when nothing special was happening, was higher even than our early primary spike records.

I don't have the numbers in front of me, but I'm pretty sure our biggest spike was at the end of his speech after the so-called "infomercial" at the end of October, which I think was somewhere around 5000 hits per second.

4. Is there a correlation between offline action – such as Obama giving a speech – and online traffic patterns?

Absolutely. In fact this was one of the things that really added to the complexity of our resource estimates that I mentioned earlier. There was a direct correlation between nearly every major media event surrounding the campaign and traffic to the website.

Regarding Obama's speeches specifically, there was always a funny cardiogram-esque "lub-dub" traffic pattern around them. Traffic would start building slowly in the half-hour or so before the speech, then suddenly drop off almost completely while he was speaking, and then finally spike through the roof at the end of the speech (it got even worse toward the end of the campaign when he started mentioning MyBO at the end of his speeches). As you can imagine, this made it really hard for us to tune the applications and the servers so that they would work smoothly and efficiently throughout such wild traffic swings.

Basically I never got to watch any of the big events, speeches, debates, etc., without my laptop nearby, constantly watching our performance graphs for any problems, ready to tweak any settings at the first sign of trouble.

5. First political job?

You're looking at it.

6. What drew you to Blue State Digital?

My short answer is that it was an opportunity to "use my powers for good." In fact I'm pretty sure I said as much in the cover letter I sent with my resume.

Basically over the course of my life I've realized two facts about myself. First is that I have a core set of liberal beliefs that are important to me, and second is that I am horrible at persuading others about those ideals. Seriously, watching me have a political discussion at a party is just plain unpleasant. I always wake up the next day angry at myself for all the things I wished I'd said. Obviously this has been pretty frustrating for me. So when BSD came along I finally saw the possibility that I could resolve that frustration and help persuade people (albeit indirectly) by leveraging the technical skills I'd built up throughout my career.

7. How has BSD changed since you started?

Size is the first thing I'd say. When I first started, I was the second person in the Boston office, and there were maybe another half-dozen people working out of the DC office, which was about the size of my living room. Now we're struggling just to have enough space for everyone's desks! Though we've agreed that getting rid of the ping-pong table would be worse for morale than just working in close quarters.

From the philosophical sense, as we've taken on larger-scale and larger-profile clients, there has been a natural evolution away from the "fast-and-loose" work environment of a startup, but to everyone's credit I think we've tried to limit that progression only to aspects of the business where it’s necessary, and everyone seems to keep that startup mindset and lifestyle wherever we still can (see above re: ping-pong).

One thing that hasn't changed, again to everyone's credit, is the nature of the people who work here. Obviously, when a company grows as fast as BSD has recently, there can be a tendency to lower hiring standards, either in terms of skills or personality. But it's been my experience that in spite of time and workload pressures we never compromise when it comes to hiring not only the folks with the exactly abilities we need, but also insist on hiring the people we WANT to work with. There are no prima donnas at BSD.

8. What has been the most interesting thing that you’ve worked on/done in your time at BSD?

It's probably an obvious thing to say, but working on the Obama campaign has been the highlight, not only of my time at Blue State, but of my professional career. You see most of the work I do is "client agnostic" meaning that nearly every thing my team and I do affects all of BSD's clients equally. So while I get to feel a certain reward when one of our clients does well, it's actually somewhat rare that I had a direct hand in their success. With Obama on the other hand, because his campaign consumed more resources than all our other clients combined (at the time), my team and I spent far more time than normal working specifically on the systems that drove his website and online tools. As a result, the level of reward I feel at this accomplishment is way beyond anything I've felt before.

9. What has been your most memorable moment you have witnessed working at BSD?

My first inclination was to talk about election night, or the excitement at witnessing Obama's inauguration in person, or the fun of attending the Inaugural Ball. But as I think about it, the most memorable aspect of my time here isn't any of those specific events, but rather a collection of events that occurred over the last few months of the presidential campaign.

As the election approached, and most of our technical staff was getting simply exhausted beyond anything most of us had experienced before, while at the same time, site traffic was steadily increasing and the campaign was sending more and more emails and trying to raise every cent they could. In the midst of all that fatigue and chaos and the struggles that we had, just to keep the site running under those conditions, there evolved a camaraderie amongst everyone working on the campaign. Every time a crisis would come up, either during the day or in the middle of the night, people would spring into action, much like firefighters rushing to their trucks. Everyone at their keyboards clicking away and calling out to each other across the room about what they were doing and what they thought the problem was. Everyone seemed to know their job and did it. Those moments of bonding and teamwork under fire collectively make up my most memorable moments here at BSD.

10. What do you like to do when you’re not ensuring the reliable performance, security and sustainable growth for BSD’s servers and network infrastructure?

I started coxing my college's rowing team and have sort-of been doing it ever since. During the warmer months of the year this takes up a good chunk of my non-work time.

Oh come on, stop being modest. I think your experience with the rowing team is a little bit more exciting than that.

I've been coxing now for about 15 years. During that time I've won two U.S. National Championships, and won the Head of the Charles four times (for those who know what that is). Starting in 2002 I began training on and off with the U.S. National Team (from which the eventual Olympic team is chosen), and in 2007 after a few difficult years and even a failed attempt at retirement, I made the U.S. Team and my crew raced to 5th place at the World Championships. I took 2008 off of rowing to focus on my work at Blue State Digital, but I'm hoping to get back in to it this year, maybe even back to Worlds. I still harbor thoughts about coxing at the London Olympics in 2012, but that's a long ways off.

11. What lessons that you learned while working for the MIT Computer Science and Artificial Intelligence Laboratory did you bring to Blue State Digital?

I think the first big lesson anyone learns at a place like MIT, that seems particularly relevant in the political sector as well, is that you never know who you might be talking to, so always treat people with professional respect. During my first few months at the AI Lab, I made an irreverent and dismissive comment to an older gentleman who'd come in to my office with a question. I later found out it was Marvin Minsky, who was not only one of the Lab's founders, but one of the "founding fathers" of the Artificial Intelligence field!

Now I freely admit that I don't know the names of all of BSD's clients, but that lesson all those years ago is always a sobering reminder to treat my work with equal care regardless of who the client is.

12. How much wood could a woodchuck chuck if a woodchuck could chuck wood?

Seven. The answer is seven. I looked it up.

13. What are some of the most common mistakes made by system administrators?

I can't really speak to "other" system administrators' mistakes, so instead I'll try to stick to the major ones I've made along the way, which I can only infer have affected others.

One of the major struggles I've always had as a sysadmin is managing my time. I think most system administrators tend to have the dual responsibility of both fighting fires and preventing them. As a result, it becomes a near-daily struggle to balance ones resources between the two. Over time, most of the really hot shot sysadmins I know have developed routines and systems for managing all their different tasks from day to day, and while I don't believe any one system is right for all sysadmins, I strongly believe that having *A* system for managing your time is critical. Once you have a system for managing your time, I've found that the balancing act between firefighting and fire-prevention becomes more manageable because you're better able to deal with interruptions. Personally I'm still honing my own systems and still struggling with the discipline it takes to stick to it.

14. Who is the public figure you most admire?

(Good) Authors, (intellectual) Filmmakers and (some) professional athletes. People who are the best at what they do at things I wish I could do. In particular, anyone who approaches their avocation with the level of (or lack of) seriousness appropriate to whatever it may be, is someone I can admire. These are all careers requiring both talent and discipline, and were a person can potentially earn fortunes, yet the most admirable ones, and the ones whose works produce the most beauty, are the ones who see the money as a happy side-effect of doing the thing itself.

15. The tech portion of BSD is based in Boston. What are some of the advantages of being in Boston as opposed to DC, where many political clients are based?

I think in much the same way that DC is known (obviously) of an epicenter of political work, the Boston area, much like, say Silicon Valley, is known as a hotbed of technology. So just as political talent tends to gravitate toward DC, people with the sort of technical skills we need gravitate to this area. I believe this makes it easier for us to hire and retain the best engineers, programmers and sysadmins. Although I suppose one could say much the same about the northern Virginia area as well.

Regardless of whether we're in DC or Boston or anywhere else, I've realized that it is helpful to have the tech office separate from the office where most of the client interactions take place. Our environment here is quite different from the DC or New York offices. We tend to get a little rowdy at times, we play loud music and as a rule we don't really dress in a manner appropriate for clients (except for me of course), but a lot of this is a necessary side-effect of the way we work. I think an environment like this would be very difficult for the people who need to spend lots of time on the phone or meeting with clients. Which isn't to say they don't have fun in DC or New York, it's just that they have doors they can close to shut out the fun. We have almost no doors and therefore cannot escape the fun.

16. BSD has offices in New York, LA, DC and London. Other than the fact that the Red Sox (clearly the best team in baseball) are from Boston, what makes it the best BSD office?

You mean aside from that I'm here?? Well, we have a ping-pong table. I hear they have one in DC too, but every time they hire a bunch of new people in DC I'm told they put the table away to make space, so basically they haven't played pong in DC since 2007. We took a poll in Boston, and we all agreed that we'd rather work with two people to a desk than ever put away the ping-pong table.

For more information about Blue State Digital's strategy & technology services, click here.