Sunday, December 7, 2008

The hacker's lament

One of the thrilling parts of working and writing in Silicon Valley is the incredible variety of people I've had the chance to meet. Sometimes, I meet someone that I feel a visceral connection with, because they are struggling with challenges that I've experienced myself. In a few cases, they are clearly smart people in a bad situation, and I've written about their pain in The product manager's lament and The engineering manager's lament.

Today I want to talk about another archetype: the incredibly high-IQ hacker who's trying to be a leader. (As always, this is a fictionalized account; I'm blending several people I've known into a single composite. And please forgive the fact that I use male pronouns to describe the archetype. There is terrible gender bias in our profession, but that's a subject for another day. Suffice to say, most of the hackers I've known have been men. As a last disclaimer, please consult the definition of the word hacker if you're not familiar with the controversies surrounding that term.)

It's common to find a hacker at the heart of almost any successful technology company. I know them right away - we can talk high-level architecture all the way down to the bits-and-bytes of his system. When I want to know about some concurrency issues between services in his cluster, he doesn't blink an eye when I suggest we get the source code and take a look. And as soon as I point out an issue, he can instantly work out the consequences in his head, and invent solutions on the fly.

This kind of person is used to being the smartest person in the room. In fact, it's a rare person who can be subjected to recurring evidence of just how stupid the people around them are, and not become incredibly arrogant. Those who have the endurance are the ones that tend to lead teams and join startups, because you just can't be successful in a startup situation without empathy. I would characterize them as intolerant but not arrogant.

When a startup encounters difficult technical problems, this is the guy you want solving them. He's just as comfortable writing code as racking servers, debugging windows drivers, or devising new interview questions. As the company grows, he's the go-to person for almost everything technical, and so he's very much in demand. He throws off volumes of code, and it works. When scalability issues arise, for example, he's in the colo until 2am doing whatever it takes to fix them.

But life is not easy, either. As the company grows, the number of things he's called on to do is enormous, and the level of interruptions are getting intense. It's almost as if he's a country that was immune to the economic theory of comparative advantage. Since he's better at everything, he winds up doing everything - even the unimportant stuff. There's constant pressure for him to delegate, of course, but that doesn't necessarily work. If he delegates a task, and it gets messed up, he's the one that will get called in to deal with it. Better just to take care of it himself, and see that it's done right.

When you're the physical backstop putting dozens of fingers in the damn to prevent it from bursting, you might get a little irritated when people try to "help" you. The last thing you need is a manager telling you how to do your job. You're not very receptive to complaints that when you take on a task, it's unpredictable when you'll finish: "you try getting anything done on schedule when you're under constant interruptions!" Worst of all, your teammates are constantly wanting to have meetings. When they see a problem with the team's process, why don't they just fix it? When the architecture needs modifying - why do we need a meeting? Just change it. And we can't hire new engineers any faster, because you can't be interviewing and debugging and fixing all at the same time!

The picture I'm trying to paint is one of a bright individual contributor stretched to the breaking point. I've been there. Trust me, it's not a lot of fun. And I've also been on the receiving end; and that's not much fun either. Yet, quite often these dynamics play out with ever-increasing amplitude, until finally something drastic happens. Unfortunately, more often than not, it's the hacker who gets fired. What a waste.

What's wrong with this picture?

One of the most exhilarating things about a startup is that feeling of intense no-holds-barred execution. Especially in the early days, you're fighting for survival every day. Every day counts, every minute counts. Even if, in a previous life, you were a world expert in some functional specialty, like in-depth market research or scalable systems design, the compressed timeline of a startup makes it irrelevant. You get to figure things out from first principles all the time, experiment wildly, and invest heavily in what works. From the outside, it looks a lot like chaos. To a hacker, it looks a lot like heaven.

But even a tiny amount of success requires growth. Even with the highest standards imaginable, there's no way to hire just genius hackers. You need a diversity of skills and backgrounds. Suddenly, things slow down a little bit. To me, this is the critical moment, when startups either accept that "process = bureaucracy" or reject that thinking to realize that "process = discipline." And it's here that hackers fall down the most. We're just not naturally that good at thinking about systems of people; we're more comfortable with systems of computers.

If you've ever been abused by a bad manager in your career, it's easy to become traumatized. I think this is the origin of the idea among hackers that managers are idiots who just get in the way. The variations on this theme are legion: the pointy-haired boss, the ivory-tower architect, and of course the infinite variety of marketroids. But whenever groups of people assemble for a common purpose, they adopt process and create culture. If nobody is thinking about it, you're rolling the dice on how they turn out. And, at first, it's OK if the person who's doing that thinking is part-time, but eventually you're going to need to specialize. The alpha-hacker simply can't do everything.

Even in the areas that hackers specialize in, this go-it-alone attitude doesn't work. Building a good application architecture is not just coding. It's more like creating a space for other people to work in. A good architect should be judged, not by the beauty of the diagram, but by the quality of the work that the team does using it. The "just fix it" mentality is counter-productive here. Every bug or defect needs to go through the meta-analysis of what it means for the architecture. But that's impossible if you're constantly fire-fighting. You need to make time to do root cause analysis, to correct the systemic mistakes all of us tend to make.

And taking on too many projects at once is a classic sub-optimization. Sure, it seems efficient. But when there is a task half-done, it's actually slowing the team down. That's because nobody else can work on the task, but it's costly to hand it off. Imagine a team working from a forced-rank priority queue. Naturally, the best person should work on the #1 priority task, right? Not necessarily. If that person is subject to a lot of interruptions, as the people working on the less-important tasks finish, they're forced to keep working down the list. Meanwhile, the #1 task is still not done. It would have been faster for the team as a whole to have someone else work on the task, even if they were much slower. And of course there's the secondary benefit of the fact that as people work on tasks they don't know anything about, they learn and become more capable.

The reason this situation reaches a breaking-point is that it's constantly getting worse. As the team grows, the number of things that can go wrong grows with it. If a single person stays the bottleneck, they can't scale fast enough to handle all those interruptions - no matter how smart they are. And the interruptions themselves make looking for solutions increasingly difficult. Each time you look for solutions, you see a conundrum of this form: you can't hire because you're too busy, but you can't delegate because you can't hire.

All is not lost, though. When I get involved in companies that struggle with this problem, here is the kind of advice I think can help:
  • Introduce TDD and continuous integration. This is one of the bedrock practices of any lean startup, and so it's a common piece of advice I give out. However, it's particularly helpful in this situation. Without requiring a lot of meetings, it changes the perspective of the team (and its leadership) from fire-fighting to prevention. Every test is a small investment in preventing a specific class of bugs from recurring; once you've been successful at building this system, it's pretty easy to see the analogy to other kinds of preventative work you could do. It also helps ratchet down the pressure, since so many of the interruptions that plague the typical hacker are actually the same bugs recurring over and over. TDD plus continuous integration works as a natural feedback loop: if the team is working "too fast" to produce quality code reliably, tests fail, which requires the team to slow down and fix them.

  • Use pair programming and collective code ownership. These are two other Extreme Programming practices that are explicitly designed to counteract the problems inherent in this situation. Pair programming is the most radical, but also the most helpful. If your team isn't ready or able to adopt pair-programming across the board, try this technique instead: whenever anyone is becoming a bottleneck (like the proverbial hacker in this post), pass a rule that they are only allowed to pair program until they are not the bottleneck anymore. So each time someone comes to interrupt them, that person will be forced to pair in order to get their problem solved. In the short term, that may seem slower, but the benefits will quickly become obvious. It's another natural feedback loop: as the interruptions increase, so does the knowledge-transfer needed to prevent them.

  • Do five whys. This is a generalization of the previous two suggestions. It requires that we change our perspective, and instead treat every interruption as an opportunity to learn and invest in prevention.

  • Hire a CTO or VP Engineering. A really good technology executive can notice problems like the ones I'm talking about today and address them proactively. The trick is to hire a good one - I wrote a little about this in What does a startup CTO actually do? Sometimes, a great hacker has the potential to grow into the CTO of a company, and in those cases all you need is an outside mentor who can work with them to develop those skills. I've been privileged to have been the recipient of that kind of coaching, and to have done it a few times myself.
At the end of the day, the product development team of a startup (large or small) is a service organization. It exists to serve the needs of customers, and it does this by offering its capabilities to other functions in the company, and partnering with them. That's only possible if those interactions are constructive, which means having the time and space for people of different backgrounds and skills to come together for common purpose. That's the ultimate task for the company's technology leadership.

I strongly believe that all hackers have the innate ability to become great leaders. All that's required is a shift in perspective: at their root, all technology problems are human problems. So, fellow hackers, I'd love to hear from you. Does this sound familiar? Are you ready to try something different?

8 comments:

  1. As usual, a fantastic article. Many thanks for posting it. In a future article, I'd love to see more about what goes into that mentoring process.

    ReplyDelete
  2. Another great article.

    I feel like I'm a similar position, though I'm coming from the opposite direction. Rather than getting to this point by growing a startup, I joined an existing company out of college and matured into the lamenting hacker.

    You make a great point that hackers shouldn't grow it alone, and should culture a great team. But there *are* bad managers, and sometimes they can't be fixed. I feel like I'm growing to the point where there are no more technical problems that can be solved without replacing management. A poorly performing CEO won't fire themselves.

    Now, I'm working to create a startup just to get away from being in the situation where I'm responsible for fixing the symptoms of problems, but don't have the authority to solve their cause. I'll be mindful of your advice. :-)

    Allen

    ReplyDelete
  3. Excellent, excellent, excellent, spot on, well-written, entertaining without forsaking depth. This essay puts PG to shame.

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. I can't agree with you more.

    I came out of the university at a relatively young age and I was recruited out of my lab by a startup. Long story short, I am the only go to guy for every technical problem in the company.

    Our online media company is small, we have 6 employees, great proven leaders, great talent, great cashflow. Our CTO is the CEO himself, but he lacks the understanding of the new software development processes and the new technologies.

    In this group, I am the only technical guru with very little industry experience. I've never been managed before since I had so much autonomy and freedom to do almost anything in my laboratory back in the university. In school I was being pounded day and night with anything that has to do with agile software development. When I entered the company, I realized that there were no process and I quickly applied what I know about agile software development. I am basically a one-man shop.

    My responsibility at the time I joined the company was to develop a product that is to be deployed on a social network (hint hint, FB). It sounded fun in the beginning and I had a ton of passion to create this product. For this product, I later realized that I was the software architect, the frontend engineer, the backend engineer, the database administrator, the systems administrator, the QA engineer, and lastly I was the only technical support guy. The product eventually became bigger and bigger and more difficult to maintain. The CEO tried to help me delegate tasks to several other people, but I found that it only makes matter worst and uses more of my time because at the end of the day I have to go back and review what they have done and I still have to fix it! I didn't give up so I worked around the clock to get it done. Luckily, my girlfriend did not leave me because I make it seem like I don't have that much work left to do when I go home every night.

    To make matters worst, there was another product waiting in the pipeline for development process to begin right after the previous one. Guess what? I went through the same things again and now I have two products under my belt. What's funny is that I make my job look so easy that the CTO never really thought about hiring someone to help me out. (I'd really prefer he had hired a VP of Engineering, or whoever that can manage me effectively.)

    The second time around however, I went around the internet and read blogs such as yours to see how I can save myself from all the chaos because I knew that at some point I will break and just leave for another company only to find that I will probably have to go through the same thing again!

    I applied what I learned to save myself. Processes such as TDD, continuous integration, pair programming (w/ my imaginary friend =P), source code management, and other organizational techniques, processes, whatever.

    I would love to have an effective CTO or VP of Engineering to manage me. I believe I am a great engineer, but what I am going through right now is really making me want to quit the company and maybe even start my own. I don't want the next guy to go through what I went through though.

    What can I do? I would love to try something different. Ah, headaches, headaches.

    ReplyDelete
  6. Very nice article, thanks a lot.

    ReplyDelete
  7. Great article. As another person that did the startup thing, I have a few lessons learned as well that maybe someone can benefit from:

    http://softwareindustrialization.com/BridgewerxProductSuccessBusinessFailure.aspx

    Cheers!

    ReplyDelete
  8. Well done, another gripping and perfectly accurate article.

    After meeting quite a few like-minded hackers whilst working in the UK for the first part of my career, I must say I miss the opportunity of running into the sort of people that being in a technologically- and intellectually- dense part of the world seems to generate. I no longer run these sorts of hackers down here in NZ!

    I agree with you regarding turning Hackers into CTOs with proper mentoring, and respect for you for taking on this role. All too often hackers stay hacking because of this lack of seeing the bigger picture or not having the support to make the 'transition'.

    I had a similar mentor, although didn't appreciate what he was trying to do at the time; I was so into being a hacker that all the talk of understanding strategy, sales, marketing, and commercial goals had little appeal over hacking away at the next project or rewriting yet-another-version-of-the-application-framework (TM) ;)

    ReplyDelete