Monday, March 28, 2011

Why I'm doing a start-up pre-tenure

Thanks to the tireless work of the entire Hadapt team, we had a very successful launch at GigaOM's Structure Big Data conference last week. In coming out of stealth, we told the world what we're doing (in short, we're building the only Big Data analytical platform architected from scratch to be (1) optimized for cloud deployments and (2) closely integrated with Hadoop so you don't need those annoying connectors to non-Hadoop-based data management systems anymore; i.e. we're bringing high performance SQL to Hadoop). Although a lot of people knew I was involved in a start-up, several people were surprised to find out at the launch how centrally involved I am in Hadapt, and I have received a lot of questions along the lines of what Maryland professor Jimmy Lin (@lintool) tweeted last week:

.@daniel_abadi wondering how the tenure track thing fits in with @Hadapt (r u on leave?) - but congrats on coming out of the Ivory tower! :)

Although Jimmy did not question my sanity in his tweet, others have, so I think it is time for me to explain my (hopefully rational) decision-making process that lead me to start a company while still on the tenure-track at Yale.

A few facts to get out the way: although I am currently on teaching leave from Yale, I am not taking a complete leave of absence, which means my tenure clock is still ticking while I'm putting all this effort into Hadapt. The time I'm spending on Hadapt necessarily subtracts from the time I have available to spend on more traditional research activities of junior faculty (publishing papers, serving on program committees and editorial boards of publication venues, and attending conferences), which means that there is a huge risk that when I come up for tenure, if I am evaluated using traditional evaluation metrics, I will not have optimized my performance in these areas, and thereby will reduce the probability of receiving tenure. When I was considering starting Hadapt, I sent e-mails to several senior faculty members in my field and asked them if they could think of an example of a database systems professor doing a start-up while still a junior faculty member, and going on to eventually receive tenure (I desperately wanted a precedent that I could use to justify my decision). Not a single one of the people I e-mailed were able to think of such a case (in fact, one of them called the chair of my department to yell at him for even thinking of letting me start a company while still pre-tenure). Starting Hadapt is a gamble --- there's no doubt about it.

So why am I doing it? I want my research to make impact, which to me means that my research ideas should make it into real systems that are used by real people. Unfortunately for me, the research I enjoy the most is research that envisions complete system designs (rather than research on individual techniques that can be applied directly to today's systems). It's hard enough to publish these system design papers; but it's almost impossible to get other people to actually adopt your design in real-world deployments unless an extensive and complete prototype is available, or your design is already proven in real-world applications. For example, there have been many papers published by academics that fall in the same general space as the Google Bigtable paper. Yet the Bigtable paper has had a tremendous amount of impact, while the other papers languish in obscurity. Why? Because when Powerset and Zvents needed to implement a scalable real-time database, they felt safer using the design suggested in the Google paper (in their respective HBase and Hypertable projects) than the design from some other academic paper that has not been proven in the real world (even if the other design is more elegant and a better fit for the problem at hand).

Therefore, if you want to publish system design papers that make impact on the real world, you seemingly only have three choices:

(1) You can use the resources in your lab to build a complete prototype of your idea. That way, when people are considering using your idea, their risk is significantly reduced by trying out your system on their application without significant upfront development cost. Unfortunately, building a complete prototype is a much harder task than building enough of a prototype to get a paper published. It involves a ton of work to deal with all of the corner cases, and to make it work well out of the box --- this amount of work is far too much for a small handful of students to do (especially if they want to graduate before they retire). Therefore additional engineers must be hired to complete the prototype. In the DARPA glory days, this was possible --- I've heard stories of database projects burning over a million dollars per year to complete the engineering of an academic prototype. Unfortunately, those days are long gone. My attempts to get just one tiny programmer to build out the HadoopDB prototype were rebuffed by the National Science Foundation.

(2) You leave academia and work for Google, Yahoo, Facebook, IBM, etc. Matt Welsh has discussed in significant detail his decision to leave Harvard and do exactly that. This is a great solution in many ways --- it increases the probability of your research making impact by orders of magnitude, and has the added bonus of eliminating a lot of the administrative time sinks inherent in academic jobs. If I didn't love other aspects of being part of an academic community so much, this is certainly what I would do.

(3) You do a start-up. This is basically the same as choice (1), except you raise the money to build out the prototype from angel investors and venture capitalists instead of from the government (which typically funds the academic lab). The main downside is that starting a company is highly non-trivial, and you end up having to spend a lot of time in all kinds of non-technical tasks --- meeting with investors, meeting with potential customers, interviewing potential employees, investing the time to understand the market, coming up with a go-to-market strategy, attending board meetings, dealing with patents, participating in boring trade-shows, etc., etc., etc. It adds up to an extraordinary amount of time. It's also more competitive than academia --- there are far more people who want to see you fail in the start-up world than in academia, and some of these people go to great lengths to increase the probability of your failure. There are all kinds of hurdles that come up, and you need to have a strong will to overcome them. If it wasn't for the most determined person I have ever met, Justin Borgman, the CEO of Hadapt, we would never have made it to where we are today. It's hard to start a company, but in my mind, it was the only viable option if I wanted my three years of research on HadoopDB to make impact (Hadapt is a commercialization of the HadoopDB research project).

If it wasn't for the fact that I spent the majority of the last decade soaking up the wisdom of Mike Stonebraker, I might not have chosen option (3). But I watched as my PhD thesis on C-Store was commercialized by Vertica (which was sold last month to HP), and another one of my research projects (H-Store) was commercialized by VoltDB. Thanks to Stonebraker and the first-class engineers at Vertica, I can claim that my PhD research is in use today by Groupon, Verizon, Twitter, Zynga, and hundreds of other businesses. When I come up for tenure, I want to be able to make similar claims about my research at Yale on HadoopDB. So I'm taking the biggest gamble of my career to see that happen. I just hope that the people writing letters for me at tenure time take my contributions to Hadapt into consideration when they are evaluating the impact I have made on the database systems field. I know that this will require a departure from the traditional way junior faculty are evaluated, but it's time to increase the value we place on building real, usable systems. Otherwise, there'll be no place left in academia for systems researchers.


[Note: Hadapt has successfully raised a round of financing and is hiring. If you have experience building real systems, especially database systems --- or even if you have built reasonably complex academic prototypes --- please send an e-mail to hackers@hadapt.com. I personally read every e-mail that goes to that address.]

18 comments:

  1. I salute your bravery in breaking away from stale conventions. Hopefully this type of disruption exposes some of the flaws inherent to the tenure system and encourages others such as yourself to keep the purity of the research first and foremost in mind.

    ReplyDelete
  2. I am an example of someone who co-founded a company (Ingres Corp with Mike Stonebraker and Gene Wong) when I was an untenured Asst Prof. I had many of the same objectives and concerns you described.

    In the end I did get tenure, but the case was delayed for one year because my publication record lagged in the 12-18 months before tenure which was just when Ingres was getting started.

    Some of it was political - that is, some faculty members really objected to doing commercial start-ups and they expressed that displeasure by holding me to a high standard.

    In the end, I am glad I did it. As you will learn, if you haven't already, you will be a much better faculty member as a teacher, advisor, and researcher as a result of your experience.

    I should comment that Mike's promotion from Assoc to Full Professor also ran into problems due to the Ingres experience.

    For further info about our experience, read the paper we wrote in one of the early Ingres Papers books that Mike edited.

    ReplyDelete
  3. Thanks Larry! That is exactly the type of precedent that I'm looking for!

    (Also, thanks Lou for your comment).

    ReplyDelete
  4. Can you give some examples of the papers that cover solutions that are "more elegant and a better fit for the problem at hand"? It'd be cool to see a series of posts about great ideas in academia whose time has come to build a real world project.

    Best of luck. I enjoy reading your blog and look forward to updates.

    ReplyDelete
  5. I have heard from several independent people that doing competitive and meaningful systems research is becoming hard over the years because:
    1- It requires a lot of resources and academia can't compete with the industry in this regard.
    2- Industry has direct access to real life problems and have more incentives to solve them.

    Do you think that, going forward, systems research performed by the academia will lag that of the research labs and companies? Is this inevitable?

    ReplyDelete
  6. @nizam: I think the type of systems research that can be done in academia is changing. There are still plenty of "technique" papers that can be published in academia and can make high impact. But for more complete system proposals, it is becoming increasingly necessary to collaborate (in some fashion) with industry.

    @Jim: One example of an underrated system in this space that I've written about in this blog (though it's not from academia) is PNUTS (see http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html). Writing about other examples in this blog is a good idea ...

    ReplyDelete
  7. Dan - congrats on this. I think it is incredibly important to go after your dreams in life, even if they jeopardize what everyone else considers to be the ultimate goal of an academic career. At the same time I don't think you are taking a risk at all. Either you get tenure or you don't. If you get tenure, great. If you don't, then Yale is being idiotic and you wouldn't have wanted tenure there anyway. And it would be no different than if you left Yale to do the startup full-time. As I see it you can't lose.

    Good luck and look forward to watching how it goes from the sidelines :-)

    ReplyDelete
  8. Just a note from another data point. I started Sleepycat in 1996 and got tenure in 2000. It's possible and I don't think you're crazy.

    ReplyDelete
  9. Matt, Margo, it is an honor to receive comments from you on my blog. Margo --- I had no idea you didn't already have tenure when you started Speepycat. That's awesome!

    ReplyDelete
  10. As a systems researcher getting sidetracked by a startup 1 year away from finishing my PhD, this story really resonates with me. I hope it works out for both of us. :)

    ReplyDelete
  11. In option (2), you'll also be limited by the problems that you can work on. I think it's very challenging to conduct work that is both impactful and interesting. But it's fun to face it and say 'challenge accepted' :)

    ReplyDelete
  12. I've been watching you from the sidelines for quite some time, and I have two comments. First, I think your decision to start a company is absolutely the right one as a systems professor. Getting insight into real problems will help you pick transferable research and put your future students on solid ground. Yes, there are other ways to get to that insight, but I cannot think of a more exciting one. Second, when you must choose between priorities of academic tenure and the startup, I hope you choose the startup. The positive feedback effects of making this choice, I believe will make success more likely for both. Goodluck.

    ReplyDelete
  13. Just saw this post. Belated congratulations on raising your round, and best of luck with your academic and commercial success! Given what you accomplished with Vertica, I expect great things from Hadapt!

    As someone who has pursued a pragmatic research agenda from within industry, I am delighted to see academic researchers devoting their waking hours to "real systems that are used by real people."

    I certainly hope your tenure committee is enlightened enough not to hold this against you -- more for their sake rather than yours. Indeed, people like you are the best hope of a continuing place in academia for systems researchers.

    ReplyDelete
  14. Randy, Zhe, Ashwin, Mehul, Daniel: thanks for your kind words.

    ReplyDelete
  15. Daniel, I'm sure everything will work out just fine. I wish you the best of luck!

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. I am only a student with no experience with startups and tenure. But still, hats-off to you, for not joining the rat race and following your passion. Reminds me of the words, "It is not the years in your life that count, but the life in the years".

    Maybe, we can read "years" as "papers". I thought of substituting "life" too, but it is a little complicated as the two occurrences of "life" have different senses(senses 3 and 4/9 in wordnet http://bit.ly/nW7VTX ). :)

    ReplyDelete