Guest: Jocelyn Goldfein, Former Director of Engineering at Facebook
As technologists, we want to build software that is friendly, fast, beautiful, reliable, secure, and scalable. And we expect ourselves to deliver it on time and under budget, because our ultimate goal is to have lots of happy customers who can do what they want: cue Daft Punk’s “Technologic!”
But time and energy are finite, and we simply cannot deliver it all at once. We need to choose our priorities, and this choice is one we should make consciously.
Evaluating our software development priorities while dealing with constraints is known as the tradeoff space.
How can you make wise tradeoffs for your product?
The choice is based on a balance between your technology stack and business model type.
“Move fast and break things!”
While this has become a popular motto, it doesn’t apply to every company.
For example, enterprise software companies that are building system-level software prioritize reliability because customers need to use them. Each change needs to be rigorously tested, and often approved before it can be released.
Meanwhile, consumer internet companies spend time and money on making their UX delightful so that people want to use them. Reliability is something they’re willing to sacrifice. Since many are web-based applications, they can iterate quickly and release changes frequently.
So yes, they can move fast and break things.
The tradeoff space may seem insurmountable, but you too can become confident about your decisions by learning from a true pro!
In the second episode of FemgineerTV, I’ve invited Jocelyn Goldfein, the Former Director of Engineering at Facebook, to talk about:
Jocelyn has led engineering teams at early to growth-stage startups like VMWare and enterprise companies like Trilogy, so she’s definitely had her fair share of dealing with constraints and having to make tradeoffs to ship product and meet business goals.
We also dig into the cost of a mistake, how to take risks, the biggest mistake Jocelyn sees technical folks making over and over again, and how to avoid making it!
Watch the episode to learn how you can make smart tradeoffs when developing software products.
Viewers Challenge!
After you’ve watched the episode, take our challenge. Let us know in the blog comments below:
The three best responses will receive a special giveaway from our sponsor Pivotal Tracker and be showcased in Femgineer’s weekly newsletter!
Submit your responses in the blog comments below by March 19th at 11:59pm PST.
The next episode of FemgineerTV airs in April. I’ve invited Ryan Hoover and Erik Torenberg, the founders of Product Hunt, to talk about: How to Build a Community of Evangelists for Your Software Product. Subscribe to our YouTube channel to know when it’s out!
Poornima Vijayashanker: Welcome to the second episode of FemgineerTV, brought to you by Pivotal Tracker. I’m your host, Poornima Vijayashanker, and the founder of Femgineer. For those of you who aren’t familiar, Femgineer is an education company, and our goal is to help innovators build software products to give them more freedom in their careers, to enrich other people’s lives, and to make the tech community a lot more inclusive. When it comes to building software, we’re fundamentally just solving problems, and there’s always more than one approach to solving a problem, but too often, we have a lot of constraints, like time, money, and talent.
And when we’re dealing with all those constraints, we want to make sure that we’re taking the right approach, so in today’s episode, we’re going to be talking about how to make smart tradeoffs when it comes to developing software products, and to help us out, I’ve invited Jocelyn Goldfein, the former Director of Engineering at Facebook, to talk to us about how to make these tradeoffs. Jocelyn has helped companies from early-stage startups all the way to growth-stage startups and enterprise companies build software products and manage engineering teams, so she’s had quite a lot on her plate, and has been able to help these companies meet their business goals and ship products on time. So thank you, Jocelyn, for joining us today.
Jocelyn Goldfein: Oh, thank you for having me.
Poornima Vijayashanker: Yeah. It’s great to have you on the show and to be talking about tradeoffs. And we met a few years ago at a dinner, and you were just getting started at Facebook, so it’s wonderful to see that, after all these years, you’ve made such a positive impact on the company, and I also want to thank you for all your encouragement in helping young women pursue degrees in computer science and careers in engineering.
Jocelyn Goldfein: Thanks.
Poornima Vijayashanker: Yeah, thank you so much.
Jocelyn Goldfein: It’s definitely a cause that I care about a lot.
Poornima Vijayashanker: Awesome. Now, before we dig in today’s topic, I want to take a couple steps back and just ask you, what got you interested in building software, and what led you to then join Facebook?
Jocelyn Goldfein: Sure. I didn’t actually start out knowing that I was going to be interested in computer science. I was a nerd in high school, but I was a really well-rounded one, and I just got to college, and I took a computer science class fall quarter of my freshman year, and I thought, “Wow, this is really fun, and I’m actually pretty good at it.” And I just sort of went from there, but even by the time I graduated, I had no conviction about what I wanted to be when I grew up, and so I just kind of thought, “Well, I’d better get a job as a software engineer and find out,” and I had a chance to work for a lot of really interesting companies, including my own startup, and I was at VMware for…I joined when it was a pretty small, medium small company of just a couple hundred people, and I grew up from an engineering manager. I stayed seven years. The company was 10,000 people when I left.
Poornima Vijayashanker: Nice.
Jocelyn Goldfein: And I was the vice president of engineering by the end, and so really grew up at that company, and then wanted to keep growing and keep learning new things, and so it just seemed like a obvious choice to me to walk away from everything I knew about making enterprise software and operating systems and go work for a consumer web app, and so I joined Facebook about four years ago and had an absolutely great time.
Poornima Vijayashanker: Yeah. So now, given your extensive background, both at early-stage to growth-stage startups, and building enterprise products, you’ve had to deal with a lot of tradeoffs, and in a very recent talk that you gave, I remember you talking about this thing called the tradeoff space, so let’s start by digging into, what exactly is the tradeoff space?
Jocelyn Goldfein: Sure. Like maybe you’ve worked with an engineering manager who said, “Oh, features, quality, schedule. Pick 2.” You have kind of a bad rap for saying things like that. I’m guilty of saying things like that.
Poornima Vijayashanker: Sure.
Jocelyn Goldfein: But it’s actually a lot more complicated than that. First of all, I mean, it’s not “yes” or “no.” You’re going to have some features, and you’re going to have some quality, and you’re going to have some schedule, but there’s also a lot of other things we care about, too. We want our product to be beautiful and easy to use, and we want it to be speedy and responsive, and we want it to be reliable and never have downtime, and we want it to have exactly the features that are going to make customers buy it. And I could go on and on, and all those things are good things, but the one thing that is finite is time and energy.
Poornima Vijayashanker: Right.
Jocelyn Goldfein: And so no matter what you do, no matter what release process you follow, no matter what culture you have, there’s no silver bullet. You can’t have everything all the time, and so getting some of each of the things we care about is a choice, and one I believe that we should make consciously, and so how much of each of those things you do is what I call the tradeoff space.
Poornima Vijayashanker: Got it. Now, when we look at the space, you’ve listed probably like 10 things that we need to think about, right? It’s really easy to get overwhelmed by it, whether we’re a manager, or whether we’re the individual contributor. How can we alleviate some of those feelings? How can we maybe reduce the problem space a little bit so that we can ship product and meet our goals?
Jocelyn Goldfein: Yeah. Well, one interesting side effect, for me, of working at such different companies, and on such different kinds of products, was I found that different environments, different technology stacks, really lend themselves to different solutions. For example—I think this will be pretty familiar to most engineers—the technology stack you’re using really makes some things easy and some things hard. If you’re on the web at the top of the stack, if you will, you get a lot of reuse. You get to sort of build a lot of functionality very quickly, because you’re relying on frameworks that other people have built. You get to deploy your code on your own servers, so you don’t have to worry about testing it in a wide environment, and if you want to make a change, you just update your own hardware, because you have control of updates.
You start working your way down the stack to middleware, to native, mobile, and desktop apps, even to operating systems or, God help you, hardware, and you get more and more control as you move down. The reason a lot of people make native apps for phones is because on the phone, you can send push notifications. You can get access to the camera with a native app. You can’t do that with a web app, but you also give something up when you go down the stack. You might be able to…you won’t get as much reuse, and as soon as you deal with installing software on somebody else’s device, you’re going through a third party to certify your application, or you’re sending a CD to someone that they have to install…
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: …now you have a lot less control over your update cycle, and so you have to think really differently about how frequently you can update.
Poornima Vijayashanker: Yeah. I know for myself, when I was getting started, my first job was actually designing electronic design automation software, so it was software for chip designers like Intel and AMD, and this was probably millions and millions of lines of code and was definitely a native app, and the sort of technology tradeoffs that we had to make were very different from my second company, which was Mint.com, right?
Jocelyn Goldfein: Mm-hmm.
Poornima Vijayashanker: Consumer internet app, web applications, so we had different ways of thinking, given our two technology stacks, so it became very, very apparent that difference of enterprise versus consumer, and most importantly, like where you are in terms of the stack.
Jocelyn Goldfein: Yeah, and I definitely think, Facebook’s well known for the phrase, “Move fast and break things,” and I think that makes total sense when, if you break something, you can update the software on your own web server. It’s not quite that simple if you’re shipping an operating system to somebody’s data center that may get updated months or years later.
Poornima Vijayashanker: Yeah. So I know that, as engineers, it’s really easy for us to think about the technology stack and develop an intuition and be able to make tradeoffs and decisions quickly, but a lot of times, we aren’t aware of the business model we’re operating under and we either get really frustrated with the way management’s making decisions, or we don’t have an understanding. What are some ways in which, as engineers or technical folks, we can develop that intuition or some understanding so that when we do have to make tradeoffs, we keep that in mind?
Jocelyn Goldfein: Yeah, I mean, I think there are sort of two sides to any decision you’re making about investing your time and effort in doing something, whether it’s a performance speed up or a test suite, and it’s, “What does it cost me, and how am I going to benefit?” Like, does the benefits outweigh the costs, right?
Poornima Vijayashanker: Right.
Jocelyn Goldfein: It’s kind of simple.
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: And I think the technology stack tells us a lot about what things cost, and I think you’re right. Engineers have great intuition for what things cost, but I think to make good decisions, you’ve got to not just understand the costs, you’ve also got to understand the benefits. And I think mostly, those are driven by your business model, so if you’re building a product for enterprises that are spending a lot of money on your software, and rely on it every day to do their jobs, then reliability is really important to them, and that’s really important to your company having a good reputation and meeting your users’ needs. At the other end of the spectrum, if you’re doing a consumer app, maybe a free app, people’s expectations are a lot lower when they aren’t paying you a lot of money for something, but because it’s not a need-to-use, it’s a want-to-use, actually, the UX, the sort of delight and enjoyment of using your app needs to be much, much higher for people to sort of opt in to wanting to use it. And so I think that’s one reason why even though consumer apps are free or very inexpensive compared to enterprise software, they’re generally much more beautiful and much more fun to use, although we have a new breed of startups that are sort of challenging that received wisdom and seeing what happens when we take consumer design philosophy and bring it to enterprise software, and is that able to make a real compelling difference? And I think it will, if we find that it’s able to meet employee needs better, and able to achieve business goals better, but knowing if that’s a good tradeoff or not, to spend a lot of time on the design of your enterprise software, means understanding it, because that’s what matters the most to your users.
Poornima Vijayashanker: Yeah, so that’s a kind of new, novel experiment people are doing, huh, where they’re actually bringing some of that rich UI back to…
Jocelyn Goldfein: The enterprise.
Poornima Vijayashanker: The enterprise. That’s interesting. It’s really interesting, because, like I said, when I was working building EDA software, it certainly was not, by any means, beautiful.
Jocelyn Goldfein: Yes.
Poornima Vijayashanker: Like not even the codebase was beautiful.
Jocelyn Goldfein: Yes.
Poornima Vijayashanker: But it worked really well, and like you said, very, very reliable, because at the end of the day, this is going to go and design 90, 40-nanometer technology, so it needs to be precise, right?
Jocelyn Goldfein: Mm-hmm.
Poornima Vijayashanker: There are so many things downstream that are impacted by it but didn’t have a beautiful user experience, and was still functional, and made millions of dollars for the company, or actually billions of dollars.
Jocelyn Goldfein: Yeah.
Poornima Vijayashanker: And then contrast that to Mint, which was consumer internet, and free application, had a very, very beautiful user experience and user interface. So there was obviously a difference here, and part of that difference also came down to the customer and what their needs were.
Jocelyn Goldfein: Yeah.
Poornima Vijayashanker: Right? So can you elaborate a little bit more on like, how can we take back some of the customer needs and apply that to the way that we think about tradeoffs as we’re building?
Jocelyn Goldfein: Yeah. I mean, I think that in some ways, to an engineer, it’s counterintuitive that the web app has the more beautiful UI than the native app. If you think about it, the further down the stack you go, the more control you have over pixels. I mean, if you want to create the richest, most immersive user experiences, you try to bypass the operating system, even, and that’s where we see video games, or, in this day and age, I’m really excited about what Oculus Rift is doing, and to deliver those incredible user experiences, you actually have to go very low in the stack. But at the same time, it’s not a technology-driven choice alone. Even though your EDA app was native software, it was lower in the stack. It had the capability to deliver a more beautiful UI than an off-the-shelf kind of web app. Nonetheless, that wasn’t what mattered to your customers. What mattered to your customers was that hardware design, actually have to go very low in the stack, but at the same time, it’s not a technology-driven choice alone. Even though your EDA app was native software, it was lower in the stack. It had the capability to deliver a more beautiful UI than an off-the-shelf, kind of, web app. Nonetheless, that wasn’t what mattered to your customers. What mattered to your customers was that hardware designers made a design that got to manufacturing and produced actual, workable components they could sell to their customers, and if the EDA software did not produce…
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: …created tremendous manufacturing waste, then they were out of business.
Poornima Vijayashanker: Totally.
Jocelyn Goldfein: And in that case, your EDA software would also be out of business. Whereas with Mint, if it’s free, if it’s not a must-have, then people have to find it easy to use or they just won’t use it.
Poornima Vijayashanker: Right. They want it to be beautiful when they log into their account and see their money, or don’t see their money, but yeah, yeah.
Jocelyn Goldfein: Yes, yes.
Poornima Vijayashanker: That’s true.
Jocelyn Goldfein: And so I think a lot of times, we imagine that it’s just about philosophy, or personality, or this set of founders care about design and that set of founders don’t, but I really think, at the end of the day, as engineers and product people, we have to be motivated by the person using our software, and we have to remember that we cannot do everything in the world that we’d like to do, so we should focus our efforts on the things that matter most to that person.
Poornima Vijayashanker: That’s so true.
Jocelyn Goldfein: And I think that when we design for the person, we do our best work.
Poornima Vijayashanker: Nice. So we’ve talked about tradeoffs in the context of the technology stack, as well as the business model, and now, taking into consideration customers, but one, I think, real concern that people have is the cost of a mistake, right? You make a mistake, you could be out of business, or the product gets end-of-lifed, and so you’ve got to have a real appetite for risk, and I know, having worked at various companies and startups as well, you’ve probably come across this appetite for risk. How should people be thinking about that and relating it back to the cost of a mistake?
Jocelyn Goldfein: Yeah. We talk about risk and failure all the time in the Valley. It’s like, “Oh, in order to innovate, you’ve got to be willing to fail,” and I’m guilty of that, too.
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: You know? I really I do think that you’ve got to be willing to take risks to do something great, but it’s sort of easy, again, to think of appetite for risk as a character trait or a moral quality, and Mark Zuckerberg’s got more of it, so Facebook’s able to innovate, and maybe the CEO of VMware is very cautious or conservative. And again, I don’t think it’s a personality trait. If you think about what is the cost of a mistake, I think it comes in two parts. One is, what does it cost you? If you make a mistake, what does it cost you to fix it? And let’s assume you’re a going concern now, like you’ve got software, you’re established, but…so you’re going to have a chance to fix it. The cost of a mistake isn’t you’re out of business.
Poornima Vijayashanker: Right.
Jocelyn Goldfein: So there’s, what does it take to fix it, and that’s like some cost we can reason about. Like there’s a bug that makes it to production, we debug it, we code up a solution, we test our solution, we deploy the solution, right? Whatever that cost is, let’s call it X, OK? And then there’s another dimension to the cost, which is not what’s my cost to my business, but what’s the pain to my customer?
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: To my user of my outage, and that’s some pain because my software is not working the way they expect it to. And so I think that if you look at, again, these two dimensions of technology stack and business model, if your code runs on hardware you control, then that cost to fix X, is whatever X is, and then as soon as you introduce, as soon as you start going down the stack, as soon as you have to run on multiple other kinds of hardware, all of a sudden, you have to have a test matrix.
Poornima Vijayashanker: Right.
Jocelyn Goldfein: All of a sudden, you have to get Apple to approve your App Store update. All of a sudden, you have to get users to be willing to deploy your software, maybe just on their phone with an over-the-air update, or maybe it’s a hardware OEM that needs to get a CD to their factory and reimage a bunch of components as they come off the assembly line, OK? And so, I want to say is, as you move down the stack and you go from the code running on your servers to depending on other people, the cost of fixing it goes up by an order of magnitude, let’s just say 10x. We’re making broad, ballpark, sweeping assumptions. And then, let’s look at the pain to your users on the business model axis. We can joke about Twitter’s Fail Whale, but the truth is, if Twitter has a little outage, we can come back later. It’s not the end of the world.
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: And so that pain is X, but as you sort of move down to software that people are relying on, like email, they can’t communicate without it, or my revenue booking software, I can’t book sales, I can’t operate my manufacturing plant while your software is down, that cost, I would say, also goes up by an order of magnitude. The more people have spent on your software, the more they’re relying on it to do their job. And then I think if you kind of combine the two axes and you’re building, and you’re in VMware’s situation…
Poornima Vijayashanker: Oh, totally.
Jocelyn Goldfein: …and you are building system software that people have paid a lot of money for, I’d love to say it’s 10x plus 10x, but it’s actually geometric expansion, right? It’s 10x times 10x, so it’s two orders of magnitude more cost. It’s 100 times more, and so I think it is perfectly reasonable for free consumer web software to make a lot of mistakes and quickly fix them and to choose, even, a path to designing a product, which is to rapidly experiment. Whereas I think at the other end of the spectrum, if you are down in this other sort of quadrant where the cost of a mistake is two orders of magnitude higher, then it makes sense to have a much, much more careful process where you are really sure you have something exactly right before you let it out the door.
Poornima Vijayashanker: Yeah, so there’s a lot going on here, right? There’s a number of ripple effects. There’s the fact that your process also needs to be bulletproof or somewhat bulletproof. Where do you see the biggest mistakes that people make, then, when it comes to these tradeoffs?
Jocelyn Goldfein: I think we arrive at an intuitive understanding. I think whether people sort of are consciously thinking about their technology stack, and their business model, and, “Let me pick a software development process that optimizes for quality or that optimizes for UX,” I think we just kind of naturally, sort of, form follows function in a very Darwinian way, like whatever experiences we have, we find our way to what optimizes for that. The trouble is, as software engineers, we tend to get really religious.
Poornima Vijayashanker: Yes.
Jocelyn Goldfein: We think can have religious wars over like tab widths and text editors…
Poornima Vijayashanker: Right.
Jocelyn Goldfein: …and so when we find a way to deliver software that really works for the product we’re working on, we get dogmatic, and we think we found the one right way to do it. And I think that that’s fine, as long as you’re applying it to the same product, but what if your company needs to pivot from web to mobile like Facebook did? Or what if your free consumer game, your freemium consumer game, turns into an enterprise messaging app, like Slack? You may sort of pivot your way right out of what you were optimizing for, and if you have, sort of, religious or moral conviction that you were shipping software the right way, now you’re going to end up with something that’s really ill-suited for what you’re trying to do, where you’re going to spend a lot of time and energy on things that are expensive and maybe you don’t need to do. I think of Microsoft doing Bing, which is free consumer web software, and I think they could have moved faster and taken more advantage of iteration, but it was hard to shake off the roots, the culture, the values of a software company that was making operating systems and needed really predictable schedules. And at the same time, I know from firsthand experience it was really hard for Facebook to move from web to native apps, and that is not because Objective-C is a more difficult programming language than PHP, it’s because of the culture that surrounded the way we released software, and that move-fast-and-break-things culture was dangerous when you—
Poornima Vijayashanker: When doing mobile.
Jocelyn Goldfein: —took it to a native app, which you cannot update twice a day. Which is, by the way, the rate at which Facebook updates its web app is twice a day.
Poornima Vijayashanker: Yeah.
Jocelyn Goldfein: And so I think that everybody’s going to try to make these tradeoffs, and I would encourage you to be self-conscious, and reflective, and do it, and sort of write down what you think is important to you and what is easy to you and figure out if you’re doing stuff that’s expensive and not important and stop doing it, because that’s an opportunity for a big win. I’d encourage you to figure out if things are really important to you and you could optimize for them more, but at the end of the day, the most important thing, realize that your release process is not a matter of ethical or moral right and wrong, that it’s just a pragmatic choice shaped by what works well for the product you’re doing now, and be prepared to pivot. Be prepared to adapt when your technology stack changes, when your business model changes.
Poornima Vijayashanker: So any final parting words for our viewers?
Jocelyn Goldfein: Well, I would say, most of all, software is really a team effort.
Poornima Vijayashanker: Right.
Jocelyn Goldfein: And so even if you’ve watched this video and kind of absorbed the tradeoffs, and you’re inspired to be adaptable, or to apply the right process, or the right values, you’ve got a whole team to think about, not just yourself. And so if you’re bringing in leaders who are going to set the tone and set the engineering culture at your company, I think being really conscious that they either come from a background that emphasizes the same priorities as your company, or that if they come from a different background, they know how to adapt, that you’re onboarding them really thoughtfully, and that you’re teaching people your engineering values so that you’re all, kind of, optimizing for the same stuff, and not making tradeoffs that cancel each other out.
Poornima Vijayashanker: Yeah, that’s a good point, and so part of that is having those engineering values, and sitting down, and writing them out if you don’t yet have those.
Jocelyn Goldfein: Yeah. Yeah, absolutely, just get everybody rowing the same direction.
Poornima Vijayashanker: All right, so just to recap, we’ve covered the topic today of how to make smart tradeoffs when you’re developing software products, and we talked about the tradeoff space, and while that can be really big and filled with many, many different dimensions, you’ve got to narrow it down to, ultimately, what’s a priority for you and your business, and so part of that is considering your technology stack. And it’s really easy, as technical folks, to develop an intuition, but we’ve also got to think about the business model that we’re operating under and factor that into the decisions we make. And then, finally, we talked about the cost of a mistake and how we want to think about that in terms of the level of risk that we can absorb as a company. Finally, we have a challenge for you. We want to know when was the last time you had to make a decision when it came to tradeoffs? What did you have to consider? What was the cost of the mistake, and what was your appetite for risk or your company’s?
Let us know your response in the comments below, and the winner of the best response will get a special giveaway from Tracker and be hosted on Femgineer’s weekly newsletter. So I’m really looking forward to hearing what everyone has to say and reading through their responses to get an understanding of the tradeoffs and the decisions you’ve had to make when building products. So I’d like to thank our special guest, again, Jocelyn Goldfein for joining us today. Jocelyn, how can people get in touch with you, or how can people learn more from you?
Jocelyn Goldfein: Well, I keep a blog which you can find at jocelyngoldfein.com, or you can always follow me on Twitter @jgoldfein.
Poornima Vijayashanker: And thanks to all of you for tuning in today. I’m really looking forward to your challenge responses, and of course, to our wonderful sponsor, Pivotal Tracker, for helping to produce this episode, and if you’ve enjoyed this episode, then please share it with your friends, your teammates, and your boss, and subscribe to the YouTube channel to know when the next episode will be out. We’ll have special guests, the founders of ProductHunt. Thanks.
Femgineer’s Confident Communicator Course will be opening up soon. This course is only taught once a year, so you won’t want to miss out. For more information and resources, visit femgineer.com/confident-communicator-course.
This pilot episode of FemgineerTV is brought to you by Pivotal Tracker—build better software faster.
FemgineerTV is produced as a partnership between Femgineer and Pivotal Tracker. San Francisco video production by StartMotionMEDIA.
