Adam Jacob on Why Scaling Is ‘The Funnest Game’

“There’s no funner game than the at-scale technology game. But if you play it, some people will hate you for it. That’s OK … that’s the game you chose to play.” — Adam Jacob At Monster Scale Summit 2025, Rachel Stephens, research director at RedMonk, spoke with Adam Jacob, co-founder of Chef and CEO of System Initiative, about what it really means to build and operate software at scale. Monster SCALE Summit 2026 will go live March 11-12, featuring antirez, creator of Redis; Camille Fournier, author of “The Manager’s Path” and “Platform Engineering”; Martin Kleppmann, author of “Designing Data-Intensive Applications” and more than 50 others. The event is free and virtual. Register and join the community for some lively chats.  The Existential Question of Scale Stephens opened with an existential question: “Does your software exist if your users can’t run it?” Yes, your code still exists in GitHub even if us-east-1 goes down. But what if: Your system crawls under load. Critical integrations constantly break. You can’t afford the infrastructure costs. “Software at scale isn’t just about throughput,” Stephens said. “It’s about making sure that your code endures, adapts and remains accessible no matter the load and location of where you’re running. Because if your users can’t use it, your software may as well not exist.” With that framing, Stephens brought in someone who’s spent his career dealing with scale firsthand: Adam Jacob. Only Scale When It Hurts Stephens asked Jacob how teams can balance quality, speed and scale under uncertainty. How do you avoid both cutting corners and premature optimization? Jacob argues that early on, it’s fine not to worry much about scale. Most products fail for other reasons before scalability ever becomes a problem. He explained: “I think of it basically through the lens of optionality. When you start building new things, it’s nice not to worry too much about scale, because you may never reach it. Most products don’t fail because they fail to scale. Think about how badly Twitter failed to scale … and yet here we are.” The first priority is to build a solid product. Once scale becomes a real issue, that’s when it makes sense to refactor and remove bottlenecks. But if you’ve been around the block a little, your experience helps you make early choices that pay off later. Jacob noted, “Premature optimization is real. But as you gain experience, there are some decisions you make early because you know that if things work out, you’ll be happier later — like factoring your code so it can be broken apart across network boundaries over time, if you need to.” Chef Scalability Horror Stories Next, Stephens asked Jacob if he would share a scaling horror story from his Chef days. Jacob obliged and offered two memorable ones. “The best was when we launched the first version of Hosted Chef. The day before the launch, we discovered it took about a minute and a half to create a new user. It didn’t take that long when we were running it on a laptop, but it did later … and we never really tested it. So, in the final hours before launch, we changed it from ‘anyone can sign up’ to a queue system with a little space robot saying, ‘Demand is so high; we’ll get back to you.’ We just papered over the scalability problem.” “Another example: That same Chef server (the one that couldn’t create accounts quickly) eventually had to work at Facebook. The original version was written in Rails, which was great to work with, but not parallel enough. At Facebook scale, you might have 40,000 or 50,000 things pointed at one Chef server. So we rebuilt it in Erlang, which is great for that kind of problem. I literally brought the Erlang version to Facebook on a USB stick. When we installed it and bootstrapped a data center, we thought it was broken because it was using less compute and finished almost instantly.” Jacob explained that if they’d tried to build the Chef server in Erlang from the start, the project probably wouldn’t have gained traction. Starting in Rails made it possible to get Chef out into the world and learn what the system really needed to do. Only later, once they understood how the system really behaved, could they rebuild it with the right architecture and runtime for scale. Growth or Efficiency: Know Which Game You’re Playing At Chef, scaling was ultimately required to land customers like Facebook and JPMorgan Chase, which operate at massive scale. Jacob advised, “Making it scale required major investment, but it worked. You can’t buy your core. If it matters to customers, you have to build it yourself. People often wait too long to realize they have a deep architectural problem that’s also a business problem. Rebuilding for scale takes months, so you have to start early.” Your own approach to scale should ultimately be driven by what game you’re playing: In the venture capital game, growth and traction come first. You can spend money to scale faster because you’re funded. In the profitability game, efficiency comes first. Overspending on compute or poor architecture hits the bottom line hard. Why Scaling Is the ‘Funnest’ Game Stephens mentioned that “when software succeeds, it stops being yours — it becomes everyone’s.” She then asked Jacob what it’s like when your tech scales to the point that people have extremely strong opinions about it. His response: “It’s hard to build things that people care about. If you’re lucky enough to create something you love and share it with the world and people love it back, that’s incredibly rewarding. Even when they don’t, that’s still a gift. “Someone once tapped me in a coffee shop and said, ‘You wrote Chef? I hate Chef.’ I said, ‘I’m sorry; I didn’t write it to hurt you.’ But at scale, that means he used it. It mattered in his life. And that’s what you want: for people to experience what you built. “I love the technology, the problem, the difficulty. Scaling adds more layers of complexity, more layers of fun. There’s no funner game than the at-scale technology game. But if you play it, some people will hate you for it. That’s OK … that’s the game you chose to play.” You can see the full talk below. The post Adam Jacob on Why Scaling Is ‘The Funnest Game’ appeared first on The New Stack.