Process Theater vs. Technical Excellence: A Recurring Software Crisis

The term “snake oil salesman” is often used to describe individuals who engage in deceptive marketing practices. Wild west characters like Clark Stanley advertised their snake oil as a wondrous cure-all remedy. But in 1916, the U.S. government’s Bureau of Chemistry tested the liniment, found it to be dramatically overpriced and of limited value, and Stanley was fined $20. Yet that’s not the end of the story. You Probably Use Snake Oil Snake oil wasn’t entirely purposeless. While it’s true that it didn’t match the claims on the bottle, certain ingredients, such as capsaicin and camphor, proved valuable when used for valid purposes. Capsaicin, derived from chili peppers, is now used in skin-applied pain relief products to relieve muscular and joint pain. It’s an FDA-approved therapeutic treatment. Camphor is also commonly used as a counter-irritant, helping relieve itching from insect bites. It’s also the go-to ingredient for makers of chest rubs, which you’ve likely used as a decongestant when you’ve had a cold. So, while snake oil failed to match the wild claims of its peddlers, it wasn’t completely useless. This is also true in the software industry, so being able to separate the valuable ingredients from debunked software delivery recipes is a crucial skill. Waterfall Is Overpriced and of Limited Value The term “waterfall” is often used as a catch-all name for phased software delivery, where tasks are performed in a sequential order that resembles a waterfall. When the lightweight rebellion overthrew the heavyweight models of the time, it created a mistaken belief that the phased software models were simply wrong. But the creators of these old models have been short-changed, since they’ve been telling us to work in this new way all along. Source: Production of Large Programs. Herbert D. Benington. 1956. In his 1956 paper, “Production of Large Programs,” Herbert Benington discusses concepts that we would now label as platform engineering. Benington decried the idea of top-down programming, in which a specification is completed before the code is written. Winston Royce, in his 1970 paper “Managing the Development of Large Software Systems,” advised people to work in small incremental changes, as this would reduce complexity and allow organizations to roll back to a previous version if they moved in the wrong direction. These ideas resurfaced in Barry Boehm’s Spiral Model. The success of Agile was largely due to how the proponents of lightweight software delivery carefully extracted the good ingredients from the heavyweight snake oil recipe used in the popular processes that dominated the industry in the 1990s. They preserved the good parts and discarded large swathes of the toxic ones. Understanding this intense filtering process is our path out of a crisis that keeps repeating itself in our industry. The Recurring Crisis The crisis comes from a tendency for management to be attracted to process and repelled by technical and cultural practices. They have a craving to reintroduce elements of waterfall that were expertly removed, and they want to discard crucial techniques that they don’t understand (or sound like hard work). Increasing process weight while decreasing technical excellence is a path to destruction. The canonical example of this management error comes from the early days of Agile. Around the time of the Agile Manifesto, the leading lightweight method was Extreme Programming (XP). It had similar process elements to Scrum, along with a map of interconnected technical practices that kept the cost of change low, which is key to sustaining agility over the long term. For managers, Scrum’s exclusive focus on process was unthreatening, while XP’s emphasis on technical skills struck fear into their hearts. When it came to management, Scrum was top dog. As a result, we spent a decade spinning wheels until Dave Farley and Jez Humble revived and renewed the ideas of XP in their landmark book, “Continuous Delivery.” Of course, it didn’t stop with Scrum. When you don’t have technical excellence, the process elements of Scrum don’t deliver the outcomes that are expected of agile software development. As a result, management responded by bulking up the process to “work at scale” or “handle Enterprise needs”. The real motivation behind this was, of course, the comfort of process working against the complexity of reality, which can only be resolved by social and technical means. When DevOps first emerged, it could be summed up as breaking down the silos between development and operations. This idea was further refined to align the two teams‘ goals and encourage them to collaborate more effectively. Everyone was on board with this until a decade of research revealed the need for those intimidating technical elements, the necessity of transformational leadership, and the value of lean product management. When DevOps got too real, the desire to run away intensified. The rush from complex realities to simplifications is a mistake we repeatedly make. Putting it unkindly, the fall of all good methods is the result of managers fleeing in terror from things they don’t understand as well as they should. Finding the Real Remedy The software industry’s snake oil problem isn’t that we have too many techniques and practices; it’s that we have too few. It’s that we’ve lost the ability to think critically about them. We adopt wholesale when we should cherry-pick. We follow prescriptions when we should experiment. The most effective software teams aren’t the ones who’ve found the perfect framework. They’re the ones who’ve learned to extract value from imperfect ones, who understand that every practice is context-dependent, and who continuously question whether what they’re doing is actually helping. Snake oil taught us an important lesson, but it wasn’t the one we thought. It’s not that old remedies are worthless. It’s what we need to look past the marketing to understand what actually works. The same applies to software practices. Behind every framework, methodology, and best practice lies a kernel of insight that addresses a real problem. Our job isn’t to mindlessly follow or unthinkingly reject. It’s about understanding, extracting, and applying wisely. The post Process Theater vs. Technical Excellence: A Recurring Software Crisis appeared first on The New Stack.