The U.S. military’s technological edge is increasingly defined by software: It rapidly creates new capabilities for tanks, planes, and weapons systems to outsmart and outmaneuver the enemy without fielding new hardware. While America’s adversaries pour billions into artificial intelligence and machine learning, the Department of Defense has yet to fully embrace its most powerful ally: the world-leading innovation of America’s commercial tech sector.
Take the department’s many software factories like the Air Force’s Kessel Run and Space Force’s Space Camp. These teams are tasked with developing and debugging mission-critical software, which requires software development platforms with built-in tools and services to help programmers. A good platform helps these developers create and release higher-quality software faster. Despite plenty of mature, commercially developed platforms, government teams too frequently fall into the trap of building platforms in-house to optimize costs. However, an agency can only achieve cost optimization if its alternative solution works, and in our experience, that’s a bad assumption.
At face value, there are sound arguments for taking a do-it-yourself approach to a platform. Agencies can own and control development processes and avoid feeling locked into a vendor’s product and unable to change to a different solution. In theory, an agency is investing in something it owns and can customize to meet unique workflows and security requirements. In reality, building platforms yourself is an incredibly costly, time-consuming pursuit — one that is likely to fail without sufficient expertise and resources. And yet, agencies are willing to pay for full-time employees and operating expenses, including more resources if timelines start to slip. Why? To avoid the ever-dreaded lock-in. Teams end up focused on building infrastructure when they should be focused on building timely mission-critical applications. Usually, however, they end up locked out of operations.
The paradox is that when the government does succeed in building software platforms, they end up locked into their homegrown solution. When it comes to software, don’t waste your time and resources building a solution you can easily buy.
It should be said upfront that as senior software executives in the private sector, we may have a commercial stake in this debate. Our perspective, however, is based on nearly 30 years of combined experience building software within government and the private sector, where we have witnessed how the preference to build in-house has invariably led to exacerbating, not solving, the critical issue of lock-in. Furthermore, while one of us works at a software company that provides commercial platforms to government agencies, the other is actually in the business of helping the government build its own software solutions in-house. One could assume, therefore, that in this regard we have competing business interests. Yet we are completely aligned where it matters most — in helping the Defense Department solve one of its greatest challenges through public-private partnership.
Based on our experience, it requires acquisition changes and significant cultural change within the department where officials are as comfortable relying on software providers as much as they depend on hardware providers that build tanks and planes. The first step of that change would be to start thinking critically about the pros and cons of some level of lock-in to technology solutions rather than avoiding it at all costs. In most cases, defense agencies can get their teams from blueprint to production faster and more cost-effectively by using a commercial platform instead of building their own.
The Sweet Spot of Flexibility
All technical choices have some degree of lock-in as a consequence. Instead, the government should focus on flexibility by managing switching costs, or the cost to switch to an alternative approach. There is a meaningful difference between technical decisions with two-day and two-decade switching costs. And through the hyper-competition in commercial markets, commercial platform providers are forced to demonstrate this. Agency officials are trained to avoid vendor lock-in, particularly to prevent getting stuck with a lousy partner or getting gouged on prices. And it’s true: You can get locked into a specific product like a database, an architecture like Kubernetes, or a particular cloud or platform vendor and its various services. But lock-in comes in many forms, including homegrown solutions and open-source technologies. Both have the potential to become lousy partners and lead to high costs.
Every technology decision weighs value versus tradeoffs. Each provider will offer unique features their competitors don’t. When those features drive real benefits or time savings to your organization, some level of lock-in may be acceptable, if not inevitable. The benefit is measured in both capability and time.
In the military, it is often the case that the cost of delay is measured in lives, and nothing could be more expensive. In the case of Kessel Run’s famous Tanker Planner Application, upon delivery, it immediately began saving $214,000 a day in fuel costs. But think about the alternative: Every day of delaying that app getting to the field “cost” the Air Force $214,000.
Agencies tend to spend more on making systems flexible than the actual switching cost. While agencies must reduce potential liability, there is a point of diminishing returns when investing to reduce lock-in. The key is to find the sweet spot between being vulnerable to high switching costs and spending too much to dodge unlikely scenarios. Legendary computer scientist David Knuth would remind us that “premature optimization is the root of all evil.”
The Challenges with Do-It-Yourself
Good platforms reduce the cognitive load on product teams, improve reliability, accelerate development, reduce risks, and enable more cost-effective use of services.
Unfortunately, plenty of custom-built platforms do the opposite. Some of them just plain suck. They’re only used because of enterprise mandates, not because teams respond to their needs. Part of the reason is that platforms are still products that require proper product management, user experience design, and documentation. Frequently, agencies don’t realize the full extent of resources necessary to maintain reliability and evolve to meet user needs. They think platforms are cheaper to build in-house, but that’s only because they’re seriously underestimating the cost and commitment of maintaining those platforms. And this is the advice they get from systems integrators and contract labor providers. While the bias of platform vendors is obvious, labor providers also have an incentive to commoditize their complements to capture a greater share of wallet, like IBM did in the 1990s.
Building a platform is challenging. It typically comes with a high price tag and can take five to 10 years to reach maturity. The talent required isn’t cheap. Agency do-it-yourself platform strategies often start out with estimates as low as 20 full-time employees. Using even these low numbers — a team of 20 and an hourly contractor rate of $175 — puts it in the ballpark of $7 million per year for labor only. Rise8 has seen these small teams quickly buckle under the demands of both new capability needs and Day 2 operations at scale, and then balloon by a factor of 10 more. That $7 million estimate turns into $70 million — money that otherwise would have been spent on mission applications.
Management, licenses, subscriptions, other direct costs, and full-time employees would all be expenses on top of that amount. Also, add in time lost to predictable delays such as onboarding, change approval boards, and obtaining Authority To Operate.
In fact, the agency has hopped into the platform business where its commercial competitors devote billions of dollars and thousands of developer hours to create new capabilities. The licensing fees procurement officials often balk at are a fraction of the enormous costs of research and development spread out across a vast customer base. The economies of scale allow technology companies to recoup their upfront costs while giving customers a more affordable product. It takes creative accounting — like counting government developer time as “free” — to create a platform for less than a commercial company’s licensing costs. Even then, it is nowhere near comparable in terms of capability or rate of evolution.
Looking at publicly available data for Palantir, it took over $5 billion in losses to build its Gotham and Foundry operational intelligence platforms. Platforms can only be built when you amortize the costs across many customers. That is to say: The government can’t realistically build its own platform cost-effectively, if it can be done at all. When you build your own, you are implicitly saying that ownership, not cost, is the motivating variable.
Are You Locked Into a Vendor or Locked Out of Outcomes?
Good engineering costs a lot, but bad engineering costs much more. The uncomfortable truth about many do-it-yourself platform projects is that many never get into production, and those that do lack maturity. These unsuccessful efforts waste time, money, and effort, effectively locking organizations out of their DevOps journey (bridging the gap between development and operations through automation, collaboration, and continuous delivery) and preventing them from realizing the benefits of modern software development practices.
We discuss cost, schedule, and performance in federal acquisition, but the overriding consideration should be value. Value is performance divided by the product of schedule and cost:
When building a new capability, performance should also factor in the likelihood of achievement and cost should factor in both effort and sacrifice that goes beyond dollars.
Building a platform is a high-effort, high-cost endeavor with a long product implementation timeline and low likelihood of achievement, making it a low-value activity for an agency.
According to our research and experience, a true subscription platform as a service model can save agencies 50 percent to 75 percent as compared to the build example we provided above. It also offers faster time to value and a higher likelihood of success while focusing agency development teams on mission-critical applications. If needed, agencies can add custom layers for specific needs, but the point is to focus agency resources on delivering capabilities to users, not changing the underlying platform infrastructure or the shibboleth of “affordability.” What America can’t afford is to lose on the battlefield.
The path to effective DevOps implementation in government agencies doesn’t have to be paved with expensive do-it-yourself platform projects. The future of government tech lies not in building every component from scratch, but in strategically adopting and adapting proven technologies. This approach saves time and resources but also positions agencies to harness the full potential of modern software development practices, ultimately leading to better services for citizens and more efficient government operations. It positions the United States to be the disruptor, rather than the disrupted, on the battlefield. It positions America to win.
Shyam Sankar is the chief technology officer of Palantir Technologies.
Bryon Kroger is the founder and chief executive officer of Rise8.
Image: Luke Allen
Comments are closed.