The Pentagon still lacks clear ways to measure if the organizations charged with innovation in the Department of Defense actually work.
In 1964, the federal government funded 67 percent of research and development in the United States. By 2020, that figure had fallen to 21 percent. With defense budgets remaining relatively flat, it is more important than ever for the Department of Defense to adopt commercial technology.
For 10 years, the Department of Defense has been standing up defense innovation organizations to make itself a better partner to tech companies. The Defense Innovation Unit was created in 2015, followed by the Chief Data and Artificial Intelligence Office, the Office of Strategic Capital, AFWERX, the Army Applications Laboratory (which I lead), NavalX, and others. The theory behind these organizations is that by eliminating barriers to entry, they can more effectively pivot commercial technology to solve military problems. But how does the Defense Department know if these organizations are succeeding?
Even in the commercial sector, measuring innovation is a challenge. In the Department of Defense, each innovation organization has a slightly different mission and does not judge success the same way. Furthermore, each stakeholder for whom innovation organizations must show value — Congress, military service leadership, companies and their investors — is concerned with something different. No one metric suits all needs.
Based on my experience at Army Applications Laboratory, I think defense innovation organizations should be grading themselves on three things. The first is impact on the warfighter: Are they transitioning usable technologies for important problems quickly? Second is impact on the civil innovation base: Are they motivating more work among companies who don’t usually work with the Department of Defense? Third is impact on the Department of Defense: Are they developing best practices that get adopted by other organizations? Which matters more depends on the specific mission of each innovation organization.
At the Army Applications Laboratory, our mission is to get the best technology in the hands of warfighters faster by expanding access to the civil innovation base. So, for us, our impact is primarily measured by technology transition and identification and support of companies with limited defense experience. An organization whose mission is to get technology to the warfighter in two years or less will focus most on the impact of their technology projects. What follows is a guide for how innovation organizations can measure all three.
Impact on the Warfighter
Impact on the warfighter is the return on investment for projects. It aims to measure if an organization is working on important or tough problems; transitioning technology that helps solve those problems and, if so, to where; decreasing the cost of developing technology or the time it takes to get to warfighters; and how likely warfighters are to use the technology.
In the commercial sector, return on investment is measured through the equation: profit equals revenue minus cost. It’s not that easy for the military. Many innovation organizations track technology transition rates, the amount of co-investment and follow-on contracts, and how long it takes from a project’s start until the technology is in warfighters’ hands as surrogates. All are good to know. But none of these benchmarks track warfighter impact. For that, the innovation organization must establish the military utility of a project and if their solution is one that warfighters are going to use. Military utility relative to cost is the return on investment for a defense innovation project.
The first step in determining military utility is developing the military business case, which clearly lays out the military problem the project will address, the utility of the technology when applied to that problem, and how to know if it is working.
By explicitly defining the military utility at the beginning of a project, solvers can ensure experiments generate data to support or refute their theory of success. Does the data suggest a tenfold improvement in battlefield capability or a 67 percent decrease in maintenance costs? Defining those metrics early makes it easier to be intellectually honest. Without that, the inclination is to choose the most attractive metric to declare success at project completion.
At least as important is that, by articulating the business case, the innovation organization develops a hypothesis it can test at the outset. Is the supposed problem real? Would the warfighter use the solution? Would those who buy things on the warfighter’s behalf buy this? Unless the answer to all the above is yes, resources are better used elsewhere.
Writing down the military business case is just the beginning. To make investment decisions, leaders need a way to clarify apples-to-oranges comparisons. How does the military utility of a battlefield capability like a munition compare to that of an institutional improvement like facilities cost savings? To determine this, innovation organizations should ask, on a scale from one to five, to what extent would the technology improve military advantage over adversaries? Is it minor, moderate, considerable, significant, or decisive?
Admittedly, this is more art than science. Some capabilities, like loitering munitions with 20 times the current range, are easy to put on this scale. Others, like facilities improvements or better training tools, are harder. But it can be done. A technology that cuts building construction costs will score well if the savings allow facilities to be built that otherwise would have been unfunded. A logistics solution that increases vehicle fleet readiness by several percentage points could also score well because it would get vehicles back in the fight sooner.
Even if organizations are solving important problems, there’s a case to be made that an innovation organization’s transition rate for those technologies can be too high. In my view, for an organization focused on technologies that are somewhere between early-stage proof of concept and minimum viable product, the Army Application Laboratory’s historic transition rate is too high at 47 percent. It shows we might not be accepting enough technical risk, accepting double instead of striving for tenfold solutions.
Plotting military utility scores against technical risk for projects can tell an innovation organization a lot about the kinds of problems it prefers, and its risk tolerance. It can also tell them if they are playing it safe. High-value and low-risk projects are ideal, and some of those solutions are ones only an innovation organization is likely to find. But, when the price is right, high-value and high-risk projects as well as low-value and low risk projects make good sense.
The technical risk assessment should account for different variables, depending on the goal. The Army Application Laboratory brings technologies to the level of a minimum viable product and then puts them on a path to the warfighter. While we would consider manufacturability, we would not consider a company’s manufacturing capacity. In cases where an organization is focused on reaching first unit equipped, something akin to the Department of Energy’s adoption readiness level framework could be used.
User adoption is the last key to military utility. A great solution to an important problem is no solution at all if the people meant to employ it do not agree. They simply will not use it. Army Applications Laboratory gauges this with the CONEX metric.
The CONEX is a standardized metal shipping container. To be ready to deploy on short notice, Army units keep the containers on hand and use them for storage. When the Army gives units tools they do not find useful, the CONEX is where they stay.
Understanding this, when getting unit feedback on technology, the Army Application Laboratory asks soldiers to rate, from zero to 10, how likely they are to take it out of the CONEX and use it under three conditions: if provided as designed, if provided as shown to them that day, and if provided as it would be if engineers incorporated the feedback the soldiers had provided.
The first question translates to, “If this does what we say it will, how likely are you to use it?” We ask this because, if they’re using a very early version, we don’t want them to score it as if it is going to be the final solution. The second question — whether they would use the technology in its current state — is useful for two reasons. First, we can see if the metric improves over time. Second, it could be a demand signal for procurement.
The last question asks, “If this does what you want it to do, how likely are you to use it?” We ask that because sometimes if a minor detail — like the button on the left should be on the right — is tanking user adoption scores, we want to know.
Over the lifetime of a project, the CONEX metric should improve: the mean should increase, and the standard deviation should narrow. When we are confident that over half of units would rather have the tool than not, it may be time for a limited issue to allow continuous experimentation. If the CONEX metric stays low, we may be solving the wrong problem or offering an inadequate solution. It could also be a signal to divest.
A common critique of innovation organizations is that their efforts are not tied to acquisition programs — the most common way to get technology to the warfighter. This is true. For a thorough description of the challenge, see the 2023 RAND research report, “Strengthening the Defense Innovation Ecosystem.” But a traditional program of record is just one way to get technology to the warfighter. Of the 20 technologies we transitioned, 22.7 percent transitioned to programs of record, and 9.1 percent transitioned directly to operational units. An additional 31.8 percent transitioned to further research and development efforts, to give their technology more time to develop.
Impact on the civil innovation base
For many innovation organizations, expanding the Department of Defense’s access to hard-to-reach parts of the U.S. civil innovation base is an end unto itself. Much has been made of the tech community’s reluctance to work on defense. But, as the United States faces down adversaries like China and Russia, sentiment has shifted. Broadly speaking, convincing U.S. companies that they should contribute to the common defense is pushing on an open door. The question, especially for smaller companies, is whether selling to the Department of Defense makes good business sense.
Innovation organizations working to address this should measure three things. First, are they successfully reaching outside the traditional defense ecosystem? Second, have they become better partners to industry? Third, are they providing investors with the kinds of returns that will motivate more investment in defense technologies?
This is much easier to measure than military utility. Innovation organizations know who is in their investment portfolios and who is applying to their solicitations. These are sometimes called vanity metrics because they seem to measure volume of activity rather than outcomes. But there is value in showing the spread of companies on contract by technology sector and geography. Innovation organizations should also know what proportion of the companies they work with are aspiring new entrants to the defense market versus established players.
To take it to the next level, innovation organizations should measure the extent to which they are enabling other Department of Defense organizations to start work with companies with whom they have not worked before. A company that has long worked with the Air Force may be new to the Army. One part of the Army may not know about a solution it can get from a company already on contract with another part of the Army. Because companies struggle to see across Department of Defense stovepipes, promising solutions die. To measure this, get data from Department of Defense stakeholders on which companies they had previously not worked with and which they had not known about.
For an innovation organization trying to bring smaller companies into defense, speed to capital is critical. Few of these companies can afford to wait one or two years from the time they reach an agreement to the time they are paid. For this, the key metrics are time from proposal down-select to contract award and time from contract award to first payment.
Meanwhile, companies’ investors are looking for meaningful returns in five to seven years. If that seems unlikely, they will steer their companies away from defense. Therefore, innovation organizations should track the number and magnitude of follow-on contracts awarded to the companies in their portfolios, wherever they come from.
Impact on the Department of Defense
Most defense innovation organizations have few or no unique authorities. They succeed by finding better ways to work through existing processes. In doing so, they tend to solve process problems. Some are unique to their mission, but most are not. Those best practices can be shared and scaled. Innovation organizations should measure the extent to which their methods are being adopted by others.
This too is relatively easy to know. When other organizations see innovation organizations doing things that they want to do as well, they call to ask how it was done. That leads to sharing products and points of contact. It tends to become a running dialogue. This makes it easy to monitor how your process innovations are being implemented elsewhere. Innovation organizations should take credit for that.
For example, some innovation organizations excel at involving warfighters in problem framing and solution development, getting acquisition professionals involved early to facilitate transition, informing requirements, or creating expeditious contract vehicles. All the above help technology cross the valley of death, and should be shared widely.
Conclusion
As the tenth anniversary of our existence approaches, defense innovation organizations are still muddling the message on their value. It is time for a more robust framework that organizations can apply quantitative rigor to. By measuring impact on the warfighter, impact on the civil innovation base, and impact on the Department of Defense, defense innovation organizations can demonstrate their value to Congress, Department of Defense and service leadership, companies, and investors in clear, easily understandable ways.
Casey Perley, PhD, is the executive director of the Army Applications Laboratory, where she has served since 2019. Prior to joining the Army Applications Laboratory, she studied biological pathogens in high containment at the U.S. Army Research Institute of Infectious Diseases.
The views expressed here are those of the author and do not represent those of the U.S. Army, the Department of Defense, or any part of the U.S. government.
Image: Lance Cpl. Richard Perez Garcia via DVIDS.
Jesus Christ is King
Comments are closed.