Performance measurement in economic development – even the standard can’t live up to the standard

Governors, mayors, and boards that oversee economic development organizations (EDOs) increasingly approach their responsibilities with a corporate mindset. They demand any initiative justify its value with direct, quantifiable impact. Virtually everywhere the standard for performance metrics is the same: job creation and investment associated with business attraction. As a result, most EDOs now release quarterly reports and dashboards that track these metrics with the precision of a stock ticker. As we’ve discussed, this is a problem for export and foreign direct investment (FDI) initiatives, because there isn’t data to measure their contribution to regional performance on these metrics (even though some places are measuring progress without data).

But current approaches to performance metrics don’t just make it hard to justify export and FDI initiatives: they are holding back broader systemic change. EDOs are struggling with major disruption, with business attraction and expansion activity down 50 percent, by one estimate. In response, as Brookings Metropolitan Policy Program Director and Vice President Amy Liu has written, EDOs are trying to remake themselves through new strategies focused on cultivating long-term economic assets and enhancing the competitiveness of firms. This work rightly points out the importance of “setting the right goal,” since goals should, in theory, dictate which strategies receive resources. But in our experience working with EDOs across the country, that’s often not the case. The pressure to quantify impact – exacerbated by declining corporate relocations that provide visible “wins”— means that regardless of overarching goals, specific initiatives (including exports and FDI) still live and die by their ability to contribute to traditional metrics. Thus, leaders are caught in a bind – they have to evolve, but are handcuffed to the limited set of strategies that appear to contribute to the existing standard for performance metrics.

Most places have tried to escape this bind by engaging in bigger incentives battles and making increasingly far-fetched attempts to claim credit for economic activity (often with disastrous results, as recently seen in Virginia). But there is a better alternative. Those who oversee and lead EDOs need to confront the fact that their performance metrics are deeply flawed. They need to jettison the standards they’ve been using to judge the impact of their strategies. They need to recognize that the current standard for performance metrics is, by and large, an illusion. Export and FDI initiatives don’t live up to it, but traditional business attraction approaches don’t either—they only appear to because EDOs expend a lot of energy and money to maintain the illusion.

To explain these flaws more concretely, we’ll use the example of an EDO claiming it created exactly 4,903 jobs in a given year. This kind of claim is common in the field and raises three key issues.

1. EDOs exaggerate their responsibility for outcomes. EDOs don’t “create” jobs, even in seemingly clear-cut cases, such as when a company is enticed to relocate. By the time an EDO is involved in such a deal, the company has usually zeroed in on two or three locations based on factors inherent to the region. EDOs make a marginal difference in these cases. Yet, if the company does relocate, all of the jobs are tallied up as a measure of performance. In contrast, a program that makes a similarly marginal impact on a firm’s exports could never claim to have “created” every job in that company.

2. EDOs buy metrics by providing financial assistance to firms. Because EDOs don’t create jobs, but performance metrics are structured as if they do, they have to somehow engineer a clear link between their efforts and a firm’s actions. They also need firms to give them data that they can use to quantify impact. Giving away incentives and subsidies, regardless of whether they’re necessary or justified, solves both problems – the EDO can point to the money as proof that it caused certain outcomes, and it can quantify those outcomes using numbers provided by the firm. We’ve seen an EDO offer a tax break to a company to offset the cost of machinery purchased months earlier so that it could claim credit for a year’s worth of “retained” jobs. Programs that bolster the competitiveness of existing firms have no such mechanism at their disposal and therefore seem ineffectual in comparison.

3. EDOs overstate how accurately they can measure impact. The metrics that EDOs produce may be precise (i.e., 4,903 jobs), but precision isn’t the same as accuracy. And most EDOs rarely, if ever, invest in measurement efforts that would enable them to accurately evaluate performance. (Tim Bartik has summarized a number of ways this can be done.) For example, when EDOs run programs that offer assistance to a group of firms, they usually claim credit for any positive outcomes that occur in the following year. They fail to account for selection bias (firms that receive assistance are more likely to achieve desired outcomes anyway) by comparing participating firms with a peer control group. These and other considerations significantly reduce the impact attributable to the EDO. A Pew report highlights state analyses that estimated that 80 percent to 90 percent of jobs attributed to incentive programs either replaced existing jobs or would have been created regardless of EDO involvement. Export and FDI strategies, along with others that address complex or long-term goals, are disadvantaged when leaders come to expect these inflated and artificially precise metrics.

This only scratches the surface. Other problems abound: EDOs don’t count job losses that occur while new ones are pursued, they claim credit for jobs that are announced but may never materialize, and they only consider gross outcomes rather than return on investment.

Metrics are not inherently problematic. By insisting that every initiative meets a standard for job creation and capital investment that was never realistic to begin with, however, those that oversee EDOs are not only holding potentially important initiatives hostage, they are also actively driving unproductive behavior.