In § 1502 of Dodd-Frank, Congress instructed the SEC to draft rules requiring public companies to disclose their use of “conflict minerals” originating in and around the Democratic Republic of the Congo (DRC). Coined the Conflict Minerals Rule, the statute is based on the notion that investor accountability paves the way for socially conscious corporate behavior. The suspicion was that money from U.S. companies was flowing to warlord-controlled mines in the region and thereby fueling human rights abuses by such groups. The hope was that improved transparency ushered in by the Rule would cause companies to turn off the spigot.

The Conflict Minerals Rule has faced a wide range of critiques, but perhaps the most widely and loudly voiced criticisms have concerned its potential cost. Early in the rulemaking process, leading oppositionists postulated that the expenditures required for compliance would cripple industry players.

The SEC’s final rule release fueled such speculation. The agency postulated therein that the Conflict Minerals Rule would cost the industry a stratospheric $3 to $4 billion in the first year. This estimate has been repeated over 10,000 times in a wide range of publications. Scholars, the press, and even the courts have deferred to its validity.

The problem? The SEC’s figures are groundless. Our forthcoming article, “Cost-Benefit Analysis and the Conflict Minerals Rule,” unpacks the SEC’s economic model. We strip away the layers and layers of baseless assumptions upon which the agency’s calculation is based. At bottom, all that remains is conjecture. We argue that our analysis is a rebuke not only of the SEC’s use of cost-benefit analysis (CBA) in this case, but also of CBA itself.

The SEC has historically weighed costs and benefits as part of its rulemaking. More recently, however, the D.C. Circuit has found that the agency has done so with insufficient rigor. In Chamber of Commerce v. SEC, 412 F.3d 133 (D.C. Cir. 2005) and Business Roundtable v. SEC, 647 F.3d 1144 (D.C. Cir. 2011), the court chastised the agency for failing to take steps to quantify the costs and benefits of its rules, even where the SEC explicitly claimed that it lacked a reliable basis for making certain estimates. While not directly mandating more exacting CBA, the court’s rulings ushered in a new era of SEC rulemaking—one focused on quantification. In 2012, the SEC circulated a memorandum to its staff, requiring its rule-makers to “monetize or otherwise quantify potential costs and benefits [of new rules] . . . whenever such quantification is practicable,” and to do so “even where the available data is imperfect and where doing so may require using estimates (including ranges of potential impact) and extrapolating from analogous situations.”

The SEC’s conflict minerals CBA was contained in a 22-page “Economic Analysis” section of the final rule release. The section made no attempt to quantify the benefits of § 1502. The SEC instead cited to Congress’s intention that the Conflict Minerals Rule would reduce human-rights abuse in the DRC. With quantification of benefits off the table, the vast majority of the section was an attempt to quantify costs.

To reach its staggering estimate of $3 to $4 billion in the first year, the SEC relied almost exclusively on calculations submitted during the rulemaking process by the National Association of Manufactures (NAM) and Tulane University’s Payson Center (Tulane). NAM suggested that the Conflict Minerals Rule would cost the industry anywhere from $9.4 billion to $16 billion in the first year. Similarly, Tulane estimated that the Rule would initially cost the industry $7.93 billion. The SEC did not engage in any independent research when creating its own cost estimate. Rather, the agency reengineered the Tulane and NAM models. The SEC retained the inputs it felt were correct in both models and modified the inputs it disagreed with, sometimes taking an input from Tulane and inserting it into NAM’s model and visa versa. The low end of the SEC’s range ($3 billion) comes from Tulane’s model, as modified by the SEC, and the high end ($4 billion) comes from NAM’s model, as likewise modified.

Our article methodically tracks every input contained in the SEC’s final calculations and analyzes every tactical step the agency took to modify the Tulane and NAM models. Our analysis reveals three major flaws that compromise the credibility of the resulting figures.

First, the SEC retained estimates for compliance costs that were not imposed by the Rule. For example, NAM included costs incurred by suppliers to audit their supply chain diligence measures. But the Rule requires no such thing.

Second, the figures that NAM and Tulane used in their models, and which the SEC maintained, lacked empirical backing. Many numbers were inserted into the models without any indication as to their basis. In those cases where NAM and Tulane did cite evidence for their figures, the studies they pointed to were either inherently unreliable or misused. For instance, NAM claims it surveyed its members to determine the average number of conflict-mineral suppliers per issuer. But NAM does not tell us how many companies responded, the size of the respondents, the questions asked, the statistical controls used to assess the representativeness of the respondents, etc. Similarly, Tulane relies heavily on a survey submitted by IPC during the SEC’s rulemaking process. But IPC warns in the study itself that it never set out to produce statistically significant data. Indeed, the organization received 60 responses from a population of 3,839—a response rate far short of typical significance thresholds. Tulane also manipulates IPC’s figures in ways the underlying data does not support.

Finally, when the SEC altered Tulane’s and NAM’s models, it provided unsatisfactory explanations for its reengineering. For example, NAM said it would cost the average issuer $1 million to update its IT systems to comply with the Rule. Tulane decided that the $1 million estimate was fair for large issuers, but claimed small issuers would only expend $205,000 for such upgrades. The SEC took issue with this assessment. It argued, without supporting empirics, that for large issuers “the appropriate estimate lies somewhere in between” $205,000 and $1 million, and posited that such companies would only spend $410,000. Pure guesswork.

The manifold flaws in the SEC’s analysis mean its cost estimate should play a muted role in the broader debate regarding the merits of the Conflict Minerals Rule and the expenses associated with supply-chain transparency efforts more generally. Post mortem studies confirm that the SEC was off-base and suggest that compliance cost far less than $3-$4 billion in the first year.

Each branch of the federal government champions the use of CBA in regulatory analysis, but our findings call its usefulness into question. Even though the SEC failed to quantify benefits, and did a poor job of quantifying costs, even if it had analyzed both sides, and done so to the best of its abilities, it would have run up against insurmountable epistemic hurdles that inhere in CBA, calling into doubt CBA’s value regardless of how competently an agency applies it in any particular rulemaking effort. In looking at benefits, what monetary value would the SEC have placed on the prevention of rape in the Congo? How would it have estimated how many rapes the Rule would have prevented?

The literature largely assumes that the cost-side is easy to estimate. But the conflict-minerals example suggests otherwise. While the agency’s decision to rely on NAM’s and Tulane’s ill-conceived models doomed its effort, estimating costs also implicates epistemic constraints. It is inherent difficulty to predict how companies might respond to unprecedented regulation like the Conflict Minerals Rule. The only source of information is the companies themselves, but they are a dubious resource: companies do not really know what they will do until they actually do it and, when asked to predict, they are likely to submit overly lofty figures.

Our study also challenges the procedural case for CBA. Advocates argue that even flawed calculations are better than none at all because they inspire a richer public debate. That did not happen here. Instead, the public was misinformed. And this is to be expected. This arc of understanding—where figures are blindly repeated before being questioned when considered more deeply—is predictable. It is easy and free to accept a regulator’s estimate, but there are extensive information-processing costs to discovering its flaws. Finally, when and if figures are carefully scrutinized, the resulting critiques may have great difficulty dislodging the existing narrative.

Because the application of CBA in each instance is so complex, it is only through case studies that we can empirically assess its usefulness. While a study like ours provides an insufficient basis for far reaching conclusions, it contributes to a growing body of research that questions the contribution that the practice makes to regulatory analysis and public discourse.

The preceding post comes to us from Jeff Schwartz, a Professor of Law at the University of Utah S.J. Quinney College of Law, and Alexandrea Nelson, a Law Clerk to the Honorable Judge Dee V. Benson, Federal District Court of Utah. The full article on which this Post is based is entitled “Cost-Benefit Analysis and the Conflict Minerals Rule” and available here.