I guess this doesn’t come as a big surprise, but the recent mini-scandal over McKinsey & Co’s survey of employers’ responses to the US health care reform act shows pretty clearly how pressure for transparency in policy research in the US depends on disagreement between the two parties.  In this case, when the survey showed a surprisingly high number of employers dropping employee health care coverage in response to the Act, Senator Max Baucus led the call for more research disclosure, arguing that “honest public discourse requires a standard level of transparency — one McKinsey simply has not met.”  Baucus released a letter to McKinsey management that included a laundry list of requirements for “standard professional practice.”  These include:

  • What was your sampling design?
  • What was the breakout of the survey by industry, geography, employer size, percentage of low-wage workers… (etc.)
  • Were there any oversamples..?  What was the margin of sampling error?
  • How were participants chosen?
  • What was the participation rate?
  • How were the interviews conducted?  What script was used to “educate respondents” before asking questions?
  • What questions were asked, including their exact wording?  What was the order of the questions?

Good questions!

Under pressure, McKinsey released the data, claimed they stood by their work, and then added that “The survey was not intended as a predictive economic analysis of the impact of the Affordable Care Act.”  Apparently the joke is on everyone else for assuming otherwise.

But kudos to Baucus for being such a stickler for the principle that public policy should be made with publicly-available data.  We’ve argued repeatedly that this should be a minimum standard for research used in policymaking, and that it should be built into the rules for submission to all policy comment processes.  Anything that doesn’t meet that threshold shouldn’t be taken seriously. (And frankly, this should have been the end of the story with the McKinsey report).  To the best of our knowledge, however, no US government agencies meet this standard.  The much-publicized Obama transparency agenda has been almost entirely about the data government produces, not the data it uses, which in many contexts comes entirely in the form of studies–or summaries of studies– submitted by private stakeholders.  Access to the commercial datasets underlying the studies?  No.

So how often have official requests for transparency been made of industry IP research?  The kind of studies, for example, that claim  $51 billion in losses to software piracy in 2009?  Or $6.2 billion in movie losses in 2005?  Or $466 million in lost music revenues in China in 2010?  To the best of my knowledge, the answer is never.  Not when IP enforcement obligations were added to the Higher Education Opportunity Act in 2008, based partly on MPAA  claims about infringement on campus that later turned out to be false.  Not when the Pro-IP Act was being debated in 2008 based on a whole raft of industry loss claims (the bill ultimately passed the House 410 votes to 11 and unanimously in the Senate).  Not when the Government Accountability Office unsuccessfully sought information about industry reporting methods in 2010.  Not when the Protect IP Act was debated in 2011.  Not when the British government published a report earlier this spring that called the evidentiary basis for IP policy “threadbare.” In short, bad data thrives without an official champion for good data.  And there isn’t one for IP policy in the U.S.

Why can’t we turn to Max Baucus on these issues?  He seems interested in good research practices.  Sadly, when Baucus isn’t sponsoring bills to sanction other countries for inadequate enforcement, he’s sponsoring his own methodologically-challenged reports about losses to Chinese piracy, conducted in this case at US Senate Finance Committee request by the International Trade Commission.  The ITC report surveyed 5000 US-owned “IP-intensive” firms about their losses to Chinese IPR infringement, and calculated that US firms lose roughly $48 billion to Chinese infringement.  Further, they argued that this results in roughly 1 million lost US jobs.  Powerful stuff in these times.

Mike Masnick has identified the main problem with this survey method: why would you rely on firms to estimate their own losses to infringement?  The incentives to over report losses are pretty clear: foreign rightsholders want to pressure  the Chinese government to ramp up enforcement efforts and they have a variety of biases that predispose them to view infringement as a theft or loss (as opposed to, say, the basis of valuable network effects in software markets).  But the problem is more circular than that.  When ITC asked how companies estimated their losses, here’s what they got:

Firms that reported IPR infringement in China generally said that their estimated revenue losses were based on information provided by third parties, such as industry associations. Almost half of firms (48.7 percent) reported that their estimates for lost sales in the U.S. market were provided by third parties, while the remaining firms largely reported using estimates they calculated for themselves, for example, by multiplying the amount of infringement or confiscated products by wholesale prices or by the products’ retail value.   (p3-11)

In other words, about half the companies that reported losses (48.7%) based their claims  directly on industry association studies (IIPA, MPAA, BSA, and so on) and the other half multiplied an estimated number of infringements (which generally also comes from industry associations) by the retail price of their goods. The latter method, it is worth noting, has been abandoned by virtually all the industry groups because it doesn’t take into account the low substitution rates associated with most counterfeit and pirated goods–ie., the likelihood that a pirated or counterfeit good substitutes for a licit sale.  This is especially significant in developing countries where price to income ratios make licit purchases very infrequent.  To their credit, some 41% of companies indicated that they couldn’t estimate losses.

So the ITC report, for all its considerable detail and sophistication, is basically just  laundering the same old industry data through a survey–and worse, data the industry groups themselves would no longer put forward as credible.  That’s significantly more than the industry groups got out of the OECD in 2007 and the GAO in 2010 in the course of similar efforts to launder industry claims.  Is there a lot of piracy in China?  Yes.  Is this a credible account of its impact?  No.  Let’s get Baucus to send himself a letter.