168x Filetype PDF File size 0.56 MB Source: www.bitss.org
American Economic Association Replication in Empirical Economics: The Journal of Money, Credit and Banking Project Author(s): William G. Dewald, Jerry G. Thursby, Richard G. Anderson Source: The American Economic Review, Vol. 76, No. 4 (Sep., 1986), pp. 587-603 Published by: American Economic Association Stable URL: http://www.jstor.org/stable/1806061 Accessed: 09/07/2010 14:14 Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/action/showPublisher?publisherCode=aea. Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. American Economic Association is collaborating with JSTOR to digitize, preserve and extend access to The American Economic Review. http://www.jstor.org in Empirical Economics: Replication Project and Banking Credit of Money, The Journal BY G. THURSBY, AND RICHARD G. ANDERSON* WILLIAM G. DEWALD, JERRY research. It This paper examines the role of replication in empirical economic and data from collected programs study that presents the of a two-year findings to Our their results. research provides authors and attempted replicate published and causes of failures to replicate about the extent information new and important in errors Our suggest that inadvertent published results in economics. findings than occurrence. a rather a rare are commonplace empirical articles published The confirmation of research findings for knowledge-does not fit within the which defines the researchers is " paradigm through replication by other puzzle-solving" Scien- research. an essential part of scientific methodology. in scientific reward structure in not awarded William Broad and Nicholas Wade Be- are tific and professional laurels trayers of Truth (1983) present examples for replicating another scientist's findings. wherein the inability of other researchers to Further, a researcher undertaking a repli- as imagination replicate published scientific findings re- cation may be viewed lacking vealed both inadvertent errors and outright and creativity, or of being unable to allocate fraud. Replications in the physical and social his time wisely among competing research sciences are attempted infrequently, how- projects. In addition, replications may be ever. Thomas Kuhn (1970) emphasized that interpreted as reflecting a lack of trust in replication-however valuable in the search another scientist's integrity and ability, as a critique of the scientist's findings, or as a Final- personal dispute between researchers. ly, ambiguities and/or errors in the docu- *Dewald is Senior Economist, Bureau of Economic may leave and Business Affairs, U.S. Department of State, mentation of the original research unable to distinguish between Washington, D.C. 20520. Thursby and Anderson are the researcher Associate and Assistant Professors, respectively, De- errors in the replication and in the original partment of Economics, The Ohio State University, study. Months of effort may yield the repli- Columbus, OH 43210. We extend our thanks to the inconclusive results the and Banking and to the pres- cator only regarding Journal of Money, Credit validity of the original study, and thus no ent editors of the JMCB for their cooperation and foundation for his future research in the decision to continue to request that authors submit data area. These circumstances nurture a natural along with papers for review. We thank the National studies. replication contract to undertake financial under reluctance Science Foundation for support SES-8112800. We extend special thanks to the authors of Money, Cred- who responded to our requests for programs and data, In July 1982, the Journal many of whom spent significant amounts of time and it and Banking, with financial support from We thank the National Science Foundation, embarked project. for this resources compiling materials upon the JMCB Data Storage and Evalua- Hashem Dezbakhsh, Roger Lagunoff, and Joseph Hohman for research assistance, and Jeanette White, of the the JMCB tion As part Project, the JMCB secretary, for supervising the receipt and Project. distribution of data sets. We thank Ed Kuh and MIT's adopted an editorial policy of requesting from the and data used in Center for Computational Research in Economics and authors programs and programs and these Management Science for loaning us the TROLL com- their articles making puter program. We also thank Benjamin Friedman, data available to other researchers on re- Lawrence Goldberg, Edward J. Kane, Thomas Mayer, quest. In a second part of the Project, we Daniel Newlon, Edward J. Ray, Anthony Saunders, attempted replication of published results George Stigler, Stephen Stigler, Roger Waud, and of data sets. Our Geoffrey Woglom for their comments and assistance. for a number the submitted ours. findings suggest that inadvertent errors in Responsibility for errors remains 587 THE AMERICAN ECONOMIC REVIEW SEPTEMBER 1986 588 inter- private however, published empirical articles are a common- For other researchers, place rather than a rare occurrence. While est prevailed and our request was either re- correction of the errors did not affect the fused or ignored. We note that NSF Policy authors' conclusions in most studies, the er- Number 754.2 requires that computer pro- rors make independent replication impossi- grams and data which have been produced ble unless the replicator errs in precisely the with the assistance of NSF grants be made either by pub- same way. available to other researchers In one crucial aspect, replications in em- lication, duplication, or loan to the re- pirical economics are simpler than in the searcher. Investigators have the first right of experimental sciences: given the researcher's publication, but the NSF rule requires that computer programs and data set, calcula- the programs and data be made available to tions may be repeated. The required pro- others. It appears that this policy is seldom grams and data are rarely available for rep- enforced and that investigators either are lication, however. Many researchers employ unaware of the policy or unafraid of the proprietary statistical packages such as SAS, penalties for failure to comply with it. SPSS, TROLL, TSP, etc., which prohibit Journals Professional Role of copying the program; replication is impossi- I. The without sub- ble for the individual researcher authors' access Professional journals disseminate university) sidized (for example, major to the same computer hardware and soft- findings throughout the world. Our results ware. Many other researchers utilize pro- suggest that journals take a more active role assistants the quality of the results pre- research in assuring grams which they or their have written in FORTRAN, Pascal, or other sented in empirical studies.' As editor of Frisch for such languages; interpretation and evaluation of Econometrica, Ragnar argued in of that "In these programs is difficult at best-and a role the first issue journal: impossible at worst-without considerable statistical and other numerical work pre- in the raw data skill, experience, and the cooperation of the sented Econometrica original Dean Leimer will, as a rule, be published, unless their original programmer. and Selig excessive. This is important in Lesnoy (1982), for example, traced the false volume is to stimulate control, and fur- conclusions of Martin Feldstein (1974) to a order criticism, we discovered ther studies" (1933, p. 3). Today, most eco- error; computer programming except the JMCB do not similar errors in some of our replication nomics journals studies. Finally, we note that some research have editorial policies which facilitate repli- of such cation of published results by requesting projects employ computer programs from enormous size and complexity as to all but programs and data sets authors. guarantee that no other researcher will at- It is a matter of public record that errors tempt replication of the study. The large- exist in published empirical studies. Recent scale macroeconometric models such as the examples include Leimer and Lesnoy, cited MPS model are members of this group. We above, and Frederick Siskind's 1977 correc- discuss below our attempts at replication of tion of Finis Welch's 1974 minimum wage a study based on the MPS model at Harvard study. Our research suggests that there are University. many more unrecorded and undiscovered Similar problems arise with data. Some cases similar to these. in data are confidential, having significant pro- The frequency and magnitude of errors prietary value due to the difficulty and/or empirical articles raise serious questions re- expense of their collection, while federal law garding the integrity of the refereeing pro- makes other data available only for the in- cess of professional journals. Referees are ternal use of employees of government concerned primarily with methodology, the- such as the Federal Reserve agencies System. In the JMCB Project, many authors fur- data had not nished their data even when the of journals is 'An argument for the role professional been fully exploited in their own research. presented by Edward Kane (1984). 76 NO. 4 DEWALD ETAL.: REPLICATION-THEJMCB PROJECT 589 VOL. oretical specification, statistical estimators, tical tests of a hypothesis, in which and importance of results; an author's pro- either the confirmation or the con- grams, data, and calculations are typically tradiction of the author's statistical assumed to be correct. While our findings tests is reported. For this task to be suggest that this assumption often is unwar- reasonably economical, any author ranted, we hesitate to suggest-due to the should be willing to provide his un- cost). (at massive amount of time which would be derlying data to other scholars required-that referees should be required Indeed, this behavior is a requirement and for responsible scholarship. programs to check an author's computer p. 1295] data. Our findings suggest that the existence [1975, of a requirement that authors submit to the journal their programs and data along with The editors subsequently added a new sec- each manuscript would significantly reduce tion to the journal devoted to verifications of We and contradictions of papers first published the frequency and magnitude errors. this section con- found that the very process of authors com- in the JPE. Invariably, piling their programs and data for submis- tained papers employing either new data sets sion reveals to them ambiguities, errors, or alternative statistical techniques; little at- the would be un- tention was paid to replication. Further, and oversights which otherwise nor facilitated making detected. JPE neither required Our experience with authors who had not programs and data available for replication prepared programs and data for submission attempts. to the JMCB prior to submission of the The JPE experiment is a classic example article is indicative of the difficulties. Many of market failure. The benefits of reduced could not locate the data for the article, frequency and magnitude of errors in em- many of the characteris- while others had lost their programming. pirical articles share the Even when the programs and data could be tics of public goods: all who read journal located, authors often had not kept a con- benefit from the knowledge that the research temporaneous record of the progress of the reported in its articles has been more care- the the quan- research and could not reconstruct their re- fully monitored by researcher; sults. In many cases-particularly in larger tity of benefits available to any single reader journal; the universities-graduate student research as- is not reduced by others reading sistants had conducted essential parts of the and it is difficult to induce the reader to value (price) for better research project, and after their departure it reveal his or her true high faces A researcher was impossible to reconstruct data sets from quality articles. single original sources. costs in time and money from undertaking In the of economic replication of a study and finds no ready principle, marketplace which the social research might be expected to provide a marketplace correctly prices check against careless, undocumented em- and individual value of the good. the sub- pirical research. Since the editorial policies An editorial policy which requires journal of professional journals had failed to address mission of programs and data to the the problems of replication of published has two significant advantages relative to a work, Edgar Feige proposed in 1975 that laissez-faire system wherein interested re- authors First, minimum must contact directly. "6... a editors searchers as journal standard, cost borne by a could explicitly publicize the necessity of it substantially reduces the full reporting of procedures and data..." researcher seeking to replicate original re- (p. 1293). In response to Feige, the editors of search. The economic self-interest of the journal the Journal Political wrote: author in satisfying the editors of the of Economy to the assures that the materials submitted We believe that the true remedy is journal are more complete and correct than resort to the powerful force of compe- what if anything might be furnished to an The provides a tition. We believe that should individual researcher. journal journals cost-effective clearinghouse for these materi- be to accept alternative statis- prepared
no reviews yet
Please Login to review.