jagomart
digital resources
picture1_Journal Articles Pdf 53416 | Replication In Empirical Economics  The Journal Of Money Credit And Banking Project Optional3


 168x       Filetype PDF       File size 0.56 MB       Source: www.bitss.org


File: Journal Articles Pdf 53416 | Replication In Empirical Economics The Journal Of Money Credit And Banking Project Optional3
american economic association replication in empirical economics the journal of money credit and banking project author s william g dewald jerry g thursby richard g anderson source the american economic ...

icon picture PDF Filetype PDF | Posted on 20 Aug 2022 | 3 years ago
Partial capture of text on file.
      American Economic Association
      Replication in Empirical Economics: The Journal of Money, Credit and Banking Project
      Author(s): William G. Dewald, Jerry G. Thursby, Richard G. Anderson
      Source: The American Economic Review, Vol. 76, No. 4 (Sep., 1986), pp. 587-603
      Published by: American Economic Association
      Stable URL: http://www.jstor.org/stable/1806061
      Accessed: 09/07/2010 14:14
      Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
      http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
      you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
      may use content in the JSTOR archive only for your personal, non-commercial use.
      Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
      http://www.jstor.org/action/showPublisher?publisherCode=aea.
      Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
      page of such transmission.
      JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
      content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
      of scholarship. For more information about JSTOR, please contact support@jstor.org.
            American Economic Association is collaborating with JSTOR to digitize, preserve and extend access to The
            American Economic Review.
      http://www.jstor.org
                                                                  in Empirical Economics: 
                                           Replication 
                                                                                                                  Project 
                                                                                        and Banking 
                                                                           Credit 
                                                      of Money, 
                            The Journal 
                       BY                                              G.  THURSBY, AND RICHARD G.  ANDERSON* 
                            WILLIAM G.  DEWALD,  JERRY 
                                                                                                                   research. It 
                           This paper examines the role of replication in empirical economic 
                                                                                                               and data from 
                                                                                      collected programs 
                                                                         study that 
                           presents the                of a two-year 
                                           findings      to                                           Our 
                                                                        their               results.        research provides 
                           authors and attempted            replicate          published 
                                                                                        and causes of failures to replicate 
                                                                   about the extent 
                                                    information 
                           new and important 
                                                                                                                               in 
                                                                                                                       errors 
                                                                     Our              suggest that inadvertent 
                           published results in economics.                 findings 
                                                                                                 than           occurrence. 
                                                                    a                   rather         a rare 
                                                               are    commonplace 
                                        empirical articles 
                           published 
                    The  confirmation  of  research  findings                    for  knowledge-does              not  fit  within  the 
                                                                                                                      which defines the 
                                                          researchers is         "                      paradigm 
                 through  replication  by  other                                   puzzle-solving" 
                                                                                                                                    Scien- 
                                                                                                                        research. 
                 an  essential  part  of  scientific methodology.                                       in scientific 
                                                                                 reward structure 
                                                                   in                                                       not awarded 
                 William  Broad  and Nicholas  Wade                     Be-                                            are 
                                                                                 tific and professional laurels 
                 trayers  of  Truth  (1983)  present  examples                   for  replicating  another  scientist's  findings. 
                 wherein the inability of other researchers to                   Further,  a  researcher undertaking  a  repli- 
                                                                                                               as            imagination 
                 replicate  published  scientific  findings  re-                 cation may be viewed              lacking 
                 vealed  both  inadvertent errors and outright                   and creativity, or of being unable to allocate 
                 fraud. Replications in the physical and social                  his  time  wisely  among  competing  research 
                 sciences  are  attempted  infrequently,  how-                   projects.  In  addition,  replications  may  be 
                 ever. Thomas  Kuhn (1970) emphasized that                       interpreted  as  reflecting a  lack  of  trust in 
                 replication-however           valuable in the search            another scientist's integrity and ability, as a 
                                                                                 critique  of  the  scientist's  findings,  or  as  a 
                                                                                                                                     Final- 
                                                                                 personal dispute between researchers. 
                                                                                 ly,  ambiguities  and/or  errors in  the  docu- 
                     *Dewald  is  Senior Economist, Bureau of Economic                                                          may leave 
                 and  Business  Affairs,  U.S.  Department  of  State,            mentation of the original research 
                                                                                                     unable to distinguish between 
                 Washington,  D.C.  20520.  Thursby and  Anderson  are            the researcher 
                 Associate  and  Assistant  Professors,  respectively,  De-       errors in  the  replication and in  the original 
                 partment  of  Economics,  The  Ohio  State  University,          study.  Months  of effort may yield the repli- 
                  Columbus,  OH  43210.  We  extend  our  thanks to  the                         inconclusive results                    the 
                                             and Banking and to the pres-         cator only                               regarding 
                 Journal of Money, Credit                                         validity  of  the  original  study,  and  thus  no 
                  ent  editors  of  the  JMCB  for  their  cooperation  and       foundation  for  his  future  research in  the 
                  decision to continue to request that authors submit data        area. These  circumstances nurture a natural 
                  along  with  papers  for review. We  thank the National                                                        studies. 
                                                                                                                  replication 
                                                                    contract                    to undertake 
                                           financial         under                reluctance 
                  Science Foundation  for            support 
                  SES-8112800.  We extend special thanks to the authors                                                 of Money, Cred- 
                  who responded to our requests for programs and data,               In July 1982, the Journal 
                  many of whom  spent significant amounts of  time and            it  and Banking, with financial support from 
                                                                   We thank       the National  Science Foundation, embarked 
                                                          project. 
                                                  for this 
                  resources compiling materials                                   upon  the JMCB  Data  Storage and Evalua- 
                  Hashem  Dezbakhsh,  Roger  Lagunoff,  and  Joseph 
                  Hohman  for  research assistance,  and  Jeanette White,                                    of the             the JMCB 
                                                                                  tion            As part            Project, 
                  the  JMCB  secretary,  for  supervising  the  receipt  and            Project. 
                  distribution of data sets. We thank Ed Kuh and MIT's            adopted  an  editorial  policy  of  requesting 
                                                                                  from               the               and data used in 
                  Center for Computational  Research in Economics and                     authors         programs 
                                                                                                                                        and 
                                                                                                                           programs 
                                                                                                    and             these 
                  Management  Science  for loaning us the TROLL com-              their articles         making 
                  puter  program.  We  also  thank  Benjamin  Friedman,           data  available  to  other  researchers on  re- 
                  Lawrence Goldberg,  Edward J. Kane, Thomas Mayer,               quest.  In  a  second  part of  the  Project, we 
                  Daniel  Newlon,  Edward J.  Ray,  Anthony  Saunders,            attempted  replication  of  published  results 
                  George  Stigler,  Stephen  Stigler,  Roger  Waud,  and                             of                    data sets. Our 
                  Geoffrey  Woglom  for  their comments  and  assistance.         for a number          the submitted 
                                                      ours.                       findings  suggest  that  inadvertent  errors in 
                  Responsibility  for errors remains 
                                                                              587 
                                                   THE AMERICAN ECONOMIC REVIEW                              SEPTEMBER 1986 
               588 
                                                                                                                          inter- 
                                                                                                                 private 
                                                                                                      however, 
               published  empirical articles are a common-                 For other researchers, 
               place  rather than  a  rare occurrence. While               est  prevailed  and our request was either re- 
               correction  of  the  errors did  not  affect  the           fused  or ignored. We note that NSF  Policy 
               authors' conclusions  in most studies, the er-              Number  754.2  requires that computer pro- 
               rors make independent  replication impossi-                 grams and data which have been produced 
               ble unless the replicator errs in precisely the             with  the assistance  of NSF  grants be made 
                                                                                                                either by pub- 
               same way.                                                   available to other researchers 
                  In  one  crucial aspect, replications in em-             lication,  duplication,  or  loan  to  the  re- 
               pirical  economics  are  simpler  than  in  the             searcher. Investigators have the first right of 
               experimental sciences: given the researcher's               publication,  but  the NSF  rule requires that 
               computer  programs  and  data  set,  calcula-               the programs and data be made available to 
               tions  may  be  repeated.  The  required pro-               others.  It appears that this policy is seldom 
               grams and data are rarely available for rep-                enforced  and  that  investigators  either  are 
               lication,  however.  Many researchers employ                unaware  of  the  policy  or  unafraid  of  the 
               proprietary statistical packages such as SAS,               penalties  for failure to comply with it. 
                SPSS, TROLL, TSP, etc., which prohibit 
                                                                                                                 Journals 
                                                                                                   Professional 
                                                                                           Role of 
                copying  the program; replication is impossi-                      I. The 
                                                       without sub- 
                ble for the individual researcher                                                                       authors' 
                                                               access         Professional journals disseminate 
                                                 university) 
                sidized (for example, major 
                to  the  same  computer  hardware and  soft-               findings  throughout  the  world.  Our results 
                ware.  Many  other  researchers utilize  pro-              suggest that journals take a more active role 
                                                           assistants                      the  quality  of  the  results  pre- 
                                                research                   in  assuring 
                grams which they or their 
                have written in FORTRAN,  Pascal, or other                 sented  in  empirical  studies.'  As  editor  of 
                                                                                                       Frisch           for such 
                languages;  interpretation and evaluation of               Econometrica, Ragnar                argued 
                                                                                    in                   of  that            "In 
                these  programs  is  difficult  at  best-and               a role      the  first issue           journal: 
                impossible  at  worst-without          considerable        statistical  and  other  numerical  work  pre- 
                                                                                    in                   the           raw data 
                skill,  experience,  and the cooperation of the            sented       Econometrica         original 
                                           Dean Leimer                     will,  as  a  rule,  be  published,  unless  their 
                original programmer.                       and Selig                      excessive.  This  is  important  in 
                Lesnoy (1982),  for example, traced the false              volume  is 
                                                                                   to stimulate               control, and fur- 
                conclusions  of  Martin Feldstein (1974) to a              order                  criticism, 
                                                     we discovered          ther studies" (1933, p. 3). Today, most eco- 
                                              error; 
                computer programming                                                             except  the  JMCB  do  not 
                similar  errors  in  some  of  our  replication            nomics  journals 
                studies.  Finally,  we note that some research              have editorial policies which facilitate repli- 
                                                             of such        cation  of  published  results  by  requesting 
                projects employ computer programs                                                       from 
                enormous  size and complexity as to all but                 programs and data sets             authors. 
                guarantee  that  no  other researcher will  at-               It is  a matter of public record that errors 
                tempt  replication  of  the  study.  The  large-            exist  in  published  empirical studies. Recent 
                scale  macroeconometric models such as the                 examples  include  Leimer and Lesnoy, cited 
                MPS model are members of this group. We                     above,  and Frederick Siskind's 1977 correc- 
                discuss  below  our attempts at replication of              tion  of  Finis  Welch's  1974 minimum wage 
                a study based on the MPS model at Harvard                   study.  Our research suggests that  there are 
                University.                                                 many  more  unrecorded  and  undiscovered 
                   Similar  problems  arise  with  data.  Some              cases similar to these. 
                                                                                                                               in 
                data are confidential, having significant pro-                 The frequency and magnitude of errors 
                prietary  value  due  to  the  difficulty and/or            empirical  articles raise serious questions re- 
                expense of their collection, while federal law              garding  the  integrity  of  the  refereeing pro- 
                 makes other data available only for the in-                cess  of  professional  journals.  Referees  are 
                 ternal  use  of  employees  of  government                 concerned  primarily with methodology, the- 
                           such as the Federal Reserve 
                 agencies                                     System. 
                 In  the  JMCB  Project,  many  authors  fur- 
                                                        data had not 
                 nished their data even when the                                                         of             journals is 
                                                                               'An argument for the role    professional 
                 been  fully  exploited  in  their own  research.           presented by Edward Kane (1984). 
                       76  NO. 4            DEWALD ETAL.: REPLICATION-THEJMCB                   PROJECT                            589 
                VOL. 
                oretical  specification,  statistical  estimators,                tical  tests  of  a  hypothesis,  in  which 
                and importance of results; an author's pro-                       either  the  confirmation  or  the  con- 
                grams,  data,  and  calculations  are  typically                  tradiction  of  the  author's  statistical 
                assumed  to  be  correct. While  our  findings                    tests  is  reported.  For  this  task  to  be 
                suggest that this assumption often is unwar-                      reasonably  economical,  any  author 
                ranted,  we  hesitate  to  suggest-due           to  the          should  be  willing  to  provide  his  un- 
                                                                                                                             cost). 
                                                                                                                         (at 
                massive  amount  of  time  which  would  be                       derlying data to other scholars 
                required-that        referees  should  be  required               Indeed,  this behavior is a requirement 
                                                                    and           for responsible scholarship. 
                                                       programs 
                to check an author's computer                                                                             p. 1295] 
                data. Our findings suggest that the existence                                                    [1975, 
                of  a requirement that authors submit to the 
                journal  their programs and data along with                   The  editors  subsequently added a new sec- 
                each  manuscript  would  significantly reduce                  tion  to  the journal  devoted  to  verifications 
                                                        of           We        and contradictions  of papers first published 
                the  frequency and magnitude                errors.                                              this  section  con- 
                found that the very process of authors com-                   in  the  JPE.  Invariably, 
                piling  their programs and data for submis-                    tained papers employing either new data sets 
                sion  reveals  to  them  ambiguities,  errors,                 or alternative statistical techniques; little at- 
                                                                                                                                    the 
                                                         would be un-          tention  was paid to replication. Further, 
                and oversights which otherwise                                                             nor facilitated making 
                detected.                                                     JPE  neither required 
                   Our experience with authors who had not                     programs and data available for replication 
                prepared programs and data for submission                      attempts. 
                 to  the  JMCB  prior  to  submission  of  the                    The JPE  experiment is a classic example 
                 article is indicative  of the difficulties. Many              of  market  failure.  The  benefits  of  reduced 
                 could  not  locate  the  data  for  the  article,             frequency  and  magnitude of  errors in  em- 
                                                                                                         many of the characteris- 
                 while  others  had  lost  their  programming.                 pirical articles share 
                                                                                                                          the 
                 Even when the programs and data could be                      tics of public goods: all who read             journal 
                 located,  authors often  had not  kept  a con-                benefit from the knowledge that the research 
                 temporaneous  record of the progress of  the                  reported in  its  articles has been  more care- 
                                                                                                        the                 the quan- 
                 research and could not reconstruct their re-                  fully  monitored by           researcher; 
                 sults.  In  many  cases-particularly  in larger               tity of benefits available to any single reader 
                                                                                                                             journal; 
                                                                                                                         the 
                 universities-graduate         student  research as-           is not reduced by others reading 
                 sistants had conducted essential parts of the                 and  it  is  difficult  to  induce  the  reader to 
                                                                                                           value (price) for better 
                 research project, and after their departure it                reveal his or her true 
                                                                                                                                  high 
                                                                                                                            faces 
                                                                                                   A          researcher 
                 was impossible  to reconstruct data sets from                 quality articles.      single 
                 original sources.                                             costs  in  time  and  money  from undertaking 
                    In               the                  of economic          replication  of  a  study  and  finds  no  ready 
                        principle,        marketplace                                           which                       the social 
                 research  might  be  expected  to  provide  a                 marketplace              correctly prices 
                 check  against  careless,  undocumented  em-                  and individual value of the good. 
                                                                                                                              the sub- 
                 pirical  research. Since  the  editorial policies                An editorial policy which requires 
                                                                                                                               journal 
                 of professional journals had failed to address                mission  of programs and data to the 
                 the  problems  of  replication  of  published                 has  two  significant  advantages relative to  a 
                 work,  Edgar  Feige  proposed  in  1975  that                 laissez-faire  system  wherein  interested  re- 
                                                                                                             authors              First, 
                             minimum                                                        must contact               directly. 
                 "6...    a                                      editors       searchers 
                      as                               journal 
                                          standard,                                                                 cost  borne by a 
                 could  explicitly  publicize  the  necessity  of              it  substantially  reduces the 
                 full  reporting  of  procedures  and  data..."                researcher  seeking  to  replicate  original  re- 
                 (p. 1293). In response to Feige, the editors of               search.  The  economic  self-interest  of  the 
                                                                                                                               journal 
                 the Journal        Political              wrote:              author in satisfying the editors of the 
                                 of            Economy                                                                          to  the 
                                                                               assures  that  the  materials submitted 
                     We  believe  that  the  true  remedy  is                  journal  are more complete and correct than 
                     resort to the powerful force of compe-                    what  if  anything  might be  furnished to  an 
                                                                                                            The            provides a 
                     tition.  We believe that                 should           individual researcher.            journal 
                                                   journals                    cost-effective  clearinghouse for these materi- 
                     be              to accept alternative statis- 
                         prepared 
The words contained in this file might help you see if this file matches what you are looking for:

...American economic association replication in empirical economics the journal of money credit and banking project author s william g dewald jerry thursby richard anderson source review vol no sep pp published by stable url http www jstor org accessed your use archive indicates acceptance terms conditions available at page info about policies jsp provides part that unless you have obtained prior permission may not download an entire issue a or multiple copies articles content only for personal non commercial please contact publisher regarding any further this work information be action showpublisher publishercode aea each copy transmission must contain same copyright notice appears on screen printed such is profit service helps scholars researchers students discover build upon wide range trusted digital we technology tools to increase productivity facilitate new forms scholarship more support collaborating with digitize preserve extend access research it paper examines role data from col...

no reviews yet
Please Login to review.