The Future (?) of Peer Review

by Thomas von Foerster

From the beginnings of modern science or "natural philosophy," in the last part of the 17th century, and through the early part of the 19th century, the give and take of science, the discussion of experiments and results, the construction and critique of theories, took place at meetings of learned societies at which the members heard reports of each others’ work and read (aloud) letters from their corresponding members. The scientific literature of the period then consisted of the books or pamphlets published by individuals concerning their investigations together with the printed reports of the meetings of the learned societies. These published minutes generally contained not only the report of a paper or letter read at the meeting, but also a summary of the discussion that followed it. These societies (such as the Royal Societies of London, and Edinburgh, and the scientific Academies of Paris, Berlin, and St. Petersburg) restricted their membership to Fellows (local and corresponding) of known reputation and stature. Indeed, being elected to one of these bodies came to be a mark of high scientific distinction; for a few academies the anointing of members, deriving honor from honoring the excellent, has become their major function.

However, with increased specialization—the division of natural philosophy into chemistry, physics, mathematics, biology, and the rest—open discussions with local savants interested in other matters became less and less useful, while interchanges with other specialists, some of them remote, became more important. Meetings thus became less useful than correspondence and publication. As a result, around the beginning of the 19th century, individual scientists began publishing scientific journals of their own, independent of the old learned societies. Thus, for example, L. von Crell began a Chemisches Journal in 1781, A.L. Crelle began his journal of mathematical researches in 1826, and J. Liebig started the Annalen der Chemie (originally as Annalen der Pharmacie) in 1836. In these journals the editors selected or commissioned the articles to be printed, only sometimes turning to colleagues or associate editors for advice. The journals thus reflected their editors’ interests and preferences and often carried the editor’s name as part of their title, as indeed some, such as Crelles Journal, still do.

It was only with the great increase in specialization of the sciences around the end of the 19th century and the rise of publications of specialized societies (such as the American Physical Society or American Chemical Society) that journal editors began to rely almost entirely on reviews by others knowledgeable in the particular area of a submitted paper. Even so it is only in the last few decades that such "anonymous peer review" has come to dominate scientific publication. The technique is so dominant, in fact, that it has come to be used almost universally in science whenever assessments of quality bare to be made (evaluating grant proposals, promotions to tenure, and so forth).

I suspect that it is the bureaucratization of science, particularly the indirect effects of the funding of scientific research by governments, that has provided much of the impetus. The bureaucratic imperative is always to diffuse responsibility and to make decisions appear "objective" and based on consensus. Thus, program officers relying on their own judgment of what might yield interesting science (which some agencies maintained until the 1960s) have been replaced with panels that rate proposals for funds according to complex multidimensional scales that are then reduced with specious precision to five- or six-digit ratings that determine the success of an application. Editors of important scientific journals, not dealing with public funds, need not go to such extremes, but they too are under the same pressures (especially if the journal is published by a society for its members) to be "objective" and "democratic."

There are some exceptions; a recent example is a new journal, Materials Research Innovations, started by Rustum Roy, who already has a reputation as a maverick, and whose negative opinion of anonymous peer review is well known. The new journal calls itself "The first Journal using super peer review ...A radically new process of peer-reviewing based on each author’s past record, ... avoiding the long delays and bias against the new inherent in traditional peer review."

In the traditional scheme of peer review, editors receive articles, send them out to one or several technical referees, and base their decision to publish or not on the reviewers’ reports. While the reports are generally sent on to the authors, the names of the referees are not. (A few journals have tried to hide the names of authors from the reviewers, but this has rarely been successful—the subject and style often give good clues to the author’s identity.) Anonymity is supposed to maintain the objectivity of the referees and the integrity of the process, but it can also lead to abuses (and has indeed led to some recent lawsuits). There are a few journals (more in the social sciences) that do not use anonymous referees, and in one or two of these, the referees’ reports are printed along with the article if there are strong differences of opinion on the merit or contents of the article. In the physical sciences, however, anonymous peer review is the standard procedure used to ensure the quality and integrity of the research literature.

The primary function of these reviews is quality control: to determine what is included in the table of contents of a particular journal. But they do serve a secondary purpose as well, namely, to provide constructive criticism for the author. In the best cases, the comments lead to improvements not only in the presentation of the ideas, but also in the arguments or data themselves. To name one recent example, the discovery of lacunae in Wiles’s proof of Fermat’s last theorem required another massive investigation to correct. That such reviews are not common can be concluded, for example, from the small number of papers in which the author thanks a reviewer for substantive comments.

Electronic "Enhancements"

Electronics—chiefly the Internet, but also the enormous reduction in the costs of telephone links and the enormous increase of computer power—has is affecting how scientists communicate their results to their colleagues. Important information is communicated by telephone, e-mail, and other means that involve no paper, let alone scientific journals. Beginning with high-energy physics, but spreading rapidly to other disciplines, preprint servers are more and more becoming the primary means of distributing recent results to one’s colleagues.

The Internet has of course also changed the paper flow associated with peer review drastically, in that much of the correspondence about articles to be published takes place via e-mail. In many cases the articles themselves are also sent to reviewers in electronic form. Although the reviewing process is less demanding on the presentation (format, language) of an article than publication, there are still some problems in using only electronic manuscripts for review because of difficulties in reading on one machine the files prepared on another. As "universal" readers (such as Techexplorer or Adobe Acrobat) improve, these difficulties will certainly decrease, and the purely electronic procedures will become more and more widely adopted. This is, however, a only a change in the procedures, and it is unlikely, in the absence of other changes, to change the relationship of editor, author, referee, and reader. On the other hand, these procedural changes are easy to implement and to improve incrementally (that is, they are adiabatic changes—in Eherenfest’s, not the thermodynamic, sense of adiabatic), so most journals in the physical sciences will probably be developing them in the near future. The journal editors with whom I have discussed this concur in that view.

Because e-mail is not as secure as paper (it is easy to alienate an address, for example, to make a message seem to come from someone other than the author), editors who use electronic means to send out papers and obtain reviews must take some precautions. An editor can, for example, give reviewers passwords by telephone or fax both to access a read-only version of a submitted article and to authenticate the reviewers’ comments. As papers submitted to a journal become more technologically demanding, some reviewers may find they do not have the means to review all aspects of a paper, and editors will have to take such limitations into account. Of course, as scientific literature becomes more dependent on computer power, not only reviewers and editors but also scientists will find that technological developments limit access to parts of the scientific literature to those with the funds to acquire the necessary technology.

One aspect of the standard procedure of peer review, however, will not be altered by the Internet: The number of publications in scientific journals has been growing at a much faster rate than the number of scientists and, consequently, orders of magnitude faster than even the most active editor’s Rolodex. Each reviewer thus is being asked to referee an ever-increasing number of papers, to the frustration of many reviewers and occasionally to the detriment of the quality of the reviews. Electronic publishing by itself does nothing to alter that relationship; it is embedded into the sociology of publishing and reviewing.

The coexistence of electronic and traditional means of communicating scientific information can lead to occasional anomalies. Thus, for example, one can easily imagine a scenario in which a referee receives from a journal editor an article that he has already seen on a preprint server, and to which he has already posted a comment on the article; it would be most natural to use the same comment in the review for the journal, thereby not only voiding the anonymity of the review but also raising the question, what has the journal added to the article that the open forum has not already provided?

For some disciplines, such as high-energy physics, much of the discussion of scientific results already takes place via the preprint servers. In these fields, computers have taken over one of the functions of the traditional journal: to serve as a forum for discussion of new ideas. Interesting preprints spawn comments from workers in the area, the original papers are improved and new versions posted, other workers are stimulated to do experiments or suggest new ideas, and the field progresses without a single tree being felled.

It seems highly likely that such electronic forums will become the norm of the future. Once a critical fraction of the leading researchers in a field make use of a preprint server, no other worker in the field can afford to ignore it, and the Matthew principle will take over. The way in which such a forum works seems more akin to the 17th and early 18th century discussions of the learned societies than to those in the learned journals of the 19th and 20th centuries. Within the forum, linked papers and commentary replace the meetings and reports of the society, and the journals’ peer review is replaced by dialogue. The discussion itself becomes the review.

Such a system works well only if the group that interacts in such a forum is sufficiently cohesive. The members must know that for the most part they can trust each other’s judgment—otherwise the posted material is not worth reading. This may be part of the reason why preprint servers find ready acceptance in some fields such as astronomy or high-energy physics but not in other, larger, fields, such as condensed-matter physics. To preserve the needed social cohesion, the forums of the future may find they have to limit access to their discussions, just as the old learned societies restricted their membership to an elite (usually) of learned members. There are of course many ways to restrict access to such a forum; one can, for example limit it to people who:

• have learned an arcane jargon (K-theory, geologic stratigraphy, Latin);
• have or do not have specified domain names in their e-mail addresses;
• have been elected by the existing participants;
• have paid dues (money or scientific contributions) to join; or
• have access to a restricted domain on the Internet (such as the Internet 2).

Limitations on access to the papers and discussions in an electronic forum, however useful they may be in keeping out crackpots and others who have no business in the field, do come up against a fundamental value in science: as John Ziman pointed out, science is public knowledge. It is precisely the openness of scientific discourse that has allowed modern science to progress. Errors are corrected (some sooner than others) and new ideas are propagated by open discussion. It is only when discussion is restricted that wrong ideas such as Lysenkoism can thrive. While workers in fields such as reaction kinetics or graph theory find it easy to separate useful papers from useless or crackpot ideas simply from the structure of the argument and the use of jargon, the separation of wheat from chaff is more difficult in other fields. Alan Sokal’s hoax in Social Text — whatever else one may think of it — is a recent example of an interloper penetrating a field’s barriers. The conflict between openness and restriction is of course not new to the electronic medium (as Sokal’s example shows), but the explosion of the Internet into the general public makes the conflict more prominent and more difficult to resolve.

Because different fields have different social structures and different ways of validating ideas, the electronic forums will also have different ways of operating, but the payoff in terms of speed of communication and ease of interaction seems so large that most disciplines in the natural sciences are likely at some point to start using a preprint server.

What of the Future?

Would there be a role for something analogous to a journal (and thus for a publisher) in such a future? I think there is. A truly open, entirely electronic literature is too unwieldy and too ephemeral. One wants material that has been graded for quality (even TV guides rate films, giving stars for those worth watching), and one wants the really good stuff readily accessible for a long time. Those interested in the development of an idea (historians of science, prize committees) also want a trail of who said what when.

There is therefore, even in the electronic future, a place for a repository for scientific papers of certified quality that has a long lifetime, stable contents, and a known means of access. That is, there is a place for, more or less, what one now calls a journal.

While a preprint server provides ready access, and possibly also a long life, for a paper, it will, by its nature, not provide the certification of quality. A journal, even an electronic journal, must thus be a separate entity, though it could of course refer to the papers on the preprint server for its contents, assuming that the preprint server can guarantee long-term accessibility for the contents. (Print on paper remains readable and understandable for centuries, and in favorable cases for millennia, so a purely electronic archive has to meet a high standard of longevity and accessibility).

Quality control in a journal based on electronic papers may be provided by some form of review and certification, for which some person or group (an "editor" or "editorial board") of known stature and discernment would take responsibility. The certification may be provided directly by one of the editors (as in Liebig’s or Roy’s journals) or by anonymous referees to whom the editor sends the paper (as in most current scientific journals).

The long life and ready access would be provided by some entity that one hopes has a long life, or at least a well-thought out plan of succession. This entity (more or less what one could call a "publisher") could be a university, a scientific society, a traditional publisher (profit-making or non-profit), or a library. The publisher would make the article available electronically and would maintain it on their servers (or some practical equivalent thereof) forever. Libraries could maintain mirrors of the publishers’ material for a licensing fee. This would add some redundancy to the archive, as well as providing readier access for some users; it could also lead to problems in verifying and authenticating what is the "true" article.

While publishers at present do not directly guarantee the quality of the traditionally printed literature — the editorial boards do that — they do provide the administrative apparatus that makes the control possible. The electronic model does not change that. Publishers also have not been guarantors of the longevity of the literature, but for the electronic media there may be incentives for publishers to assume that function. Providing the electronic versions of articles on their servers and restricting access to paying customers may be a good way to preserve their investments in the production of the literature. In addition, by providing efficient search engines for the data stored in their own archives and tailored to the research communities they serve, publishers can add considerable value to the literature they control.

The reviewing and certification of articles for such an electronic journal could take any of several forms that can be grouped into two categories: transmission-based forms, such as the electronic version of the traditional peer review, with editors accepting or rejecting manuscripts submitted to them, based on their own ideas or on comments from referees they select; or collection-based forms, in which editors or other readers periodically browse through articles and associated comments posted in any appropriate preprint archive, certifying specific articles for their "journal" or personal collection as they see fit. In either case, the editor adds value to the accepted article by certifying its interest and merit. The latter form is already to some extent in use: most WWW home pages for individual researchers or research groups have a list of relevant publications, web sites and the like. In a sense, even the simplest of these home pages already provides a primitive form of electronic journal.

Which form most of the reviewing and certification will take will probably depend on social factors more than technology, on what researchers, including journal editors, like to use: this will vary from discipline to discipline, since, as we know, each has its own sociology. I suspect, however, that the "collection" forms will seem more "natural" to electronic communication, and that we may see more of these arising n the future.

More than this I would hesitate to predict.