Casting a Jaundiced Eye on Peer Review

Frederic W. Grannis Jr., MD

Recently a colleague who had read one of my blogs on Cancer Network asked me why I was “wasting my time” writing pieces that are not peer-reviewed.* He followed up by reciting for me the many purported benefits of peer-reviewed publication, including maintenance of scientific rigor and academic credit.

On reflection, I believe he asked a reasonable question, deserving of an answer. As background, I should comment that I have a good amount of experience on both sides of the peer review divide, having published approximately 75 peer-reviewed articles and having served as an editorial reviewer for about the same number of manuscripts written by others. This experience has jaundiced my view of the peer review process.

The first obvious problem with peer review is delay. I have spent as long as 2 years working to get a manuscript through editorial review and into print. Part of the delay is with reviewers who fail to meet deadlines, but this appears to be improving in recent years as editors have gotten tougher in demanding timely review. (I confess that I have not always served as a role model in this respect.) Delays are problematic. A first report might lose priority. Patients might not get the benefit of a new treatment until it is too late.

If delays resulted in higher quality and objectivity of review, it might represent an adequate trade-off, but my experience tells me that this is seldom the case. On more than one occasion, questions and objections from reviewers have mystified me. Had they even read the manuscript? On the same and other occasions, I observed what appeared to be strong and persistent bias on the part of reviewers that ultimately led to rejection or intrusive revision of the manuscript. Reviewer bias is obviously much more of a problem when the author is presenting new information that challenges current paradigms. On other occasions, reviewers did not have strong objections to the content of the manuscript but complained that questions they were interested in were not addressed. In such circumstances, the author can either revise the manuscript to write the article the reviewer wants to read, or send the manuscript elsewhere. My former colleague Rod Schwarz advises never accepting rejection. Just send the manuscript on to the next journal in line, and so on, ad infinitum, until it is ultimately accepted. Each such submission, however, requires time and effort in revisions in style and construction, as well as time-consuming communication with co-authors.

In the worst circumstances, review processes appear to be improper. We have had a manuscript rejected from a society journal by an editor who did not submit it to reviewers, because he personally disagreed with the conclusion-despite the fact that the material had been selected for an oral report to the same society’s annual meeting. A letter of complaint to the editor-in-chief stating that we had not been provided the level of editorial review promised in the submission guidelines went unanswered.

My most frustrating experience to date involved a reviewer for a European journal who raised 10 major objections and advised rejection of the article. The other reviewer had questions and critiques but would accept a revised manuscript. I wrote to the editor and explained that I could easily identify the reviewer from the tenor of his comments, since I had acrimonious debate in print with him on prior occasions. I suggested that it was improper for him not to have revealed this prior conflict to the editor-as I had done when requested to review his work. I then wrote a rebuttal to his objections that was approximately three times the length of the original manuscript and returned it with a letter to the editor, explaining that I was certain that this reviewer would next provide a list of 10 new objections, and that I would continue to respond to each such rejection response seriatim. My prediction proved accurate. After a second round of objections, and even more lengthy responses, the editor finally made the decision to publish over the reviewer’s objections. This was a satisfying but Pyrrhic victory, considering the many months and enormous amounts of effort and frustration involved. I am quite certain that a younger, less stubborn author would have been defeated by such tactics.

My experiences as a reviewer have taught me further lessons about editorial practice. First, I have come to abhor anonymous review that allows some reviewers to bully the authors and treat them disrespectfully without fear of public identification. There is no excuse for such conduct. It is almost always possible to praise what is good in a manuscript and make constructive suggestions as to how the manuscript might be improved, even if it would need to be resubmitted to another, perhaps less prestigious journal. In the case of authors writing in English as a second language, I have often offered to do a limited rewrite, following reflection on how my manuscript would read if I attempted to publish in German, French, or Mandarin.

One advantage of looking at the manuscript from the reviewer’s point of view is that it is possible to defend a manuscript from what one considers to be unrealistically harsh criticism or unreasonable editorial bias from other reviewers. I have not been shy in telling the editor when I considered the critiques of another reviewer to be unfair, inaccurate, or biased. I have never, however, been able to convince an editor to reverse a rejection decision under such circumstances. One candid editor explained that it was so difficult to obtain reviewers, that he was unwilling to override a review for fear of losing the reviewer. I did not receive a response to my follow-up question as to the wisdom of continuing to employ such a reviewer for future reviews.

I have also learned that editors will reject published material if they are concerned that they will get negative feedback from editorial boards and sponsoring associations. For example, when I wrote a letter to the editor of a society journal following a print address to that organization’s annual meeting by a US Senator, exposing the senator’s tobacco industry associations and dismal voting record on public health issues, I was told frankly by the editor that publishing such a letter would mean loss of his job. I had no recourse but to search for the email addresses of all members of the society and email the unpublished letter to them.

My most recent editorial review, however, has been the most frustrating. It requires a bit of explanation. As noted earlier, with most journals, editorial review is anonymous and can provide a blind for intellectual bullies or can enable score-settling. In this case the journal, to its credit, identifies reviewers. The other reviewer and I were generous in our praise of the manuscript, from a European randomized trial group performing an economic analysis of a screening trial. Despite the two favorable reviews, the manuscript was rejected. My complaint elicited a response from the journal editor who explained that the final decision to publish resides with the editor rather than the reviewer and that sometimes other considerations override quality of manuscripts.

In this setting, at least I was able to tell the authors that I believed their article was of high quality and advised them to submit it elsewhere. They asked and I agreed to allow them to submit my editorial comments to the next journal. Although this story will almost certainly come to a happy ending with publication of the manuscript, the delay in publication may have serious adverse consequences.

On July 30, 2013, The US Preventive Services Task Force posted a draft version of an updated guideline on population lung cancer screening in the United States. One of the considerations that has delayed implementation of lung cancer screening has been the question of whether screening would be cost-effective. The unpublished article in question contains powerful evidence that I believe would inform questions about cost-effectiveness of screening in the context of a randomized controlled trial, during the USPSTF’s 30-day public comment period on the draft guideline. Unfortunately, delay in publication has squandered a golden opportunity to inform a vital USPSTF public policy decision, specifically with regard to who should be eligible for Medicare and Medicaid coverage of lung cancer screening.

A final personal consideration is that print publication leaves me in the dark as to how many people are reading my articles, and what they think of them. Letters to the editor are informative, but in my experience, it is often more difficult to get a letter to the editor published than an original paper. A new Web site www.researchgate.net is very helpful in letting authors know how many readers their print media publications reach.

When I write a blog for Cancer Network, editorial review has been rapid and reasonable. The Web site allows me to determine how many people have accessed the article and how many enjoyed it. Questions and critiques from readers are posted as comments to which I can respond or provide supplementary information almost instantaneously where appropriate.

Last, with impending retirement, I no longer need peer-reviewed publications to pad my bibliography to fuel academic promotion.

*Editor’s Note:While blogs and news items that appear on Cancer Network are not peer-reviewed, review articles that appear in the journal ONCOLOGY and chapters from Cancer Management: A Multidisciplinary Approach do go through a peer review process.