Thoughts of a first-time peer reviewer

Most of my time is spent tirelessly chipping away at the scientific rock face, probably bogged down fixing a bug in my code or staring at some noisy looking data. Every now and then it all comes together and I want to tell people about it. So I write up my results as best I can, spend hours tinkering with figures, another few hours getting the fonts right on the axes, and after drafts and re-drafts, eventually I'll send it away to a journal to be published. This is where I become caught up in the process of peer review.

Usually it goes like this: The editor of the journal will check that the paper is basically interesting and then send it out to two reviewers who are chosen for their expertise in your area. These reviewers, or referees, will then read the paper, check it for basic errors and then comment on its originality and its pertinence to the field. This is sent back to the editor who will decide whether or not to publish. Usually the referees make you fix something, sometimes nothing, sometimes you can have a right old ding dong.

The main point is that the process is anonymous and behind closed doors. This is good and bad. Better blogs than this one discuss different options. It's not really my intention to criticise or support peer review, just to share my experiences.

Recently I was sent my first ever article to review. I can't say anything about the details, but it has been strange crossing over to the other side. I've had to ask questions that I have never thought much about before. So I wanted to put it down before I forget what all the fuss is about.

Reviewed

Up to now my only experience has been on the reviewed side of peer review. I've certainly had mixed experiences here. The first paper I had reviewed went through after lots of useful comments by the reviewers. It gave us more work but it made the paper better. Good experience. Another time a reviewer spotted a small error in our equations - also a good experience.

My worst experience involved two bad scenarios. Our first reviewer had not understood the paper, nor taken the time to follow the references that would allow him/her to do so. Instead of passing on to someone more qualified they just said it didn't make sense and was not interesting. The second referee had some interesting points but appeared to block it mainly on the basis that it didn't agree with other (presumably their) results. As you can see, I'm still bitter about this paper! It took 18 months to eventually get it through by which time it was thoroughly buried.

Of course, I'm biased, our paper could have been crap. Either way, the experience was bad enough that I was close to leaving science because of it. Receiving sneering anonymous reviews is a crushing blow to your ego - even if they're right.

Reviewer

So now I've reviewed my first paper. I won't say what I did, most of the questions I found myself asking would apply to any paper.

I'm quite used to reading other people's work, occasionally making a scoffing remark, or more likely not fully understanding it. The prospect of checking a paper for errors and assessing its quality filled me with dread. The only way I could deal with it was telling myself that it doesn't matter if I don't understand absolutely everything. The main thing is to check that they haven't done anything completely stupid.

This part of peer review I think is not too bad. There is an element of trust that someone has collected their data properly, but checking that it's not completely upside down is not too difficult or controversial.

Where it starts to get subtle is questioning the interpretation. Pulling someone up on their conclusion requires quite a bit of guts. Or I suppose an over inflated ego - of which there are many in science. This is related to another question, when should a scientific argument happen before publication and when should it happen afterwards? If the signal to noise ratio is to kept reasonably high then some things will need to be filtered out before hitting public view. I have not worked out an answer to this.

The final problem I had was with the question, "is this work of sufficient quality to be published in journal X?". Again this is really tricky, scientists can be real bitches when deciding what is or isn't interesting. On the other hand some scientists can try and get away with putting out any old crap just to lift their publication count. I found being asked to be the arbiter of quality quite stressful. Most results need to be on the scientific record somewhere, but should something be blocked for being too "incremental"? I suppose this is the journal's decision.

Is it worth it?

Apart from some initial stress I found the whole experience quite enjoyable. It makes you feel part of the scientific collective and it really tunes your critical skills. It will be interesting to see what becomes of peer review in the web 2.0 era, I would quite like to see it open up a little. I worry that unregulated, open anonymous comments, could be unhelpful. People are arseholes when they're anonymous, just ask a peer reviewer.