Disclaimer: I am not involved in the decision process at all this year, so my advice is merely that which I’ve accumulated and will give out to my own students and collaborators when responding. It is not official advice for NIPS and has not been discussed with any members of this year’s program committee.

First stage NIPS reviews are out today and the author rebuttal section is beginning. This seems an appropriate time to think about the best way to respond to a set of reviews.

Firstly

It can be very dispiriting when you first receive your set of reviews. Most people are in a heightened state of anxiety when they receive them. This means it can be very difficult to read them objectively. However, you must never write your response in that anxious state, and you need your points to be as objective as possible. To control for this my advice is to use the first 24 hours after reading to digest the circumstances of your paper.

  1. Read through the reviews. Don’t yet try and address the specific points that are being made. Don’t even make notes. Just get the general impression of where the paper sits.

  2. Wait for 24 hours.

  3. Read the reviews again, carefully this time. Extract specific points from each review. Try and find if there are comments that are common to all the reviews. Note down what these are.

  4. Re-read your paper in the light of the reviewer comments.

  5. Draft an initial response and discuss with your collaborators.

Reviewers aren’t Idiots

When you go through the process above, you may find yourself feeling that one or more of the reviewers is an idiot. They aren’t.

If a reviewer has missed the point of your paper, it may be because the paper wasn’t clear. If they are uncertain about the paper, it may be because it wasn’t clear. If they missed something in the paper that is actually there, it may be because it wasn’t clear. In your response, if any of these things has happened, politely point out where the reviewer has misunderstood and if there is space you can state how you will clarify.

It may be the case that a particular reviewer makes you very angry, and you would like to tell everyone what an idiot he or she is. If that’s the case you can write a very angry response, and discuss what an idiot the reviewer is with your collaborators. But don’t submit such responses. Delete the text once you’ve written it. Whilst it is highly unlikely that the reviewer is an idiot, it is very possible that a reviewer might have made an honest mistake. If this is the case, then the mistake may be a result of either poor explanation, or bad sign-posting in your paper. You can politely point out the error and apologise for not making that section clearer.

If you find yourself thinking that all the reviewers have each made a lot of mistakes, it may be that the paper itself is the problem. Note I didn’t say the idea was the problem. But the paper. The reviewers don’t have direct access to your mind, they are reading things through the restricted window of the presentation that the paper gives them. It is surprisingly easy to forget this!

The Nature of Reviewers

Firstly, all reviewers are volunteers. Never forget that. They are doing their best to accept quality material. However, how they approach this role differs according to the individual.

In a recent discussion on GitXiv, with samin and Roelof Pieters I suggested how that reviewers broadly tend to divide. They split into gate keepers, and space-curators. I’ll try and expand on that now and suggest what it might mean for your responses.

Gate Keeper Reviewers

A gate keeper reviewer is one who is concerned for the reputation of the conference (or journal etc) in terms of the quality of the papers. They take the view that there is limited space, so you must have had to do something ‘difficult’ to get in. As a reviewer, if a paper is roughly in your domain of expertise it is quite easy to be this type of reviewer, so this is the form that most reviewers take. Gate-keepers review the paper from the perspective “if the author had been me, how would I have done this paper, and how long would it have taken me to write it”. These reviewers are very impressed by technicality and benchmark-performance (or what I refer to as ‘engine testing’).

Space Curator Reviewers

A space-curator reviewer is one who is much more concerned with the diversity and interest of the conference. They are looking for new ways of thinking and they try and project themselves onto the community to judge why the paper might be important. They are concerned with whether a paper might change the debate or move things forward in an interesting new direction. They are less concerned about the technical gymastics a particular result required and they are aware that new directions are difficult to benchmark. They will want to ensure that your work is not trivial or known, but their main focus will be whether they find it inspiring or useful.

What it Means for your Reviews

In practice a community benefits from both types of reviewers. If you have satisfied both these reviewer types, then congratulations! You have written a very strong paper that would probably be in the top 5% of papers submitted to a conference like NIPS! You should still write a response, but you are most of the way there already (but don’t get complacent … reviewers smell complacency).

If your average score is less than five, then you are facing a real uphill struggle. Many papers with scores at this level are withdrawn at this stage. Note that it doesn’t mean that the paper was bad, it means that the reviewers adjudged it wasn’t suitable for NIPS. Sometimes this is because there’s a misunderstanding of what the conference is looking for, or the paper wasn’t ready yet, and sometimes this is because you were unlucky with the reviewers.

If you haven’t satisfied the reviewers, and you have a score average of around 6 (between 5 and 7), you will probably need to be clever in your responses.

My experience is that gate-keeping is a much easier job than space-curating, so most reviewers (unfortunately) will naturally fall into that role. This means that when you read your reviews they will mostly focus on the perceived negatives in your paper. Reading such comments can be fairly grating, but the very worst idea is to respond to the reviewer ‘in kind’. At this point you need to have some faith in the system. You need to rely on a champion.

Find Your Champion and Cross your Fingers

Actually, no one wants to attend a conference that is filled with only highly technical papers or only papers that perform well on some very standard benchmarks. Most people want to see cool ideas that inspire them. Some reviewers are aware of this and they may be prepared to argue for your paper. This type of reviewer is called a ‘champion’ (think Knight on a white charger saving a ‘paper in distress’). If there is a supportive reviewer that likes your idea, make sure they have the armour and weaponary in your rebuttal to defend and rescue your paper. Ensure you cover all the technical points from the other reviewers. Where the aims of your paper have been misunderstood politely restate them.

Even if you don’t have a champion in the reviewers, Area Chairs are normally selected for their abilities as space-curators. They will be chosen by Program Chairs because they have experience and good taste. If your paper has caught the eye of the Area Chair and you are able to deal with the issues that the reviewers have identified, then your work also has a chance of being accepted.

When it Doesn’t Work

Even after all the thought you’ve put in, your paper may not be accepted. This doesn’t mean its a bad paper. It can just mean you were unlucky. But the reviewers comments are still useful, they help you see where typical misunderstandings are. They can help you improve the introduction, or reinforce your message where it has been missed or misunderstood. Or it may be that the conference just wasn’t the right venue for your work.

Receiving reviews can be one of the most dispiriting events in your post-graduate research career. This applies as much for the senior professor receiving reviews of a major program grant as to the PhD student receiving their first set of reviews for a submitted conference paper.

If you feel the reviews are unfair you feel victimised and isolated, but ironically, you are not alone. Everyone else also feels like that, just about a different paper and different set of reviewers. It’s how you cope next that counts!

Good luck!