The NIPS Experiment

Examining the Repeatability of Peer Review

[Neil Lawrence](http://inverseprobability.com/) [@lawrennd](http://twitter.com/lawrennd)

with Corinna Cortes (Google, NYC)

MLPM Summer School

21st September 2015

# rerun Fernando's script to load in css
%run talktools

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import binom
from IPython.display import HTML

NIPS

• NIPS is premier outlet for machine learning.
• Particularly at interface with Neuroscience and Cognitive Science

NIPS in Numbers

• In 2014 NIPS had:
  1678 submissions
  414 accepted papers
  20 oral presentations
  62 spotlight presentations
  331 poster presentations
  19 papers rejected without review
  

NIPS in Numbers

• To review papers we had:
  • 1474 active reviewers (1133 in 2013)
  • 92 area chairs (67 in 2013)
  • 2 program chairs

The NIPS Experiment

• How consistent was the process of peer review?
• What would happen if you independently reran it?

The NIPS Experiment

• We selected ~10% of NIPS papers to be reviewed twice, independently.
• 170 papers were reviewed by two separate committees.
  • Each committee was 1/2 the size of the full committee.
  • Reviewers allocated at random
  • Area Chairs allocated to ensure distribution of expertise

Notes on the Timeline for NIPS

AC recruitment (3 waves):

17/02/2014 08/03/2014 09/04/2014

We requested names of reviewers from ACs in two waves:

25/03/2014 11/04/2014

Reviewer recruitment (4 waves):

14/04/2014 28/04/2014 09/05/2014 10/06/2014 (note this is after deadline ... lots of area chairs asked for reviewers after the deadline!). We invited them en-masse.

06/06/2014 Submission Deadline 12/06/2014 Bidding Open For Area Chairs (this was delayed by CMT issues) 17/06/2014 Bidding Open For Reviewers 01/07/2014 Start Reviewing 21/07/2014 Reviewing deadline 04/08/2014 Reviews to Authors 11/08/2014 Author Rebuttal Due 25/08/2014 Teleconferences Begin 30/08/2014 Teleconferences End 1/09/2014 Preliminary Decisions Made 9/09/2014 Decisions Sent to Authors

Decision Making Timeline

Deadline 6th June

1. three weeks for paper bidding and allocation
2. three weeks for review
3. two weeks for discussion and adding/augmenting reviews/reviewers
4. one week for author rebuttal
5. two weeks for discussion
6. one week for teleconferences and final decisons
7. one week cooling off

Decisions sent 9th September

Speculation

• To check public opinion before experiment: scicast question

Results

		Committee 1
		Accept	Reject
Committee 2	Accept	22	22
	Reject	21	101

4 papers rejected or withdrawn without review.

Summarizing the Table

• inconsistency: 43/166 = 0.259
  • proportion of decisions that were not the same
• accept precision $0.5 \times 22/44$ + $0.5 \times 21/43$ = 0.495
  • probability any accepted paper would be rejected in a rerunning
• reject precision = $0.5\times 101/(22+101)$ + $0.5\times 101/(21 + 101)$ = 0.175
  • probability any rejected paper would be rejected in a rerunning
• agreed accept rate = 22/101 = 0.218
  • ratio between aggreed accepted papers and agreed rejected papers.

Reaction After Experiment

• Public reaction after experiment documented here

• Open Data Science (see Heidelberg Meeting)

• NIPS was run in a very open way. Code and blog posts all available!

• Reaction triggered by this blog post.

A Random Committee @ 25%

		Committee 1
		Accept	Reject
Committee 2	Accept	10.4 (1 in 16)	31.1 (3 in 16)
	Reject	31.1 (3 in 16)	93.4 (9 in 16)

Stats for Random Committee

For random committee we expect:

• inconsistency of 3 in 8 (37.5%)
• accept precision of 1 in 4 (25%)
• reject precision of 3 in 4 (75%) and a
• agreed accept rate of 1 in 10 (10%).

Actual committee's accept precision markedly better with 50% accept precision.

Uncertainty: Accept Rate

rv = binom(340, 0.23)

x = np.arange(60, 120)
fig, ax = plt.subplots(figsize=(10,5))
ax.bar(x, rv.pmf(x))
display(HTML('<h3>Number of Accepted Papers for p = 0.23</h3>'))
ax.axvline(87,linewidth=4, color='red') 
plt.show()

Number of Accepted Papers for p = 0.23

Uncertainty: Accept Precision

• How reliable is the consistent accept score?

rv = binom(166, 0.13)
x = np.arange(10, 30)
fig, ax = plt.subplots(figsize=(10,5))
ax.bar(x, rv.pmf(x))
display(HTML('<h3>Number of Consistent Accepts given p=0.13</h3>'))
ax.axvline(22,linewidth=4, color='red') 
plt.show()

Number of Consistent Accepts given p=0.13

Bayesian Analysis

• Multinomial distribution three outcomes.
• Uniform Dirichlet prior.
  • (doesn't account for implausability of 'active inconsistency')

def posterior_mean_var(k, alpha):
    """Compute the mean and variance of the Dirichlet posterior."""
    alpha_0 = alpha.sum()
    n = k.sum()
    m = (k + alpha)
    m /= m.sum()
    v = (alpha+k)*(alpha_0 - alpha + n + k)/((alpha_0+n)**2*(alpha_0+n+1))
    return m, v

k = np.asarray([22, 43, 101])
alpha = np.ones((3,))
m, v = posterior_mean_var(k, alpha)
outcome = ['consistent accept', 'inconsistent decision', 'consistent reject']
for i in range(3):
    display(HTML("<h4>Probability of " + outcome[i] +' ' + str(m[i]) +  "+/-" + str(2*np.sqrt(v[i])) + "</h4>"))

Probability of consistent accept 0.136094674556+/-0.0600011464859

Probability of inconsistent decision 0.260355029586+/-0.087455352832

Probability of consistent reject 0.603550295858+/-0.150347100243

def sample_precisions(k, alpha, num_samps):
    """Helper function to sample from the posterior distibution of accept, 
    reject and inconsistent probabilities and compute other statistics of interest 
    from the samples."""

    k = np.random.dirichlet(k+alpha, size=num_samps)
    # Factors of 2 appear because inconsistent decisions 
    # are being accounted for across both committees.
    ap = 2*k[:, 0]/(2*k[:, 0]+k[:, 1])
    rp = 2*k[:, 2]/(k[:, 1]+2*k[:, 2])
    aa = k[:, 0]/(k[:, 0]+k[:, 2])
    return ap, rp, aa

ap, rp, aa = sample_precisions(k, alpha, 10000)
print ap.mean(), '+/-', 2*np.sqrt(ap.var())
print rp.mean(), '+/-', 2*np.sqrt(rp.var())
print aa.mean(), '+/-', 2*np.sqrt(aa.var())

0.508753122542 +/- 0.128980361541
0.822081388624 +/- 0.0531283853908
0.184137068656 +/- 0.0694158213505

fig, ax = plt.subplots(1, 3, figsize=(15, 5))
_ = ax[0].hist(ap, 20)
_ = ax[0].set_title('Accept Precision')
ax[0].axvline(0.25, linewidth=4)
_ = ax[1].hist(rp, 20)
_ = ax[1].set_title('Reject Precision')
ax[1].axvline(0.75, linewidth=4)
_ = ax[2].hist(aa, 20)
_ = ax[2].set_title('Agreed Accept Rate')
ax[2].axvline(0.10, linewidth=4)

<matplotlib.lines.Line2D at 0x10dd28910>

Conclusion

• For parallel-universe NIPS we expect between 38% and 64% of the presented papers to be the same.
• For random-parallel-universe NIPS we only expect 25% of the papers to be the same.

Discussion

• Error types:
  • type I error as accepting a paper which should be rejected.
  • type II error rejecting a paper should be accepted.
• Controlling for error:
  • many reviewer discussions can be summarised as subjective opinions about whether controlling for type I or type II is more important.
  • with low accept rates, type I errors are much more common.
• Normally in such discussions we believe there is a clear underlying boundary.
• For conferences there is no clear separation points, there is a spectrum of paper quality.
• Should be explored alongside paper scores.