Evaluate Papers Fast

Pattern Name: Evaluate Papers Fast -- Jens Palsberg, mailto:palsberg@theory.lcs.mit.edu

Problem: Assign an overall-evaluation score to a research paper in less than an hour.

Context: As a member of a program committee one receives a batch of papers to evaluate, typically 20-60. Each paper must be assigned an overall-evaluation score, and the task has to be completed fairly quickly, typically in 5-7 weeks.

Forces: One should ensure that a paper reflects a novel idea, that it is important, that it tackles a difficult problem, and that it does so correctly. Unless one has little else to do, a complete investigation, understanding, and comparison of the papers is out of the question. At the same time, one wants to do as good a job as possible.

Solution: Read the title, the author's names, the abstract, the list of references, the related work, the introduction, the conclusion, in that order. Then, spend time on the technical part.

Rationale: The first thing to find out is: does the paper present something new? This is likely to be apparent from the abstract, the list of references, and the related work. The next thing to find out is: is the new stuff important? This is likely to be apparent from the introduction and the conclusion. If the paper at this point still seems to say something new and important, the next thing to find out is: is the new stuff difficult? Unless the paper has solved an open problem, then the answer is to be found in the technical part of the paper. While reading this part, one checks for correctness as much as time permits.

Resulting context: By focusing on the most important questions about the quality of the paper, one spends the available time as well as possible.

Sketch:

           Title
           Authors
           Abstract -------------+
  +------- Introduction <------+ |
  |        Technical stuff <-+ | |
  |   +--> Related work -----|-+ |
  +---|--> Conclusion -------+   |
      +--- References <----------+

-- Entered by JimCoplien

I agree with almost all of this except the new part. If all you are looking for is groundbreaking work, then what about the work that refines or rediscovers corners of the world that others have only partially surveyed? Doesn't only looking for new work play in to the hands of the NoveltyVampires? Or is this the boundary between the worlds of patterns miners and researchers?

-- KyleBrown

Whereas I agree with most of it, I think it would be better not to read the authors' names at all until you've decided what you think of the paper. -- GarethMcCaughan