Assigning Papers To Reviewers

One of the main jobs of the program chair is assigning papers to program committee members to review.

It would be nice if every member could have read every paper. Unfortunately, since there are usually several hundred submissions, this is impossible. Moreover, OOPSLA papers are from a wide variety of topics, and it is pretty unlikely that one person could review them all. So, the program chair usually has a list of topics that each member wants to see, puts each paper in a set of those topics, and then assigns the papers in such a way that members are likely to review papers that interest them.

People would be more likely to get papers that interest them if they could pick their own. The problem is one of time. There is not very much review time and papers need to be sent out right away.

One possibility is that authors could send all abstracts in electronically, and the program chair could send those out to the reviewers so they can pick what they want to review. I bet that it would be possible to make a very nice WWW interface for committee members to browse the abstracts. When I was program chair, I just assigned papers myself.

For whatever reason, OOPSLA has a high rejection rate. A few papers are universally accepted, the majority are universally rejected, and a fairly large amount are controversial.

One of the problems is that the controversy is usually just between a few people on the committee, because most of them haven't read the paper. The year I was program chair, I addressed this problem by assigning three people to review each paper, and asked them to send me reviews ASAP. If any of them voted to accept, I asked three more people to review the paper. This meant that any paper likely to have a discussion had six people who had read it, not just three.

The main problem I had in implementing this pattern was that I didn't explain it in advance to people. I should have asked people to scan through their papers and review the ones they liked best first, so I could find out about those papers and assign other people to review them. I should have told them "send me the reviews by XXX and I will send you some more papers after that". In general, it is always good to let people know what is going on.

-- RalphJohnson