Best Practices for Running the Conference Review Process Based on USENIX ATC 2018

I got the opportunity to participate in the review process as a program committee member for USENIX ATC 2018. USENIX ATC is a big conference in term of submissions: this year, it received more than 400 complete submissions. To handle such a huge load on reviewing, the chairs had purposedly doubled the number of program committee members to be 70 this year. Still, it requires a careful planning to run the paper reviewing process smoothly.

1. Two rounds reviews. The load is roughly 3 days per paper for each program committee member. Each paper got 2 reviews during the first round and another 2, if a paper entered the second round (as long as a paper got at least one accept, whether it is an accept or a weak accept, it was qualifed to enter the second round).
2. One week buffer time was introduced between the end of the first round and the start of the second round. The primary benefit or goal is to give people some time to turn in late reviews. This is very effective: in average, everyone was supposed to turn in 12 reviews for the first round. However, when the deadline of the first round was past, in average, each PC member had turned in only 4 reviews. The number went to 10 reviews per person by the end of the next week. Deadline is the primary productive force！
3. Another week was introduced between the end of the second round and the final PC meeting. The primary goal is to give people some time for online discussion, to reach consesuses (pre-accept or pre-reject) before in person meeting. This can reduce the number of papers that need to be discussed at the final PC meeting.
4. Collected those who will come to the PC meeting and assign them as discussion leaders for papers that entered the second round. Right after the second round reviews ended, discussion leaders were assigned and they were encouraged to start online discussions as soon as possible.