Guide to Reviewing CHI Papers and Notes

Introduction

The CHI review process for 2010 will continue to make use of the new subcommittee structure and contribution types that were introduced for CHI 2009. This review structure is different from prior years and from many other HCI conferences. The key points in undertaking reviews under this structure are:

Your primary criterion for judging a paper is: Does this submission provide a strong contribution to the field of HCI?
In recognition of the diversity of different contributions that can be made in the interdisciplinary work of HCI, authors indicate a contribution type. As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type (see: Selecting a contribution type).
In recognition that there are range of different types of expertise in our field, and in an attempt to have each paper judged by true experts in its topic, authors submit their papers to subcommittees which are clustered around different topics (see: Selecting a subcommittee). However, matches of papers to subcommittees are not always clean or perfect. So as a reviewer you should not judge a paper by how well it fits the subcommittee theme.
Reviewers rate the paper using a 5-point ranking scale; your written appraisal must support your numeric ranking.

Contribution Types

There has been increasing concern over the years that CHI reviewers favor certain styles of papers over others; the perception is that only papers following a certain narrow ‘formula’ can get accepted. To discourage this narrowing of what makes an acceptable paper, each author can now specify a contribution type for their submission (see: Selecting a contribution type). The description of each contribution type includes a set of questions that you, as a reviewer, should consider when evaluating the contribution of the paper. What is important about these questions is that they may push you to think about the contribution in ways that differ from what you may perceive as the CHI formula paper.

However, these questions are not a checklist. The submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI? This is important. The contribution type and its associated questions are a guide to assist your thinking; they are not a strict criteria by which you should measure acceptance. They are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.

What you will see on the review form is:

A summary of the contribution type questions (i.e., review criteria), followed by a Contribution Type Specific Rating. By filling in your score, you are judging the paper according to its contribution type.
Immediately following this is an Overall Rating. By filling this in, you are judging the overall contribution of the paper. While we suggest you use the contribution type as a starting point for appraising the contribution, the paper does not need to exactly match the contribution type criteria.
The various other boxes ask you to provided details. Again, while we suggest you use the contribution type as a starting point for appraising the contribution, we stress that the paper does not need to exactly match the contribution type criteria.

Subcommittees

To improve the reviewing process, authors are now asked to submit their papers to sub-committees, where each sub-committee is responsible for various topic areas typically seen at CHI (see: Selecting a subcommittee for details). Each sub-committee will comprise of a sub-chair and associate chairs who are knowledgeable in these topics. The idea is that, as specialists in the theme areas, they should be able to find good referees (such as you) for each submission, and that as specialists they should be able to better handle the meta-review process.

However, as a reviewer, you should not judge the paper by how well it fits the subcommittee theme(s). Many papers will not cleanly fit into a particularly sub-committee for a variety of reasons, and we do not want to penalize authors for this. Remember, the subcommittee organization is there only to try to improve reviewer matches and to better handle the volume of submissions. If you have a paper that does not fit the subcommittee theme, evaluate it as best you can. Any topic is valid, as long as it fits within the interests of a reasonable fraction of the overall CHI audience.

Papers and Notes

Papers and Notes are both reviewed within the same rigorous review process and at the highest level are judged by very similar criteria (i.e., does this paper or note provide a strong contribution to the field of HCI?). However, it is important as a reviewer to realize that the type of content that is appropriate for each is somewhat different. In particular, Notes present brief and focused research contributions that are noteworthy, but may not be as complete or provide the same depth of results as a full paper. For details about these differences see Papers versus Notes: What is the difference?.

Numeric Ratings and Written Appraisals

The numeric ratings on the review form are on a 5 point scale (with half steps) starting at 1.0 (the lowest ratings) to 5.0 (the highest). Be as accurate with your ratings as possible. As in previous years, it is your written appraisal that is crucial. Make sure your review is balanced, and that its details reflect the numeric rating. The appraisal of a paper should always indicate why the paper deserves that rating. Your review may be discounted if, for example, you numerically rate a paper highly but only indicate its flaws. As well, be polite to authors. Even if you rate a paper poorly, you can critique it in a positive voice. As part of polite reviewing practice, you should always state what is good about a paper first, followed by your criticisms. If possible, you should offer suggestions for improvement along with your criticism.

References

The debate about what makes a good CHI paper has been ongoing for several years, and the introduction of contribution types are an attempt to address this debate. If you are interested, the papers below touch upon this debate and contain references to additional papers that concern it.

Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. CHI '08. ACM, 111-120. DOI=http://doi.acm.org/10.1145/1357054.1357074
Olsen, D. R. 2007. Evaluating user interface systems research. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology. UIST '07. ACM, 251-258. DOI=http://doi.acm.org/10.1145/1294211.1294256
Dourish, P. 2006. Implications for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '06, ACM, 541-550. DOI=http://doi.acm.org/10.1145/1124772.1124855
Newman, W. 1994. A preliminary analysis of the products of HCI research, using pro forma abstracts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). ACM, New York, NY, 278-284. DOI=http://doi.acm.org/10.1145/191666.191766