ACM UbiComp 2014 Policy on Anonymization in the Review Process
Background: Anonymization is sometimes used in academic review processes in a number of ways.
First, reviewer identities may be kept from the authors; this allows reviewers to write critical
reviews without the authors identifying them. Second, reviewer identities may be kept from other
reviewers; this can help more junior reviewers share their frank assessment of the paper without
worrying that they are contradicting more senior people who they perhaps normally would not
contradict. Third, author identities may be kept from reviewers; this can help avoid bias (or
unintentional bias, or just perceived bias) towards authors (or their institutions) by reviewers.
In the text below, “reviewer” includes both “PC member” and “external reviewers” (who are not
PC members).
None of the policies below override the conflict policy – in no case will a reviewer identity
be shared with anyone who has a conflict with a paper.
UbiComp has the following policies on anonymization:
- Reviewer identities will always be kept confidential from authors and conflicted PC
members/PC chairs. The only exception to this is during a “shepherding” process,
where a PC member is asked to help the authors through direct communication to make
changes to a paper. This process MAY be done using non-anonymized means (e.g. email)
if the relevant PC member agrees.
- External reviewer identities will be kept confidential from external reviewers. PC
member identities MAY be kept confidential from external reviewers (but this is often
not possible, e.g. due to reviewing system limitations and because the external
reviewer obviously knows who invited them to review).
- Reviewer names WILL NOT be kept confidential from PC members (except those conflicted).
This is because (a) typically both PC members assigned to a UbiComp paper allocate
reviewers, so it helps them to know the identity of the other reviewers so they can
solicit reviews from a broad range of viewpoints (e.g. not unwittingly inviting another
member of the same institution to review), and (b) during the PC meeting, it sometimes
helps to know who a review is by, and (c) it enables excellent reviewers to receive some
recognition since PC members will see their identities. Note that typically only PC
members assigned to a paper will see reviewer identities during the review phase; only
during the PC meeting phase is it necessary for all PC members to see reviewer identities.
- Author identities WILL NOT be hidden from PC members reviewing the paper, or (at the PC
meeting) from all PC members. The former is so that PC members can assign non-conflicted
external reviewers, and it is important because the external reviewers themselves often
cannot tell if there is a conflict if the paper is anonymized. The latter is for
conflict detection at the PC meeting – by showing the author list of a paper prior to its
discussion, it is easier to reliably exclude conflicted people.
- During the review process, the authorship of the paper should be de-emphasised so that
focus is on the paper content only. This includes (a) requiring authors to leave blank
the author list area of a paper’s first page, so that it is not part of reviewers’ first
impressions of a paper, (b) after the reviewer assignment phase is complete, if possible,
turning off the option for PC members to see author identities, and (c) informing all
reviewers that author and institution identities must not be used to influence review
scores or any other aspect of the review process.
- Authors MAY fully anonymize their paper (i.e. not just the normal author list area, but
the full paper) if they wish. If this is done, external reviewers will not find out who
the authors are. This is accomplished by e.g. anonymizing faces or names in figures, and
anonymizing all self-references by referencing the prior work neutrally (e.g. "In previous
work, Smith et al. found..." rather than "In previous work, we found..."). The reason this
is optional is that some papers are hard to fully anonymize (e.g. because the systems they
refer to are well-known to the community), and in some cases anonymization can be
detrimental to fair review (e.g. for a study about a well-known system, it makes a big
difference to the perceived contribution if the authors are those who built the system or
not)
- Authors MUST NOT remove relevant self-references from a paper for anonymization reasons,
e.g. by saying “blank for blind review”. Papers using "blank for blind review" may be
rejected simply because it is impossible to evaluate the contribution of a paper if other
related papers are not accessible. Authors who wish to be anonymous to external reviewers
should reference prior work neutrally(e.g. "In previous work, Smith et al. found..." rather
than "In previous work, we found..."). In the case of simultaneous related (but not identical)
submissions, authors may remove author names from the citation, but if so they must provide
the PC chairs with an anonymized copy of the other paper. Please also refer to the Policy on
Novelty.