Reviewer Instructions for AISTATS 2016
Reviews must be entered electronically through the CMT system:
Each review should begin with a paragraph providing an overview of the
paper, and summarising its main contributions. In particular some thought
should be given to how the paper fits with the aims and topics of the
conference (not interpreted overly narrowly). The ideas presented should be
related to previous work in the field.
The next section of the review should deal with major comments, issues that
the reviewer sees as standing in the way of acceptance of the paper, or issues
that should be addressed prior to publication, or reasons for rejecting the
The final section of the review should deal with any minor issues, such as
typographical errors, spelling mistakes, or areas where presentation could be
For AISTATS 2016, reviewers may request code (and as necessary, accompanying
data) in the initial reviews, which will be provided as part of the author
response. This might be, for instance, to check whether the authors' method
works as claimed, or whether it correctly treats particular scenarios the
authors did not consider in their initial submission. Code/data should only be
requested in the event that this is the deciding factor in paper acceptance.
The request should be reasonable in light of the duration of the discussion
period, which limits the time available for review. The SPC member in charge
of the paper will confirm whether a code/data request is warranted and
reasonable. Authors may only submit separate code and data at the invitation of
a reviewer; otherwise, the usual restrictions apply on author response length.
The conference chairs will enable the anonymous transfer of code and data to
the relevant reviewers.
Contributions of AISTATS papers can be categorised into four areas a)
algorithmic, b) theoretical, c) unifying or d) application.
Algorithmic contributions may make a particular approach feasible for the
first time or may extend the applicability of an approach (for example allowing
it to be applied to very large data sets).
A theoretical contribution should provide a new result about a model or
algorithm. For example convergence proofs, consistency proofs or performance
A unifying contribution may bring together several apparently different
ideas and show how they are related, providing new insights and directions for
Finally, an application contribution should typically have aspects that
present particular statistical challenges which require solution in a novel way
or through clever adaptation of existing techniques.
A paper may exhibit one or more of these contributions, where each of them
are important in advancing the state of the art in the field. Of course, at
AISTATS we are also particularly keen to see work which relates machine
learning and statistics or highlights novel connections between the fields or
even contrasts them.
One aspect of result presentation that is often neglected is a discussion of
the failure cases of an algorithm, often due to concern that reviewers will
penalize authors who provide this information. We emphasize that description of
failure cases as well as successes should be encouraged and rewarded in
When reviewing bear in mind that one of the most important aspects of a
successful conference paper is that it should be thought provoking. Thought
provoking papers sometimes generate strong reactions on initial reading, which
may sometimes be negative. However, if the paper genuinely represents a
paradigm shift it may take a little longer than a regular paper to come around
to the author's way of thinking. Keep an eye out for such papers, although they
may take longer to review, if they do represent an important advance the effort
will be well worth it.
Finally, we would like to signal to newcomers to AISTATS (and to
machine-learning conferences generally) that the review process is envisioned
in exactly the same spirit as in a top quality journal like JRSS B, JASA, or
Annals of Statistics. Accepted contributions are published in proceedings, and
acceptance is competitive, so authors can rightly include these contributions
in their publication list, on par with papers published in top quality
journals. Further, AISTATS does not give the option to revise and resubmit: if
a paper cannot be accepted with minor revisions (e.g. as proposed by the
authors in their response to the reviews), it should be rejected.
Given the culture gap between the statistics and machine learning
communities, we thus want to emphasize from the start the required levels of
quality and innovation. All deadlines are very strict, as we cannot delay an
overall tight schedule.
Confidentiality and Double Blind Process
AISTATS 2016 is a double blind reviewed conference. Whilst we expect authors
to remove information that will obviously reveal their identity, we also trust
reviewers not to take positive steps to try and uncover the authors' identity.
We are happy for authors to submit material that they have placed online as
tech reports (such as in arXiv), or that they have submitted to existing
workshops that do not produce published proceedings. This can clearly present a
problem with regard to anonymisation. Please do not seek out such reports on
line in an effort to deanonymise.
The review process is double blind. Authors do not know reviewer identities,
and this includes any authors on the senior programme committee (the area
chairs). However, area chairs do see reviewer identities. Also, during the
discussion phase reviewer identities will be made available to other reviewers.
In other words, whilst the authors will not know your identity, your
co-reviewers will. This should help facilitate discussion of the papers.
If a reviewer requests code from the authors, this code should be anonymized
(e.g. author names should be removed from the file headers). That said, we
understand that it might be difficult to remove all traces of the authors from
the files, and will exercise reasonable judgment if innocent mistakes are made.
The AISTATS reviewing process is confidential. By agreeing to review you
agree not to use ideas, results, code, and data from submitted papers in your
work. This includes research and grant proposals. This applies unless that work
has appeared in other publicly available formats, for example technical reports
or other published work. You also agree not to distribute submitted papers,
ideas, code, or data to anyone else unless permission is gained from the
program chairs. If you request code and accompanying data, you agree that this
is provided for your sole use, and only for the purposes of assessing the
submission. All code and data must be discarded once the review is complete,
and may not be used in further research or transferred to third parties.
The CMT Reviewing System
The first step in the review process is to enter conflicts of interests.
These conflicts can be entered as domain names and also by marking specific
authors with whom you have a conflict. The use of double blind reviewing means
you may not able to determine the papers you have a conflict with, so it is
important you go through this list carefully and mark any conflicts. You should
mark a conflict with anyone who is or ever was your student or mentor, is a
current or recent colleague, or is a close collaborator. If in doubt, it's
probably better to mark a conflict, in order to avoid the appearance of
impropriety. Your own username should be automatically marked as a conflict,
but sometimes the same person may have more than one account, in which case you
should definitely mark your other accounts as a conflict as well. If you do not
mark a conflict with an author, it is assumed that you do not have a conflict
CMT also requests subject information which will be used to assist
allocation of reviewers to papers. Please enter relevant keywords to assist in
You can revise your review multiple times before the submission. Your formal
invite to be a reviewer will come from the CMT system. The email address used
in this invite is your login, you can change your password with a password
reset from the login screen.
Supplementary material is allowed by AISTATS 2016. For example, this
supplementary material could include proofs, video, source code or audio. As a
reviewer you should feel free to make use of this supplementary material to
help in your review.
Simultaneous submission to other conference venues in the areas of machine
learning and statistics is not permitted.
Simultaneous submission to journal publications of significantly extended
versions of the paper is permitted, as long as the publication date of the
journal is not before June 2016.
Some parts of these reviewer instructions are based on those for NIPS.