Intructions for Reviewers

Thank you for reviewing for EACL 2021! In order to ensure the quality of reviews, we would like to share with you the following instructions for reviewing EACL 2021 papers. Please read carefully these instructions before you start reviewing EACL papers.

Confidentiality

Please note that the content of any submission to EACL 2021, information regarding the participants,and the content of discussions on submissions, are strictly confidential.

Furthermore, note that at EACL we employ a double-blind reviewing process. So, in your review, please refrain from mentioning any details that may disclose your identity (including describing your own work, excessive self-references, personal remarks, etc.)

The Review Form

Our review form and instructions are roughly based on previous ACL events. To improve the efficiency of both reviewing and meta reviewing we purposefully employ a more succinct review form, containing two main sections only:

Section 1. In-Depth Review: This section is for you to give your overall assessment of the paper and to provide evidence to support your opinions. It has four items concerning: high-level comments, low-level comment, reasons to accept, and reasons to reject. This section will be shared with both the authors and the area chairs.

In this section, there are 7 subsections:

  • The core review (high-level comments): This is the most important part. It should include your view of the main contributions that the paper is intended to make and how well it succeeds at making these contributions. From your point of view, what are the significant strong and weak aspects of the paper and the work it describes? This could be a 2 paragraph (or longer) essay and/or bullet points. Remember to describe how the work advances the state of knowledge in computational linguistics and/or highlights why it fails to make a sufficient contribution If you have any questions to the author(s), please also include them in this section as well. These questions will be important for the author response period, and and they may also point to future directions and suggestions for improvements of the paper.
  • Typos, Grammar, Style and References (low-level comments): include here your comments concerning how the paper may be improved. Specifically, suggestions concerning the organization and style of the writing and possible language issues. Feel free to mention typos and any other suggestions that could help the authors to present their work in a clearer and more readable manner. Also, this is the place to mention issues with references, including references format, missing citations and incomplete items in the bibliography.
  • Reasons to accept: please briefly summarize the main reasons why this paper should be accepted for the conference, and how the NLP community would benefit from it. You may refer back to your review to provide more context and details.
  • Reasons to reject: please briefly summarize the main reasons that this paper cannot be published and presented in its current form. What are the parts that would need to be improved in order to advance the state of knowledge?
  • Overall recommendation: Here you are asked to synthesize the above and come up with your own recommendation for the paper.
    • Like EMNLP 2020, we have used a 5 point scale with a half point increments. The detailed explanation for each point level is provided in the review form. These numbers are just a concise way of expressing your overall opinion and relative importance of the factors mentioned above.
    • Similar to EMNLP 2020, we are allowing a rating of 3 (ambivalent) but, please try to take a stand on whether the paper is above or below the borderline, e.g., by selecting 2.5 or 3.5. However, as much as we would like you to do that, if you think this is indeed a borderline paper or you are not able to decide, you should use 3.
    • Decisions will be made not just on the scores and certainly not on average scores, but will also take into account the whole review, reviewer discussion and Area Chair meta-reviews and recommendations. However it is important to align your recommendation with the reasoning given above, so that authors will be able to understand the motivation for the recommendations and how decisions were arrived at.
  • Reviewer confidence: This section should be used to inform the committee and authors how confident you are about your recommendation, taking into account your own expertise and familiarity with this area and the paper’s contents.
  • Author response: There will be an author response period. It is important that you check whether author responses have cleared up your questions or misunderstandings. This may influence your overall recommendation and the core review. If that’s the case, please update your recommendation and review accordingly (and state in your review any new decisions you made so the Area Chairs are aware).

Section 2. Confidential Information: Here we ask you about recommendations for awards, the recommended presentation mode, and confidential comments to the area chairs and PC chairs. Your answers to questions in this section will not be shared with the authors, only with the Meta-Reviewers, Track Chairs and PC chairs.

For Best Paper Award, please be open minded and feel free to nominate good quality papers even though they may not be the typical types. These can be a survey paper, an opinion paper, a paper about resources and datasets, a paper for low resource language, an analysis paper, etc. A committee will evaluate best paper candidates, and we would like to have a wide variety of paper types in the candidate pool, not just vanilla empirical research papers. We will have separate long and short paper awards.

Supplementary Materials

Supplementary materials are allowed as a stand-alone document uploaded as an additional file. Supplementary materials are, as the name suggests, supplementary, and you have no obligation to read them. You should treat them like other citations in submissions that may be helpful in understanding background or details beyond the scope of the paper itself.

In addition to that, authors may provide additional information about their datasets and experiments in Appendix, and attach a zip file with resources such as code and data. Please take some time to check those, if applicable. Papers providing code and data are preferred to those not providing sufficient basis for reproducibility. Uploading data and/or code alongside paper submissions is preferred over supplying a hyperlink, as the latter could violate the double-blind review practice.

What Makes A Good Review?

The below is a non-conclusive list of characteristics of what we consider to be a good review.

  1. The review is written in a clear way, it is clearly organized, and uses constructive language. Specifically, the written review respects the authors and the work they had put into the paper. A good review helps the authors improve their presentation in the camera ready or improve future submissions of the reported work.
  2. The review clearly demarcates the essence of the contribution, that is: how it will advance the NLP community, where it has the potential to make impact, what would be follow up research questions that could be promoted if the paper is to be published.
  3. The review clearly outlines the strengths of the paper. This includes anything that was well done in your opinion, including: phrasing the questions answered or the hypothesis being tested, novel and sound model design, proper experimental setup and execution, excellent comparison with previous work, exceptional results, great potential for impact on the greater community, social and ethical implications, etc.
  4. The review clearly outlines the weaknesses of the contribution and provides concrete suggestions as to how these may be overcome in future revisions. Note: please avoid generally dismissive non-concrete phrases such as: “evaluation is weak” or “results are low”. Replace such general phrases with specific indicators and evidence, such as: “I suggest that the following X experiment is missing, requires Y metrics or Z data” or “I suggest to compare the results in table X to the experiment in paper Y with metric Z”.
  5. The review takes a clear stand towards acceptance or rejection (rather than ambivalent-3)
  6. The reviewing doesn’t end in writing the review! A good reviewing process proceeds into the discussion, helping the Area Chairs and PC chairs make a quality decision.
  7. The review takes into account the Author Response and respond to it (as applicable).
  8. The review considers the discussion with fellow reviewers (even if retaining its original score)

You may find additional advice in the following post on writing good reviews by the EMNLP 2020 PC chairs, and the resources therein. ACs will be instructed to flag poor reviews, ask reviewers to revise their reviews or provide objective reasons to justify your positions.

Secondary Reviewers

As in most previous NLP conferences, you are allowed to solicit help from others. However, when it comes to writing the final review and giving the final scores, we expect you to take the secondary reviewer’s review and rewrite it using your own words and adjust the scores when you see fit. Essentially, the final review should reflect your own opinions about the paper, and you need to be able to justify the opinions you present in the final review.

Best Reviewers

As in most previous NLP conferences, we are going to select a list of best reviewers and provide honorable mention of those in the written and oral program.

Format of Submissions

The program chairs and area chairs have already identified submissions that violated our formatting guidelines and have desk-rejected those submissions. Therefore, you do not need to worry about formatting issues with the submissions assigned to you. If you think the paper violates the format guidelines, please contact your area chairs or PCs. Otherwise, the paper format shouldn’t be used to weigh down your evaluation of the paper.

Important Dates

Your reviews are due Monday, November 16, 2020 (11:59pm anywhere on Earth). Please note that there is a reviewer discussion period from November 29 to December 2 after the author response. Your duties are listed below. Don’t leave reviewing to the last minute!

  • October 29 – November 16: Review Period
  • November 22 – November 25: Author response period