Recommendations for Reviewers

This document aims to provide a set of ideas and recommendations (as well as pointers to some references) that can assist in reviews with more constructive feedback, while avoiding that authors of non-accepted papers are discouraged from continuing to improve their work and seek new submission opportunities. 

All reviewers are recommended to read carefully this material  and its suggestions. It is important to observe simple examples of shallow and inadequate reviews, as well as suggestions on how to make feedback constructive. Finally, an FAQ with some answers to common reviewer questions is also presented. Furthermore, if you have any recommendations for reviewers or questions that could be added to the FAQ, please contact us.

It is crucial to remember that the role of reviewers is to evaluate the quality of the submitted work, clearly justifying its strengths and weaknesses. In addition, whenever possible, the reviewer may also provide questions and insights that can contribute to the improvement or evolution of the authors’ work. Reviews should preferably be constructive and thorough. The reviewer should aim to use the same scientific rigor expected in the work when writing the review. When writing the review, the reviewer should be concerned with using the same scientific rigor that they expect to find in the work being reviewed.

Finally, it is important to emphasize that the evaluation should be appropriate to the type of work being assessed, thereby avoiding the application of criteria meant for a main track article to an undergraduate research article, where a student is just beginning their journey in research and needs a lot of encouragement and words of support, even if the article is rejected due to some serious issue.

Recommended readings (in Portuguese only)

• Algoritmo estimula revisão por pares construtiva e sem grosserias
• Cinco Princípios para uma Revisão Responsável
• Vamos falar sobre a importância da revisão por pares nos eventos científicos e as discrepâncias entre revisões?
  • Especially from section “O papel do Comitê de Programa”.

Some planning tips before and during the review

You must reserve time in your schedule to do the reviews. Delaying reviews can disturb the planning of the entire event. This is both for coordinators, who need to organize the schedule, and for authors, who need to organize their travel arrangements if their paper is accepted. The time spent on each review is difficult to predict. If this is your first time as a reviewer, plan to review as soon as you can access the papers and register the time spent on each one. After a few reviews, you will be able to predict the time needed to review all the papers assigned to you, allowing you to better plan for future events. If you realize that you will not be able to review a paper by the deadline, let the program committee coordinators know as soon as possible.

Save your review progress as you write. Avoid writing directly in the submission system interface to avoid losing all your work if there is a problem with your connection.

Before finalizing the review, don’t forget to use a spell checker and/or grammar checker on it.

General aspects to consider in the review

• Clarity and structure
  • General organization: evaluate if the work is well-organized, with a clear structure including an abstract, introduction, methods, results, discussion, and conclusion.
  • Clarity of writing: assess whether the text is clear and easy to understand. Verify logical flow and coherence in arguments.
• Content and relevance
  • Objectives and contribution: the work should clearly define its objectives and contributions; once these have been defined, evaluate if the work fulfills its objectives and effectively delivers the alleged contributions.
  • Relevance to the field: ensure that the research topic is relevant and contributes to the field.
  • Originality: consider if the research provides new insights or advances existing knowledge.
  • Depth and breadth: evaluate if the study covers the topic with sufficient depth and breadth. 
• Literature review
  • Coverage: ensure that the literature review covers relevant and up-to-date sources. If you know of other references that seem pertinent, indicate them in the review.
  • Context: verify if the authors have adequately contextualized their work with respect to existing literature and if the context is relevant and applicable.
• Contribution to the field
  • Significance: verify if the authors have adequately contextualized their work with respect to existing literature and if the context is relevant and applicable.
  • Practical implications: evaluate the practical implications and possible applications of the research.
  • Reproducibility: ideally, an academic work should have all the elements that allow its reproduction by other authors under similar conditions. Check if the submission provides elements to reproduce its results; if it does not, try to point out the missing elements.
• Constructive feedback
  • Specificity: provide specific comments and suggestions for improvement in future versions of the work. Also, make it clear in the review which are the most critical issues in the work, i.e., which corrections/improvements will make the most significant difference to the work.
  • Tone: use a respectful and constructive tone, aiming to help the authors improve their work. Be formal in your writing to avoid misunderstandings by the authors. Do not assume that the authors have malicious intentions. There are various reasons behind deficient submissions, including submission deadlines arriving before the work is 100% complete, lack of review by more experienced co-authors, and pure unfamiliarity with techniques, results, or references (or even how to write a good article). Even if you are convinced that an author has significantly failed in some aspect of the work, consider the possibility that there is a benign explanation for this. Seek guidance, even with a view to a possible future submission, rather than condemning or (worse yet) humiliating. Whenever possible, cite good sources for authors to consult, avoiding recommending works of your own co-authorship unless you are convinced that they are highly relevant.
  • Balance: highlight both strengths and areas for improvement in the manuscript.

Examples of superficial and inadequate comments

It is crucial that the reviewer understands the importance of their role and the necessity of performing work that helps improve the authors’ ability to understand what is lacking and to know where to start improving in future opportunities.

• Example 1: vague and useless comments
  • “Your work is not very good. It lacks depth and is not well written.”
  • Why it is not constructive: this review is vague and does not specify which aspects of the work lack depth or how the writing can be improved. It also does not provide guidance on which specific areas need more work. Even in cases where the text has sentences with many errors, for example, grammatical errors, point out some of them, indicating the page where they appear and suggest how they could have been written (it is not necessary to point out all of them if there are many). 
  • this comment does not provide any useful feedback on the work itself. If a reviewer feels unqualified or uninterested, they should decline to review instead of providing useless comments. One way to avoid this in the future is to correctly fill in your areas of interest in the JEMS system as requested by the coordinators. Even though this does not completely solve this type of problem, it greatly reduces the chance of reviewers being assigned to work outside their areas).
  • In this case it is useful to recommend the use of a spell checker and/or grammar checker.
• Example 2: excessive criticism without suggestions
  • “The methodology section is completely flawed, and the data analysis makes no sense. This work should not be accepted or published.”
  • Why it is not constructive: this review is harsh and disdainful without explaining what is wrong with the methodology or data analysis. It does not offer any suggestions on how the authors could improve these sections.
• Example 3: very brief and without details
  • “I didn’t like reading this work. The topic is not interesting or relevant.”
  • Why it is not constructive: the reviewer does not explain why the topic is not interesting or relevant. This type of feedback is very brief and subjective, offering no actionable advice.
• Example 4: focused on personal preferences
  • “I do not like the writing style. It is too informal for my taste.”
  • Why it is not constructive: the reviewer does not explain why the topic is not interesting or relevant. This type of feedback is very brief and subjective, offering no actionable advice.
• Example 5: general praise without substance
  • “This is a great work. Well done!”
  • Why it is not constructive: while positive feedback is important, this review is overly general and does not explain specifically what is good about the work. It does not help the authors understand their strengths or how to replicate them in future work.
• Example 6: non-specific criticism
  • “The results section is confusing.”
  • Why it is not constructive: the feedback does not explain what is confusing about the results section or how it can be improved. It leaves the authors guessing about what changes to make.
• Example 7: dismissive comments
  • “This research is useless. I see no value in it.”
  • Why it is not constructive: this review dismisses the research without providing reasons or suggestions for improvement. It does not help the authors understand why the reviewer thinks the research has no value.
• Example 8: ignoring the purpose of the review
  • “I do not think this topic fits my area of expertise. I am not interested in this subject.”
  • Why it is not constructive: this comment does not provide any useful feedback on the work itself. If a reviewer feels unqualified or uninterested, they should decline to review instead of providing useless comments.  (One way to avoid this is to fill in your areas of interest correctly. Even though this does not completely solve this type of problem, it greatly reduces the chance of reviewers being assigned to work outside their areas).

How to make feedback constructive

To make feedback constructive, reviewers should:

• Be specific: clearly identify what works well and what does not work and justify it (avoid ambiguity in comments just as we do not want to see in articles).
• Provide examples: support criticisms (or praises of the work) with examples, such as presenting examples of which results or graphics have problems, which writing errors were made, or which important related works were overlooked.
• Offer suggestions: whenever possible, provide helpful advice on how to solve the problems. This can be written in different ways: “To make the article stronger, the authors could evaluate the impact of the threshold on the FP rate.” or “The article would benefit from defining a threat model.” or “An evaluation in a real system would give more confidence in the results.” or “A comparison with [xyz] would make the contribution clearer compared to the state-of-the-art.”.
• Balance criticism with praise: the challenge for good reviews is to find technical problems and limitations that were not perceived by the authors. However,  it is also important for the reviewer to correctly acknowledge the strengths listed in the work.
• Be respectful and professional: use an encouraging and supportive tone. If criticizing, always refer to the work (its current state) and never the authors.

Contributors: Diego Kreutz (UNIPAMPA), Altair Santin (PUCPR), Marinho Barcellos (University of Waikato), Marco A. Amaral Henriques (UNICAMP), Rafael Obelheiro (UDESC), Lourenço A. Pereira (ITA), Daniel Macêdo Batista (USP)