Review a Research Paper: Constructive Critique in Five Steps

I’m helping organize the ICCQ conference this year for the fourth time, with the in-cooperation support of the IEEE Computer Society. Based on this short-term experience, I can assert that reviewing research papers is a skill that even some reputable and experienced academicians either don’t possess or are too lazy to apply. We often encounter sketchy, subjective, and disputable reviews that don’t assist authors but only frustrate and discourage them. In this short blog post, as an absolute amateur in the subject matter, I will try to summarize how to review an academic research paper (thus mostly helping other newbies).

Structure your review as a plain text of five paragraphs, each answering one question:

How did you understand the thesis of the paper?
What are the positive points?
What are the major inconsistencies?
What are the minor mistakes?
What’s next?

First, provide a brief summary of the paper. The main purpose of this paragraph is to ensure that you, the reviewer, have actually read the paper and understood what it’s about. Such summarization helps build rapport between you and the readers of your review—the authors of the paper, whom you intend to criticize. The better your summary, the more they will respect your negative points, taking them constructively.

Then, identify the positive points of the paper, again demonstrating that you have read and appreciate it. Here is a cheat list of the most typical merits a good research paper may have (most important at the top):

The idea is new and contributes significantly to the field.
The research question(s) are clear, well-defined, and relevant.
The method is robust, appropriate, and clearly described.
The paper thoroughly reviews existing literature.
The data found effectively supports the conclusions.
The results are significant.
The conclusion effectively summarizes the research.
The paper is well-organized.
The paper is well-written with clear and concise language.

Next, highlight the major inconsistencies. This list of typical mistakes may help you (the most severe ones are at the top):

Certain parts of the paper are obvious plagiarism.
The purpose of the research is unclear.
The need for this research in the field is not obvious.
The research questions are not clearly defined.
Literature review misses important related work.
Conclusions are not fully supported by the data.
Data is distorted or cherry-picked to fit a hypothesis.
The viewpoints presented are biased and/or lack balance.
Important details of the research method are missing.
Not all limitations or threats to validity are identified.
The structure is not coherent.
Some statements are ambiguous.

Then, mention minor mistakes. The difference between minor and major problems is that a minor problem is not a “show stopper”: a paper with minor mistakes but without major ones may be accepted for publication, while the opposite is not true. A paper with major issues must be rejected with a suggestion for rework by the authors. Here is a list of the most typical minor issues:

The abstract is inadequate.
Sources are misquoted or incorrectly cited.
The title of the article is inappropriate.
There are grammatical errors, typos, and unclear language.

Finally, conclude your review: what should be done next? Do you suggest publishing the paper? Do you think the authors are moving in the right direction? Should they continue working on this topic, or would it be better to abandon it for something more meaningful? Be honest and sincere; don’t be afraid of offending them: the review is anonymous anyway.

Obviously, I’m joking. It’s easy to offend an author, especially a young one. Thus, as a good reviewer, you must understand your mission: the review you provide should help the authors by encouraging and educating them. Making them feel miserable is definitely not the purpose of the review, though it is sometimes an unfortunate side effect. Try to minimize it.

Here is a toy example:

In their research, the authors claim that all programmers 
are lazy and selfish creatures, grounding their conclusion 
on a survey of 150 respondents.

Pros:
  - An important topic was addressed.
  - The reasoning is clear and concise.
  - The conclusion is very true.

Major cons:
  - Similar research done by Dean [1] is overlooked.
  - It's obvious that they are lazy; why another research?
  - Only Java programmers were interviewed.

Minor cons:
  - Typos and broken English here and there.
  - The font in most figures is too small.

Even though the subject of the research is important, 
I believe this paper is not yet ready for publication 
and requires significant rework.

[1] Dean et al., Programmers Are Super Lazy, 2022