# Review a Research Paper: Constructive Critique in Five Steps

Source: https://www.yegor256.com/2023/12/17/how-to-review-research-paper.html

I'm helping [organize]({% pst 2021/apr/2021-04-14-how-we-organized-iccq %})
the [ICCQ conference](https://www.iccq.ru) this year for the fourth time, with the
in-cooperation support of the IEEE Computer Society. Based on this short-term
experience, I can assert that reviewing research papers is a skill that even some
reputable and experienced academicians either don't possess or are too lazy to
apply. We often encounter sketchy, subjective, and disputable reviews that
don't assist authors but only frustrate and discourage them. In this short blog
post, as an absolute amateur in the subject matter, I will try to summarize how
to review an academic research paper (thus mostly helping other newbies).


{% jb_picture_body %}

Structure your review as a plain text of _five_ paragraphs,
each answering one question:

  1. How did you understand the thesis of the paper?
  1. What are the positive points?
  1. What are the major inconsistencies?
  1. What are the minor mistakes?
  1. What's next?

First, provide a brief summary of the paper. The main purpose of this paragraph
is to ensure that you, the reviewer, have actually read the paper and
understood what it's about. Such summarization helps build _rapport_ between you
and the readers of your review—the authors of the paper, whom you intend to
criticize. The better your summary, the more they will respect your negative
points, taking them constructively.

Then, identify the **positive points** of the paper, again demonstrating that you
have read and appreciate it. Here is a cheat list of the most typical merits a
good research paper may have (most important at the top):

  * The idea is new and contributes significantly to the field.
  * The research question(s) are clear, well-defined, and relevant.
  * The method is robust, appropriate, and clearly described.
  * The paper thoroughly reviews existing literature.
  * The data found effectively supports the conclusions.
  * The results are significant.
  * The conclusion effectively summarizes the research.
  * The paper is well-organized.
  * The paper is well-written with clear and concise language.

Next, highlight the **major** inconsistencies. This list of typical
mistakes may help you (the most severe ones are at the top):

  * Certain parts of the paper are obvious [plagiarism](https://en.wikipedia.org/wiki/Plagiarism).
  * The purpose of the research is unclear.
  * The need for this research in the field is not obvious.
  * The research questions are not clearly defined.
  * Literature review misses important related work.
  * Conclusions are not fully supported by the data.
  * Data is distorted or cherry-picked to fit a hypothesis.
  * The viewpoints presented are biased and/or lack balance.
  * Important details of the [research method]({% pst 2023/oct/2023-10-11-method-of-research %}) are missing.
  * Not all limitations or threats to validity are identified.
  * The [structure]({% pst 2022/aug/2022-08-24-research-paper-template %}) is not coherent.
  * Some statements are ambiguous.

Then, mention **minor** mistakes. The difference between minor and major
problems is that a minor problem is not a "show stopper": a paper with minor
mistakes but without major ones may be accepted for publication, while the
opposite is not true. A paper with major issues must be rejected with a
suggestion for rework by the authors. Here is a list of the most typical minor
issues:

  * The abstract is inadequate.
  * Sources are misquoted or [incorrectly cited]({% pst 2023/sep/2023-09-05-style-checker-for-bibtex-files %}).
  * The title of the article is inappropriate.
  * There are grammatical errors, typos, and unclear language.

Finally, conclude your review: what should be done **next**? Do you suggest
publishing the paper? Do you think the authors are moving in the right
direction? Should they continue working on this topic, or would it be better to
abandon it for something more meaningful? Be honest and sincere; don't be
afraid of offending them: the review is _anonymous_ anyway.

Obviously, I'm joking. It's easy to offend an author, especially a young one.
Thus, as a good reviewer, you must understand your mission: the review you
provide should help the authors by _encouraging_ and _educating_ them. Making
them feel miserable is definitely not the purpose of the review, though it is
sometimes an unfortunate side effect. Try to minimize it.

<hr/>

Here is a toy example:

```text
In their research, the authors claim that all programmers
are lazy and selfish creatures, grounding their conclusion
on a survey of 150 respondents.

Pros:
  - An important topic was addressed.
  - The reasoning is clear and concise.
  - The conclusion is very true.

Major cons:
  - Similar research done by Dean [1] is overlooked.
  - It's obvious that they are lazy; why another research?
  - Only Java programmers were interviewed.

Minor cons:
  - Typos and broken English here and there.
  - The font in most figures is too small.

Even though the subject of the research is important,
I believe this paper is not yet ready for publication
and requires significant rework.

[1] Dean et al., Programmers Are Super Lazy, 2022
```