INFT 4017 CIS Research Methods

The reviewing process

In this section we look at how papers are reviewed. This is important to us because:

You will have seen in the video shown by library staff in week 3 how it is important to work out whether something you read from online sources is reliable, and usually the indicator of reliabiltiy is whether a paper has been "peer reviewed". Saying a paper has been "peer reviewed" means that it has been closely looked at by experts in the field, with those experts chosen by other experts to give their opinion on the work in the paper. These reviewers are generally not paid for doing this work, and do it as a form of community service to the research community.

The peer review process is intended to give us an assessment of whether the work in a paper is reliable and frequently whether it is going to be useful. We look at the peer review process here.

How the quality of journals can be assessed

Firstly we will look at the various judgements on paper quality.

How the quality of papers is assessed

Now we will look at the reviewing process itself. This will give us useful insights into how the papers we include in our literature survey were assessed, and how any papers we write might be assessed.

Journals and conferences often have different reviewing processes, which are largely down to the presence of deadlines. In Computer Science areas, these deadlines are about the only real difference between journal and conference papers, perhaps the length of submitted papers is upper-bounded by conferences whereas in journals it generally is not.

However in other disciplines, there is a significant difference between journal and conference publications, and often the peer-reviewing process for conferences in other disciplines is not as selective as it usually is in Computer Science.

Conference editorial processes usually work to a publication deadline, so each paper tends to be reviewed only once, and either it is accepted for publication or else it is "sudden death" where a paper is rejected without chance of reworking it to be considered again. (Some conferences will offer a short paper instead, which are not as prestigious but still not too bad in some cases).

In contrast, journals often have papers go through one or more revisions before being accepted. This is because the journal is continuously being produced, and there is plenty of time for any paper to be revised to bring it up to the journal standards.

There are generally 4 or 5 journal outcomes: accept, accept with minor revisions, accept with major revisions, revise and resubmit, and reject.

In-class exercise

You should have brought one of the papers from your bibliography into the class today (as per the email on Tuesday). You will be applying the review process to your selected paper.

Depending on whether you have brought a journal paper or a conference paper, you will use either the ACM TWEB review form or the Easychair review form.

Once you have performed the review, we will look at the journal or conference rankings, and the citations for the paper.

Last update hla 2009-03-31