Literature search for related work

When conducting a literature review on a given topic, your goal is to find all related work on this subject. This can seem daunting — there is lots of research out there! Fortunately, there are some tricks to making this process more manageable.

Seed your queue with general searches

For many people, the first and last search tool they’ll need is Google Scholar. Take advantage of its “Advanced Search” features, such as the source: keyword for limiting your searches to specific venues.

However, there are other search tools out there, such as CiteSeerX.

You can also try dblp. Its search is limited to titles (and authors and venues), so it’s less useful than search engines that also search abstracts and texts, but because it’s limited to “major computer science journals and proceedings,” it can be good if you’re feeling overwhelmed with results from other searche.

Many CS papers are published by the ACM and IEEE, which have their own venue-specific search engines: ACM Digital Library and IEEE Xplore.

Narrowing down your searches

It’s normal for literature review to turn up dozens of relevant papers. But if you have way more than that (e.g., hundreds), here are some suggestions for finding the ones to focus on.

Prioritize specific venues

Good research can be published anywhere, but there are certain venues that are more likely to contain relevant work in usable security and privacy.

These are also good places to check if you have the opposite problem from having too many papers: your keyword-based searches aren’t turning up any relevant work.

Security

• IEEE Security & Privacy
• USENIX Security
• ACM CCS

Privacy

• PETS

Human-computer interaction

• ACM CHI
• ACM UbiComp (IMWUT)
• ACM CSCW

Usable security

• SOUPS @ USENIX
• USEC
• EuroUSEC

Consider citation counts

The number of citations are a flawed metric for evaluating research quality or significance, but they can be useful for finding some of the best-known papers you may need to cite. Highly-cited papers are also useful because you can see what other papers cite them (see below).

Finding related work by following the citation tree

1. Start by identifying a paper that you consider highly relevant
2. Read through its Related Work section and add the papers from there to your queue
   • Citation management software like Zotero is a good way to keep track of papers you find.
3. Enter the paper’s name into Google Scholar. Under the paper’s listing, click on Cited by X to get a list of all papers that cite this one. Add those to your queue.
4. Repeat steps 2 and 3 until your queue is empty
   • You don’t actually need to do this for every paper you find, just the ones you consider notable and relevant (see below).

Which papers are worth citing?

It’s often impossible to cite every paper in existence on a particular topic.

• My rule of thumb is that the more closely related a paper is, the more you need to cite it.
  • i.e., cite more of those papers — don’t skip them even if they seem similar to the others.
  • In the related work section, they merit a more in-depth description (e.g., what they did, what were the contributions). You’ll need to explain how your work is different.
• In contrast, “further away” papers can be catalogued less exhaustively.
• More important papers may be worth more attention
  • Importance can be gleaned from number of citations or publication venue, but importance is relative — it depends on a particular project and its research questions.

What should you write in the related work section?

You’ve collected lots of relevant papers and read them. Now you’re ready to write the Related Work section — but how should you do it?

Here is some guidance for how to write a Related Work section, originally appearing on the wiki for Barbara Engelhardt’s research group:

A related work section is essential to most research articles (and every methods paper). A fundamental contribution in methods papers is methodological novelty and editors will commonly ask how the method in questions differs from an already published method. A well-written related work section will enable reviewers to confidently answer this question.

Goals of a related work section

The overall goal is to describe the related research areas and to place your method’s contributions to the field in this context. By clearly describing previous work, you can better describe the current limitations and the need for new methodology. It also gives you an opportunity to demonstrate knowledge of the area and helps others relate your current work to other scientific areas. The section should include methods that formulated the problem, addressed a central or related problem, used a similar methodology as your work to a similar problem, or if your work was inspired by their work. Be extremely generous in including related work — the proverbial scorned author turned reviewer is sometimes a real problem.

Structure of a related work section

There should be a common theme to each paragraph in this section. For instance, you can describe related methods development work chronologically starting from early work and their assumptions as the models get more complex. Subsequent paragraphs can describe how newer methods improved on previous work, relaxed assumptions, used fundamentally different approaches (frequentist versus Bayesian), or tried to solve similar problems. When describing each method, say what it is and how it builds on previous approaches before describing its limitations. Better to teach here and tell a story than just to establish dominance in a (possibly) crowded area. Related work sections are often judged on their intellectual honesty (at least by me).

The final paragraph should then place your contributions squarely in the context of the preceding research. It should be simple and clear, given the background from this section: “The main contributions of our work to the problem of Blah are i) …” and then later “The impact of our approach that addresses these existing limitations is …”

As with the abstract, viewing other published articles in the target journal on related problems will give a good sense of what is expected.