S2 Week 4 Workshop - Critical evaluation

7 February: Answers now available - scroll to end of page for link. Discuss the answers in this thread on Piazza.

The aim of the S2 Week 4 workshop and the associated preparation is to help you achieve the learning outcome:

  1. Critically evaluate data-driven methods and claims from case studies, in order to identify and discuss a) potential ethical issues and b) the extent to which stated conclusions are warranted given evidence provided.

Note also that the paper discussed in this workshop will be referred to in the exam!

If you can, please bring a laptop and HDMI dongle with you to the workshop, so that you can look at relevant parts of the paper together on the screens at your tables.

Before the Workshop

1A. Learn about reading academic papers

Reading academic papers is a crucial skill that you'll need to use later on in this course, including the exam, and are likely to use later on in your degree programme, particularly in your 4th year project. We suggest you spend about 20-30 minutes reading the following resources (only if you haven't already read them for the Week 9 workshop in Semester 1):

The key ideas are:

  • Read strategically - depending on your purpose, you don't always need to read all of a paper thoroughly
  • Read critically, challenging assumptions, methods and findings in the paper
  • Read in multiple passes, examining the paper in more detail at each pass
    • On a first or second pass, if you encounter a concept you don't understand, make a note of it, and keep reading - hopefully the rest of the paper will still make some sense
    • Before moving to the next pass, look up the concepts you've noted. Wikipedia is often a reliable source, but you may wish to find another source to check it - Wikipedia always provides references where you can check things.

1B. Read an academic paper

Do a first pass read of the relevant paper: Menni et al. (Real-time tracking of self-reported symptoms to predict potential COVID-19, . Nat Med 26, 1037–1040 (2020). https://doi.org/10.1038/s41591-020-0916-2). Specifically, do the following:

  • Read the title and abstract very carefully
  • Read the first 3 paragraphs (up to "Between 24 March and 21 April 2020...") and the last 1 paragraph (starting with "Our work suggests that...") of the Main section carefully - these serve as the introduction and conclusion of the paper respectively
  • Read the rest of the Main section, but loosely - if there's something you don't understand, make a note of it, but then move on
  • Look at the Methods and Data Availability sections, but only enough to see what they're talking about, and don't go into more details

Now try to answer the questions - if you can't answer a question fully, don't worry; there will be time to refine them later.

  1. Question: What question does the paper address?
  2. Category: What type of paper is this? A paper collecting survey data, a randomised controlled trial, an analysis of an existing dataset?
  3. Context: What dataset(s) does it use? How were the datasets collected?
  4. Which methods were used to analyse the problem?
  5. Correctness: Do the assumptions appear to be valid?
  6. Contributions: What are the paper’s main contributions?
  7. Clarity: Is the paper well written?
  8. Your own understanding: From what you've looked at so far, is there anything you still don't fully understand? It's very unlikely that the answer to this will be "No" - even experienced researchers don't understand every part of every paper they read immediately. Furthermore, there is at least one method - ROC - that hasn't been covered in lectures yet, so it's likely you won't know exactly what it means at this point.

During the Workshop

2A. Discuss your initial thoughts with your group

As a group, consider the questions outlined in part 1B. Is everyone on the same page? Are there any disagreements?

If you're not sure about any of the above questions, or there is something you don't fully understand, make sure to bring it up with the group. If after discussion there are still concerns, then bring it up with one of the tutors in the workshop.

These questions are crucial for properly understanding any paper, so now would be the time to make sure you understand them properly.

2B. Discuss these more advanced (but still important!) questions with your group

As a group, consider the following questions. You may need to read specific parts of the paper that you haven't read fully before, if you've been following the workshop steps.

  1. What is the methodology of the study and what are the results?
  2. Identify the statistical methods used in the study and explain how they are applied to the data.
  3. Does the paper use confidence intervals? If so, where, and do they make sense? If not, where would be good to use them?
  4. Provide a critical discussion of the paper. Evaluate how strongly the data and analysis support the stated conclusions. Identify limitations of the study.
  5. Identify and explain the ethical issues investigated in the study, or connected with how the study was conducted. How well do the authors discuss these issues?
  6. Try to figure out what the ROC is (there is information about the ROC in Chapter 22 of the Lecture Notes) and what its corresponding results mean. In papers and articles, you may often come across terms or methods that you're not familiar with - trying to figure them out is an important skill.

Answers for these questions are now available.

License
All rights reserved The University of Edinburgh