Part 1: From the original Cochrane risk of bias tool to RoB 2

Summary

This webinar discusses the development and application of the Risk of Bias Tool (RoB 2), a systematic tool used to assess bias in studies. The presenters provide insights into the historical context, evolution, and improvements of the tool, addressing challenges such as inconsistencies, unclear judgments, and missing data. The tool’s enhancements offer clearer guidance, better usability, and applicability to various trial types. With an emphasis on reducing bias and improving reliability, the RoB 2 aims to be more effective for systematic reviewers, incorporating recent methodological advances and addressing critical issues like unblinded trials and outcome measurements.

Takeaways

• 😀 Acknowledgment of contributors: Matt, Roy, Daniel, Isabel, Luke, and Vincent for their work on developing the Rob2 tool, with funding from the Medical Research Council in the UK.
• 😀 The purpose of the webinar is to introduce the Rob2 tool, its historical context, and the need for a new version of the previous risk of bias tool.
• 😀 Bias is defined as a systematic error or deviation from the truth in research, which leads to overestimating or underestimating the effect of an intervention.
• 😀 The distinction is made between internal validity (whether the study's results reflect the truth) and external validity (the generalizability or relevance of the study to other situations).
• 😀 The risk of bias is about the result of a study, not the study itself, and should be assessed only if a result is present.
• 😀 A key improvement in the Rob2 tool is that it addresses more comprehensive biases, including those in unblinded trials, and now offers versions for different trial types, including crossover and cluster randomized trials.
• 😀 A clearer structure and guidance within the Rob2 tool have been introduced to help users make better risk of bias judgments and improve reliability.
• 😀 The original risk of bias tool, developed in 2008, was critiqued for being used too simplistically, with low reliability and issues like inconsistent judgments and complexity in certain domains.
• 😀 Rob2 has been improved based on empirical evidence, modern developments in causal inference, and feedback from systematic reviewers and researchers, aiming for more consistent and useful assessments.
• 😀 The tool now includes an overall risk of bias judgment that feeds into primary analyses, with sensitivity analysis that can show how risk of bias influences the results.
• 😀 The webinar outlines the introduction of new resources for learning more about the Rob2 tool, as well as a focus on conflicts of interest in research that should inform risk of bias assessments.

Q & A

What is the main goal of the Risk of Bias 2 (ROB 2) tool?

The ROB 2 tool is developed to assess the risk of bias in randomized trials. Its primary goal is to evaluate how systematically accurate the results of a study are, ensuring that they reflect the true effect of an intervention.

Who were some of the key contributors to the development of the ROB 2 tool?

Key contributors to the development of the ROB 2 tool include Matt Roy, Daniel, Isabel, Luke, and Vincent, with the funding support from the Medical Research Council in the UK.

What is the main distinction between internal and external validity in the context of risk of bias?

Internal validity refers to whether the results of a study accurately reflect the true effect of an intervention, while external validity, or generalizability, refers to how relevant the study’s results are to other situations or populations.

What are some of the limitations of earlier risk of bias tools before the development of ROB 2?

Earlier tools were often used inconsistently, lacked detailed guidance, and didn't address specific trial designs like crossover or cluster-randomized trials. They also struggled with issues like low inter-rater reliability and inadequate treatment of certain biases.

How does ROB 2 improve upon its predecessors?

ROB 2 improves upon previous tools by offering more comprehensive and structured guidance for assessing biases, better inter-rater reliability, versions for different trial designs, and clearer judgment criteria. It also incorporates modern methodological developments in causal inference and addresses issues in unblinded trials.

What role does 'unclear' judgment play in the original risk of bias tool, and how has it been addressed in ROB 2?

The 'unclear' judgment in the original tool was a common issue, as it indicated ambiguity in bias assessment. In ROB 2, clearer guidance and structure are provided to reduce uncertainty and improve reliability in making bias judgments.

What are some of the specific domains covered by the ROB 2 tool?

ROB 2 covers five specific bias domains that are assessed to evaluate risk of bias, with further webinars and resources dedicated to detailing each of these domains.

What does the risk of bias assessment focus on, and why is the result more important than the study itself?

The ROB 2 tool focuses on assessing the risk of bias for a specific result or outcome from a study. The result is the key factor because it may be systematically overestimated or underestimated due to biases in the study design, data collection, or analysis.

Why is it important to differentiate between bias and random error in systematic reviews?

It is crucial to differentiate between bias (systematic error) and random error because bias leads to consistently incorrect results, whereas random error is reflected in statistical uncertainty and is accounted for through confidence intervals.

How does the ROB 2 tool incorporate modern developments in causal inference and epidemiology?

ROB 2 aligns with modern causal inference methodologies by improving its treatment of biases in various study designs, such as unblinded trials, and addressing new issues like missing data, which have become increasingly important in evidence synthesis.