> For the complete documentation index, see [llms.txt](https://ambitious-impact.gitbook.io/ambitious-impact-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ambitious-impact.gitbook.io/ambitious-impact-docs/ambitious-impact-eval/evidence-review.md). # Evidence Review Guidance on evidence reviews ## Practical guidance {% stepper %} {% step %} ### Figure out what to ask first For most interventions, we can break down the question into the following key parts to be evaluated separately: * **Can a charity effectively deliver this intervention?** \ For example, can a new charity successfully train health workers/TBAs on the benefits of KMC so that it is regularly delivered through healthcare facilities? * **Will the intervention have the expected impact?** \ E.g.,. Does KMC actually reduce neonatal mortality? How does this compare to conventional neonatal care? If you have reviewed a ToC, then you will have already identified a few assumptions that need validation. That is a good start. You may have identified other critical uncertainties in addition to the two main questions above. For example, if the intervention may cause harm, the risk should be fully understood and explored. Base these questions on what your uncertainties were during the ToC exercise. Use the ToC for the intervention and identify crucial considerations and/or assumptions that need more research, where we have gaps in knowledge along the ToC and research them. {% endstep %} {% step %} ### Gather evidence in a structured way Gather evidence as systematically as possible while trading this off with efficiency. We strongly recommend using a spreadsheet such as [this one](https://docs.google.com/spreadsheets/d/1ndDh2SgW_ZbUyvLuoP8zBJVqYcJaKqSK-H_Ex0ERJbI/edit?gid=1399670834#gid=1399670834), and downloading all potential references, including and excluding sources as per predetermined criteria. {% endstep %} {% step %} ### Decide what to evaluate in depth {% endstep %} {% step %} ### Evaluate the evidence {% endstep %} {% step %} ### Summarize {% endstep %} {% endstepper %} ## Theory Evidence reviews are an intrinsic part of virtually all research projects, whereby researchers seek to assess the relevant existing evidence pertaining to a research question. There is no catch-all answer to "good evidence," as this depends on the specific research question one is asking. However, there are conventions around the hierarchy of evidence for attributing cause-and-effect relationships. #### What is an evidence review? An evidence review is a catch-all term for a somewhat structured process that reviews existing evidence in relation to a research question. As practitioners of impact-oriented research, evidence reviews are an intrinsic part of virtually all research projects we undertake, as we need to know what existing research can tell us about the decisions we face. Most of the time, we will be interested in understanding what the evidence for a specific causal relationship is. For example, when AIM researcher Vicky Cox looked into banning low-welfare animal product imports, she needed to assess whether a non-profit’s work was likely to cause a change in import policy ([Cox, 2023](https://9475dbf4-555e-4808-9886-5f8ee815cc82.usrfiles.com/ugd/9475db_ed65351464a349b2afbc5791e26885f2.pdf)). When AIM researcher James Che looked into participatory learning and action groups for maternal support, the main question he faced was whether putting these groups in place was likely to cause declines in maternal and neonatal mortality ([Che, 2024](https://9475dbf4-555e-4808-9886-5f8ee815cc82.usrfiles.com/ugd/9475db_e4707b83ae454fb68a837f20e805f7db.pdf)). Evidence reviews of intervention effectiveness are closely tied to the ToC for the intervention itself, as you try to understand the logic and assumptions sustaining the intervention through the ToC and the evidence backing it. Other times, we will be looking at evidence to try to gain a descriptive understanding of a topic. This may involve gauging the evidence on the burden on a specific problem, the barriers leading to an issue, or the case studies for how a specific intervention works in practice. For instance, AIM researcher Morgan Fairless had to spend quite some time reviewing the evidence substantiating how big of a problem snakebites were, given the lack of consensus ([Fairless, 2023](https://9475dbf4-555e-4808-9886-5f8ee815cc82.usrfiles.com/ugd/9475db_ae0af1944af947359ec1534227051e4f.pdf)). Evidence reviews are usually structured in steps: 1. Set a research question 2. Search for published evidence about the research question 3. Assess the studies collected 4. Draw conclusions based on the studies as evidence ### What is “good evidence”? The nightmare question that keeps researchers up at night. The best answer to this question is “it depends” – which can be a terribly useless (albeit often used) answer. However, we want you to take home the message that the quality of evidence depends – to a large extent – on the questions you are asking (e.g., do you want to learn how people feel about a certain disease plaguing a community? Perhaps a participatory research exercise may give you the richest detail on this; do you want to make a causal inference about the quality of a vaccine? There’s no question about it, a randomized trial or collection of studies will be your go-to). Quality of evidence is also relative and often practical, while randomized-controlled trials may be the “gold standard” for causal inference, they are often impractical to run, or cannot be run because of ethical concerns – sometimes it is straight-up impossible to randomize the treatment, and in those cases, a quasi-experimental approximation may be the best possible evidence for a particular question. Please remember this point on context and research questions as we review this section. You will note we will delve mostly into the quality of evidence for causal attribution, which does have a relatively straightforward hierarchy of quality. #### **The hierarchy of evidence** The hierarchy of evidence is a framework used in evidence-based medicine and social science research to assess the quality and reliability of different types of evidence for causal attribution. It organizes various study designs based on their methodological rigor, potential for bias, and ability to provide reliable answers to causal attribution research questions. The hierarchy allows researchers and healthcare professionals to determine the strength of evidence supporting a particular intervention or treatment. The hierarchy typically consists of several levels, with higher levels representing stronger evidence. Note that the hierarchy of evidence is a rough heuristic applied specifically to causal attribution. Different studies are better suited for specific research questions. You probably would not weigh a qualitative examination of focus group data very highly if you are trying to determine how a vaccine affects rates of disease, but you would probably value its insight in determining what feelings people have in relation to getting their children vaccinated. Additionally, it is good practice to avoid any brute applications of the hierarchy in terms of quantity of studies. One excellent RCT may be better than four badly designed ones, a quasi-experimental study may be more informative than a meta-analysis of observational studies, and so on. While the specific levels may vary slightly depending on the source or field of study, a common hierarchy includes the following:
**Core materials** * Why is it so hard to know if you're helping? ([Hoel, 2024](https://www.theintrinsicperspective.com/p/why-is-it-so-hard-to-know-if-youre)) (\~17 minutes) * How many people die from snakebites? ([Dattani, 2023](https://www.scientificdiscovery.dev/p/14-how-many-people-die-from-snakebites)) (\~12 minutes) ### A step-by-step guide to evidence reviews Here we discuss the four key steps involved in an evidence review. Sometimes you only have a couple of minutes or hours to conduct a review. Fret not, nobody expects you to run a fully systematic review in five minutes. Note this is an ideal-type explanation, and that you will be able to adapt and adjust the process to different research capacities as you see fit. Evidence reviews follow a research question. Sometimes, these research questions are not explicitly stated, but there are good reasons to make research questions explicit: * Reasoning transparency (see [section 1](https://docs.google.com/document/d/1WD0eb1QmjqgoG7IksWRZ1Ed3KmQppSDK-MHF3xjVk_0/edit?tab=t.0#heading=h.yg4eo5e6mwwf)) * Helps focus the evidence-gathering process * Helps focus analysis onto responding to a concrete question, instead of exploring a topic Good research questions are clear, focused and concise. They are also researchable (i.e., they can be investigated and answered) ([George Mason University, n.d.](https://writingcenter.gmu.edu/writing-resources/research-based-writing/how-to-write-a-research-question)). A link to previous week's work: A good way of defining research questions is to look at your ToC and the assumptions/uncertainties in it. Each uncertainty can typically be linked to (at least) one research question. The next step is to gather the relevant studies pertinent to your question. This step can be done to different standards depending on how much time is available to the researcher. Adjusting the research process you use to the amount of time and capacity you have available is a key skill in applied research. The ideal-type evidence-gathering process involves something close to a systematic literature review, where a search protocol is developed and executed aiming to find all pertinent literature on a subject, and then combed through to find applicable studies. Executing a search like this is really transparent (because you can communicate your search protocols), and minimizes risks of missing important studies. Most of the time, you only have limited time and need to move on to analysis quickly. In those cases searching for specific types of studies that summarize information on a question first (such as systematic reviews) is helpful, and your search for relevant literature may be less structured. The steps for gathering literature on a question can be summarized as follows: 1. Think of search terms pertinent to the research question 2. Identify where you will search for published literature ([Elicit](https://elicit.com/) can help in this process for searching) 3. We also like to check grey literature published by a few organizations that do similar research to ours, such as GiveWell, Rethink Priorities, Open Philanthropy, Animal Charity Evaluators, Happier Lives Institute, and Founders Pledge. 4. Use a spreadsheet or literature review assistance software (; ; [CADIMA](https://www.cadima.info/index.php/area/evidenceSynthesisDatabase)) to collect all relevant data from the studies (perhaps Author, Year, DOI/URL, Abstract, Method) 5. Use some form of decision-making process to decide which papers you will review (such as prioritizing the top methodology to answer a question, cutting off resources from a certain year and below, etc.) Once you have identified which studies you will review and in what relative order, you can proceed with the analysis. We suggest creating separate sheets in your spreadsheet for different types of evidence. The specific columns you use will vary slightly depending on the review you are conducting, but some usual criteria to collect information are: * Title of paper, authors, and year of publication. * Method * Details on the intervention design * Study Period * Location of study * Context notes * Outcome variable * Control or Comparison Group * Study size * Description of the population * How is the Effect size measured? * Follow up period between end of intervention and main outcome * Effect size of treatment/change * Baseline * P-value * Confidence interval * Statistical power * Was the study pre-registered? * Do you perceive a risk of researcher or funder bias? * Comments on External validity to RQ/Context We discuss how to evaluate different study designs in [section 3.4.2](https://docs.google.com/document/d/1WD0eb1QmjqgoG7IksWRZ1Ed3KmQppSDK-MHF3xjVk_0/edit?tab=t.0#heading=h.eectb1up1ztq). Once you have collected and analyzed your evidence, you can draw conclusions and write up of your process and answers. After you have collected all the information in your spreadsheet you can then summarise your findings in writing in your report and put an overall credence score on your conclusion and the evidence base for the intervention you are looking at. Of course, making conclusions on the evidence base of an intervention is somewhat more complicated than counting the number of RCTs that exist for an intervention. A more nuanced approach to interpreting and comparing evidence bases of different interventions is discussed in the core material. **Core materials** * Comparing two bodies of evidence ([Falk and Hausen, 2023](https://drive.google.com/file/d/1SvJE2o5eszU28Cc2tuyChV6stvZlt0xA/view)) (video, \~13 minutes) * Does X cause Y? An in-depth evidence review ([Karnofsky, 2021](https://www.cold-takes.com/does-x-cause-y-an-in-depth-evidence-review/)) * Database Search Tips: Overview ([MIT, n.d.](https://libguides.mit.edu/c.php?g=175963\&p=1160724)) **Further materials** * Guidance on Conducting a Systematic Literature Review ([Xiao and Watson, 2017](https://journals.sagepub.com/doi/full/10.1177/0739456X17723971) - [ALT LINK](https://drive.google.com/file/d/1sWYQLPFBa9XTA2S8lqHCPxNwm208rBm3/view?usp=sharing)) (\~60 minutes) * Cochrane Handbook for Systematic Reviews of Interventions ([Higgins and Thomas, 2023](https://training.cochrane.org/handbook/current)) (click through handbook, very long but you can pick and choose) ### Interpreting and evaluating studies #### **Methodologies** The table below details the different study methodologies you are likely to encounter. | Method | Description | Strengths | Limitations | Core material | | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Meta- analysis |

“Meta-analysis is a research process used to systematically synthesize or merge the findings of single, independent studies, using statistical methods to calculate an overall or ‘absolute’ effect.2 Meta-analysis does not simply pool data from smaller studies to achieve a larger sample size. Analysts use well-recognized, systematic methods to account for differences in sample size, variability (heterogeneity) in study approach and findings (treatment effects) and test how sensitive their results are to their own systematic review protocol (study selection and statistical analysis).” (Shorten and Shorten, 2012, abstract)


A meta-analysis is necessarily quantitative.

|

An advantage of meta-analysis is that it increases statistical power by combining data from many studies, providing a more precise estimate of the effect and increasing the likelihood that an effect will be observed if it exists

Another advantage is that it will give you a more accurate overall view of a topic, because whilst the chance that a given individual study is flawed and misleading is significant, it’s far less likely that all of the studies in a metastudy are flawed.




|

The methodologies used to create meta-analyses are subject to ongoing discussion and iteration in econometrics, the fact that something is a meta-analysis does not necessarily mean it's a well-done one.


Meta-analyses and systematic reviews are only as strong as the studies that feed them. If the studies themselves are not trustworthy, statistical techniques of aggregation can limit but not eliminate bias.


Heterogeneity: Systematic reviews often include studies with varying methodologies, populations, and outcomes, resulting in heterogeneity. This heterogeneity can make it difficult to pool the results of studies accurately and draw meaningful conclusions. For instance, a behaviour-change intervention can work very well in one population but not at all in another – and it may not make sense to try to get an average.


Publication bias: If the studies included in the review come from a literature riddled with publication bias – that is, studies with large effect sizes being published while those with small or null effect sizes remain unpublished – the results of a meta-analysis will also be biased.

| Systematic Reviews and Meta-Analyses - How to Interpret the Results ([Bishop, 2015](https://www.youtube.com/watch?v=yRzAvV0guT4)) (\~8 minutes) | | Systematic reviews |

“Systematic reviews re-examine the evidence on a clearly formulated question using systematic and explicit methods to identify, select, and critically appraise relevant primary research and extract, collate, and analyze data from included studies.” (Armstrong et al., 2008, abstract)


A systematic review is necessarily comprehensive, but may not be quantitative.

| | | | | Randomized controlled trials (RCTs) | RCTs involve an experimental study design that randomly assigns participants to different groups or conditions and compares the outcomes between them. In cluster RCTs (cRCTs), randomization happens at the cluster level (say, 6 villages are treated and 6 are not) | RCTs are considered the gold standard for establishing causality in social science research. They are basically creating an experiment which allows them to take two statistically identical groups and the only factor they change is the intervention which then very clearly allows them to assess the impact for that particular factor rather than measuring all the other things that might be different in the two different groups. |

Unalterable characteristics: If you are studying the effects of a characteristic that cannot be altered (like the impact of age or ethnicity on a health outcome), an RCT is not possible since you can't randomly assign these characteristics to people.


Behavioural or lifestyle interventions: If the study involves an intervention that requires a significant behaviour change from participants (like a new diet or exercise regime), it may be difficult to ensure compliance. While we can design a study where participants are forced to perform or not to perform some actions, the results of such studies might not be valid, as the conditions of the study are highly artificial.


Complexity and cost: RCTs can be expensive and time-consuming to conduct, and require a lot of resources. They also require expertise in study design, data collection, and statistical analysis.


Ethical considerations: In some cases, it's unethical to withhold a treatment from a control group if that treatment is known to be effective, or to expose participants to a potentially harmful intervention


|

Randomized Control Trials and Confounding (Global Health with Greg Martin, 2013) (video, \~4 minutes)


How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) (Marginal Revolution University, 2020) (video, \~13 minutes)



| | Quasi- experimental studies |

“Quasi-experimental designs identify a comparison group that is as similar as possible to the treatment group in terms of baseline (pre-intervention) characteristics. The comparison group captures what would have been the outcomes if the program/policy had not been implemented (i.e., the counterfactual)”


Different quasi-experimental designs use distinct approaches to identifying the two groups and counterfactuals, such as regression discontinuity designs (RDD), the difference in difference (DD), and propensity score matching.

|

Quasi-experimental studies can provide valuable evidence when randomization is not feasible or ethical, they are essentially a “next best”.



| However, they are considered slightly weaker in terms of internal validity compared to RCTs. Quasi-experimental methods basically argue that a certain group mimics the counterfactual without creating that group through an experimental design. Usually, this is done by meeting certain assumptions (e.g., that treatment was as good as random in RDDs). These assumptions are usually debatable. |

Quasi-experimental designs (UvA Research Methods And Statistics, 2016) (video, \~7 minutes)


Regression Discontinuity Design for policy evaluation (EU Science Hub - Joint Research Centre (JRC), 2017) (video, \~2 minutes)


Difference-in-Differences method for policy evaluation (EU Science Hub - Joint Research Centre (JRC), 2017) (video, \~ 2 minutes)

| |

Observational studies (longitudinal, case/control, and pre-post tests)


| Observational studies do not randomize or attempt something close to randomization; instead, they use different data gathering and statistical techniques to find treatment effects associated with different variables. |

Observational studies, when well done, can provide clues about the associations and potential causal relationships between variables. When other higher-quality evidence is not available, observational studies can provide guidance as to plausible causal relationships for a researcher to further explore by triangulating evidence from other sources.


Observational studies of high quality can also provide insight into real-world behavior and descriptive statistics on variables of interest.

| Observational studies can only go so far in establishing solid correlational relationships, they cannot provide causal attribution. |

Correlational designs (UvA Research Methods And Statistics, 2016) (video, \~4 minutes)


| | Qualitative studies | Qualitative studies are observational studies that are conducted using qualitative methodologies, such as text analysis, focus group discussion, interviews, etc. | Qualitative studies, when well delivered, can provide a lot of insight onto lived experiences other than our own, and deep analysis of how theory applies to real-world events. | Qualitative studies cannot provide us with any reliable causal attribution. | Fundamentals of Qualitative Research Methods: What is Qualitative Research ([Yale University, 2015](https://www.youtube.com/watch?v=wbdN_sLWl88\&list=PLqHnHG5X2PXCsCMyN3_EzugAF7GKN2poQ)) (video, \~14 minutes) | | Expert views and consensus |

Module on experts next week!


| | | | ### **Primer on elements of analysis** | Element | Meta-analyses and Systematic Reviews | RCTs | Quasi- experiments | Observational designs (Quant) | Observational designs (Qual) | | ------------------------- | ------------------------------------ | ---- | ------------------ | ----------------------------- | ---------------------------- | | Outcome variables | ✅ | ✅ | ✅ | ✅ | ❌ | | Hypothesis testing | ✅ | ✅ | ✅ | ✅ | ❌ | | Effect sizes | ✅ | ✅ | ✅ | ✅ | ❌ | | Measures of uncertainty | ✅ | ✅ | ✅ | ✅ | ❌ | | Publication bias | ✅ | ❌ | ❌ | ❌ | ❌ | | Preregistration | ✅ | ✅ | ✅ | ✅ | ❌ | | Researcher & funding bias | ✅ | ✅ | ✅ | ✅ | ✅ | | External validity | ✅ | ✅ | ✅ | ✅ | ✅ | | Internal validity | ✅ | ✅ | ✅ | ✅ | ✅ |
This section discusses some key elements of analysis applicable to several of the methodologies we discuss above. Outcome variables: Research studies tend to measure various kinds of quantitative variables and try to assess how much one variable affects another one, for example, how much smoking cigarettes affects life expectancy. The magnitude of how much one variable affects another is called the “effect size.” To understand scientific studies, it’s critical to understand what types of outcome variables there are and how effect sizes are measured. There are two main types of outcome variables: * Binary outcome: Every observation can only take one of two values: 0 or 1 (“no” or “yes”). These can measure whether or not a participant attended a training program, whether they caught a disease, or whether they died during a study. * Linear outcomes: Each observation can take a whole range of values. These include outcomes like income, age of death, or blood concentration of a nutrient. Pointers * When evaluating a study, prioritize identifying the variables it is assessing, and what are its outcome variables. * When you have identified a variable, consider whether it’s a linear or binary one. Hypothesis testing: Most studies you encounter will be testing some research question against one or more hypotheses. In the context of quantitative studies, hypothesis testing is a statistical method used to assess whether a claim or hypothesis about a population is supported by sample data. It involves formulating a null hypothesis (H0) representing the absence of an effect, an alternative hypothesis (Ha) suggesting the presence of an effect, and choosing a significance level. Data is collected and analyzed using statistical tests to calculate a test statistic, which is then compared to a critical region determined by the significance level. The conclusion is drawn based on whether the test statistic falls within the critical region, leading to either rejection or acceptance of the null hypothesis. Hypothesis testing helps researchers make informed decisions and inferences about populations based on sample data, but it is important to consider its assumptions and limitations in interpreting the results. For further information, watch [this video](https://www.youtube.com/watch?v=0oc49DyA3hU) (\~15 mins). Pointers * Most quantitative studies involve hypothesis testing. * The null hypothesis is your starting point for understanding what the study aims to test. * Beware of language that suggests certainty; we can never confirm a null hypothesis - only fail to reject it. * Failure to reject the null doesn't mean it's true, just that we don't have strong evidence against it. Effect sizes: The results of a study are usually expressed in terms of the effect size of one variable over an outcome variable. Effect sizes are described using different summary statistics, depending on the variable type and study. * Binary variable's effect sizes are usually described in terms of Risk Ratios (also called Relative Risks, RR) or Odds Ratios (OR) RRs describe the likelihood of a certain event happening in one group compared to the likelihood of the same event happening in another group (e.g., eating red meat has a RR for colorectal cancer of 1.20, i.e. a 20% increase in the probability). ORs describe the likelihood of a certain event happening in one group compared to the likelihood of the same event happening in another group, expressed in terms of betting odds, i.e. OR = 3.0 means a 3x higher odds ratio (e.g. a 9:1 odds instead of 3:1 odds) To understand the real-life effect of these effect sizes, you need to know the baseline likelihood of the event in the untreated population. [Here’s](https://docs.google.com/document/d/1CoQkVTB6rtR1caD8nQ-DtLiG51QPTa_dK79GNUi9ojw/edit#bookmark=id.zdiuqzbxjmrc) some guidance on how to do this. These videos describe [RRs](https://www.youtube.com/watch?v=ELNt0XZYsAQ) (\~4 minutes) and [ORs](https://www.youtube.com/watch?v=ixKhS0Silb4) (\~7 minutes) in more detail. * Linear variables’ effects are usually described in terms of the variable itself (e.g. IQ point difference, the difference in test scores) or Standardized Mean Difference (Cohen’s D) Cohen’s D is an effect size metric that measures the standardized difference between two means, expressed in the number of standard deviations. An absolute change (e.g., 8cm) is often more useful than a Cohen’s D (e.g., 0.14), but sometimes you only get given the latter. To get from Cohen’s D to an absolute increase in a linear value, you need to know the baseline standard deviation. This is just like how with binary outcomes, to get from odds ratio or relative risk to an absolute increase in probability, you need to know the baseline probability. A common interpretation of Cohen’s D is that small effect sizes are (d <= 0.2), medium (0.2 <= d <= 0.5), and large (d >= 0.5). This [video](https://www.youtube.com/watch?v=tTgouKMz-eI) (\~5 minutes) goes into more detail. Pointers * Binary outcome effect sizes will usually be given as a risk ratio (RR) or an odds ratio (OR) * Linear outcome effect sizes will usually be given as the size of the change in terms of the variable itself, or Cohen’s D. * For RRs, ORs, and Cohen’s D, you will need to know more about the baselines at hand to understand the real-life effect size on the variable of interest. * A common interpretation of Cohen’s D is that small effect sizes are (d <= 0.2), medium (0.2 <= d <= 0.5), and large (d >= 0.5). Uncertainty: As you will know by now, everything in life is uncertain. If a study found that a new drug changed cancer mortality by -50% but its confidence interval ranges from -80% to +20%, then it’s entirely possible that the drug didn’t work at all – and there’s even a possibility it increased mortality! However, if a study estimates an effect as -50%, ranging from -60% to -40%, then you can be quite confident that the drug works. The three most important (and interrelated) elements to understand are confidence intervals, p-values, and statistical power). * Confidence intervals are a way of expressing quantitative estimates, using a range of values and a probability that the actual value lies within that range. For example, if the 95% confidence interval for the height of a random Australian woman is 155-177cm, then there’s a 95% probability that a random Australian woman’s height is in that range, based on the statistical evidence. The wider the confidence interval for a given confidence level, the more uncertain the estimate. More on confidence intervals in this [video](https://www.youtube.com/watch?v=ENnlSlvQHO0) (\~5 minutes). * P-values measure “the probability of obtaining the observed results, assuming that the null hypothesis is true. The lower the p-value, the greater the statistical significance of the observed difference. A p-value of 0.05 or lower is generally considered statistically significant.” ([Beers, 2023, key takeaways](https://www.investopedia.com/terms/p/p-value.asp#:~:text=A%20p%2Dvalue%20measures%20the,is%20generally%20considered%20statistically%20significant.)). This [video](https://www.youtube.com/watch?v=vemZtEM63GY) (\~11 minutes) goes into more detail. * Statistical power is the probability that a statistical test will correctly reject the null hypothesis when the alternative hypothesis is true. It's essentially the ability of a study to detect an effect when there is one. It is defined in terms of a specific effect size, e.g. for a decrease in mortality of 20%, statistical power of 90% means there’s a 90% chance that you’ll find an effect of that size or higher. If you’re looking for a decrease in mortality as low as 0.1%, that same study might have a statistical power of 30%, meaning that it’s unlikely to identify the effect even if it does exist. This [video](https://www.youtube.com/watch?v=Rsc5znwR5FA) (\~8 minutes) goes into more detail. Pointers * Study results carry uncertainty. * Confidence intervals are a way of expressing quantitative estimates, using a range of values and a probability that the actual value lies within that range. * P-values tell us the probability that the null hypothesis is true. * Studies are designed to have a certain statistical power to correctly reject the null hypothesis. A usual standard is to design studies to have 80% power. Publication bias: Publication bias refers to the tendency of researchers, journals, and other decision-makers to publish studies based on the direction or statistical significance of their results. It occurs when studies with positive or significant findings are more likely to be published, while studies with negative or non-significant results are less likely to be published or given less prominence. Publication bias is a huge problem for evidence-based policymaking because it distorts the overall body of evidence available on a particular topic. It creates an overrepresentation of positive or significant results in the published literature, while suppressing or underreporting studies with non-significant or negative findings. This means that even results of a meta-analysis can be very biassed, unless the analysis makes an active effort to correct for publication bias. A good rule of thumb is: If it looks too good to be true, it probably is! Preregistration: One of the best ways of preventing publication bias is preregistration. Preregistration refers to the process of publicly registering a research study's hypotheses, methods, and analysis plans before conducting the study or analysing the data. It involves submitting a detailed protocol or plan to a recognized registry or platform specifically designed for pre registration purposes. This means that (1) everyone can see that a study is being conducted and expect to see its results once it's finished, (2) the way the study will be analysed and reported is decided ahead of time and not once the researcher sees the data. Some journals are more likely to publish non-significant results for studies that have been preregistered (and some journals nowadays even require preregistration!). If a study follows a preregistered plan, it should say so somewhere in the paper. Unfortunately, preregistration has only become more common in the past 5-10 years so the majority of published research is not preregistered. Researcher & funding bias: The last kind of bias can come from the researcher themselves and/or the body that is funding the research. For instance, if a researcher has built their career around studying a certain psychological phenomenon, they may have a strong incentive to publish research that supports the existence of the phenomenon. Or a pharmaceutical company, which wants to obtain a patent for a new drug, has an incentive to publish research demonstrating that the drug works. It is therefore advisable to look at who the funders of a study are or if the study has a stated conflict of interest (though note that, unfortunately, many studies don’t state conflicts of interest even if they have them). If the topic of the study is controversial, it may also be good to look up the main author and see if they have a particular stance on the issue or a vested interest in supporting one side of the argument. External validity: When we do applied research, we are often trying to figure out whether the evidence we are reviewing is relevant to the intervention or program we are assessing. This gets to the difficult question of generalizability. Bettle ([2023](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)) discusses some key points on generalizability (with specific regard to RCTs): “I suggest estimating the degree to which an RCT result will generalise by examining the mechanism by which the intervention appears to work; do the necessary local conditions for this intervention hold in the new context? Some interventions will require more local conditions to hold than others, and therefore may have very heterogeneous effects across different areas. For example, vitamin supplementations work in a similar fashion across contexts; regions where large numbers of people lack that nutrient are therefore likely to benefit. On the other hand, behavioral or educational interventions may require lots of conditions to hold. For example, the presence of motivated teachers to carry out the intervention, presence of misconception about a particular health-related behavior, motivation to learn the new information, etc. Duflo & Banerjee (2017) outline four key ways that RCTs may fail to generalise, which are outlined below as a method to estimate external validity. These are specific sample differences, hawthorne effects, equilibrium effects and special care effects. Sample differences refer to differences between the RCT population and the population in the new context that will affect the success of the intervention; e.g. if a population that a radio intervention plans to expand to does not have high rates of radio listenership, the intervention will likely be less effective. Hawthorne effects refer to the way in which being observed during an RCT may alter people’s patterns of behavior; e.g. obeying hand-washing instructions more often if a person is aware that they are being observed. General equilibrium effects refer to emergent effects that appear when a program is being operated at a larger scale; e.g. a cash transfer program having effects upon the local economy once enough cash transfers have been received. Finally, special care effects refer to when an intervention is implemented differently at scale relative to how it is during an RCT; e.g. a teaching intervention that is carried out less rigorously when it is scaled out to an entire teaching district.” (p.21) Pointers * External validity refers, broadly speaking, to the level of trust we can have that the effects found in a study will replicate in other contexts (i.e., how generalizable they are). * To assess generalizability, one must think about the mechanics and context of the study, and how these differ from real-world implementation in other contexts. * External validity can also be assessed by considering “specific sample differences, hawthorne effects, equilibrium effects and special care effects” ([Bettle, 2023, p.21](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)). Internal validity: Internal validity refers to “the extent to which a given study supports the claims that it is making. If an exact replication of the study was carried out, would we see the same effect? Studies that are poorly designed, for instance with a low sample size, are likely to have lower internal reliability.” ([Bettle, 2023, p.9](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)). Many of the elements we have discussed above, and in particular uncertainty markers, feed our assessment of internal validity. There are two types of error that relate to internal validity. * Type M errors are errors in the magnitude of the effect found (under or over estimation). In the context of most scientific studies, the likeliest scenario is that of overestimation (the real effect size is lower than found) ([Bettle, 2023](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)). Bettle ([2023](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)) provides more details: “Type M error is strongly affected by the power of the study, and the presence of publication bias. As a rule of thumb, the lower the power of the study, the more inflated the effect size is likely to be if the threshold of statistical significance (<0.05) was reached. This is due to an effect called the ‘winner’s curse’ (see Fig 9), and is liable to occur whenever there is a benchmark statistical significance level to be reached. For example, imagine that an effect really exists with an effect size of 0.2. Our study only has the power to detect this effect size 20% of the time. The results of any study are subject to sampling variation and random error, meaning that the study could hypothetically find an effect size that is either smaller or larger; say 0.1 or 0.3. However, due to low power, the study does not successfully detect the 0.1 or 0.2 effect sizes; only the third case (0.3) reaches statistical significance. The ‘winner’s curse’ means that the filter of 0.05 significance level means that the under-powered studies will only tend to find statistically significant results when the ‘lucky experimenters’ happened to obtain an effect size that is inflated due to random error (Button et al., 2013). Because statistically significant results are more likely to be published, this means that there is an overrepresentation in the literature of larger effect sizes at p values < 0.05. We should be most worried about winner’s curse for studies that are low-powered, for studies where it is plausible that other similar studies have been undertaken and the results not published, and for studies that provide the first evidence for an effect; bear in mind that studies which provide the first evidence for an effect are often highly-cited.” (p.10) * Type S errors are errors in the estimation of the direction of the effect. “These errors are plausible when power is low (below around 0.2); see Fig 3. When evaluating a study, red flags for the possibility of a Type S error are (1) the intervention is not well studied (i.e. this is the first published RCT on the topic), such that there is considerable uncertainty around the effect size, (2) the RCT data is noisy, with a high standard error, (3) the RCT sample size is small, relative to the expected effect size, (4) a negative direction of effect is plausible for the metric of interest. For example, it would make sense to check for the possibility of a Type S error for an under-powered RCT examining behavioral change in response to a new intervention. It would not make sense to check for a Type S error if there are multiple high-powered RCTs on the intervention.” ([Bettle, 2023, p.16](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)) * There are other types of validity concerns to care about, and you may want to go deeper into this literature at your own pace. For instance, construct validity concerns relate to how the questions asked in data collection exercises correspond to the outcomes of interest ([Wikipedia Contributors, n.d.](https://en.wikipedia.org/wiki/Construct_validity)). Sometimes surveys are poorly designed, and do not actually capture what we care about. Pointers * Internal validity refers, broadly speaking, to the level of trust we can have on the effects found in a given study. * Type M error refers to under or over estimation. * Type S error refers to the errors in the direction of the effect. * These two concepts are intricately related to statistical power and other measures of uncertainty. Finally, here are some guiding questions we use when evaluating an evidence review. | Has the author arrived at reasonable conclusions regarding the research questions they are investigating? (see reasoning transparency for how those conclusions are presented) | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

Has the author appropriately identified the key research questions pertinent to the stated aims of the report?

| |

Has the author (in the summary of evidence and/or in the write-up) correctly interpreted the results of studies?

| |

Has the author missed key studies?

| |

Has the author investigated additional types of evidence?

| |

Has the author actively looked for evidence against it?

| **Core materials** * Interpreting Scientific Studies ([AIM, 2024](https://docs.google.com/document/d/1CoQkVTB6rtR1caD8nQ-DtLiG51QPTa_dK79GNUi9ojw/edit?usp=sharing)) (\~27 minutes) * Replicability & Generalisability: A Guide to CEA discounts ([Bettle, 2023](https://docs.google.com/document/d/1eJBSmNG-iRJ-twoHaoztQUmB4pmEHut8_oT3xgtldnI/edit?usp=sharing)) (\~1 hour) **Further material** * Overview: Strategies for Causal Attribution ([Rogers, 2014](https://www.unicef-irc.org/publications/pdf/brief_6_overview_strategies_causal_attribution_eng.pdf)) [link](https://www.betterevaluation.org/sites/default/files/2022-03/Overview_Strategies_for_Causal_Attribution_ENG.pdf) that works * Randomized Controlled Trials (RCTs) ([White et al., 2014](https://www.unicef-irc.org/publications/pdf/brief_7_randomized_controlled_trials_eng.pdf)) * Quasi-Experimental Design and Methods ([White & Sabarawal, 2014](https://www.unicef-irc.org/publications/pdf/brief_8_quasi-experimental%20design_eng.pdf)) * Comparative Case Studies ([Goodrick, 2014](https://www.unicef-irc.org/publications/pdf/brief_9_comparativecasestudies_eng.pdf)) * Eva Vivalt’s research suggests social science findings don't generalize. So evidence-based development–what is it good for? ([80,000 Hours, 2018](https://80000hours.org/podcast/episodes/eva-vivalt-social-science-generalizability/#related-episodes)) (podcast, \~2 hours) * Rachel Glennerster on a year’s worth of education for under a dollar and other ‘best buys’ in development ([80,000 Hours, 2018](https://80000hours.org/podcast/episodes/rachel-glennerster-best-buys-in-international-development/)) (podcast, 1.5 hours) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://ambitious-impact.gitbook.io/ambitious-impact-docs/ambitious-impact-eval/evidence-review.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.