A procedure for when two credible sources disagree

Most apparent conflicts dissolve here. Different populations, doses, durations or outcome definitions produce different answers to questions that share a summary sentence.

Establish they are answering the same question

Most apparent conflicts dissolve here. Different populations, doses, durations or outcome definitions produce different answers to questions that share a summary sentence. Check the question before weighing the answers.

Compare designs before conclusions

A randomised trial and an observational study disagreeing is the expected case rather than a puzzle, and the trial usually deserves more weight on causation. The observational work may still be better on long-term or rare outcomes.

Check dates in both directions

The newer source is not automatically right. A large well-run older study can outweigh a small recent one, and the question is what each added rather than when each appeared.

Look for the common ancestor

Sources often trace to the same original study described differently. Following both back frequently reveals one piece of evidence and two interpretations, which is a different situation from two independent findings.

Weight by what could be wrong

For each source, ask what would have to be true for it to be mistaken, and how likely that is. A source whose failure mode is implausible deserves more weight than one whose failure mode is routine.

Consider who would be wrong publicly

Sources that commit to checkable specifics expose themselves to correction. A source that stays general cannot be shown wrong and should carry less weight for exactly that reason.

Unresolved is a legitimate conclusion

Some questions are genuinely open, and reporting them as open is more accurate than choosing a side. A reader is better served by knowing the state of the evidence than by a false resolution.

Write down what would change your mind

Naming in advance the finding that would settle it prevents reading subsequent evidence selectively, and it makes the eventual answer easier to recognise.

Check whether the disagreement is about size or direction

Sources disagreeing about whether an effect exists are in genuine conflict. Sources agreeing on direction and differing on magnitude are usually compatible, and the apparent dispute is often an artefact of how each summarised its interval. Separating the two resolves a large share of conflicts before any weighing is needed.

Prefer the source that would be embarrassed

Where one source has a stake in the answer and still reports an inconvenient finding, that finding is unusually credible. Where a source reports only what suits it, agreement with it is weak evidence regardless of how carefully it is argued.

Trace both to their source

Conflicting sources frequently rest on the same original study, described differently. Following both back often reveals one piece of evidence and two interpretations, which is a different problem from two independent findings.

Unresolved is a legitimate answer

Some questions are genuinely open. Reporting them as open is more accurate, and more useful, than manufacturing a resolution the evidence does not support.

Recency is not automatically authority

A newer source may incorporate more evidence, or may simply be a less careful restatement of an older one. Checking whether the recent source adds data or only adds confidence is the distinction that matters, and it is usually visible in whether it cites anything the earlier one did not.

Weigh by what each source could know

A party with direct access to a record, and a reason to be accurate about it, outranks a commentator reasoning at a distance. Weighting by access rather than by tone resolves most conflicts faster than weighting by how certain each source sounds.

The code is HEALTHYLIFE10, worth 10% off the subtotal sitewide at growthguys.com. It carries no minimum and no exclusions, and every test of it is dated and published — the evidence index, and the full code page.

Related