GRADE Evidence Profiles

Purpose

An evidence profile transparently records a human panel's judgments about a body of evidence for each important outcome. It is not an article-scoring shortcut and does not produce a patient-care recommendation.

Use the current GRADE Book and the GRADE Working Group as the controlling methodology. The GRADE Book is replacing the older handbook with progressively updated content.

Non-Automation Rule

Never:

• infer certainty from keywords, abstracts, p-values, journal name, or study design alone;
• count checklist items to calculate certainty;
• treat one study's risk-of-bias judgment as certainty in a body of evidence;
• equate certainty with recommendation strength;
• assign recommendation strength without an Evidence-to-Decision process and a responsible panel;
• invent source citations or downgrade/upgrade rationales.

The bundled checker verifies structure, allowed labels, human attribution, rationale, and source linkage. It does not alter or endorse a judgment.

Unit of Assessment

Rate certainty separately for every critical or important outcome. Include desirable and undesirable effects. Different outcomes may have different:

• bodies of evidence;
• risk-of-bias concerns;
• directness;
• precision;
• reporting bias;
• certainty.

Do not collapse all outcomes into a single study-level grade.

Required Profile Fields

Question

• Population
• Intervention/exposure/index approach
• Comparator/reference
• Outcomes and time horizons
• Setting and decision context

Sources

For every source include:

• stable source ID;
• full citation;
• URL or DOI;
• publication type;
• version/date;
• access date when content is living.

Effect

For each outcome record:

• measure and direction;
• absolute and relative effects when appropriate;
• confidence or credible interval;
• participants and studies;
• follow-up/horizon;
• missingness;
• whether the estimate is adjusted;
• applicability limits.

Do not convert an effect into a clinical instruction.

Certainty Domains

Every domain entry requires a judgment, rationale, source IDs, and human reviewer role.

Risk of Bias

Use a design-appropriate tool. Describe how limitations could change the estimated effect. Do not use a numeric quality score as a substitute.

Inconsistency

Examine the direction and magnitude of effects, interval overlap, heterogeneity, and plausible explanations. A statistical heterogeneity value alone is not the judgment.

Indirectness

Compare population, intervention/exposure, comparator, outcome, time horizon, setting, and evidence pathway with the framed question.

Imprecision

Use decision-relevant thresholds and the range of effects compatible with the interval. Do not apply unsupported universal event-count rules.

Publication Bias

Consider missing studies/results, selective reporting, small-study effects, sponsorship patterns, registrations, protocols, and reporting availability.

Upgrading Considerations

When the selected GRADE approach permits, a panel may consider large effects, dose-response gradients, or plausible residual confounding. Each requires explicit methodology, rationale, and citations. “Statistically significant” is not an upgrading reason.

Final Certainty

Allowed labels:

• high;
• moderate;
• low;
• very low.

Record:

• the final human judgment;
• who made it and in what role;
• date;
• domain-to-final-rating rationale;
• dissent or unresolved issues;
• source IDs.

The label describes confidence in an estimate for an outcome in a defined context. It is not a recommendation and does not imply safety, effectiveness, or authorization.

Evidence to Decision

Recommendation development is outside the automated helper. A qualified panel using an applicable GRADE Evidence-to-Decision framework must explicitly consider, as relevant:

• priority of the problem;
• desirable and undesirable effects;
• certainty of evidence;
• values and variability;
• resources and cost effectiveness;
• equity;
• acceptability;
• feasibility.

Keep the evidence profile and any later recommendation record separate and traceable.

Quality-Control Checklist

• Search and selection methods are documented.
• Outcome definitions and horizons match the question.
• All important benefits and harms are represented.
• Effect estimates include uncertainty.
• Each domain has a human judgment and rationale.
• Every rationale links to source IDs.
• The final certainty is outcome-specific.
• Conflicts of interest and panel roles are recorded.
• Disagreements and updates are versioned.
• No patient-specific or treatment directive appears.

Helper

python3 scripts/evidence_profile_check.py assets/evidence_profile_template.json

The distributed template intentionally contains unresolved judgments. A non-zero result is expected until qualified humans complete it.