coreSR.AIcoreSR
Usage Guide

PROBAST

How to assess risk of bias for prediction model studies

1

What PROBAST is and when to use it

  1. 1PROBAST (Prediction model study Risk Of Bias Assessment Tool) — Wolff et al., Ann Intern Med 2019, version 15/05/2019
  2. 2The right instrument for studies that develop, validate, or update a multivariable prediction model that produces individualised predictions (probabilities, classifications, or expected values)
  3. 3NOT for studies that use multivariable models only to identify predictors without producing individualised predictions — PROBAST explicitly excludes these
  4. 4NOT for randomised trials evaluating treatment effects (use RoB 2.0 / ROBUST-RCT) or non-randomised studies of interventions / exposures (use ROBINS-I / ROBINS-E)
  5. 5Unit of assessment: one PROBAST assessment per (publication × distinct prediction model × outcome of interest). One paper that develops two competing risk scores produces two PROBAST assessments.
Tip: If a paper updates an existing model AND validates the update, treat it as Development + Validation: the update IS development; the validation IS the validation.
2

The 4-step structure of every PROBAST assessment

  1. 1Step 1 — Specify the review question. Project-level. Four free-text fields: intended use, participants and setting, predictors, outcome. All assessments in the project inherit the default review question.
  2. 2Step 2 — Classify the study evaluation. Per-assessment. Pick exactly one of: Development only / Development and validation / Validation only. Determines which signaling questions are instantiated.
  3. 3Step 3 — Assess 4 domains × 20 signaling questions. Each question takes Y / PY / PN / N / NI. All questions are phrased so that "Y" indicates absence of bias (this invariant is what makes the rollup work).
  4. 4Step 4 — Reach an overall judgment. Per (assessment × evaluation context). Includes the Dev-only §7.1 special downgrade (described below).
3

The 4 domains and what they assess

  1. 1Domain 1 — Participants. Were the data sources and inclusions appropriate for predicting the outcome? Risk-of-bias rating + applicability rating.
  2. 2Domain 2 — Predictors. Are predictors defined, measured, and timed appropriately? Available at the intended-use timepoint? Measured without knowledge of outcome? Risk-of-bias + applicability.
  3. 3Domain 3 — Outcome. Was the outcome ascertained appropriately, with a standard definition, and not contaminated by predictors? Risk-of-bias + applicability.
  4. 4Domain 4 — Analysis. Sample size, predictor handling, missing data, model-building strategy, performance measures, internal validation, and coefficient correspondence. Risk-of-bias ONLY — Domain 4 has no applicability rating per PROBAST §6.4.13.
Tip: Most studies fail Domain 4 — sample size, internal validation, and continuous-predictor handling are commonly inadequate. PROBAST is famously strict here; do not soften the published rules.
4

The 20 signaling questions

  1. 1D1 (2 questions): 1.1 data sources, 1.2 inclusions/exclusions
  2. 2D2 (3): 2.1 predictor definitions, 2.2 blinding to outcome, 2.3 availability at intended-use timepoint
  3. 3D3 (6): 3.1 outcome ascertainment, 3.2 standard definition, 3.3 incorporation bias check, 3.4 same method for all participants, 3.5 blinding to predictors, 3.6 time interval
  4. 4D4 (9): 4.1 sample size (computed), 4.2 predictor handling, 4.3 inclusion in analysis, 4.4 missing data, 4.5 univariable selection, 4.6 complexities, 4.7 performance measures, 4.8 overfitting/optimism, 4.9 coefficient correspondence
  5. 5Dev-only questions (only asked when the assessment exercises model development): 4.5, 4.8, 4.9
  6. 6For Dev+Val assessments, every "both" question is asked TWICE — once in the Development context, once in the Validation context — because the same paper's Dev cohort and Val cohort can have different bias profiles
Tip: You can hover the (i) icon next to any signaling question in the assessment view to see a one-paragraph excerpt from the PROBAST E&E paper explaining what to look for.
5

Sample-size adequacy (Q4.1) is computed, not AI-judged

  1. 1PROBAST §6.4.3 gives numeric thresholds; CoreSR applies them deterministically rather than asking AI to read them
  2. 2EPV heuristic (default): Development binary outcomes — EPV ≥ 20 → Y, 10-20 → PY, 5-10 → PN, < 5 → N
  3. 3Validation: ≥ 200 events → Y, 100-200 → PY, < 100 → PN/N
  4. 4When candidate predictors are missing or ambiguous in the paper, the answer is NI — CoreSR will NEVER infer a denominator from prose
  5. 5Riley pmsampsize (advanced): toggle "Sample-size method" on the project header to switch from EPV to the pmsampsize R package. Falls back to EPV if R or the package is unavailable.
  6. 6The result is stored in computedJudgments JSONB with method="epv_v1" or "pmsampsize_v1" and the inputs used, so historical assessments stay reproducible even if you change the project default later
6

Phase A → Phase B → Phase C — how AI fits in

  1. 1Phase A (extraction): one structured AI extraction per domain. Each extraction has its own Zod schema for the descriptionFields slice. The 4 domains run in parallel — a malformed JSON in one domain does not poison the others; failed slices stay marked "failed" so the reviewer fills them manually.
  2. 2Phase B (scoring): per signaling question, structured AI scores against the decision rules. Each prompt receives ONLY the relevant descriptionFields slice plus the spec's decision-rules table — no global document context, which keeps the assessment focused.
  3. 3Phase C (rollup): pure deterministic code in server/probast/rollup.ts. No AI calls. Domain rating uses strict per-domain rules: any "N" → High; any "PN" → High in every domain (Domain 4 strictness is built in).
  4. 4Idempotency: every AI run is keyed on a hash of (classification, document text, decision-rules version). Re-running with no upstream change short-circuits and reuses prior results, no re-billing.
Tip: The reviewer's userAnswer always wins over aiAnswer. Re-running AI never overwrites a manual override.
7

Domain rating algorithm (deterministic)

  1. 1For each domain, the rollup looks at the applicable signaling answers (NA codes are skipped — that question doesn't apply)
  2. 2If ALL applicable answers are Y or PY → Low risk of bias
  3. 3If ANY answer is N → High
  4. 4If ANY answer is PN → High (D1-3 with rationale; D4 strictly)
  5. 5If only Y/PY/NI codes (no N/PN) and at least one NI → Unclear
  6. 6No applicable answers yet → Unclear
  7. 7The reviewer's userRating overrides the algorithm rating. Both are persisted; the dashboard shows the algorithm rating in italics when an override is in place.
8

Applicability ratings (Domains 1-3 only)

  1. 1Each of Domains 1, 2, 3 carries a separate applicability rating (Low / High / Unclear) alongside its risk-of-bias rating
  2. 2Domain 4 has NO applicability rating per PROBAST §6.4.13 — the schema enforces this; the UI hides the field for Domain 4
  3. 3Applicability is reviewer-judged in M1; Phase A extraction in M2/M3 will pre-populate algorithmApplicability based on the description fields
  4. 4Overall applicability is the worst of D1/D2/D3: any High → High overall; otherwise all Low → Low; otherwise Unclear
9

Step 4 — Overall judgment + the Dev-only §7.1 special downgrade

  1. 1Overall RoB: any domain High → High; all four Low → Low; otherwise Unclear
  2. 2CRITICAL EXCEPTION (PROBAST §7.1): for Development-only assessments where all four domains are Low, the rollup default-downgrades to High
  3. 3The verbatim §7.1 rule: "If a prediction model was developed without any external validation, and it was rated as low risk of bias for all domains, consider downgrading to high risk of bias. Such a model can only be considered as low risk of bias, if the development was based on a very large data set and included some form of internal validation."
  4. 4To undo the downgrade and restore Low overall, the reviewer must: (1) check "very large dataset" (2) check "internal validation present" (3) supply a non-empty rationale describing why
  5. 5The server REJECTS empty rationales with a 400 — the rule cannot be quietly bypassed
  6. 6Every override fires a probast_audit_events row with type "dev_only_downgrade_overridden" + before / after JSON + reviewer's rationale text
Tip: For Dev+Val assessments, the §7.1 rule does NOT trigger because external validation is part of the assessment. The combined-context overall row picks the worse of the two contexts on each axis.
10

Override workflow — every level is editable + audited

  1. 1Per signaling answer: dropdown override of any Y / PY / PN / N / NI code; clears with one click
  2. 2Per domain: override RoB rating (Low / High / Unclear) + applicability rating (D1-3) + free-text notes
  3. 3Per assessment: override the overall judgment + Dev-only downgrade fields
  4. 4Click "Recompute" after edits to refresh the algorithm-suggested ratings; userRating is preserved
  5. 5Every override writes a probast_audit_events row: assessment id, user id, event type, before-state JSON, after-state JSON, rationale (when supplied), timestamp
  6. 6The audit log is append-only — there is no UPDATE path. Use it to trace exactly who changed what and when, and to defend reviewer decisions in a methods appendix.
11

Hosmer-Lemeshow flag and PROBAST E&E quotes

  1. 1PROBAST E&E specifically warns that Hosmer-Lemeshow alone is insufficient evidence of calibration (underpowered with small samples, overpowered with large ones)
  2. 2CoreSR detects when a paper's analysis description mentions H-L without any other calibration measure (plot, intercept, slope) and surfaces a yellow warning beside Q4.7
  3. 3Each signaling question has a tooltip (info icon) with a paraphrased excerpt from the published E&E paper — hover to see what to look for
  4. 4These are reviewer reminders, NOT a substitute for reading the full E&E document
12

Bulk operations and project export

  1. 1"Run AI on all" — sequences AI runs across every assessment in the project that has files and a classification; skips others with a clear reason
  2. 2"Export CSV" — RFC 4180 CSV with study metadata, classification, per-domain RoB + applicability ratings, overall RoB + applicability, dev-only downgrade flags, and reviewer override rationale
  3. 3AI run history per assessment shows status (succeeded / partial / failed), token in/out, and cost in cents. Risk-of-bias runs use the configured production model pricing.
  4. 4Per-domain "Re-extract" button clears that slice from descriptionFields so the next Run AI re-extracts only that domain — useful when one domain produced bad output but the others were fine
13

Pulling included references straight from screening

  1. 1In a full-text screening batch, click "Send Included to Risk of Bias" → PROBAST
  2. 2CoreSR finds-or-creates one PROBAST project per (batch × tool), auto-creates a default review question, and populates one assessment per included reference
  3. 3PDFs are copied from screening storage into the PROBAST project's file table with extracted text already populated
  4. 4Idempotent: re-pushing the same batch only adds NEW references; previously-pushed assessments are reused unchanged
  5. 5After import: fill in the four review-question fields, set classification per assessment, and click "Run AI on all"
14

Reviewer best practice

  1. 1Read the AI rationale and evidence quotes BEFORE accepting an answer — the rules table is dense and AI can cite the wrong field
  2. 2When AI confidence is "Low", spend extra time on that row — the cached reviewer panel shows the model answer and rationale
  3. 3For Q4.1, sanity-check the totalN, events, and candidatePredictors numbers in descriptionFields.analysis before accepting the computed answer — these are common targets for extraction error
  4. 4The Dev-only downgrade override should be RARE — most prediction-model development studies do not meet the "very large dataset AND internal validation" bar. Defaulting to High is usually correct.
  5. 5For Dev+Val assessments, you will see two columns side-by-side. They are independent — a study can be Low for Development but High for Validation if the validation cohort had a different ascertainment method, for example.
Tip: PROBAST is famously strict by design. Most published prediction-model studies land at High overall RoB because of Domain 4. Do not soften the published rules just because every study fails — that is the published instrument working as intended.
coreSR.AICoreSR

AI-powered evidence synthesis