Usage Guide
QUIPS
How to assess risk of bias for prognostic factor studies
1
What QUIPS is and when to use it
- 1QUIPS (Quality In Prognosis Studies) — Hayden et al., Ann Intern Med 2013
- 2Use for studies of prognostic factors: observational research that asks whether and how strongly a factor predicts a future health outcome
- 3NOT for treatment-effect studies (use RoB 2.0 / ROBINS-I), diagnostic accuracy studies (use QUADAS-2), or prediction model development/validation (use PROBAST)
- 4Unit of assessment: one QUIPS assessment per (publication × prognostic factor × outcome). The bias profile can differ across PF–outcome pairs in the same paper, so the same paper may produce N×M assessments.
Tip: QUIPS is deliberately judgment-driven. The CoreSR rollup gives you a deterministic SUGGESTION; reviewer override is expected as the primary path, not an exception.
2
The 6 domains
- 1Domain 1 — Study Participation: bias from non-representativeness of the study sample
- 2Domain 2 — Study Attrition: bias from differential loss to follow-up by PF status or outcome
- 3Domain 3 — Prognostic Factor Measurement: bias from misclassification of the PF
- 4Domain 4 — Outcome Measurement: bias from differential or invalid outcome measurement
- 5Domain 5 — Study Confounding: the most consequential domain in practice; bias from unmeasured or unadjusted confounders
- 6Domain 6 — Statistical Analysis and Reporting: bias from inadequate model, inappropriate analytic strategy, or selective reporting
3
Hinge items per domain
- 1Each domain has 1-2 "hinge" items that the rollup treats specially — when the hinge is rated N (or sometimes PN), the domain rating jumps to High regardless of other items
- 2D1 hinge: 1.5 (participation rate)
- 3D2 hinges: 2.1 (extent of loss to follow-up) + 2.4 (differential attrition between completers and lost)
- 4D3 hinges: 3.2 (PF measurement validity) + 3.5 (PF missingness)
- 5D4 hinges: 4.2 (outcome measurement validity) + 4.3 (consistency of ascertainment / blinding)
- 6D5 hinges: 5.1 (critical confounders measured) + 5.7 (critical confounders adjusted for)
- 7D6 hinges: 6.3 (model adequacy for the design) + 6.4 (selective reporting)
Tip: In the review panel, hinge items are tagged with a small "hinge" badge and a tooltip. The reviewer can still override the auto-suggested domain rating.
4
Setting up Step 1 (review question)
- 1Open Risk of bias → QUIPS and click "Create new QUIPS project"
- 2Add at least one review question with five fields: Population, Prognostic factor, Outcome, Confounding context, and Moment of prediction
- 3The first review question becomes the project default; you can add more for projects covering multiple PFs or outcomes
- 4Optionally upload a protocol PDF and click "Extract from PDF" to pre-fill the five fields
5
Step 1b — Locking expected confounders (Domain 5 prerequisite)
- 1Click "Generate for outcome" and supply an outcome label (e.g. "5-year mortality after MI"); add the PF and clinical context for better AI gap analysis
- 2CoreSR looks the outcome up in shared/confounder-knowledge-base.json (the same KB that powers ROBINS-I) and seeds the list with critical + important confounders
- 3AI gap analysis suggests context-specific additions the static KB might miss
- 4Edit the list — add, remove, or change importance (critical / important / optional)
- 5Click "Lock" to freeze the list. Domain 5 AI scoring will refuse to run until the list for an outcome is locked.
- 6When the reviewer unlocks + edits later, a NEW VERSION is created and Domain 5 is flagged stale on existing assessments using that outcome — re-run AI on Domain 5 only or acknowledge that the new list does not change your judgment
Tip: The lock uses a race-safe compare-and-set on lockedAt — if two reviewers try to edit at once, the second save fails cleanly with a 409 instead of clobbering.
6
Step 2 — Per-assessment metadata
- 1Each assessment carries four metadata fields: study design, prognostic factor type, outcome type, analysis type
- 2These fields gate Domain 6 model-adequacy logic (e.g. logistic regression on time-to-event data is PN; Cox on time-to-event is Y) and Item 3.3 NA logic (cutpoint handling is NA when PF type is binary)
- 3Set these on the project page or on the assessment detail page. Changing them re-evaluates which items are NA and updates the item-answer stubs accordingly.
7
How AI scoring works (Phase A → B → C)
- 1Phase A: 6 parallel per-domain evidence extractions. Each domain has its own Zod schema; failed slices stay marked failed and the reviewer fills them manually.
- 2Phase B: per-item structured AI scoring. Domain 5 prompts include the locked expected-confounders list inline. Domain 6 prompts include the metadata so model-adequacy gating works correctly.
- 3Phase C: deterministic rollup in server/quips/rollup.ts — six per-domain heuristics with hinge-item weighting; computeOverallSummary produces the CoreSR convention output.
- 4Idempotency: every AI run is keyed on (metadata, doc text, expectedConfounders version, calibration, decision-rules version). Re-running with no upstream change short-circuits.
8
Numeric thresholds — per-project calibration
- 1Several items use numeric thresholds that QUIPS itself does not specify: participation rate (1.5), loss-to-follow-up (2.1), PF missingness (3.5), confounder missingness (5.5)
- 2CoreSR ships defaults from epidemiological convention (≥80% / 60-80% / 40-60% / <40% participation, etc.) but allows per-project overrides
- 3Open the project settings to tune thresholds for context-specific reviews (e.g. emergency-department studies have lower acceptable participation rates)
- 4The chosen calibration is recorded in the AI prompt + audit log so historical assessments stay reproducible
9
Domain rollup — auto-applied with reviewer override
- 1When the heuristic produces a suggestion, it becomes the algorithmRating immediately and finalRating mirrors it (PROBAST-style auto-apply)
- 2Reviewer userRating overrides the algorithm. Both are persisted; the dashboard shows the algorithm rating in italics when an override is in place.
- 3NA items are skipped in the rollup — the autoSetReason is shown inline so reviewers know why
- 4Reviewer can override any item answer or domain rating one-click; rationale is encouraged and audit-logged
10
Domain 5 stale-confounders flow
- 1When you supersede a locked expected-confounders list, every Domain 5 row for assessments using that outcome is marked stale
- 2Stale Domain 5 rows show a yellow banner with two options: re-run AI on Domain 5 (preferred) or acknowledge that the new list does not change your judgment for this study
- 3Acknowledge requires a free-text rationale and is audit-logged
- 4Sign-off REFUSES (409) when any Domain 5 is still stale — the platform forces you to deal with it before locking the assessment in
11
CoreSR overall summary (collapsed by default)
- 1QUIPS does NOT mandate an overall risk-of-bias rating — domain ratings are the canonical output
- 2CoreSR offers a computed convention (any High → High; any Moderate → Moderate; else Low) for downstream consumers (dashboards, GRADE)
- 3The tile is COLLAPSED BY DEFAULT (codex audit recommendation) so domain ratings dominate the page
- 4CSV exports use the column name synthero_overall_convention (not overall_risk) so it can't be confused with a QUIPS-mandated rating
- 5Reviewer override on the overall is supported and logged
12
Sign-off and audit
- 1Click "Sign off" on the overall tile to mark the assessment complete
- 2Sign-off records signedOffBy + signedOffAt and is audit-logged
- 3Domain 5 must not be stale; assessment must have a final rating per domain
- 4The append-only audit log captures every metadata change, override, lock, supersede, AI run, and sign-off — useful for methods appendix and reviewer accountability
13
Bulk operations and project export
- 1"Run AI on all" enumerates which assessments are ready (have files + locked confounders + outcome label) and which are skipped (and why)
- 2"Export CSV" produces RFC 4180 CSV with study metadata, six per-domain ratings, the CoreSR overall convention, sign-off status, and which expected-confounders version Domain 5 was scored against
- 3Per-domain "Re-extract" button clears that slice from descriptionFields so the next Run AI re-extracts only that domain
14
Pulling from screening
- 1In a full-text screening batch, "Send Included to Risk of Bias → QUIPS" creates a QUIPS project keyed on (batch × tool), auto-creates a default review question, and one assessment per included reference
- 2Idempotent: re-pushing the same batch only adds NEW references
- 3After import: fill in the five review-question fields, generate-and-lock expected confounders for each unique outcome, set per-assessment metadata, then run AI
15
Reviewer best practice
- 1Read the AI rationale and evidence quote BEFORE accepting an answer — the QUIPS items are dense and AI can cite the wrong field
- 2When AI confidence is low or evidence is sparse, spend extra time on that row
- 3For Domain 5: even if CoreSR auto-marks an item Y, manually inspect the per-confounder coverage subpanel — semantic equivalence (e.g. "diabetes" vs "type-2 diabetes") may not be detected by the string-match heuristic
- 4Hinge items deserve disproportionate attention — they drive the domain rating
- 5For Domain 6, the model-adequacy answer (6.3) depends on the metadata you set in Step 2. If Step 2 is wrong, 6.3 will be wrong. Re-check.
- 6QUIPS does not mandate an overall — the domain ratings are what reviewers and downstream consumers should care about. Treat the CoreSR summary as a convenience, not a verdict.