Royal College of Surgeons in Ireland
2 files

A simplified approach to the pooled analysis of calibration of clinical prediction rules for systematic reviews of validation studies.

Version 2 2022-02-14, 17:33
Version 1 2019-11-23, 10:34
journal contribution
posted on 2022-02-14, 17:33 authored by Borislav D. Dimitrov, Nicola Motterlini, Tom FaheyTom Fahey

OBJECTIVE: Estimating calibration performance of clinical prediction rules (CPRs) in systematic reviews of validation studies is not possible when predicted values are neither published nor accessible or sufficient or no individual participant or patient data are available. Our aims were to describe a simplified approach for outcomes prediction and calibration assessment and evaluate its functionality and validity.

STUDY DESIGN AND METHODS: Methodological study of systematic reviews of validation studies of CPRs: a) ABCD(2) rule for prediction of 7 day stroke; and b) CRB-65 rule for prediction of 30 day mortality. Predicted outcomes in a sample validation study were computed by CPR distribution patterns ("derivation model"). As confirmation, a logistic regression model (with derivation study coefficients) was applied to CPR-based dummy variables in the validation study. Meta-analysis of validation studies provided pooled estimates of "predicted:observed" risk ratios (RRs), 95% confidence intervals (CIs), and indexes of heterogeneity (I(2) ) on forest plots (fixed and random effects models), with and without adjustment of intercepts. The above approach was also applied to the CRB-65 rule.

RESULTS: Our simplified method, applied to ABCD(2) rule in three risk strata (low, 0-3; intermediate, 4-5; high, 6-7 points), indicated that predictions are identical to those computed by univariate, CPR-based logistic regression model. Discrimination was good (c-statistics =0.61-0.82), however, calibration in some studies was low. In such cases with miscalibration, the under-prediction (RRs =0.73-0.91, 95% CIs 0.41-1.48) could be further corrected by intercept adjustment to account for incidence differences. An improvement of both heterogeneities and P-values (Hosmer-Lemeshow goodness-of-fit test) was observed. Better calibration and improved pooled RRs (0.90-1.06), with narrower 95% CIs (0.57-1.41) were achieved.

CONCLUSION: Our results have an immediate clinical implication in situations when predicted outcomes in CPR validation studies are lacking or deficient by describing how such predictions can be obtained by everyone using the derivation study alone, without any need for highly specialized knowledge or sophisticated statistics.


Health Research Board Ireland



The original article is available at

Published Citation

Dimitrov BD, Motterlini N, Fahey T. A simplified approach to the pooled analysis of calibration of clinical prediction rules for systematic reviews of validation studies. Clinical Epidemiology. 2015;7:267-80

Publication Date


PubMed ID



  • General Practice
  • HRB Centre for Primary Care Research

Usage metrics

    Royal College of Surgeons in Ireland



    Ref. manager