Applying advanced psychometric approaches yields differential randomized trial effect sizes: secondary analysis of individual participant data from antidepressant studies using the Hamilton rating scale for depression

Objectives: As multiple sophisticated techniques are used to evaluate psychometric scales, in theory reducing error and enhancing the measurement of patient-reported outcomes, we aimed to determine whether applying different psychometric analyses would demonstrate important differences in treatment effects.

Study design and setting: We conducted a secondary analysis of individual participant data (IPD) from 20 antidepressant treatment trials obtained from Vivli.org (n = 6843). Pooled item-level data from the Hamilton Rating Scale for Depression (HRSD-17) were analyzed using confirmatory factory analysis (CFA), item response theory (IRT), and network analysis (NA). Multilevel models were used to analyze differences in trial effects at approximately 8 weeks (range 4-12 weeks) post-treatment commencement, with standardized mean differences calculated as Cohen's d. The effect size outcomes for the original total depression scores were compared with psychometrically informed outcomes based on abbreviated and weighted depression scores.

Results: Several items performed poorly during psychometric analyses and were eliminated, resulting in different models being obtained for each approach. Treatment effects were modified as follows per psychometric approach: 10.4%-14.9% increase for CFA, 0%-2.9% increase for IRT, and 14.9%-16.4% reduction for NA.

Conclusion: Psychometric analyses differentially moderate effect size outcomes depending on the method used. In a 20-trial sample, factor analytic approaches increased treatment effect sizes relative to the original outcomes, NA decreased them, and IRT results reflected original trial outcomes.

Plain language summary: This study aimed to determine if using advanced psychometrics methods would inform any clinically or statistically important differences in clinical trial outcomes when compared to original findings. We applied factor analysis (FA), item response theory (IRT), and network analysis (NA) to the most commonly used measure of depression in clinical settings - the Hamilton Rating Scale for Depression (HRSD) - to identify and remove nonperforming survey items and calculate weighted item scores. We found that the efficacy reported in trials increased when using FA to removed items, but decreased when using NA. There was almost no change in efficacy when using IRT. Using weighted scores based on respective models offered no additional utility in terms of increasing or decreasing efficacy outcomes.