Aim for clinical utility, not just predictive accuracy

The predictions from an accurate prognostic model can be of great interest to patients and clinicians. When predictions are reported to individuals, they may decide to take action to improve their health or they may simply be comforted by the knowledge. However, if there is a clearly defined space of actions in the clinical context, a formal decision rule based on the prediction has the potential to have a much broader impact. Even if it is not the intended use of a developed prediction model, informal decision rules can often be found in practice. The use of a prediction-based decision rule should be formalized and compared to the standard of care in a randomized trial to assess its clinical utility, however, evidence is needed to motivate such a trial. We outline how observational data can be used to propose a decision rule based on a prognostic prediction model. We then propose a framework for emulating a prediction driven trial to evaluate the utility of a prediction-based decision rule in observational data. A split-sample structure can and should be used to develop the prognostic model, define the decision rule, and evaluate its clinical utility.


Step-by-step Guide
To make things concrete, we will consider the setting of major abdominal surgery in Crohn's Disease (CD). Major abdominal surgery due to CD is considered a serious adverse outcome, and is responsible for high health care costs and decrease in quality of life in people with CD. Identifying individuals at high risk for surgery may allow for targeted use of early therapeutic interventions to offset this natural course. We have simulated data to mimic a large national cohort of CD patients with clinical information, demographics, treatment history, and up to 10 years of follow up. Here we outline the steps that we will undertake to develop and evaluate a prediction based decision rule.

Preparation and planning
1. Decide on eligibility criteria for the potential randomized clinical trial to assess a prediction based decision rule. This includes both patient characteristics and the time at which the decision will be made, e.g., within 2 weeks of diagnosis. Record this information in a draft protocol.
2. Assemble a cohort of patients who meet the eligibility criteria. Define covariate data that is available before the time at which the decision is to be made, and then the clinical outcome of interest and the treatments that are observed at the time when the decision is to be made.
Record this information in the draft protocol.
3. Randomly allocate patients in the cohort into 3 groups: Model, decision rule, and clinical utility.

Development of prognostic model
1. In the first model subcohort, decide on the statistical techniques that will be used to develop prediction models. Record this information in a protocol.
2. Apply the statistical techniques to develop a prediction model for the clinical outcome using the covariates that are available before the decision.
3. In a cross-validation framework, assess the accuracy of the prediction model development process that was used in the previous step.
4. Report the prediction model algorithm and its cross-validated performance. Test that the algorithm is reproducible and can be applied to new data in the same format.

Development of decision rule
1. In the decision rule subcohort, apply the prediction algorithm developed in the previous step.
Examine the distribution of the predictions in this new cohort. Use predictiveness curves and clinical consultation to decide on cutoffs for high risk versus low risk. Record this decision making process and the results in a protocol.

Evaluation of the decision rule
1. In the clinical utility cohort, apply the prediction algorithm and cutoffs to obtain the classification into high risk and low risk.
2. Specify the grace period as the time from eligibility to when a treatment was received (e.g., two weeks), and use this to define the observed treatment for each individual in the sample. See eFigure 1 below.
3. In the subgroups of individuals classified as high and low risk separately, specify and fit a regression models for the outcome as a function of the observed treatment and observed confounders of the treatment-outcome association. Make sure to include relevant treatment-covariate interactions. 4. Using the models estimated in step 3, obtain predictions for each subject by risk group using their observed covariates, and setting their treatment to the treatment prescribed by the proposed decision rule. These are the predicted potential outcomes.

##
Step 7 M1 -M2 } data <-samp_data() mainest <-estimate_utility(data) The estimated clinical utility of the decision rule is -0.11 95% CI: -0.17 to -0.05. This is interpreted as the estimated difference in the proportion having the outcome comparing the prediction based decision rule arm to the standard of care.