Can Predictive Modeling Tools Identify Patients at High Risk of Prolonged Opioid Use After ACL Reconstruction?

BACKGROUND: Machine-learning methods such as the Bayesian belief network, random forest, gradient boosting machine, and decision trees have been used to develop decision-support tools in other clinical settings. Opioid abuse is a problem among civilians and military service members, and it is difficult to anticipate which patients are at risk for prolonged opioid use.

QUESTIONS/PURPOSES: (1) To build a cross-validated model that predicts risk of prolonged opioid use after a specific orthopaedic procedure (ACL reconstruction), (2) To describe the relationships between prognostic and outcome variables, and (3) To determine the clinical utility of a predictive model using a decision curve analysis (as measured by our predictive system's ability to effectively identify high-risk patients and allow for preventative measures to be taken to ensure a successful procedure process).

METHODS: We used the Military Analysis and Reporting Tool (M2) to search the Military Health System Data Repository for all patients undergoing arthroscopically assisted ACL reconstruction (Current Procedure Terminology code 29888) from January 2012 through December 2015 with a minimum of 90 days postoperative follow-up. In total, 10,919 patients met the inclusion criteria, most of whom were young men on active duty. We obtained complete opioid prescription filling histories from the Military Health System Data Repository's pharmacy records. We extracted data including patient demographics, military characteristics, and pharmacy data. A total of 3.3% of the data was missing. To curate and impute all missing variables, we used a random forest algorithm. We shuffled and split the data into 80% training and 20% hold-out sets, balanced by outcome variable (Outcome90Days). Next, the training set was further split into training and validation sets. Each model was built on the training data set, tuned with the validation set as applicable, and finally tested on the separate hold-out dataset. We chose four predictive models to develop, at the end choosing the best-fit model for implementation. Logistic regression, random forest, Bayesian belief network, and gradient boosting machine models were the four chosen models based on type of analysis (classification). Each were trained to estimate the likelihood of prolonged opioid use, defined as any opioid prescription filled more than 90 days after anterior cruciate reconstruction. After this, we tested the models on our holdout set and performed an area under the curve analysis concordance statistic, calculated the Brier score, and performed a decision curve analysis for validation. Then, we chose the method that produced the most suitable analysis results and, consequently, predictive power across the three calculations. Based on the calculations, the gradient boosting machine model was selected for future implementation. We systematically selected features and tuned the gradient boosting machine to produce a working predictive model. We performed area under the curve, Brier, and decision curve analysis calculations for the final model to test its viability and gain an understanding of whether it is possible to predict prolonged opioid use.

RESULTS: Four predictive models were successfully developed using gradient boosting machine, logistic regression, Bayesian belief network, and random forest methods. After applying the Boruta algorithm for feature selection based on a 100-tree random forest algorithm, features were narrowed to a final seven features. The most influential features with a positive association with prolonged opioid use are preoperative morphine equivalents (yes), particular pharmacy ordering sites locations, shorter deployment time, and younger age. Those observed to have a negative association with prolonged opioid use are particular pharmacy ordering sites locations, preoperative morphine equivalents (no), longer deployment, race (American Indian or Alaskan native) and rank (junior enlisted).On internal validation, the models showed accuracy for predicting prolonged opioid use with AUC greater than our benchmark cutoff 0.70; random forest were 0.76 (95% confidence interval 0.73 to 0.79), 0.76 (95% CI 0.73 to 0.78), 0.73 (95% CI 0.71 to 0.76), and 0.72 (95% CI 0.69 to 0.75), respectively. Although the results from logistic regression and gradient boosting machines were very similar, only one model can be used in implementation. Based on our calculation of the Brier score, area under the curve, and decision curve analysis, we chose the gradient boosting machine as the final model. After selecting features and tuning the chosen gradient boosting machine, we saw an incremental improvement in our implementation model; the final model is accurate, with a Brier score of 0.10 (95% CI 0.09 to 0.11) and area under the curve of 0.77 (95% CI 0.75 to 0.80). It also shows the best clinical utility in a decision curve analysis.

CONCLUSIONS: These scores support our claim that it is possible to predict which patients are at risk of prolonged opioid use, as seen by the appropriate range of hold-out analysis calculations. Current opioid guidelines recommend preoperative identification of at-risk patients, but available tools for this purpose are crude, largely focusing on identifying the presence (but not relative contributions) of various risk factors and screening for depression. The power of this model is that it will permit the development of a true clinical decision-support tool, which risk-stratifies individual patients with a single numerical score that is easily understandable to both patient and surgeon. Probabilistic models provide insight into how clinical factors are conditionally related. Not only will this gradient boosting machine be used to help understand factors contributing to opiate misuse after ACL reconstruction, but also it will allow orthopaedic surgeons to identify at-risk patients before surgery and offer increased support and monitoring to prevent opioid abuse and dependency.

LEVEL OF EVIDENCE: Level III, therapeutic study.