Target Variables

To predict the severity of AP, this research uses the definitions of RAC which categorise the disease into mild, moderate, or severe. Binary classification tasks with two classes were formed by combining the RAC subgroups, where moderate and severe are one class, and mild is the second class. The formation of binary classes provides the opportunity to compare with the clinical scoring systems directly.

Machine Learning Algorithms and Evaluations

This research uses three machine learning algorithms: support vector machine (SVM), random forest (RF), and XGBoost (XGB). SVM is an algorithm that assigns labels to objects from learning by example. SVM uses kernel functions to transform the input data into a higher-dimensional feature space [8]. RF is made of multiple decision trees, combining their outputs to gain a single result. RF uses bagging, also known as bootstrapping, which refers to sampling with replacement [9]. XGBoost is a scalable boosting system that builds on decision trees, ensemble learning, and gradient boosting [10].

Three evaluation metrics are discussed in this research: sensitivity (SEN), specificity (SPE), and accuracy (ACC). SEN describes a model’s probability of gaining a positive result, where a high SEN indicates a reduced likelihood of missing a positive diagnosis or a lower false negative rate [11]. SPE describes a model’s probability of gaining a negative result, where a high SPE indicates a reduced likelihood of misdiagnosing a negative classification, leading to a lower false positive rate [11]. ACC measures the model’s ability to correctly diagnose patients, where a high ACC may indicate a useful classifier. It is a proportion of correctly classified samples in the total samples [12].

Experimental Setup

This research was implemented using Python. The sci-kit library implements the SVM and RF classifiers, and hyperparameter tuning. The XGBoost classifier, meanwhile, runs based on the XGBoost library. All experiments employ nested stratified cross-validation, reporting the average performance of each run. We defined a fixed random seed to ensure experiments and results are reproducible. The code written for this research was executed on Jupyter Notebook.

Results

Table 1 presents results of machine learning algorithms based on RAC using binary classification. The SVM and RF models have high SPE compared to XGBoost, indicating a reduced likelihood of misdiagnosing a negative classification. On the other hand, the XGBoost model excels in SEN, suggesting that XGBoost is particularly apt for screening purposes, and can accurately pinpoint patients who are not severe cases. Overall, all machine learning models demonstrate a high level of accuracy, indicating good discrimination.

Table 2 presents the results of severe AP patients classified using clinical scoring systems. In comparison to machine learning models, as shown in Table 1, the accuracy and sensitivity of clinical scoring systems are poor, and the specificity scores are similar or slightly better. In terms of overall scoring, machine learning models appear to be comparable or better than clinical scoring systems indicating the potential of machine learning algorithms for predicting the severity of acute pancreatitis.

Discussion

This research provided experimental evidence showcasing the capabilities of machine learning algorithms predicting severe AP cases using routinely collected hospital data. We also provide a direct comparison with clinical scoring systems, where the performance of machine learning models are consistently better. This research has many future directions, including obtaining data from other hospitals for further validation of the ML models, discussions with experts to design a deployment plan, and incorporating clinician feedback.

Data

This study uses a retrospective dataset comprising patient data from 2009 to 2013 in a mainland Chinese hospital. Adult patients admitted to the hospital and diagnosed with AP, as per the 2012 updated Atlanta criteria, were included in the study. The dataset contains 2,581 deidentified patient data with more than 50 variables, including diagnoses, decisions, vital signs, routine laboratory test results, and calculations using scoring systems. The dataset also includes the patient level of AP for patients, categorised using RAC, and comprises 65.2% male and 34.8% female patients, with an average age of 47.2 years and an age range of 18 to 80 years. Figure 2 provides an overview of the distribution of data based on the age and gender of patients.

Among 50+ variables, 34 features were selected through expert recommendation. These included risk factors such as patient’s gender, age, and abdominal pain onset time; triage information such as high dependency unit and intensive care unit; and vital signs including temperature, respiration, and heart rate. Information on organ failure, the site of the organ failure (lung, kidney, heart, etc.), and selected variables from the laboratory test results were also included.

Table 1: Machine learning models for predicting severe AP based on RAC using binary classification. Mean values with standard deviations are presented. The best scores for each evaluation metric are bolded.

Acknowledgements

I am incredibly grateful for the guidance and support provided by my supervisors, Professor Gill Dobbie, Dr Vithya Yogarajan, and Professor John Windsor, throughout my summer research scholarship. Their continuous assistance and feedback made the project a whole lot less daunting, which I undoubtedly appreciated. I’d also like to thank the Master’s student I worked with, Xiaoheng Ji, for being so kind and receptive to all of my questions.