Potential of machine learning methods to identify patients with nonvalvular atrial fibrillation

Atrial fibrillation (AF) is the most common form of arrhythmia, particularly in the elderly [1,2]. AF increases the risk of ischemic stroke, in particular embolic stroke, which is in general the most severe form of stroke with the poorest outcomes [3,4]. For prevention of cardioembolic stroke, anticoagulation therapy as well as appropriate AF management is essential. However, there are still many patients who are not diagnosed with AF before having a cardioembolic stroke [5,6]. Unfortunately, paroxysmal AF, reportedly constituting 50% or more of any-cause AF [7], is often asymptomatic and difficult to detect in general clinical practice [8], with electrocardiography is the only definite method for diagnosis. Shorter duration 12-lead electrocardiogram (generally for several minutes) is not sufficient to find paroxysmal AF and longer monitoring such as event recorder electrocardiography is generally required. However, wearing a device for a long period of time can be burdensome and is not always practical. Taking into consideration these limitations, a two-step approach might be useful, whereby patients at high risk for AF are first identified and then screening by electrocardiography is undertaken among these high-risk patients to diagnose AF.

Although classical cardiovascular risk factors for AF have been investigated in previous studies, the overall identification ability of these factors may not be sufficient to justify screening large populations who present with such risk factors [9–12]. It is possible that a machine-learning approach could be used to develop a better identification model for AF by allowing the consideration of a larger number of variables, although to the best of our knowledge there are currently no reports of machine leaning-based AF detection in the literature. The objective of this study was therefore to assess the identification ability of a machine learning-based algorithm for the detection of nonvalvular AF (NVAF) using data obtained from a Japanese health claims database.

Materials & methods

Data source

This study used de-identified health claims and Diagnosis Procedure Combination (the flat-free payment system) data from 314 acute care hospitals across Japan, obtained from the Medical Data Vision (MDV) Co, Ltd (Tokyo, Japan). The details of this database have been described elsewhere [13,14]. In brief, the database includes administrative data from 15 million patients in Japan. The distribution of demographic characteristics including the age and gender distributions of patients included in the database is similar to national statistics in Japan. The database includes inpatient and outpatient administrative data from participating hospitals. Although the data contain clinical and demographic characteristics, medical procedures and medications prescribed for all patients, laboratory data (clinical laboratory test results) and other clinical tests (such as blood pressure values) are not always available, with availability varying by hospital. Diseases were identified using International Classification of Disease, 10th Revision (ICD-10) codes and/or the local standard disease codes to supplement ICD-10 codes. The codes used in the current study are shown in Supplementary Table 1. Medical procedures were defined with the procedure codes shown in Supplementary Table 2.

Patient inclusion

Criteria for selection of NVAF and non-AF cohorts are shown in Figure 1. NVAF patients were defined as patients with at least two diagnosis codes for AF (I48), but not with valvular AF (standard disease code 8846941), postoperative AF (8847772), mechanical-valvular AF (T820) or chronic rheumatic heart diseases (I05-I09) between April 2008 and August 2017. Patients with a history of procedures involving prosthetic heart valves were also excluded. According to these criteria, 33,885 patients were selected into the NVAF cohort. The date of AF diagnosis (that is, first appearance of the I48 code during a whole available period in the database) was defined as the index date. The final NVAF cohort included, patients who had data for all essential variables and at least 60% of ‘optional variables’ within a period of ‘index date–6 months’ were selected (Table 1). Patients who had never been diagnosed with AF (no ICD code of I48 but having at least two visits and at least 1 year of available data) within the study period were defined as non-AF patients. Patients diagnosed with valvular AF, postoperative AF, mechanical-valvular AF or chronic rheumatic heart diseases and those with a history of valve replacement procedures were also excluded from this cohort. The index date for each non-AF patient was defined the date of last visit where laboratory data were available. Patients who had data for all essential variables and for at least 60% of ‘optional variables’ within a period of ‘index date–6 months’ were selected (Table 1). In total, 27,592 patients were included in the NVAF cohort and 463,750 patients in the non-AF cohort.

Variable selection

Variables were selected based on known associated with NVAF, and included demographic characteristics (age, sex, height and weight) [12], comorbidities (hypertension, diabetes [15], ischemic stroke [16], heart failure [16], anemia [17], hyperthyroidism [18], chronic obstructive pulmonary disease [19] and ventricular premature contraction [20], coronary artery disease [21] and peripheral artery disease [16]) and laboratory data related to renal function (creatinine and glomerular filtration rate), metabolic disease (cholesterol and triglyceride) [12] and others (hemoglobin A1c [15], brain natriuretic protein [22] and uric acid [23]) were selected. Additional laboratory test values (hematocrit, hemoglobin, mean corpuscular volume, mean corpuscular hemoglobin, mean corpuscular hemoglobin concentration, white blood cell count, red blood cell count, platelet count, serum total protein, GOT, GPT, gamma-GTP, ALP, LDH, CRP, urinary nitrogen, K⁺, Cl⁻, Na⁺ and Fe²⁺) were selected as these tests are commonly used in general clinical practice.

Statistical analysis

Datasets

The raw data were processed to create datasets for analysis. First, missing data were imputed by multivariate imputation by chained equations. The proportion of missing values for each variable is shown in Supplementary Table 3. The dataset was then randomly divided in three at a 40:40:20 ratio, a training dataset 1, training dataset 2 and a validation dataset, respectively. And the ratio of NVAF/non-AF was kept consistent among three datasets. These datasets were used for development of both model 1 and model 2. Overall steps are shown in Figure 2. For model 2, to alleviate imbalance in the dataset (the number of non-AF patients was much larger than NVAF patients), random undersampling of the majority class was applied to the dataset so that the ratio of NVAF to non-AF was 1:1. After undersampling, the final training dataset contained 11,036 patients with NVAF and 11,036 non-AF.

Model-1

For Model 1, a ‘deep learning’ (DL) method was used to create multiple features to predict NVAF in Step 1 using the training dataset 1. DL-based features created in the Step 1 were then applied to Step 2, where Lasso regression (LR) was used to develop a classifier using the training dataset 2. In Step 3, the developed algorithm was validated using the validation dataset.

Model-2

For Model 2, stacking of multiple single classifiers was used, which included LR, ridge regression, support vector machine, random forest, DL, AdaBoost and gradient boosting. These multiple models were trained using the training dataset 1. Each model provided probability of AF as an outcome which were then cast into a second level model (LR model), where the outcomes obtained in the first step became features in the second step. The developed algorithm was validated using validation dataset in model validation step.

Variable-feature relation analysis

We investigated the relationship between features developed by stacking model 1 or 2 and each of the variables included in the analytical dataset. The relationships were described using a Pearson correlation coefficient.

Receiver operating characteristic (ROC) analysis was conducted to estimate area under the ROC curve (AUC), sensitivity and specificity, for each of the developed algorithms. All of the above analyses were performed using Python version 3.6.

Results

Patient characteristics are shown in Table 1. As expected, there were marked statistically significant differences between patients in the NVAF and non-AF cohorts in terms of key clinical and demographic characteristics. In general, patients in the NVAF cohort tended to be older with a higher comorbidity burden.

The ROC curve suggested a small difference in performance between the stacking model and other methods (Figure 3). AUC, sensitivity and specificity are shown in Table 2. Stacking models 1 and 2 had an AUC of 0.90, specificity and sensitivity were also well-balanced (Table 2). The AUC for each single model was >0.8, and the sensitivity of single models varied between 0.09 and 0.84, and specificity between 0.79 and 0.99.

We visualized the correlation between study variables and each of the features engineered by the DL algorithm for stacking model 1 (Supplementary Figure 1) and each of the single model algorithms developed for stacking model 2 (Supplementary Figure 2). For the deep-learning developed features used in stacking model 1, almost all of the developed features were associated to used variables, although associations were strongest with arterial disease and COPD (Supplementary Figure 1). Importantly though, some of the developed features had no correlation with used variables (for example, features 10 and 15 in Supplementary Figure 1). Age and select comorbidities including heart failure, hypertension, diabetes and coronary arterial disease were most strongly correlated with each of the single model algorithms (Supplementary Figure 2). Lab tests tended to be weekly correlated, although among the lab tests considered brain natriuretic protein, urinary nitrogen, uric acid showed the strongest associations.

Discussion

In this Japanese study, we found that machine learning methods can be used to develop an algorithm which detects patients with NVAF from patients without NVAF with a level high accuracy. All machine learning algorithms developed in this study had at least an 80% chance of correctly identifying NVAF patients. These results should be considered exploratory and require further validation, but encouragingly demonstrate the potential utility of machine-learning approaches to help detect patients with NVAF in the general population.

AF risk scores have been reported from both Western countries and Japan. The CHARGE-AF study has proposed a validated score for AF by using 5 US and European cohorts and its c-statistic was about 0.7 [10]. A Japanese risk score to predict a 10-year risk of AF proposed by the Suita study also had a c-statistic of 0.7 s [24]. Both scores have been prepared for the purpose of being used to effectively screen for AF in general outpatient clinics without 12-lead ECG measurement, and the c-statistics could be large enough for this purpose.

In the CHARGE-AF and the Suita study risk scores, variables that are established cardiovascular risk factors, or are considered as factors associated with AF, were selected and used for preparation of the scores. However, in the present study, factors available from the database irrespective of their potential association with AF, were used for development of the algorithm. As a result, our algorithm contained more information on comorbid disease and laboratory tests, however, it is uncertain how these differences affected our model performance. Unfortunately, due to the limited availability of variables, we did not perform cross-validation between our algorithm and these previously reported risk scored. Performance of the algorithm should be examined by the cross-validation in the future.

In this study, we used an administrative claims database, which includes real-world data related to patient health status and provision of clinical care collected in general clinical practice. AF is difficult to be diagnosed in clinical practice and there might be a considerable number of patients with AF who have not yet diagnosed. These patients with undiagnosed AF were included in the non-AF cohort in the present study and might have an impact on the result. This is a critical limitation of this study. A prospective, probably ‘interventional’, study may be necessary to actively diagnose NVAF to address this issue.

In the present study, we did not show which factors/features contributed the most to identification of NVAF. Some variables that are not directly significantly associated with the outcome (NVAF in this study) are sometimes considerably contributed to the outcome when a machine learning-based approach, particularly DL or stacking, is applied to development of algorithms and vice versa. This is a critical difference between conventional statistics-based approach (such as risk scores based on risk factors) and our DL-based approach.

There are limitations to our study that should be considered. First, as already mentioned above, our model was prepared and validated only by using claims data from the MDV database. This dataset might contain patients with misdiagnosed AF and this could affect model validity. Second, the MDV dataset is limited to patients treated within an acute care hospital setting who may be sicker and have more comorbid disease than would otherwise be observed in the broader Japanese population. Therefore, inferences gleaned from this study may not be applicable to patients solely seen in primary care or people who have had no recent healthcare interaction, and any comparisons of the algorithm performance seen in this study with those of CHARGE-AF and the Suita study should be viewed in this context. The algorithm developed in this study distinguished patients with diagnosed NVAF from those without a diagnosis, however it remains possible that patients with NVAF who are currently undiagnosed are distinct to those known. This possibility as well as algorithms to identify such patients should be explored in future studies.

We selected variables based on known risk factors and data clinically available in general hospital practice. However, requiring patients to have at least 60% of test data available may have introduced bias if there are systematic differences between patients with a lab test recorded and those without, and therefore patients with >60% of data available may not be representative of a general hospital-admitted population. Additionally, there is very limited information on potential confounding factors such as socioeconomic status, smoking [25] or alcohol intake [26] which might improve or maximize the machine learning based algorithm to avoid missing latent AF patients. As we comment above, our results should be regarded as speculative rather than definitive: they represent results from what can be done using very large-scale, routinely collected administrative data and require further work, in different study designs, to confirm or refute the findings.

Conclusion

Machine-learning based algorithms can detect NVAF with accuracy. Although additional validation is needed, this methodology could encourage a new approach to improve the detection of patients with NVAF in the general population in an aging society like Japan.

Future perspective

Earlier detection of NVAF is key to prevent stroke and improve outcomes of patients, particularly in increasingly aging societies like Japan. Efforts to improve screening and earlier detection of at-risk patients will continue to evolve in coming years. As health-related data proliferate and advanced analytical techniques to handle such data continue to evolve alongside such efforts, the ability of these innovate approaches to be applied in everyday clinical settings to support decision-making for the detection of the NVAF may become realized. Indeed, machine learning is playing an increasing role within many areas of healthcare, including but not limited to earlier diagnosis and has the potential to transform clinical care.