Augmentation of a multidisciplinary team meeting with a clinical decision support system to triage breast cancer patients in the United Kingdom

Background

The advent of a new generation of machine learning (ML) methods and algorithms applied to healthcare [1,2] has led to the development of AI-enabled computerized clinical decision-support systems (CDSS), that present therapeutic options to support oncologists who are treating cancer patients [3]. IBM developed a CDSS, Watson for Oncology (WFO) that used natural language processing and machine learning to generate ranked, evidence-based therapeutic options [4]. This CDSS provided a range of therapeutic options for consideration in cancer treatment plan decisions for all key treatment modalities (surgery, chemotherapy, radiotherapy, hormone therapy and immunotherapy) and specific treatment types (e.g., types of surgical procedures or chemotherapies) for several tumor types. It was developed in collaboration with experts at Memorial Sloan Kettering Cancer Center (MSKCC) and a detailed description of the WFO system is published elsewhere [5].

One potential application of an AI-enabled CDSS is the streamlining of cancer multidisciplinary team (MDT) meetings. Cancer care by an MDT has been mandated in the United Kingdom's (UK) National Health Service (NHS) since 2000 [6]. As the complexity and volume of cancer cases have increased, and as care has become more personalized, MDTs have struggled to operate effectively [7]. A study by Cancer Research UK found that around half of patients were discussed for 2 min or less, with insufficient time to discuss more complex patients [8]. In 2020, NHS England published guidance on streamlining MDTs [7]. It proposed a process of “introducing Standards of Care as a routine part of MDT to stratify patient cases into those which require full multidisciplinary discussion in the MDT, and those cases which can be listed but not discussed in the MDT, as patient need is met by a Standard of Care (SoC)”. A SoC is defined as “a point in the pathway of patient management where there is a recognized international, national, regional or local guideline on the intervention(s) that should be made available to a patient”. The guidance states that for a patient to be assigned for ‘no discussion at the MDT’, the SoC must have been reviewed by an appropriate person or triage group. The guidance does not specify who/what should make up this triage group, instead it is decided locally.

This study examines and quantifies the extent to which an AI-based CDSS (the CDSS used in this study was WFO) can be used by the MDT to enhance the structured decisions around clinically complex patients, and ultimately reduce the prevailing workload and time pressures by streamlining some patients to ‘no discussion at the MDT’. The study objective was to identify a set of less complex, non-metastatic breast cancer cases in which there was less decision conflict, and suitable for triage in a smaller quorum of clinicians. It assessed the concordance of therapeutic options generated for early-stage non-metastatic breast cancer made by two triage groups, each comprising of an oncologist and surgeon, with the therapeutic options provided by the CDSS, in a retrospective sample of breast cancer patients treated at Guy's Cancer Centre.

Methods

Study design & patient selection

The retrospective concordance study compared local best practice breast cancer MDT treatment decisions with treatment decisions made by: two MDT triage teams, each comprising one consultant breast medical oncologist and one consultant breast surgeon, with and without decision support from the CDSS; the CDSS acting ‘alone’; and the historical MDT decision. MDT triage teams were blinded to each other's and historical MDT decision outcomes. Local best practice MDT breast cancer treatment decisions, or ‘consensus panel decisions’ were derived at the end of the study from the consensus decisions of the two consultant medical oncologists and two consultant breast surgeons with knowledge of the historical MDT outcome and the CDSS therapeutic options, and these decisions were considered the gold standard. The teams reviewed 213 MDT retrospective cases. We included patients discussed at the MDT between 2017 and 2018 apart from three who opted out of MDT triage. Female patients with Stage 1–3 invasive breast cancer were selected, excluding patients who could not be analyzed by the CDSS due to unavailable treatment options. These were patients with recurrent breast cancer, bilateral breast cancer, male patients and adenoid/colloid/secretory/tubular histology. Although the CDSS supported metastatic patients on the first line of therapy, these were also excluded as advanced disease was not deemed appropriate for triage in a real-world setting.

The study was discussed with the GSTT Information Governance team and was classified as a service evaluation and did not require ethical or Institutional Review Board (IRB) approval. In accordance with information governance recommendations, all eligible patients were invited by letter to opt-out, and three patients chose to do so.

WFO

The breast cancer module contained a repository of up to 270 clinical attributes that influenced the WFO algorithm. Examples include demographic/family/medical histories, comorbidities, functional status, endocrine status, genetic profiling, tumor characteristics such as tumor, node, metastasis (TNM) staging/grade/receptor status, prior treatment modalities including their timing and responses to treatments, and major organ functional status. For a given case, WFO used a dynamic subset of attributes to derive a treatment recommendation.

WFO provided a list of US-based treatments, which may differ from UK practice. We were not able to retrain WFO to learn and adapt to UK practice within the constraints of this study. We addressed these differences in practice by ‘post-localization’ analysis which we describe in the Results section of this paper.

Data collection

Patient attributes were extracted from electronic health records and transcribed into a version of the CDSS to support research, the Concordance Case Tool. Treatment options were collected and managed as study data using Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Guy's and St Thomas Hospital. REDCap is a secure, web-based software platform designed to support data capture for research studies [9,10]. Study data were categorized by ‘treatment plan’ (surgery, chemotherapy, endocrine therapy, radiotherapy) and ‘treatment type’ (e.g., for surgery, wide local excision versus mastectomy).

Concordance analysis

Concordance was compared between local current best practice MDT treatment decisions and: the historical MDT decisions; the MDT triage team decisions before and after being shown the CDSS therapeutic options; and the CDSS therapeutic options. The consensus panel reviewed all cases where CDSS therapeutic options were discordant with local best practice. Several clinical scenarios were identified where CDSS US-based treatments differed from UK practice. We were not able to retrain the CDSS to learn and adapt to UK practice within the constraints of this study; however, such localization is feasible, and to assess the potential impact of CDSS adaptation to UK practices these scenarios were re-classified as concordant in a further ‘post-localization’ analysis.

The two independent MDT triage teams were presented with the same patient attributes as the CDSS, and their agreed treatment plan and type were recorded by an independent researcher in REDCap. The teams were then shown the CDSS output and allowed to revise their prior decisions. This generated ‘pre’ and ‘post’ CDSS output for each team.

Data analysis & statistics

The characteristics of each breast cancer case were summarized using counts and frequencies. Concordance is reported with 95% confidence intervals (CIs) approximated using the Wilson interval. Fisher's exact test is used to assess differences in concordance between the MDTs and the CDSS and to assess differences in concordance within case characteristics. Estimates for performance metrics were obtained by randomly sampling 50% of the data 1000 times and taking the median performance of the decision tree on these subsamples. 95% CIs for the performance metrics were calculated by taking the 2.5th and 97.5th percentiles of the 1000 values. All analyses were conducted using the R software version 3.6.3.

A decision tree that decides whether to triage breast cancer patients to the CDSS or the MDT was created using fast-and-frugal trees (FFTs) that were constructed with the R package FFTrees [11]. FFTs [12,13] are supervised learning algorithms for binary classification tasks. We chose age, prior early-stage treatments, clinical stage, histology, estrogen receptor (ER) status, progesterone receptor (PR) status and human epidermal growth factor receptor 2 (HER2) status as variables for the decision tree because they had particularly high or low decision concordance and were clinically relevant. The outcome variable being classified was the CDSS concordance with consensus panel MDT treatment recommendations.

The values of a confusion matrix of case concordance and the prediction by the CDSS were used to calculate the performance metrics that in turn were used to assess the predictive ability of the FFTs. The performance of decision trees was assessed using sensitivity, specificity, balanced accuracy, weighted accuracy and percentage of patients whose cases were sent to be analyzed by the CDSS.

In this study, a false positive translates to a breast cancer patient case being sent to the CDSS and receiving treatment therapeutic options discordant with the consensus panel MDT. Conversely, a false negative translates to a breast cancer patient case being sent to an MDT despite the CDSS providing therapeutic options concordant with the consensus panel MDT. A false positive will result in poorer patient outcomes than a false negative; therefore, we aimed to create a decision tree that minimized the number of false positives, in other words, that maximized specificity.

Weighted accuracy considers both sensitivity and specificity and, depending on a weight, w, can be weighted in favor of one or the other. Values of w closer to 1 weight in favor of predictive tools with high sensitivity and values of w closer to 0 weight in favor of tools with high specificity. We chose weighted accuracy with a weight of 0.3 to be the performance metric used in building and choosing FFTs. The most accurate MDT triage team achieved 96.2 and 93.9% concordance for treatment plan and type, respectively. These values were therefore selected as the minimum benchmarks for specificity in the study. After the best performing FFT was chosen, an estimate of each performance metric was calculated with 95% CIs.

Results

Sample characteristics

The characteristics of each case of breast cancer included in the study can be found in Table 1.

Post-localization

Four main clinical practice changes to the CDSS's therapeutic options were identified in order to adapt the CDSS to conform to local best practice (referred to here as ‘localization’). In 13 cases this was a change in treatment plan, while in 36 cases this was a change in treatment type. These were: use of neoadjuvant carboplatin for triple negative breast cancer patients (TNBC; 0/13 for treatment plan, 22/36 for treatment type), avoidance of adjuvant Capecitabine after neoadjuvant chemotherapy in ER positive (ER+) patients (7/13 for treatment plan, 7/36 for treatment type), avoidance of adjuvant capecitabine after neoadjuvant chemotherapy in patients that had received neoadjuvant carboplatin (6/13 for treatment plan, 6/36 for treatment type), and use of adjuvant pertuzumab in node negative (N0) patients (0/13 for treatment plan, 1/36 for treatment type).

Concordance

Figure 1 displays the concordance of each group.

The CDSS achieved 85% concordance with local best practices for treatment plan recommendations and 70% concordance for treatment type recommendations. When localized to local clinical best practice, the CDSS achieved much higher concordance, and no statistical difference with either MDT triage team was observed (treatment plan decisions: 92% CDSS vs 96% team 1 vs 92% team 2, not significant (NS); treatment type decisions: 89% CDSS vs. 93% team 1 vs 82% team 2, NS).

Tables 2 and 3 display for treatment plan and treatment type, respectively, the CDSS's concordance before and after localization to local clinical best practice.

The CDSS showed high treatment plan concordance with local best practices in treatment naive patients, patients with prior chemotherapy, and in Stage 1 disease. After adapting to UK treatment practice (localization) high treatment plan concordance was also seen in ductal histology. For treatment type high concordance was demonstrated in patients with prior chemotherapy, Stage 1 disease and multicentric tumors. Statistically significant positive differences in concordance were also found in cases who were ER+, PR+, and HER2+ but concordance levels were low (79.7, 80.2 and 86.1%). Post-localization, the CDSS showed higher treatment type concordance with local best practice in cases with Stage 1 cancer (100%).

FFT for treatment plan

FFT analysis was used to identify clinical parameters in which the CDSS therapeutic options showed high concordance with MDT outcomes. This would allow identification of a subset of patients that could be triaged to the CDSS and would not require an MDT discussion.

Figure 2 displays the decision tree that decides whether a case should be triaged to the CDSS or an MDT for their treatment plan. A case should be triaged to the CDSS for their treatment plan if:

• The case has a stage 1 cancer;

• The case has stage 2 or 3 cancer, is under 50 years, and HER2+;

• The case has stage 2 or 3 cancer, is under 50 years, HER2-, and ER-.

The performance of this decision tree is presented in Table 4. The sensitivity of the decision tree was 43.4% (95% CI: 36.4–50.5%) and the specificity was 100.0% (95% CI: 100.0–100.0%). On average, 40.2% (95% CI: 33.6–46.7%) of cases were triaged to the CDSS. Because the specificity of this tree did not fall below 100%, we can say that on average, we would expect 40% of cases to be correctly triaged to the CDSS for a treatment plan.

FFT for treatment type

Figure 2 displays the decision tree that decides whether a case should be triaged to the CDSS or an MDT for their treatment type. A case should be triaged to the CDSS for their treatment type if:

The performance of this decision tree is presented in Table 4. The sensitivity of the decision tree was 38.9% (95% CI: 32.6–46.3%) and the specificity was 100.0% (95% CI: 100.0–100.0%). On average, 34.6% (95% CI: 29.0–41.1%) of cases were triaged to the CDSS. Because the specificity of this tree did not fall below 100%, we can say that on average, we would expect 35% of cases to be correctly triaged to the CDSS for treatment type.

Discussion

This study is one of the first to evaluate an AI-based oncology CDSS tool in its utility in streamlining workloads for MDTs. We found that up to 40% of eligible breast cancer cases can be triaged by a CDSS straight to the treating clinician, avoiding MDT discussion and potentially reducing MDT workloads or increasing time for discussion of more complex treatment decision cases. The reduction in the size of the MDT list by 40% may not necessarily equate to a 40% reduction in MDT time, if the time required to discuss a case is proportional to its complexity, and less complex patients are triaged to the CDSS. This may not be a reasonable assumption however, and a prospective study would be required to establish the actual time saving.

The validity of this CDSS's therapeutic options has been assessed in studies in India, Korea and China by measuring the level of concordance between the CDSS and local multidisciplinary tumor Boards [14–19]. Results have been variable, ranging from 20.2% concordance in colon cancer patients aged 70 and over at Gil Medical Centre, South Korea [14], to 98% in HER2 positive metastatic breast cancer patients at Manipal Hospital, India [17]. Several explanations have been offered for this observed variance in concordance including tumor type, disease stage, patient age, availability of drugs and other local practice differences, different release versions of the CDSS [14–19] and an MSKCC bias towards US published studies [20].

Although published evidence is limited, it is not unreasonable to suggest that CDSSs may have a place in routine cancer care. This is supported by a study from Thailand which found 70% of CDSS therapeutic options acceptable or identical to local practice where the gold standard was made by an expert panel of clinicians rather than concordance with the MDT only [5], and a systematic review of all previous concordance studies showed the CDSS had greater concordance with MDTs than individual clinician decisions [21].

A recent meta-analysis found the highest consistency in the CDSS ‘recommended’ and ‘for consideration’ MDT decisions was in breast cancer cases [22]. Although direct comparison is limited due to our separating treatment plan versus treatment type, their subgroup analysis was consistent with our own decision tree. They found there was greater consistency in lower stage cases, and when they looked at “recommended” treatment options only they found greater consistency in hormone receptor negative patients (including hormone receptor negative, HER2+) than hormone receptor positive patients, which also echoes our own decision tree for treatment type and sending cases straight to the CDSS. Papers also highlighted the need for localization as well as the role of CDSSs in supporting rather than replacing clinician decision making [22].

Our study examined a specific application of a CDSS within a UK cancer pathway designed to address a pressing NHS need of cancer care providers: how best to introduce standards of care to streamline MDTs so that some patients, once reviewed by an appropriate person or triage group to have met recognized national or local guidelines or ‘Standard of Care’ protocols, can be assigned for ‘no discussion at the MDT’ (NHS England, 2019). Currently, there is no consensus or evidence base on what constitutes the most appropriate person or triage group. We compared the treatment plan and treatment type decisions of a consensus panel of two breast medical oncologists and two breast surgeons, which we defined as local best practice, with the decisions made by those same clinicians, when they formed two independent two-person oncologist/surgeon triage teams. We assessed consensus panel – triage team concordance both with and without clinical decision support from a CDSS. Both teams achieved very high concordance for treatment plan decisions (96% and 93%). Concordance for treatment type decisions was slightly lower than the treatment plan for team 1 (93%), but was markedly lower for team 2 (82%). This suggests that streamlining MDT recommendations for treatment type using two-person triage teams may introduce unacceptable variations in clinical decision making for those patients who are not discussed at a full MDT. Access to the CDSS changed the teams' decisions in only one case for team 1 and 3 cases for team 2, suggesting that the CDSS offered minimal clinical benefit as a support tool to the triage teams.

When the CDSS was assessed as a stand-alone triage tool for use with breast cancer patients meeting the eligibility criteria, it initially failed to achieve the levels of concordance with local best practices seen in the human triage teams. However after the tool was localized to take account of four key differences between MSKCC and local MDT clinical best practice, our post-localization CDSS analysis suggests that the CDSS approached the performance levels seen in our two human triage teams for treatment plan decisions and may have actually performed better than one of the teams for treatment type decisions. However, our FFT analysis provides the most promising evidence for the potential value of an AI CDSS within a cancer pathway. Our findings suggest that with a simple triage decision tree that could be administered by a trained non-clinical MDT coordinator/supervising clinician, or automated within the electronic patient record that uses clinical stage, patient age, HER2, ER and PR status, and with localization of a CDSS to account for local best practice, it might be possible to streamline a breast cancer MDT pathway such that 40% of eligible patients could be referred to a CDSS for treatment plan therepeutic options and assigned for ‘no discussion at the MDT’. This would require the appropriate clinical information including staging to be available at the time of triage. Our FFT analysis predicts that this CDSS would be 100% concordant with local MDT best practice for this sub-group of cases, and would therefore require minimal clinical supervision. If treatment type therapeutic options are required, 34% of eligible patients could be safely referred to the CDSS, again with 100% concordance with local MDT best practice. We estimate that if the triage decision tree, combined with the eligibility criteria for the version of the CDSS used in this study were applied to the current Guy's Cancer Centre breast cancer pathway, the clinical burden on the MDTs would be reduced by approximately 30%. This could be translated into shorter MDTs and more time for clinical work, or longer discussions on more complex cases. This could be investigated in a prospective study implementing AI-CDSS triage, by the MDT meeting being kept the same length initially and the clinical team evaluating if there is more value to patient care by the meetings being shorter, or the meetings being the same length but having more time to discuss fewer cases.

There were several methodological limitations in our study design. We made some assumptions when considering the potential value of a CDSS for routine UK breast cancer care, which may not hold true. Concerning the study design, we used the consensus decisions/a ‘consensus panel’ of our two participating medical oncologists and two surgeons as a proxy ‘ground truth’/gold standard for local best practice, and these same clinicians then formed the two triage teams. The two oncologists and one of the surgeons are amongst the most senior members of the local breast MDTs. All four clinicians are core MDT members; therefore, it is not unreasonable to use their decisions as a proxy for local best practices. A more rigorous design could have included other clinical members of a full MDT in the consensus panel. Still, we have no reason to believe their participation would materially alter the consensus decisions. We could also have identified triage teams comprising clinicians not part of the consensus panel, and from other UK centers to establish the ‘ground truth’/gold standard. However of note, as standards of care to streamline MDTs are introduced in the UK, the triage groups likely asked to review patients will similarly comprise clinicians drawn from the full MDTs.

Our post-localization CDSS analysis assumes that the machine learning algorithms used to generate CDSS therapeutic options can be adapted to consider local clinical practice differences from MSKCC practice, such as those identified in our study. While this is technically quite feasible, the resource implications in re-training a CDSS for adaptation to every national or regional setting in which such a tool is implemented may make localization commercially non-viable unless cheaper computational methods other than machine learning (such as expert systems) can be applied.

Cancer stage is a vital patient characteristic driving the triage decision tree that resulted in a sizeable sub-group of patients being sent to the CDSS (avoiding the MDT) such that 100% concordance with local MDT best practice was achieved. This presupposes that accurate clinical staging is possible early in the clinical pathway prior to the MDT discussion. This may not be the case for all local breast cancer pathways. It may require that the pathway be re-designed, for example, making imaging or histology results available at an earlier point in the pathway.

The impact we suggest a CDSS could make on the clinical MDT burden is based on the mix of breast cancer patients seen at the Guy's Cancer Centre. The Centre serves as both a secondary referral center for the local population of South London, but also as a tertiary referral center with a large early phase clinical trials programme. However, since patients with metastatic disease were excluded from this study, we believe our findings are generalizable to other secondary care cancer centers. It may also be the case that compliance with nationally agreed standards of care is more likely to increase in such settings, where there may be less access to specialized medical oncology expertise. Our study was not designed to test this hypothesis. Still, our finding that the post-localization CDSS treatment type therapeutic options had a higher concordance with our consensus panel decisions than one of the two-person triage teams provides some indirect support.

Conclusion

Before AI-enabled, computerized CDSSs can be recommended as effective guideline implementation interventions to promote evidence-based medicine, they must themselves be shown to be evidence-based. Previous prospective concordance studies for the CDSS we used in this study provide some preliminary evidence for its clinical effectiveness in specific settings, and this retrospective study builds on that by demonstrating how, when used as a triage tool and adapted to local best practice guidelines, AI-enabled CDSSs can potentially reduce the clinical workload for a breast cancer MDT by up to 40%. Before such tools can be deployed routinely within a UK cancer pathway, and similarly for deployment globally, they would need to be appropriately localized and clinically validated in prospective studies designed to evaluate both clinical effectiveness and economic impact. Localization and tumor group specific prospective evaluation would be required to identify subgroups with the greatest concordance. These tools are meant to augment decisions of the MDT and unlikely to replace the MDT. However, it can help reduce the cognitive burden and decompress decision processes. For rapid NHS adoption a coordinated multicenter evaluation approach, for example through the NHS Transformation Directorate, would seem appropriate.