1. Introduction

Palliative care specialist consultation is one of the few evidence-based interventions that improves quality-of-life (QOL) in cancer patients. In 2010, Temel et al. [1] (2010) demonstrated enhanced QOL and mood in metastatic non-small cell lung cancer patients with early integrated outpatient palliative care. Subsequently, the ENABLE III trial showed that early palliative care integration was associated with increased one-year survival rates among advanced cancer patients [2]. A third large, randomized trial including patients with gynecological cancers demonstrated that early outpatient palliative care was associated with improved QOL, satisfaction with care, and symptom control [3]. Inpatient and outpatient palliative care specialist consultations for patients with gynecological cancers also improve symptom control across a wide range of domains: pain, anxiety and depression, appetite and nausea and fatigue [4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. Based on these and additional studies, the American Society of Clinical Oncology (ASCO), National Comprehensive Cancer Network (NCCN) and Society for Gynecologic Oncology (SGO) have recommended systematic early integration of palliative care in evidence-based oncology care [14, 15, 16, 17, 18, 19].

Despite the growing body of evidence confirming the value of palliative medicine consultation and society recommendations encouraging consultation, outpatient palliative care specialists remain underutilized among patients with gynecological cancers, even in high-resource settings (Problem Statement) [20, 21]. In prior studies, 30–50% of eligible patients with gynecological cancers had a palliative care consultation as part of their oncologic care [22, 23, 24, 25]. Additionally, fewer than one-fourth of patients with gynecological cancers have a “timely” palliative care consultation, defined by ASCO as consultation within eight weeks of advanced stage and/or recurrent cancer diagnosis [23, 26, 27, 28, 29, 30]. ASCO reaffirmed these guidelines in 2024 based on an updated systematic literature review [31]. In the revised guidelines, ASCO stressed the central role of outpatient palliative care specialists as a key indicator of early palliative care engagement, emphasizing the need to evaluate adherence to ASCO guidelines in the outpatient setting [31].

More recently, researchers have leveraged implementation science and quality improvement (QI) methods to assess factors contributing to breakdowns in systematic early and timely referrals to outpatient palliative care specialists. Previously, our group defined a process map and sequential flow diagram required for the successful uptake of timely, early integrated outpatient palliative care specialist consultation [32]. This included five key components: (1) the presence of a need for palliative care specialist such as physical and/or psychological symptom management, serious illness communication, and/or coping skills/family management, (2) recognition of the need for prompt referral to palliative care specialists (often by comparison against standardized referral criteria), (3) patient with willingness and ability to attend outpatient palliative care specialist appointment, (4) adequate available outpatient palliative care specialist resources to see patients, and (5) standardized flags to prompt reassessment of patients not identified to have palliative care specialist needs on initial assessment [32]. Several studies have evaluated screening tools to prompt outpatient palliative care specialist consultations for patients with gynecological cancers [33, 34, 35, 36, 37, 38, 39, 40, 41, 42]. However, there is limited data on implementation and QI outcomes to guide the operationalization of associated workflows.

The objective of this study was to perform a self-assessment of outpatient palliative care specialist referrals for patients with gynecological cancers to assess (1) the ability to identify patients meeting standardized criteria for outpatient palliative care specialist referral using an automated electronic medical record (EMR) report (2) recognition of the need for palliative care specialist referral in the course of usual clinical care relative to standardized criteria (3) rates of referral completion, and (4) timeliness of referral completion. Study objectives were undertaken with the secondary goal of identifying tools and strategies for QI intervention evaluation that are broadly applicable and can be implemented at other institutions to increase the reach, timeliness, and appropriateness of the deployment of outpatient palliative care specialist resources.

2. Materials and methods

We conducted a single institution retrospective cohort study to assess our ability to identify patients with gynecological cancers with palliative care specialist needs and evaluate referral patterns for outpatient specialist palliative consultations. The period evaluated spanned from January 2019 to December 2022. We chose this timespan to (1) follow the introduction of a consistent outpatient palliative care specialist referral workflow process at our institution, (2) coincide with a period of stability in terms of gynecologic oncology and palliative care workforce and availability, and (3) ensure that internal data validation had been completed before data analysis. The institutional review board at the University of Kansas Medical Center reviewed the study protocol and waived the requirement for informed consent due to the study’s retrospective nature and primary purpose of quality improvement.

We used an automated approach to identify eligible patients, i.e., patients with gynecological cancers which met the standardized criteria for outpatient palliative care specialist referral. The patient population (study denominator) was ascertained through the use of an automated EMR report in Epic Systems software (hereafter referred to as “EMR report”). Patients with a gynecologic cancer diagnosis evaluated in an outpatient gynecologic oncology clinic between January 2019 and December 2022 were identified using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis codes for the following cancer sites: ovarian, fallopian tube, primary peritoneal/mullerian origin, uterine, endometrial, cervical, vulvar and vaginal. Patients were included if advanced stage (stage III or IV by the American Joint Committee on Cancer staging) at the time of diagnosis or if diagnosed with disease recurrence during the study period. These standardized criteria were selected to correspond to ASCO’s definition of advanced cancer in the clinical oncology practice guideline update recommending early integrated palliative care [43]. We assessed the true positive rate of the EMR report by performing a secondary chart review of patients identified in the EMR report to verify that the stage, recurrent disease status and categorization of meeting ASCO criteria in the automatically generated EMR report was valid based on clinical data as reported in progress notes, cancer diagnosis problem list, and cancer staging entered by hand in the EMR in usual clinical care.

We then reviewed an internal outpatient palliative care specialist referral tracking database (PC database) created and maintained by our institution’s Division of Palliative Care in the Research Electronic Data CAPture (REDCap) electronic data capture system [44, 45] hosted at The University of Kansas Medical Center. The REDCap system is a secure and Health Insurance Portability and Accountability Act (HIPAA)-compliant web-based software platform designed to support validated data capture, auditing trials to track data manipulation and export, automated export procedures, and procedures for data integration and interoperability with external sources [44, 45]. Data collected within the database from retrospective chart review by the database manager included cancer site, source of referral, date of outpatient palliative care specialist referral, date of initial outpatient palliative care visit, number of outpatient palliative care visits, date of hospice enrollment, date of death, and patient status (active, discharged, referred and not seen, or deceased). We reviewed the charts of patients with gynecological cancers identified in the PC database as having an outpatient palliative care specialist referral. Our goal was to assess their cancer site, cancer stage, and whether the disease was primary or recurrent to determine if they met ASCO referral criteria. All unique patients meeting ASCO criteria identified in the EMR report and the PC database were summed to determine the denominator for this study, i.e., the population of patients with gynecological cancers eligible for outpatient palliative care specialist referral. This permitted calculation of the EMR report’s sensitivity in identifying patients meeting ASCO criteria for outpatient palliative care referral having now ascertained the number of true positive patients in the EMR report and the number of false negatives in the EMR report, i.e., the number of patients meeting ASCO criteria in the PC database that were not in the EMR report. We also determined the proportion of patients with gynecological cancers referred to outpatient palliative care specialists during the study period who did not meet ASCO criteria for referral. For patients meeting ASCO criteria for a referral who were referred to outpatient palliative care specialists but did not complete an outpatient palliative care specialist visit, the reason they were not seen was determined.

Primary outcomes included quantification of the gynecologic cancer patient population eligible for outpatient palliative care specialist referral, rate of outpatient palliative care specialist referral placement for eligible patients, and rates of referral completion among referred patients. Secondary outcomes related to the timing of outpatient palliative care specialist referral as assessed by the times from referral placement to first outpatient palliative care visit, hospice enrollment and/or death. Descriptive variables were reported as absolute numbers with percentages or as medians with interquartile ranges as appropriate [46]. We assessed differences in referral rates and completion by cancer type using Chi-Square tests with statistical significance set as p < 0.05. This manuscript was prepared in accordance with Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guidelines [47].

3. Results

The EMR report identified 324 patients with gynecological cancers meeting ASCO criteria for outpatient palliative care specialist referral from January 2019 to December 2022. There were 151 (46.5%) patients with ovarian, fallopian tube and/or primary peritoneal cancer, 107 (32.9%) with uterine cancer, 40 (12.3%) with cervical cancer, 15 (4.6%) with vaginal or vulvar cancer and 11 (3.7%) with unclassified primary gynecologic cancer site. Most patients in the EMR report had primary advanced disease; only 110 of 324 (34%) were classified as having recurrent disease. We identified an additional 110 patients meeting ASCO criteria for outpatient palliative care specialist referral from the PC database that were not in the EMR report, including 49 (44.5%) with ovarian, fallopian tube and/or primary peritoneal cancer, 35 (31.8%) with uterine cancer, 18 (16.4%) with cervical cancer, 8 (7.3%) with vaginal or vulvar cancer and 92 (83.6%) of patients with recurrent disease. The true positive rate of the EMR report was 100%, as all patients in the EMR report met the ASCO criteria for referral to outpatient palliative care specialists on secondary data review. The EMR report’s sensitivity for identifying patients meeting ASCO criteria was 74.7%, as the EMR report failed to include 110 of 434 patients with gynecological cancers meeting ASCO criteria for referral identified in the PC database.

Only 192 of 434 (44%) patients with gynecological cancers meeting ASCO criteria were referred to outpatient palliative care specialists during the study period. Of note, 29 referred patients (15%) were not included in the PC database at the time of data analysis due to the lag in manual data entry relative to receipt of patient referrals during the usual database maintenance by the palliative care department. Among referred patients, 147 of 192 (77%) patients ultimately completed an outpatient visit with a palliative care specialist. Fig. 1 presents a Sankey chart demonstrating the cumulative influence of various sources of palliative care referral attrition. The most frequently identified reasons that patients did not utilize outpatient palliative care specialist services after referral included hospice enrollment (44%), patient preference (41%) and unable to be contacted (9%) (Fig. 2). The other patients (less than 5% in each category) were not seen due to their needs being adequately met via telephone encounter with a palliative care nurse, an inability to reach the patient, lack of clinic availability or another unspecified reason.

Fig. 1.Outpatient palliative care specialist referral attrition. EMR: electronic medical record; PC: palliative care; ASCO: American Society of Clinical Oncology.

Fig. 2.Reasons underlying incomplete outpatient palliative care specialist referrals.

There was no significant difference in the proportions of eligible patients with gynecological cancers referred to outpatient palliative care specialists by cancer site (p > 0.05), though patients with vulvovaginal cancers more commonly had a referral (Fig. 3). Most patients (66%) with recurrent disease had outpatient palliative care specialist referrals relative to the proportion (25%) of patients with advanced primary disease. Most patients with gynecological cancers (78%) were referred to outpatient palliative care specialists by their gynecologic oncologist. The remainder were referred by the inpatient palliative care team at hospital discharge (12%), medical oncology (5.1%) or radiation oncology (2.8%) (Fig. 4).

Fig. 3.Eligible patient population versus population referred to outpatient palliative care specialist by cancer type.

Fig. 4.Sources of referrals for patients with gynecological cancers to outpatient palliative care specialist.

The timeliness of referrals was also assessed (Table 1). The median time from referral to appointment was 20 days (interquartile range (IQR) 12–29 days). The median time from referral to hospice enrollment was 112 days (IQR 30–329 days) and the median time from hospice enrollment to death was 18 days (IQR 9–38 days).

4. Discussion

In a single institution retrospective cohort study, under half of advanced or recurrent patients with gynecological cancers meeting ASCO criteria were referred to outpatient palliative care specialists as part of routine clinical care. When eligible patients were referred, over 70% completed a timely outpatient palliative care specialist consultation. An automated EMR report and a manually maintained PC database tracking referrals provided valuable data for evaluating outpatient palliative care specialist referral workflows within our institution’s Division of Gynecologic Oncology. Both data sources were inadequate when used independently without cross-referencing and verifying merged data, limiting their utility for systematic use to chart improvement against institutional quality improvement goals in their current form. However, this systematic data profiling and evaluation of process measures relative to an established process flow diagram [32] and established QI metrics readily demonstrated persistent gaps and areas for targeted improvement within ongoing QI workflows. Such deficiencies can be addressed in future Plan-Do-Study-Act (PDSA) cycles [48] at our institution and can be leveraged to bolster data collection and analysis in other institutions.

Our study is unique in its concurrent assessment of quality metrics across all steps required for successful integration of timely outpatient palliative care specialist referral, including identification of palliative care specialist needs, recognition of need by a treating physician, prompt for referral and referral placement, patients accepting and able to present to outpatient palliative care specialist referral, adequate palliative care specialist resources, and regular reassessment of palliative care specialist needs. A process map to guide quality improvement work is crucial when considering outpatient palliative care specialist referrals since the process is multifactorial, including interdisciplinary teams across multiple specialties. Process maps can also be tailored to reflect variations in a process in the local contexts of specific healthcare systems. Facilitators and barriers to uptake and completion of outpatient palliative care specialist can vary significantly across different care settings. For example, multiple studies have identified barriers to outpatient palliative care specialist referral uptake among patients with gynecological cancers, including late or lukewarm collaboration from gynecologic oncologists who are wary about the possibility of conflicting messages or patients feeling abandoned, as well as perceived patient stigma, hopelessness, or unrealistic expectations [49, 50, 51, 52, 53, 54, 55, 56, 57]. In our study, over 70% of patients with gynecological cancers referred to outpatient palliative care specialists completed an outpatient palliative care consultation, suggesting that barriers at the level of gynecologic oncologists and patients are not the predominant contributors at our institution. However, assessments at other institutions may yield differing results, suggesting the need for provider- and/or patient-level education and intervention. Similarly, in our institution, palliative care specialist needs for patients with gynecological cancers have yet to exceed the capacity of our robust Division of Palliative Medicine. This aligns with the 95% of National Cancer Institute (NCI)-designated cancer centers with outpatient palliative care specialist clinics available; in contrast, only 40% of non-NCI-designated centers have similar resources [58].

To date, multiple prior studies have examined standardized criteria to prompt referral to outpatient palliative care specialists for patients with gynecological cancers with mixed results. Multiple studies included patients with gynecological cancers as a small subpopulation within a larger advanced solid tumor study cohort [36, 37, 59]. In contrast, only a few smaller pilot studies evaluated the use of automatic palliative care specialist referral prompts specifically for patients with gynecological cancers in the outpatient setting [34, 35, 38]. The type and complexity of referral criteria employed vary widely across studies. Hui et al. [60, 61] (2016) published an initial extensive systematic review that identified six themes amenable to defining twenty referral criteria, including needs-based categories such as physical symptoms, psychological distress, performance status, end-of-life care planning, and two time-based categories defined as cancer trajectory and prognosis. These criteria were adopted in multiple systems, including an International Delphi consensus panel pilot tested in outpatient settings [62] as well as modified NCCN Referral Criteria piloted in inpatient and outpatient settings [63] and the Palliative Care Referral Protocol (PCRP) used in breast and patients with gynecological cancers in Brazil [36]. Additional questionnaires that have been used as outpatient palliative care specialist referral prompts include the Edmonton Symptom Assessment System-revised (ESAS-r) [37] and Patient-Reported Outcomes Measurement Information System (PROMIS®) instruments [38]. Shorter prompts have also been evaluated, including the “surprise question” (Would you be surprised if this patient died in the next 6-12 months?) [34, 35, 64, 65] as well as panels of fewer than five criteria including some elements of ASCO criteria with elements of healthcare utilization (hospitalization within thirty days or lasting longer than seven days), poor symptom control or complex clinical situations (malignant bowel obstruction, pelvic exenteration) [26, 66, 67]. However, shorter prompts have been evaluated more commonly in inpatient rather than outpatient settings. Together, studies demonstrate that using standardized criteria to prompt outpatient palliative care specialist referral is feasible, but data demonstrating their effectiveness and efficiency in implementation remain sparse [33].

It is difficult to evaluate the above-proposed outpatient palliative care specialist screening and referral criteria systems in terms of their ability to prospectively identify patients with palliative care specialist needs when their performance has only been evaluated in a retrospective and ad hoc fashion. Most commonly, studies have validated referral criteria by evaluating patients already enrolled in outpatient palliative care specialist care to determine how many referral criteria were present at or before the time of referral [36, 59, 65, 68, 69]. Alternatively, studies have noted that the use of standardized criteria would have identified palliative care specialist referral needs earlier in the disease course to minimize end-of-life interventions for gynecologic cancer patient decedents [24, 70]. Though these studies affirm the potentially clinically useful positive predictive value of proposed outpatient palliative care specialist screening and referral criteria, they provide limited information on the validity of chosen criteria to prospectively identify the relevant total denominator of eligible patients with gynecological cancers with specialist palliative care needs. We lack sufficient data to determine how accurately existing criteria identify patients who require specialized palliative care and avoid misidentifying those who have false negatives.

Though the EMR report was valid in accurately identifying patients with gynecological cancers meeting ASCO criteria for outpatient palliative care specialist referral, the report had suboptimal sensitivity and positive predictive value as calculated. Only 44% of patients identified as meeting ASCO criteria were referred to outpatient palliative care specialist. A subsequent chart review is planned to understand whether patients meeting ASCO criteria who were not referred lacked outpatient palliative care specialist needs or to understand why needs were unrecognized or did not prompt referral to palliative care specialists. The population meeting ASCO criteria at our institution likely includes a subset of patients who only completed surgery at our institution and received chemotherapy locally and thus may not have found it feasible to follow with palliative care at an institution far from home. There were an additional 12 patients referred to outpatient palliative care specialists during the study period who did not meet ASCO criteria. Similarly, future retrospective chart review will determine if the system failed to recognize that these additional 12 patients also met ASCO criteria for referral or if their outpatient palliative care specialist needs were not captured by using ASCO criteria alone. Recurrent disease status can be inconsistently documented in the EMR, which may account for some of the discrepancies between data sources; the EMR report was comprised primarily of advanced primary disease patients whereas the majority of patients unique to the PC database that were not captured in the EMR report had recurrent disease. The study cohort also had more primary than recurrent disease patients, suggesting that both sources may not capture all recurrent disease patients treated at our institution. If a large subset of the 56% of patients meeting ASCO criteria ultimately did not have outpatient specialist care needs, that would indicate that ASCO criteria holds low positive predictive value in determining the need for outpatient palliative care specialist referral at our institution. This suggests that ASCO criteria may need to be paired with additional criteria to optimize the sensitivity of systematic screening for outpatient palliative care specialist needs. The example of advanced endometrial cancer patients supports this hypothesis. Though patients with FIGO stage III endometrial cancer meet ASCO criteria, many are cured with surgery and adjuvant chemotherapy/immunotherapy with minimal long-term symptom burden.

Several previous studies have demonstrated limitations in the practical utility of ASCO criteria as a standalone tool to support timely integrated palliative care specialist consultation. Lefkowits et al. [71] (2014) piloted the use of ASCO criteria to assess the need for palliative care specialist referral for gynecologic cancer inpatients. Even in an institution with an overall high palliative care consultation rate and strong collaborative relationships between the palliative care and gynecologic oncology departments, the overall referral rate for patients meeting two or more ASCO criteria was 53% with referral rates ranging from 59–79% for patients with one criterion present [71]. This suggests that ASCO criteria may be an insufficient proxy for near-term palliative care specialist needs and/or that decision support tools may be needed to aid recognition of ASCO criteria. The increasing complexity of usual clinical care may also cloud clinicians’ ability to recognize patients meeting ASCO criteria for outpatient palliative care specialist referral. Though the same group reported a high overall rate of palliative care specialist referral (70%) in a platinum-resistant ovarian cancer cohort that, by definition, meets ASCO criteria, approximately half of consultations occurred within three months of death, only 16% occurred within eight weeks of diagnosis, and the rate of outpatient versus inpatient consultations was not reported [27]. Similarly, in a study of the implementation of Delphi consensus criteria at a tertiary cancer center palliative care clinic, the only criteria associated with a prolonged median time to referral (greater than one month) were those based on “objective” measures of disease stage, e.g., “within three months of diagnosis of advanced cancer” and “progressive disease despite second line systemic therapy [59]”.

Clinicians’ frequent oversight of patients meeting ASCO criteria in routine care emphasizes the high probability of inconsistent access to palliative care specialists when referrals are solely clinician dependent. An international Delphi survey of palliative care experts sought feedback on previously validated need-based and time-based criteria for outpatient palliative care specialist referral [72]. Although a consensus was reached that automatic referral criteria may increase the number of referrals, facilitate earlier referrals, and help set benchmarks for quality improvement, over half of the respondents expressed concern that automatic referrals would decrease clinician autonomy and create conflict between oncologists and palliative care specialists [72]. Ultimately, the panel recommended parallel or augmented referral models combining clinician-based and automatic referrals with institution-specific referral criteria mutually agreed upon by oncology and palliative care departments [72]. In this setting, regular auditing to ensure referral thresholds are consistent and well-matched to the availability of local palliative care specialist resources is critical. Several groups have also investigated the use of ESAS-r [37] and PROMIS® [38] instruments to leverage patient-reported data to overcome inherent bias and time/resource gating from physician-dependent assessment of outpatient palliative care specialist needs. Though results have been mixed as to whether the use of patient-reported outcomes significantly increases the rate of outpatient palliative care specialist referrals [37, 68, 73, 74], data suggest they may decrease the number of identified patients who ultimately decline outpatient palliative care specialist referrals due to inadequate need for services [37].

Despite a supportive culture including active integration of palliative care training in fellowship education [75, 76, 77, 78] and institutional buy-in for early integrated palliative care specialist involvement in oncologic care, we still identified opportunities to improve performance in our institution. Of 192 referrals, 26 (13.5%) patients were not seen due to immediate hospice transition or death, suggesting that a subset of patients identified with palliative care specialist needs still did not receive timely referrals. However, the median time from outpatient palliative care specialist referral to hospice enrollment of 112 days suggests that the majority of patients with gynecological cancers at our institution do receive timely palliative care referrals. Timely referral to palliative care specialists is crucial due to its improved efficacy in symptom management, psychological care, patient education, and serious illness conversations with longitudinal relationship building [51, 58, 59, 64, 65, 66, 67, 68, 69, 70, 79]. In contrast, most patients enrolled in hospice services less than one month prior to death, which presents a potential opportunity to improve the timeliness of hospice referral for hospice-eligible patients. However, there also may have been a lesser need for hospice services, given improved physical and psychological symptom control due to the involvement of outpatient palliative care specialists.

Once an initial quality improvement study is completed, it is helpful to map its key findings (i.e., baseline benchmarks relative to defined QI metrics, areas of missing or inadequate data, and facilitators and barriers to meeting care quality goals) to a concrete framework for implementation. Our group has previously defined a process map with a sequential flow diagram outlining the minimum required steps for identification, referral, and uptake of timely integrated outpatient palliative care specialist consultation [52, 58]. We demonstrate the use of our study findings for root cause analysis in a Fishbone quality improvement diagram (Fig. 5). This analysis can facilitate idea generation for actionable solutions to improve outpatient palliative care specialist referrals tailored to the biggest deficits and/or barriers in the context of a specific institution or health system.

Fig. 5.Process map with sequential flow diagram of required steps for outpatient palliative care specialist referral (inset) with Fishbone quality improvement diagram of root cause analysis for our local institution.

The primary barrier at our institution was quickly identifying patients who met the ASCO criteria for outpatient palliative care specialist consultation. The EMR report could not automatically identify all patients meeting ASCO criteria for outpatient palliative care specialist consultation. Thus, we were required to perform supplemental chart review to assess discrepancies between the EMR report and the PC database and identify factors that could be associated with failure to identify patients with gynecologic cancers meeting ASCO criteria for referral. The key driver of discrepancies between the EMR report and the PC database was disease recurrence. Notably, 83.6% of discrepant cases (patients meeting ASCO criteria identified in the PC database but not in the EMR report) had recurrent disease. This suggests that our automated EMR report (and potentially those at other institutions) is limited in effectively capturing patients with recurrent disease, which represents a large sub-population of patients meeting ASCO criteria. The remaining discrepant cases were due to delayed entry secondary to the administrative burden for the database coordinator, which was discovered on root cause analysis following data analysis. Multiple prior studies have demonstrated limited explicit recording of cancer recurrence in the electronic medical record and cancer registries [80, 81]. Lee et al. [80] (2024) recently developed an automatic rule-based detection algorithm to analyze automatically extracted raw electronic medical record data for the presence of imaging or biomarker abnormalities or multiple chemotherapy lines to identify patients with ovarian cancer recurrences. The algorithm achieved over 90% sensitivity, specificity and accuracy and saved 32 hours per 100 patients in chart review. Other groups have considered text mining algorithms for words related to disease progression [82] or combinations of diagnosis codes for secondary neoplasm after index surgery and/or procedure or treatment codes for surgery, chemotherapy, and/or radiation and/or treatment dates [39, 83, 84, 85]. Further research is needed urgently needed to determine how to leverage an optimal algorithm to identify all patients with recurrent or progressive disease via an efficient and reproducible strategy. Without this capability, we will be effectively unable to evaluate the false negative rate of ASCO criteria as a clinical prompt for outpatient palliative care specialist referral in real-time to further quality improvement.

Similar claims-based methods have been used to assess severe adverse effects of treatment or other significant symptom burdens [86]. However, this needs-based approach depends on accurate International Classification of Diseases and Hierarchical Condition Category coding, which depends on significant effort from physicians and billing/coding departments. An alternative approach may be to incorporate documentation of validated patient-reported outcomes scores [87, 88, 89] (e.g., Edmonton Symptom Assessment Scale [90, 91]) or performance status (e.g., Palliative Performance Scale [92]) in refreshable flowsheets that can easily be queried and tracked. This has been implemented in all outpatient cancer centers in Ontario, Canada, since 2008 [93]. However, significant institutional buy-in, multidisciplinary stakeholder support, and technological and data management infrastructure are needed to implement such complex quality improvement initiatives [94]. An alternative technological approach could be an automated reminder pop-up in the EMR ordering system to consider ordering an outpatient palliative care specialist referral when a gynecologic oncologist creates a second-line or later chemotherapy or supportive care plan. This is an appealing option at our institution, where there is high buy-in for the role of palliative care specialist involvement among the gynecologic oncology department, but it may be a less effective solution at other institutions where there is no established collaborative relationship between departments.

The strength of this study is its use of rigorous quality improvement methodology to define structure, process and outcome measures to evaluate the fidelity of timely outpatient palliative care specialist integration into gynecologic oncologic care. Quality improvement projects, including those involving multidisciplinary collaboration between palliative care and oncology departments with the engagement of trainees, have demonstrated efficacy in increasing palliative care use and augmenting institutional and stakeholder support for key workflows [95, 96, 97, 98]. Few studies define quality metrics per Centers for Medicare and Medicaid standards [99], including greater specificity than high-level society guidelines to define measurement settings, numerators assessing adherence to the quality measure, denominators quantifying the eligible patient population and reporting requirements with triangulation [100]. This permitted mapping to an established process flow diagram, which can help support integrating quality improvement and implementation science data to help guide the adaptation of study findings to unique institutional models of palliative care delivery [75]. Such QI and implementation data are essential in the ongoing effort to make the business case for palliative care to secure institutional buy-in from clinicians and leadership, program administrative support and other resources to ensure long-term department sustainability [61, 101].

A key limitation of this study is the two independent data sources to determine our study numerator and denominator. However, even an imperfect assessment of the denominator (i.e., the number of patients with gynecological cancers eligible for outpatient palliative care specialist referral) represents a crucial step forward. Most prior studies claiming to validate referral criteria to outpatient palliative care specialists only evaluated patients already enrolled in outpatient care with a palliative care specialist [36, 59, 65, 68, 69]. Such study designs provide no information on patients who were missed, i.e., patients meeting the criteria for outpatient palliative care specialist consultation who were not referred. We transparently documented the flow of patients from both sources using a novel Sankey diagram. Multiple study members confirmed the data accuracy with an independent chart review. Though discrepancies due to recurrent disease status persisted, this itself is a finding of clinical significance. Additionally, the twenty-nine patients with gynecologic cancer in the EMR report were referred to outpatient palliative care specialists that were not in the PC database and had been referred within thirty days of the data pull for this QI project. The data manager updated this for the PC database. We included this resolved discrepancy to use real-world data to demonstrate the quality improvement process. Similarly, patients in the PC database who did not meet ASCO criteria for referral but were also referred to outpatient palliative care specialists had appropriate indications for referral (i.e., poorly controlled symptoms and/or serious illness communication needs). We plan to explore why many gynecologic cancer patients meeting ASCO criteria were not referred in a subsequent study. We hypothesize that many advanced-stage gynecologic cancer patients who achieve remission with upfront surgery with or without adjuvant chemotherapy and/or radiation have minimal persistent symptom burden and low clinical need.

A retrospective cohort study is inherently limited due to the nature of data collection after the outcome of interest has occurred. This is associated with potential selection bias and misclassification bias. We attempted to mitigate this by having two study team members independently review charts to verify and cross-reference data accuracy and classification. This also limits the generalizability of study results. Our approach to data collection was utilitarian, using the minimum necessary data elements to answer the key study objectives. As a result, we lack information regarding the current or past treatment status or performance status of evaluated patients with gynecological cancers. Reasons for referral to outpatient palliative care specialists were also inconsistently tracked in our institutional database. This limits our ability to assess and address potential confounding demographic and clinical variables. Finally, we conducted our study at a single institution, which limits the generalizability of specific research findings to other clinical practices. However, the quality improvement process for data collection and analysis could be adapted and applied across various institutions. Future directions include identifying characteristics of patients meeting ASCO criteria for outpatient palliative care specialist referral who did not have palliative care specialist needs, defining additional EMR report variables to better contextualize and triage the patient population eligible for palliative care specialist consultation, and developing customizable data collection and analysis templates for use by other institutions.

5. Conclusions

Fewer than half of advanced or recurrent patients with gynecological cancers meeting ASCO criteria for outpatient palliative care specialist referral had a referral placed by their gynecologic oncologist. Yet, over three-fourths of patients with gynecological cancers who were referred completed an outpatient visit with a palliative care specialist. Outpatient palliative care specialist referrals were usually completed in a timely manner, with most patients scheduling a first appointment within thirty days of referral and well before hospice enrollment. A quality improvement project in collaboration between gynecologic oncology and palliative care to assess referral patterns can identify individual-, culture- and systems-level drivers and barriers to palliative care referral and measure performance against institutional care quality goals, including timeliness. Institutions can replicate this process to create actionable, context-sensitive referral solutions that are practical and designed to secure institutional buy-in.

We remain limited in our ability to identify patients meeting ASCO criteria for palliative care specialist referral from billing and electronic medical record data. This is particularly true for patients with recurrent gynecologic cancers. Leveraging data collected during usual clinical care to find and track patients most likely to benefit from palliative care can improve the delivery of comprehensive cancer care and improve patients with gynecological cancers’ overall QOL. This requires easily accessible and analyzable systematic data collection and recording around the patients’ symptom burden, performance status, recurrent disease status and markers of poor prognosis. Based on our study results, our institution plans to evaluate two potential changes in parallel. We are investigating supplementing and/or replacing ASCO criteria as the primary prompt for outpatient palliative care specialist referral and adding updated documentation of Eastern Cooperative Oncology Group performance status in a searchable flowsheet as a requirement to close outpatient gynecologic oncology visit charts. We will evaluate the impact of these changes in a subsequent study and PDSA cycle.

Abbreviations

PC, palliative care; ASCO, American Society of Clinical Oncology; CPT, Current Procedural Terminology; IQR, interquartile range; QOL, quality-of-life; NCCN, National Comprehensive Cancer Network; SGO, Society for Gynecologic Oncology; EMR, electronic medical record; QI, quality improvement; ESAS-r, Edmonton Symptom Assessment System-revised; ICD-10-CM, International Classification of Diseases, Tenth Revision, Clinical Modification; REDCap, Research Electronic Data CAPture; HIPAA, Health Insurance Portability and Accountability Act; PROMIS®, Patient-Reported Outcomes Measurement Information System; STROBE, Strengthening the Reporting of Observational studies in Epidemiology; PDSA, Plan-Do-Study-Act; NCI, National Cancer Institute; PCRP, Palliative Care Referral Protocol.

Availability of data and materials

A deidentified dataset from which information was collected may be shared by contacting the corresponding author.

Author contributions

RR and LS—designed the research study. EP and CS—performed data collection and cleaning. RR and EP—performed data analysis. RR—wrote the manuscript. IC, MJ and AJ—contributed to study conceptualization, writing, review and editing. All authors provided help and advice on study design and data reporting and reviewed and contributed to editorial changes in the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

The institutional review board (IRB) at the University of Kansas Medical Center reviewed the study protocol and waived requirement for informed consent based on determination that the study qualified for designation as quality improvement (IRB correspondence dated 03 January 2023). Please note that the University of Kansas Medical Center does not issue reference numbers for quality improvement studies per institutional protocol. However, the approval letter from the IRB was included with this manuscript submission.

Acknowledgment

The authors wish to acknowledge Susan Boring Van Unen for her ongoing maintenance of our institutional palliative care referral database and for her assistance in data entry, verification, and export to support data analysis for this project. We also acknowledge Gillian Luevano for her help in editing and reference management for this manuscript.

Funding

This research received no external funding.

Conflict of interest

RR and LS are members of the Society for Gynecologic Oncology Education Committee’s Palliative Care Subcommittee, which are unpaid volunteer positions.

The authors declare no conflict of interest. Lori Spoozak is serving as one of the Guest editors of this journal. We declare that Lori Spoozak had no involvement in the peer review of this article and has no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to EH.