Paper and electronic versions of HM-PRO, a novel patient-reported outcome measure for hematology: an equivalence study.

Aim: To determine measurement equivalence of paper and electronic application of the hematological malignancy-patient-reported outcome (HM-PRO), a specific measure for the evaluation of patient-reported outcomes in HMs. Patients & methods: Following International Society of Pharmacoeconomics and Outcomes Research ePRO Good Research Practice Task Force guidelines, a total of 193 adult patients with different HMs were recruited into a multicenter prospective study. The paper and the electronic version of the instrument were completed in the outpatient clinics in a randomized crossover design with a 30 min time interval to minimize the learning effect. Those who completed the paper version first, completed the electronic version after 30 min and vice versa. Instrument version and order effects were tested on total score of the two parts of the HM-PRO (Part A: quality of life and Part B: signs & symptoms) in a two-way ANOVA with patients as random effects. Intraclass correlation coefficients (95% CI) and Spearman's rank correlation coefficients were used to evaluate test-retest reliability and reproducibility. The effects of instrument version and order were tested on total score of the two parts of HM-PRO. Results: The questionnaire version and administration order effects were not significant at the 5% level. There were no interactions found between these two factors for HM-PRO (Part A [quality of life]; p = 0.95); and (part B [signs and symptoms]; p = 0.72]. Spearman's rank correlation coefficients were greater than 0.9, and intraclass correlation coefficients ranged from 0.94 to 0.98; furthermore, the scores were not statistically different between the two versions, showing acceptable reliability indexes. Noteworthy, the difference between the completion time for both paper (mean = 6:38 min) and electronic version (mean = 7:29 min) was not statistically significant (n = 100; p = 0.11). Patients did not report any difficulty in completing the electronic version during cognitive interviews and were able to understand and respond spontaneously. Conclusion: Measurement equivalence has been demonstrated for the paper and electronic application of the HM-PRO.

care, which includes understanding health outcomes, monitoring performance and informing clinical and policy decisions [1,2]. In addition, understanding signs and symptoms early in the course of the disease is important to identify disease state and severity in hematological malignancies (HMs). Furthermore, such symptoms can impact different aspects of a patient's life, such as physical, social, emotional and functional well-being, and often go undetected or are underestimated during the clinical consultation [3][4][5]. Although PROs in clinical practice can be very useful to overcome such issues, there are several barriers for their implementation, such as accessibility, userfriendliness, processing the collected information and finally the interpretation. Integration of an electronic version of PROs could offer a potential solution to the problem [6] by increasing accessibility to patients through mobile smart devices or a dedicated webpage. The information collected in such a manner would be easy to process and record, would provide immediate interpretation of the outcome to the clinicians and, in turn, back to the patients. In addition, the electronic versions also help in reducing the 'missing items' in the responses [7].
A systematic review conducted by Gwaltney et al. concludes that the electronic and the paper versions of PROs are equivalent and have the same measurement properties [8]. However, US FDA and other regulatory authorities require evidence to support such equivalence. Hence, adapting the paper version of the instrument to its electronic version requires demonstration of equivalent measurement properties so that both versions can be accepted and used interchangeably. The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) provides recommendations on evidence needed to prove the equivalence between electronic and paper version of PROs [9].
The newly developed HM-PRO is a specific instrument for patients with HM in routine clinical practice to evaluate benefits-risks of treatment, capture unmet patient's needs and facilitate physician-patient communication, which is currently undergoing an assessment to meet the minimum standards set out by the US FDA. The use of an electronic application of a PRO measure not only provides a quick and most reliable assessment of quality of life (QoL), but would also enhance the integrity and accuracy of data and is also incited by regulatory authorities. The aim of this study was to evaluate the psychometric properties of the electronic version of the HM-PRO, in particular, reliability and face validity, in order to demonstrate equivalence between paper and electronic version of the instrument.

Ethics
Multicenter ethics approval was obtained from the NRES South West Bristol, UK (ref 14/SW/0033) followed by individual R&D approvals from all the participating centers. Relevant research passport was obtained to carry out patient interviews at the hospital sites. Signed informed consent was obtained from all the study participants.

Recruitment
This mixed method study was conducted in seven secondary care hospitals in the UK, following the ISPOR ePRO Good Research Practice Task Force Guidelines on evidence needed to support measurement equivalence between electronic and paper-based applications of a PRO instrument. The inclusion criteria were ability to give informed written consent; diagnosed with HM as per the most recent WHO classification [10]; state of the disease, defined as stable, progressing or remission; any stage of the treatment (due to start the treatment, on treatment and finished treatment); and able to read and write in English. A purposive sampling methodology was adopted to recruit patients with different HMs, disease state and treatment, both in-and out-patients. A randomized crossover design between the paper and the electronic versions of the HM-PRO was adopted. Patients were approached by the member of the clinical practice staff or by a member of the research team. Those who agreed to participate gave written informed consent. Anonymity of each patients was observed by allocating them a unique identification number.

Instrument
The HM-PRO is a composite measure consisting of two scales: Part A (impact); and Part B (signs and symptoms). Part A measures the impact of the HM and its treatment on a patient's health-related QoL, and Part B captures the severity of different disease symptoms and treatment side effects. Based on factor analysis as well as RASCH modeling, Part A has a total of 24 items in four domains: physical behavior (7); social behavior (3); emotional behavior (11); and eating and drinking habits (3). Patients' responses are recorded on a three-point Likert scale (0 = not at all to two = a lot) and not applicable as a separate response option. Part B consists of 18 items in a single domain and the responses are captured on a three-point severity Likert scale (0 = not at all to two = severe) [11,12].
The third item of the 'eating and drinking habits' domain in Part A, in other words, 'my drinking habits have changed', and ninth item of Part B related to 'skin problems' are not included in the scoring system but collected for additional information. Although a misfit in the RASCH modeling, these two items were kept based on their high prevalence in the qualitative phase as reported important by the patients. The HM-PRO has shown good reliability with Cronbach's α and ICC greater than 0.8 for all four domains of Part A and for Part B. Using the anchor-based approach, individual score bands for Part A (0-7, 8-25, 26-41, 42-74, 75-100, representing 'no impact' to 'very large impact' on patients' life) and Part B (0-3, 4-16, 17-29, 30-65, 66-100, representing 'no effect' to 'very large effect' of signs and symptoms on patients' life) were established for translating the science of PROs in clinical practice. The bands are valid across gender and different age groups [13]. Furthermore, HM-PRO is capable of detecting small but clinically important change in patients' HRQoL over time [14]. The minimum clinically importance difference (MCID) of '6' was proposed based on standard error of the mean for both Part A and Part B of the HM-PRO [14].
The electronic version of HM-PRO was very similar to the paper version, except that each domain of Part A was displayed on a separate page and the whole of Part B on a single page. Patients are unable to skip any response and move to the next page. The general formatting, font type, font size, colour, instructions were similar in both the paper and the electronic version of the instrument ( Figure 1).

Data collection procedures & HM-PRO administration
All the patients received the study 'patient information sheet' and a flow chart of the study procedure prior to their participation. The study participants attending the outpatient clinics were also informed about the length of time they were required to stay in the clinic to complete the study. All patients gave written informed consent in person and completed the demographic information sheet. The paper and the electronic versions of the HM-PRO were then completed in a randomized crossover design with a 30 min time interval to minimize the learning effect. The random numbers were generated using RANDOM.ORG webpage before starting the patient recruitment in each center. Those who completed the paper version first, completed the electronic version after 30 min and vice versa. Ipads installed with the electronic application of the HM-PRO were provided to the patients. All the patients completed both the paper and electronic version of the HM-PRO on the same day.
In addition, a separate online survey was conducted in order to better understand the patients' general impression and preference following completion of the paper-based and electronic application of the HM-PRO. The responses to the survey were automatically recorded electronically. Moreover, ten patients were randomly selected for cognitive interviewing. The interviews were audio recorded and transcribed verbatim.

Statistical analysis Statistical tests
All statistical analyses of the data collected were performed using IBM SPSS statistics software, version 23. The descriptive statistical analysis was performed on the demographic information. The scores of the paper and the electronic versions of the HM-PRO were calculated for the scales and the individual domains. The scores between the two versions of the HM-PRO were compared using the nonparametric Wilcoxon test for paired samples. A two-way ANOVA with patients as a random effect was used to test the 'version effect' of the HM-PRO (paper vs electronic), and 'administration order effect' (i.e., paper first vs electronic first). Spearman's rank correlation coefficient (Spearman's rho) and ICC were used to assess reliability between the two versions of the HM-PRO. Descriptive statistics were performed for the survey responses and content analysis of the transcribed interviews was performed using NVivo 11, a qualitative analysis software. In addition, analysis of 'completion time' was carried out between the paper and the electronic versions of the HM-PRO.

Health-related quality of life scores
A total of 192 pairs of completed questionnaires (electronic and paper version) were available for analysis. The summary statistics of scores for the four domains and both scales (Part A: impact, and Part B: signs and symptoms) are presented in Table 2. The mean difference between the scores of both versions was lowest for social behavior (0.02) and eating and drinking habits (0.02), whereas the highest difference was observed for the overall score of Part A (0.28). However, the difference in the mean scores of all the domains and the two scales was not statistically significant according to Wilcoxon signed rank tests for paired samples, even without Bonferroni correction for multiple testing (Table 2). This suggests that the scores are comparable for both the electronic and the paper versions of the HM-PRO in the sample. The Spearman's rank correlation coefficients were greater than 0.8 for all future science group www.futuremedicine.com  the domains and the two scales, ranging from 0.87 (social behavior) to 0.96 (Part A: impact). The ICC was greater than 0.9 for all the four domains and the two scales, showing a strong reliability ( Table 2). Being a randomized crossover design, a total of 96 patients completed the electronic version first and the remaining 96 completed paper version first. Hence, to assess the effects of the type of the questionnaire (electronic or paper) and the order of administration, using a random-effect two-way ANOVA, a total of 96 combination of cases were available. There were no interactions between the type of questionnaire and the order of the administration for both Likewise, there was no effect of the type of questionnaire for Part A (F 1, 96 = 0.09, p = 0.76) and Part B (F 1,96 = 0.80, p = 0.78). Furthermore, no effect of administration order was found for Part A (F 1, 96 = 0.099; p = 0.752) and Part B (F 1,96 = 3.62; p = 0.058). This confirms that scores between the two versions of the HM-PRO were not statistically different, irrespective of their order of administration.
The distribution of scores for Part A and Part B obtained from both the paper and the electronic versions of the HM-PRO was analyzed using linear regression. The distribution of individual scores of Part A was: R 2 linear = 0.94 ( Figure 3); and R 2 linear = 0.92 for Part B (Figure 4), accounting for more than 90% of the variance. This means that the coefficient of determination for the relationship between the scores of HM-PRO for the electronic and the paper versions is above 0.9 and only less than 10% of the variance in the scores might be due to other factors such as type of the instrument (electronic or paper) or change in the environment while completing the HM-PRO.
The HM-PRO practicality & applicability A total of 165 patients responded to the applicability and practicality questions, and the results are shown in (Table 3). The majority of the patients were able to read the statements easily, able to respond spontaneously and found the display on the screen of the electronic version understandable and easy to follow. Most 160 (97%) patients found the font size appropriate. Overall 143 (86.7%) showed preference toward completing the electronic version of the HM-PRO compared with its paper version. Only 20 (12.1%) preferred the paper version. Furthermore, the cognitive debriefing interviews confirmed that the patients were able to understand the items as intended and found responding to them straightforward.  The HM-PRO completion time A total of 100 patients recorded completion time for the paper version of the instrument, whereas the completion time for the electronic version was recorded automatically for all 193 patients. Therefore, 100 patients whose completion time was available for both versions were included in the analysis. The median time to complete the paper version of the HM-PRO was 5 min (mean = 6.4 min, range = 2.0-25.0 min, interquartile range = 4.2 min).
For the electronic application of the HM-PRO, the median completion time was 6.5 min (mean = 7.3 min, range = 1.27-16.17 min, interquartile range = 3.6). The difference between the completion time for paper and electronic version of the HM-PRO was not statistically significant (p = 0.114). Although statistically insignificant, the completion time for the electronic version was slightly higher than the paper version because each domain is displayed on a separate page, which requires additional number of screen touch, and filling in the comment box electronically consumes more time than writing on the paper version.

Discussion
According to the ISPOR ePRO Good Research Practice Task Force report [9], adapted from Shields et al. [16], the rationale of a minor change is that 'the modification can be justified on the basis of logic and/or existing literature. No change in content or meaning'. The electronic version of the HM-PRO is no different with respect to its content (including the instruction for completion) or meaning to the paper-based version of the instrument. The only difference in the electronic version is that the items are presented by one domain per a page, which is under the definition of the 'minor changes' according to the ePRO task force report. The level of evidence recommended for such minor changes are 'cognitive debriefing and usability testing' [9]. This study goes beyond the standard recommendations and assesses the measurement equivalence by calculating the ICC, comparison of mean scores, Cronbach's α and test-retest reliability. The design of this study was chosen as randomized crossover design based on task force recommendations so to have greater statistical power even with a smaller sample. The demographics of the recruited patients showed a good mix of patients with different diagnoses and different disease states which confirms that the study sample is representative of the target population, in other words, patients with HMs; hence, the heterogeneity in the sample should be considered as a strong point. Furthermore, the age range of 18-89 years shows that the HM-PRO has been validated in a vast age distribution. The cognitive debriefing interviews are performed to assess that the items are perceived in the same way irrespective of the version of the instrument. The cognitive debriefing of ten patients confirmed that the items are understood and responded as intended. No patient reported any difference between the versions of the instrument.
The findings suggest that the electronic version of the HM-PRO fulfills the requirements of measurement equivalence with the paper version of the instrument. In fact, the intraclass correlation was greater than 0.9 and the correlation coefficients were greater than 0.8 for all domains of PART A and for both scales, Part A and Part B, confirming good test-retest reliability for all the domains and the scales. The difference in mean score between the paper and the electronic version was insignificant even before Bonferroni corrections for multiple testing, further confirming reliability. The practicality testing of the eHM-PRO confirmed that it is easy to follow, understand and respond spontaneously. The general outlook of the electronic version was also found to be userfriendly. Interestingly, 95% of the study participants preferred the electronic version of the HM-PRO compared with the paper version.
The usability testing of the electronic version of the HM-PRO confirmed that it is easy to follow, understand and respond spontaneously. The general outlook of the electronic version was also found to be userfriendly. Interestingly, 87% of the study participants preferred the electronic version of the HM-PRO compared with the paper version.
The electronic version of the HM-PRO can be of great significance not only in daily clinical practice but also in clinical research. Patients' change in scores can be incorporated in the healthcare records and monitored over time very easily. The electronic version allows the immediate calculation and interpretation of scores; hence, its use would be very useful at the time of consultation to facilitate the communication between the doctor and the patients. The individual signs and symptoms in Part B can be discussed based on the severity reported by the patients. Further, patients as well as clinicians could more readily use the electronic version of the HM-PRO to keep a record as part of the general electronic health record to monitor changes in their QoL over time in relation to the treatment they are receiving at any given time. The additional benefit of the electronic version of HM-PRO for all the stakeholders would be its straightforward incorporation into the patient's health records which could be accessed at the point of care during the consultation. The patients can complete it in the comfort of their home prior to attending the outpatient clinic. Such an approach would also facilitate the periodic review of such patients' records which is considered of paramount importance in regulating the treatment and other needs by the care team.
One bias of the study may be the difficulty to recruit patients who were not comfortable with using an electronic device, though the number of such patients was very small. Furthermore, in the paper version patients could respond to the items randomly compared with the electronic version where the responses were to be given in the presented order. However, random response in the paper version could not be detected and was not observed even while calculating the differences in the mean scores.

Conclusion
The electronic application of the HM-PRO is equivalent to the paper-based version for its measurement properties, with minor changes as per the ISPOR ePRO good research practice task force report. The electronic version has future science group www.futuremedicine.com good reliability, face validity and comparable scores to the paper version of the HM-PRO. The interest of patients in completing the electronic version of the instrument over the paper version is an interesting finding and would help the implementation of the electronic version of HM-PRO both in daily clinical practice and clinical research.

Summary points
• The electronic version of the hematological malignancy-patient-reported outcome (HM-PRO) is equivalent to the paper version for its measurement properties, with minor changes as per the The International Society of Pharmacoeconomics and Outcomes Research electronic PRO good research practice task force report.
• The electronic version has good reliability, face validity and has comparable reproducibility of scores.
• The Spearman's correlation coefficient was greater than 0.8 between the score of the two versions of the instrument suggesting that they are highly related.
• Administration order and type of questionnaire effects were insignificant.
• The cognitive debriefing of results confirmed that the items are understood and responded as intended.
• The patients' preference of the electronic version of HM-PRO over the paper version is an interesting finding and it is hoped to facilitate the use of eHM-PRO in daily clinical practice and clinical research.