Psychometric Properties of EQ-5D-3L Applied through Phone Follow-Ups: An Experience in PERSIAN Traffic Cohort

Introduction

Road Traffic Crashes (RTCs) annually leave 20 to 50 million people with non-fatal injuries, with many suffering a health problem, and injuries account for 41.2 million years of healthy life lost [1-3]. Moreover, over 2600 person years of life lost are attributed to road traffic injuries (RTIs) [4], which causes the second most deaths in Iran [5, 6]. The RTCs should be of a specific concern in Iran, where only in 2007, more than 27000 people lost their lives and approximately 277000 were injured [7]. To draw a surprising contrast, Nordic countries are universally in the front position on injury prevention, while Iran has globally been prominent for its top road traffic injury and death rates [8], and the first in the Eastern Mediterranean region [9]. Low and middle income countries (LMICs) are the place for 90% of increasing trauma-related deaths and disability-adjusted life years, which decrease national incomes [10]. Hospitalized patients due to RTCs often suffer severe injuries which may lead to functioning problems in the near future or in a long run [11]. In terms of their physical and mental health, the patients’ quality of life is required to be measured using an easy-to-understand and precise tool post injury. The most commonly-used tool to assess health related quality of life (HRQoL) is the EQ-5D in the world. It has presented good validity and responsiveness in a variety of populations [12-16]. Its original version has three response levels for its 5 dimensions, called EQ-5D-3L (European Quality of Life 5-Dimension 3-Level Version). Considering the heavy burden of RTIs in Iran, it is substantial to evaluate the psychometric properties of EQ-5D-3L for accurately assessing RTIs patients’ quality-adjusted life years (QALYs).

The way applied to collect data such as phone surveys comparing with in-person interviews may affect an instrument’s psychometric properties [11]. Therefore, it is essential to assess a tool’s psychometric properties for various data collections. As a major concern, very sparse research study were done through phone survey on health status of injured patients post-crash and post discharge. For such assessing, there is a rare opportunity to have in-person visits, in some settings particularly in low and middle income countries, and the resources do not support in-person interviews. However, we had a great opportunity of a national cohort study. This setting helps to apply the tool and assess its psychometric properties through national cohort platform of Post-crash PERSIAN (Prospective Epidemiological Research Study in IrAN) Traffic Safety and Health Cohort Study (PTC) [17]. Then, the key action was to select and apply reliable and valid tools in measure health-related quality of life after RTIs. In order to assess HRQoL, generic instruments include the Health Utility Index, the EQ-5D-3L, or the Short Form 6D, creating utilities that produce a quantity that combines both the length and quality of life [18]. The cohort study requires a complete tool, short enough to use for phone surveys, such as EQ-5D-3L with great sensitivity. As the brief form of this globally approved tool takes short time, and includes five different health dimensions, we psychometrically assessed it through phone interviews on three follow-up phases (one, six, and 12 months after injury). To our knowledge, the paucity of such all-inclusive tool on creating the health profile of people involved in RTCs is sensible reason to examine its internal consistency and stability reliability among study population through phone surveys at three different time periods post injury. Then, it would pave the study way of the health status in a cost-effective manner.

Materials and Methods

Assessment of Reliability

The psychometric properties of EQ-5D-3L was evaluated at three times after crash through phone interviews. This study was conducted in a clinical population. The source population were RTIs survivors enrolled from post-crash PTC [17]. All participants hospitalized in one of two referral trauma centres, were followed up through the PTC at one-, six-, and twelve- months follow-up post injury. All hospitalized patients have been recorded in the database of integrated road traffic injury registry system (IRTIRS) since 2019 [19]. The minimum number of sample required was 50 to detect the value of 0.4 for Intra-class correlation coefficient (ICC), with alpha and power defined at 0.05 and lower than 90%, respectively [20]. Patients were randomly recruited from those who must be called via phone to complete follow-ups forms. We included 50 patients per each follow-up. The subjects of each follow-up was different from others, then, we had 150 patients in total who participated in test and re-test phases between May-June 2020. In the participants’ selection process for each follow-up phase, we regarded both gender, four age groups (18-24; 25-44; 45-64; 65 and above), and educational levels (illiterate, school education, academic education) to be included in the sample. A number of ten persons rejected to participate (six patients) or died (one patient) or did not answer the call (three patients). They were substituted by the same sex, age group, and education level. The average time between two measurements was 12 days (10 to 14 days) [21]. They did not have any treatment or changes in their medication during that period. As inclusion criteria, participants were hospitalized due to RTIs for at least 24 hours, one or six or twelve months prior to their interview date dependant on their follow-up phase, registered, aged 18 or over, and informed consent was required. Specific exclusion criteria included having any disorders like psychiatric problems affect the validity of the responses or an event experienced during the test-retest interval significantly increases the level of disability (e.g. disease, car crash, fall etc.). The data collection was performed by a trained interviewer with broad experience (≥4years) [22]. The completeness of tools was mostly guaranteed since the interviewer filled them out. The selected participants were replaced by the same sex and age if they refused to participate.

Measures

The European Quality of Life 5-Dimension 3-Level Version

The EQ-5D-3L has been developed since 1980s by a network of global multidisciplinary researchers, “EuroQol group”. It is a broadly applied generic tool for measuring respondents’ diverse aspects of health outcome on the day of the survey [21] by five dimensions of Mobility, Self-care, Usual Activities, Pain/ Discomfort, and Anxiety/ Depression. Each dimension has three levels (no/some/extreme problems), characterised 1–3 [23]. This tool defines 243 (5³) unique health states. We calculated the health status scores for test and re-test data at total (n=150) and follow-up (n=50) levels.

Socio-demographic Data

Basic socio-demographic and crash-related information such as used/ counterpart vehicle, mechanism, and injured person’s role were collected.

Statistical Analysis

Initial analyses of the tool and baseline characteristics data were performed using descriptive statistics. Distribution of demographic variables were tested based on the value of skewness and kurtosis. Kruskal-Wallis test and Wilcoxon rank-sum (Mann-Whitney) test were applied to test the hypothesis that there are not statistically significant differences between groups of independent demographic variables including sex, age, education levels, and marital status, on the dependent variable of quality of life score on the basis of test phase data. To measure the psychometric properties of the tool, we assessed internal consistency reliability and stability reliability. We also evaluated the agreement of test and re-test scores using kappa agreement coefficient and Bland-Altman Plot. Data were analysed using STATA statistical package version 15 (StataCorp LLC, Texas) and a significance level of 0.05 was adopted.

Scale Score Reliability

Internal Consistency Reliability

Internal consistency indicates the extent to which the items in a subscale are correlated. The homogeneity was evaluated using Cronbach’s Alpha statistic [24]. A value between 0.5 and 0.75, 0.75 and 0.9, and greater than 0.90 specify moderate, good, and excellent reliability, respectively [25].

Test–retest Reliability

Another measurement used to directly assess the scale score reliability of the tool was repeatability assessment using the test-retest method. Comparing the results of both measurements examined the consistency over time. The test-retest correlation was measured through tau Kendal b and Spearman- Brown correlation coefficient. Spearman- Brown coefficient of greater than 0.7 is considered as acceptable [26]. A common metric, ICC derived from a two way mixed model with an absolute agreement definition was calculated between the two phases to quantify the test-retest reliability of disability measurements at three follow-ups and total level [27]. Scores of test-retest were analysed using the “single-rater type (k = 1), absolute-agreement, 2-way mixed-effects model to produce the intra-class correlation with 95% confidence intervals [28]. An ICC coefficient of >=0.75 was considered as evidence of measurement stability. ICC between 0.4 and 0.75 indicates fair to good reliability, and ICC <0.4 indicates poor reliability [29].

Agreement between Test and Retest Data

The Kappa coefficients and percentages of agreement were calculated to measure the agreement between test and re-test responses for EQ-5D-3L. To measure the amount of agreement on the ordinal variable between two phases, we applied the Kappa coefficients [30]. Kappa value above 0.81 specifies a nearly perfect agreement, 0.61-0.80 substantial agreement, 0.41–0.60 moderate agreement, 0.21–0.40 fair agreement, below 0.20 indicates a slight agreement [31]. Besides, regarding restrictions of ICC and correlation coefficient in assessing the agreement between the tool’s test and retest data [32], Bland-Altman method was applied to examine their agreement. The result is a scatter plot XY, in which the Y axis displays the difference between the two paired measurements (A-B) and the X axis indicates the average of these values ((A+B)/2) [33]. Thus, its vertical and horizontal axes indicate difference between and the average of test and retest scores, respectively. The limits of agreement were also estimated as mean difference ±1.96 standard deviations of the difference. A good agreement between would be represented by a near to zero mean difference and with a maximum of 5% or less points residing outside the limits of agreement [34].

Floor and Ceiling Effects

The floor and ceiling effects were derived from the percentage of respondents with the lowest and highest possible scale scores for individual items. A ceiling effect is considered by 15% or more participants with the highest score [34]. Ceiling and floor effects were considered moderate up to 25% and substantial when higher than 25% [35].

Results

Sample Characteristics

Our sample composed of 120 men participants (80%). The total mean age (SD) was 41 (14.7) years old. The majority of participants were adults between 25-64 years old (46%), or married (72%), or educated (86%), or employed (78%) (Table 1). At each follow-up phase, the most common crash mechanism was vehicle-vehicle collision (50%-56%), followed by Vehicle-pedestrian collision (20%-30%). The majority of subjects were motorcycle and car drivers (44%-52%) and Passenger/ pillion passenger (24%-32%).

As represented in Table 2, the whole patients’ quality of life showed that there are not statistically significant differences between groups of independent demographic variables including sex, age, education levels, and marital status, on the dependent variable of quality of life score on the basis of test phase data.

Scale Score Reliability

Floor and Ceiling Effects

The proportion of patients (N=150) with a full health state (11111) and worst health state (33333) were 2.7% and 8%, respectively. There was reported no floor and ceiling effects in each follow-up (floor and ceiling effects in F-up1: 0, 16; F-up2: 2, 4; F-up3: 6, 4, respectively).

Internal Consistency Reliability

Cronbach’s Alpha Coefficient

For the total test data (N=150), the overall Cronbach’s α coefficient resulted in 0.81 showing excellent internal consistency. The Cronbach’s α coefficients were estimated for each follow-up phase (N=50) as 0.62 at one month after crash, 0.69 at six months after crash, and 0.87 at 12 months after crash.

Stability Reliability

Testing the test-retest reliability, the Spearman-brown correlation coefficients were above 0.9 at one, six, and 12 months after crash (0.97, 0.98, and 0.97, respectively) as well as at total score level (0.98).

The reliability of the test-retest method for the EQ-5D-3L was confirmed using the ICC as described in the materials and methods. The ICC showed the absolute agreement between test and retest data at three post-crash follow-ups. For total data, it was excellent at both total and item levels. The coefficients suggested good to excellent reliability and stability (Figure 1). The minimum ICC at item levels was 0.85 for Anxiety.

Fig. 1. Test-retest reliability of the European Quality of Life 5-Dimension 3-Level Version (EQ-5D-3L) at three follow-up phases in road terrific injuries (RTI) patients.ICC: Intraclass Correlation Coefficient

Testing the correlation between total test and retest data (N=150), Kendall’s tau-b showed 0.92. At item level, it ranged from 0.81 for Anxiety to 0.97 for Mobility, with p>0.0001 (Table 3). The kappa agreement coefficients were estimated moderate to perfect ranged from 0.6 at F-up 3 to 0.8 at F-up 2, with p>0.0001. The percentage of agreement was high enough (Table 3) [36].

Bland-Altman Plot

To assess agreement between results of test and retest data, we used Bland-Altman plot. In this graph, the majority of the points with higher score fall within the range of ±2SD of the mean differences and represent high agreement between the tool’s test and retest scores. As the score increases (health status gets worse) more points gather around the mean difference (-0.073), with 95 % limits of agreement (-1.21, 1.06; SD=0.58). We have similar number of outlier differences distributed above the high limit and below the low limit (Figure 2).

Fig 2. Bland-Altman plot (agreement between test and retest scores). Y axis, difference between test and retest scores; X axis, average of test and retest scores

Discussion

This study assessed the psychometric properties of available Persian version of EQ-5D-3L through phone surveys at three various times after crash as a special feature of the paper. This questionnaire was translated in different languages even in Persian [15, 37] but neither specifically through phone survey nor in RTIs patients. Then, we assessed and revealed that Persian EQ-5D-3L is psychometrically reliable to the great extent in phone-based assessment. Adequate internal consistency discloses that the tool measures health status in five dimensions and delivers high reliable score at item level and total score in our representative sample. The instrument is robust to changes in the same subjects over time and less prone to measurement error as time passes by, as ICC showed excellent reliability and the Bland-Altman Plot showed high agreement. The stability reliability of the tool was approved in phone surveys at three different times after crash. This research showed better results than studies in different populations [15, 38] and in in-person interviews [31, 39]. Patients with acute conditions such as road traffic injuries could report different health status from those with chronic diseases. Moreover, time span between test and re-test and different sample size could affect the results.

To collect valid data and prevent from underestimating the burden of disease [40], it was generally preferred to apply not only reliable and valid, but also short and simple instruments in the setting of the national PTC along with other tests. Therefore, it was a proper choice to use EQ-5D-3L in the current study, to examine the patients’ quality of life, regarding its better reproducibility than its five-level version [38] and a good agreement between both versions [16]. The elderly, illiterate patients or those with lower education levels had more troubles in discriminating the five levels and remembering them with their corresponding number [41, 42]. As the majority of RTI victims have only school education or none, and 8% are the elderly [43], the 3-level version is preferable. Moreover, less time-consuming technique of data collection like phone interviews is critical in long-lasting projects, because frequent connections with patients are needed to complete defined tools. Phone surveys enable data collection without unnecessary travel of patients to the study site/ hospital or travel of interviewers to the place of patients’ residence, due to patients’ movement restrictions. Simultaneously, it prevents from exposure to pathogens in the Pandemic COVID-19. Likewise, in-person interview is not financially an easily-conducted method for every researcher/ participant. Additionally, phone interviews require interviewers with special communication skills to continue the communication. However, elderly patients and those with hearing problems may have difficulties with phone survey. Therefore, it is recommended to reassess the tool in this specific target population.

The study found no ceiling and floor effects at any follow-ups and better results compared with studies in China, Greece and Thailand [31, 34, 38]. Contrarily, a research in rural Chinese patients published a high-ceiling effect [16]. Different diseases in various severity could result in different ceiling effects. The inter-countries comparisons may be affected by cross-cultural response pattern, too.

The agreement represented that responses are identical. Satisfactory agreement was revealed between test and re-test results at all three times post injury similar to a study in Thai diabetes [38] in which only the retest data were collected via telephone. However, lower agreement was reported by a Korean study in breast cancer patients for self-administered EQ-5D-3L [44]. Quality of life measured using EQ-5D-3L would be affected by types of injuries.

This study examined the psychometric properties of EQ-5D-3L in patients at one, six, 12 months post injury via phone surveys. We realized that the tool is highly reliable and suitable to apply in phone interviews during follow-ups of injured patients above 18 years. Applying it in the traffic cohort study, we would decline the costs of research since its data could be collected remotely. Also, it would be applied by patients to self-rate their health status.

Regarding the limitations, because the pattern of answers may vary by disease features [38], the finding extrapolating to other groups of patients as well as under 18 RTI patients should be made with caution. It is also recommended to re-assess the tool in specific target population such as elderly people through phone survey due to their possible hearing problems.

Declaration

Ethics approval and consent to participate: The study respected ethical precepts in research and was performed consistent with the principles of the Declaration of Helsinki. It was evaluated and approved by the Ethics Committee of the Road Traffic Injury Research Center, Tabriz University of Medical Sciences (cohort ethics code: IR.TBZMED.REC.1398.543). An informed consent was required from all participants.

Consent for publication: All authors read and approved the final manuscript to be published, and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of interest: None declared.

Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Authors’ contributions: HSB conceived and designed the study. HSB and NS conducted the analyses, interpreted the results, and drafted the manuscript. MAJ, MF, SBH, AR, MG, and FP contributed in the critical drafting and revising the manuscript for important intellectual content.

Acknowledgements: The authors would like to thank the participants in the study.