Pragmatic Free Trial Meta
Pragmatic Free Trial Meta is a non-commercial, open data platform and infrastructure that supports research on pragmatic trials. It collects and distributes cleaned trial data, ratings, and evaluations using PRECIS-2. This allows for diverse meta-epidemiological studies to evaluate the effects of treatment across trials with different levels of pragmatism.
Background
Pragmatic trials are becoming more widely recognized as providing real-world evidence for clinical decision-making. Pragmatic KR " however, is a word that is often used in contradiction and its definition and assessment require further clarification. The purpose of pragmatic trials is to guide clinical practice and policy decisions, rather than to prove an hypothesis that is based on a clinical or physiological basis. A pragmatic trial should try to be as close as is possible to the real-world clinical practice that include recruiting participants, setting, design, delivery and implementation of interventions, determining and analysis results, as well as primary analyses. This is a major difference between explanatory trials as described by Schwartz & Lellouch1, which are designed to confirm the hypothesis in a more thorough way.
The most pragmatic trials should not conceal participants or clinicians. This can lead to bias in the estimations of the effects of treatment. The pragmatic trials also include patients from different healthcare settings to ensure that their outcomes can be compared to the real world.
Additionally, clinical trials should be focused on outcomes that matter to patients, like the quality of life and functional recovery. This is particularly relevant for trials involving the use of invasive procedures or potential for serious adverse events. The CRASH trial29, for instance focused on the functional outcome to compare a 2-page case-report with an electronic system to monitor the health of patients admitted to hospitals with chronic heart failure, and the catheter trial28 focused on urinary tract infections caused by catheters as the primary outcome.
In addition to these features, pragmatic trials should minimize the trial's procedures and data collection requirements to reduce costs. Furthermore pragmatic trials should try to make their results as relevant to actual clinical practice as possible by ensuring that their primary analysis is the intention-to-treat approach (as described in CONSORT extensions for pragmatic trials).
Despite these requirements however, a large number of RCTs with features that challenge the notion of pragmatism were incorrectly labeled pragmatic and published in journals of all kinds. This can lead to false claims of pragmatism, and the use of the term needs to be standardized. The creation of a PRECIS-2 tool that provides a standardized objective assessment of pragmatic features is a good start.
Methods
In a practical study it is the intention to inform clinical or policy decisions by showing how an intervention can be integrated into routine treatment in real-world situations. This differs from explanation trials that test hypotheses regarding the causal-effect relationship in idealized settings. Therefore, pragmatic trials might have lower internal validity than explanatory trials and may be more susceptible to bias in their design, conduct, and analysis. Despite these limitations, pragmatic trials can contribute valuable information to decision-making in the context of healthcare.
The PRECIS-2 tool evaluates the degree of pragmatism in an RCT by assessing it on 9 domains, ranging from 1 (very explicit) to 5 (very pragmatic). In this study, the domains of recruitment, organisation, flexibility in delivery, flexible adherence and follow-up received high scores. However, the primary outcome and the method for missing data were scored below the practical limit. This suggests that a trial can be designed with good practical features, yet not harming the quality of the trial.
However, it's difficult to judge the degree of pragmatism a trial is since pragmaticity is not a definite characteristic; certain aspects of a trial may be more pragmatic than others. Furthermore, logistical or protocol modifications made during a trial can change its pragmatism score. In addition 36% of the 89 pragmatic trials discovered by Koppenaal and colleagues were placebo-controlled or conducted prior to licensing and most were single-center. This means that they are not very close to usual practice and can only be described as pragmatic in the event that their sponsors are supportive of the absence of blinding in these trials.
A common aspect of pragmatic studies is that researchers attempt to make their findings more meaningful by analyzing subgroups within the trial sample. However, this can lead to unbalanced comparisons with a lower statistical power, which increases the risk of either not detecting or incorrectly detecting differences in the primary outcome. This was a problem during the meta-analysis of pragmatic trials because secondary outcomes were not adjusted for covariates that differed at baseline.
In addition practical trials can present challenges in the gathering and interpretation of safety data. This is due to the fact that adverse events tend to be self-reported and are susceptible to delays, inaccuracies or coding differences. It is therefore important to improve the quality of outcome ascertainment in these trials, in particular by using national registry databases instead of relying on participants to report adverse events in the trial's database.
Results
While the definition of pragmatism may not require that all trials be 100 percent pragmatic, there are some advantages to incorporating pragmatic components into clinical trials. These include:
Enhancing sensitivity to issues in the real world, reducing cost and size of the study and allowing the study results to be faster implemented into clinical practice (by including patients from routine care). However, pragmatic trials can also have drawbacks. For example, the right type of heterogeneity could help a study to generalize its findings to a variety of settings and patients. However, the wrong type of heterogeneity could reduce assay sensitiveness and consequently reduce the power of a study to detect small treatment effects.
Numerous studies have attempted to categorize pragmatic trials, with various definitions and scoring systems. Schwartz and Lellouch1 created a framework to distinguish between research studies that prove a clinical or physiological hypothesis, and pragmatic trials that help in the selection of appropriate therapies in real-world clinical practice. Their framework included nine domains that were scored on a scale of 1 to 5, with 1 indicating more lucid and 5 indicating more pragmatic. The domains included recruitment, setting up, delivery of intervention, flex adherence and primary analysis.
The original PRECIS tool3 featured similar domains and scales from 1 to 5. Koppenaal et al10 developed an adaptation of this assessment, dubbed the Pragmascope that was simpler to use for systematic reviews. They found that pragmatic systematic reviews had a higher average score in most domains but lower scores in the primary analysis domain.
This difference in primary analysis domain can be due to the way in which most pragmatic trials analyze data. Certain explanatory trials however do not. The overall score for systematic reviews that were pragmatic was lower when the areas of organization, flexible delivery, and follow-up were merged.
It is important to note that a pragmatic trial does not necessarily mean a low-quality trial, and in fact there is an increasing number of clinical trials (as defined by MEDLINE search, however this is not specific or sensitive) that employ the term 'pragmatic' in their title or abstract. The use of these terms in titles and abstracts may suggest a greater awareness of the importance of pragmatism but it isn't clear if this is reflected in the contents of the articles.
Conclusions
In recent years, pragmatic trials have been becoming more popular in research as the value of real-world evidence is increasingly recognized. They are randomized trials that compare real world alternatives to clinical trials in development. They are conducted with populations of patients that are more similar to those who receive treatment in regular care. This approach can overcome the limitations of observational research such as the biases that come with the reliance on volunteers and the lack of the coding differences in national registry.
Pragmatic trials offer other advantages, such as the ability to use existing data sources and a greater probability of detecting meaningful differences from traditional trials. However, pragmatic tests may be prone to limitations that undermine their effectiveness and generalizability. Participation rates in some trials could be lower than anticipated because of the healthy-volunteering effect, financial incentives, or competition from other research studies. The necessity to recruit people quickly restricts the sample size and impact of many pragmatic trials. Practical trials aren't always equipped with controls to ensure that the observed variations aren't due to biases in the trial.
The authors of the Pragmatic Free Trial Meta identified RCTs published from 2022 to 2022 that self-described themselves as pragmatic. They assessed pragmatism using the PRECIS-2 tool, which consists of the eligibility criteria for domains and recruitment criteria, as well as flexibility in intervention adherence, and follow-up. They found that 14 of these trials scored as highly or pragmatic sensible (i.e. scores of 5 or more) in one or more of these domains, and that the majority were single-center.
Studies that have high pragmatism scores tend to have broader criteria for eligibility than traditional RCTs. They also include patients from a variety of hospitals. The authors claim that these traits can make the pragmatic trials more relevant and applicable to daily practice, but they don't necessarily mean that a pragmatic trial is free of bias. The pragmatism principle is not a fixed characteristic; a pragmatic test that doesn't have all the characteristics of an explicative study could still yield reliable and beneficial results.
