PCORI Methodology Award Scientific Results for “New Methods for Planning Cluster Randomized Trials to Detect Treatment Effect Heterogeneity” [PCORI ME-2020C3-21072]

Last updated: December 2025

Background and Basic Idea

Pragmatic cluster randomized trials (CRTs) are increasingly conducted in healthcare delivery systems where patients are nested within providers or clinics. While the average treatment effect (ATE) has been the cornerstone in comparative effectiveness research, there is growing interest in understanding whether the treatment effect varies among pre-specified patient subgroups, addressing the goal of “what works best, and for whom.”  This PCORI-funded methodological project (2021-2025, PCORI project page here) aims to develop new sample size formulas and methods for assessing confirmatory heterogeneity of treatment effect (HTE) in various types of CRT designs. The development is largely based on a covariate-by-treatment interaction test derived from random-effects regression models and properly accounts for the multilevel and longitudinal correlation structures of both the outcome and the baseline effect modifiers. The central finding is that the power of the HTE interaction test and hence sample size depend on two types of intraclass correlation coefficients (ICCs):

  • Outcome ICC (o-ICC): the correlation between outcomes collected from individuals within the same cluster; this is the typical ICC in the classical CRT literature;
  • Covariate ICC (c-ICC): the correlation between effect modifiers (e.g., baseline characteristics) collected from individuals within the same cluster; this is the less familiar ICC that should be considered for determining the power of the interaction test.
Figure 1. Visualization of design effects for testing ATE versus HTE (with a patient-level effect modifier) as a function of the o-ICC, c-ICC and the cluster size. In CRTs, it is apparent that the design effect for testing the ATE is unbounded and increases indefinitely as o-ICC increases. By contrast, the design effect for testing the HTE (interaction between a cluster-level treatment & patient-level covariate) is bounded from the above, and thus the variance inflation due to clustering is generally much smaller. More details in Yang et al (2020).

The procedure involves the following steps: (1) Specify design parameters, including number of clusters, cluster size, o-ICC, c-ICC, and the target HTE effect size; (2) Use the developed analytical formulas (or software) to calculate the required sample size or power; (3) Conduct sensitivity analyses by varying ICC parameters over plausible ranges. To support implementation, an R Shiny App (CRT HTE Calculator) and an R package (CRThtePower) have been developed by our team to support this procedure. At this stage, the R Shiny App is more comprehensive and covers almost all types of CRT designs, but only considers a single binary or continuous effect modifier (arguably the most common case). The R package provides functions to support calculations with multivariate effect modifiers, but is currently refined to three-level parallel-arm designs and stepped wedge designs. The companion tutorial article for the R Shiny App can be accessed here and is under revision at International Journal of Epidemiology.

Publication Deliverables

We have developed a collection of new sample size formulas and methods under the simplest parallel-arm cluster randomized trial design, addressing different aspects of complications encountered in statistical practice. These publications include:

Given many modern healthcare system trials extend beyond the simple parallel-arm CRT design, our team has also provided a set of recent generalizations to address the power of an interaction test in individually-randomized group treatment trials (IRGTs), cluster randomized crossover trials (CRXO Trials), and stepped wedge cluster randomized trials (SW-CRTs). These publications include:

Figure 2. A schematic illustration of the data structure in hypothetical cross-sectional and cohort stepped wedge trials with 6 clusters, 4 periods, and 3 treatment sequences. Here, XijkX_{ijk} and YijkY_{ijk} are the effect modifiers and outcome, 𝕊ij\mathbb{S}_{ij} represents the set of unique individuals recruited in a cluster-period under cross-sectional sampling (Panel A), and 𝕊i\mathbb{S}_i represents the individual cohort recruited at the beginning of the trial for each cluster under closed-cohort sampling (Panel B). Details in Li et al. (2024).

In companion, this project supported several translational tutorial and empirical publications to assist application of the proposed new methods:

Figure 3. Screenshot of the CRT HTE Shiny Calculator interface and power curves from calculator for a SW-CRT with a continuous outcome and a binary effect modify covariate. Details in Baumann et al. (2025+)

Finally, this project also cultivates new methods for estimation; beyond design and sample size considerations, we have contributed approaches for both confirmatory and exploratory HTE analysis in CRTs. The study team is continuing to explore this frontier, and two example methodological developments in this area are:

Primary Investigator Team

  • Principal Investigator: Fan Li, PhD, Associate Professor, Department of Biostatistics, Yale School of Public Health
  • Co-Investigator: Patrick Heagerty, PhD, Professor, Department of Biostatistics, University of Washington
  • Co-Investigator: Rui Wang, PhD, Lagakos and Zelen endowed Chair and Professor, Harvard Pilgrim Health Care Institute and Harvard Medical School
  • Co-Investigator: Denise Esserman, PhD, Professor, Department of Biostatistics, Yale School of Public Health
  • Co-Investigator: Mary Ryan Baumann, PhD, Assistant Professor, Departments of Population Health Sciences and Biostatistics & Medical Informatics, University of Wisconsin – Madison

Disclaimer and Acknowledgement

This project was supported by the a competitive Patient-Centered Outcomes Research Institute (PCORI) Award ME-2020C3-21072 (total cost $1,069,312, approved in July 2021). The official PCORI webpage can be found here, where one can also locate the peer-reviewed version of the Final Research Report (expected to be online in 2026). All members of the study team expressed sincere gratitude to PCORI for supporting this project and making it possible, thereby enabling the development of this new strand of methodological research. To our knowledge, this work represents the frontier of efforts to advance HTE design and estimation and the broader study of heterogeneous effects in cluster-randomized trials.

The Investigator Team would also like to express their deep gratitude to all collaborators and stakeholder partners who have contributed their expertise and dedication throughout this award. It has been a genuine privilege to work with such an engaged and interdisciplinary team. The stakeholder discussions, in particular, have profoundly shaped the direction of this research and have motivated ideas that extended well beyond the initial proposal. Their insights have enriched the project in ways we did not anticipate and have directly contributed to the fact that we now have far more results and progress than originally envisioned. We are truly thankful for their partnership.

Disclaimer: The statements in this website are solely the responsibility of the authors and study team and do not necessarily represent the views of PCORI, its Board of Governors, or Methodology Committee. The principal investigator Fan Li extends special thanks to Yukang Zeng and Hao Wang for their valuable assistance in developing the initial version of this webpage.