Power and Sample Size for Multilevel and Longitudinal Study Designs

Power and Sample Size for Longitudinal and Multilevel Study Designs, a five-week, fully online course covers innovative, research-based power and sample size methods, and software for multilevel and longitudinal studies. The power and sample size methods and software taught in this course can be used for any health-related, or more generally, social science-related (e.g., educational research) application. All examples in the course videos are from real-world studies on behavioral and social science employing multilevel and longitudinal designs. The course philosophy is to focus on the conceptual knowledge to conduct power and sample size methods. The goal of the course is to teach and disseminate methods for accurate sample size choice, and ultimately, the creation of a power/sample size analysis for a relevant research study in your professional context.

Power and sample size selection is one of the most important ethical questions researchers face. Interventional studies that are too large expose human volunteer research participants to possible, and needless, harm from research. Interventional studies that are too small will fail to reach their scientific objective, again bringing possible harm to research participants, without the possibility of concomitant gain from the increase in knowledge. For observational studies in which there are no possible harms to the participants, such as observational studies, proper power ensures good stewardship of both time and money.

Most National Institutes of Health (NIH) study sections will only fund a grant if the grantee has written a compelling and accurate power and sample size analysis. The Institute of Education Sciences (IES), the statistics, research, and evaluation arm of the U.S. Department of Education, also offers competitive grants requiring a compelling and accurate power and sample size analysis (Goal 3: Efficacy and Replication and Goal 4: Effectiveness/Scale-Up).

At the end of the online course, learners will be able to:
• Use a framework and strategy for study planning
• Write study aims as testable hypotheses
• Describe a longitudinal and multilevel study design
• Write a statistical analysis plan
• Plan a sampling design for subgroups, e.g. racial and ethnic
• Demonstrate the feasibility of recruitment
• Describe expected missing data and dropout
• Write a power and sample size analysis that is aligned with the planned statistical analysis

This is a five-week intensive and interactive online course. We will use a mix of instructional videos, software demonstration videos, online readings, quizzes, and exercise assignments. The final course project is a peer-reviewed research study you design for future power or sample size analysis.

• Week 1: Introduction to Multilevel and Longitudinal Designs
• This first module introduces all course participants to the online course, its structure, its learning objectives, and your peers within the course. As noted, the course is composed of multiple activities to reach the learning objectives. Next, we review basic statistical concepts (e.g., hypothesis testing), and explore the fundamentals of both multi-level and longitudinal studies. Conceptual knowledge is covered to provide a framework for analyzing and synthesizing research study designs. This module lays a foundation for subsequent learning. The module concludes with an introduction to the GLIMMPSE software for conducting your own power and sample size analyses. You will walk through a fully guided exercise problem to solve for power for a single level cluster design.
• Week 2: Foundations of Complex Multilevel and Longitudinal Designs
• In the second module, we are going to dive into the many facets of research design, and important considerations related to power and sample size analysis. Specifically, we will examine between, within, and interactions; type 1 error, type 2 error, and power; and standard deviation, variance, and correlation structure. We will explore the appropriate statistical tests for use in specific models, criteria for evaluating these different tests, and how to choose an appropriate test for a data analysis problem. Finally, we will note how clusters of observations or multivariate designs can induce correlation. This module provides the details for specifying research designs, and the beginning steps in aligning the research design to sample size and power analysis. The module concludes with summarizing research designs for GLIMMPSE software. You will walk through a guided exercise problem to solve for sample size analysis for a longitudinal study.
• Week 3: Model Assumptions, Alignment, Missing Data, and Dropout
• The third module includes a wide variety of topics related to power and sample size analysis. First, we examine multivariate and mixed models, their assumptions, and how this assumption impact power. After we focus on aligning the features of data analysis and power analysis as well as the consequences of misalignment. Then we focus on missing data from sources like participant drop-out, machine failures or data entry errors; and how to account for missing data by adjusting your sample size. This module highlights several important features to consider in power and sample size analysis. To conclude the module, you will walk through an exercise problem to solve for power for a multilevel study independently.
• Week 4: Inputs to Analysis, Recruitment Feasibility, and Multiple Aims
• Our emphasis in the fourth module includes the many sources of inputs for power and sample size analysis from the empirical literature, internal pilot studies, planned pilot studies, and computer simulations. Each of these approaches is discussed in detail in relation to power and sample size analysis, including the overall benefits and challenges associated with each approach. Next we talk about recruitment feasibility and its critical importance to sample size calculations by discussing some key factors such as health, socioeconomic, and demographic factors that can be predictive of recruitment difficulty. Next, we deal with research studies that address multiple aims (e.g., hypotheses) and how to address this situation in your sample size analysis. Lastly, you will walk through a fully independent exercise problem to solve for sample size analysis for a multilevel study with longitudinal repeated measures.
• Week 5: Ethics and Using Power and Sample Size Analysis to Get Funded
• The fifth and final module first introduces the ethics of sample size analysis, including overpowered and underpowered research studies and the importance of early planning. Next, we walk through the process of structuring a sample size section of a proposal in a grant application. Then we dive into power curves again and discuss how to decide to incorporate a graphic to best tell your story. After, we explore subgroup analyses such as gender or race, and how to incorporate these design features into a power and sample size analysis. We close our last lecture on searching and applying for funding opportunities, and how a clear design and analysis plan improves your chances for funding. As this is our last module, you will walk through a fully independent exercise problem to solve for sample size analysis for a planned subgroup analysis. You will review at minimum two of your peers’ research design and sample size analyses documents, and finally, complete the final exam in the course.