Meta-Analysis of Controlled Clinical Trials

By Anne Whitehead

John Wiley & Sons

ISBN: 0-471-98370-5

Chapter One



Meta-analysis was defined by Glass (1976) to be 'the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings'. Although Glass was involved in social science research, the term 'meta-analysis' has been adopted within other disciplines and has proved particularly popular in clinical research. Some of the techniques of meta-analysis have been in use for far longer. Pearson (1904) applied a method for summarizing correlation coefficients from studies of typhoid vaccination, Tippet (1931) and Fisher (1932) presented methods for combining p-values, and Yates and Cochran (1938) considered the combination of estimates from different agricultural experiments. However, the introduction of a name for this collection of techniques appears to have led to an upsurge in development and application.

In the medical world, the upsurge began in the 1980s. Some of the key medical questions answered by meta-analyses at this time concerned the treatment of heart disease and cancer. For example, Yusuf et al. (1985) concluded that long-term beta blockade following discharge from the coronary care unit after amyocardial infarction reduced mortality, and the Early Breast Cancer Trialists' Collaborative Group (1988) showed that tamoxifen reduced mortality in women over 50 with early breast cancer. By the 1990s published meta-analyses were ubiquitous. Chalmers and Lau (1993) claimed: 'It is obvious that the new scientific discipline of meta-analysis is here to stay'. They reported a rise in the number of publications of meta-analyses of medical studies from 18 in the 1970s to 406 in the 1980s. Altman (2000) noted that Medline contained 589 such publications from 1997 alone.

The rapid increase in the number of meta-analyses being conducted during the last decade is mainly due to a greater emphasis on evidence-based medicine and the need for reliable summaries of the vast and expanding volume of clinical research. Evidence-based medicine has been defined as 'integrating individual clinical expertise with the best available external clinical evidence from systematic research' (Sackett et al., 1997). A systematic review of the relevant external evidence provides a framework for the integration of the research, and meta-analysis offers a quantitative summary of the results. In many cases a systematic review will include a meta-analysis, although there are some situations when this will be impossible due to lack of data or inadvisable due to unexplained inconsistencies between studies.

The Cochrane Collaboration, launched in 1993, has been influential in the promotion of evidence-based medicine. This international network of individuals is committed to preparing, maintaining and disseminating systematic reviews of research on the effects of health care. Their reviews are made available electronically in the Cochrane Database of Systematic Reviews, part of the Cochrane Library (

Within the pharmaceutical industry, meta-analysis can be used to summarize the results of a drug development programme, and this is recognized in the International Conference on Harmonization (ICH) E9 guidelines (ICH, 1998). In accordance with ICH E9, meta-analysis is understood to be a formal evaluation of the quantitative evidence from two or more trials bearing on the same question. The guidelines indicate that meta-analysis techniques provide a useful means of summarizing overall efficacy results of a drug application and of analysing less frequent outcomes in the overall safety evaluation. However, there is a warning that confirmation of efficacy from a meta-analysis only will not usually be accepted as a substitute for confirmation of efficacy from individual trials. Certainly the magnitude of the treatment effect is likely to be an important factor in regulatory decision-making. If the treatment effect is smaller than anticipated, then statistical significance may not be reached in the individual trials. Even if statistical significance is reached in the meta-analysis, the magnitude of the treatment effect may not be clinically significant, and thus be considered insufficient for approval.

Fisher (1999) considered the two conditions under which one large trial might substitute for the two controlled trials usually required by the Food and Drug Administration (FDA) in the USA. The first relates to the strength of evidence for demonstrating efficacy. He showed that if the evidence required from the two controlled trials is that they should each be statistically significant at the twosided 5% significance level, then the same strength of evidence is obtained from one large trial if it is statistically significant at the two-sided 0.125% level. The same type of argument could be applied to combining trials in a meta-analysis. It would seem reasonable to set a more stringent level of statistical significance corresponding to proof of efficacy in a meta-analysis than in the individual trials.

The second condition discussed by Fisher relates to evidence of replicability, and he proposes criteria which need to be met by the one large trial. A meta-analysis will always involve at least two trials, and it will be important to assess the consistency of the results from the individual trials. The extent of any inconsistencies amongst the trials will be influential in the choice of model for the meta-analysis and in the decision whether to present an overall estimate. These issues are discussed in detail in Chapter 6 of this book.

A recent 'Points to Consider' document (Committee for Proprietary Medicinal Products, 2001) has provided guidance on when meta-analyses might usefully be undertaken. Reasons include the following:

To provide a more precise estimate of the overall treatment effects.

To evaluate whether overall positive results are also seen in pre-specified subgroups of patients.

To evaluate an additional efficacy outcome that requires more power than the individual trials can provide.

To evaluate safety in a subgroup of patients, or a rare adverse event in all patients.

To improve the estimation of the dose-response relationship.

To evaluate apparently conflicting study results.

There is much to be gained by undertaking a meta-analysis of relevant studies before starting a new clinical trial. As Chalmers and Lau (1993) note, this allows investigators to ascertain what data are needed to answer the important questions, how many patients should be recruited, and even whether a new study is unnecessary because the questions may have already been answered. Meta-analysis also has a useful role to play in the generation of hypotheses for future studies.

The conduct of a meta-analysis requires a team, which should include both statisticians and knowledgeable medical experts. Whilst the statistician is equipped with the technical knowledge, the medical expert has an important role to play in such activities as identifying the trials, defining the eligibility criteria for trials to be included, defining potential sources of heterogeneity and interpreting the results.

Most meta-analyses within the field of medical research have been conducted on randomized controlled trials, and this is the focus of this book. Other application areas include epidemiological studies and diagnostic studies. The special problems associated with observational studies are outside the scope of this book, and the interested reader is referred to Chapter 16 of Sutton et al. (2000) and Chapters 12-14 of Egger et al. (2001).

Over the last twenty years there have been great strides in the development and refinement of statistical methods for the conduct of meta-analyses, as illustrated in the books by Sutton et al. (2000) and Stangl and Berry (2000). A number of different approaches have been taken, giving the impression that the methodology is a collection of distinct techniques. The present book is self-contained and describes the planning, conduct and reporting of a meta-analysis as applied to a series of randomized controlled trials. It attempts to present the various approaches within a general unified framework, and to place this framework within mainstream statistical methodology.


Meta-analyses are often performed retrospectively on studies which have not been planned with this in mind. In addition, many are based on summary statistics which have been extracted from published papers. Consequently, there are a number of potential problems which can affect the validity of such meta-analyses.

A major limitation is that a meta-analysis can include only studies for which relevant data are retrievable. If only published studies are included, this raises concern about publication bias, whereby the probability of a study being published depends on the statistical significance of the results. Even if a study is published, there may be selective reporting of results, so that only the outcomes showing a statistically significant treatment difference are chosen from amongst the many analysed. If the outcomes of interest have not been defined or recorded in the same way in each trial, it may not be appropriate or possible to combine them. Even if identical outcomes have been recorded in each trial, the way in which the summary statistics have been calculated and reported may differ, particularly with regard to the choice of the subjects included and the mechanism of dealing with missing values. Matters can be improved if time and effort are devoted to obtaining data from all (or nearly all) of the randomized trials undertaken, irrespective of their publication status. Retrieving individual patient data from trial investigators is especially advantageous.

Typically, the objective of a meta-analysis is to estimate and make inferences about the difference between the effects of two treatments. This involves choosing an appropriate measure of the treatment difference, for example the log-odds ratio for binary data or the difference in means for normally distributed data, and calculating individual study estimates and an overall estimate of this difference. In a retrospective meta-analysis the available studies may vary in design, patient population, treatment regimen, primary outcome measure and quality. Therefore, it is reasonable to suppose that the true treatment difference will not be exactly the same in all trials: that is, there will be heterogeneity between trials. The effect of this heterogeneity on the overall results needs to be considered carefully, as discussed by Thompson (1994). Great care is needed in the selection of the trials to be included in the meta-analysis and in the interpretation of the results.

Prospectively planning a series of studies with a view to combining the results in a meta-analysis has distinct advantages, as many of the problems associated with retrospective meta-analyses then disappear. The individual trial protocols can be designed to be identical with regard to the collection of data to be included in the meta-analysis, and individual patient data can be made available.

In drug development, a co-ordinated approach to the trial programme, in which meta-analyses are preplanned, would seem to be a natural way to proceed. The results of a meta-analysis will be more convincing if it is specified prior to the results of any of the individual trials being known, is well conducted and demonstrates a clinically relevant effect.

Within the public sector, collaborative groups are beginning to form in order to conduct prospective meta-analyses. For example, the Cholesterol Treatment Trialists' Collaboration (1995) reported on their protocol for conducting an overview of all the current and planned randomized trials of cholesterol treatment regimens. In such cases it is unlikely that the meta-analysis can be planned before the start of any of the trials, but certainly the preparation of a protocol prior to the analysis of any of them offers considerable advantages.

The conduct of both retrospective and prospective meta-analyses will be discussed in this book. Many of the analysis methods are common to both, although methodological difficulties tend to be fewer and more manageable for the prospective meta-analysis.


One of the controversies relating to meta-analysis has concerned the choice between the fixed effects model and the random effects model for providing an overall estimate of the treatment difference. The topic has usually been discussed in the context of a meta-analysis in which the data consist of trial estimates of the treatment difference together with their standard errors. In the fixed effects model, the true treatment difference is considered to be the same for all trials. The standard error of each trial estimate is based on sampling variation within the trial. In the random effects model, the true treatment difference in each trial is itself assumed to be a realization of a random variable, which is usually assumed to be normally distributed. As a consequence, the standard error of each trial estimate is increased due to the addition of this between-trial variation.

The overall estimate of treatment difference and its confidence interval based on a fixed effects model provide a useful summary of the results. However, they are specific to the particular trials included in the meta-analysis. One problem is that they do not necessarily provide the best information for determining the difference in effect that can be expected for patients in general. The random effects model allows the between-trial variability to be accounted for in the overall estimate and, more particularly, its standard error. Therefore, it can be argued that it produces results which can be considered to be more generalizable. In principle, it would seem that the random effects model is a more appropriate choice for attempting to answer this question. However, there are some concerns regarding the use of the random effects model in practice. First, the random effects model assumes that the results from the trials included in the meta-analysis are representative of the results which would be obtained from the total population of treatment centres. In reality, centres which take part in clinical trials are not chosen at random. Second, when there are only a few trials for inclusion in the meta-analysis, it may be inappropriate to try to fit a random effects model as any calculated estimate of the between-study variance will be unreliable. When there is only one available trial, its analysis can only be based on a fixed effects model.


Excerpted from Meta-Analysis of Controlled Clinical Trials by Anne Whitehead Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.