Major Constraints in Evidence Synthesis and Meta-Analysis Care Imposed by Coventional Meta-Analysis

Μeta-analysis is a two-steps process for quantitative synthesis of a set of studies that address the same research question. It involves the calculation of one or more study-specific effect sizes and their subsequent synthesis as a weighted average; each study is attributed a weight according to the amount of information it carries (as this is reflected by the sample size, number of events, precision etc).
There are several considerations to be taken into account during this process, such as the choice of the appropriate measure of association and the integration of between-study differences into the summary effect. Methods to measure and account for heterogeneity are central to meta-analysis; in fact, the exploration of sources of heterogeneity has been proven particularly helpful in understanding the conditions under which the intervention or risk factor might alter its postulated impact on the studied condition. Meta-analysis offers the potential to increase precision and consolidate conclusions, but the reliability of the result is often limited by the quality of the included studies.
The practical difficulties are numerous. Individual studies have missing data or they present outcomes in incompatible ways that make synthesis challenging. Statistical tools have been developed tackling some of these problems within the conventional meta-analysis framework. There is a vast statistical literature offering solutions to challenging or controversial issues in meta-analysis (such as alternative random-effects distributions, missing data imputation techniques etc).
The main drawback of the current state of the art is that evidence synthesis and meta-analysis focus on comparing only two alternatives, e.g. two interventions. For example, consider the situation of smoking cessation therapies; there are more than twenty-two treatments. There are twenty-one Cochrane reviews, each one of them synthesizing studies that compare smoking cessation therapy A versus therapy B. Each of these reviews can be informative as to whether a given intervention is better than another. However, this is only a fragment of the big picture. Ideally, clinicians would like to know how all the different options rank against each other and how big the differences are between all the available options. The choice of the best intervention should ideally also take into account genetic and epidemiological differences across populations.In this context, one needs to consider the entire set of those twenty-one head-to-head comparisons. Another example is the selection of the best treatment for breast cancer; several chemotherapy regiments are possible and their effectiveness is evaluated in several pair-wise meta-analyses of randomized trials. However, it is believed that the effectiveness of some interventions is altered by the HER-2 status. Therefore, evidence including all possible treatment alternatives should be combined with meta-analysis of genetic factors and interactions should be addressed.
Unfortunately, the restricted picture described above is encountered in all organisations supporting guidelines and systematic reviews that focus solely on comparing ‘option A’ versus ‘option B’. Even if many such head-to-head comparisons are considered, this fragmented approach restricts the applicability and relevance of systematic reviews for developing guidelines and recommendations. Moreover, relevant indirect evidence from other studies (through for example an indirect comparator) is ignored with substantial loss of information.

Within a better framework, all possible health technologies for the same health condition should be considered within their environmental and genetic context, all relevant experiments should be identified and their results synthesized. The need for a new robust framework which answers directly critical decision-making questions has been identified by core national and international agencies such as WHO, The Cochrane Collaboration and NICE in the U.K. A new form of reviews was introduced: the Overviews of Reviews developed within the auspices of the Cochrane Collaboration that summarize evidence for the relative effectiveness of several interventions for the same condition. This emerging review format is not popular yet because of important difficulties with the necessary statistical component, the Multiple-Treatments Meta-analysis (MTM). The first attempts in this direction appear to be very promising but controversial: the recent evaluation of the relative effectiveness of all new generation antidepressants attracted enthusiastic supporters as well as scepticism; for discussion see (Barbui et al., 2009, Cipriani et al., 2009).


Barbui, C., Cipriani, A., Furukawa, T. A., Salanti, G., Higgins, J. P., Churchill, R., Watanabe, N., Nakagawa, A., Omori, I. M., & Geddes, J. R. (2009). Making the best use of available evidence: the case of new generation antidepressants: a response to: are all antidepressants equal? Evid. Based. Ment. Health 12, 101-104

Cipriani, A., Furukawa, T. A., Salanti, G., Geddes, J. R., Higgins, J. P., Churchill, R., Watanabe, N., Nakagawa, A., Omori, I. M., McGuire, H., Tansella, M., & Barbui, C. (2009). Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet 373, 746-758

Additional information