Final Project – Part 3

Final Project – Part 3

Revise your chosen methodology, design, and data-collection tools as needed. Also, define the process from the selection of participants (and stakeholders as applicable) through data collection to data analysis, including any limitations, delimiters, and assumptions. Include all ethical considerations. Paint a picture of the entire process to show how you will conduct the study.

Guidelines for Submission: Your paper must be submitted as a five to seven page Word document with double spacing, 12-point Times New Roman font, one-inch margins, and at least three sources cited in APA format.

International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079

Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.

International Journal of Psychological Research 151

Meta-analysis in Psychological Research.

El meta-análisis en la investigación psicológica.

Julio Sánchez-Meca and Fulgencio Marín-Martínez University of Murcia, Spain

ABSTRACT

Meta-analysis is a research methodology that aims to quantitatively integrate the results of a set of empirical

studies about a given topic. With this purpose, effect-size indices are obtained from the individual studies and the

characteristics of the studies are coded in order to examine their relationships with the effect sizes. Statistical analysis in

meta-analysis requires the weighting of each effect estimate as a function of its precision, by assuming a fixed- or a random-

effects model. This paper outlines the steps required for carrying out the statistical analyses in a meta-analysis, the different

statistical models that can be assumed, and the consequences of the assumptions in interpreting their results. The statistical

analyses are illustrated with a real example.

Key words: Meta-analysis, effect size, fixed-effects models, random-effects models, mixed-effects models.

RESUME�

El meta-análisis es una metodología de investigación que pretende integrar cuantitativamente los resultados de un

conjunto de estudios empíricos sobre un determinado problema. Con este propósito, se calculan índices del tamaño del

efecto y se codifican las características de los estudios con objeto de examinar su relación con los tamaños del efecto. El

análisis estadístico en meta-análisis requiere ponderar cada estimación del efecto en función de su precisión asumiendo un

modelo de efectos fijos o de efectos aleatorios. En este trabajo se presentan las etapas necesarias para realizar un meta-

análisis, los diferentes modelos estadísticos que pueden asumirse y las consecuencias de asumir dichos modelos en la

interpretación de sus resultados. Finalmente, los análisis estadísticos se ilustran con datos de un ejemplo real.

Palabras clave: Meta-análisis, tamaño del efecto, modelos de efectos fijos, modelos de efectos aleatorios, modelos de

efectos mixtos.

Article received/Artículo recibido: December 15, 2009/Diciembre 15, 2009, Article accepted/Artículo aceptado: March 15, 2010/Marzo 15/2010 Dirección correspondencia/Mail Address: Julio Sánchez-Meca, Dpto. Psicología Básica y Metodología, Facultad de Psicología, Campus de Espinardo, Universidad de Murcia, 30100-Murcia, Spain, E-mail: jsmeca@um.es Fulgencio Marín-Martínez, University of Murcia, Spain

INTERNATIONAL JOURNAL OF PSYCHOLOGICAL RESEARCH esta incluida en PSERINFO, CENTRO DE INFORMACION PSICOLOGICA DE COLOMBIA, OPEN JOURNAL SYSTEM, BIBLIOTECA VIRTUAL DE PSICOLOGIA (ULAPSY-BIREME), DIALNET y GOOGLE SCHOLARS. Algunos de sus articulos aparecen en SOCIAL SCIENCE RESEARCH NETWORK y está en proceso de inclusion en diversas fuentes y bases de datos internacionales. INTERNATIONAL JOURNAL OF PSYCHOLOGICAL RESEARCH is included in PSERINFO, CENTRO DE INFORMACIÓN PSICOLÓGICA DE COLOMBIA, OPEN JOURNAL SYSTEM, BIBLIOTECA VIRTUAL DE PSICOLOGIA (ULAPSY-BIREME ), DIALNET and GOOGLE SCHOLARS. Some of its articles are in SOCIAL SCIENCE RESEARCH NETWORK, and it is in the process of inclusion in a variety of sources and international databases.

International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079

Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.

152 International Journal of Psychological Research

Meta-analysis in Psychological Research

1. Introduction

In the last 30 years meta-analysis has become a very useful methodological tool for accumulating research

on a given topic. The huge growth of research in

psychology has made it very difficult to synthesize the

results in any field without the help of statistical methods to

summarize the evidence. Unlike traditional reviews on a given topic, which are essentially subjective in nature,

meta-analysis aims to imbue the research review with the

same scientific rigor that is demanded of empirical studies:

objectivity, systematization and replicability. Thus, meta-

analysis is a method used to quantitatively integrate the

results of a set of empirical studies on a given research question. With this purpose, the results of each individual

study included in a meta-analysis have to be quantified in

the same metric, usually by calculating an effect-size index,

and then the effect estimates are statistically analyzed in

order to: (a) obtain an average estimate of the effect

magnitude, (b) assess heterogeneity among the effect

estimates, and (c) search for characteristics of the studies

that can explain the heterogeneity (Cooper, 2010; Cooper,

Hedges, & Valentine, 2009; Hunter & Schmidt, 2004;

Lipsey & Wilson, 2001; Petticrew & Roberts, 2006).

As meta-analysis aims to integrate single studies, the analysis unit is not the participant, but the single study.

Therefore, the sample size in a meta-analysis is the number

of studies that it has been possible to recover regarding the

research question.

Meta-analysis is being applied in many different

fields in psychology, but especially in evaluating the

effectiveness of treatments, interventions, and prevention

programs in such settings as mental health, education,

social services, or human resources. Other psychological

fields where meta-analysis is also being applied include areas such as gender differences in childhood, adolescence

or with adults of many aptitudes and attitudes;

psychometric validity of employment tests, and reliability

generalization of psychological tests in general (Cook,

Cooper, Cordray et al., 1992). Nowadays, it is very

common to find meta-analytic studies on very different topics in any scientific psychology journal. Therefore,

clinicians and researchers should have a sufficient

knowledge base for correctly interpreting and/or carrying

out meta-analyses.

This article is divided into four sections. Firstly,

the phases in which a meta-analysis is carried out are

presented. Then we outline the main statistical methods in

meta-analysis. In the next section statistical methods for

meta-analysis are illustrated using a real example. Finally,

we present some concluding remarks.

2. Phases in a Meta-analysis

A meta-analysis is a scientific investigation and,

consequently, it involves carrying out the same phases as in an empirical study. However, some of the phases have a

few specificities that it is necessary to mention. Basically,

we can conduct a meta-analysis in six phases: (1) Defining

the research question; (2) literature search; (3) coding of

studies; (4) calculating an effect-size index; (5) statistical

analysis and interpretation, and (6) publication (Cooper,

2010; Egger, Davey Smith, & Altman, 2001; Lipsey &

Wilson, 2001; Littell, Corcoran, & Pillai, 2008; Sánchez-

Meca & Marín-Martínez, 2010.

(1) Defining the research question. As in any empirical study, the first step in a meta-analysis is to define

the research question as clearly and objectively as possible.

This implies proposing conceptual and operational

definitions of the different concepts and constructs related

to the research question. For example, in a meta-analysis

about the efficacy of psychological treatments of obsessive-

compulsive disorder (OCD), constructs such as

psychological treatment, obsessive-compulsive disorder,

and the measurement tools to assess efficacy were defined

in this phase (Rosa-Alcázar, Sánchez-Meca, Gómez-

Conesa, & Marín-Martínez, 2008).

(2) Literature search. Once the research question

is formulated, the next step consists of defining the

eligibility criteria of the single studies, that is, the

characteristics a study must fulfill in order to be included in

the meta-analysis. The selection criteria will depend on the

purpose of the meta-analysis, but it is always necessary to

specify the types of study designs that will be accepted

(e.g., only experimental designs, or also quasi-experimental

ones, etc.). For example, in the meta-analysis on OCD

(Rosa-Alcázar et al., 2008) in order to be included in the

meta-analysis the studies had to fulfill several criteria: (a) to apply a psychological treatment to adult patients with OCD;

(b) to include a control group with OCD patients; (c) to

report statistical data for calculating the effect sizes; (d) to

have at least 5 participants in each group, and (e) to be

published between 1980 and 2006.

In this phase the different strategies used to locate

the single studies are also specified. No meta-analysis is

complete without a search of electronic databases

specifying the keywords used (e.g., PsycInfo, MedLine,

ERIC). This search strategy is usually complemented by carrying out searches by hand of relevant journals and

books for the topic of interest, and by checking the

references of the papers included in the meta-analysis.

Additionally, it is very advisable to try to locate

unpublished papers that might fulfill the selection criteria,

in order to counteract publication bias. This can be done by

International Journal of Psychological Research, 2010. Vol. 3. No. 1. ISSN impresa (printed) 2011-2084 ISSN electrónica (electronic) 2011-2079

Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological Research. International Journal of Psychological Research, 3 (1), 151-163.

International Journal of Psychological Research 153

sending letters to well-known researchers in the field

requesting unpublished papers about the topic.

(3) Coding of studies. Once we have the single studies included in the meta-analysis, the next step is to

record the main characteristics of the studies in order to

later explain the heterogeneity exhibited by the effect sizes.

The characteristics of the studies, or moderator variables,

are classified as substantive, methodological, and extrinsic

variables. Substantive characteristics are those related to the

research question of the meta-analysis, whereas

methodological variables are characteristics related to the

study design. Finally, extrinsic variables refer to those

characteristics that, despite are not related with the subjects

nor the study design, could also have an influence in the results. In the OCD example (Rosa-Alcázar et al., 2008),

substantive characteristics coded in the studies included the

type of psychological treatment (e.g., cognitive therapy,

exposure techniques), the mean age of the participants and

the illness history (in years). Some of the methodological

characteristics coded included the type of design

(experimental versus quasi-experimental), attrition in the

posttest, and the sample size. Moreover, extrinsic variables

such as the country where the study was carried out and the

education profile of the main author were also coded.

The coding norms of the moderator variables are

written in a codebook. Some study characteristics are

difficult to code due to incomplete or ambiguous reporting

in the single studies. Therefore, the reliability of the coding

process should be analyzed. To this end, two (or more)

researchers should independently apply the codebook to all

or a sample of the single studies. Then, using the coding

records made by the researchers, agreement indices are

applied (e.g., kappa coefficients, intraclass correlations) in

order to assess the reliability of the coding process.

(4) Calculating an effect-size index. In the coding process of single studies, an effect-size index also has to be

calculated in order to quantify the results of each study in a

common metric. Depending on the study design and the

type of dependent variables (continuous, dichotomous),

different effect-size indices can be applied. Thus, when the

studies have a two-group design and the outcome measure is continuous, the most appropriate effect-size index is the

standardized mean difference or d. This is defined as the

difference between the two means divided by a pooled

within-study standard deviation. Furthermore, when the

dependent variable is dichotomous, several risk indices can be applied: (a) the risk difference, rd, defined as the

difference between the failure (or success) proportions for

the two groups; (b) the risk ratio, rr, defined as the ratio

between the two proportions, and (c) the odds ratio, or,

defined as the ratio between the odds of the two groups.

Finally, when the study applied a correlational design, a correlation coefficient can be used as the effect-size index

(e.g., the Pearson correlation coefficient, its Fisher’s Z

transformation, the point-biserial correlation coefficient, the

phi coefficient, etc.). Table 1 presents some of the usual

effect-size indices applied in meta-analysis together with

their estimated sampling variances, 2

i σ̂ , as they are used in

the statistical analyses of a meta-analysis (cf. Borenstein,

Hedges, Higgins, & Rothstein, 2009; Cooper et al., 2009).

Once the effect-size index most appropriate to the characteristics of the studies has been selected, it is applied

to each single study and its sampling variance is also

calculated with the corresponding formulas (cf., e.g.,

Borenstein et al., 2009). When a meta-analysis includes

studies with different designs (e.g., correlational and two-

group designs), there are formulas to transform different

effect-size indices into each other. For example, it is

possible to transform correlation coefficients into d indices,

and vice versa; or odds ratios into d indices (Sánchez-Meca,

Marín-Martínez, & Chacón-Moscoso, 2003).

(5) Statistical analysis and interpretation. The dataset in a meta-analysis is composed of a matrix where

the rows are the studies and the columns are the moderator

variables, the effect-size index calculated in each study, and

its sampling variance. With these data it is possible to carry

out statistical analyses, which have the following three

main objectives: (1) to calculate an average effect size and

its confidence interval; (b) to assess the heterogeneity of the

effect sizes around the average, and (c) to search for

moderator variables that can explain the heterogeneity

(Sutton & Higgins, 2008). The main characteristic of meta-

analysis is that statistical methods are used for integrating the study results. More details about how to statistically

analyze a meta-analytic database are presented in the next

point of this article.

(6) Publication. Finally, the results of a meta-

analysis have to be published following the same structure as any other scientific paper: Introduction, method, results,

and discussion and conclusions (Botella & Gambara, 2006;

Rosenthal, 1995). A literature review on the topic is

outlined in the introduction, together with definitions of the

constructs and variables implied in the research question,

and the objectives and hypotheses of the meta-analysis. In

the method section the following should be included: the

selection criteria of the studies, the search strategy of the

studies, the coding process of the study characteristics, the

effect-size index calculated in the single studies, and the

statistical analyses that were carried out in the meta-

analytic integration. In the results section the characteristics

of the studies are presented, together with the effect-size

distribution, the mean effect size, the heterogeneity

assessment, and the results of the statistical analyses for

searching for moderator variables related to the effect sizes.

Finally, in the discussion and conclusion section the results

154 International Journal of Psychological Research

of the meta-analysis are compared with previous ones, the

implications for future research are mentioned, and the

limitations and the main conclusions of the meta-analysis

are also outlined.

Table 1. Effect-size indices and their respective estimated within-study sampling variances

Effect-size index Ti Estimated sampling variance, 2

i σ̂

Mean difference CE

yyD −=

C

C

E

E

n

S

n

S DV

22

)( +=

Standardized mean

difference S

yy

� d CE

−

− −=

94

3 1

)(2 )(

2

CECE

CE

nn

d

nn

nn dV

+ +

+ =

Risk difference CE pprd −=

C

CC

E

EE

n

pp

n

pp rdV

)1()1( )(

− +

− =

Natural logarithm of the

risk ratio )/(

CEe ppLogLrr =

CC

C

EE

E

pn

p

pn

p LrrV

− +

− =

11 )(

Natural logarithm of the

odds ratio)

− −

= )1(

)1(

EC

CE

e

pp

pp LogLor

dcba LorV

1111 )( +++=

Pearson correlation

coefficient

rxy 2

)1( )(

22

− −

= �

r rV

xy

xy

Fisher’s Z

− +

= xy

xy

er

r

r LogZ

1

1

3

1 )(

− = �

ZV r

E y and

C y : means for experimental and control groups.

2

E S and

2

C S : variances for experimental and control

groups. nE and nC: sample sizes for experimental and control groups. S: pooled standard deviation of the two groups. ‘ = nE

+ nC. pE and pC: success (or failure) proportions for experimental and control groups. a, b, c, and d: cell frequencies of success and failure for experimental and control groups.

3. Statistical Methods in Meta-analysis

The main characteristic of meta-analysis is the use

of statistical methods to integrate the study results. In order

to do this, an effect size estimate is calculated from each

single study as well as a set of moderator variables

(substantive and methodological characteristics) that can

explain the variability in the effect size distribution. The statistical analysis in a meta-analysis proceeds in three steps

(Lipsey & Wilson, 2001): (1) the obtaining of an average

effect size and a confidence interval around it; (2) the

assessment of the heterogeneity of the effect sizes, and (3)

if there is a large heterogeneity, the search for moderator

variables that may be related to the effect sizes.

The effect sizes obtained from the single studies

differ among themselves in terms of their precision, as they

are calculated from different sample sizes. Effect sizes

obtained from large samples are more accurate than those

obtained from small ones. As a consequence, statistical

methods in meta-analysis take into account the accuracy of

each effect size by weighting them as a function of its

precision (Marín-Martínez & Sánchez-Meca, in press;

Sánchez-Meca & Marín-Martínez, 1998). In particular,

statistical theory shows that the most appropriate method

(in terms of the minimum variance estimate) for weighting

effect sizes in a meta-analysis involves using the inverse variance of each effect size estimate as the weighting factor

(Cooper et al., 2009; Hedges & Olkin, 1985).

(1) Averaging effect sizes. The first step in the

statistical analyses consists in calculating an average effect

size that summarizes the overall effect magnitude of the meta-analyzed studies. The statistical model for carrying

out these calculations assumes a random-effects model,

which considers that the effect size, Ti, in each single study

is estimating a different population effect size, θi, that is, Ti

= θi + ui, where ui represents the sampling error in Ti due to

International Journal of Psychological Research, 2010. Vol. 3. No. 1.

ISSN impresa (printed) 2011-2084

ISSN electrónica (electronic) 2011-2079

Sánchez-Meca, J., Marín-Martínez, F., (2010). Meta-analysis in Psychological

Research. International Journal of Psychological Research, 3 (1), 151-163.

International Journal of Psychological Research 155

the fact that the single study is based on a random sample

selected from the population of potential participants (Field,

2003; Hedges & Vevea, 1998; Schmidt, Oh, & Hayes,

2009). The sampling error is quantified through the within-

study sampling variance, 2

i σ . Thus, it is assumed that in a

given meta-analysis the included studies constitute a

random sample of the studies which could have been

carried out about the same topic. Moreover, for the included

studies it is almost sure that the research conditions differ someway (e.g., in the therapist’s experience, the treatment’s

design and length, etc.), so it is reasonable to suspect that

the effect sizes could vary owing to these differences. Thus,

a distribution of population effect sizes, θi, with a mean

population effect size, µθ, is assumed, that is, θi = µθ + εi, with εi being the deviations of the population effect sizes

from its mean. The variability of the population effect sizes

is called the between-studies variance, τ2, or heterogeneity variance. Hence, in a random-effects model it is assumed

that each effect size estimate includes two variability

sources: the within-study variance, 2

i σ , and the between-

studies variance, τ2. The statistical model can be formulated as:

Ti=µθ+εi+ui. (1)

When εi = 0, then the random-effects model

becomes a fixed-effects model, where there is only one

variability source, the within-study variance 2

i σ , and all of

the studies are estimating the same population effect size.

Thus, the statistical model is simplified to Ti = µθ + ui, and µθ = θi.

In practice the meta-analyst will have to decide

which statistical model to apply, the fixed- or the random-

effects model. The consequences of assuming a random-

effects model or a fixed-effects one concern the

interpretation of the results and also the results obtained

themselves. On the one hand, a meta-analyst that applies a

fixed-effects model is assuming that his/her results can only

be generalized to an identical population of studies to that

of the individual studies included in the meta-analysis,

whereas in a random-effects model the results can be

generalized to a wider population of studies. On the other

hand, the error attributed to the effect size estimates in a

fixed-effects model is smaller than in a random-effects

model, which is why in the first model the confidence

intervals are narrower and the statistical tests more liberal than in the second one. The main consequence of assuming

a fixed-effects model when the meta-analytic data come

from a random-effects model is that we may attribute more

precision to the effect size estimates than is really

appropriate and that we may find statistically significant relationships between variables that are actually spurious.

Consequently, it is more realistic to assume random-effects

models in meta-analysis.

In order to apply statistical inference, it is usually assumed that the effect size distribution, Ti, in a random-

effects model follows a normal distribution with population

mean µθ and variance equal to the sum of the two variability sources,

2

i σ + τ2, that is, Ti ∼ ‘(µθ;

2

i σ + τ2).

Thus, the uniformly minimum variance unbiased

estimator of µθ, UT , is given by (Viechtbauer, 2005):

∑

∑ =

i

i

i

ii

w

Tw

TU , (2)

where wi are the optimal weights, defined as

( )22 ii

τσ1 +=w . The variance of UT is given by:

∑ =

i