Bodong Chen

Crisscross Landscapes

Notes: Clark - 2014 - Quantitative research designs: recognizing the overall plan for a study



Citekey: @Clark2014-nn

Clark, V. L. P., & Creswell, J. W. (2014). Understanding research: A consumer’s guide. Pearson Higher Ed. (Chapter 6)



This chapter first introduces you to two major types of quantitative research. It then focuses on the characteristics of five quantitative research designs that you will likely encounter as you read research. Knowing about different quantitative research designs will help you read the Method sections of quantitative reports and understand and evaluate the research designs used in quantitative studies. (p. 1)

quantitative research studies can have differ- ent purposes, such as explaining the effect of an intervention, the relationships among variables, or the trends in a population. (p. 2)

researchers may use different procedures (p. 2)

researchers select an overall plan, called a research design, to guide their choices among all possible procedures that they could use. By knowing how to recognize the different research designs used in quantitative studies, you have a better understanding of how the particular methods and results follow from a study’s purpose (p. 2)

How Do You Identify the Research Design in a Quantitative Study? (p. 2)

A research design is a logical set of procedures that researchers use to collect, analyze, and report their data in a research study. They are considered logical, because all the different procedures fit together in a coherent way to address a specific kind of research purpose. (p. 2)

when the process of research involves quantitative research, then the researcher’s procedures usually include deciding what to study; stating specific, narrow research questions or hypotheses; collecting numeric data from a large number of participants or time points; analyzing the numbers using statistics; and providing objective expla- nations about group differences, relationships, and trends among variables. (p. 2)

Figure 6.1 summarizes the general characteristics of quantitative research (as introduced in Chapter 2 ), and indicates how research designs fit within this process of quantitative research. (p. 2)

Quantitative research designs, such as experiments or survey research, provide research- ers with blueprints to guide how they select participants and collect, analyze, and report data in their studies so that they can address their specific research purposes. (p. 2)

blueprints, not strict scripts (p. 2)

, a well-written quantitative research report includes supporting infor- mation about the design choice in addition to specifying which design has been chosen. (p. 3)

What Characteristics Distinguish the Different Research Designs Used in Quantitative Studies? (p. 4)

The develop- ment of quantitative research designs has grown alongside the developments in statisti- cal and measurement procedures. In the late 1800s, researchers began working out basic statistical procedures for relating variables and comparing groups. (p. 4)

By 1963, Campbell and Stanley had identified several major types of quantitative research designs; these designs’ procedures, strengths, and weak- nesses continue to be refined and expanded today ( Campbell & Stanley, 1963 ; Kerlinger, 1964 ; Shadish, Cook, & Campbell, 2002 ). (p. 4)

fall within two major categories: experimental research and nonexperimental research. Experimental research is used when a researcher intends to test the effect of an interven- tion by manipulating the conditions experienced by participants. Nonexperimental research is used when researchers intend to describe variables without manipulating the conditions experienced by participants. Thus, quantitative research designs differ in terms of: ■ the intent (e.g., to test for cause and effect of an intervention or to describe variables), ■ the use of manipulation (e.g., the researcher does or does not manipulate the conditions experienced by participants), and ■ the procedures used (e.g., procedures for selecting participants, assigning them to groups, collecting data, and analyzing data). (p. 4)

In addi- tion, by understanding the different procedures that are used, it is easier to assess the kinds of claims that can rightly be made at the conclusion of different quantitative research studies (e.g., claims that an independent variable caused an effect in a depend- ent variable and claims that the results generalize to a larger population). (p. 4)

Experimental quantitative research is used when the research purpose is to deter- mine whether an independent treatment variable (e.g., taking a special medicine or not) causes an effect in a dependent outcome variable (e.g., feelings of depression). (p. 4)

manipulation of the conditions (p. 4)

measurement of the dependent outcome variable (p. 4)

Experimental research is also referred to as intervention research (p. 4)

carefully controlled (p. 4)

Table 6.1 provides a summary of several different types of experimental research designs found in the literature. (p. 4)

When well implemented, experimental design procedures and their emphasis on control can provide strong to moderate evidence that an independent treatment variable caused an effect in another variable. (p. 4)

TABLE 6.1 Intent and Procedures for Different Types of Experimental Research Designs (p. 5)

There are many research situations when it is not possible for the researcher to manipulate the conditions experienced by participants. (p. 5)

Nonexperimental research is also referred to as descriptive or observational research because the variables are described or observed as they happen to occur. Instead of focusing on manipulating and controlling the conditions, nonexperimental research- ers focus attention on such procedures as carefully selecting the participants for the research. To describe trends in a large population, participants are selected so that they are representative of the population; in this way, strong claims can be made that the study results generalize to that population. (p. 5)

Table 6.2 provides a summary of several different types of nonexperimental research design (p. 6)

you generally cannot conclude that one of the variables caused an effect in the other because of the lack of manipulation and lack of control in these designs (p. 6)

TABLE 6.2 Intent and Procedures for Different Types of Nonexperimental Research Designs (p. 6)

How Do You Understand Five Common Quantitative Research Designs? (p. 7)

As summarized in Figure 6.2 , the five quantitative research designs that we consider are: ■ true experiment design, ■ quasi-experiment design, ■ single-subject research design, ■ correlational research design, and ■ survey research design. (p. 7)

The True Experiment Research Design (p. 8)

Random assignment means that each person who participates in the study has an equal chance of being assigned to the treatment or control conditions. (p. 8)

Researchers Use True Experiments When They Need to Be Certain That a Treatment Causes an Effect. (p. 8)

Researchers use a true experiment when they want to be highly certain that the treatment caused the outcome because true experiments provide the best procedures for controlling all the variables that might influence the outcome. (p. 8)

the “gold standard” for conducting quantitative research because the use of random assignment makes them the best designs for deter- mining whether a treatment causes an effect. (p. 8)

Identifying the Characteristics of a True Experiment in a Research Report. (p. 8)

You can note the use of a true experiment when researchers label their design as a true experiment, intervention trial, or randomized controlled trial (RCT). (p. 8)

■ The research problem called for a test of an intervention. (p. 8)

■ The experimental researcher randomly assigned individual participants to differ- ent conditions. (p. 8)

■ The experimental researcher manipulated the conditions experienced by partici- pants. (p. 9)

■ The experimental researcher statistically compared the groups in terms of the out- come variables. (p. 9)

■ Strong claims about whether the treatment caused an effect can be made. (p. 9)

The Quasi-Experiment Research Design (p. 10)

unable to randomly assign (p. 10)

he researcher assigns existing groups to the different conditions, but does not randomly assign individuals because groups cannot be artificially created for the exper- iment. (p. 10)

The use of intact groups, however, is a weakness of this design because it introduces the possibility of other influences that might affect the outcomes. (p. 10)

Therefore, quasi-experiments are only able to provide moderate to weak conclusions about the treatment causing a measured effect. (p. 10)

Researchers Use Quasi-Experiments to Determine the Effect of a Treatment for Intact Groups. When you read the report of a quasi-experiment, the focus on determining whether a treatment causes the desired effect is readily apparent. (p. 10)

they do provide useful evidence of interventions that work within naturally occurring groups like those found in your own practice setting. In some quasi-experiment reports, authors refer to their research design generically as an experiment. This means that you need to recognize the identify- ing characteristics to determine whether it is a true or quasi-experiment. (p. 11)

■ The research problem called for a test of an intervention. (p. 11)

■ The experimental researcher used intact groups of participants. (p. 11)

■ The experimental researcher manipulated the conditions experienced by the groups. (p. 11)

■ The experimental researcher statistically compared the groups in terms of the out- come variables. (p. 11)

■ No more than moderate claims about whether the treatment caused an effect can be made. (p. 11)

Even if a difference is found, at best the researchers only can make moderate claims that the treatment caused the measured effect because they cannot account for all other variables that may have influenced the outcome. (p. 11)

The Single-Subject Research Design (p. 12)

Single-subject research is a quantitative research design that involves the study of single individuals, the administration of an interven- tion, and the careful monitoring of the individuals’ behaviors before, during, and after the intervention to determine whether the treatment affects the behavior. (p. 12)

the individual becomes his/her own “control” in a single- subject experiment in which the researcher collects many measurements of his/her behavior over time. (p. 12)

Researchers Use Single-Subject Designs When They Want to Affect an Individual. (p. 13)

Identifying the Characteristics of Single-Subject Research in a Research Report. In addition to the term single-subject research, you may find that researchers refer to this design as N-of-1 research, applied behavior analysis, or multiple baseline design. Authors also use letters to indicate variants of this design, such as ABAB single-subject design, where A stands for baseline measurements made without the intervention and B stands for measurements made during the intervention. (p. 13)

■ The research problem called for an intervention to change a behavior that is a prob- lem for an individual. (p. 13)

he researcher states a research purpose that focuses on testing an intervention for the individual to address this need. (p. 13)

■ The experimental researcher studied a single individual or a few individuals who would benefit from a change in behavior. (p. 13)

■ The experimental researcher established a baseline of behavior and then manipu- lated the conditions experienced by the individual. (p. 13)

Could the intervention change its ‘intensity’ during the study? Should it stay the same in one study? (p. 13)

■ The researcher plotted the individual’s behavior over many points of time on a graph and visually inspected the data for change. (p. 13)

expect the researcher to note how the behavior of the individual changed during the intervention, after withdrawing the intervention, or during multiple interventions. (p. 13)

■ Strong claims about whether the treatment caused a meaningful change in the indi- vidual’s behavior can be made. (p. 13)

Because of the high level of control of when and how the intervention was administered, the single-subject researcher often makes strong claims of cause and effect in the Conclusion section. (p. 13)

The Correlational Research Design (p. 14)

unable or uninterested in manipu- lating conditions (p. 14)

Correlational research designs are nonexperimental procedures in quantitative research in which investigators measure the degree of association (or relationship) between two or more variables using the statistical procedures of correlational analysis. (p. 14)

Correlational researchers can conclude that two variables are related to each other, such as when higher motivation scores tend to be associated with higher performance scores. They cannot, however, conclude cause and effect because these designs lack the control found in experimental research. (p. 15)

Researchers Use Correlational Research When They Need to Describe the Relationships Among Variables. (p. 15)

This design allows researchers to describe the variables that predict a dependent variable. (p. 15)

Right, prediction does not require causation! (p. 15)

Identifying the Characteristics of Correlational Research in a Research Report. Correlational research is probably the most common type of quantitative research (p. 15)

However, sometimes studies using this design can be identified by correlational terms such as correlation, association, or prediction. Research studies that use a correlational design share the following characteristics: (p. 15)

■ The research problem called for describing the relationship among variables that cannot readily be manipulated. In the Introduction of a correlational study, note that the author usually argues that there is a need to understand how specific vari- ables are related to, associated with, or predict one another. (p. 15)

■ The correlational researcher studied one group of participants. (p. 15)

■ The correlational researcher collected information for each variable. (p. 15)

■ The correlational researcher statistically analyzed the data to test for relationships. (p. 15)

■ Claims are limited to the extent to which variables are related to each other; claims of cause and effect are not warranted. Researchers that used a correlational design cannot correctly conclude that some variables cause an effect in other variables because they did not use random assignment or manipulate the conditions (as in an experi- mental design). (p. 16)

Unfortunately, some researchers err by making cause-and-effect claims in their correlational reports (p. 16)

Once the data were gathered, the researchers used correlation and multiple regression statistical pro- cedures to determine that several variables, including delinquency-related behaviors, age, and substance use, together were significant predictors of nonmedical use of pre- scription drugs by the youth. (p. 16)

The Survey Research Design (p. 17)

Survey research designs are nonexperimental quantitative proce- dures that researchers use to administer a survey questionnaire to a smaller group of people (called the sample) in order to describe trends in attitudes, opinions, behaviors, or characteristics of a larger group of people (called the population). A hallmark of this design is that the researchers carefully select people to study to ensure that they are rep- resentative of a larger population that is of interest. (p. 17)

The word survey is used exten- sively in quantitative research reports. Often it refers to a type of questionnaire used to gather data. Any quantitative study can use a survey ques- tionnaire to gather data. In contrast, a survey research design includes procedures for who to study, how to collect and analyze data, and how to report results. (p. 17)

Important distinction (p. 17)

Researchers Use Survey Research When They Need to Describe Trends in a Population. Researchers choose to use survey research designs when they need to describe trends in a population to understand a research problem. (p. 17)

Identifying the Characteristics of a Survey Research Design in a Research Report. Researchers use different names to refer to the use of a survey research design, including descriptive research or population study. Survey research designs can be difficult to distinguish because researchers using any of the quantitative research designs often refer to their questionnaires for gathering data as a “survey.” (p. 17)

■ The research problem called for describing the attitudes, opinions, or behaviors of a large group. (p. 17)

■ The survey researcher selected a large number of participants who are representative of the group. (p. 17)

■ The survey researcher used a survey questionnaire to gather information from the participants. (p. 18)

■ The survey researcher analyzed the data for trends. (p. 18)

■ Strong claims about trends in the larger population can be made. (p. 18)

In the study’s Conclusion section, expect these claims to be strongly stated if the researcher selected the study sample so that it was representative of the larger population. (p. 18)

How Do You Recognize the Research Design in a Quantitative Research Report? (p. 19)

The following four steps can help identify a study’s quantitative research design: (p. 19)

  1. Look to see if the author named the design in the title or abstract of the report. (p. 19)

  2. Examine the purpose statement to see if it names or suggests the study’s research design. Researchers select their research designs to match their studies’ pur- poses. (p. 19)

  3. Read the beginning paragraphs of the Method section and look for a statement that identifies the design. (p. 19)

  4. Examine the procedures described within the Method and Results sections. Quantitative researchers often do not name their designs in their reports. When this occurs, look for procedures that suggest a specific research design as you read the Method and Results sections (p. 19)

A good strategy is to first consider whether the study used an experimental or nonexperimental approach, and then consider the specific design within that approach. (p. 19)

■ Did the researcher manipulate the conditions by providing a treatment to individuals (true, quasi-, or single-subject experi- mental design) or collect data without manipulating the condi- tions experienced by participants (correlational or survey nonexperimental design)? ■ Did the researcher study two or more groups (true or quasi- experiment), one group (correlational or survey design), or one or a few individuals (single-subject design)? ■ Did the researcher randomly assign individuals to different condi- tions (true experiment) or assign intact groups to different condi- tions (quasi-experiment)? ■ Did the researcher randomly select individuals to participate (usu- ally survey design) or use individuals who were conveniently avail- able (usually experiments or correlational design)? ■ Was the focus of the data analysis on comparing groups (usually true or quasi-experiment), inspecting graphs of an individual’s data (single-subject design), describing the relationship among var- iables (usually correlational design), or describing trends (usually survey design)? (p. 19)

How Do You Evaluate the Research Design in a Quantitative Study? (p. 20)

in a good, rigorous quantitative study, the research design matches the purpose, and the methods of collecting, analyzing, and reporting data match the selected design. A good, rigorous report makes it clear that the researcher had a reason for selecting the design for the study, and that this design is identified, explained, and provided a good overall plan for conducting the study. In addition, the claims that the researcher makes at the end of the study need to be consistent with the overall research design used. (p. 20)

Table 6.3 lists criteria that are useful to consider when evaluating the research desig (p. 20)

Use the rating scale in Figure 6.3 to apply the quality criteria to the research design used in any quantitative research report. For each of the criteria you locate, assign a quality rating from fair (1) to excellent (3) and document your evidence and/or reasoning behind the rating. If one of the criteria is missing or very poorly stated, then indicate poor (0) as your rating. (p. 20)

TABLE 6.3 Criteria for Evaluating the Research Design in a Quantitative Research Report (p. 21)

  1. The choice of the research design is appropriate and justified. (p. 21)

  2. Good quantitative procedures are used to select and assign participants. (p. 21)

  3. Good quantitative data collection procedures are used. (p. 21)

  4. Good quantitative data analysis procedures are used. (p. 21)

  5. Good quantitative results and conclusions are reported. (p. 21)

General Evaluation 6. The study used a rigorous research design. (p. 21)

1 All elements of the study from problem to purpose to methods to results to conclusions fit together in a logical, coherent way. (p. 21)

  1. The use of the quantitative research design addressed the study’s purpose. (p. 21)