read

References

Citekey: @Edelson2002

Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105–121. doi:10.1207/S15327809JLS1101_4

Notes

An important piece on design research articulating different types of theories in DR, as well as ‘twin intentions/goals’ of advancing both design theories and domain theories in DR.

Highlights

The emerging design research paradigm treats design as a strategy for developing and refining theories. (p. 3)

This new wave of research, characterized by iterative design and formative research in complex real world settings, has been variously called design experiments (Brown, 1992; Collins, 1992), design research (Cobb, 2001), and development research (Richey & Nelson, 1996; van den Akker, 1999). (p. 4)

The conventional role of design in educational research has been as a strategy for testing theories. Cobb (2001) (p. 4)

As described by this ideal, the role of design in research is to implement a theory so the theory can be evaluated and, if necessary, refined. This role assumes a fully developed theory that maps directly into design. (p. 4)

In the recent literature on design experiments and design research, researchers present a different picture of the role of design in their research (Brown, 1992; Kelly & Lesh, 2000; Richey & Nelson, 1996; van den Akker, 1999). They describe a process in which design plays a critical role in the development of theories, not just their evaluation. (p. 4)

Importantly, these hypotheses and principles are not detailed enough to determine every design decision. In addition, these guiding principles are not followed slavishly if accumulated evidence, specific circumstances, or informed intuition lead the designers to believe they do not apply. In this way, the design researchers proceed through iterative cycles of design and implementation, using each implementation as an opportunity to collect data to inform subsequent design. Through a parallel and retrospective process of reflection upon the design and its outcomes, the design researchers elaborate upon their initial hypotheses and principles, refining, adding, and discarding—gradually knitting together a coherent theory that reflects their understanding of the design experience. (p. 4)

This research has led to both applied (design-oriented) theories and basic theories on topics such as cognition, motivation, and social context. (p. 5)

An important characteristic of design research is that it eliminates the boundary between design and research. In the traditional theory-testing paradigm, design and research are distinct processes that happen sequentially. Design takes place first as the implementation of the theory, followed by the evaluation-oriented research. The design process is not regarded as an opportunity for learning. In contrast, design research explicitly exploits the design process as an opportunity to advance the researchers understanding of teaching, learning, and educational systems. Design research may still incorporate the same types of outcome-based evaluation that characterize traditional theory testing, however, it recognizes design as an important approach to research in its own right. (p. 5)

In this article, I focus on a different question: What kinds of lessons can we learn from the design process in design research? (p. 5)

WHAT WE LEARN WHEN WE ENGAGE IN DESIGN (p. 6)

These opportunities for learning are the direct result of the specific decisions that must be made in the course of a design. (p. 6)

Design is a sequence of decisions made to balance goals and constraints. In the course of any design, the design team makes three sets of decisions that determine the results of the process. These are decisions about (a) how the design process will proceed, (b) what needs and opportunities the design will address, and (c) what form the resulting design will take. (p. 6)

On the other hand, in challenging or innovative design, these decisions can be complex, and as Schon pointed out, interdependent, requiring extensive investigation, experimentation, and iterative refinement on the part of the designers. In these cases, the designers inevitably acquire substantial new understanding. (p. 6)

I refer to the three collections of decisions that determine a design outcome: the design procedure, the problem analysis, and the design solution. (p. 6)

I do not mean to imply that they exist as such in any actual design nor that design is a neat, rational process of making these decisions (Schon, 1990). (p. 7)

Similarly, the problem analysis may not be constructed before the design solution, as an idealized view might suggest, but may be developed hand-in-hand. (p. 8)

Since 1995, my colleagues and I on the Supportive Inquiry-Based Learning (SIBLE) Project at Northwestern University have been investigating the design and use of software to foster reflection in extended computer-supported inquiry activities (Loh et al., 2001). (p. 8)

There have been two types of design products from this research. The first is a software environment called the Progress Portfolio. The Portfolio is a database that allows students to record the process and intermediate products of an investigation, and to annotate, organize, and create presentations from those records. The second design product is a set of curriculum units that incorporate the Progress Portfolio as a support for extended investigations involving computer-based inquiry tools. (p. 8)

We have found the voice of teachers in this design process to be particularly valuable. (p. 9)

Although this example of our research on the Progress Portfolio and reflective inquiry has specific elements that reflect the particular goals of the design research program, it is characteristic of design research in general. The research was undertaken with the twin goals of understanding the process of reflective inquiry and developing software tools with practical utility for supporting that process. (p. 10)

THEORIES FROM DESIGN RESEARCH (p. 10)

That is, the goal of ordinary design is to use the lessons embodied in a design procedure, problem analysis, and design solution to create a successful design product. Design research retains that goal but adds an additional one, the goal of developing useful, generalizable theories. (p. 10)

For each of these three elements of design, there is a corresponding type of theory that design research can develop. I call (p. 10)

these three types of theories domain theories, design frameworks, and design methodologies. (p. 11)

Domain Theories (p. 11)

A domain theory is the generalization of some portion of a problem analysis. Thus, a domain theory might be about learners and how they learn, teachers and how they teach, or learning environments and how they influence teaching and learning. (p. 11)

it is a theory about the world, not a theory about design per se. As such, it is descriptive, not prescriptive. Design research can contribute to two types of domain theories, context theories and outcomes theories. (p. 11)

A context theory characterizes the challenges and opportunities presented by a class of design contexts. For example, in the Progress Portfolio research, we have been developing a context theory that describes the challenges facing students who are engaging in extended scientific investigations for the first time. (p. 11)

An outcomes theory characterizes a set of outcomes associated with some intervention. (p. 11)

Design Frameworks (p. 12)

A design framework is a generalized design solution. Although domain theories are descriptive, design frameworks are prescriptive. They describe the characteristics that a designed artifact must have to achieve a particular set of goals in a particular context. A design framework is a collection of coherent design guidelines for a particular class of design challenge. (p. 12)

Van den Akker (1999) also described design frameworks, which he called substantive design principles, as a distinguishing characteristic of design research. Some prominent examples of design frameworks include: anchored instruction (Cognition & Technology Group at Vanderbilt, 1997), for creating meaningful problem contexts for extended problem solving; concept-oriented reading instruction (Guthrie & Alao, 1997), for motivating reading instruction through interest; for conducting embedded performance assessments (Sloane, Wilson, & Samson, 1996); and goal-based scenarios (Schank, Fano, Bell, & Jona, 1993/1994), for creating learning-by-doing software and in-person learning environments. (p. 12)

Design Methodologies (p. 13)

A design methodology is a general design procedure. Like a design framework, it is prescriptive. However, a design methodology provides guidelines for the process rather than the product. A design methodology describes (a) a process for achieving a class of designs, (b) the forms of expertise required, and (c) the roles to be played by the individuals representing those forms of expertise. (p. 13)

Van den Akker (1999) uses the term procedural design principles to describe a design methodology. The creation of design methodologies is common in many design fields, including traditional instructional design (e.g., Gagné, Briggs, & Wager, 1992), as a mechanism for ensuring that the design process addresses all the essential issues, includes all of the necessary expertise, and progresses efficiently. (p. 13)

FROM DESIGN TO DESIGN RESEARCH (p. 14)

To be useful, lessons from design must apply beyond the specific context in which they were learned, and they must serve an audience beyond the designers themselves. (p. 14)

I describe four features that distinguish design research from simple design and their benefits for generating valuable research results. (p. 14)

Research driven. For a design research program to yield useful results, it must be informed by prior research and guided by research goals. (p. 14)

Systematic documentation. Although it is standard procedure in most engineering disciplines to keep comprehensive records of the design process, there are no such conventions in educational design. Therefore, for most educational designs the only record of the process available for analysis is the design itself. To support the retrospective analysis that is an essential element of design research, the design process must be thoroughly and systematically documented. (p. 14)

Formative evaluation. In principle, formative evaluation should be a part of all design, but like systematic documentation, it is often left out because of limited time or resources. Formative evaluation is critical in design research because it can identify inadequacies in the problem analysis, the solution, and the design procedure that cannot be exposed through analytical processes alone. (p. 15)

Generalization. The final element of design research is the process of generalization. In this process, the designer–researcher expands his or her focus beyond the current design context to look for generalizations to other contexts. (p. 15)

CERTAINTY IN DESIGN RESEARCH (p. 15)

Unlike results from the theory-testing tradition, the form of design research I have described here does not lead to results with statistically determined confidence levels. This concern highlights two important contrasts between design research and experimental research. (p. 15)

The first is that the objective of design research is different from traditional empirical research. Therefore, they should not be judged by the same standards. (p. 15)

goal of design research is the generation of new, useful theories. Thus, two important evaluation metrics for design research are novelty and usefulness. A design research program should yield new theories that have utility for resolving important problems. The point of design research is to generate theories that could not be generated by either isolated analysis or traditional empirical approaches. (p. 16)

The second contrast between design research and traditional empirical research is their source of strength. Traditional empirical methods gain their strength from statistical sampling. As others have pointed out, the strength of theories developed through design research comes from their explanatory power and their grounding in specific experiences (Cobb, 2001; Steffe & Thompson, 2000). A design research theory is compelling to the extent that it is internally consistent and that it accounts for the issues raised during the design and evaluation process. (p. 16)

Finally, though the point of this article is to highlight their differences, design research is not, in fact, incompatible with traditional outcome-based evaluations. If the nature of any theory is such that a minimum level of certainty is required before it should be applied, then the theory should be evaluated empirically before it is applied, whether the theory was developed through design research or otherwise. My argument for the usefulness of this method of theory development is not intended to question the usefulness of theory-testing methods. Evaluation of theories is essential, particularly as Cobb (2001) pointed out, for convincing audiences such as administrators and policymakers. However, there are two risks for the educational research community of overemphasizing such evaluative research. First, the pressure to evaluate theories could lead to useful theories being discarded because they were evaluated and found wanting before they could be fully developed. Second, overvaluing evaluative research can lead us as a community to overlook the important contribution made by research approaches that develop, rather than evaluate, theories. (p. 16)

WHY DESIGN? (p. 16)

The first reason for engaging in design research is that it provides a productive perspective for theory development. Design provides a productive focus in three ways. First, the practical demands of design require that a theory be fully specified. If a theory is incompletely specified, it cannot meet the needs of designers. Second, the process of design reveals inconsistencies more effectively than analytical processes. The practical process of applying a theory to construct a design naturally exposes inconsistencies because the theory will provide the designer with conflicting guidance. Third, the goal-directed nature of design provides a natural (p. 16)

focus for theory development. (p. 17)

The second argument for design research is the usefulness of its results. At its heart, education is a design endeavor. (p. 17)

If the ultimate goal of educational research is the improvement of the education system, then results that speak directly to the design of activities, materials, and systems will be the most useful result. (p. 17)

The third argument for engaging in design research is that design research directly involves researchers in the improvement of education. (p. 17)

Blog Logo

Bodong Chen


Published

Image

Crisscross Landscapes

Bodong Chen, University of Minnesota

Back to Home