Bodong Chen

Crisscross Landscapes

Notes: Boote - 2005 - Scholars Before Researchers



Citekey: @Boote2005-kt

Boote, D. N., & Beile, P. (2005). Scholars Before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation. Educational Researcher , 34(6), 3–15.



This article suggests criteria to evaluate the quality of dissertation literature reviews and reports a study that examined dis- sertations at three universities. Acquiring the skills and knowledge required to be education scholars, able to analyze and synthesize the research in a field of specialization, should be the focal, integrative activity of predissertation doctoral education. Such scholarship is a prerequisite for increased methodological sophistication and for im- proving the usefulness of education research. (p. 1)

The Role and Purpose of the Literature Review in Education Research (p. 1)

“Good” research is good because it advances our collective under- standing. (p. 1)

all heard the joke before—as we move through graduate school, we learn more and more about less and less until we know everything about nothing. It is expected that someone earning a doctorate has a thorough and sophisticated understanding of an area of research and scholarship. (p. 1)

Shulman argues that generativity—along with discipline, publication, and peer review—is one of the hallmarks of schol- arship (1999, p. 162–163). He defines generativity as the ability to build on the scholarship and research of those who have come before us. Generativity grants our work integrity and sophistica- tion. (p. 1)

Yet the messy, complicated nature of problems in education makes generativity in education research more difficult than in most other fields and disciplines (Berliner, 2002) (p. 1)

reparing students to analyze and synthesize research in a field of specialization is crucial to understanding ed- ucational ideas. Such preparation is prerequisite to choosing a productive dissertation topic and appropriating fruitful methods of data collection and analysis. (p. 1)

Education Research and Doctoral Preparation (p. 2)

As the foundation of any research project, the literature review should accomplish several important objectives. It sets the broad context of the study, clearly demarcates what is and what is not within the scope of the investigation, and justifies those deci- sions. It also situates an existing literature in a broader scholarly and historical context. It should not only report the claims made in the existing literature but also examine critically the research methods used to better understand whether the claims are war- ranted. (p. 2)

When considering the criteria and standards used to evaluate a dissertation, we need to keep in mind that most people with doctorates in education do not go on to pursue research careers. Most teach, administer, or lead (Passmore, 1980). (p. 2)

More- over, this type of review allows the author not only to summarize the existing literature but also to synthesize it in a way that permits a new perspective. Thus a good literature review is the basis of both theoretical and methodological sophistication, thereby improving the quality and usefulness of subsequent research. (p. 2)

uccessful doctoral candi- dates need to be “comprehensive and up to date in reviewing the literature” (Barry, 1997) and that their dissertations demonstrate this prowess (p. 2)

Yet it is apparent that for many, if not most, doctoral candi- dates and dissertation committees, the literature review is of sec- ondary importance. (p. 2)

Graduate education in the United States developed in a period when German universities were ascendant and when “America copied the German version of advanced studies … unfortunately the period when the Berlin positivists were in the ascendancy” (Berelson, 1960, p. 12; see also Storr, 1969). As a result, the U.S. doctorate was designed to focus on research training, and the dissertation became a vehicle for demonstrating research prowess.2 (p. 2)

Our secret is made pub- lic by editors and reviewers who openly lament the inadequacy of literature reviews in manuscripts submitted for journal publi- cation (Alton-Lee, 1998; Grant & Graue, 1999; LeCompte, Klingner, Campbell, & Menk, 2003). (p. 2)

Schoenfeld (1999) identifies a number of difficulties and dilemmas facing doctoral education. Among these are specialization that leads to compartmentaliza- tion, theorizing that leads to superficiality, and simplistic ap- proaches to methodology that hinder a deep understanding of what it means to make and justify a claim about educational phe- nomena. (p. 2)

On the basis of his experience coordinating and teaching in a doctoral program, Labaree (2003) outlined some of the general problems facing doctoral education. He framed the problems in terms of a clash between school and university cul- tures that occurs when we ask teachers to shift from a normative to an analytic way of thinking, from a personal to an intellectual relationship with educational phenomena, from a particular to a universal perspective, and from an experimental to a theoretical disposition. (p. 3)

Quite to the point. And special for education. (p. 3)

In other words, with the very few ex- ceptions noted below, most graduate students receive little or no formal training in how to analyze and synthesize the research lit- erature in their field, and they are unlikely to find it elsewhere. (p. 3)

Richardson (2003), who develops the concept of doctors of ed- ucation as stewards of both the field of study and the enterprise of education. (p. 3)

That doctoral candidates would espouse such naïve concep- tions of literature reviewing and perceive it as relatively low in importance would seem to be a product of the culture of doctoral programs in education. (p. 3)

The Literature Review: A Necessary Chore? (p. 3)

ibrary skills (p. 3)

bibliographic skills (p. 3)

hese chapters typically indicate the importance of the literature review, albeit in vague terms, and briefly summarize techniques for searching electronic databases and methods for abstracting prior research. (p. 3)

but place a much greater emphasis on an understanding of methods of data collection and analysis. (p. 3)

Creswell (1994) suggests that the literature review should meet three criteria: “to present results of similar studies, to relate the present study to the ongo- ing dialogue in the literature, and to provide a framework for comparing the results of a study with other studies” (p. 37). To accomplish these criteria Creswell (2002) recommends a five- step process: “identifying terms to typically use in your literature search; locating literature; reading and checking the relevance of the literature; organizing the literature you have selected; and writing a literature review” (p. 86). (p. 3)

Both Zaporozhetz and Labaree note that ed- ucation doctoral students tend to be mature, accomplished pro- fessionals who are committed to improving education practice. Yet these qualities make it more difficult for them to admit that they may lack library search and information synthesis skills and knowledge. (p. 3)

ature with more explanatory and predictive power than is offered by existing perspectives. Finally, it satisfies the formal criteria of a good theory. Standards such as consistency, parsimony, elegance, and fruitfulness characterize a good synthesis. (p. 4)

A product of this doctoral program culture is that the litera- ture review is not valued, and because it is not valued it is rarely an explicit part of doctoral curriculums. (p. 4)

Library instruction has tended to focus on the mechanics of database search strategies and on the varieties of in- formation available. (p. 4)

On en- countering an inadequate literature review, examiners would proceed to look at the methods of data collection, the analysis, and the conclusions much more carefully. In that way, Mullins and Kiley found that for examiners there was a tacit link between candidates’ knowledge of the field and their ability to do sub- stantive, well-justified research. (p. 4)

Instead, we need to under- stand that the ability to write a thorough, sophisticated literature review is a form of scholarship requiring a broad range of skills and knowledge—skills and knowledge that we ought to expect of anyone earning a doctorate. (p. 4)

The Literature Review: Our Foundation and Inspiration (p. 4)

In an editorial in Review of Educational Research, LeCompte and colleagues (2003) wrote on the importance of convincing emerg- ing scholars that state-of-the-art literature reviews are legitimate and publishable scholarly documents. Too many new scholars believe that empiri- cal research is the only “real” research; they avoid the deep levels of investigation needed to create the kinds of manuscripts sought by RER. (p. 4)

Hart contends (p. 27) that a disserta- tion literature review should clearly articulate what research needs be done in a field and why it is important, articulate the practical significance of the research, synthesize prior research to gain a new perspective on it, and critically analyze the research methods used.3 (p. 4)

Strike and Posner (1983, pp. 356–357) further suggest that a good synthetic review has three characteristics. First, it clarifies and perhaps resolves the problems within a field of study rather than glossing over those problems. Second, it results in a “pro- gressive problem shift” that yields a new perspective on the liter- (p. 4)

Hart clearly articulates that doctoral students must be successful scholars—able to critically synthesize ideas and meth- ods in their field—before they are to have any chance of being generative researchers (p. 5)

Contrast this with the most common con- ception, which seems to entail a mechanical process of summa- rizing a supposedly exhaustive collection of prior studies. (p. 5)

Yet library search skills are not enough. Too often, coverage is interpreted by doctoral students as exhaustive coverage of every- thing previously written about their topic (Bruce, 2001a). This naïve approach to searching and selecting prior research can make it very difficult for researchers to critically synthesize the literature in their field, (p. 5)

Standards and Criteria of a Literature Review (p. 5)

Bruce suggests that coverage should be looked at more broadly. He proposes eight criteria: topicality, comprehensiveness, breadth, exclusion, relevance, currency, avail- ability, and authority. (p. 5)

Hart’s criteria were adapted and incorporated into our 12-item scoring rubric, which can be grouped into five categories (see Table 1). (p. 5)

The first category, “Coverage,” consists of a single criterion that was not one of Hart’s. Criterion A assessed how well the au- thor of the dissertation justified criteria for inclusion and exclu- sion from review. (p. 5)

The second category, “Synthesis,” consists of Criteria B through G and is designed to gauge how well the author summarized, analyzed, and synthesized the selected literature on a topic. The individual criteria ask how well the author (B) distinguished what has been done in the field from what needs to be done, © placed the topic or problem in the broader scholarly literature, (D) placed the research in the historical context of the field, (E) acquired and enhanced the subject vocabulary, (F) articulated important vari- ables and phenomena relevant to the topic, and (G) synthesized and gained a new perspective on the literature. (p. 5)

here are interesting differences among the ways that authors search the literature and make decisions about suitabil- ity and quality. (p. 5)

Relative novices to a topic of the review, mea- sured by the number of previous publications on the topic, tend to be very explicit about their search strategies and criteria and are more likely to use databases and indexes to identify and se- lect research to review. Relative experts, on the other hand, tend to not be as explicit about their search strategies and criteria and often rely on personal communications with leading researchers as their main means of identifying relevant research (p. 5)

Criteria H and I constitute the third category, “Methodology.” Criterion H measures how well the author identified the main methodologies and research techniques that have been used in the field, and analyzed their advantages and disadvantages. (p. 5)

At minimum, an author should recognize how previous researchers’ methodological choices affected the research findings. Any sophisticated review of literature should also con- sider the research methods used in that literature and consider the strengths and weaknesses of those research methods (p. 5)

Literature Review Analysis Findings (p. 7)

After developing our rubric, we initially examined 30 dissertations awarded in the year 2000 from three state-funded colleges of ed- ucation in the United States; we selected 12 of those dissertations for full analysis. (p. 7)

The fourth category, “Significance,” includes Criteria J and K, which measure how well the dissertation rationalized the practi- cal (J) and scholarly (K) significance of the research problem. (p. 7)

Our analy- sis of dissertation literature reviews supports Schoenfeld’s (1999) contention that doctoral students may not be learning what it means to make and justify educational claims.5 (p. 7)

The final category, “Rhetoric,” also consists of a single item, Criterion L, which measures whether the literature review was written with a coherent, clear structure that supported the re- view. (p. 7)

Such lack of sophistication does not bode well for the students’ ability to stay abreast of research in their field as teachers, administrators or leaders—let alone to lead productive research careers after re- ceiving their doctorates. (p. 7)

More generally, Granello (2001) has argued that focusing on the formal aspects of writing is a means of increasing students’ cog- nitive complexity, moving students from lower to higher levels of Bloom’s taxonomy of the cognitive domain (see also Libutti & Kopala, 1995). (p. 7)

These high-quality reviews lead us to believe that our cri- teria and standards are not unreasonable. (p. 7)

Taken together, these twelve criteria and associated stan- dards set ambitious expectations for doctoral dissertation liter- ature reviews. (p. 7)

Paltridge (2002) differentiates among four general dissertation formats, each of which is seen among education dissertations. (p. 8)

The traditional simple dissertation presents a single study in five chapters (p. 8)

What Paltridge (2002) calls the traditional complex format pre- sents several related studies, each presenting its own introduction, methods, results, and conclusions. (p. 8)

The topic-based dissertation is also often used in education, especially for theoretical, philosophical, humanities-based, and qualitative dissertations (Paltridge, 2002). (p. 8)

Refining Our Conception of Literature Reviewing (p. 8)

Finally, the compilation of research articles format for disserta- tions, advocated in these pages by Duke and Beck (1999), pre- sents a number of discrete articles often written in the format of journal articles, framed with introductory and concluding sec- tions. (p. 8)

One of the most common concerns raised about our research is whether the criteria and standards that we have developed should apply to the two types of doctoral degrees in education and to various dissertation formats. (p. 8)

indeed, it is quite unclear to us what, exactly, earning a doctorate might signify if one does not know the literature in one’s field. (p. 8)

The second concern often raised is whether our criteria ought to apply to the various formats of dissertations. (p. 8)

Whatever format the author chooses, a thorough, sophisticated review ought to be influential and ev- ident in the entire dissertation. (p. 8)

It is productive insight that distinguishes a synthetic review, in Lather’s (1999) sense, from the plodding research summaries that characterize most dissertations. Productive insight can never be routine. But we contend that requiring doctoral candidates to engage in substantive, thorough, sophisticated literature reviews creates and fosters conditions that will greatly increase the likeli- hood of their developing productive insight. (p. 9)

Looking Forward (p. 9)

Doctoral students must be scholars before they are researchers. First and foremost, a dissertation should demonstrate a thorough and sophisticated grasp of one’s field of study; secondarily and antecedently, it should demonstrate the ability to do research that advances the collective understanding of important educa- tion issues. (p. 9)

As Bargar and Duncan (1986) write, a “thor- ough understanding and sincere commitment to problems of im- portance can and very often do lead to pedestrian, unimaginative solutions” (p. 35). (p. 9)

Yet the most obvious means of improving the situation— adding a class on literature reviewing to doctoral programs—is the least likely to be effective. (p. 9)

That is, to review the literature in the way that we have sug- gested here is a very complex task that requires the integration and application of a variety of skills and knowledge that few in- dividual faculty members have mastered. (p. 9)

We need to stress that a good literature review is necessary but not sufficient for good research. (p. 9)

For example, this approach to literature reviewing requires ad- vanced bibliographic methods for searching and locating re- search from a variety of sources (p. 10)

also requires diverse skills on the part of doctoral faculty. These include a substantive under- standing of the topic being reviewed, the skills and knowledge required to critically evaluate and synthesize concepts, advanced understanding of writing and rhetoric, and the sophisticated un- derstanding of research methodology that is required to critically evaluate methods used in prior studies and suggest means of overcoming prior methodological limitations. (p. 10)

Rather, literature reviewing should be a central focus of predissertation coursework, inte- grated throughout the program. (p. 10)

Our concern is that by focusing on methodological issues, the education research community is addressing the symptom rather than the cause. That is, researchers must understand prior research in their field, and its strengths and weaknesses, before they can be expected to choose appropriate methods of data collection and data analysis. (p. 10)

Only recently have libraries redefined library instruction, with its traditional emphasis on mechanical searching skills, to include information literacy, which employs a more conceptual approach to information use. (p. 10)

As a result, empiricism and methodological issues have been ascendant at the expense of scholarship, generativity, and theory building. Theorizing is fundamental to research and scholarship. (p. 10)

Taking the idea a step further, Isbell and Broaddus (1995) discuss the possibility of integrating writing instruction into the process. (p. 10)

Doctoral faculty expect doctoral students to possess not just bibliographic skills but also advanced skills in the mechanics of writing and the art of rhetoric. (p. 10)