read

References

Citekey: @Mandinach2016

Mandinach, E. B. (2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 1–27. http://doi.org/10.1016/j.tate.2016.07.011

Notes

A foundational piece in the special issue laying out an evolving framework of Data Literacy for Teachers (DLFT). This paper introduces the history of the framework, presents its components, and discusses next steps different parties should take to foster data literacy among teachers.

Highlights

This article reports on the evolution of a conceptual framework for a construct called data literacy for teachers. Data use has become an emphasis in education but few educators have received sufficient training or preparation pertaining to data literacy skills. This article lays out the framework, identifying the specific knowledge, skills, and dispositions teachers need to use data effectively and responsibly. It concludes with a call to schools of education and teacher preparation programs to begin to integrate data literacy into curricula and practical experiences. (p. 1)

  1. Integrating data literacy into teacher preparation (p. 1)

Yet, education has struggled to define what it means to be a data literate educator (Mandinach & Gummer, 2011, 2013a, 2016; Schildkamp & Kuiper, 2010). Until recently, the field has paid more attention to assessment literacy (Brookhart, 2011; Christoforidou, Kyriakides, Antoniou, & Creemeers, 2013; Gotch & French, 2014; Kahl, Hofman, & Bryant, 2013; Mandinach, 2014; Plake, 1993) (p. 1)

Data-literate educators continuously, effectively, and ethically access, interpret, act on, and communicate multiple types of data from state, local, classroom, and other sources to improve outcomes for students in a manner appropriate to educators’ professional roles and responsibilities. (p. 1) (p. 1)

The North Carolina Department of Public Instruction is the only state in the United States that has deemed data use sufficiently important to develop a webpage on which they lay out to educators the meaning of data literacy. Data literacy is defined as “one’s level of understanding of how to find, evaluate, and use data to inform instruction” (p. 1). (p. 1)

Our work attempts to change the focus to all education data, not just assessment data, to provide a more comprehensive depiction of students. (p. 1)

The Data Quality Campaign’s definition (p. 1)

To fill that gap and address the need, we (Gummer & Mandinach, 2015; Mandinach & Gummer, 2016) have laid out a definition (p. 1)

data literacy for teachers (DLFT) (p. 2)

barriers and enablers to the enculturation of data in at least five European countries, research is beginning to emerge, reflecting the growing importance of data use globally (Schildkamp & Lai, 2013; Schildkamp et al., 2013). (p. 2)

Data literacy for teaching is the ability to transform information into actionable instructional knowledge and practices by collecting, analyzing, and interpreting all types of data (assessment, school climate, behavioral, snapshot, longitudinal, moment-to-moment, etc.) to help determine instructional steps. It combines an understanding of data with standards, disciplinary knowledge and practices, curricular knowledge, pedagogical content knowledge, and an understanding of how children learn. (p. 2)

Much attention from policymakers in the United States has been given to the importance of teachers using data. Former Secretary of Education Arne Duncan (2009a, 2009b, 2009c, 2010a, 2010b, 2012) has spoken widely about the need for teachers to use data and the importance of such evidence-driven practice. In fact, at a national conference sponsored by the Data Quality Campaign, Duncan (2012) publically challenged schools of education to focus on and accelerate efforts to educate educators to use data. Further, data use is one of the four pillars in the American Recovery and Reinvestment Act (ARRA, 2009) and in the Race to the Top (U.S. Department of Education, 2009), two major initiatives from the U. S. Department of Education. (p. 2)

Our work is grounded on three propositions that have emerged from our six years of studying data literacy. First, there is no question that educators must be armed with data. (p. 2)

Professional organizations such as the Council for Accreditation of Educator Preparation (CAEP) and the Council of Chief State School Officers (CCSSO) have included the capacity to use data among their recommendations and standards. The National Board of Professional Teaching Standards also has advocated for teachers’ data literacy (Aguerrebere, 2009; Thorpe, 2014). A Blue Ribbon Panel (2010) report released by NCATE, CAEP’s predecessor, and endorsed by Duncan (2010b) recommended that teacher candidates know how to make data-driven decisions. (p. 2)

Second, there is a problematic conflation between assessment literacy and data literacy. (p. 2)

Third and most relevant to teacher preparation around use data, our findings have indicated that simply relying on professional development to enhance teachers’ data literacy is inadequate (Mandinach & Gummer, 2012, 2013a, 2013b, 2016). (p. 2)

CCSSO (2013, 2015) released the InTASC (Interstate New Teacher Assessment and Support Consortium) standards for teaching that articulated 10 standards infused with data literacy, each standard identifying various forms of knowledge, dispositions, and performance skills that are required of teachers. (p. 2)

Thus, it is our belief based on our prior work, that teacher preparation programs must begin to introduce data use to teacher candidates, with professional development further enhancing the skills. (p. 2)

1.1. Positioning data use in policy and research (p. 2)

1.1.1. Policy and policymakers (p. 2)

This issue speaks to the conflation of assessment and data literacy, a major issue plaguing the comprehensive adoption of data literacy in education; that is, people think that data are only about assessments, and this is far from the case (Mandinach & Gummer, 2016; Mandinach et al., 2015). (p. 3)

The Data Quality Campaign, a bipartisan advocacy organization, has been a strong proponent of improving data literacy among educators. They have provided multiple supports and activities including: A paper, Empowering Teachers with Data: Policies and Practices to Promote Educator Data Literacy (Data Quality Campaign, 2014a). A national event to promote and increase awareness about data literacy was held (Data Quality Campaign, 2014b). A webinar to debate the differences between data literacy and assessment literacy (Data Quality Campaign, 2014c). (p. 3)

  1. Conceptual framework for data literacy for teachers (p. 3)

2.1. Evolution of the framework (p. 3)

Our first effort to understand data literacy involved convening 55 experts to determine how they defined the construct (Mandinach & Gummer, 2012, 2013a). We first examined how published materials, resources, and books dealt with data literacy. Second, we required each expert to provide a definition of data literacy. Third, we asked them to differentiate between data literacy and assessment literacy. (p. 3)

The review of the professional materials indicated that the texts focused on inquiry cycles, different types of data, and the role of data analysis to inform decisions. (p. 3)

1.1.2. Research (p. 3)

In some ways, policymakers are further along in their thinking about data literacy among educators than are researchers. (p. 3)

Many articles and studies have noted the importance for teachers to know how to use data effectively to inform their practice and the need to build educators’ capacity to use data (Baker, 2003; Choppin, 2002; Feldman & Tung, 2001; Hamilton et al., 2009; Ikemoto & Marsh, 2007; Mandinach & Honey, 2008; Mason, 2002; Mandinach, 2009, 2012; Miller, 2009). (p. 3)

the knowledge and skills were sorted into the following categories: inquiry process, habits of mind, data quality, data properties, data use procedural skills, transformation of data to information, and transformation of information to implementation. (p. 3)

In a subsequent project, we conducted an analysis of state licensure documents to understand how data literacy skills were described, if at all (Mandinach, Parton, Gummer, & Anderson, 2015a). (p. 3)

Though having good professional development is important, there also is a pressing need for the infrastructure to support the infusion of data use into schools and districts (Marsh, Pane, & Hamilton, 2006) and in teacher preparation programs (Duncan, 2012; Mandinach & Gummer, 2013b, 2016). (p. 3)

Our examination yielded interesting yet unsurprising findings. Assessment literacy was a more prevalent skill required by states than data literacy. However, some states articulated the knowledge and skills that pertain to data literacy as part of their requirements. Some states (Kansas, Kentucky, North Carolina, Ohio, South Dakota, Tennessee) even had an explicit data competency and a developmental continuum from novice to expert for data use. (p. 3)

While the existing literature focuses on the practicing cohort of teachers with an emphasis on in-service training, and professional development, little attention had been devoted to teacher preparation and data literacy in pre-service programs. (p. 3)

we convened a meeting of key stakeholders to discuss what schools of education can do to prepare educators to use data. The outcome of that meeting was a call to action (Mandinach & Gummer, 2011) that appeared in the Educational Researcher (Mandinach & Gummer, 2013b). (p. 3)

A message that emerged following our analyses of the data from the meeting with the experts and our review of licensure requirements is that for teachers, data literacy must be integrated with other essential aspects of teaching, namely knowledge of content domain and pedagogical content knowledge (Shulman, 1986, 1987). (p. 3)

We therefore conceptualized an interaction among three components, content knowledge, pedagogical content knowledge, and data use for teaching (what we termed the knowledge and skills around data use). (p. 4)

2.2. What DLFT looks like now (p. 4)

The conceptual framework (Mandinach & Gummer, 2016) therefore was elaborated to include the full scope of Shulman’s (1987) foundational knowledge. (p. 4)

We have now included seven key knowledge areas that integrate with data use in the inquiry process: content knowledge; general pedagogical knowledge; curriculum knowledge; pedagogical content knowledge; knowledge of learners and their characteristics; knowledge of educational contexts; and knowledge of educational ends, purposes and values. These areas provide essential information and feed into the data use for teaching domain of the DLFT construct. The data use for teaching domain is then comprised of five components under which we have associated specific knowledge and skills. The domains include: identify problems and frame questions, use data, transform data into information, transform information into a decision, and evaluate outcomes. (p. 4)

2.2.1. Identify problems and frame questions (p. 4)

The underlying knowledge and skills initiate the iterative inquiry process. Teachers should be able to: Articulate a problem of practice about a student, group of students, a topical area, the curriculum, or an aspect of instruction. They should be able to identify the problem and explain the issue or question. Understand the context at the student level to understand the problem with a student or a group of students. Contextualize the learning, behavioral, or motivation issues students may be having will help them to better understand the situation, concretizing the problem that can lead to a decision and subsequent action. (p. 4)

Understand the context at the school level. This is a different level of aggregation from student level context. Understand the larger context of a school in which teachers’ practice is embedded provides a broad view toward seeking solutions to problems of practice. Involve other participants or stakeholders, including students. Other educators, parents, and students can provide valuable insights into students’ performance. Consultation with them is an important part of the decision-making process. Understand student privacy. It is increasingly important for teachers to understand the regulations around the protection of student privacy and confidentiality. Data breaches have been increasing at alarming rates and education is not immune. This skill set includes knowing how to discuss data and with whom, understanding data sharing, among other topics. (p. 5)

2.2.2. Use data (p. 5)

The largest component in the inquiry cycle is Use Data. This component pertains to the fundamental knowledge and skills that most directly relate to actual data use. (p. 5)

We have identified 27 skills under the Use Data component. (p. 5)

Identify possible sources of data. Teachers must be able to evaluate the right data for the particular problem of interest, data that are aligned at the problem of practice and actionable. Understand the purposes of different data sources. Teachers must understand the purpose of different data because different data have different uses and utility. Understand how to generate data. Generating data is important because not all data are already produced. Teachers generate a plethora of data every day, ranging from moment to moment assessments of students to more long-term determinations of students’ understanding. Understand assessment. Teachers need to understand what makes a sound assessment. They must understand the different kinds of assessments, their purposes, and their uses. ▪ Use formative and summative assessments. Teachers must know how to use both formative and summative assessments, including understanding when which sources of data are appropriate for instructional versus assessment/grading decision making is a key skill for teachers. These sources of data yield different kinds of information that may be more or less aligned to instructional practice. ▪ Develop sound assessment design and implementation. Teachers must be able to develop assessments whether by hand or through the use of assessment systems. This set of knowledge and skills entails knowing how to design items and combine them into a meaningful test that yields reliable and valid data. ▪ Understand data properties. Data have characteristics and qualities that teachers need to understand and have alignment with the purposes of data use. Data have different levels (e.g., total score, composite, strand, item-level) and these levels may be more appropriate for different questions. ▪ Use multiple measures/sources of data. This is a foundational concept in data-driven decision making and educational measurement; that is, it is important not to rely on just one measure but to triangulate among multiple sources of data to obtain a better and more accurate depiction of the situation. ▪ Use qualitative and quantitative data. Most educators think of data as numbers that can be quantified. Data are much more. It is important to recognize that not all data are quantitative and that qualitative indices can be valued and informative sources. ▪ Understand specificity of data to question/problem. This set refers to the knowledge and skills teachers must have to understand that some data can address the issue being examined, while other data will not. ▪ Understand what data are appropriate. Teachers must be able to understand that not all data are appropriate or applicable for every given circumstance. It is important to recognize when specific data are or are not relevant for the problem at hand. Understand data quality. Data quality has many aspects such as validity, timeliness, and consistency of the data. Foundational to data use is knowing that the data being used are “clean”, timely in terms of from data collection to use, and valid for the purposes of use and interpretation. Data must not be misleading or out of range; that is, if the highest score possible is 100 and a score is entered as 110, teachers must recognize that there is a problem. Understand elements of data accuracy, appropriateness, and completeness. This is a subskill from the more general one about data quality. Data must be accurate. They also must be appropriate to the problem of practice or the question being addressed and as complete as possible. Using data that lack any of these qualities can invalidate the conclusions drawn from the data collection and analysis process. Understand how to access data. Teachers must know how data are stored and made available. They must be able to navigate across multiple data systems. Increasingly, educational data are stored in electronic formats for easy and safe access and analysis. Note that accessing data is different from generating data. The former entails data retrieval. The latter entails the actual creation of data. Find, locate, access, and retrieve data. Teachers must have the ability to locate the data needed to address a problem of practice or educational question and be able to pull out those data for subsequent examination. ▪ Use technologies to support data use. Teachers must know how to use technologies to support data use through data warehouses, assessment systems, student information systems, instructional management systems, data dashboards, simple spreadsheets, apps, and other relevant technologies that provide access to, analysis, and reporting of data. Understand how to analyze data. Teachers must understand what the data mean. Analyzing data enables the user to understand what the data mean, whether qualitative or quantitative. Analysis is one of the most foundational skills in data literacy. Understand statistics and psychometrics. Teachers need not be expert statisticians or psychometricans, but they must have a fundamental knowledge of both topics. By statistics we mean simple statistics such as central tendency and dispersion, not more advanced techniques like regression or ANOVA. By psychometrics, we mean understanding topics such as reliability, validity, and error of measurement. Manage data. Teachers must know how to manage data because there is such a multitude of data, that the wealth of data must be handled with accuracy, coded, stored, and arranged in a coherent manner for latter access and examination. Organize data into a meaningful and manageable representation of the information. Prioritize data because not all data are as relevant or as important as others. It is therefore important to arrange the data according to the utility of the issue being addressed. Examine data. Data examination means to scrutinize or inspect them in a meaningful way to address a particular question, hypothesis, or issue. (p. 5)

Integrate data by examining, analyzing, and combining them in some meaningful way for sense making. Manipulate data. Data manipulation entails handling or treating the data as part of the examination process. Drill down into data. Total test scores do not tell the complete story. Teachers need to drill down to strand or item levels to really understand misconceptions and understandings. Aggregate data. Teachers must understand that there are times when looking to whole group data is important. Disaggregate data. Teachers must understand the need to examine subgroup differences. They must know how to break down data into subgroups to discern differences across group performance. (p. 6)

Determine next instructional steps. Based on the evidence that teachers have acquired, they must use those data to plan and determine what are the next logical steps to take instructionally. Monitor student performance. Teachers need to be able to watch student performance over time to determine if differences have occurred or changed behavior. Diagnose what students need. Teachers must know how to determine students’ learning strengths and weakness from performance over time. Make instructional adjustments. Teachers must know how to make instructional adjustments based on data they have at hand. This means knowing what instructional actions are appropriate given the information gained from examining data. Understand the context for the decision. Teachers must understand the setting into which their decision is being fit. This includes knowing about the content, the curriculum, the scope and sequence, and other classroom contextual information. (p. 6)

2.2.3. Transform data into information (p. 6)

2.2.5. Evaluate outcomes (p. 6)

Nine skills were identified. (p. 6)

The final component in the conceptual framework pertains to examining the impact of the decision making process. (p. 6)

Consider the impact and consequences because the outcome of the inquiry process and decision is an impact on the circumstances that has consequences which can be either expected or unexpected. Generate hypothetical connections to instruction. Teachers must foresee how what they plan to do instructionally fits into the flow of students’ learning and progress. To do this, they must know how to project hypothetically, what will happen if they take various courses of action. This is based on hypotheses or educated guesses about why things are happening the way they are and what might happen with different instructional steps. Test assumptions early in the inquiry cycle to help determine if teachers are on the right track or off target. Understand how to interpret data. Teachers must know what the data mean. Interpretation gives meaning to the data and provides explanations. Understand and use data displays and representations. Teachers need to know how to use data displays because data are often graphically depicted, in chart, tables, graphs, and other displays. Assess patterns and trends. Teachers must be able to discern patterns and trends from data, especially when displays in charts and graphs. Probe for causality, as an attempt to understand why the behavior, performance, or situation has happened. Use statistics. Teachers must be able to use simple statistics such as central tendency and dispersion to understand student performance. Synthesize diverse data. Teachers must know how to synthesize diverse data because often times, disparate data are applied to a problem and must be pulled together in a coherent manner in order for them to make sense. Articulate inferences and conclusions from the information analyzed during the inquiry process. Summarize and explain data to pull together an explanation of what the data and information mean. (p. 6)

Re-examine the original question or problem. Teachers must recognize the need to look back at the original problem or question to determine if the data and decision have addressed the original issue and what steps need to occur informed by that re-evaluation. Compare performance preand post-decision. Part of the inquiry process is checking to see if there has been a change from before a decision was made and action taken to afterwards. This is to determine the impact of that decision making process. Monitor changes in classroom practices. Part of evaluating the outcomes is observing what has happened in the classroom based on the actions taken from the decision. Teachers must continue to be aware of the changes to determine if they have been in the desired direction. Monitor student changes in performance. Teachers must observe changes in their students following a decision to determine if the decision and ensuing intervention has had the desired effect. Consider the need for iterative decision cycles. Fundamental to the decision making process is the notion of iterative cycles of inquiry. This means that decisions do not have an end point. Data are collected, analyzed, interpreted, and acted upon. Then the cycle begins anew. Teachers must recognize the iterative nature of the teaching and learning process. (p. 6)

2.2.6. Dispositions, habits of mind, or factors that influence data use (p. 6)

Substantial research has been conducted in this area and reflects an international perspective (Datnow, Park, & Kennedy-Lewis, 2012; Jimerson, 2013; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; Schildkamp et al., 2013; Schildkamp & Kuiper, 2010; Schildkamp & Teddlie, 2008; Vanhoof, Verhaeghe, Van Petegem, & Valcke, 2012). These components are dispositions, beliefs, and habits of mind around data use (p. 6)

2.2.4. Transform information into a decision (p. 6)

Five skills were identified. (p. 6)

We are calling them dispositions as we agree with InTASC (2013) that these are more than just beliefs as they strongly structure what actions teachers undertake. Six dispositions were identified: (p. 7)

From our work, we recommend an integrated approach of data into pre-service preparation courses, especially because of the integration of Shulman’s (1987) seven areas of teacher knowledge. (p. 7)

Belief that all students can learn. Teachers must believe that all students have the capacity to learn and what actions they take can have a positive effect on student performance. Belief in data/think critically. Teachers need to believe in the use of data, that data can help them do their jobs more effectively and to think deeply about the problem at hand while using the inquiry cycle to use data to inform a decision. Belief that improvement in education requires a continuous inquiry cycle. Improvement is a continuous cycle of tweaking things to see if they work, observing outcomes, and making modifications as needed. Teachers need to understand that their work is not a linear, cause-and-effect process, but a cyclical and ongoing process. Ethical use of data, including the protection of privacy and confidentiality of data. Of the utmost importance is the knowledge about how to protect student data in terms of privacy and confidentiality. This skill is increasingly important now that technological applications are making data use more efficient but also somewhat riskier in terms of data security. Teachers must understand the fundamental ways to use data securely and responsibly. Collaboration (vertically and horizontally). Collaboration is considered an important and valued component in the inquiry process where educators work together to examine data and seek solutions to a particular problem. Communication skills with multiple audiences. Teachers must be able to discuss results and performance with various audiences, using empirical evidence in the discussion. Audiences include students, parents, guardians, other educators, and other relevant stakeholders. They must be able to adapt their communication to the particular audience. (p. 7)

It is our opinion that integration has the best chance of providing the triangulated approach. (p. 7)

3.2. How to integrate data literacy (p. 7)

At the heart of both formative assessment and data literacy for teachers is the teacher knowledge and skills of learners and their characteristics. Without an understanding informed by the learning sciences of how students think and act, the teacher is at a disadvantage in determining how to structure learning experience. (p. 7)

  1. Implications: a systemic perspective on change (p. 7)

This section presents looming questions and next steps that can be extrapolated from our work. (p. 7)

3.1. The timing of introducing data literacy (p. 7)

We now believe that the earlier in teachers’ careers, the better for the introduction of data-driven decision making. (p. 7)

Knowledge of educational contexts is key to understanding how to act. Data at the school and district level can be incorporated into educational foundation courses to help pre-service teachers with experiences of what they will encounter when they start to teach. (p. 7)

Schools of education are the central and the major player in the system. (p. 8)

Knowledge of education ends, purposes, and values is key to understanding from where the current focus on educational standards and tests has emerged. The data in the standards is not numerical, but the statements of what students need to know and be able to do that make up the Common Core State Standards in English Language Arts and Mathematics (Common Core State Standards Initiative, 2010; 2015) and the Next Generation Science Standards (NGSS Lead States, 2013) are at the heart of how teachers use data in the classroom to determine how well students are attaining educational goals and objectives. (p. 8)

Data literacy is only one of many topics that now must be addressed as teacher preparation programs are reconsidered and restructured for continuous improvement. (p. 8)

State education agencies and their licensure components play a major role in the system. They design the regulations and requirements around the skills and knowledge teachers are required to have, and therefore what schools of education are required to provide to teacher candidates. (p. 8)

We strongly considered leaving out general pedagogical knowledge and curriculum knowledge from our framework. Yet we asked ourselves how a teacher could use data to inform instruction without knowledge what curriculum is being used and how that curriculum relates to others. (p. 8)

School districts impact what teacher preparation programs do by exerting pressure to produce teacher candidates with certain skill sets to meet local needs. (p. 8)

In part, where we have emerged on these two issues reflects why schools of education and teacher preparation programs have a unique role in developing the capacity among teachers to use data effectively. (p. 8)

Testing organizations in the United States produce tests that teacher candidates must pass to obtain their certification. We know that the three major tests, Praxis and NOTE produced by Educational Testing Service (Sykes & Wilson, 2015), and edTPA (SCALE 2013) have introduced components that require test takers to demonstrate their ability to use and knowledge of data use concepts. (p. 8)

Professional organizations can leverage the importance of data literacy by impressing upon their members the need for educators to use data effectively and responsibly. (p. 8)

3.3. A systemic approach (p. 8)

We will summarize here some of the major players in the United States that are needed to facilitate systemic change. (p. 8)

Professional development providers can assist schools of education through the provision of existing materials and resources. (p. 9)

A final player in the system is the U.S. Department of Education. Federal policymakers have made it known that teachers must use data and that education must become an evidence-driven profession. Yet the Department has done little to improve the capacity of educators to use data, short of providing the technological infrastructure at the state level to support data use. (p. 9)

3.4. Next steps (p. 9)

First, there is a need for more research. We do not intend to have our conceptual framework be a static model. It will evolve and be modified over time as more knowledge of the field becomes further clarified. (p. 9)

importance of protecting the privacy and confidentiality of student data (Mandinach et al., 2015a; Phi Delta Kappa, 2015). (p. 9)

Second, research could inform the field about how best to address the differing needs of teachers and teacher candidates across the developmental career continuum. (p. 9)

Third, research could address how role-based data literacy is; that is, how the knowledge and skill sets might differ for teachers, administrators, and other educators. (p. 9)

Fourth, it is unclear what the continuum of data literacy looks like. (p. 9)

In terms of practical applications, schools of education will need assistance as they introduce data literacy into their programs. Faculty will need help as they plan how to integrate data literacy concepts into multiple courses. This will require retooling existing course materials. It will require faculty to acquire their own data literacy. The virtual course and the MOOC mentioned above can provide resources. However, we strongly advocate for the development of materials and resources that can be integrated into courses to minimize the potential resistance from faculty members. (p. 9)

Finally, it is important for a strengthening of the discourse and messaging around data literacy in teacher preparation programs to occur. (p. 9)

Blog Logo

Bodong Chen


Published

Image

Crisscross Landscapes

Bodong Chen, University of Minnesota

Back to Home