read

References

Citekey: @Bannert2014

Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacognition and Learning, 9, 161–185. doi:10.1007/s11409-013-9107-6

Notes

A nice article applying Process Mining (PM) on the study of self-regulated learning. Starting from discussing the importance of studying temporal and sequential information in self-regulated learning, this article moves on to introduce the key ideas of PM. Then it illustrates the application of PM, using a software called ProM v5, to study a group of students’ learning in a hypermedia environment. The results mainly include comparisons, presented (visualized) in models, between two extreme groups representing successful vs. unsuccessful students. (The models look like results from lag-sequential analysis, even though they may be quite different according to the authors; the distinction is not that clear to me at this moment.) Overall, a successful one-stone-two-birds attempt to (1) introduce PM, and (2) investigate self-regulated learning. Many insights valuable for my ongoing work on self-regulated learning in MOOCs.

Highlights

Abstract Referring to current research on self-regulated learning, we analyse individual regulation in terms of a set of specific sequences of regulatory activities. Successful students perform regulatory activities such as analysing, planning, monitoring and evaluating cognitive and motivational aspects during learning not only with a higher frequency than less successful learners, but also in a different order—or so we hypothesize. Whereas most research has concentrated on frequency analysis, so far, little is known about how students’ regulatory activities unfold over time. Thus, the aim of our approach is to also analyse the temporal order of spontaneous individual regulation activities. In this paper, we demonstrate how various methods developed in process mining research can be applied to identify process patterns in self-regulated learning events as captured in verbal protocols. We also show how theoretical SRL process models can be tested with process mining methods. Thinking aloud data from a study with 38 participants learning in a self-regulated manner from a hypermedia are used to illustrate the methodological points. (p. 161)

regulation should be seen in terms of events rather than in terms of traits and aptitudes. The phenomena worth explaining in this perspective are those relating to “the very actions that learners perform, rather than descriptions of those actions or of mental states that actions generate” (Winne 2010, p. 269). In need of explaining are regularities and patterns on the event level, rather than differences between learners in respect to their aptitudes (Winne and Perry 2000). Differences in learners’ (meta-/cognitive, motivational, emotional) dispositions are seen as (p. 161)

less relevant than before for SRL research, with the possible exception of epistemic beliefs (Greene et al. 2010). Methodologically, this change in perspective has been progressing with an increasing focus on behavioral and verbal process data, and a decreasing interest in questionnaire methods (Azevedo 2009; Bannert 2009; Veenman et al. 2006). The interest in event data has been fueled by technical advances that make the recording of learning-related behavior on a level close to quantitative analysis almost effortless for the researcher, and largely un-obtrusive for the learners (e.g., Winne and Nesbit 2009). (p. 162)

The contribution we hope to make with this paper is to demonstrate how a specific class of analysis methods known as Process Mining (PM) can be applied to a type of data often found in SRL research—(coded) think aloud data. We argue that PM methods are appropriate in cases where, for theoretical reasons or qua instructional design, students’ regulative behavior can be seen as being driven by a holistic model of a process, akin to what Winne (2010, p. 270) calls a “patterned contingency”. (p. 162)

Components and processes of self-regulated learning (p. 162)

Self-regulated learning involves a complex interplay of cognitive, metacognitive, and motivational regulatory components (Boekaerts 1997). According to recent theoretical approaches, regulatory activities during learning include orientation in order to obtain an overview over the task and resources, planning the course of action, evaluating the learning product and monitoring and controlling all activities. Research has revealed that successful learning corresponds with all of these regulatory activities (e.g., Azevedo et al. 2004; Bannert 2009; Manlove et al. 2007; Moos and Azevedo 2009). (p. 162)

Commonly, cycles with the phases of forethought–performance–reflection (Zimmerman 2000) are distinguishable, even though more elaborated models exist (e.g., Winne 1996; Borkowski 1996). In this research, we are building on previous work on the temporal structure of self-regulated learning (e.g., Biswas (p. 162)

et al. 2010; De Jong 1994; Hadwin et al. 2007; Kapur 2011; Schoor and Bannert 2012; Winne and Nesbit 1995). (p. 163)

In an exploratory case study Hadwin et al. (2007) showed differences in the ways eight selected students regulated their learning over time by means of activity transition graphs. (p. 163)

De Jong (1994) was able to show differences in learning sequences between high and low performing students by means of concordance analysis. He analysed the frequencies and patterns of self-regulatory activities of successful and less successful students and found significant differences in respect to the kind and variability in learning patterns. That is, the manner in which successful students used regulation activities (especially testing and monitoring activities) in much more flexible ways than less successful students. He also found that different kinds of tasks were related to different sequences in self-regulatory activities. (p. 163)

Biswas et al. (2012) also identified different learning patterns for high vs. low performers by means of an exploratory data mining method. (p. 163)

Group performance was predicted by 30–40 % on learning activities that were shown at the beginning of learning. In a recent study using lagsequential analysis, Kapur (2011) found that different temporal patterns of collaborative problem-based learning were significantly related to different group performance. (p. 163)

Moreover, we conclude from these studies that students display a very high degree of variation regarding the sequential organization of their learning and regulation activities. Therefore, we argue that it is not appropriate to aggregate across all learners, but to compare subgroups that probably share a similar regulation behavior. (p. 163)

Furthermore, research on temporal patterns in SRL is at an early stage and largely explorative at the moment. In absence of prior research and knowledge about detailed expected effects, an analysis of extreme groups is a suitable approach (Preacher et al. 2005). (p. 163)

In general, we expect that process patterns of high performers are similar to the ideal behavior described by SRL models (e.g., Zimmerman 2000) and that they show more active regulation, whereas the patterns of less successful students should be far from optimal regarding an ideal order. (p. 164)

Process mining: how is it done and how has it developed? (p. 164)

There are numerous methods available for sequential and temporal analysis of SRL data (Sanderson and Fisher 1994; Langley 1999) so the question why it is necessary to introduce a new method needs to be answered. (p. 164)

With granularity, we are referring to the difference between treating a number of events that follow each other over time as (i) a sequence, as (ii) generated by a process, or as (iii) part of a narrative (Reimann 2009). (p. 164)

Our motivation to suggest process mining as a method in SRL research is that it allows for the expression of the assumption that a sequence of events is generated by a particular process of self-regulation in a formal manner, and also, to test assumptions about such processes. (p. 164)

Process mining (PM) refers to methods of data mining (Robero et al. 2010) that build on the notion of a process model. The purpose of engaging in PM is to identify, confirm or extend process models based on event data. PM is increasingly used in educational contexts, in particular in research on ICT-supported learning and teaching (Reimann and Yacef 2013; Trčka et al. 2010). (p. 164)

A process model in this sense does not refer to a cognitive architecture, nor does it mean that the processes described have to be mental or cognitive in nature. The reason that we nevertheless suggest to employ PM methods in the context of SRL research is that in principle, these methods are compatible with conceptualizations of SRL measured by event logs (Azevedo et al. 2010; Winne and Perry 2000) rather than as changes in (continuous) variables over time (Mohr 1982; Reimann 2009). (p. 164)

The type of model used in much of PM research is a very abstract one, namely variants of Petri Nets (Reisig 1985). (p. 164)

This is an important distinguishing feature of process mining compared to sequence mining, or sequential pattern analysis (Agrawal and Srikant 1995; Perera et al. 2009; Zhou et al. 2010); these methods, as well as stochastic methods such as lag-sequential analysis (Bakeman and Gottman 1997) do not contain the assumption of a (latent) process. (p. 165)

PM enable us to express that the process as a whole matters; it is a more holistic view of a process than a process-as-sequence view affords (Reimann 2009). The PM view of process is hence particularly relevant when we (p. 165)

have reasons—hopefully based on theory—to believe that the performance of the system that generates the event sequence is driven by something that resembles a plan for action. In the context of SRL, this ‘something’ can for instance, be a learning strategy (a mental structure), or can be a resource given to the learner from outside, such as prompts (Bannert 2009) or a worksheet. In group learning, the equivalent would be a collaboration script (Kollar et al. 2006). (p. 166)

Fortunately, very comprehensive software support is readily available for process mining analysis in form of the ProM workbench (http://www.processmining.org/prom/). This software package, maintained at the University of Eindhoven, provides access to a large range of process mining and process analysis algorithms, under a unified framework for data import. (p. 166)

Mapping
How to map SRL strategies (2000 article) with MOOC behaviors? I guess this is the first step, and an important step/contribution, to take/make in our paper. (p. 166)

Empirical study (p. 166)

Participants and procedure (p. 167)

Dependent on how quickly they adopted the thinking aloud methods, students were tasked with carrying out three to five search tasks using a program similar to the one employed in the learning session. (p. 167)

Learning material and learning performance measures (p. 167)

Learning outcomes were measured on three different levels based on Bloom’s Taxonomy of the cognitive domain (Bloom 1956): Knowledge of basic terms and concepts (free recall method), comprehension of basic terms and principles (multiple-choice test) and application of basic concepts and principles in new situations (transfer test). (p. 167)

Coding scheme for analysing the learning activities (p. 168)

Students’ verbal protocols were segmented and coded according to the coding scheme presented in Table 1. It is based on our theoretical framework of self-regulated hypermedia learning (Bannert 2007) and differentiates between the main categories metacognition, cognition, motivation, and others. Metacognition contains students’ utterances referring to orientation (ORIENT), goal specification (SETGOAL), planning (PLAN), searching information (SEARCH) and judgment of its relevance (EVALUATE), evaluation (EVAL), and finally, monitoring and regulation (MONITOR). Cognition contains the sub-categories reading (READ), repeating (REPEAT), and deeper processing, i.e., elaboration (ELABORATE) and organization (ORGANIZATION). The REPEAT event, when following a READ event, is best understood as ‘rehearsing’ the information read. The category motivation (MOT) includes all positiveand negative motivational utterances on task, situation or oneself. Finally others (REST) include task-irrelevant aspects such as questions to the experimenter about the overall procedure of the study or ambiguously utterances (‘not classifiable’) as well as phases of silence. (p. 168)

Results (p. 168)

Frequency analysis of coded learning events (p. 168)

Referring to research and theoretical assumptions of SRL processes (e.g. Zimmerman 2000), successful students should show an ideal regulatory behavior, whereas the process patterns of less successful students should be far from optimal. (p. 168)

SEARCH (p. 169)

EVALUATE (p. 169)

EVAL (p. 169)

MONITOR (p. 169)

READ (p. 169)

Process analysis of coded events (p. 171)

In a further step we analysed the coded verbal protocols by taking the temporal order of regulatory activities into account, using the ProM-software V 5.0 (2008). (p. 171)

The Fuzzy Miner algorithm, when applied to an event sequence, yields a transition diagram that is in essence quite comparable to the graphs suggested first in Winne and Nesbit (1995). (p. 171)

Parameters and metrics of the Fuzzy Miner (p. 172)

The Fuzzy Miner Algorithm uses this data to generate a complete model which consists of nodes (event classes) and edges (relations between event classes) by taking the relative importance and the temporal order of all events into account. The algorithm uses two fundamental metrics, significance and correlation, for computing a process model for the given data set. (p. 172)

Significance measures the relative importance of the occurrence of event classes and of relations between events. For example, events which occur more frequently are assessed as more significant (Günther and van der Aalst 2007). Correlation is only calculated for edges and it indicates “how closely related two events following one another are” (Günther and van der Aalst 2007, p. 333). The basic concepts of significance and correlation are embedded in a metrics framework which calculates three primary types of metrics: unary significance (of nodes/event classes), binary significance (of edges/relation of two event classes) and binary correlation (of edges). These metrics are described in detail by Günther and van der Aalst (2007) and summarized in Schoor and Bannert (2012). (p. 173)

Mining process models of successful and less successful students (p. 173)

As theoretical SRL models postulate, one can see in Fig. 2 that successful students show a variety of metacognitive activities (ORIENT, EVALUATE, PLAN, EVAL, MONITOR). For successful students reading (READ) is mainly connected with monitoring (MONITOR) and elaboration (ELABORATE), however repeating (REPEAT) is also listed in their process model. There is a double loop of monitoring (MONITOR), reading (READ) and elaboration (ELBORATE). Less successful students (Fig. 2, right part) do not only show less events (as already listed in Table 4), but they also display less event types in their process model. For them, reading (READ) is mainly connected with repeating (REPEAT). Reading is also strongly connected with monitoring (MONITOR), and repeating (REPEAT) is also connected with elaboration (ELABORATE). (p. 174)

Since current models of SRL operate mainly with the three phases of forethought– performance–reflection (i.e., Zimmerman 2000), we aggregated several of the coded categories in a further step as follows: The metacognitive activities orientation (ORIENT), planning (PLAN) and goal specification (SETGOAL) were combined into the new category analysis (ANALYSIS), and the cognitive activities elaboration and organization of information into the new category deeper processing (PROCESS). In addition, motivation (MOT) was excluded from further analysis due to low frequencies. (p. 174)

Successful students (left part of Fig. 3) show high frequencies of preparing activities (ANALYSIS) which are monitored (MONITOR). Reading (READ) is highly connected with monitoring (MONITOR) and deep processing (PROCESS). There are also evaluation activities (EVAL) which are connected to preparing activities (ANALYSIS). Moreover, there is a triple loop with analyzing (ANALYSE), monitoring (MONITOR), reading (READ) and deep processing (PROCESS). This model comes close to Zimmerman’s model of ideal SRL (2000). For less successful students the process model with aggregated categories (right part of Fig. 3) also includes 5 categories. However, their type, occurrence and relationships are quite different. Here we can see less preparing activities (ANALYSIS) which are connected to reading (READ). Reading (READ) is followed by repeating (REPEAT) and monitoring (MONITOR). Whereas successful students show deep processing activities when reading, the less successful students show mainly surface processing by repeating (REPEAT). There is a double loop between analysis (ANALYIS), reading (READ) and repeating (REPEAT). Their process model does not include evaluation (EVAL) which is different from successful students. (p. 174)

Illustrating model-testing approaches (p. 175)

Testing a full model (p. 176)

First, a process model was constructed in form of a Petri Net (Fig. 4), using WOPED (http://www.woped.org/) as the editor. WOPED saves Petri Nets in a format that can be imported into ProM. (p. 176)

It is a somewhat ‘ideal’ sequence of steps one would like to see learners engage in: (1) ANALYSE the learning task, (2) then perform any of the actions SEARCH, READ, PROCESS or REPEAT, (3) MONITOR your progress, (4) EVALuate if you are done, and stop or perhaps continue with a similar cycle. (p. 176)

In order to compare this model against the empirical trace data, we used the ProM module Conformance Checker; the methods that this software module implements, and the metrics it is able to calculate are described by Rozinat and van der Aalst (2008). The only metric used for our analysis is model fit. (p. 177)

Firstly, we applied the Conformance Checker to our data of the less successful students in order to compare the full model (Fig. 4) with our empirical data (sequences of coded think aloud data from 6 cases). This comparison yields a fitness value of f=0.41, which can be interpreted as the model accounting for approximately 40 % of the observed event sequences. (p. 177)

One visualization is trace-based: Fig. 5 shows the first process steps for all 6 less successful participants, highlighting mismatches with the model. (p. 177)

Another type of visualization is model-based (Fig. 6): the results of the analysis are over-laid on the model (see Fig. 4). The numbers on the edges of the graph correspond to numbers of traversals. The (red) numbers in the place representations (the circles) indicate how often the constraint at that place was not obeyed. (p. 177)

Figure 7 shows the model view for the data from the 5 successful students (762 events). The fitness metric here takes a value of 0.48. Hence, it is only marginally better than the value for the less successful group. We can interpret this finding as saying that the differences between the successful and the less successful group, as far as they are due to differences in the temporal structure of their performance, are not expressed in the model. The successful group does not adhere (overall) more to the model than the less successful group. (p. 178)

Testing a partial model (a sequence pattern) (p. 179)

In order to perform partial model checking, that is, in order to check for the existence of sequences of events, one needs a language for sequence patterns. ProM provides such a language, namely LTL (Language for Temporal Logic). In this language, one can formulate logical formulae akin to first-order predicate logic, such as: Always_when_A_then_B, or Eventually_A_next_B. (p. 179)

Discussion (p. 180)

In addition to introducing selected methods of process mining, the aim of this paper was to analyse students’ processes of self-regulated learning. Besides detecting differences in frequencies of SRL events, we wanted to gain insight into the temporal patterns in students’ self-regulated learning. Our analysis contrasted the processes of the most successful and least successful students and considered whether these extreme groups differed in their SRL activities during hypermedia learning. (p. 180)

From a methodological point of view we have only shown the tip of the iceberg: Process Mining comprises many other algorithms for sequence data and temporal data than those applied here, and also many more methods for model analysis and comparison. (p. 181)

Also, we have focused only on process logic; we have not taken into account data that capture aspects of the process environment (such as what it read in the hypertext), nor have we taken into account quantitative temporal aspects (duration of events). For both aspects, specific analysis and data mining methods are available in the ProM-Framework (e.g., Rozinat and van der Aalst 2006). (p. 181)

One of the disadvantages of process mining models as introduced here is that they are not directly related to statistical testing, such as significance testing. (p. 181)

We are not discussing these methods of lag-sequential analysis further at this juncture as they are already described in the psychological method literature (see Bakeman and Gottman 1997, for an excellent introduction) and also as they work with a sequence view of events, rather than a process view. (p. 181)

The kind of process measures considered is also of importance. Commonly, students’ navigation behavior is mined (Biswas et al. 2010; Winne and Perry 2000; Wirth 2004). (p. 182)

Love this paragraph!

We hesitate to recommend using the results of process mining or other event-based temporal analysis methods directly for instructional purposes. A strategy often suggested in educational data mining and learning analytics research is to use patterns in performance that have been found to discriminate between more and less successful learners directly for instruction. That is, in the sense that patterns found in more successful students are used as recommendations for future learners. This strategy is not only limited in scope—recommendations so derived can only work in the same environment and context in which they have been identified—but also raises ethical concerns; we know that inductively identified regularities can be spurious and brittle, and should critically reflect if we want to ground instructional advice on such a basis alone.

Blog Logo

Bodong Chen


Published

Image

Crisscross Landscapes

Bodong Chen, University of Minnesota

Back to Home