Bodong Chen

Crisscross Landscapes

Notes: Ali. (2012). A qualitative evaluation of evolution of a learning analytics tool



Citekey: @Ali2012

Ali, L., Hatala, M., Gašević, D., & Jovanović, J. (2012). A qualitative evaluation of evolution of a learning analytics tool. Computers & Education, 58(1), 470–489. doi:10.1016/j.compedu.2011.08.030



In our research, we have been specifically interested in investigating the use of semantic technologies for learning analytics. Semantic technologies can enable meaningful linking of students’ learning activities and their interactions with the learning contents as well as with the other participants of their learning processes. To develop such a learning analytics tool, we made use of the ontology framework, named LOCO (Learning Object Context Ontology), which we developed in our previous work to allow for modeling of learning contexts (Jovanovi c, Gasevi c, Knight, & Richards, 2007). The key notion of LOCO is learning context, which is defined as an interplay of a learning activity, learning content, and participants (e.g., learners and/or educators); thus, LOCO enables us to represent and meaningfully interlink learning context data from different learning environments, or from different services (e.g., chat rooms and discussion forums) inside the same learning environment. On top of the LOCO framework and by leveraging semantic annotation (Popov et al., 2003) of diverse kinds of learning resources, we developed a learning analytics tool LOCO-Analyst.7 (p. 471)

Teacher ADVisor (TADV) and Student Inspector are two examples of the tools, which focused on feedback provisioning through analysis of log data and visualization of the outcomes of such analyzes. The Teacher ADVisor (TADV) framework uses LCMS tracking data to extract student, group, or class models (Kobsa, Dimitrova, & Boyle, 2005). TADV relies on a set of predefined conditions to identify situations that require teachers’ intervention. When such conditions are satisfied, TADV informs teachers and provides them with an advice how to assist the students. TADV focuses on day-to-day activities of teachers. The Student Inspector tool analyzes and visualizes the log data of student activities in online learning (p. 471)

environments in order to assist educators to learn more about their students (Zinn & Scheuer, 2007). (p. 472)

For example, Zaine and Luo (2001) made use of novel data mining techniques for the analysis of access logs of an LCMS aiming at extracting and visualizing patterns that might be valuable for evaluating and interpreting learning activities in online courses. Similarly, TADA-Ed (Tool for Advanced Data Analysis in Education) combines different data visualization and mining methods in order to assist educators in detecting pedagogically important patterns in students’ assignments (Merceron & Yacef, 2005). These systems, however, focus exclusively on one type of learning activities (e.g., reading and exercises). The work presented in (BenNaim, Bain, & Marcus, 2009) aimed to develop a system that will assist educators to improve their feedback for students in adaptive tutorials, in cases when students encounter known “error-states” or when new “error-states” emerge. (p. 472)

CourseViz is developed to work with WebCT to visualize student tracking data (Mazza & Dimitrova, 2007); GISMO follows a similar approach for Moodle (Mazza, & Milani, 2005). Both of these tools try to assist educators in exploring social, cognitive, and behavioral insights of student activities in Web-based learning systems. (p. 472)

Although the above tools try novel ideas, it is not easy to judge how effective these are, unless a thorough evaluation is carried out. Only two systems mentioned above report their evaluation results at a sufficient level of details. (p. 472)

Generally speaking, both student and educator-centered feedback types provided by learning analytics tools can be divided into the two main groups (Melis & Ullrich, 2003): local and global. Typically, local feedback is an immediate response to some students’ activities (e.g., answering the questions of quizzes) in e-learning systems. On the other hand, global feedback is not necessarily immediate and is used to inform students and/or educators about a learning process as a whole or some of its sub-processes. (p. 472)

LOCO-Analyst is a learning analytics tool aimed at providing educators with feedback on the relevant aspects of the learning process taking place in a web-based learning environment, and thus helps them improve the content and the structure of their web-based courses. It provides educators with feedback regarding: (1) all kinds of activities their students performed and/or took part in during the learning process; (2) the usage and the comprehensibility of the learning content they had prepared and deployed in the LCMS; and (3) contextualized social interactions among students (i.e., social networking) in the virtual learning environment. (p. 473)

The generation of feedback in LOCO-Analyst is based on the analysis of user tracking data. These analyses are grounded in the notion of learning context, which is about a student (or a group of students) interacting with learning content by performing a certain activity (e.g., reading, quizzing, chatting) with a particular purpose in mind (Jovanovi c, Gasevi c, Knight, et al., 2007). The purpose of learning object context is to facilitate abstraction of relevant concepts from user-tracking data of various e-learning systems and tools. (p. 473)

LOCO-Analyst is implemented as an extension of Reload Content Packaging Editor,10 an open source tool for creating courses compliant with the IMS Content Packaging (IMS CP)11 specification. (p. 473)

  1. The first evaluation of LOCO-Analyst (2006) (p. 473)

RQ1. To what extent do educators perceive a learning analytics tool useful for improving their course content and instruction in their everyday practice? RQ2. To what extent does the user interface of a learning analytics tool impact the perceived value of the tool? (p. 473)

The study used the LOCO-Analyst tool and a questionnaire to evaluate all the features of the tool. (p. 474)

The questionnaire consisted of 16 items (i.e., questions): first three items asked participants to specify their role and experience; nine items were related to the tool’s feedback features and functionality; and remaining four items focused on the GUI and intuitiveness features. Eight of the items (listed in Table 2) had two parts: nominal scale responses (i.e., quantitative) and open-ended (i.e., qualitative) part prompting the participants to give their observations in the free text form. (p. 474)

Category Perceived usefulness of the tool for improving the course content/instruction Perceived value of the tool’s 3GUI General perception of the tool (p. 475)

we were able to categorize the themes and keywords into a coding scheme which consisted of three general categories: (1) Positive comment – expressing positive opinion without any concerns; (2) Positive comment with some observations – expressing positive opinion, but also has some observations that question some decisions or suggest some improvements; and (3) Negative comment– expressing either negative observations or some concerns questioning the decisions made in the design. As the participants’ comments were either related to the feedback type or the user interface, or were generic in nature, we further categorized the codes into 3 subcategories, namely: (1) Feedback feature – observations about specific feedback mechanisms supported by the user interface of LOCO-Analyst; (2) Intuitiveness – observations about the intuitiveness of the user interface; and (3) General comments – conceptual comments not necessarily directly applicable to the user interface and implementation of LOCO-Analyst. (p. 476)

A slightly different coding scheme was applied to the question, which sought suggestions for improving the tool’s GUI (Q8). Each response was broadly recognized as either a suggestion or a compliment. Both of these categories were further sub-divided as: (1) mainly related to improving data visualization/graphics; (2) mainly related to improving interface design; (3) mainly related to improving annotations; and (4) other. (p. 476)

the dominant area for improvement is data visualization (S1) (p. 478)

  1. Improvements in the second version of LOCO-Analyst (p. 478)

Table 5 A summary of suggestions participants gave in the 2006 survey for enhancing LOCO-Analyst’s features. No. Suggestion S1 Improve the data visualization. S1.1 Use graphics [for feedback presentation]. S1.2 Allow instructors to get insights by using different graphical chart, bar, XY diagrams etc. S1.3 Improve the visualization of analysis results [i.e., feedback] S1.4 Use more colors for representing data. S2 Improve the Graphical User Interface. S2.1 Keep the structure on the right hand side (RHS) of the tool window unchanged. [The RHS part of the tool’s user interface is used to display the feedback.] S2.2 Organize the contents in a way that the interface does not look overburdened. Try using more panels, windows, drop-down lists, if needed. S2.3 Use color codes and tool tips which pop up on hovers (such as on the hyperlinks) to give further idea of the feature or functionality available on clicking. S3 Further enhance the feedback. (p. 479)

The 2009 version of the LOCO-Analyst tool divided the user interface window into two vertical panels. These panels display the Reload editor on the left and the LOCO-Analyst functionalities on the right, as in the previous version. We used lower half of the left panel to display annotation feedback (Fig. 2A). (p. 479)

The major improvement in the new version was the way we present and communicate feedback to the educators (i.e., data visualization). (p. 480)

  1. The second evaluation of LOCO-Analyst (2009) (p. 480)

RQ1. To what extent the implemented interventions do affect the perceived value of a learning analytics tool? RQ2. To what extent the variables characterizing the perceived utility of a learning analytics tool are associated? (p. 480)

The study procedure was the same as in the 2006 study (Section 3.1.4), except hat besides the written instructions we also used video clips to introduce the tool and its functionality. (p. 481)

  1. Comparison of the two evaluations (p. 484)

This finding supported our expectations that visual representation of feedback would enable educators to quickly and easily get an overall insight into different aspects of the learning process. (p. 484)

As for the negative responses, we refer to two participants who questioned the usefulness of statistics. For instance, one of them stated: “you can determine that these students are passive, but how is this finding useful?” It is undisputable that statistics in isolation represent only one aspect of any real-world situation. To make more meaningful interpretations, we often need to look at the two or more statistics together. Therefore, LOCO-Analyst allows educators to consider different kinds of statistics related to either courses they teach or individual students attending their online courses. (p. 484)

In this section, we discuss two main types of threats to validity commonly analyzed in empirical research – internal and external validity. With respect to internal validity of our experiment, we are interested in checking if some confounding factors significantly influence the analysis of the collected data (Chin, 2001). In our studies, two main confounding factors are difference in experience with using similar tools and motivation. (p. 486)

This section is dedicated to the recent research that can be very valuable for the improvement of existing and introduction of new types of feedback provided by LOCO-Analyst. For example, Macfadyen and Dawson (2010) conducted an analysis of usage tracking data collected by an LCMS in order to determine the best predictors of students’ academic performance based on their activities in Web-based learning systems. Total number of posted discussion messages, sent email messages, and completed assessments are identified as the variables that can best predict (i.e., explain most of the variability of) the final grade of students. This empirical result is very important for LOCO-Analyst, as it can be directly applied to the feedback types listed in Table 1, and especially to those related to the analysis of social interactions. (p. 487)

Learning analytics has been recognized as an important research topic for workplace learning, as well. However, in that context, common assessment approaches (e.g., formal exams) are not easily applicable (Murray & Efendioglu, 2007). A very important work in this domain is presented in (Macfadyen & Sorenson, 2010) who proposed Learner Interaction Monitoring System (LIMS) (p. 487)

o be able to identify such patterns from the data, we intend to make use of the wellestablished pedagogical frameworks that define and explain interactions in educational settings. In particular, we intend to rely on the Community of Inquiry framework (Garrison, Anderson, & Archer, 2000) and the Interactivity Triangle (Anderson & Garrison, 1998), which we have already made use of in our recent research to: i) explore the role and potential benefits of social semantic web technologies on interactivity in online learning environments (Jovanovic, Gasevic, Stankovic, Jeremic, & Siadaty, 2009) and ii) investigate the role and influence that the concept of online presence (Jovanovic, Gasevic, Torniai, Bateman, & Hatala, 2009) could have on collaboration in personal learning environments.20 (p. 488)

we can conclude that the significance of this research is in providing clear evidence that a success of a tool for supporting educational processes can depend strongly on the visualization aspects of the system. Although this conclusion seems rather intuitive, in the technology-enhanced learning domain the focus over the last decade was predominantly on pedagogical, adaptive and responsive aspects of learning systems. We argue that an emergence of the learning analytics field is timely and further research is warranted on how we can utilize the existing knowledge from the human–computer interaction and visual analytics fields to improve the effectiveness of learning analytics (for both educators and learners) and study which particular interaction and visualization techniques work best for learning and education. (p. 488)