Citekey: @Worsley2012

Worsley, M. (2012). Multimodal learning analytics. In Proceedings of the 14th ACM international conference on Multimodal interaction - ICMI ’12 (p. 353). New York, New York, USA: ACM Press. doi:10.1145/2388676.2388755


Worsley’s brief description of his plan of dissertation work.


In order to develop a more appropriate measure and means for assessing SPBL I want to construct a complete picture of how learning takes place in SPBL environments through the use of multimodal learning interfaces and techniques. (p. 1)

Furthermore, this research builds on the work that I have done during the past three years which has found that there are meaningful cues in student speech [15][16][17], gaze [18], programming state [19]; epistemological beliefs and identity [20]; and in a combination of modalities [21][22]. It also builds on a number of published and unpublished research tools that enable: object tracking [23]; user localization [24]; and multi-modal data capture of collaborative work [25][26][27]. To this end, my previous research has looked at a variety of modalities in isolation, but as I move forward into my dissertation work I want to begin to better leverage the integration of different data streams in extended analyses of student learning. (p. 1)

will be studying high school and undergraduate students as they participate in engineering design workshops. During these workshops students do bifocal modeling, computational modeling, digital fabrication, robotics and introductory electronics, computer programming, wood working, polymer casting and more. (p. 2)

Digital Design Drawing Data Using digital pens and paper, I will capture continuous inking streams of student drawings. This will enable me to have the full evolution of their designs in a high precision fashion. These drawings will be aligned with speech data. (p. 2)

Student Gaze Data, Explanations and NetLogo logs – students will be asked to participate in a pair of studies in which they explore STEM phenomena in Netlogo, an agent based modeling environment. I will log their gaze, verbal explanations and Netlogo actions through synchronous data capture. (p. 2)

Student Wifi-based Localization – Using a custom Android application, I will capture student’s relative locations at halfsecond increments as they move around the lab. This information will be useful for studying relative student collaboration, as well as for bootstrapping other data streams. (p. 2)

Student Motivation and Sentiment – This data will come from a mobile phone survey platform. Students will be asked multiple choice, likert scale and free response questions through periodic polling. This information is useful for grounding some of the interpretations that I make from looking at students speech and actions near the time of polling. (p. 2)

Student Dialogue Capture – An array of microphones arrays, lapel and head-mounted microphones will be used to capture student dialogue throughout the lab. I am still trying to find the best solution to do this synchronously, in a relatively large space, and for a large number of users. (p. 2)

Student Location Capture – Using Kinect Sensors and a Teachscape Panoramic Camera, I will capture student location and actions. (p. 2)

Blog Logo

Bodong Chen



Crisscross Landscapes

Bodong Chen, University of Minnesota

Back to Home