Bodong Chen

Crisscross Landscapes

Notes: Zhang2016-ps-Understanding the dynamics of MOOC discussion forums with simulation investigation for empirical network analysis (SIENA)

2017-09-01


References

Citekey: @Zhang2016-ps

Zhang, J., Skryabin, M., & Song, X. (2016). Understanding the dynamics of MOOC discussion forums with simulation investigation for empirical network analysis (SIENA). Distance Education, 37(3), 270–286. https://doi.org/10.1080/01587919.2016.1226230

Notes

Interesting and well-grounded study of MOOC forums using SIENA. It adopts a hypothesis-testing paradigm. The discussion of the literation is extensive and useful. The findings could be further discussed. For example, I would like to see the authors’ discussion of why reciprocity declined across the time periods. More details about the course design and its potential impact on forum activities would also be useful. Overall, a nice study.

Highlights

Four network effects— homophily, reciprocity, transitivity, and preferential attachment—were tested to explain the dynamic mechanisms of interaction in the MOOC forum. (p. 2)

A major purpose of the discussion forum and its activities is to allow students to engage in an exploration of their ideas to develop their knowledge and understanding of the subject matter, hopefully resulting in improved learning. (p. 3)

A promising approach for the analysis of the dynamics of such free-flowing discussion forums is simula- tion investigation for empirical network analysis (SIENA), which enables insight into the changes of the studied network using a family of actor-driven models, that is, creating, maintaining, or terminating ties by an individual actor from other actors (Snijders, van de Bunt, & Steglich, 2010). (p. 3)

this study sought to measure the dynamic mecha- nisms by which networks are formed among MOOC participants. (p. 3)

Potential of network science for studying MOOC interactions (p. 4)

lassic types of social network analysis (SNA) studies (e.g., Xu, Zhang, Li, & Yang, 2015) have tended to be descriptive (p. 4)

More recent studies have realized the importance of exploring the changes in discussion forums. (p. 4)

or example, Yang, Sinha, Adamson, and Rosé (2013), in their preliminary work by applying SNA to measure the net- work graph at different points in time, examined how students interacted with their com- munity as it existed at the time of their entry. This type of research is by nature exploratory rather than hypothesis-driven. (p. 4)

Since 2014, collaboration between network scientists and educational researchers has fortuitously resulted in the development of an interdisciplinary approach to understanding the dynamics of networks in educational settings. As an example, the work by gillani et al. (2014) adopted complex network analysis to explore the vulnerability and information dif- fusion potential of the MOOC discussion forums. (p. 4)

Potentials of understanding network dynamics for social learning (p. 5)

The network view offers important contributions to knowledge about how learning is occurring among participants, as it provides measures such as hierarchy, clustering, or connectivity to describe what holds the network together (Haythornthwaite & De Laat, 2010). (p. 5)

learning is a social process which takes place beyond the individual mind (see Levy, 2011; Viswanathan, 2012), and SNA sits comfortably within this broad per- spective of social learning theory. (p. 5)

In the social-ecological systems literature, the concepts of sustainability, resil- ience, and robustness are implicitly similar or, at least, highly related terms (Berkes & Folke, 1998). These concepts refer to the capacity of a system to retain its basic functions when subject to unforeseen change. In MOOCs, if highly active learners drop out, the discussion network is likely to encounter a sudden change in its structure, and the discussion might shift in a way where others jump in to lead the conversation, move in a different direction, or fizzle out. (p. 6)

Effects of network dynamics (p. 6)

This study focuses specifically on using network configurations (reciprocity, transitivity, and preferential attachment) and actor covariant (homophily) to examine the overall change in relationships formed in a MOOC discussion forum, and explaining how these four network effects could be used as metrics to inform the design of a better social learning environment. (p. 6)

Hypothesis 1 (H1) There is a tendency towards reciprocation in the studied MOOC network (i → j and j → i). (dyadic level) (p. 7)

Hypothesis 2 (H2) There is a tendency towards network cohesiveness (i.e., increasing transitivity and reducing distance between actors; i → j, j → k and i → k). (triadic level) (p. 7)

Homophily, which is a term coined by Lazarsfeld and Merton (1954), refers to the tendency of individuals to associate with those similar to themselves. (p. 7)

Homophily is one of the most pervasive and robust tendencies of the way in which people interact with others (see McPherson, Smith-Lovin, & Cook, 2001). (p. 7)

Hypothesis 3 (H3) There is a tendency towards an increasing volume of interactions between students. (p. 8)

Preferential attachment represents the tendency of heavily connected nodes to receive more connections in a network. (p. 8)

Hypothesis 4 (H4) There is a tendency towards preferential attachment within the studied network. (p. 8)

The context of this study is the ‘Introduction to Psychology’ (30700313X) Spring 2014 course at Tsinghua University (https://www.xuetangx.com/courses/TsinghuaX/30700313X/_/about), which is a required undergraduate course for majors in the Department of Psychology. (p. 8)

As there was no group assigned by the instructor, the discus- sion forum served as an optional, open, and loosely structured environment for participants to communicate and collaborate if they wished to do so. (p. 9)

this course had a rather popular discussion forum where students asked and answered questions by others. About 5000 discussion messages were posted online during the first offering of the course (April 2014–September 2014). Some of these messages were in the form of enquiries about subject matter, exercises, course mate- rials, and the logistics of the course. Students also used the discussion forum as a platform to report on their own study, as well as seek out social activities. Some of the messages also included feedback on the course. (p. 9)

ach one of these discussions may consist of one or more posts, which might be responded to with replies or comments. (p. 9)

Method (p. 9)

This study adopted SIENA, which performs statistical estimation of stochastic actor-driven models for repeated measures of empirical networks (Snijders et al., 2010). The family of the dynamic actor-driven models assumes that participants who are represented by the nodes in the network play a crucial role in changing their connections to others (Steglich, Snijders, & Pearson, 2010). The behaviors of actors (treated as dependent variables) affect network dynamics, and in return, the network dynamics influence their behaviors, which is seen as a co-evolution of networks and behavior (Snijders, 2011). The method is implemented in the R package RSiena (SIENA version 4), developed by Snijders et al. (2010; https://www. stats.ox.ac.uk/~snijders/siena/). (p. 9)

The underlying assumption of actor-driven models to the analysis of data is that all net- work changes can be broken down into very small steps, so-called ministeps, in which one actor creates or terminates one outgoing tie. These ministeps are probabilistic and made sequentially. Whether an actor makes a ministep, as well as which ministep an actor will make, is estimated by means of an objective function that actors are assumed to optimize. (p. 9)

The objective function, which is used in this research, depends on two types of effects: structural and covariate. Structural effects capture endogenous network mechanisms. (p. 10)

the following structural effects were used: • Reciprocity is represented by the number of reciprocated ties (measure of mutuality). Reciprocity estimates the probability of participant B replying to participant A, given that A has replied to B. • Transitive triplets or transitivity is represented by the number of ties to participants who are the friends of my friends (measure of network closure; that is, transitivity estimates the probability of A replying to C, if A has replied to B, and B has replied to C). • Activity of alter is defined by the sum of the outdegrees of the others to whom the actor is tied (measure of activity attraction; that is, participants who have already received many replies are likely to be replied to by others). (p. 10)

Covariate effects estimate the network dynamics based on exogenous factors, for exam- ple, the role of actors. In this research, one dyadic constant covariate effect (same role) was used: • Same role is coded as 0 for students, and the non-student role (i.e., instructor and TA) was coded as 1. (p. 10)

Based on the conjectured hypotheses, three models were defined. (p. 10)

In SIENA, the conditional method of moments estimation (MoM) was used to decide how significant the structural and covariate effects were (Ripley, Snijders, Boda, Voros, & Preciado, 2015). The conditioning variable was the total number of observed changes (distance) in the network variable. (p. 10)

The evolving network was examined using SIENA with six time periods (see Observation time in Table 1), with approximately equal number of messages (i.e., posts, replies and com- ments) being posted in each time period. As interactions in MOOCs are asynchronous and replies to messages can occur later, the network grows with time. Thus, the accumulated number of links was used for analysis in each period. (p. 11)

Results (p. 11)

In this MOOC course, 1915 participants posted 5251 messages in total, of which 217 are original posts, 2553 are replies to the original posts, and 2481 are comments to the replies (see Table 2). (p. 11)

SIENA estimations (p. 11)

The rate parameter estimates the speed with which the dependent variable will change over two sequential time periods. (p. 11)

The reciprocity effect is significant and, in model 1, varies from 2.9055 to 5.8586 in different time periods. Thus, hypothesis H1, which asserts that there is a tendency to create reciprocal links, is accepted. (p. 13)

but is it true that the network became less reciprocal over these time periods? (p. 13)

The transitivity effect is significant with a positive coefficient ranging from 1.2575 to 3.1882. At the triadic level, it is confirmed (p. 13)

As shown in Table 3, same role is a significant covariate effect (p < 0.001) with the negative coefficients in six periods of obser- vational time. Thus, the considered network is heterophilic; that is, participants do not have a preference for creating links with those similar to themselves. (p. 13)

the definition of same-role is a bit narrow, because there are many other learner characteristics that could be considered (edu levels, regions, etc) and the learner-nonlearner contrast is not well balanced in number. (p. 13)

the activity of alter effect is significant ( p < 0.005), with a coefficient that is positive but relatively small: it varies from 0.0545 to 0.2920 in different time periods. (p. 13)

Discussion and conclusion (p. 13)