Citekey: @Gasevic2014-ki

Gasevic, D., Kovanovic, V., Joksimovic, S., & Siemens, G. (2014). Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative. The International Review Of Research In Open And Distributed Learning, 15(5), 134–176.


Summarize: This review is unique compared to typical literature analysis as it focuses on grant proposals submitted to the MRI initiative. Applying a wide range of analytical techniques, this paper uncovers key themes, leading authors (being cited in proposals), representation of disciplines, status of interdisciplinarity, and the impact of the breadth of coverage (defined by SNA measures of resulting citation networks within clusters of disciplines) on likelihood of being funded.

Assess: Unique and interesting analysis tackling on various research questions. Helpful to be considered together with Veletsianos (2015).

One interesting thing with the citation network in this paper, is that it treats co-authorship and citation with equal status when constructing networks. A future direction is to decouple them.


This paper reports on the results of an analysis of the research proposals submitted to the MOOC Research Initiative (MRI) funded by the Gates Foundation and administered by Athabasca University. (p. 1)

The results revealed the main research themes that could form a framework of the future MOOC research: i) student engagement and learning success, ii) MOOC design and curriculum, iii) self-regulated learning and social learning, iv) social network analysis and networked learning, and v) motivation, attitude and success criteria. (p. 1)

The submissions were dominated by the researchers from the field of education (75% of the accepted proposals). Not only was this a possible cause of a complete lack of success of the educational technology innovation theme, but it could be a worrying sign of the fragmentation in the research community and the need to increased efforts towards enhancing interdisciplinarity. (p. 1)

Distance education and online learning have been clearly demonstrated to be an effective option to traditional classroom learning1. (p. 2)

This paper is an exploration of MOOCs; what they are, how they are reflected in literature, who is doing research, the types of research being undertaken, and finally, why the hype of MOOCs has not yet been reflected in a meaningful way on campuses around the world. (p. 2)

we are confident that the conve rsation about how MOOCs and online learning will impact existing higher education can be moved from a hype and hope argument to one that is more empirical and research focused. (p. 2)

Massive Online Open Courses (MOOCs) (p. 3)

The public conversation following this MOOC was unusual for the education field where innovations in teaching and learning are often presented in university press releases or academic journals. (p. 3)

Research communities have also formed around learning at scale 4 suggesting that while the public conversation around MOOCs may be fading, the research community continues to apply lessons learned from MOOCs to educational settings. (p. 3)

As such, there are potential insights to be gained into the trajectory of online learning in general by assessing the citation networks, academic disciplines, and focal points of research into existing MOOCs. (p. 3)

MOOC Research (p. 3)

Much of the early research into MOOCs has been in the form of institutional reports by early MOOC projects (p. 3)

Recently, some peer reviewed articles have explored th e experience of learners (Breslow et al., 2013; Kizilcec, Piech, & Schneider, 2013; Liyanagunawardena, Adams, & Williams, 2013). (p. 3)

We settled on using the MOOC Research Initiative as our dataset. (p. 3)

MOOC Research Initiative (MRI) (p. 4)

Specific topic areas that the MRI initiative targeted included: i) student experiences and outcomes; ii) cost, performance metrics and learner analytics; iii) MOOCs: policy and systemic impact; and iv) alternative MOOC formats (p. 4)

Research Objectives (p. 4)

In this paper, we report the findings of an exploratory study in which we investigated (a) the themes in the MOOC research emerging in the MRI proposals; (b) research methods commonly proposed for use in the proposals submitted to the MRI initiative, (c) demographics (educational background and geographic location) characteristics of the authors who participated in the MRI initiative; (d) most influential authors and references cited in the proposals submitted in the MRI initiative; and (e) the factors that were associated with the success of proposals to be accepted for funding in the MRI initiative. (p. 4)

Methods (p. 4)

Content Analysis (p. 4)

both automated a) and manual b) content analyses. (p. 4)

Automated content analysis of research themes and trends. (p. 5)

through the use of scientometric methods (Brent, 1984; Cheng et al., 2014; Hoonlor, Szymanski, & Zaki, 2013; Kinshuk, Huang, Sampson, & Chen, 2013; Li, 2010; Sari, Suharjito, & Widodo, 2012) (p. 5)

Among different techniques, t he one based on the word co-occurrence (p. 5)

has been gaining the widespread adoption in the recent literature reviews of educational research (Chang, Chang, & Tseng, 2010; Cheng et al., 2014). (p. 5)

Specifically, we based our analysis approach on the work of Chang et al. (2010) and Cheng et al. (2014). Generally speaking, our content analysis consisted of the three main phases: 1. extraction of relevant key concepts from each submission, 2. clustering submissions to the important research themes, and 3. in- depth analysis of the produced clusters. (p. 5)

For extraction of key concepts from each submission, we selected Alchemy, a platform for semantic analyses of text that allows for extraction of the informative and relevant set of concepts of importance for addressing research objective c, (p. 5)

This allowed us to rank the concepts and select the top 50 ranked (p. 5)

After the concept extraction, we used the agglomerative hierarchical clustering in order to define N groups of similar submissions that represent the N important research themes and trends in MOOC research, as aimed in research objective c. (p. 6)

our representation of each submission was a vector of concepts that appeared in a particular submission (p. 6)

a large submission-concept matrix where each row represented one submission, and each column represented one concept, while the values in the matrix (MIJ) represented the relevance of a particular concept J for a document I. (p. 6)

we used the popular cosine similarity which is essentially a cosine of the angle θ between the two submissions in the N-dimensional space defined by all unique concepts. (p. 6)

We used the GAAClusterer (i.e., Group Average Agglomerative) hierarchical clustering algorithm from the NLTK python library (p. 6)

Finally, in order to assess the produced clusters and select the key concepts in each cluster, we created a concept-graph consisting of the important concepts from each cluster. The nodes in a graph were concepts discovered in a particular cluster, while the links between them were made based on the co-occurrence of the concepts (p. 7)

Content analysis of important characteristics of authors and submissions. (p. 8)

The content analysis afforded for a systematic approach to collect data about the research methods and the background of the authors. (p. 8)

Specifically, each submission was categorized into one of the four categories (p. 8)

  1. qualitative method, which meant that the proposal used a qualitative research method such as grounded theory; 2. quantitative method, which meant that a proposal followed some of the quantitative research methods on data collected through (Likert-scale based) surveys or digital traces recorded by learning platforms in order to explore different phenomena or test hypotheses; 3. mixed-methods, which reflected a research proposals that applied some combination of qualitative and quantitative research methods; 4. other, which comprised of the research proposals that did not explicitly follow any of these methods, or it was not possible to determine from their content which of the three methods they planned to use. (p. 8)

their home discipline and the geographic location (p. 8)

Citation Analysis and Success Factors (p. 9)

It entailed the investigation of the research impact of the authors and papers cited in the proposals submitted to the MRI initiative (Waltman, van Eck, & Wouters, 2013). In doing so, the counts of citations of each reference and author, cited in the MRI proposals, are used as the measures of the impact in the citation analysis. (p. 9)

Citation network analysis – the analysis of s0 -called co-authorship and citation networks have gained much adoption lately (Tight, 2008) – was performed in order to assess the success factor of individual proposals to be accepted for funding in the MRI initiative (p. 9)

Social network analysis was used to address research objective e. In this study, social networks were created through the links established based on the citation and co- authoring relationships, (p. 9)

Actors occupying central networks nodes are typically associated with the higher degree of success, innovation, and creative potential (Burt, Kilduff, & Tasselli, 2013; Dawson, Tan, & McWilliam, 2011). (p. 9)

Nodes in the network represent the authors of both submissions and cited references, while links are created based on the co-authorship and citing relations. (p. 10)

more ideally could be a two-mode social network (p. 10)

We created a citation network for each cluster separately and analyzed them by the following three measures (p. 10)

All social networking measures were computed using the Gephi (p. 10)

Phase 1 Results (p. 11)

In total, there were nine research themes with a sim ilar number of submissions, from 19 (i.e., “Mooc Platforms” research theme) to 40 (i.e., “Communities” and “Social Networks” research themes). (p. 11)

More than half of the papers from the “Social Networks” research theme moved to the second phase and finally 25% of them were accepted for funding, while none of the submissions from the “Education Technology Improvements” theme was accepted for funding. (p. 11)

the most common research methodology type is mixed research, while the purely qualitative research is the least frequent. (p. 16)

Phase 1 citation analysis. (p. 17)

Figure 2 shows the list of most cited authors, while Table 8 shows the list of most cited papers in the first phase of the MRI initiative. (p. 17)

Phase 1 success factors. (p. 19)

We looked at the correlations between the centrality measures of citation networks (Table 9) and the second phase acceptance rates. (p. 19)

Interesting approach. Why would they be correlated? (p. 19)

These results confirmed the expectation stated i n the citation analysis section that research proposals with the broader scope of the covered literature were more likely to be assessed by the international review board as being of higher quality and importance. (p. 19)

Interesting interpretation. If the networks were constructed for each cluster, the size of the cluster could be a varialbe as well. Also, the interpretation of diameter/path length could go either way: coverage, or disconnectedness. (p. 19)

Phase 2 Results (p. 20)

Phase 2 research themes. (p. 20)

Research theme 1: engagement and learning success (p. 21)

The main topics in this cluster are related to learners’ participation, engagement, and behavioral patterns in MOOCs. (p. 21)

Research theme 2: MOOC design and curriculum (p. 23)

Research theme 3: Self-regulated learning and social learning (p. 24)

Analyzing cognitive (e.g. , memory capacity and previous knowledge), learning strategies and motivational factors, the proposals from this cluster aimed to identify potential trajectories that could reveal students at risk. Moreover, this cluster addressed issues of intellectual property and digital literacy. (p. 24)

Research theme 4: SNA and networked learning (p. 24)

Research theme 5: Motivation, attitude and success criteria (p. 25)

Phase 2 research methods. (p. 25)

mixed methods was the most common methodological approach followed by purely quantitative research (p. 25)

Research design. (p. 25)

Phase 2 demographic characteristics of the authors. (p. 26)

Phase 2 citation analysis. (p. 27)

At the centre of the network is L. Pappano, the author of a very popular New York Times article “The Year of the MOOC” (p. 27)

We also analyzed citation networks for each research theme independently and extracted common network properties such as diameter, average degree, path and density (Table 17). (p. 27)

Discussion (p. 30)

Emerging Themes in MOOC Research (p. 30)

The results also confirmed that social aspects of learning in MOOCs were the most successful theme in the MRI initiative (see Table 9). (p. 31)

Research Methods in MOOC Research (p. 32)

The high use of mixed methods is a good indicator of sound research plans that recognized the magnitude of complexity of the issues related to MOOCs (Greene, Caracelli, & Graham, 1989) . (p. 32)

Our results revealed that most of the proposals planned to use conventional data sources and data collection methods such as grades, surveys on assessments, and interviews. (p. 33)

learning analytic and educational data mining (LA/EDM) (Baker & Yacef, 2009; Siemens & Gašević, 2012) (p. 33)

Interestingly, the most successful themes (Clusters 3-4 in Phase 2) in the MRI initiative had a higher tendency to use the LA/EDM methods than other themes. (p. 33)

Importance of Interdisciplinarity in MOOC Research (p. 33)

an overwhelmingly low balance between different disciplines (p. 33)

ould this be a sign of the networks to which the leaders of the MRI initiative were able to reach out? Or, is this a sign of fragmentation in the community? Although not conclusive, some signs of fragmentation could be traced. (p. 33)

learning at scale indicate that the conference was dominated by computer scientists 8. (p. 34)

A fragmentation would be unfortunate for advancing understanding of a phenomenon such as MOOCs in particular and education and learning in general, which require strong interdisciplinary teams (Dawson et al., 2014). (p. 34)

Conclusions and Recommendations (p. 34)

Research needs to come up with theoretical underpinnings that will explain factors related to social aspects in MOOCs that have a completely new context and offer practical guidance of course design and instruction (e.g., Clusters 2, 4, and 5 in Phase 2). (p. 34)

The connection with learning theory has also been recognized as another important feature of the research proposals submitted to MRI (p. 35)

The new educational context of MOOCs triggered research for novel course and curriculum design principles as reflected in Cluster 2 of Phase 2. Through the increased attention to social learning, (p. 36)

Finally, it is important to note that the majority of the authors of the proposals submitted to the MRI were from North America, followed by the authors from Europe, Asia, and Australia. (p. 36)

Blog Logo

Bodong Chen



Crisscross Landscapes

Bodong Chen, University of Minnesota

Back to Home