Bodong Chen

Crisscross Landscapes

Notes: Graesser - 2011 - Coh-Metrix



Citekey: @Graesser2011

Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011). Coh-Metrix: Providing Multilevel Analyses of Text Characteristics. Educational Researcher, 40(5), 223–234. doi:10.3102/0013189X11413260


An overview of measures provided by Coh-Metrix.


Two important caveats must be raised before describing the computer analyses of text characteristics. First, computers obviously cannot identify and scale texts on all levels of language, discourse, and meaning. Available computer systems cannot fully comprehend the deep metaphors, literary devices, and historical contexts of Shakespeare’s plays, for example. Some characteristics of texts require humans to provide informed, deep, critical analyses. Second, successful text comprehension involves much more than an analysis of text characteristics alone because prior knowledge, inference mechanisms, and skills of readers are also critically important (Graesser, Singer, & Trabasso, 1994; McNamara et al., in press; Pressley, Wood, Woloshyn, Martin, King, & Menke, 1992; Snow, 2002). (p. 224)

Traditional Computer Metrics ofText Ease/Difficulty (p. 224)

The traditional approach to scaling texts is to have a single metric of text ease or difficulty. This is the approach taken by the three most popular metrics, namely Flesch-Kincaid Grade Level or Reading Ease (Klare, 1974–1975), Degrees of Reading Power (DRP; Koslin, Zeno, & Koslin, 1987), and Lexile scores (Stenner, 2006). (p. 224)

Coh-Metrix Analyses on Multiple Levels of Language and Discourse (p. 224)

The simplicity of a single dimension of text difficulty (p. 224)

Coh-Metrix was developed to analyze texts on multiple characteristics and levels of language-discourse. The original inspiration in developing Coh-Metrix was to have an automated metric of text cohesion (hence the label Coh-Metrix). (p. 224)

The Flesch-Kincaid metrics (Grade Level and Reading Ease) are based on the length of words and length of sentences. For example, Flesch-Kincaid Grade Level is computed as 11.8 * Syllables + .39 * Words − 15.59. Word length is measured as the mean number of syllables per word (which is highly correlated with number of letters); sentence length is the mean number of words per sentence. (p. 224)

The measures of Coh-Metrix are aligned with various language-discourse levels proposed in multilevel theoretical frameworks (Graesser & McNamara, 2011; Kintsch, 1998; Snow, 2002). These theoretical frameworks identify the representations, structures, strategies, and processes at different levels of language and discourse. Five levels have frequently been proposed in these frameworks: (1) words, (2) syntax, (3) the explicit textbase, (4) the situation model (sometimes called the mental model), and (5) the discourse genre and rhetorical structure (the type of discourse and its composition). A sixth level of pragmatic communication between speaker and listener, or writer and reader, is often also included, but this level is not tightly connected with text characteristics and will not be addressed in this article. The five levels are elaborated later and in Graesser and McNamara (2011). (p. 224)

DRP and Lexile scores relate characteristics of the texts to performance of readers in a cloze task. In the cloze task, the reader receives the text with words left blank and is asked to fill in the words by generating them or by selecting a word from a set of options. A text is at the reader’s level of proficiency if the reader can perform the cloze task at a corresponding threshold of performance (e.g., 75%). (p. 224)

The large suite of measures ends up nicely funneling into factors that are aligned with Levels 1–5 of the multilevel theoretical framework (Graesser & McNamara, 2011). It is beyond the scope of this article to specify precisely how each measure is computed. (p. 225)

Words (p. 225)

Word frequency. Words that occur with a higher frequency in the English language are more familiar to the reader, are processed more quickly, and are linked to richer bodies of world knowledge (Beck et al., 2002; Haberlandt & Graesser, 1985; Perfetti, 2007). Word frequency in Coh-Metrix is computed from multiple corpora of texts that we have documented in Coh-Metrix publications. The logarithm of word frequency is computed because reading times are linearly related to the logarithm of word frequency, not to raw word frequencies (Haberlandt & Graesser, 1985; Just & Carpenter, 1987). (p. 225)

it is important to analyze words on multiple characteristics that have relevance to reading development and construction of meaning. (p. 225)

Parts of speech. Each word is assigned a syntactic part-of-speech category. These syntactic categories are segregated into content words (such as nouns, main verbs, adjectives, adverbs) and function words (e.g., prepositions, determiners, pronouns). (p. 225)

Psychological ratings. Coh-Metrix measures words on characteristics in an established psycholinguistic database (Coltheart, 1981), a collection of ratings of thousands of words along several psychological dimensions: age of acquisition, meaningfulness, concreteness, imagability, and familiarity. (p. 225)

Semantic content. The semantic content of words is analyzed by semantic categories provided by WordNet (Miller, Beckwith, Fellbaum, Gross, & Miller, 1990), a large lexicon with words annotated by experts on linguistic and psychological features. (p. 225)

Coh-Metrix computes the relative frequency of each word category (both syntactic and semantic categories, described later) by counting the number of instances of the category per 1,000 words of text. (p. 225)

Some syntactic categories have repercussions at deeper levels of comprehension. (p. 225)

Co-reference. Coh-Metrix tracks different types of word coreference: content word overlap, noun overlap, argument overlap, and stem overlap. (p. 226)

Syntax (p. 226)

Theories of syntax assign words to part-of-speech categories (e.g., nouns, verbs, adjectives, conjunctions), group words into phrases (noun phrases, verb phrases, prepositional phrases, clauses), and assign syntactic tree structures to sentences (Jurafsky & Martin, 2008). (p. 226)

Lexical diversity. Lexical diversity is related to cohesion because a higher number of different words in a text means that new words need to be integrated into the discourse context. The most wellknown lexical diversity index is type-token ratio, but a number of analogous measures are included in Coh-Metrix that control for text length (McCarthy & Jarvis, 2007). (p. 226)

Latent semantic analysis. Coh-Metrix measures conceptual overlap between sentences by a statistical model of word meaning called Latent Semantic Analysis (LSA; Landauer, McNamara, Dennis, & Kintsch, 2007; Millis, Magliano, & Todaro, 2006). (p. 226)

LSA is a mathematical, statistical technique for representing word and world knowledge, based on a large corpus of texts. The central intuition is that the meaning of a word is captured by the company of other words that surround it in naturalistic documents. Two words have similarity in meaning to the extent that they share similar surrounding words. (p. 226)

Sentence syntax is easier when there are shorter sentences, fewer words per noun phase, fewer words before the main verb of the main clause, and fewer logic-based words. (p. 226)

Textbase (p. 226)

The textbase consists of the explicit ideas in the text: the meaning rather than the surface code of wording and syntax (Kintsch, 1998; van Dijk & Kintsch, 1983). (p. 226)

content (i.e., main verbs that signal state changes, events, actions, and processes). (p. 227)

Situation Model (p. 227)

The situation model is the subject matter content or the narrative world that the text is describing. In narrative text, this includes characters, objects, spatial setting, actions, events, processes, plans, thoughts and emotions of characters, and other details of the story. In informational text, the situation model is the substantive subject matter being described. (p. 227)

Genre and Rhetorical Structure (p. 227)

Genre refers to the category of text, such as whether the text is narration, exposition, persuasion, or description (Biber, 1988; Pentimonti, Zucker, Justice, & Kaderavek, 2010). (p. 227)

Zwaan and Radvansky (1998) proposed five dimensions of the situation model that apply to narrative text: causation, intentionality (goals), time, space, and protagonists. (p. 227)

Cohesion is facilitated by these signaling devices, which clarify and stitch together the actions, goals, events, and states conveyed in the text (Louwerse, 2001). However, if the reader has enough background knowledge about the subject matter, the lack of cohesion devices may not disrupt comprehension and can sometimes improve comprehension by stimulating inferences (McNamara & Kintsch, 1996; O’Reilly & McNamara, 2007; Ozuru et al., 2009). (p. 227)

Coh-Metrix analyzes the extent to which a text is classified as narrative as opposed to informational. Texts cannot always be crisply categorized as one or the other; some have elements of both. (p. 227)

A break in cohesion occurs when there is a discontinuity on one or more of these situation model dimensions. Whenever such discontinuities occur, it is important to have connectives, transitional phrases, adverbs, or other signaling devices that convey to the readers that there is a discontinuity; we refer to these forms of signaling as particles. Different sets of particles are associated with causal (e.g., because, enable), intentional (in order to, so that), and temporal (before, later) cohesion, although some particles may be applicable to more than one type of cohesion. Cohesion is facilitated by particles that clarify and stitch together the actions, goals, events, and states conveyed in the text. (p. 227)

Coh-Metrix computes the ratio of cohesion particles to the relative frequency of relevant referential (p. 227)