TxReadability Project
The Development of A Multi-Language Readability Tool
Summary
TxReadability is an application developed by the Accessibility Institute at the University of Texas at Austin (UT), in conjunction with the Texas Technology Access Program/Texas Center for Disability Studies at UT. This readability analysis tool assesses the ease with which text is read. Analyzing readability in terms of educational levels at which text is written, this tool serves as a quantitative gauge for users to determine whether their written text or Web content is suitable for their target audience.
Description of the TxReadability Web Application
TxReadability can be accessed online as a Web application. The Web site consists of three pages-Home, Submission, and Results. At this time, the tool supports analysis of content written in English, Spanish, and Japanese; the Home page of TxReadability thus provides welcome messages in all three languages.
By choosing one of the three languages on the Home page to interface with the tool, users are guided to the Submission page, presented in one of the three languages they chose. Users can (i) submit content written in one of the three languages, and (ii) either submit a URL or paste text into a form. Selecting the 'Analyze' button returns the Results page showing the readability score(s) of the submitted content and additional information to aid in interpretation.
Supplemental pages containing information about readability and topics related to the tool itself are also available as links from the Web site.
Purpose of a Readability Analyzer
TxReadability is designed to support increased access to Internet users with learning or other disabilities, particularly those with limited reading abilities (cf. Boldyreff, Burd, & Donkin, 2001). TxReadability is not just beneficial to Internet users with disabilities, but also to a broad range of users. This tool is useful for Web designers as it serves as a way to provide feedback for whether Web content is written at a level suitable for their intended audience.
The Web Content Accessibility Guidelines (WCAG 2.0, draft version), established to promote accessibility of Web content, includes a guideline which addresses making text readable and understandable (Guideline 3.1). TxReadability provides information to help users determine if a given text meets the success criterion for compliance with this guideline.
By no means is the tool limited to analyzing content that is already on the Web. An option is available for users to paste in text for analysis. Teachers and students may find the tool useful for identifying the level of reading difficulty of instructional material or papers. This readability tool can also be helpful to government agencies and industries required to provide information in a format that people with limited education can read and understand.
Background on Readability
The term 'readability' can be interpreted in many ways, as conveyed by the broad definition of 'readable': "such as can be read; legible; fit or suitable to be read; worth reading; interesting" (Webster's Revised Unabridged Dictionary, 1913). Needless to say, it is important to define how 'readability' of Web content and other kinds of text is to be analyzed by this tool. Text difficulty, assessed in terms of vocabulary, grammar, sentence structures, etc., and appeal, reflected through legibility, font size, or layout of content, or realized through particular styles of writing, all seem to be valid factors that can influence the ease with which text is read.
The development of TxReadability was based on an established method of measuring readability, that is, using readability formulas. Unfortunately, no readability formulas are available that included all factors related to difficulty and appeal as they influence readability.
Analyzing with Readability Formulas
Readability formulas determine reading ease based on calculated scores that are mapped to educational grade levels and have been in use since the 1930s (Zakaluk & Samuels, 1988). Readability formulas single out a few factors as critical for determining reading difficulty. Although readability formulas have been critiqued as targeting only certain features of text for analysis, they are easy to use and have been employed in a variety of research contexts. Globally, research on the assessment of educational material or medical literature as related to the public and studies which test reading proficiency among school children often cite the use of readability formulas (e.g., Allan, McGhee, & van Krieken, 2005; Berland et al., 2001; Parker, Hasbrouck, & Weaver, 2001; Pérez & Couto, 2002). Different readability formulas have also been developed for use with analyzing a variety of languages. The wide applicability of readability formulas as well as their availability in many languages make them a suitable means for assessing readability of texts written in different languages.
In the development of TxReadability several factors were considered in the selection of appropriate formulas for each of the languages analyzed by the tool:
- The extent of a formula's applicability with respect to Web content
- The feasibility of developing algorithms to analyze components calculated by formulas
- The ability to reflect the mapping of scores to educational levels as online information
- The extent to which a formula applies to a broad range of educational levels
- The extent to which usage of a formula is widespread
- The extent to which a formula is known to the general public
Factor 1: Applicability to Web content
There are no known readability formulas for analyzing Web content. Therefore, it was necessary to consider if the factors analyzed by the available readability formulas for regular text were also relevant to the assessment of Web content. In general, formulas that incorporate the use of vocabulary lists to identify the difficulty level of words (e.g., Dale-Chall formula for English (Chall & Dale, 1995; Dale & Chall, 1948) and Spaulding's Spanish readability formula (Spaulding, 1951, 1956)) were not suitable. This was because lists of words of varying grade level difficulties were not updated to include words, phrases, and Web-related terminologies that make up a significant portion of contemporary Web content. In addition, another concern involving formulas with vocabulary lists is the lack of available soft copies of the lists for storage in a database from which to compare with submitted content.
Many formulas developed for English and Spanish analyze the number of words per sentence as a factor of readability. While sentences are commonly delimited by sentence-ending punctuation, sentence delimiters do not occur as consistently on Web pages as they do on regular passages of text. Therefore, in researching the applicability of formulas to Web content, the extent to which accuracy of results may be affected by fewer tokens of sentence-ending punctuation was taken into account. To this end, formulas that did not analyze sentences (i.e., require algorithms that scan content and count sentence delimiters) were preferred over ones that did. The Forcast Grade Level formula (Caylor, Sticht, Fox, & Ford, 1973) for analyzing English text was thus selected as one of the formulas to use in TxReadability as it was developed to analyze word difficulty in terms of words with more than one syllable and does not analyze sentences. Unfortunately, most formulas do in fact factor in components that are analyzable only with respect to sentences. Therefore, the lack of 'non-sentence-analyzing' formulas made it necessary to include 'sentence-analyzing' ones.
Factor 2: Feasibility of developing algorithms to analyze formula components
Whether or not algorithms can be developed effectively to account for all the components to be calculated by the formula was also important. Readability formulas developed for regular passages of text call for manual counts of number of words, syllables, or in the case of Japanese certain types of characters. In place of manual counting, algorithms were thus necessary to obtain automated counts of those elements so as to analyze text or Web address submitted online. In this respect, separate sets of algorithms for counting elements within text had to be created for each language analyzed by the tool.
Factor 3: Ability to map scores to educational levels
A formula also needs to have corresponding information about the mapping of scores to actual educational levels. Furthermore, the information had to be in a format that could be easily presented in the format of a Web page. Some formulas require users to plot variables on a graph (cf. 'Fry Graph for Estimating Reading Ages' for English (1968; 1969; 1977) and also adapted to Spanish (Crawford, 1984; Fry, 1968, 1969, 1977; Garcia, 1977; Gilliam, Peña, & Mountain, 1980; Vari-Cartier, 1981), and Crawford Readability Graph (used for Spanish content (Crawford, 1984)). The reading of graphs on Web pages may be challenging and could decrease accessibility of the information. As such, formulas which provide the information in tabular format that can be directly accessed on a Web page were selected.
Factor 4: Formulas' application to broad range of educational levels
The extent to which a formula applies to text written for a broad range of educational levels is also important. Some formulas are formulated specifically to target text written at a narrow grade level range, e.g., elementary levels only or have been validated against certain grade levels only. Formulas which provide mapping of results to the widest possible range of educational levels were thus preferred.
Factors 5 & 6: Frequency of use and the public's knowledge of the formulas
The last two factors take into consideration the general reception by the public to a given formula. Factor 5 evaluates a formula's frequency of use in published research while Factor 6 considers how well-known a formula is in general. A sampling of research done in the last decade shows that the Flesch Reading Ease and Flesch-Kincaid Grade Level formulas (Flesch, 1948; Kincaid, Fishburne, Rogers, & Chissom, 1975) are two of the most commonly used formulas for English text. These two formulas are also recognizable by general users as they are used to determine readability by a widely used word processing software, Microsoft Word. The Flesch Reading Ease and Flesch-Kincaid Grade Level formulas were selected for use in analyzing English content for their widespread use and pre-existing exposure to the general public. Further, the two formulas provide straightforward interpretation of scores in relation to educational levels, and the variables involved can also be easily analyzed.
Popular readability formulas for Spanish content include the Spaulding Formula, used widely in Latin America (Zakaluk & Samuels, 1988) as well as various Spanish adaptations of the Fry Readability Graph, e.g., Vari-Cartier's FRASE graph (1981). Unfortunately, these formulas do not map scores to educational levels directly. Consequently, the formula selected for TxReadability was the Huerta Reading Ease. This formula was adapted from the Flesch Reading Ease formula for English. In addition to having similar advantages to the Flesch formula, the Huerta Reading Ease Formula has also proven to be useful specifically for testing Web content. Pérez & Couto's (2002) research, for instance, uses the Huerta formula to assess readability of health Web pages.
While there are a number of formulas for English and Spanish content to choose to use, we were only able to obtain access to two Japanese readability formulas. Both readability formulas, documented in research conducted in the late 1980s and early 1990s, were used in research involving text processing (Hayashi, 1992; Tateisi, Ono, & Yamada, 1988). Unfortunately, the published findings contained incomplete information on the readability formulas. As a result, only one of the formulas was used in TxReadability. Feedback and information regarding Japanese readability formulas or improvements to the calculation of Japanese readability are thus welcomed.
Development of TxReadability
Prior to the calculation of readability scores filters and algorithms were implemented to strip away html codes, convert html character encodings into uniformed, analyzable codes (e.g., Unicode characters). Where stipulated by parameters associated with specific formulas, algorithms were also coded to divide text or Web content into analyzable chunks.
In the development of the algorithms, elements that were specific to each language were first identified. The three languages analyzed by TxReadability have different character- and letter- inventories and are structurally distinct from one another. For example, for English and Spanish, their corresponding formulas call for a count of the number of syllables. As different vowels constitute syllables in the two languages, different letters and combinations of letter that form possible syllable configurations had to be identified and coded separately in the algorithms for syllable count. Because different types of characters make up written Japanese, algorithms were made to identify characters as Hiragana, Katakana, Kanji, or Alpha-numerical, so that accurate counts of each type of character could be obtained. Different types of sentence-ending punctuation were also coded to enable counts of sentences (for the English and Spanish formulas) or to identify sentence boundaries which aid in determining the cut-off for a run of characters in Japanese. The results for the separate components of the formulas (i.e., number of words per sentence or number of syllables) were thus obtained separately. The final stage of the calculation then involved plugging in the separate values into the applicable formulas.
Conclusions
Overall, the TxReadability tool was well received during our user testing, and the positive reaction to the ability to analyze readability in different languages suggested that such a tool will be useful to multiple audiences. The tool serves as a pilot project for the incorporation of multiple languages into a readability analyzer. It is our expectation that the TxReadability tool:
- will be useful in determining whether the readability of Web content is appropriate for a targeted audience
- will help determine compliance with the Web Content Accessibility Guidelines 2.0 (draft) of Success Criterion 3.1.5
- will grow to include other languages to support the multiple languages in which Web content is developed
References
Dale, E., & Chall, J. S. (1948). A formula for predicting readability. Educational Research Bulletin, 27, 11-20, 37-54.
Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32, 221-233.
Fry, E. B. (1968). A readability formula that saves time. Journal of Reading, 11, 513-516.
Fry, E. B. (1969). The readability graph validated at primary levels. The Reading Teacher, 22, 534-538.
Fry, E. B. (1977). Fry's readability graph: Clarifications, validity, and extension to level 17. Journal of Reading, 21(3), 242-252.
Garcia, W. F. (1977). Assessing readability for Spanish as a second language: The Fry graph and cloze procedure. Dissertation Abstracts, 38(136a).
Gilliam, B., Peña, S. C., & Mountain, L. (1980). The Fry graph applied to Spanish readability. The Reading Teacher, 3, 426-430.
Hayashi, Y. (1992). A three-level revision model for improving Japanese bad-styled expressions. Proceedings of the 14th Conference on Computational Linguistics, 2, 665-671.
Kincaid, J. P., Fishburne, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. CNTECHTRA Research Branch Report, 8-75.
Parker, R. I., Hasbrouck, J. E., & Weaver, L. (2001). Spanish Readability Formulas for Elementary-Level Texts: A Validation Study. Reading & Writing Quarterly, 17, 307-322.
Pérez, A. B., & Couto, U. G. (2002). Legibilidad de las páginas Web sobre salud dirigidas a pacientes y lectores de la población general. Rev Esp Salud Pública, 76(4), 321-331.
Spaulding, S. (1951). Two Formulas for Estimating the Reading Difficulty of Spanish. Educational Research Bulletin, 30(5), 117-124.
Spaulding, S. (1956). A Spanish Readability Formula. The Modern Language Journal, 40(8), 433-441.
Tateisi, Y., Ono, Y., & Yamada, H. (1988). A computer readability formula of Japanese texts for machine scoring. Proceedings of the 12th Conference on Computational Linguistics, 2, 649-654.
Vari-Cartier, P. (1981). Development and validation of a new instrument to assess the readability of Spanish prose. The Modern Language Journal, 65, 141-148.
Zakaluk, B. L., & Samuels, S. J. (1988). Readability: Its past, present, and future. Newark: International Reading Association.
