Developing a Concordance Between the ACT Assessment and the SAT I: Reasoning Test for The University of Texas at Austin

by 

Note: The authors wish to thank Drs. James Maxey, Assistant Vice President of Applied Research, Robert Ziomek, Director of Program Evaluation, and James Sconing, Director of Statistical Research, and the late Merine Farmer of ACT, Inc., and Drs. Steve Fitzpatrick and Charles Gaede of the Measurements and Evaluation Center, and Cindy Hargett of Student Information Systems of the University of Texas at Austin for their valuable assistance in the production of this report.  

Printer Friendly PDF Version

Introduction 

The Office of Admissions of the University of Texas at Austin (UT-Austin) is responsible for the admission and readmission of undergraduate students to the University.  As with all Texas public colleges and universities, and in accordance with Texas Education Code §51.803, students are automatically admissible as first-time freshmen if they (1) graduate in the top ten percent of their high school class (provided they are from an accredited Texas high school) and (2) submit all required credentials comprising a completed application by a deadline established by the university. At UT-Austin, applicants not eligible for automatic admission must have completed a specified group of high school courses, and like those automatically admitted, must present credentials before established deadlines.[i]

One of the credentials required of all UT-Austin applicants is the submission of a standardized college admissions test score. The ACT Assessment and the SAT I: Reasoning Tests are the only nationally-validated standardized tests used in the college admissions process, not only in Texas, but throughout the United States.[ii]

The ACT Assessment was developed in the late 1950s by college faculty, chief among them E.F Linquist, associated with the University of Iowa. Linquist was also the primary founder of the American College Testing Program, Inc., later changed to ACT, Inc.

ACT’s current literature characterizes the ACT Assessment as “tests of educational development” measuring readiness for postsecondary education.   It is more popularly known as an achievement test since it is organized by subject matter area and its scaled scores are reported that way. There are four tests in the curricular areas of English, Mathematics, Reading and Science Reasoning.   The contents of the ACT tests are based on the judgments of high school and college instructors about the academic knowledge and skills that students need to succeed in typical first-year courses and that are taught in typical college-preparatory programs in high school. Since 1988, during the development phase of what was then called the “Enhanced ACT Assessment,” ACT conducted a National Curriculum Study and has since continually analyzed (every three years) state curriculum guides, scopes and sequences, and state-approved textbooks to determine the content of the ACT Assessment and its related assessments.[iii] This is consistent with Linquist’s vision of college-admissions testing: that what should be measured is what the student can do with what they have actually been taught.

The SAT I: Reasoning Tests consist of two tests that measure verbal and mathematical reasoning abilities of college-bound students. The College Board, primarily Carl Campbell Brigham, first developed it in 1926.[iv] Historically, the SAT I has been considered an “aptitude” test emphasizing critical thinking and problem solving skills that are deemed essential for college-level work. Indeed, “SAT” was once an acronym for “Scholastic Aptitude Test,” suggesting no tie to a particular curriculum. In fact, the Board argues, this approach compensates for non-standard, and by implication, sub-standard, grading practices and differences in secondary school quality and curriculums. The Board further argues that while the SAT is not intended to be curricular, the test nonetheless reflects acceptable educational content, standards and practices. (Content and technical specification differences of the ACT and the SAT are described in greater detail below in a separate section.) The Preliminary SAT/National Merit Scholarship Qualifying Test (PSAT/NMSQT) is the only other standardized exam directly related to the SAT. It was developed and normed for college-bound high school juniors and is made up of retired SAT questions.

The University of Texas at Austin requires the submission of a complete set (either ACT Assessment or SAT I) of college admissions test scores of all its applicants. Substantial numbers of students send at least one set of both. While the university does not “prefer” either in an admission decision, there are instances in which there is a need for a common and standard measure for all UT-Austin applicants, admits, and enrollees. Since some students send only an SAT, and some others send only an ACT, it becomes necessary to have some basis for comparison of the scores on the two tests. Historically, for research and other reporting purposes, since UT-Austin receives far more SAT than ACT score reports, ACT scores have been concorded to the SAT scale.

As Dorans has pointed out, there are two ways in which the degree of exchangeability of two different tests can be evaluated: there should be an investigation into the processes that produced the scores; and statistical measures such as correlations should be produced to establish empirical relationships among the scores.[v]

The purposes of this study are to describe the skill and content similarities of the ACT Assessment and the SAT I: Reasoning Test, investigate the skill and content relationships between the two, establish the appropriateness of developing a table that will permit a comparison of the scores, and finally, to construct a table for use in determining comparable SAT scores for given ACT scores for use in admissions and research routines at the University of Texas at Austin.   The comparison will be limited to use at The UT-Austin and will be developed via a method known as concordance.

Comparing Content and Skill Specifications 

ACT English

ACT English is a 75-question, 45-minute test that measures the understanding of the conventions of standard written English. This, in turn, is separated into two categories: usage/mechanics and rhetorical skills.[vi] Usage and mechanics focuses on editing skills such as punctuation, grammar and usage, and sentence structure, or judgments about what is “right” or “wrong” in writing.   For example, in a given sentence, the subject and verb agrees or they do not. Rhetorical Skills tends to require more sophisticated judgments from students. In these questions students are required to make decisions about written material. ACT describes these decisions as those of strategy, organization and style. These questions do not necessarily focus on the “correctness” of writing, but effectiveness, or whether a given sentence, phrase, or paragraph can be made better.

Three scores are reported for the ACT English test: a total test score based on all 75 questions and placed on a scale of 1-36; a usage/mechanics subscore based on 40 questions (53%) and placed on a scale of 1-18; and a rhetorical skills subscore based on 35 questions (47%) and placed on a scale of 1-18.[vii]

ACT Mathematics

The ACT Math test is a 60-question, 60-minute test to assess skills students are expected to have learned in courses taken up to the beginning of the senior year of high school.   It requires computational and reasoning skills necessary to solve practical problems. Knowledge of basic formulas and certain computational skills are assumed—not tested as such. Certain types of calculators are permitted.

The math test is comprised of three parts that define its major content areas. Pre-Algebra/Elementary Algebra (40%) is the most elementary, often involving content normally taught in middle school. Intermediate Algebra/Coordinate Geometry (30%) and Plane Geometry/Trigonometry (30%) make up the more advanced content. Like the ACT English test, each content cluster yields a subscore on a scale of 1-18. A math score based on all 60 questions is reported on a scale of 1-36.

ACT Reading

ACT Reading is a 40-question, 35-minute test that measures reading comprehension. In describing the reading test to students, ACT focuses on skills by characterizing the questions as “referring” or “reasoning.” Referring questions have answers that are explicitly stated in the text of the reading passage. Reasoning answers, on the other hand, require students to derive meaning implicitly from the passage.   Two reading subscores, however, are reported for content, not skill, areas. Social Studies/Sciences and Arts/Literature each make up half of the test and are reported on a scale of 1-18. A reading score based on all 40 questions is reported on a scale of 1-36.

ACT Science Reasoning

ACT Science Reasoning is a 40-question, 35-minute test that measures the interpretation, analysis, evaluation, reasoning, and problem-solving skills required in the natural sciences. Seven sets of scientific information are presented and are followed by multiple-choice questions. The information is conveyed in one of three formats: Data Representation (38%) involves graphs, tables and other methods of presenting data; Research Summaries (45%) is more text-oriented and usually involve a report of one or more related experiments; Conflicting Viewpoints (17%) involves the expressions of opposing ideas and require students to evaluate the merits of arguments. In a departure from its achievement testing slant, ACT literature has indicated that scientific knowledge is not as important as an ability to navigate the scientific process and evaluate arguments and data. There are no subscores associated with the science reasoning test; one score based on all 40 questions is reported on a 1-36 scale.    

Computing the ACT Composite and Other ACT Scores

The ACT Composite is not directly derived by all 215 questions of the four tests. The Composite is merely the sum of the four test scores divided by four and rounded to the nearest whole number. Thus, the Composite is based on a 1-36 scale and each of the four tests carry equal “weight” in determining the overall score.  

It is worthy to note, however, that since each of the four test scores are placed on a 36-point scale, that the ACT scale is really a 144-point scale.

All ACT scale scores are computed through the conversion of a raw score, or the number of correct answers on a given test form, to a scale score. Since only the number of correct answers is considered, there is no “penalty” for guessing wrong on the ACT.

SAT I Verbal

The SAT Verbal test consists of two 30-minute and one 15-minute section, for a total of 75 minutes of testing time. (Unlike the ACT, which administers the whole of each of its four tests in a specified order, the SAT administration alternates between the verbal and math.) Verbal questions test the student’s ability to understand and analyze prose, recognize sentences that make logical sense, and see relationships between pairs of words. A good vocabulary is prerequisite to doing well on this section, but it is not always specifically tested.[viii] That is why the rote memorization of lists of words is not likely to significantly increase verbal scores. (Unless, of course, an individual happens to memorize the definitions of specific words on a given form of the SAT—which is very unlikely.)

In order to measure the abilities stated above, the SAT Verbal test uses three types of questions: analogies (19), sentence completions (19), and critical reading (40). Analogies are designed to measure a student’s knowledge of the meanings of words, the relationship in a pair of words, or the recognition of a similar or parallel relationship. In its literature, the College Board encourages students to look closely at the pairs of words and create a sentence or phrase that states the relationship. (e.g., “A crumb is a piece of bread.) In that respect, memorized definitions of words are unlikely to be of much help to students. Moreover, only occasionally do the analogies contain esoteric, polysyllabic words; they are far more likely to contain common words students readily know the definitions of.   Words such as “ERASER : PAGE” and “DRAMA : ACTS” are typical of SAT analogies because measuring relationships between words is more important than measuring definitions. (Saying SAT “measures analogies” is overly simplistic and specious. It is more accurate to say that SAT uses analogies to measure a student’s grasp of the relationships between pairs of words.)

Sentence completions, like analogies, measure knowledge of the meanings of words. They also measure the ability to understand how the different parts of a sentence fit logically together. Students are presented with a sentence with one or two blanks. Students must then fill in the blank(s) with a word(s) that fits the logic of the sentence. (e.g., “Medieval kingdoms did not become constitutional republics overnight; on the contrary, the change was --------.” From five choices the correct answer is “gradual.”)

Slightly more than half of the verbal section of the SAT I is critical reading. It measures the ability to read and think carefully about several different reading passages. There can be more than one passage for a given set of questions, but generally it is one of about 400 to 850 words in length. There are many different types of questions ranging from demonstrating knowledge of how a single word is used to the meaning of the passage as a whole. Critical reading on SAT verbal is very similar to ACT’s Reading test.

The SAT Verbal test produces a scale score between 200 and 800 reported in intervals of ten—or a 61-point scale. 

SAT I Math

There are three types of questions on the mathematics sections of the SAT I: standard five-choice multiple-choice questions (35); four-choice quantitative comparison questions that emphasize the concepts of equalities, inequalities and estimation (15); and student-produced responses, sometimes called the “grid-in” questions (10).

In its review for the SAT I math test, the College Board listed mathematics concepts that students should know in the following areas: arithmetic, algebra, geometry and other concepts like logical reasoning. There is no mention of advanced mathematics like college algebra, trigonometry or calculus.[ix] True to its non-curricular focus, the emphasis is on skills identified as data interpretation and applied math, i.e., reasoning. According to collegeboard.com, the following constitutes the makeup of the SAT I math test: arithmetic reasoning (18-19); algebraic reasoning (17); geometric reasoning (16-17); and miscellaneous reasoning (7-9).

The SAT I Math test does distinguish itself from the ACT Assessment and most other standardized tests in that ten of the questions are not multiple choice, but student-produced. The “grid-in” questions require the production of an answer, which on its face eliminates any vestige of corruption by good guessing. The student then must grid in the response. (There is no penalty for guessing in this section.) This answer collection method, however, may have problems of its own; Taking the SAT I: Reasoning Test, the free student preparation guide that accompanies an SAT registration packet, has two full pages of directions and hints for filling in the answer grids.[x]

The SAT Math test produces a scale score between 200 and 800 reported in intervals of ten—or a 61-point scale. 

Computing the SAT I Score

Like the ACT Assessment, the computation of the individual test scores are done separately and raw scores are converted to scaled scores. However, there is a major difference in the SAT computation in that students providing wrong answers are “penalized” (or “adjusted”). In short, if a student answers a question correctly, s/he earns one point towards a final raw score. If a student provides no answer, no credit is given. If a student answers a question and gets it wrong, there is an “adjustment” for guessing. This adjustment equals a –1/4 point if the question was a five-option multiple choice; or a –1/3 point if the question was a four-option multiple choice.  (Again, there is no adjustment for the student-produced responses in the math test.) The justification for this procedure is that, in theory, random guessing in a multiple choice format should yield 20-25% correct answers, resulting in a raw score that is 20-25% of the total, suggesting that students knew 20-25% of the content of the test.   Adjustments of this sort mean that random guessing (the equivalent of “knowing nothing”) should result in a raw score closer to zero.

Curiously, while the SAT I Total score (scale=400 to 1600) is one of the most recognized scaled scores in the world today, the College Board seldom, if ever, reports its scores in that way. Student score reports do not, and have never, reported SAT Verbal and Math (V+M) as a single score. The Educational Testing Service (ETS) does not report combined (V+M) in its data transfer to colleges and universities. They also do not routinely report percentiles for those scores. The adding of the verbal and math scores into an SAT Total score is done at homes, high schools, by the media, and by colleges and universities wishing to secure a single measure of performance for a student.  

Content and Skill Similarities Justifying the Construction of a Concordance

It is the primary purpose of both the ACT Assessment and the SAT I: Reasoning Test to provide measures for college admissions officials to construct a process (most often a regression equation) through which a freshman GPA can be predicted within a reasonable error band.   If both instruments serve that purpose, it is logical that there should be a strong correlation between scores of students having taken both—and there is such a strong relationship.  

The difference in philosophy and emphasis between ACT, Inc. and College Board/ETS complicates a comparison between the ACT Assessment and the SAT I. The difficulty lies, however, mostly in how each firm chooses to describe its content. ACT defines its content in curricular terms; SAT defines itself in terms of skills—“English” versus “Verbal Reasoning” and “Mathematics” versus “Quantitative Reasoning.” Those distinctions aside, both assessments use commonly taught material from America’s middle and high schools to construct objective questions measuring higher-order thinking skills. For example, after a facial review, it is likely that only intensively-coached students in both the ACT and the SAT are likely to detect any meaningful difference between the ACT Reading passages/questions and the critical reading section of the SAT I Verbal test.   At UT-Austin, lower (but still strong) correlations between Verbal/English (r=.76) and Verbal/Reading (r=.77) scores are likely due to ACT’s separating content which overlaps with the single SAT Verbal into two tests. When ACT English and Reading scores are combined, forming a test with a greater skill and content overlap with the SAT Verbal, the relationship increases (r=.82). Table 4 below illustrates the same trends nationally in a study conducted jointly by ACT, Inc. and ETS.

In short, the skill levels of the ACT and the SAT are very similar in that they both require problem solving and higher order thinking skills such as analysis, synthesis, evaluation, and the application of logical thinking and reasoning. In math, they use mostly arithmetic, algebra and geometry; on the language arts side the emphasis is on critical reading.

The skill level, content, and statistical relationships between the ACT Assessment and the SAT I: Reasoning Test more than justify the construction of a concordance table, based on data provided by UT-Austin applicants, for use at the University of Texas at Austin.

Research scientists from ACT, Inc. and ETS agree: “In particular, users of the ACT Assessment and SAT I scores who have sufficiently large samples of test-takers with both ACT and SAT I scores may wish to investigate the feasibility of developing their own concordance tables.”[xi] 

Constructing a Concordance Table 

Sample

The Office of Admissions Research, using UT-Austin’s Data Warehouse, constructed the dataset used for this study. The data consisted of records for each student who had scores on both tests who had applied for admission for the summer/fall semesters of 1999 and 2000.   UT’s Data Warehouse captures only the ACT Composite of each student. Since this study intends to concord more than just the SAT Total and the ACT Composite, the file was forwarded to the Research Division of ACT, Inc., and matched to the ACT database using student-identifying information. The ACT English, Mathematics, Reading and Science Reasoning scores were appended to the file.

At present, the University of Texas at Austin is the largest university in the United States. Officially, UT-Austin had a total enrollment of 49,996 for the fall of 2000. For the summer/fall 2000 semesters, UT enrolled 7,686 first-time freshmen from 13,256 admitted students from 23,040 applications.[xii] Without question, UT has a large enough population to develop its own concordance. Table 1 below illustrates the summer/fall-combined 1999/2000 applicants of UT-Austin’s entering freshmen class. The “dataset” represents those applicants who sent both ACT and SAT scores.    

Table 1
Applications
 Summers and Falls Combined
1999 and 2000