Printer friendly PDF Version

Background

The University of Texas at Austin (UT Austin) changed the way it admitted students beginning with the freshman class of 1997. The development of this process was driven both by the need to manage growing enrollments at the University and to respond to the Hopwood decision.[i] The new process dramatically reduces the influence of standardized test scores; class rank is more important; and for students not automatically admitted through the “top 10% rule” a holistic review of a student’s entire application is factored into an admissions decision.

Nearly four years later, on February 18, 2001, Dr. Richard C. Atkinson, the President of the University of California System, delivered the 2001 Robert H. Atwell Distinguished Lecture at the 83^rd Annual Meeting of the American Council on Education (ACE) in Washington, D.C. His address, Standardized Tests and Access to American Universities, sent shockwaves across university campuses throughout the United States because he concluded, “…America’s overemphasis on the SAT is compromising our educational system.”[ii] Dr. Atkinson called for much of what was already implemented at UT Austin: a “holistic” approach to defining merit. But unlike the UT Austin routine, he also called for the use of five SAT IIs (Subject Tests) in the admissions process instead of the more common SAT I: Reasoning Test.

In his letter of appointment to each Task Force member, President Larry Faulkner described a limited charge: “The objective of this group is to consider the role that standardized test scores play in our admissions process and to insure continued fairness in this process for undergraduates.” He continued by stating, “I have no preconceived ideas about how strong of a role admission tests should play and simply want an assessment of the facts with any recommendations about their continued use.” [iii]

The Mission of The University of Texas at Austin

It is the mission of The University of Texas at Austin to achieve excellence in the interrelated areas of undergraduate education, graduate education, research, and public service.[iv] As the state’s flagship university, it has a special responsibility and relationship with its community and state. As a major source of well-rounded, highly educated individuals, UT Austin appreciates the crucial role it plays in the future of the State of Texas. UT Austin is simultaneously a selective institution, with some of America’s finest colleges and schools, and the largest university in the United States; its influence extends well beyond the boundaries of Texas.

The mission of the University cannot be achieved without the exposure of its students, faculty, and staff to diversity of talent, ideas, and people. The extent to which undergraduate students become more able to function and contribute globally depends, in part, on their interaction with classmates and other cohorts of differing backgrounds, attitudes, and cultures. This crucial educational axiom is the source of the University’s commitment to a comprehensive process of admitting students and enrolling entering freshman classes consistent with its mission, desire, and capacity to represent all of the people and areas of Texas—including its rich cultural mix. In short, diversity is an educational necessity.[v]

The Expanded Concept of Merit in the Admissions Process

The admissions process of any university is an exercise in both selecting qualified students with a high probability of success, and crafting an entering class that also meets the university’s mission. Given the costs associated with a year of college, consciously admitting students with low probabilities of success is not good public policy and is a cruel act towards individuals. Admissions is necessarily, therefore, both an attempt to predict success and a message to applicants about what attributes are highly valued by the institution, i.e., merit.

The University has faced many challenges in meeting its broadly defined diversity goals since the landmark decision Sweatt v Painter.[vi]  The Hopwood decision is the most recent judicial chapter, but the immediate and pressing challenge is managing enrollment growth while attaining desired diversity—within the confines of the law. Each new chapter challenges previous definitions of merit.

Even before 1997 and the Hopwood decision, UT Austin had begun to refine it’s definition of merit, which when applied to an admission decision, consisted of high standardized test scores and high class rank. This approach, often called the “classic model,” was common among colleges and universities and consistent with an era of high stakes testing and accountability in all levels of education, especially in Texas at the elementary and secondary school levels.[vii] It is well documented that a simple distribution of students by standardized test scores will certainly result in the depression of diversity, not only racially and ethnically, but economically as well.[viii]

Shortly after Hopwood, the Texas State Legislature exerted its own definition of merit. Through the passage of HB 588, the “Texas Top 10% Automatic Admissions Law,” any student graduating in the top 10% of his/her Texas high school class is automatically admissible to any public college or university in the state. Since 1997, automatically admitted students have accounted for approximately one-half of all entering freshmen at UT Austin. Again. the challenge for the University family was to fashion an admissions policy that resulted in a diverse freshman class, while assuring a high probability of success for each of the students admitted, and staying within the law.

The Evolution of UT Austin’s Admissions Policy

Before 1997, The University of Texas at Austin believed a meritorious applicant was one who scored high on either the ACT Assessment or the SAT I: Reasoning Test and who had graduated near the top of his/her high school class. At that time, students were sorted according to a formula that predicted a freshman grade point average (GPA). Students with the highest predicted GPAs were admitted first. The process was efficient as it required almost no human intervention or judgment. The University (and almost all other selective public institutions) became comfortable with a computer-driven method of admitting students that had been in place for a number of years.

Today, the UT Austin admissions process is much more elaborate and the concept of merit is much broader. With the entering class of 1997, for those not automatically admitted, the idea of merit was expanded to include the following factors:

1. The Academic Index (AI)
   High School Record:
   • Class rank
   • Completion of UT required high school curriculum
   • Extent to which students exceed the UT required units
   • SAT/ACT score
2. The Personal Achievement Index (PAI)
   • Scores on two essays
   • Leadership
   • Extracurricular Activities
   • Awards/honors
   • Work experience
   • Service to school or community
   • Special circumstances:
     • Socio-economic status of family
     • Single parent home
     • Language spoken at home
     • Family responsibilities
     • Socio-economic status of school attended
     • Average SAT/ACT of school attended in relation to student's own SAT/ACT

Merit still includes high test scores and high class rank, but it also includes the ambition to tackle rigorous high school coursework, the production of quality prose, and the desire to make a difference in one’s school, home, or community. Evidence of employability (work), and some sense of having excelled in any number of areas are also considered. Moreover, admissions officials place these attributes in the context of the circumstances under which the student lived. Since 1997, the rational, thoughtful, and reasoned judgments of people complemented prediction formulas. Throughout the United States, many social scientists, educators, and reformers, chief among them the University of California System President Richard Atkinson, call this the “holistic approach” and advocate reducing the emphasis on test scores.[x] The University of Texas did precisely that in 1997 and since then has analyzed the qualities each applicant would bring to an entering freshman class. Furthermore, retention rates and performance have remained stable or increased since the implementation of this expanded admissions routine.[xi]

The Role of Standardized Tests in the College Admissions Process at UT Austin

There are only two nationally-validated, standardized college admissions tests in the United States today: the ACT Assessment and the SAT I: Reasoning Test.

The ACT Assessment is owned and administered by ACT, Inc. (formerly called the American College Testing Program, Inc.). It is widely known as an achievement test based on a curriculum commonly taught in American high schools. It includes four tests: English, Mathematics, Reading, and Science Reasoning. Students receive a scaled score in each area and an overall Composite score. The scale ranges from 1-36 and the national average is 21.0.

The SAT I: Reasoning Test is owned by the College Entrance Examination Board (College Board) and administered by the Educational Testing Service (ETS). It is designed to be and is recognized as an aptitude test not connected to any curriculum. It includes two tests: Verbal and Quantitative. Students receive scaled scores on each that range from 200-800 in intervals of ten. Many schools combine the two scores to form the more familiar 400-1600 score. The national average is a 1020 combined score.

A primary purpose of both instruments is to add to the accuracy of predicted freshman grade point average. Neither ACT, Inc. nor the College Board has ever suggested that its scores become the sole criteria for any high stakes decision. On the contrary, their literature actively discourages such an application.[xii]  

Concerns over the fairness of the ACT/SAT would be much more acute if students were admitted as a result of a formula based on scores alone. No undergraduate is admitted to UT Austin that way. Indeed, even in those cases where the ACT/SAT is considered, it is merely part of a comprehensive approach:

1. About one-half of entering freshmen are automatically admitted as "Top 10%" students. For those students, test scores play no role in the admission decision.
   
2. For all others, a decision is based on an academic and/or personal achievement index. (See above.)
   • Test scores are not considered in the computation of the PAI; and
     
   • In the AI, where test scores are used and a formula is developed, high school class rank and course selection are considered as well.

Standardized tests play an important, but secondary role in the admissions process.

Validating the Use of Standardized Admissions Test Scores at UT Austin

The development of the formulas used to predict freshman GPAs includes the computation of the relationship between test scores and actual earned GPAs. (UT Austin uses three explanatory variables in its formulas: the high school percentile, which is derived from the school reported class rank and size; the ACT English or SAT Verbal, and the ACT Math or the SAT Quantitative, whichever was submitted by the student.[xiii]) In its Guide to the College Board Validity Study Service, the Board describes three types of information every validity study should have. The first is the correlation between the predictors and the criteria; the second is the prediction equation, which is used to predict an individual’s freshman GPA; and the third is the “error band” or the standard error of estimate.[xiv]

The Correlation Between Test Scores and GPA

Research conducted by UT’s Office of Admissions and the Measurement and Evaluation Center, and later independently validated by ACT, Inc., has established that there is a moderate to strong relationship between test scores and freshman GPAs (r=.37-.41).[xv] High School Percentile Rank showed a slightly stronger relationship (r=.45). The UT Austin experience is typical of virtually all other universities conducting validity studies: no single predictor by itself is strong enough to adequately predict freshman GPAs. Because of this limitation the best approach is to combine variables into a prediction, i.e., multiple regression, equation.[xvi]

The Prediction Equation

UT Austin has ten undergraduate colleges. For purposes of applying prediction formulas, Liberal Arts, Communications, Fine Arts, Social Work, and Education share the same equations because of similarities in course requirements. For the same reason, Nursing, Natural Sciences, and Architecture share another, and Business and Engineering are large and unique enough for each to have its own. The correlations between the prediction models and freshman GPAs are strong, ranging from .47 to .57, and are typical of other universities.[xvii]

The Standard Error of Estimate

The range of the standard error of estimate in UT Austin’s prediction equations is from 0.55 to 0.72 of a grade point. The national median standard error of estimate for schools using the ACT Assessment, for example, is approximately 0.75.[xviii] This is strong evidence that UT Austin formulas, as currently configured, are serving the admissions goal of accurately predicting freshman GPA.

The ultimate validation, however, is that performance (actual freshman GPA) and persistence (the percentage returning for their sophomore year) for every racial/ethnic and gender group has held steady or improved since 1996. Since the implementation of the University’s present admissions routine, one-year retention rates have increased each year—from 87.9% in 1996 to 92.0% in 2000.[xix]

Fairness: The Relationship Between Class Rank, Test Scores, Grade Point Average, and Gender and Racial/Ethnic Groups and Access

The legitimate concern for fairness pervades all aspects of testing, especially including the development of the tests, access to the tests, the standardization of the conditions under which the test is administered, and the use of the test scores.

Both ACT, Inc. and the College Board have elaborate fairness reviews to screen out items determined to have geographic, gender, culture, or racial bias.[xx] Each test takes a minimum of two years to develop. The process includes intensive screening, editing, reviewing and field-testing. Both firms utilize a method called Differential Item Functioning (dif) to determine if a question “behaves” differently among diverse groups with the same level of expertise with respect to what is being tested.[xxi]

In order to validate its entire admissions routine, UT’s Office of Admissions and the Measurement and Evaluation Center analyzed the relationships between the elements of its prediction formulas (high school class rank and test scores) and freshmen GPAs. (See Table 1 below.) When broken down by gender and racial/ethnic groups, the relationships ranged from positive correlations of .21 to .47. Only the relationship of ACT Math to African American GPA (.21) was found to be less than moderate and then only slightly so. Consistent with other research, class rank was found to have the strongest relationship to GPA, except for Hispanics and females.

By comparison, an ACT analysis of the elements of UT Austin’s PAI, which consists of the “Leadership Score” (an evaluation by an admissions officer of the student’s resume), and the mean score of the two essays students submit, shows a weaker relationship than test scores. (Leadership Score: r=.22 and Essays: r=.26)[xxii]

The UT Austin experience is a textbook example of how no predictor by itself, whether objective or subjective or standardized or performance-based, has sufficient power to predict freshman GPA accurately. The best possible approach for all students is to assimilate as many predictors as possible into the admissions process.

Another issue of concern to the Task Force was the possible effects of “coaching” or test preparation. If the SAT or the ACT is sensitive to short term test preparation, like what is available from expensive test preparation firms, then test takers from poor homes, i.e., those not able to afford prep classes, are at an unacceptable disadvantage.

While there is a dearth of independent and credible research in this area, what is currently available strongly suggests that the effect of coaching on the SAT I and the ACT Assessment is minimal and within the standard error of measurements of the tests.[xxiii] Additionally, since the reliability of both tests is high (about .92 for the SAT I and .96 for the ACT), there is no credible evidence that retesting has consistently significant benefits for students.[xxiv]

Neither coaching nor retesting has as much effect on raising scores as does decisions by students to prepare themselves for college by taking the most challenging coursework available to them.[xxv] Because of this sensitivity to course-taking patterns, and variations in the quality of what is available to high school students, score differences will result if there is inequity in the availability of quality teaching and rigorous coursework among UT Austin’s feeder schools.

A final fairness issue concerns access to the tests and whether registration fees unfairly exclude students, who cannot afford to pay, from taking the tests. Both ACT, Inc. and the College Board make available fee waivers for economically disadvantaged students during their junior and/or senior year of high school.[xxvi]

The SAT II: Subject Tests

The SAT II: Subject Tests were once called the “Achievement Tests.”During his address to the American Council on Education, Dr. Atkinson stated, “The SAT II begins to approximate what I judge to be an appropriate test for the University’s admissions process. It tests students on specific subjects that are well defined and readily described.”  He continues by asserting, “We have found that the SAT II is a better predictor of first-year college performance.”[xxvii]

UT Austin requires SAT II scores in math and separately in writing for nearly all of its entering freshmen for purposes of course placement. Interest in the use of SAT IIs for admissions is a relatively new phenomenon. The College Board designed the SAT I: Reasoning Test for admissions and SAT II: Subject Tests for course placement. Throughout its history the ACT Assessment has been used routinely for both throughout the nation.[xxviii]

The central question before this Task Force, however, is whether the SAT II, when combined with high school class rank, does a significantly better job of predicting freshman GPA.

• Among students who took SAT I and II, the correlation between SAT Verbal and Math and freshman GPA is strong (.44); for SAT II Math and Writing it is strong (.48);
  
• Among students who took the ACT and SAT II, the correlation between the ACT Tests and freshman GPA is strong (.47); for the SAT II Math and Writing, the correlation is strong (.48).
  
• In both of the cases above, the addition of the SAT II leads to insignificant increases in predictive powers. (From .47 to .49 for ACT and from .44 to .48 for SAT I.)
  
• Among students taking the ACT, SAT I and SAT II the correlations (without including class rank) with freshman GPA are .37, .37, and .38 respectively.

At UT Austin, such small differences in predictive validity between the SAT I and SAT II, and even smaller differences between SAT II and the ACT, are insufficient for this Task Force to recommend the use of the SAT II in the area of admissions. Further, for purposes of admissions, there is no evidence of any meaningful difference between achievement and aptitude testing; the correlation between the ACT Composite and the SAT I Combined Score is .92 nationally and .87 at UT Austin.[xxix]

Conclusions and Supporting Evidence

The Task Force proceeds from the premise that more information on applicants is better than less. The use of the ACT Assessment and the SAT I: Reasoning Test increases the confidence, reliability and accuracy of attempts to predict freshman GPA. Discontinuing their use would reduce information with which admissions decisions are made.

Further, the Task Force affirms that standardized college admissions test scores provide admissions decision-makers and faculty with information that is not available from any other source:

• standard measures across geographical borders;
  
• standard measures over time;
  
• nationally-validated rigorous instruments measuring higher-order thinking and problem-solving;
  
• instruments with established reliability; and
  
• results obtained under secure conditions using many equated forms.

The predictive power of the ACT Assessment and the SAT I has been validated through research conducted on campus by the UT Austin Office of Admissions Research and the Measurement and Evaluation Center, and externally by ACT, Inc. and the College Board/Educational Testing Service.[xxx] These validity studies are on-going.

Reforms most often associated with attempts to provide greater access to traditionally underrepresented students (moderating standardized test scores by considering them in the context of overall academic performance in high school and evaluating applicants in the context of their personal experiences—the holistic approach) were instituted at the University of Texas at Austin with the entering freshman class of 1997. This Task Force affirms our current admissions process as one that considers the student in relation to his/her life experiences.

Since the implementation of the Texas Top 10% Law and a refined and expanded definition of merit, UT Austin has enrolled freshman classes about as diverse as classes enrolled before 1997 when affirmative action was used.

Success, in terms of retention, for the entering freshman classes enrolled at UT Austin since 1996, has improved annually while performance levels have remained stable.

Like many other universities, UT Austin had found the classic model of admissions (distributing students according to a formula based on test scores and class rank alone) to be easy and efficient. Since 1997, however, the University has individually reviewed all applications of students not automatically admitted.

The developers of the ACT and the SAT agree that test scores should not be used as a sole criteria for making high stakes decisions but that scores should be used in conjunction with as many other variables as possible. This Task Force affirms that UT Austin does this through the use of the Personal Achievement Index (PAI).

The Task Force has established that test scores have played no role in the admission of approximately one half of UT’s entering freshmen classes since 1997.

Test scores play a limited role in the admissions process: they make up about one half of the prediction formulas that make up one half of the matrix used to admit about one half of entering freshmen.

There is a moderate to strong relationship between test scores and freshmen GPAs, but the best method of prediction is to combine test scores with high school class rank. At UT Austin the resulting prediction equations have a strong relationship to freshman GPA.

Correlations between the elements of UT’s prediction formulas (test scores and class rank) show a moderate to strong predictive relationship with freshman GPA for all gender and racial ethnic/groups.

The use of differential item functioning (dif), and other procedures safeguarding against racial/ethnic, gender, cultural and geographic bias in the development of the ACT and SAT I, are adequate to assure fairness for students taking the tests.

ACT, Inc. and The College Board provide adequate access to its tests for all students through the distribution of free tests for preparation purposes and the use of waivers for students who cannot afford to pay test fees.

There is no replacement for the reasoned judgment of professionals in the admissions process. While the elements of the Personal Achievement Index (Leadership Score and Essays) have a moderate relationship to freshman GPA, their use contributes to the educational mission of the University.

The predictive power of the SAT II: Subject Tests is comparable to that of the ACT Assessment or the SAT I. Combining the SAT II with either the ACT or the SAT I adds little to the strength of prediction formulas currently in use.

Recommendations to the President

• The Task Force recommends that the University continue the use of the ACT Assessment and the SAT I: Reasoning Test, subject to ongoing validation for that purpose, in the admissions process and that the University show no preference for either.
  
• The Task Force recommends the continued use of professional reviews of applications, resulting in a Personal Achievement Index score, subject to ongoing validation for that purpose, for students not automatically admitted to UT Austin.
  
• The Task Force does not recommend the use of the SAT II: Subject Tests for admissions purposes, but does recommend the continued use of SAT IIs for course placement, subject to ongoing validation for that purpose.