Responding to Hopwood:
Using Policy Analysis Research to Re-design Scholarship Criteria

Gary R. Hanson, Ph.D.
Lawrence Burt, Ed.D.

The University of Texas at Austin

"Our country respects individual achievement, but it also recognizes that what people have achieved often depends on the families they have grown up in, the neighborhoods in which they have lived, and the schools which they have attended, as well as on their own ability and hard work." (Bowen & Bok, 1998, p. xxiii.)

Table of Contents

• Background
• The Hopwood Decision
• Purpose
• Values Guide Policy Development and Research Analysis
• Clarify the Policy Research Analysis Expectations
• Searching for informaiton and data
• Selecting data analysis techniques
• Simulating Policy Decisions
• Validating the policy outcomes
• Conclusions

Background

According to Brademas (1983), the first institutional scholarship fund was established with a £100 gift to Harvard College for the purpose of supporting a needy student. Similarly, Wick (1993) reviewed the history of financial aid and concluded that "since the beginning of the academy, scholarships have been used to attract students known or perceived to be needy and deserving" (p.2). Scholarships were linked to student financial need for most of higher education's history in the United States and only recently have been used to recruit students based on their academic merit. Huff (1975) reported that by 1975, 54% of the institutions he surveyed were awarding some scholarships without reference to need. However, McPherson and Shapiro (1998) reported that two competitive forces pushed colleges and universities to offer merit-based financial aid. One of these forces was an attempt by schools of lesser reputation or quality to "buy" students from more prestigious schools through offers of merit aid. The effect of this force was to redistribute students among institutions thereby increasing the number of high-ranking students at less prestigious institutions and decreasing the number at the more prestigious schools. The second force was competition among schools of roughly equal quality or reputation for the most meritorious students in the schools' combined applicant pool. The effect here was to increase an individual institution's relative standing among peer institutions, at least with respect to the quality of the student body. When enrollment declines occurred in the 1980s, the competition for students increased, as did the use of merit-based financial aid. At the same time that the competition for academically able students intensified both internal and external pressures caused institutions of higher education to diversify their student bodies to more accurately reflect society's ethnic and socioeconomic populations. Several states, including Texas, were mandated to pursue aggressively the recruitment of students from under-represented ethnic populations. (e.g. see the Texas Initiative, 1996). One component of these affirmative action plans was the use of merit-based financial grants and scholarships. The success of these merit-based scholarship programs is evidenced in the increased minority enrollment during the period from 1985 to 1995. At the author's institution, the Hispanic representation among the undergraduate population increased from approximately 6% in 1985 to nearly 13% in 1995. During this time period, the African American ethnic representation increased from 2% to 4%. While this representation is far below the ethnic distribution in the state, it represents substantial progress in a relatively short period of time. It was within this context that affirmative action came to a halt within the state of Texas.

Return to Table of Contents

The Hopwood Decision

Prior to 1996, The University of Texas at Austin used two scholarship programs -- the Texas Achievement Award (TAA) and the Texas Academic Honors Award (TAHA) -- to recruit and retain underrepresented minority students. These scholarship programs were one part of a larger affirmative action plan to increase the ethnic diversity of the campus. The goals of the scholarship program were to attract bright, well-prepared African American and Hispanic students not only to enroll but to persist through graduation. The underlying philosophy of this program was based on academic excellence. Recipients were awarded scholarships on the basis of high academic accomplishment as indicated by high school class rank, standardized admissions test scores and a track record of challenging courses in a college preparatory high school curriculum. Indicators of student leadership in various extracurricular activities were part of the selection criteria as well. The scholarship program provided annual, renewable funds to each student who met a specific academic performance criterion of a 2.25 cumulative grade point average. The scholarship amounts varied from $1,000 to $5,000 per year for up to five years of assistance. Each year more than a million dollars was distributed to students through this program. On March 18, 1996, the Fifth Circuit Court rendered the Hopwood decision, and a subsequent ruling by the Texas Attorney General's Office prohibited public colleges and universities from using a student's racial or ethnic background as a consideration in any recruitment, admission or retention activities. Consequently, the scholarship programs were dismantled. The University was faced with developing a new scholarship policy within the constraints of the Hopwood decision that would embrace the University's commitment to a diverse campus (The Texas Strategic Plan, 1995-1996). When racial background could no longer be used as one of the selection criteria, how could the policy driving the award program be changed yet conform to the letter and intent of the law? What new standards could be set? How would recipients be selected? Good policy development necessarily must begin with a statement of philosophy and values. The financial aid director crafted a statement of philosophy used to generate the underlying policy (Burt, 1996). This statement of philosophy shifted the scholarship program from one based on academic merit and ethnic background to one based on financial need. Academic merit was redefined and evaluated on the basis of the student's ability to overcome adverse socioeconomic conditions and to perform at high levels of academic achievement relative to peers with the same circumstances. Clearly, this change in philosophy would dictate new policy. This written statement of philosophy not only became the foundation for the development of new policy, but it also guided the policy analysis research process from beginning to end.

Return to Table of Contents

Purpose

The purpose of this paper is to illustrate how policy analysis research was used to guide new policy development for a financial aid scholarship program at The University of Texas at Austin. The rest of this paper will follow the five stages of the policy analysis research outlined by Hanson (1998). The five stages of policy analysis are:

1. Identify the policy decision-makers and clarify the goals and purposes of the policy analysis research.
2. Identify the data elements and the data sources for constructing a policy analysis research data base.
3. Identify appropriate statistical data analyses for the policy questions.
4. Conduct policy simulations by applying the policy standards to cohorts of students in the policy analysis database and examining the policy outcomes.
5. Validate the policy decision on a subsequent cohort of students to determine if the new policy standards produced the desired outcome on another sample of students.

Return to Table of Contents

Values Guide Policy Development and Research Analysis

Traditionally, college and university financial aid award scholarship programs have been based on meritocracy. Only students who performed at high levels of academic performance, typically defined in absolute terms using an admission test score, a "required" high school grade point average or a minimum standing within one's high school graduating class, are considered for the scholarship funds. One outcome of this financial award philosophy was that students who attended the best high schools were more likely to earn the scholarships. Historically, these students were middle class and attended good suburban high schools with a rich college preparatory curriculum. Prior to 1996, most of the TAA and TAHA scholarship dollars were awarded to middle class minority students. Students from disadvantaged socioeconomic backgrounds were less likely to apply to college or to receive these scholarships when they did. The goal of the new Presidential Achievement Scholarship (PAS) program was to identify students from economically disadvantaged backgrounds who may have attended an academically inferior high school, but found a way to excel academically at much higher levels than their peers within the same high school and socioeconomic circumstances. The commitment to and valuing of diversity remained a high priority for the university, but the policy guiding the selection criteria had to be rewritten. To achieve diversity, a new population of students had to be identified. The university chose to place a high value on helping students who had not considered a major research university as possible college choice. Absolute academic meritocracy was redefined in terms of academic excellence in the face of adverse socioeconomic circumstances. At this point, the director of financial services approached the author and requested assistance in conducting the basic policy analysis research to develop, simulate, test and evaluate the policies for this new scholarship program, one that emphasized a student's ability to overcoming adversity and achieve high levels of academic performance.

Return to Table of Contents

Clarify the Policy Research Analysis Expectations

The policy analysis research began by defining and clarifying what outcomes would be. In this project, the author and the director discussed how to set the policy standards for selecting student scholarship recipients. What data indicators would summarize a student's socioeconomic background? Who is "economically disadvantaged?" How would we recognize such a student? What distinguishes an academically excellent high school from an inferior one? What indicators define excellence relative to one's peers in similar circumstances? These discussions were necessary at the very beginning, but they continued throughout the many iterations of policy simulation and evaluation. Numerous suggestions were discussed, debated, and discarded but eventually the categories of information that defined the policy standards for selecting scholarship recipients were defined. The award policy included two key indicators: (1) an "overcoming adversity" index and (2) a measure of academic performance. The specific outcome the director of financial services wanted was a numerical summary of the student's socioeconomic background, the quality of the high school attended and an indication of how well prepared each student was for college relative to his or her peers in the same high school. The final policy analysis product would consist of a set of computer algorithms and summary statistics that would define the policy enabling the director to select the student recipients at each scholarship award level. Without these agreements, the policy analysis research could not begin. With them, the next step required the identification of information and data base elements to operationalize the adversity index.

Return to Table of Contents

Searching for information and data

To translate these goals, values and vision into a concrete set of policy guidelines that aided the implementation and eventual evaluation of the policy meant taking abstract concepts and building practical data definitions that summarized the concepts. Then, data sources had to be identified in order to build computer data bases for the policy analysis and simulation stages. In this project, the author worked closely with the director of financial services to identify possible sources of data for building a numerical indicator that would operationalize the adversity index. For the first year, indicators of the student's socioeconomic status were collected as part of the College Board and The American College Testing Program's standardized admissions assessment programs. A data tape with several socioeconomic indicators for all admitted students from the previous year was obtained from College Board. Information about the student's high school was available in a state wide data base provided by the Texas Educational Agency (TEA). This data base provided "quality" indicators for each public high school in the state of Texas using the Academic Excellence Information System (AEIS). The data were downloaded from an internet web page for approximately 1,200 Texas public high schools. Information about the students' standardized admission test scores and high school rank were available from the admissions application and were stored in the campus student information system. For this project, the primary problem was not finding the data sources, but reducing the volume of data to simple, summary indicators. For example, the high school state-wide data base had hundreds of data elements on each school. The information we examined included the number of students who passed state-mandated achievement tests, the amount of money spent per pupil, the number of students qualifying for the federal lunch program, the percentage of students who took and achieved an SAT test score above 1000 as well as many others. Once the data sources were identified, we initiated the complex task of bringing these disparate data elements together into a working data base, making decisions about which data elements would best define the overall concepts of high academic achievement and adversity. In building these data bases, we needed a balance between adequate representation of the underlying concept and the need for a few simple, understandable data elements. In the end, we identified five indicators of school quality, three socioeconomic indicators, one indicator of peer performance and one indicator of high academic achievement. See Table 1 for the nature of the indicator, the data elements used and the source from which they came. To arrive at these indicators, we tried and discarded many other possible indicators during the policy simulation stage. In addition, we held repeated discussions with university decision- makers in an open debate at many points during the development stage to ensure that the final set of indicators reflected the values (stage 1) of the University. One consequence of this method of policy research analysis is that more data must be included in the construction of the data base than may be needed in the final analysis. Unlike traditional educational research where only a pre-defined set of data elements are included for analysis, policy analysis research requires a greater breadth in quality and kind of data elements. Because the purpose of policy analysis research is to identify the best combination of factors to shape a given policy, the researcher must conduct exploratory data analysis. It is easier to put a wide variety of data into a data base at the beginning than it is to reconstruct a data base at a later point in time.

Table 1
Source of Adversity Index Indicators

One of the problems with building a policy analysis data base is that the disparate data elements must be linked by individual student record. Hence, information from the admissions data base had to cross-link to the state-wide high school data base. For this project, the data indicators from the TEA high school data base were identified by a high school code and linked to a different high school code used during the admissions testing phase. A cross-link data base was constructed in order to bring the two sets of data into a common database for the next stage of the policy research, the analysis of data.

Return to Table of Contents

Selecting data analysis techniques

In the third stage of this policy analysis research project we selected the appropriate statistical analyses for building and analyzing the academic merit and adversity indicators. The selection of an appropriate data analysis technique is a function of many factors, but one of the most important is the form of the data comprising the policy data base. The form of the data elements limits the statistical analysis, yet policy analysis research routinely demands data in many different forms, much of it is categorical or qualitative in nature. Consequently, many of the traditional inferential statistical methods could not be used. For this project, we used the exploratory data analytic tools suggested by Tukey (1977).

For each of the data elements, we generated simple frequency distributions to understand the range and shape of the data. The goals of the program were to identify no more than 5% of the student applicants for possible financial scholarship awards, yet within that small group we had to make very discriminating decisions. Hence, the adversity index had to be constructed in such a way as to set a cutting score at a high level, yet provide a way to spread students across three scholarship categories. To achieve the desired shape and form for this index, we looked at each data element and assigned points giving the top 5% of the data distribution a large number of points. The next 10% of the students in the distribution would receive fewer points, the next 10-15% only a small number of points and the bottom 70-75% of any given data distribution would receive almost no points. For example, we generated the distribution for the percentage of students participating in the federal lunch program within the Texas high and assigned points as shown in the table below. If students attend a high school in which more than 70% of the students were served by the federal lunch program, they received a larger number of points because that high school represented a more "adverse" environment. This kind of point assignment was made for each data element. The indicators, the points assigned and the percentages of students within each point category are shown in Table 2.

Table 2
1998 Adversity Index Point Values

The points were summed within each of the three broad categories - parent, school and peer performance indicators -- to provide an overall adversity index that ranged in value from 0 to 69. That distribution was then used to determine three award levels. These three award levels, called Extreme, Substantial and Moderate Adversity were combined with the academic merit indicator of high school rank. Hence, students who were in the top 5% of their graduating class and in the top 5% of the adversity index would receive a $5,000 renewable scholarship for four years. Students in other combinations of adversity or high school class rank would receive differing amounts of scholarship money. Table 3 below shows the possible categories of financial aid award.

Table 3
Presidential Achievement Scholarship Award Categories

In the next stage, called policy analysis simulation, we assigned points to each adversity indicator based on the percentage of students that fell within certain data distribution values. Through repeated iterations of assigning points, re-examining the data distributions and adjusting the number of points assigned, we achieved a balance between the sub-components of the family socioeconomic, the school and the peer performance indicators. This is a particular strength of policy analysis research and will be described in some detail in the next section.

Return to Table of Contents

Simulating Policy Decisions

The fourth stage in conducting policy analysis research is to conduct policy simulations by creating the conditions under which the new policy might be applied, examining the results, adjusting the policy to further refine the possible outcomes and then re-applying the "new" policy and re-examining the outcomes again. Figure 1. shows a graphical example.

Policy Simulation Analysis

Figure 1. Policy simulation analysis flow chart.

The number of simulations is unpredictable and depends on how well the policy produces the expected results. For the financial scholarship program, points were assigned to the data elements and new cutting points established numerous times. By applying the points to an entire population of admitted students, the number of awards and the amount of money could be related to the pool of potential recipients. In addition, part of the simulation provided for the opportunity to generate statistical profiles of these potential recipients. The statistical profiles were based on the data elements comprising the selection criteria for the scholarships. Hence, at the end of each policy simulation, the potential pool of award recipients could be compared with those who had not qualified under that simulation analysis. These statistical profiles allowed the decision-maker and research policy analyst an opportunity to "fine tune" the outcomes, not only in the number of students who received awards, but also in terms of the selection criteria defining the adversity index. If the criteria selected too many students for a given category of scholarship recipients, the cutting points for the award selections were adjusted. Likewise, if the policy simulation analysis provided awards to the "wrong" students, the number of points could be re-assigned to yield a more desirable mix.

These policy simulations form the heart of policy analysis research because the results of "applying" the policy to a particular group of students can be evaluated, adjusted and re-applied before having to make a "final" decision about the policy standards. In the case of the financial scholarships, we were able to estimate the number of students qualifying within a pool of admitted students. In addition, a statistical profile of the potential award recipients allowed the decision-makers an opportunity to make judgments about the "correctness" of their policy decisions. That is, were the "right" students ending up with the awards? At one point during the policy simulation analysis for this project, we shared the results with the President and executive officers of the University. They raised questions about how points were assigned to students when they came from a single parent rather than a two-parent family. We examined the statistical profile and made an adjustment to the points for the estimated family. We then tried the "new" point system for the entire admitted class to see if a different point weighting system provided a more equitable selection of students from single-parent families. Arriving at an appropriate number of points for this category of student within the larger population required several iterations, but ultimately led to an acceptable outcome for the decision makers. Without these many simulations of the policy, the final outcome for a dramatically new and different approach to select scholarship recipients would not have been possible.

A brief example from one stage of the policy simulation analysis is provided here to illustrate how changes in the policy standards yielded different policy outcomes. Our goal was to achieve two important outcomes. First, because we had a finite amount of money to award, we had to identify a specific number of students for each award type. Second, we wanted to achieve a particular balance among the contributing family, school and peer performance factors. By adjusting the total adversity index cutting score used to define the low, moderate, substantial and extreme levels of adversity, we could modify the number of students who qualified within each high school rank category. By adjusting the point values for the data base elements used to define the three subscale categories, we could change the relative importance of the family, school and peer performance in the final adversity numerical score. During the initial iterations, we manipulated the point values to achieve the type of balance among the contributing factors we thought was most important. Once we achieved about the right mix among the three components, we worked to adjust the adversity index cutting score in combination with the high school rank to achieve the number of students within each award category.

This example shows data from the "middle" of the policy simulation analyses. The first table shows the number of students and the mean adversity index subscale scores by high school rank. This table was used to summarize how many students fell into each potential financial aid award category and to show the average value of the adversity index subscores. These latter values were examined to determine if the relative amount of weight given to the parental factors, the school factors and the peer performance index were in about the right proportion. During these two iterations, we intentionally tried to reduce the number of students in the "extreme" adversity categories within each high school rank category. By raising the total adversity index cutting score, we reduced the number of students with extreme adversity from the top 5% of the high school class eligible for the $5,000 scholarship from 298 to 221. The relative mean values of the three contributing factors to the adversity index - the parent index, the school index, and the peer performance index - changed very little, though the school index changed more than the other two factors.

Table 4
Average Adversity Sub-scale Points by High School Rank and Total Adversity Categories

Through repeated iterations we were able to identify an appropriate number of students with the type of qualities we were seeking to match the amount of money available. The next stage, validating the policy, is important if we want to evaluate how well our new policy standards worked.

Return to Table of Contents

Validating the policy outcomes

While the policy simulations allow the policy to be applied and evaluated "before" it is implemented, all possible outcomes can not be anticipated. Ultimately, the test of time must be applied and the "real" results gathered and evaluated. If the outcomes are successfully achieved, the policy may remain in effect without further modification. However, the more likely scenario is that unanticipated outcomes occur and additional refinements must be made in the policy.

One way to validate the results of the policy analysis project was to generate "profiles" of students selected for the awards. These profiles helped answer the question, "Did we get the 'right' students?" The goal was to select students with economic need who would do well in college. Though the Hopwood decision and subsequent interpretation prevented the use of race as a criteria in the selection of scholarship recipients, the intent of the policy was to pursue the goal of increasing the ethnic diversity of the campus as well. By identifying economically disadvantaged students, we expected to include a substantial number of minority students. By comparing the demographic profiles of the scholarship recipients with the profile of the first-time entering freshmen who did not receive this scholarship, we were able to evaluate whether our policy simulations produced the expected results.

Table 5 shows the data for the Presidential Achievement Scholarship for the 1997-1998 year. When we examined the aggregate statistics for the variables contributing to the Adversity Index Point system, we found that socio-economically disadvantaged students were identified. For example, nearly 52% of the mothers of the PAS students stopped their education with a high school degree or less compared to less than 15% of the other freshmen. In contrast, nearly six out of 10 (59%) of the non-PAS students' mothers completed at least a four year degree compared with only 16% of the PAS recipients. More than 75% of the non-PAS students' fathers had at least a four-year college degree compared with only 16% of the PAS students' fathers. The family income of the parents of these two groups differed dramatically as well. Using the Adversity Index, we clearly identified economically needy students. For example, 76% of the mothers of PAS students earned $25,000 or less compared with 52% of the mothers of regular students. Approximately, 40% of the non-PAS students' fathers earned more than $55,000 dollars, while only 11% of the PAS students' fathers earned that much.

Table 5

A Comparison of the Demographic Characteristics of Presidential Achievement Scholar Recipients With All Other Entering Freshmen Fall Semester, 1997

Another way to validate our policy was to examine the academic performance of these students in the classroom. Because more weight was given to financial need and less weight given to academic merit, we could not predict how well these students would fare in the classroom. The data in Table 6 show the academic outcomes of the PAS students versus the Non-PAS students.

Table 6
A Comparison of the Preparation Level and Academic Outcomes of PAS Students with Non-PAS Students

When compared to the non-scholarship holders, the recipients had lower standardized test scores, but higher high school class rank. Because class rank was weighted nearly twice as heavily in the predicted grade point average than the SAT test score, the PAS students were predicted to do slightly better than the non-PAS students (PAS=2.90 and Non-Pas = 2.82). These students had a history of overcoming the adversity of a lower socioeconomic status, but they may have attended a school that did not prepare them for college. One type of evidence used to evaluate the policy was whether or not these students would succeed academically and return for their sophomore year. At the end of the first year, the PAS students earned a 2.78 GPA, slightly lower than the predicted GPA of 2.90, while the Non-PAS students earned a 2.87 GPA which was just slightly higher than their predicted GPA of 2.82. In terms of the retention indicators, the percentage of PAS students who returned to start their sophomore year in good academic standing was nearly identical to the Non-PAS students (79.9% versus 79.7%). A slightly larger percentage of PAS students than Non-PAS students returned on academic probation (6.6% versus 5.6%). In addition, a slightly larger percentage of PAS students than Non-PAS students left on academic probation and or dismissal and a smaller percentage left in good academic standing. However, most of the differences in academic achievement and retention status are very small. Overall, the PAS students and the Non-PAS students were highly similar in the level of academic achievement and their retention status at the start of their second year of college. Given that these PAS students were economically disadvantaged, had attended poorer high schools on average and were more likely to be first generation college students, the policy standards developed for this scholarship program succeed very well in identifying students. The policy analysis research conducted to establish the scholarship award criteria identified students with a track record of handling adversity.

Return to Table of Contents

Conclusions

The results of this study show that policy analysis research was a useful tool in establishing new scholarship criteria to replace race-based merit awards. Application of the new policy standards produced a cohort of economically needy students who performed well and were retained at rates comparable to the rest of their entering class. More importantly, this cohort of students represented a "new" category of student previously underrepresented at the university. Clearly, students with this level of economic "disadvantagedness" would not have had an opportunity to consider attending a flagship institution in the past. An additional benefit was that this new scholarship program also contributed to the ethnic diversity of the freshmen class. While it is difficult to determine how many of these students would have missed the opportunity to attend a flagship institution without the scholarship offer, it is clear that this entering class of students now contains students who were not part of previous classes of students. We plan to monitor their progress and expect them to graduate in large numbers. The ability to overcome adversity in high school should aid the achievement of their educational aspirations in college.

Return to Table of Contents

References