|
|
|
Contents:
Definition of Program
Outcomes-Based Assessment
Educational Program Framework
Approach
Worksheets
Definition of "Program"
"Programs" are defined here as undergraduate and graduate degree programs. SACSCOC uses
the description (in Comprehensive Standard 3.4.1) of programs as ones for which academic credit is awarded,
are approved by the faculty and the administration, may include majors, and includes on-campus, off-campus,
and distance learning programs.
Outcomes-Based Assessment
Comprehensive Standard 3.3.1 states that: The institution
identifies expected outcomes for its educational programs …; assesses whether
it achieves these outcomes; and provides evidence of improvement based on
analysis of those results. This criterion has been in place for some time
and in his memorandum of February 23, 2000
to the deans, Provost Sheldon Ekland-Olson called for all academic programs to
identify assessment outcomes and to review their programs on a regular basis to
determine how well programs meet the evolving scholarly demands of their disciplines
and the shifting nature of the marketplace students will enter.
To elaborate, outcomes-based criteria are based on continuous improvement. One
of the consequences of this approach is a continuous effort to show that the
Outcomes/Learning Goals are being achieved through
information gathering, determination of areas of improvement needed, making changes
to achieve those improvements, and then showing that those changes were effective.
This “closing the loop” approach takes time (two to
three years per “loop”), but at accreditation review time, the program evaluators
will want to see evidence that assessment processes are in place and that the program
has cycled through this loop a couple of times at least.
Educational Program Framework
The framework for outcomes-based assessment can be thought of as an interrelated set of parts as shown in the diagram below for an undergraduate program.
(Click on image to view larger size)
Remembering that a program is defined here as a degree-granting program),
Program Educational Objectives (PEOs) are established
which describe what graduates of that program are able to do several years after
graduation. Such PEOs can be rather general statements such as be educated citizens,
be productive members of society, be leaders in the state, be able to pursue advanced
work in graduate programs, or they may be more specific such as be employed by industry
– perhaps particular industry important to the region, be able to pursue a career in
higher education, and so forth).
Graduates of these programs need to know and be able to do certain things, have
certain values or attitudes, and exhibit certain attributes and behaviors to be able
to accomplish the PEOs of the program. These Program Outcomes
as they are called are statements that describe fairly specifically what knowledge, skills,
abilities, attitudes, behaviors, and/or capabilities the students should be able to demonstrate
at the time they graduate, and these Outcomes are ones that students need to have to be able
to achieve the Program Educational Objectives (PEOs). For each PEO
there should be one or more POs that contribute to it so that through a simple matrix of PEOs
and POs it is easy to note which POs support with PEOs. An example might be for three PEOs and four POs:
| |
PEO1 |
PEO2 |
PEO3 |
| PO1 |
X |
|
|
| PO2 |
|
X |
X |
| PO3 |
X |
|
X |
| PO4 |
|
X |
|
Note that each PEO is supported by at least one PO; if a Program Outcome is not linked to any
Program Educational Objective, then that Objective will not be achieved within the curriculum
assuming the curriculum is constructed to achieve the Outcomes. These PEOs and POs are developed
by the faculty within the program with advice and counsel from their major constituents which can
include their students, employers, alumni, and others the faculty decide to consult.
Based on the POs and the PEOs, a curriculum is developed by the faculty for that program to
provide students the education they need to achieve the POs at the time they graduate and the PEOs
within several years. (If a curriculum is already in place, then the faculty review the curriculum
and update it as needed.) This curriculum contains the Core Curriculum courses (courses that are
part of the General Education Requirement) as well as courses in the major – each with learning
objectives defined so that students and faculty know what students should know and be able to do
once they finish a particular course. These courses, designed to be offered at different levels of
content and cognition, are combined into sequences, groups, and other modes to be taken by students
so that at the end of the program the students are indeed able to demonstrate mastery of the POs and
then after graduation the PEOs. Of course, excellent teaching, advising, and other academic support
activities as well as co-curricular activities play a role in this success.
How do faculty know that students are learning what needs to be learned, have the knowledge,
skills, abilities, that they want students to have by the time they finish the degree program? By
gathering and analyzing various types of data (e.g., student work, normed examinations, surveys,
etc.), the faculty demonstrate that POs are being met through the program and the courses in the
program and whether students are achieving the PEOs within a few years of graduation. Faculty
decide how those PEOs and POs will be measured and the standard that is set for students to
demonstrate that they have achieved them. Where POs and/or PEOs are not being achieved at the
desired level, it is the task of the faculty to find out why, what to change in the curriculum
(or perhaps co-curricular activities) so they can be achieved, to make those changes, and then to
show that the changes were effective. This of course applies to individual courses and students’
achieving their learning objectives as well. As noted earlier, this “closing the loop” approach
takes time (for academic programs, about two to three years per “loop”), but at accreditation review
time, the program evaluators will want to see evidence that assessment processes are in place and that
the program has cycled through this loop a couple of times at least. This improvement cycle can also
take place in response to changes in the field, etc. so that improvements might occur in the absence
of any data collected from students.
Approach
Developing Program Educational Objectives, Program Outcomes, and an Assessment
Plan that shows whether they are being achieved is a rather straightforward
process, not necessarily easy, but straightforward. The Assessment
for Quality Assurance diagram below, developed by Gloria Rogers, provides
an excellent framework to describe the approach to outcomes-based assessment
as applied to academic programs, and significant portions of this text
have been drawn from her “Assessment Planning Flow Chart©” CD
and her article “One model
for EC2000 assessment report©(pdf).”
Click on image to view larger size)
The steps involved are as follows:
- Mission – assessment planning begins with the mission of the program.
That mission is typically captured in the mission statement for the department,
school, and/or college in which the program resides, and these in turn are
consistent with the mission of the institution. The institutional mission
statement describes the communities that are served by it, the purposes
of the institution, and other characteristics that define the institution.
For example, the Civil Engineering degree program is housed in the Department
of Civil, Architectural, and Environmental Engineering, and that department’s
mission, in part, is “to educate our students to be engineering leaders
and to solve complex problems of importance to society.” Its teaching mission
is aligned with the College of Engineering’s, which is “to provide an educational
experience that inspires students to reach for the highest levels of intellectual
attainment and personal growth throughout their lives.” And the College’s
educational mission is in turn aligned with that of The University of Texas
at Austin which “provides superior and comprehensive educational opportunities
at the baccalaureate through doctoral and special professional educational
levels.” Thus, all educational programs should have missions that align
with the University’s as well as those of the school/college and department
in which they are housed.
- Develop Program Educational Objectives (PEOs) – as noted above, PEOs are
statements that describe the expected accomplishments of graduates during
their first few years after graduation, and these objectives should be consistent
with the mission of the program and the institutions. Thus, to develop PEOs,
faculty need to address the question, “What role(s) will our graduates play
in society with an education in our particular discipline?” More specifically,
are they expected to be informed, productive members of society, leaders in
their fields, or workers to support the economy of the region and the state?
Are they expected to be able to seek advanced degrees, work in specific types
of industry, contribute in ways unique to the discipline, and so forth? Faculty
can address these questions to some extent, they know what their expectations
are of their graduates, but they can address them better by consulting with
their alumni and the employers of their graduates. Development offices and
alumni associations can help faculty locate their graduates, and placement
offices can provide lists of employers who hire their graduates. By inviting
representatives from these two groups to campus for discussions, faculty can
test their PEOs and sharpen them as well as develop PEOs suggested by the
visitors. The number of PEOs need not be long; some 3 to 5 PEOs should be
adequate, and they should be ones that can be measured to show whether they
are being achieved. Example lists of Program Educational Objectives can be
found in the Resources section.
Program Educational Objectives need
to be assessed and evaluated periodically, perhaps every 3 to 5 years, to
determine their continued relevance to the needs of the program’s constituents.
This is generally done through alumni, employer, recruiter, and/or advisory
board assessment. If done every four years, then any changes needed can
be incorporated into the Undergraduate Catalog during its update cycle,
assuming they are published therein.
- Develop Program Outcomes (POs) – because POs are statements describing the
knowledge, skills, abilities, attitudes, attributes, values, and/or behaviors
that faculty want students to have at the time they graduate and because they
know these Outcomes are ones that their graduates will need to be able to
achieve the PEOs set for the program, faculty can begin the process of developing
the POs in brainstorming sessions listing examples of outcomes. These can
be distilled into 5 to 8 Outcomes, although some programs will have more and
should have more. Program Outcomes should be as discipline specific as possible
and should include those considered critical to the graduates of a given program.
It will be important that the Outcomes developed be measurable, like the PEOs,
to show whether they are being achieved. Example lists of Program Outcomes
also can be found in the Resources section.
- Measurable Performance Criteria – performance criteria are specific, measurable
statements identifying the performance(s) required to meet an outcome. For
each outcome, faculty should determine the performance criteria. These criteria
are high level indicators that represent the knowledge, skills, abilities,
attitudes, or behaviors students should be able to demonstrate by the time
of graduation that indicate competence related to the outcome. This step is
critical, for the performance criteria define what the faculty mean by each
of the outcomes (e.g., what is meant by “effective communication”). Once outcomes
are defined by performance criteria, assessment techniques can be developed
or identified for the collection of evidence of student achievement. Student
performance across the program may then be consistently and systematically
assessed. The criteria become the metrics used to determine achievement of
outcomes.
- Educational Practices/Strategies – based on the performance criteria for
each outcome identified, educational practices are reviewed to determine which
ones may prove useful to provide data for assessment of outcomes – practices
such as examinations, term papers, homework, portfolios, artistic work, performances,
etc. Strategies are developed to determine how best to sample those educational
strategies, strategies such as create a questionnaire, survey, rubric, etc.
to collect data from faculty (assessment of student work), students, alumni,
and/or employers. The items on the assessment instrument should reflect the
performance criteria established in the previous step. Understanding the alignment
between educational practices and strategies promotes efficient and effective
assessment practices. This can be accomplished by mapping educational strategies
(which could include co-curricular activities) to learning outcomes.
- Assessment: Collection, Analysis of Evidence – strategies for data collection
and analysis need to be developed that are consistent with the assessment
question, resources available, appropriate validity, and utility of findings.
This plan for data collection and analysis should include who is going to
be assessed, in what class/situation, how often, etc. Faculty need to be careful
to collect only the data needed for assessment; more data are not always better.
Be efficient and know the long-term strategy involved. Compiling the data
gathered in a single notebook or file for each outcome is an efficient way
to store and to present the data gathered.
- Evaluation: Interpretation of Evidence – evaluation is the process that
is used to determine the meaning of the assessment results. This includes
the implications of assessment results related to program effectiveness and
recommendations for improvement. This process is analogous to developing the
conclusions and recommendations from data gathered in a research project.
Evaluation should include those faculty and others who can implement improvement
strategies, for making needed changes in the curriculum, in particular courses,
in teaching methods, and so forth is usually the result of this step.
- Closing the Loop – “closing the loop” is using information from various
parts of the assessment process to improve the educational program and showing
the impacts of changes made to generate that improvement. For example, output
from the Evaluation: Interpretation of Evidence step can provide evidence
of a problem that exists because a Program Outcome is not being achieved at
the performance level desired. It can also show that resolving the cause of
the problem has enhanced student performance so that Program Outcomes are
achieved. This feedback process is critical to creating and maintaining a
systematic quality assurance system. When successfully implemented, all elements
of the quality assurance process interact with one another.
While it is very important that outcomes-based assessment be done carefully and thoroughly
o achieve continuous improvement of academic programs, it is also important to keep it simple
so that it does not overburden faculty or staff and detract from their primary work for the
institution. Thus, Dr. Gloria Rogers in her workshops held in October 2005 focused on the key
elements of outcomes-based assessment, while Dr. Mi-Suk Shim in her Spring 2006 workshops showed
how to develop Program Outcomes and an assessment program. Materials from both workshops are
available in the Resources section.
Worksheets
Information about Program Outcomes, assessment methods, levels of performance,
etc. will be captured initially in a worksheet (.doc)
before transferring the information to the database established to house
outcomes-based assessment information from programs across campus. See
the Resources and FAQ sections.
Download Word Viewer
|
|
|
|
|
