Click here to view main document in portable document format (PDF)

82


REPORT FROM THE FACULTY COUNCIL AD HOC COMMITTEE
ON COURSE INSTRUCTOR SURVEYS: EVALUATING TEACHING
EFFECTIVENESS AND EXCELLENCE


Patricia A. Alvey (Advertising), Chair of the ad hoc Committee on Course Instructor Surveys, has filed with the Secretary of the Faculty Council the report set forth below. Notice is hereby given that this report will be presented to the Faculty Council for action at its meeting on November 15, 1999.



<signed>

John R. Durbin, Secretary
The Faculty Council





Posted on the Faculty Council web site (http://www.utexas.edu/faculty/council/) on November 3, 1999. Paper copies are available on request from the Faculty Council Office, FAC 22, F9500.


83


REPORT FROM THE FACULTY COUNCIL AD HOC COMMITTEE
ON COURSE INSTRUCTOR SURVEYS: EVALUATING TEACHING
EFFECTIVENESS & EXCELLENCE



EXECUTIVE SUMMARY
Working under the guidance offered by The University of Texas System Faculty Advisory Study of Student, Faculty, and Administrator Attitudes Regarding the Student Evaluation of College Teaching, the original goals of this committee were to increase the efficiency and quality of teaching evaluation methods, improve the quality of information gathered, and improve the administrative uses of resulting information. Focusing on the original mandates to inform course choice for students, inform administrative and curricular decisions, and improve professorial development, the committee set about a four-phase plan. First, the existing literature on teaching evaluation was reviewed. Second, the materials and methods used on this campus were reviewed. Third, all colleges and student groups were presented with a draft version of recommendations and interviewed for their input. Lastly, comments were considered and these recommendations drafted.

We present here the recommendations, with pertinent background information, and rationale available in the appendices. Recommendations are aimed at serving three audiences„students, faculty, and administrators. We believe this report offers approaches that will provide: increased efficiency in the information gathering process; greater and more consistent access to information for students, faculty and administrators; more useful information for students, faculty and administrators; respect for both the public and private needs of students, faculty and administrators.

The first section of recommendations addresses revision and consolidation of the current Course Instructor Survey Forms. The second section of recommendations discusses alternative methods available for assessing teaching excellence and effectiveness. The third section of recommendations reconsiders administrative use of information gathered; privacy issues are also discussed in this section. Supporting materials are in the form of appendices attached to the report.

INTRODUCTION
In 1997 the Faculty Council, as mandated by the report The University of Texas System Faculty Advisory Study of Student, Faculty, and Administrator Attitudes Regarding the Student Evaluation of College Teaching, appointed a committee to examine and suggest improvements in the evaluation of teaching effectiveness at the University of Texas at Austin. (The full text of the UT System report will be made available on the Center for Teaching Effectiveness web site. http://www.utexas.edu/academic/cte/.) The broad goal was to ensure that students, teachers, and administrators all benefit from processes that are informative, efficient, and fair. This charge was broken down to several more specific objectives. For students, the objective would be to provide better data„and guidance in its use„with which to inform their choices of courses and teachers. Teachers would have access to feedback that would facilitate their efforts at becoming more effective in their craft. Administrators and academic departments, faced with difficult decisions regarding curriculum development and promotion, would likewise benefit from better information. In the interest of fairness and efficiency, the committee was asked to ensure uniformity of measurement and comparability of results across campus„in part by reducing the proliferation of Course Instructor Survey instruments in use. Fairness would also be a fundamental concern in regard to appropriate access to and interpretation of teaching evaluation data. The original charge to the committee is found in Appendix A.

Our research and deliberations, informed by generous feedback from various constituencies of the University community, led us to focus on the following three areas:

I.
Improving the Course Instructor Survey instruments and reducing the proliferation of forms. There are 24 CIS forms in use, and the data they generate cannot be readily compared. Research indicates that important indicators of teaching effectiveness are not always measured by these instruments, while questionable or spurious measures are given equal weight.


84


II.
Identifying appropriate alternative methods of evaluating teaching effectiveness. Robust and reliable as revised course instructor surveys may be, they can tell only part of the story. Teaching portfolios, peer review of syllabi and other materials, exit and alumni surveys, and other such alternative methods can be especially useful for professorial development, curriculum planning, and promotion decisions.

III.
Administrative Uses: Privacy, Fairness, and Consistency. Evaluation data are powerful and must be employed fairly. Decision makers„including administrators and students„should be fully informed as to what conclusions can and cannot be drawn from the data. Students should understand their role in providing fair and productive feedback. Consistent and transparent policies across campus regarding the use of data are essential to the professional relationship of faculty and administrators. Finally, privacy concerns over release or other misuse of studentsÍ written comments must be addressed.


SECTION I. IMPROVING THE COURSE INSTRUCTOR SURVEY INSTRUMENTS AND REDUCING THE PROLIFERATION OF FORMS.

COMMITTEE CHARGE 1. To consider proposals for a new common form that would be more informative to students, faculty, and departmental review committees.

There are currently 24 different CIS forms in use across the university in addition to the five-item common form. A list of the forms and their frequency of use during the fall of 1997 and spring of 1998 is contained in Appendix B1. A list of the forms and their frequency of use during the fall of 1998 and spring of 1999 is contained in Appendix B2. These forms have a large amount of overlap in the content of the items. While many of the items that overlap across the forms are applicable to almost every type of class, there are items on several forms that are only appropriate to specific types of classes.

The committee recommends that the items that are applicable to teaching in general be consolidated into two forms: a Basic CIS Form with 11 items to replace the existing common form, and an Expanded CIS Form with 24 items. It is recommended that all classes be surveyed with one of these two forms. The 11 items on the basic form are also on the expanded form, and the ratings on nine of these items would be automatically released and put online for use by the university community. Of these nine items, all except one have been shown to be related to student learning. The items on the proposed basic and expanded forms, along with their correlations with student learning are shown in Appendices C1, C2, and C3.

Items that are unique to specific types of classes or teaching methods are contained on specific forms that may be used in addition to either the basic or expanded form. The names of the proposed specific forms are:

Student Teaching
Foreign Language
Studio Art and Design
Applied Music
Discussion Section
Laboratory Nursing: Clinical Rotation
Teaching Assistant

These forms are designed to answer essential questions about every course offered on campus while offering important information to students and maintaining professorial freedom for faculty. ItÍs important to note here that no choices are eliminated; faculty members are invited/encouraged to add their own questions. Numerous additional survey items can be found on the MECÍs Course Instructor Survey web site at http://www.utexas.edu/academic/mec/cis under the heading "Using your own questions."


85


COMMITTEE CHARGE 2. To consider the proposal from the Cabinet of College Councils that every professor in every organized course be evaluated by the common CIS form every semester. If this is done, in what special circumstances should courses be exempted, and by whom? There have since been additional requests from students that all resulting information be available online.

Although on-line information is currently available to anyone in the university community, only faculty members who use the common form are represented. During the 1997-1998 and 1998-1999 academic years approximately 65% of organized courses used the common form and 79% used either the common form or another CIS form. We propose that all organized courses be evaluated with either the Basic CIS Form or the Expanded CIS Form. Should faculty members want or need additional questions or discipline or teaching method specific forms, they should either add questions or use one of the supplemental forms available from the Measurement and Evaluation Center. Exceptions may be approved by the dean or department chairperson.


SECTION II. IDENTIFYING APPROPRIATE ALTERNATIVE METHODS OF EVALUATING TEACHING EFFECTIVENESS.

COMMITTEE CHARGE 3. To consider whether there are other measures we should adopt for simplifying or improving course evaluation.

In an attempt to explore this topic, we have reviewed the literature on faculty evaluation, polled departments for new ideas, and brainstormed ideas using an instructor characteristic matrix as a heuristic. In truth, there are only a few workable alternatives to offer and most are time-consuming and require departmental agreement on criteria. However, the table in Appendix D contains a summary of the possible sources of data that could be used in evaluating different aspects of teaching that the literature has shown to be related to student learning.

The primary vehicles for collecting information about teaching are student surveys, alumni surveys, major retrospective surveys, peer review of materials, self-report in the form of portfolios, and midsemester course adjustment data. The last of these is primarily useful for formative evaluation and should probably not be included in evaluation for promotion and tenure except as requested or provided by the instructor for comparative purposes.

Review of the literature indicates that one data source that has become more widely used on campus could benefit significantly from enhanced administration methods. In many departments, the concept of peer review has been taken to mean peer observation of in-class instruction. The literature on teaching evaluation cautions institutions that this type of evaluation is fraught with problems and should be used only as support for other data. In addition, departments are recommended to precede any widespread use of peer observation with a department-wide discussion of critical parameters to observe. Such discussion would actually be a worthwhile activity in terms of creating a common understanding of what "effective teaching" might mean for a given department. (For example, does the department value depth of coverage over stimulation of interest or both and to what degree?) Once the parameters of observation have been agreed upon, some training of observers is recommended so that there is reliability between different observers. It is also more appropriate for this type of observation to rely on observable behaviors rather than inferred (e.g. "The instructor spoke with animation" rather than "The instructor was enthusiastic.").

Recognizing that in any appropriate evaluation environment multiple data sources should be used, we are recommending increased use of alternative methods across all colleges and departments. We are also recommending that it would be appropriate to gather different types of data from different sources since not all sources can be depended upon for all types of questions. The Center for Teaching Effectiveness has literature available that describes each of these methods. A sample mid-semester report that is currently used in one campus department is presented in Appendix E.


86



SECTION III. ADMINISTRATIVE USES: PRIVACY, FAIRNESS, AND CONSISTENCY.

COMMITTEE CHARGE 4. To consider guidelines for administering CIS, particularly for informing students prior to the process.

COMMITTEE CHARGE 5. To consider guidelines for professorial preparations for and understanding of the course instructor survey as well as alternative methods of evaluating teaching effectiveness.

COMMITTEE CHARGE 6. To consider guidelines for administrative uses of information.


Currently, neither students nor faculty receive guidance on how to request, provide or evaluate teaching effectiveness or student learning. The committee recommends preparation of materials aimed to help students understand the importance of giving helpful feedback and the importance of understanding how to read results made available. The committee sees two places where helpful information can reinforce students' giving and receiving of information: 1) on the evaluation form itself, a very succinct explanation as to what the information is used for and to whom itÍs distributed, 2) in the online environment, when a student visits online results, a similar explanation of the results and what the reader can infer from them. The Measurement and Evaluation Center will prepare these explanations and include them on the new forms and online material. Equally important is information made available to the student volunteers who administer course evaluations. Instructions to administrators will also be revised. Both the Measurement and Evaluation Center and The Center for Teaching Effective online pages will address these issues and provide appropriate links.

Probably even more critical to understanding how to give feedback is how to ask for it. Most faculty members receive no guidance in preparing their classes to evaluate the effectiveness of the classroom experience. Many temporary lecturers get to the end of their first semester without understanding that evaluation will take place. The committee recommends making information easily available to faculty that will guide them in critical issues of timing, presentation, etc. The Center for Teaching Effectiveness is preparing both online and printed tutorials for faculty.

One strategy for improving the value of the evaluation information is preparing those who will be using it to use it wisely. These users are the students, the faculty, and the decision-makers of the institution. Students need to be prepared both to give and interpret information of the CIS. Faculty need to be prepared to gather and interpret information gathered by the CIS and other vehicles. Decision-makers need to be prepared to interpret evaluation data of all kinds. The Center for Teaching Effectiveness is in the process of preparing guidelines for the following situations:

  1. Instructions to be placed on the CIS form to alert students to its potential uses and to encourage their serious participation;
  2. Instructions to be placed in the Web database to advise students about how to interpret information they seek about various course evaluations;
  3. Instructions to faculty about preparing students to do an in-class evaluation so that there will be sufficient participation and useful information gathered;
  4. Instructions to faculty about interpreting a given CIS result and about making the best use of CIS results and other evaluative feedback for course revision and professional development;
  5. Instructions to decision-makers about the use of various forms of evaluative data and statistics so that the information is used appropriately;
  6. Instructions about other forms of data collection that can be used to provide information about teaching. As these forms become more widespread, there will be guidance about their administration and use. An example of this is already available online through the CenterÍs website describing the preparation of teaching portfolios.

COMMITTEE CHARGE 7. To consider the request of various faculty members that the privacy of faculty be protected in regard to written comments on evaluations: that such written comments be the property of the professor


87


in question, and that no evaluating committee or administrator at any level may see them without the permission of the professor and that such permission must be voluntary

Written comments on student evaluation forms are not governed by the Open Records Act that governs access and release of answers to survey items. This communication is offered and received as a private act of communication between student and faculty. The committee recommends that the privacy of both the students and faculty be respected and preserved and in so doing it becomes the purview of the faculty member as to how and when these are released.


CONCLUSION
The committee believes this set of recommendations offer increased efficiency, more consistent information and potential for improved educational experience for students, faculty, and administrators. We would like to applaud the original UT System report for its broad vision, in-depth background, and forward thinking. The information gathered and distilled in this report is only possible as a result of the gracious cooperation of all involved parties. Every school and college responded with insightful information. Each student organization offered helpful suggestions. Most importantly, the Faculty Council has been graceful and patient while this group undertook an arduous and important task. We applaud the University community for embracing the exciting and difficult task of change in pursuit of excellence.


List of appendices to be viewed as PDF file.
  1. APPENDIX A: Faculty Council Ad Hoc Committee on Course-Instructor Surveys Approved by the Executive Committee of the Faculty Council, 1/21/97 Charge

  2. APPENDIX B1: Participation in the Course-Instructor Survey (CIS) System: Counts within Colleges of the Number of Classes Using CIS Questionnaires, Fall 1997 and Spring 1998

  3. APPENDIX B2: Participation in the Course-Instructor Survey (CIS) System: Counts within Colleges of the Number of Classes Using CIS Questionnaires, Fall 1998 and Spring 1999

  4. APPENDIX C1: The Basic Form

  5. APPENDIX C2: Basic From Item Rationale

  6. APPENDIX C3: Expanded Form

  7. APPENDIX D: Assessment of Critical Characteristics Related to Student Achievement

  8. APPENDIX E: Sample Mid-Semester Evaluation