View in portable document format.
D 11450-11460


DOCUMENTS OF THE GENERAL FACULTY 



REPORT FROM THE EDUCATIONAL POLICY COMMITTEE ON REQUIRED AND PROPOSED REVISIONS TO THE BASIC FORM COURSE EVALUATION

On February 27, 2014, and on behalf of the Educational Policy Committee, Professor Mary Rose (committee chair, sociology) submitted the following report recommending modifications to the Basic Form Course Evaluation.

The Faculty Council will discuss the report at its meeting on March 17, 2014.
neikirk signature
De an Neikirk, Secretary
General Faculty and Faculty Council



Posted on the Faculty Council website on March 11, 2014. 


REPORT FROM THE EDUCATIONAL POLICY COMMITTEE ON REQUIRED AND PROPOSED REVISIONS TO THE BASIC FORM COURSE EVALUATION

Background, Proposal, and Policy Rationale

In 2013, UT System sent a memorandum to campuses informing them that five questions would now be required to appear on campuses’ course evaluation forms. That memo appears as Appendix A (the memo also discusses online course evaluation; the EPC is currently not making a motion on this issue but may do so – or may be required to do so – at a later time). The new items must be the first items on the form, must be asked in precisely the way they were written, and must use the response categories provided. Thankfully, UT currently already uses the response categories mandated by System.



In October, the Educational Policy Committee (EPC) began the task of looking at our existing Basic Course Evaluation form in order to determine what items could be deleted so as to accommodate the new mandated items. In addition, we looked for any ways that other items on the form could be simplified or asked in better way. Committee members took this job very seriously and debated issues intensively. Discussion consumed fully four meetings (see minutes for October, November, December, and January meetings), and we concluded remaining business via an emailed vote.

Here are the new mandated items from System:

 
Strongly Disagree
Disagree
Neutral1
Agree
Strongly Agree
  1. The instructor clearly defined and explained the course objectives and expectations.
1
2
3
4
5
  1. The instructor was prepared for each instructional activity.
1
2
3
4
5
  1. The instructor communicated information effectively.
1
2
3
4
5
  1. The instructor encouraged me to take an active role in my own learning.
1
2
3
4
5
  1. The instructor was available to students either electronically or in person.
1
2
3
4
5


1Please Note: “Neutral” does not mean “Not Applicable.”


For reference, the items that have appeared in the UT Austin Basic Form are included in Appendix B. In revising the form, we were guided by a few principles. First, Harrison Keller reported that he had heard that department chairs and others wanted no changes to the two “overall” questions (overall instructor rating, overall course rating) that currently appear in the second section of the form (see Appendix B). Departmental leaders felt that these items are such an integral part of teacher evaluation (for promotion, awards, etc.) that they should not be changed. Second, we looked for any redundancies in the above new items and those we currently have. Third, we used the opportunity to address concerns that have come to EPC before but, at the time, were too costly to implement (i.e., we could not change the course evaluation just for those issues). In particular, students had expressed a desire for evaluation forms to rate the usefulness of assigned course materials (e.g., textbooks), particularly because such items can be costly and students wanted to be able to communicate whether these items were useful for the course. In addition, representatives from Fine Arts have long noted that their students often use the “Neutral” category to indicate that a particular item being assessed (e.g., concerning return of tests and assignments) is “Not applicable” (because those courses have no assignments or tests). This artificially gives the instructor a lower score on that item. We believed that, if changes could address these issues, we should aim to do so given that we were revising the form anyway. Finally, we hoped to keep the form about as long as it currently is – although there is room to add one or two additional questions because the current form has a lot of space for “optional questions provided by the instructor.”

EPC proposes that the new Basic Form look as follows:

Item
Deleted/Kept
Rationale
  1. The course was well organized
Deleted
We viewed this item as partially redundant with a new one, which asks about whether the course was organized in a specific way (making goals clear). In addition, some felt that courses may look “disorganized” to students but be intended pedagogically to be somewhat unpredictable. We thus preferred the new item’s focus on a specific way the course was organized to a general, broad assessment.
  1. The instructor communicated information effectively.
Kept
This item appears verbatim in the new mandate.
  1. The instructor showed interest in the progress of students.
Deleted
While asking about a laudable quality, this item generated a good deal of debate about what it means for instructors to “show interest” in students’ progress: That they give timely and helpful feedback?  That they seem to care?  That their tests are not too hard?  In addition, committee members who taught large courses felt that they are subject to penalty on this item because their interactions with students are often less individualized by necessity. We replaced this item with one, drawn from evaluations from other campuses, that asked about whether instructors seemed interested in “student learning,” which seemed appropriate to assess for classes of any size.
  1. The tests/assignments were usually graded and returned promptly.
Deleted
Committee members identified this question as having one good feature: if professors know that they will be evaluated on returning tests and assignments within a reasonable time frame, they may be more motivated to attend to this issue. At the same time, this particular item was deemed problematic because “promptly” is highly subjective, and many students seem to equate “prompt” with “immediate.” Further, some courses have no tests/assignments (e.g., in Fine Arts), and students sometimes mark a “3” (neutral) to mean “not applicable,” which artificially lowers scores for these sections. Given the need to reduce items, the Committee felt this item was both problematic and offered less value to course assessment.
  1. The instructor made me feel free to ask questions, disagree, and express my ideas.
Kept
The committee felt this was an important dimension of evaluation to preserve because free expression and debate are such key aspects many college-level class, and the ability to ask questions is crucial for all college classes. Additionally, given that faculty are sometimes criticized for allegedly “stifling” discussion (e.g., by having strong political views), the Committee felt it important to have a clear assessment that supports or refutes this perception for any particular class.
  1. At this point, I feel that this course will be (or has already been) of value to me.
Deleted
Like the tests/assignments item above, the Committee felt that although this item has some potential value, its limitations outweighed its potential strengths. “Value” is a challenging thing to judge, as some courses do not reveal their value until long after they were taken. It also seems to penalize courses that are required for graduation, because students may not easily see value in a course they did not choose to take. Given space limitations and a desire to measure other factors, this item was deleted.

Section 1.

 
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree
  1. The instructor clearly defined and explained the course objectives and expectations.
1
2
3
4
5
  1. The instructor was prepared for each instructional activity.
1
2
3
4
5
  1. The instructor communicated information effectively.
1
2
3
4
5
  1. The instructor encouraged me to take an active role in my own learning.
1
2
3
4
5
  1. The instructor was available to students either electronically or in person.
1
2
3
4
5
  1. The instructor made me feel free to ask questions, disagree, and express my ideas
1
2
3
4
5
  1. The instructor was genuinely interested in student learning.
1
2
3
4
5
  1. The course materials (e.g., text and supplemental materials) were helpful to me.
1
2
3
4
5


Section 2.

  1. Overall, this instructor was
Very unsatisfactory
Unsatisfactory
Satisfactory
Very Good
Excellent
  1. Overall, the course was
Very unsatisfactory
Unsatisfactory
Satisfactory
Very Good
Excellent
  1. In my opinion, the workload in this course was
Excessive
Somewhat high
About right
Somewhat light
Insufficient
  1. My overall GPA to date at UT is
Less than 2.00
2.00 – 2.49
2.50 – 2.99
3.00 – 3.49
3.50 – 4.00
  1. My probable grade in this course is
A
B
C
D
F


To make the changes clearer, we next present items from the old form, with the changes explained:

Section 1:

We added two other questions to the form, one mentioned above – “The instructor was genuinely interested in student learning” – to replace a rating about interest in student “progress.” The second was designed to address students’ long-standing desire to have a way to report back to professors if assigned supplemental materials are used and seemingly useful. A committee member who oversees a program that has multiple sections also wanted to be able to compare across classes to see if some sections use materials that are rated more highly than others. Thus, the final question in Section 1 is: “The course materials (e.g., text and supplemental materials) were helpful to me.”

As a bloc, the above revisions to Section 1 received a unanimous vote.

The new items added by System did not affect Section 2 of the form (e.g., there were no redundancies between the new items and current items in Section 2). However, the Committee used the opportunity to make some slight editorial changes to this section:

Item

Change

Rationale

  1. Overall, this instructor was

(none)

It was communicated to the Committee that this item should be kept as is.

  1. Overall, this course was

(none)

It was communicated to the Committee that this item should be kept as is.

  1. In my opinion, the workload in this course was

Revised the response scale slightly

The current response scale has options of “Excessive, High, Average, Light, and Insufficient.” The committee discussed the fact that the item appeared to be asking for two things simultaneously: A normative judgment about whether there was, in some absolute sense, “too much” or “too little” work in the course, as well as a comparative judgment about whether the workload matched other courses, which is suggested by the word “Average” in the response scale. As a fix, the response scale could be corrected, or the item could be rewritten to ask for a comparative judgment of how the workload in the current course compares to other courses. There was support for either approach (as well as some support for just leaving it as it is), but revising the response scale to match the normative assessment received more votes (6 in favor versus 4 in favor of changing to a comparative item versus 3 in favor of keeping as is). The new response scale changes the middle category and also, to improve the variability of the scale (to better demarcate “high” from “excessive” and “light” from “insufficient”) changes two other categories slightly. We propose that the response categories should read:

“Excessive, A Little High, About Right, A Little Low, Insufficient.”

  1. My overall GPA to date at UT is

(none)

The Committee entertained a motion to delete this item. Although research shows that other schools do ask the same question, there was concern that students regard it as intrusive and/or may simply skip it. However, because there was also a feeling that the item may contextualize ratings and give additional information, a final (close) vote favored keeping it (7 to keep vs. 6 to delete).

  1. My probable grade to date in this course is

Deleted “to date”

We felt the language in the question was contradictory: the word “probable” requires a guess about how students think they will do, whereas the words “to date” ask for an assessment of how they have done before the final (when evaluations are done). The Committee voted unanimously to delete the words “to date.”



If Faculty Council approves the Basic Form as submitted, these changes will also be automatically applied to the Extended Form. EPC is currently examining the Extended Form to consider other possible deletions or additions in order to reduce redundancies, make items clearer, and introduce other possible valuable dimensions on which to evaluate faculty teaching (but also aiming to keep the Extended Form the same length).


Policies for Evaluating Faculty: Recommendations for
Incorporating Student and Peer Reviews in the Faculty Evaluation Process

Overview

Effective teaching is the core of any outstanding university and is very important at every institution in The University of Texas System. UT System invests significant resources in rewarding outstanding teaching, and effective teaching is a requirement for the promotion and tenure of every faculty member. The System campuses have centers which support good teaching, and many departments work collegially to improve and augment the development of teaching within specific programs. Thus, it is entirely appropriate that excellence in teaching serves as an important foundation for a System-wide task force of faculty members and students.

Task Force Background

In 2011, The University of Texas System Chancellor unveiled his Framework for Advancing Excellence. The Framework is an action plan to implement and measure the effectiveness of nine overarching goals aimed at advancing UT institutions. Included among the Framework goals is an item addressing faculty excellence, specifically, to strengthen performance evaluations. The Chancellor appointed two task forces in 2012 to recommend ways to address this strategy.

The Task Force on the Evaluation of Faculty Teaching was charged to:

  1. Identify an appropriate, consistent, and limited set of faculty teaching evaluation questions that can be administered System-wide;
  1. Recommend a process consistent across all campuses that incorporates the critical questions which evaluate faculty teaching at the end of the semester; and
  1. Identify mechanisms to provide faculty feedback throughout the semester.
The Task Force on Faculty Peer Observations of Teaching was charged to develop a policy that every academic campus could adopt regarding faculty peer evaluations, including guidelines for implementation and a template form.

In February 2013, a work group was organized to review the recommendations of both task forces and develop a set of instructions for campuses to follow to implement the recommendations. This document provides the guidelines developed by the work group and approved by the Chancellor. Each campus is expected to incorporate these items into their policies addressing faculty evaluations and begin applying the student evaluations policies in Fall 2013 and the faculty peer review policies in Fall 2014 or earlier.


Appendix A – System Memorandum
Guidelines for Student Evaluations of Faculty

Task Force Background

Texas Education Code Section 51.974 requires institutions of higher education to conduct end-of-course faculty evaluations and make the evaluations available on the institution’s website. Most universities have accommodated this request by providing summary responses to a general overall evaluation question. The UT System would like to expand beyond the overall question, but maintain consistency across campuses.

In Spring 2012, a task force was created to identify a consistent method of evaluating faculty teaching across the UT System. The Task Force on the Evaluation of Faculty Teaching consisted of representatives from across the UT System, including students and faculty from academic and health institutions. The group met regularly throughout the spring and summer to identify a common set of evaluation questions, recommend an evaluation process, and identify mechanisms for providing continuous feedback between faculty and students. Based on the recommendations presented in the task force report, the following information is provided to assist institutions in complying with the new requirements affecting student evaluations of faculty teaching.

General Points
  • For the purposes of student evaluations, faculty members are defined as the courses’ instructors of record. Faculty members deliver the curriculum and are identified by the campus as the courses’ responsible parties.
  • Confidentiality of student evaluations of faculty teaching must be protected, and it is important that the methods used to maintain confidentiality are clearly demonstrated to students. Evaluations will not be administered for any class containing fewer than five people, as of the day after the final university drop date. If a class contains five or more students, but fewer than five completed the evaluations, the evaluation data will be utilized.
Mandatory Survey Questions

Each campus will incorporate the following five questions in every end-of-course student evaluation survey. The questions should be the first five questions of every end-of-course evaluation. The questions must be in this specific order with this specific wording:
  • The instructor clearly defined and explained the course objectives and expectations.
  • The instructor was prepared for each instructional activity.
  • The instructor communicated information effectively.
  • The instructor encouraged me to take an active role in my own learning.
  • The instructor was available to students either electronically or in person.
The response scale for each question should appear as follows:

1
2
3
4
5
Strongly
Disagree
Disagree
Neutral
Agree
Strongly Agree


All five questions should be considered mandatory. Any additional questions, specific to each institution, college, department, or faculty member may follow. Institutions should consider that long surveys typically lead to lower response rates and less accurate responses. 

Encouraging Student Participation

Student participation is crucial, as survey results are used in the formal evaluation of faculty. Students need to understand that their responses can also help instructors improve teaching styles and course materials. To incentivize student participation, institutions are encouraged to withhold a student’s access to grades until the student completes all course evaluations. Mandatory completion of course evaluations is not new – most UT health institutions already have such a system in place. Understanding that mandatory course evaluations will require a cultural shift on most campuses, the following suggestions may help to encourage acceptance and participation:
  • Encourage faculty to inform students of the importance of completing course evaluations. Students have indicated repeatedly that the faculty member’s emphasis on the importance of completing evaluations is the most compelling reason for compliance.
  • Encourage faculty members to note on the course syllabus that course evaluations are required.
  • Encourage faculty members to allow class time to complete the evaluations.  Make students aware of this time allocation in advance, so that they may bring phones, tablets, laptops, etc. in order to comply.  Reserve a computer room, even for a portion of the class time, to encourage compliance.
  • Consider applying an incentive at the course level.
  • Consider having the President, Provost, or VP for Student Affairs send a memo or email communication to all students towards the end of each semester informing them of the importance of course evaluations. Remind students that course evaluations enhance academic excellence, impact faculty’s professional development, and affect faculty’s overall evaluations at the institutional level.
  • Collaborate with campus student governments in promoting the importance of completing course evaluations. Student government promotional campaigns aid student understanding of the goals and the process of course evaluations.  Ultimately, this awareness helps to increase student participation and acceptance.
  • Accentuate completion as a positive:  Indicate that students that complete course evaluations by a certain date will have priority access to grades. One institution currently locks its online grading system two weeks before finals, allowing priority access one week after finals and releasing grades to all students one week later.  These timeframes can be adjusted based on the campus processes.

Electronic Course Evaluations

We strongly recommend institutions to utilize an online system for course evaluations. An online system is more economical and sustainable than a paper-based system, providing quicker results and offering greater ability to perform data analytics. It is often the case that the response rates to online course evaluations are lower than those of paper-based evaluations, but the suggestions listed above will encourage student participation and help to improve online response rates. The UT System administration will collect the responses to the five required survey questions and an online system will allow the sharing of data in a more efficient manner. The recommendations for encouraging student participation are particularly important if an online system is used to administer course evaluations.

Timeframe

Each campus is expected to incorporate these five questions into their student evaluations for the Fall 2013 semester.

Online Student Comments

Student comments are not required and should not be forwarded to UT System. A faculty member may want to gather comments from his or her class, but the institution should develop policies and procedures to oversee this feedback.  In developing these processes, institutions should be clear to students that providing in-class comments to an instructor is separate from the course evaluation.

Continuous Feedback

A survey of past recipients of The University of Texas System Regents Outstanding Teaching Awards revealed that systematic and frequent faculty-student feedback should be regarded as an integral component of every course. Students should receive feedback from professors and have many opportunities to provide feedback to faculty. Institutions are encouraged to use available continuous feedback mechanisms and MyEdu is developing the functionality to accommodate continuous feedback.


Appendix B

Basic Form Course Evaluation Currently in Use a UT Austin


 
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree
  1. The course was well organized
1
2
3
4
5
  1. The instructor communicated information effectively.
1
2
3
4
5
  1. The instructor showed interest in the progress of students.
1
2
3
4
5
  1. The tests/assignments were usually graded and returned promptly.
1
2
3
4
5
  1. The instructor made me feel free to ask questions, disagree, and express my ideas.
1
2
3
4
5
  1. At this point, I feel that this course will be (or has already been) of value to me.


A second section read as follows:

  1. Overall, this instructor was
Very unsatisfactory
Unsatisfactory
Satisfactory
Very Good
Excellent
  1. Overall, this course was
Very unsatisfactory
Unsatisfactory
Satisfactory
Very Good
Excellent
  1. In my opinion, the workload in this course was
Excessive
High
Average
Light
Insufficient
  1. My overall GPA to date at UT is
Less than 2.00
2.00 – 2.49
2.50 – 2.99
3.00 – 3.49
3.50 – 4.00
  1. My probable grade to date in this course is
A
B
C
D
F