It may sounds strange, but a new study states that better teachers possibly get worse student evaluations. In  this new paper a team led by Michela Braga of Bocconi University in Italy evaluated the content of the students’ evaluations by contrasting them with objective measures of teacher effectiveness.

The highlights of the study are:

• The researchers contrasted measures of teacher effectiveness with students’ evaluations.Teacher effectiveness is estimated using random allocation of students to teachers.Teacher effectiveness is negatively correlated with students’ evaluations. To put it short: A 1-standard-deviation increase in university teachers’ effectiveness in boosting student performance reduces the students’ evaluations of their professors’ teaching quality by about half of a standard deviation.
• The least performing (or able) students are more likely to dislike (good) teachers (who ask more of them).
• The researchers also found that student evaluations improve… when there is fog and as the weather gets warmer, and they deteriorate on rainy days.

The conclusion is clear:

“These empirical findings challenge the idea that students observe the ability of the teacher in the classroom and report it to the administration when asked in the questionnaire. A more appropriate interpretation is based on the view that good teachers are those who require their students to exert effort; students dislike it, especially the least able ones, and their evaluations reflect the utility they enjoyed from the course.

Overall, our results cast serious doubts on the validity of students’ evaluations of professors as measures of teaching quality or effort. At the same time, the strong effects of teaching quality on students’ outcomes suggest that improving the quantity or the quality of professors’ inputs in the education production function can lead to large gains.”

Abstract of the study:

This paper contrasts measures of teacher effectiveness with the students’ evaluations for the same teachers using administrative data from Bocconi University. The effectiveness measures are estimated by comparing the performance in follow-on coursework of students who are randomly assigned to teachers. We find that teacher quality matters substantially and that our measure of effectiveness is negatively correlated with the students’ evaluations of professors. A simple theory rationalizes this result under the assumption that students evaluate professors based on their realized utility, an assumption that is supported by additional evidence that the evaluations respond to meteorological conditions.