This week, I, along with Melissa Clinedinst, NACAC’s assistant director for research, am attending the Association for Institutional Research annual conference in Toronto.
Has introducing counselor assessments of colleges and universities to the U.S. News & World Report (USNWR) "Best Colleges" created more complications in the rankings? According to a NACAC survey released last week
, college admission counseling professionals view the peer assessment portion as one of the weakest elements of the USNWR ranking methodology. In 2011, for the first time, USNWR began including a counselor assessment
in its peer assessment measure, which had previously included only higher education leaders and administrators. Counselors from 1,787 public schools in 43 states were invited to rate colleges “to take into account the insights they use to direct students to particular colleges in addition to their knowledge about these schools in general.”
The result, at least for some colleges, was a subtle shift in the peer assessment scores. Jason Sullivan, Senior Statistical Information Specialist at Ohio State University
, was asked by campus leaders to explore the variance between counselor assessments and higher education peer assessments, as counselors rated OSU less favorably than expected.
reveal patterns in the data between higher education peer assessments (presidents, provosts, admission deans) and school counselor assessments. In general, according to Sullivan, the two sets of assessments were highly correlated. However, there were differences between counselor and postsecondary peer assessments based on institutional control. Higher education peers tended to rate public universities higher than counselors, while counselors tended to rate private universities higher than the higher education peer group. What accounts for the difference?
Sullivan’s hypothesis is that the high schools selected, at least in Ohio, tended to yield a restricted range. One important set of schools that the USNWR selection criteria ended up omitting in Ohio were large, public high schools that send large numbers of students to state universities, but also have significant numbers of underperforming students in some subgroups. Such characteristics would not qualify a school for the U.S. News Best High Schools ranking
, which requires student performance across all subgroups of students to meet or exceed the rankings’ benchmarks. Sullivan believes that selection ‘bias’ may have altered the population of counselors assessing Ohio universities, which, in turn, affected the assessment scores of institutions in the state.
While the purpose of the session was to illustrate the potential power of global information systems (GIS) to analyze data, it also highlighted important questions about the USNWR rankings. In this case, how does the selection of counselors affect the peer assessments of postsecondary institutions? Some attendees wondered whether counselors are biased toward private institutions, or whether private institutions pay more attention to school counselors than public institutions. Others wondered whether higher education peers rated large, public research institutions more highly than smaller, private liberal arts colleges, due to a bias in favor of the research function at postsecondary institutions.
No matter the answer, the subjectivity of USNWR rankings is an important theme of both this presentation and the NACAC survey results. Counselors, students, and families are well-advised to understand that rankings—no matter the source—are not objective measures of institutional quality, and are subject to influence by a number of factors, whether intentional or not.