Scratch beneath the surface of college rankings

If an athletic team dropped in rankings from number 2 to 18 in just a year, would alarm bells go off? This unprecedented decline is the case of Columbia University in New York City. It fell to 18 in the U.S. News Best College Rankings 2023 after being second last year and claiming a top spot for years.

A Columbia math professor, Michael Thaddeus, asserted the college was self-reporting false data on class size, instruction spending and full-time faculty. Shortly thereafter, Provost Mary C. Boyce acknowledged that Columbia relied on “outdated and/or incorrect methodologies” and consequently overstated class size and faculty degrees.

Thaddeus explained that submitting false data helped to increase Columbia’s rankings. His report published a detailed analysis exposing “inaccurate, dubious or highly misleading” statistics submitted to U.S. News. Thaddeus questioned how Columbia was able to catapult its ranking compared to other top universities: ranked 18 in 1988, top five in 2011 and second in 2021.

In response to these false data revelations, U.S. News initially designated Columbia as being unranked; however, later it demoted Columbia to be ranked 18 in its published magazine and website.

Although U.S. News is not the only college ranking publication, it is one of the most popular. It focuses on schools’ priorities that include more subjective measurement criteria. For example, it is the only ranking survey that separates national universities and liberal arts colleges, each with their own ranking list. This allows liberal arts colleges not to be directly compared against universities.

Liberal arts colleges are usually smaller than national universities. Liberal arts colleges mostly focus exclusively on the undergraduate experience. Universities typically have graduate schools whose professors, resources and research can trickle into the undergraduate programs.

Columbia’s disputed data factored into the Undergraduate Reputation criteria, which represents 20 percent of the U.S. News’ overall score. This information is self-reported by each school’s president, provost and dean of admissions. This reputation measurement could be biased on how each school defines each attribute. For example, class size could average in lab sessions, which typically have fewer students.

Surprisingly, U.S. News weights Graduate Indebtedness at 5 percent of the overall score. With the attention on escalating student loans and debt, assigning lower weights to these factors lessens their impact and enables colleges to sweep under the rug high student debt. The Wall Street Journal reports that 40 million people have federal student loan debt.

Another rankings list to analyze is the Forbes America’s Top Colleges for 2022. It combines universities and colleges into one list with its own unique measurement criteria. Columbia University is ranked 5, which is consistent with last year’s 2021 ranking. This ranking analysis was published online on August 30, 2022, before the Columbia dispute erupted. Forbes published Columbia’s student-to-teacher ratio at 7 to 1, which was doubted by Thaddeus.

The Forbes data primarily focuses on student outcomes. It collects information principally from two federal databases: the Integrated Postsecondary Education Data System and College Scorecard. Additional data is compiled from: PayScale; Third Way; the National Center for Science and Engineering Statistics; and Forbes’ own lists.

Alumni Salary represents 20 percent of the Forbes calculation. Interestingly, U.S. News does not include salary as a measurement criterion.

Forbes computes 15 percent each to: Debt; Graduation Rate; Forbes American Leaders List; and Return-on-Investment. Finally, Retention Rate and Academic Success each account for 10 percent

Families should have a frank conversation to identify the priorities for their student’s college experience. In some cases, the student’s goals may be different from their parents. For example, if a student wants to be recruited for a varsity sport, liberal arts colleges may be a better fit to compete on a team. Liberal arts colleges are typically Division-2 or Division-3, compared to many national universities as Division-1.

Not all ranking criteria are created equal. Each publication has different variables with distinct focuses. Each list has specific definitions and weights: it can be like comparing apples to oranges. Especially with self-reported information, it is important to understand the data components.

College planning can be an emotional journey, where rankings might cloud the process. Various ranking publications provide numerical guidance to compare and prioritize. Colleges can include specific rankings in their marketing materials to reinforce the school’s positioning and goals. Scratching beneath the surface of ranking criteria can uncover structural biases that are not obvious.

(Margo Bartsch founded College Essay Coach, a full-service college admission business, and has been an adjunct professor in business at Champlain College and at Middlebury College.)