Graduation rate is one of the most common metrics across PBF programs and general PSE accountability frameworks.
A graduation rate metric was part of the reporting associated with SMA2 and SMA1.
From a student-centred perspective, it makes intuitive sense that as graduation concludes a student’s academic experience it would be a natural metric to select.
It is measured by all institutions in some form, and in the United States it was codified in federal law in the 1990 Student Right-to-Know Act. The act requires degree-granting postsecondary institutions to publish the percent of students that graduate within 150 percent of the normal time for completion of or graduation from the program, the student has completed or graduated from the program, or enrolled in any program of an eligible institution for which the prior program provides substantial preparation. (Student Right-to-Know and Campus Security Act, 1990, s. 103[a])
Organizations such as the U.S. National Student Clearinghouse Research Center, and the Consortium for Student Retention Data Exchange track both the normal four-year graduation rate and the six-year 150% graduation rate.
SMA3 defines graduation rate as measured in the University Statistical Enrolment Report (USER):
Proportion of all new, full-time, year one undergraduate university students (domestic and international) of bachelors (first-entry), or first professional (second-entry) degree programs who commenced their study in a given fall term and graduated from the same institution within 7 years.
(MTCU, 2019b as cited in University of Windsor, 2019, p. 24)
The specifics in the SMA3 measure both extend the duration for students to complete an undergraduate degree from the typical six years (150%) to seven years (175%) and targets students who started at the same institution, excluding transfer students. The Consortium for Student Retention Data Exchange SMA2 and SMA1 graduation metrics also excluded transfer students.
Familiar to PBF
Some form of graduation rate is present in most U.S. PBF programs, and the general availability of graduation rate data has made it a frequent point of comparison.
Rutherford and Rabovsky (2014) examined 568 institutions across all 50 states from 1993 to 2010 and found that graduation rates slightly declined (0.16% per year) in PBF states.
Dougherty and Natow (2019) have identified that higher education institutions subject to pressures to improve graduation rates may consider reducing academic rigour and inflating grades and reducing the number of difficult courses required to graduate.
It’s a better measure of what you did before PSE
There is evidence to suggest that graduation rate is not an output of a university that is predominantly affected by the student’s experience at the university, rather it is predicted by student backgrounds and experiences that occur before a student even applies to university (Zhang, 2009).
Umbricht et al. (2017) have argued that when university administrators draw this conclusion it can lead to a shift in attention from improving curriculum, student supports and student-to-faculty ratios, to a focus on university admissions.
Students with favourable pre-admission socioeconomic conditions have proven to possess an academic advantage that improves their likelihood of graduating, improving related PBF metric results. This realization creates an admissions selection incentive (Dougherty, 2016; Kelchen & Stedrak, 2016; Umbricht et al., 2017). This can not only lead to an admissions bias towards students that are more likely to graduate for nonacademic reasons it can lead to decreasing the number of students accepted, allowing an institution to only accept the “cream of the crop” and not the risk of weaker students (Dougherty et al., 2014; Umbricht et al., 2017).
Both biases are counter to access goals that might be focused on underrepresented populations or universality or growth goals.
A 2020 systemic analysis by Ortagus et al. synthesized 20 years of reports on PBF programs in the United States and concluded that PBF led to either no impact or a modest increase in retention and graduation rates, but also led to access and equity issues for disadvantaged students, and incentivized institutions to engage in these types of games.
Anti-access
Many PBF programs include metrics related to students receiving Pell Grants, or other underrepresented groups such as first-generation students or underrepresented racial groups, or other at-risk groups to improve access goals and mitigate this selectivity. (Boelscher & Snyder, 2019; Hagood, 2019; Kelchen, 2018). Indiana, North Carolina, and New Mexico’s PBF programs have included an At-Risk Degree Completion metric. which also includes transfer students.
This has led some states to move to graduation numbers because graduation rates are more easily gamed through strategies such as shrinking acceptance numbers (Dougherty & Natow, 2019). Related metrics such as retention rates or completion of milestone courses offer a more immediate assessment of student progression.
Implementation of the graduation rate
Across the five years of SMA3, the average weight given to the graduation rate metric is 14% in the first year, 9% in the subsequent years.
Western University has consistently weighted graduation rate the highest in the province, 30% in the first year and 25% in the subsequent years. By consistently weighting this metric as the maximum, Western is indicating that it is an area of stability or potential growth. Western’s 2018–19 graduation rate was 84.28% and its 2020–21 allowable target is 83.55%.
Brock University, Lakehead University, Nipissing University, OCAD University, Trent University, Ontario Tech University, Wilfred Laurier, and York University have consistently weighed the graduation rate metric at the lowest rating available. Université de Hearst also consistently weighted the graduation rate metric the lowest, and has the lowest graduation rate in the province, reporting an average of 46.59% in the three years provided in the historic data. The next lowest is Algoma University, at an average over the three years of 55.8%, and the highest is Queen’s University at an average of 89.07%.
Université de Hearst’s as an example of how targets are calculated
The Université de Hearst’s graduation rate provides an interesting example of how targets are calculated and the impact of variances.
The Université de Hearst’s has the lowest historical graduate rate figures provided, and the most variance: 35.71% in 2016–17, 50.00% in 2017–18, and 54.05% in 2018–19, averaging 46.59%. The Université de Hearst’s allowable performance target for graduation rate is 38.46%, much lower than its most recent year and three-year average.
The absolute change between the years, 14.29% and 4.05% means that the average percentage of change is 24.06%, the highest average percentage of change of any university.
This average percentage of change is subtracted from the three-year average plus smallest change target 50.64% (that is, 46.59% average plus 2017–18 to 2018–19’s 4.05%) to bring the lowest target to an even lower allowable performance target of 38.46%. Almost as low as the Université de Hearst’s 2016–17’s 38.46% graduation rate and much lower than projections that could be extrapolated from those three data points.
Carleton University, Lakehead University, Queen’s University, University of Guelph, University of Ottawa, University of Toronto, University of Waterloo, Western University, and Wilfrid Laurier University all had such little variation they were assigned the minimum band of tolerance 1%.
Table: Graduation rate, historic results, targets, tolerance
The first four columns are included in each SMA3, and the columns noted as calculated indicate that they are an average of the provided historical data or the absolute differences between provided data. The target is calculated through SMA3’s formula of the three most recent data points averaged, and the smallest of the absolute variations between Year 1 & Year 2, and Year 2 & Year 3, is added to the average. The band of tolerance is an average of the recent three years’ percentage change.
Graduation Rate as Included in Each SMA3 With Each Element of Allowable Performance Target CalculationThese are the elements that calculate the allowable performance target included for each metric in each SMA3, calculated as target – (target * band of tolerance).
Graduation Rate as Included in Each SMA3 With Each Element of Allowable Performance Target CalculationBoelscher, Scott, and Martha Snyder. “Fiscal Year 2019 State Status & Typology Update.” HCM Strategies, 2019. http://hcmstrategists.com/promising-policy/wp-content/uploads/2019/01/HCM_2019_DBO_Final.pdf.
Bradley, Bill. “S.580 – 101st Congress (1989-1990): Student Right-to-Know and Campus Security Act.” Webpage, August 11, 1990. 1989/1990. https://www.congress.gov/bill/101st-congress/senate-bill/580.
Dougherty, Kevin J., Sosanya M Jones, Hana Lahr, Rebecca S. Natow, Lara Pheatt, and Vikash Reddy. “Performance Funding for Higher Education: Forms, Origins, Impacts, and Futures.” The Annals of the American Academy of Political and Social Science 655 (2014): 163–84. https://www.jstor.org/stable/24541755.
Dougherty, Kevin J., and Rebecca S. Natow. “Performance-Based Funding for Higher Education: How Well Does Neoliberal Theory Capture Neoliberal Practice?” Higher Education 80, no. 3 (December 23, 2019): 457–78. https://doi.org/10.1007/s10734-019-00491-4.
Dougherty, Kevin James. Performance Funding for Higher Education. Book Collections on Project MUSE. Baltimore, Maryland: Johns Hopkins University Press, 2016.
Hagood, Lori Prince. “The Financial Benefits and Burdens of Performance Funding in Higher Education.” Educational Evaluation and Policy Analysis, March 25, 2019. https://doi.org/10.3102/0162373719837318.
Kelchen, Robert. “Do Performance-Based Funding Policies Affect Underrepresented Student Enrollment?” The Journal of Higher Education (Columbus) 89, no. 5 (2018): 702–27. https://doi.org/10.1080/00221546.2018.1434282.
Kelchen, Robert, and Luke J. Stedrak. “Does Performance-Based Funding Affect Colleges’ Financial Priorities?” Journal of Education Finance 41, no. 3 (2016): 302–21. https://doi.org/10.1353/jef.2016.0006.
Ontario Ministry of Training, Colleges and Universities. “Ontario’s Postsecondary Education System Performance/Outcomes Based Funding – Technical Manual.” Ontario Ministry of Training, Colleges and Universities, September 2019. http://www.uwindsor.ca/strategic-mandate-agreement/sites/uwindsor.ca.strategic-mandate-agreement/files/performance_outcomes-based_funding_technical_manual_-_v1.0_-_final_september_419_en.pdf.
Rutherford, Amanda, and Thomas Rabovsky. “Evaluating Impacts of Performance Funding Policies on Student Outcomes in Higher Education.” The Annals of the American Academy of Political and Social Science 655, no. 1 (2014): 185–208. https://doi.org/10.1177/0002716214541048.
Umbricht, Mark R., Frank Fernandez, and Justin C. Ortagus. “An Examination of the (Un)Intended Consequences of Performance Funding in Higher Education.” Educational Policy 31, no. 5 (July 1, 2017): 643–73. https://doi.org/10.1177/0895904815614398.
Zhang, Liang. “Does State Funding Affect Graduation Rates at Public Four-Year Colleges and Universities?” Educational Policy 23, no. 5 (2009): 714–31. https://doi.org/10.1177/0895904808321270.