On January 21, 2021, I blogged about the SMA3 document’s Strength and Focus sections. My initial interest was the contrast between SMA2 and SMA3’s strength and focus sections. I’ve included those initial charts again.
In the PBF program categorization that Adam (2020) provided, a distinction is made between three primary types of PBF programs:
- Results: Output or outcome-based funding formula or performance formula;
- Performance set-asides or reservation; and
- Performance contracts or agreements (also compacts).
Ontario’s SMA3 program uniquely blends all three, and the institutional strength/focus metric is the biggest departure from the mostly results-based, output or outcome-based funding metrics (with some elements of a set-aside model for underperformance), and a return to the contract-basis of the performance component of SMA2.
Only 4% of funding was performance-based in SMA2, despite there being 23 system-wide metrics and 244 institutional targets and metrics created system-wide, of which 228 were unique to a single institution.
Where SMA2 asked institutions to nominate metrics and measures, SMA3 asks institutions to nominate a subject area based on institutional strength and focus but requires a system-wide measure, the “proportion of enrolment (FTEs, domestic and international) in an institution’s program area(s) of strength [as reported in the University Statistical Enrolment Report (USER)]” (MTCU, 2019, p. 24).
Weingarten and Deller’s (2010) roadmap’s principles of differentiation were reflected in the proliferation of institutionally specific metrics in SMA2, but not in their linking to funding decisions or any other form of impactful PSE system-level actions. Many metrics contained within the 21 university SMA2s could not be measured by any party other than the institution itself (measures lacked the actual values of the baselines that the targets were relative to or were wholly internal measures) or have outcomes compared to peer institutions.
As a result of these limitations, the accountability and utility university SMA2 metrics provided were comparably low. In contrast, SMA3’s single institutional strength/focus metric offers a clear, teaching-related differentiation that in the final years of SMA3 can represent as much as 25% of the 60% of performance-based funds an institution receives (as much as 15% overall).
Different measures replaced with a measure of differentiation
This metric begins to answer the question how differentiated is each Ontario university, according to each university? Because each institution nominates its own area of strength, while the USER data source and proportion of enrolment measure are consistent, this metric begins to provide a common differentiation measure in SMA3, where SMA2 mostly provided different measures.
Two potential flaws
- Ever growing targets: The target setting methodology’s assumption of growth could turn a genuine strength into a liability. With all positive and negative variations being averaged with their absolute value to set a growth target, all metrics have the potential to have targets grow during a period of decline, potentially creating a distorting negative feedback loop.
- Statistical strength: The ability for institutions to game the metric by not nominating a program or an area of enrolment that represents an academic, regional, proportional, or otherwise intrinsic institutional strength, but instead a statistical strength in an area where targets can easily be met.
Implementation of the Institutional strength/focus Metric
Across the five years of SMA3, the average weight given by universities to the institutional strength/focus metric is 21% in the first year, 16% in the second, and 15% in the subsequent years. The most often allocated weight by universities in the first year was 20%, 15% in the second year, and again 20% in the subsequent years.
Nipissing University weights this metric higher than other universities, at the maximum 35% and 30% in the first year two years but drops to 10% as all metrics become available in the third year. McMaster University and the University of Waterloo consistently weight this metric at the lowest value available, the Université de Hearst starts at year two.
Within the top 10 universities by the percentage of students enrolled in the area strength as calculated as a 2020-21 allowable performance target, the most often described element of institutional strength is “health,” including Brock University, Ryerson University, McMaster University, and York University. Of that top 10 in target percentages, only York included “engineering.”
When all institutions’ strengths are reviewed thematically, engineering jumps to number one, with six institutions including it.
The top 10 universities all set a target above 40%. Considering the commitment to differentiation, regardless of theme, those top 10 intuitions have demonstrated operationalization of differentiation.
Big net or little niche?
Themes of health, arts (University of Toronto and Wilfred Laurier University), and design (OCAD University and Ryerson University) are broad themes that many programs can fit within (see Appendix B, Figure B1).
The University of Ottawa identified its metric is targeted at only 9.21%, the lowest 2020–21 target, but it is also in the particularly narrow focused area of “Analytics and Artificial Intelligence” (MCU & University of Ottawa, 2020, para. 30).
Adam, Edmund. “‘Governments Base Performance-Based Funding on Global Rankings Indicators’: A Global Trend in Higher Education Finance or a Global Rankings Literature Fiction? A Comparative Analysis of Four Performance-Based Funding Programs.” International Journal of Educational Development 76 (2020): 102197-. https://doi.org/10.1016/j.ijedudev.2020.102197.
Ontario Ministry of Training, Colleges and Universities. “Ontario’s Postsecondary Education System Performance/Outcomes Based Funding – Technical Manual.” Ontario Ministry of Training, Colleges and Universities, September 2019. http://www.uwindsor.ca/strategic-mandate-agreement/sites/uwindsor.ca.strategic-mandate-agreement/files/performance_outcomes-based_funding_technical_manual_-_v1.0_-_final_september_419_en.pdf.
University of Ottawa, and MTCU. “2017-20 Strategic Mandate Agreement: University of Ottawa | Ontario.Ca,” 2017. https://www.ontario.ca/page/2017-20-strategic-mandate-agreement-university-ottawa.
Weingarten, Harvey P, and Fiona Deller. “The Benefits of Greater Differentiation of Ontario’s University Sector,” October 26, 2010. https://heqco.ca/pub/the-benefits-of-greater-differentiation-of-ontarios-university-sector/.