Managing the Register

Following the successful application for inclusion of three new agencies, the Register grew to a total of 48 quality assurance agencies in 2019. The map of the EHEA Key Commitment thus also showed an increase of three new countries where higher education institutions are regularly reviewed by registered QA agencies: Hungary, Georgia and Cyprus. In total, by the end of 2019, 28 EHEA countries fulfilled the EHEA Key Commitment that all higher education institutions are subject to regular external quality assurance in line with the ESG, through relying on EQAR-registered agencies.

With the applications of new agencies for EQAR registration and the opening of legal framework allowing the use of a suitable EQAR-registered agency, we note progress in countries fulfilling the commitment that higher education institutions are externally quality assured in line with the ESG.

Green: A fully functioning quality assurance system is in operation nationwide, in which all higher education institutions are subject to regular external quality assurance by an agency that has successfully demonstrated compliance with the Standards and Guidelines for Quality Assurance in the EHEA (ESG) through registration on EQAR

Yellow: A fully functioning quality assurance system is in operation nationwide but only some higher education institutions are subject to regular external quality assurance by an agency that has successfully demonstrated compliance with the ESG through registration on EQAR

Orange: A quality assurance system is in operation nationwide, but has not (yet) been fully aligned to the ESG

Red: No quality assurance system is in operation

A look at the external QA activities carried out or ‘on offer’ by the 48 EQAR-registered quality assurance agencies indicates that 25% of QA agencies carry out between 1 to 3 forms of evaluations, while 30% of registered QA agencies conduct 9 to 15 types of reviews within the scope of the ESG. There are over 300 external QA activities on offer by agencies registered in EQAR. Overall, this picture suggests that the evaluation instruments of quality assurance agencies, and by extension the quality assurance systems where they operate, are more complex and diversified than ever.

In the monitoring of registered agencies’ activities, EQAR has been also collecting data on the number of external QA activities carried out within and outside European higher education systems. This has allowed to identify a number of trends and changes throughout the years and to map the spread of QA activities.

Trends and changes in external QA activities

In 2019, a total of 11.312 external QA activities were conducted by EQAR-registered agencies at programme, institutional or joint programme level. Compared to the previous year we observe an increase in the volume of external QA activities by 22%. The increase can be attributed to the increase in programme level reviews which spiked by 30 % while institutional level reviews has dropped by 20%. The share of programme/institutional level external QA is however comparable to the figures collected for 2017 from registered agencies. The disparities in the frequency of reviews can be largely explained by the change in the review cycle for each agency, the introduction and increase in certain activities and due to changes in external QA within certain higher education systems.

While yearly changes in the volume of external QA activities of agencies are common, a few agencies have recorded a continuous decrease for the past four years (A3ES, ACSUG, AKKORK and QAA) or a continuous increase (AQ Austria, NVAO, and QQI).

Some higher education systems have also recorded a significant increase in their programme level reviews in 2019 compared to the previous year (of over 40 %) in Spain, France, the Netherlands, Romania, Belgium, Austria and Russia, while some higher education systems recorded a significant decrease in their institutional level reviews in 2019 compared to the previous year i.e. over 40% drop in the institutional audits or evaluations in United Kingdom, Kazakhstan, Portugal, Germany and Slovenia.

Spread of external QA activities

The 48 quality assurance agencies carried out reviews in 68 higher education systems covering most of the higher education systems within EHEA (at least one review in 34 countries). The spread of activity of EQAR-registered QA agencies in 2019 was similar to the previous years.

The analysis also revealed that about half (51%) of the total number of review carried out in 2019 took place within only two countries, that have large higher education systems i.e. France and Spain. These activities were carried out by a quarter of the registered agencies.

See our interactive visual tool for more information about the external QA activities within your country by agency and type (from 2014 to 2019).

Annex: Data on Activities of EQAR Registered-Agencies’ Activities within EHEA in 2019

In 2019, the Register Committee held three meetings (one in Bucharest and two in Brussels) and considered 21 applications for inclusion and renewal of registration. The Committtee took final decisions on 17 applications. The remaining four applications were deferred pending additional representation on the grounds for possible rejection; final decisions are expected in 2020.

The majority of applications was however successful with the Committee concluding that the agencies substantially complied with the ESG. As a result, three QA agencies were newly added to the Register and 11 QA agencies had their registration renewed on EQAR; one agency was re-admitted to the Register after a successful focused review. A list of all final decisions taken in 2019 is available here.

Compared to previous years, EQAR recorded a significant increase in the applications for renewal of registration, i.e. twice as many applications compared to 2018. The expiry of agencies’ five year registration on EQAR coincided with the large number of reviews that had been carried before the adoption of the revised ESG, in 2015. As of 2020, all EQAR-registered agencies will have been reviewed against the ESG 2015.

The number of initial applications followed a similar trend as in the previous years, with ca. 5-6 new applications each year. This shows a continuous interest of QA agencies to be registered on EQAR. New applications are expected for the next years as well, as the Register Committee confirmed the eligibility for four initial applications in 2019. These agencies’ reviews will be concluded in 2020.

The full decisions can be consulted here.

A closer look into the 17 completed applications of 2019 reveals that one in three agencies was only partially meeting the requirements of standards ESG 2.4 (Peer-review experts) and ESG 2.7 (Complaints and appeals). Other standards were the Register Committe found frequent deficiencies (in 5 of the 17 cases) were ESG 2.6 (Reporting), ESG 3.1 (Activities, policy and processes for quality assurance), ESG 3.3 (Independence) and ESG 3.4 (Thematic Analysis). For the remaining standards, with a few exceptions of partial compliance (ESG 2.3, ESG 2.5, ESG 3.5 and ESG 3.6), the Register Committee found that most agencies complied substantially.

A recurrent issue in complying with standard 2.4 Peer-review experts is the lack of student involvement in some forms of external QA activities, most often ex-ante procedures. In other cases, while students were involved in all (recent) reviews, an exception was made at the request of the reviewed institution. Due to the lack of consistent processes in the involvement of students in review panels, the Register Committee could not conclude on substantial compliance with the standard.

In analysing complience with standard 2.6 Reporting the Register Committee found that a few agencies did not meet the requirement of the standard as they published only summary reports. While such summaries included the decision, the main findings and recommendations of the review, the full detailed evaluation results were not available to the public at large. Due to a national requirement of compliance with European Unions’ General Data Protection Regulation, one QA agency was expected to adapt its internal reporting templates and procedures for its reports, which delayed the publication of its reports.

With a view to the requirements of 2.7 Complaints and appeals, the Committee noted that in some cases appeals were only possible in those review methods that resulted in a formal judgement, but not for methods that results in a report alone. In other cases, the agency did not give the possibility to put forward substantiated concerns about the evaluation process or conduct of review experts, unless the report ended in a negative judgment. Further concers were related to the the composition of the agencies’ appeals’ body and the full transparency in the agency’s handling of appeals.

The deficiencies in meeting the requirements of standard 3.1 Activities, policy and processes for quality assurance were the lack of student involvement (as stakeholders) in the governance and work of agencies, and the lack of clarity between the different types of the evaluations. Agencies are expected to ensure that the communication of their activities is fully clear to all stakeholders and thus avoid any misrepresentation in the separation between their external QA and other fields of work (according to Annex 5 to the EQAR Policy on the Use and Interpretation of the ESG).

While the independence (ESG 3.3) of decision-making was ensured by all agencies, the Register Committee noted that concerns remained as some agencies still lacked sufficient operational and organisational independence from the ministry or the founding body, which were responsible for, e.g., appointing or dismissing the agency’s governing body, overseening agency’s staff recruitment and finances, ensuring the general infrastructure.

The main concern with regard to ESG 3.4 Thematic analysis was the modest progress of agencies in developing and regularly publishing such analyses using the general findings of their external quality assurance activities. While some agencies produce a plethora of publications and reports, there were cases where none or very few of those were actually analysing the findings of the agencies’ quality assurance activity.

The Register Committee considered 19 substantive change reports in 2019. Nine of those were considered collectively, as a clustered analysis, since they all related to the new legal framework in Germany.

The change in the accreditation system in Germany meant that the agencies carrying out activities in the country no longer would take the accreditation decisions themselves, but were expected to now prepare the assessment report on the basis of which the German Accreditation Council (GAC) would take the decision. The reports showed that the changes in the way agencies  carried out these assessments remained largely similar to their earlier practice. A few agencies had, however, adapted  the structure of their decision-making or appeals committees. The Register Committee took note of the information provided in each case.

The remaining change reports largely addressed aspects concerning the introduction of new external QA activities (for nine QA agencies) and changes to the organisational structure of the agency (for six QA agencies). The change reports illustrate that many agencies have been adapting and expanding their type of external quality assurance activities, responding to changes of legal frameworks in countries where they operate but also introducing innovative types of reviews as a way of reducing the fatigue with a certain type of activity.

In 2019 EQAR did not receive any third-party complaints on the work of registered agencies.

EQAR maintained an active dialogue with ENQA, as the organisation that coordinates the vast majority of external reviews used for applications for inclusion on the Register.

In addition to regular meetings and dialogue between the staff in charge, EQAR and ENQA held their annual official meeting in September 2019. The meeting allowed to discuss current issues in relation to agency reviews and other matters of shared concern.

ENQA had further launched an external review of its agency reviews in 2018, with a site visit by an external panel in 2019. EQAR was interviewed during the site visit and shared its reflections on the use by the Register Committee of ENQA-coordinated reviews.

In those cases where agencies choose to work with another review coordinator, EQAR aims to have a briefing conversation with the review coordinator before the process, in order to ensure that the review corresponds to EQAR requirements and expectations, and to avoid common pitfalls.