SQA Statistics User Survey 2024

Official Statistics

 

Background

Scottish Qualifications Authority (SQA) became an Official Statistics producer in December 2019 and conducted its first statistics user consultation in May 2020.

Following this consultation, improvements to our publications were made which included improving the timeliness of publications, the publication of attainment for National 5, Higher and Advanced Higher by Scottish Index of Multiple Deprivation (SIMD) in the Equalities Monitoring Report, and a redesign of our statistical webpages.

In the years since the initial user consultation, further improvements to the production of our publications have taken place. These included the development of reproducible analytical pipelines for all our Official Statistics and Information publications, the production of our publications via R Markdown, a move towards the use of version control software, and further improvements to publication timings.

Given the extent of the changes since our last user consultation, SQA ran a further user consultation survey to understand how its statistical publications and information are being used and whether they are meeting users’ needs. The survey ran for six weeks, closing on 17th May 2024, and was advertised primarily via ScotStat.

Responses

105 valid responses were received, where at least one question was answered. Of these, 32 responses were full responses, with respondents answering all required questions in the survey and 73 were partial responses where at least one question was answered.

 

User demographics

The majority of respondents came from Education / Teaching, followed by Statistician / Data Analyst, and Research / Policy occupations.

Table 1. Survey respondent occupations
Role Responses
Education / Teaching 12
Statistician / Data Analyst 10
Research / Policy 8
Parent 3
Education Authority Director of Education (or equivalent) 2
Student 2
Other 6

Of those who responded Education/Teaching, a summary of roles (Table 2) and centre types (Table 3) is below:

Table 2. Education respondents’ roles
Occupation Responses
Teacher / Lecturer 9
Management / Leadership 2

 

Table 3. Education respondents’ centre types
Centre Type Responses
Education Authority School 6
Independent School 3
College 1
Special School 1

Publications

Respondents were asked which of our publications they used, for which they could select as many as applicable (Table 4).

Table 4. Publications used by survey respondents
Publication Responses
Attainment Statistics 39
Grade Boundaries 27
Results Services / Appeals 23
Component Marks 18
Equalities Monitoring Report 13
Assessment Arrangements 11
Estimates 11
Progression Tables 11
None of the above 10

 

Excluding responses where ‘None of the above’ were selected, the most selected publication used by respondents was Attainment Statistics with 25.5% of the total publications selected.

Follow up questions were asked for those selecting Attainment Statistics to understand the use of provisional August (Table 5) and final December post appeals (Table 6) publications.

Table 5. August Attainment Statistics publications used by respondents
Publication Type Responses Percentage (%)
Overall Summary 29 34.1
Education Authority Breakdown 21 24.7
Sex Breakdown 20 23.5
Centre Type Breakdown 15 17.6

Of those selecting August publication breakdowns, the Overall Summary was most selected by respondents. This contains the headline figures for attainment at a national level.

 

Table 6. December Attainment Statistics publications used by respondents
Publication Type Responses Percentage (%)
Overall Summary 26 36.6
Sex Breakdown 16 22.5
Education Authority Breakdown 16 22.5
Centre Type Breakdown 13 18.3

 

Similarly for those selecting December publication breakdowns, the Overall Summary was most selected by respondents. Education Authority (EA) and Sex breakdowns were selected at a similar level with Centre Type breakdowns selected less frequently.

Users selecting Progression Tables were asked a follow up question on their use of the publication to inform its transition to an Official Statistics in Development publication in 2025. Answers included using it for ‘comparative stats’ to compare schools versus a national picture; highlighting that ‘in its current format, it is satisfactory’; and is used to give ‘realistic expectations’ to pupils and their parents. Another comment mentioned a growing interest in ‘the best way to ascertain the path of a pupil from BGE to senior phase’.

Uses of our statistics

For those that selected one or more publication and responded to a follow up question on use, a summary of responses is below. Respondents could select one or more use per publication selected:

Table 7. Publication uses
Publication Monitoring Performance (%) Personal Interest (%) Policy Making (%) Research Analysis (%) Subject / Course Choice (%)
Attainment Statistics 25.9 18.5 18.5 31.5 5.6
Grade Boundaries 24.3 16.2 16.2 24.3 18.9
Results Services / Appeals 19.2 15.4 19.2 34.6 11.5
Component Marks 30.8 15.4 15.4 23.1 15.4
Equalities Monitoring Report 22.2 16.7 27.8 27.8 5.6
Assessment Arrangements 7.7 7.7 30.8 38.5 15.4
Estimates 20.0 20.0 0.0 50.0 10.0
Progression Tables 33.3 8.3 16.7 25.0 16.7

Satisfaction with publications

Of the 42 responses received on satisfaction with our publications, 71.4% were very satisfied or satisfied with our publications. Of the users that were very dissatisfied or dissatisfied, further information on how our publications could be improved to increase their satisfaction was not provided in the follow up questions.

Table 8. User satisfaction with publications
Satisfaction level Responses Percentage (%)
Very satisfied 9 21.4
Satisfied 21 50.0
Neutral 8 19.0
Dissatisfied 2 4.8
Very dissatisfied 2 4.8

Further comments on satisfaction

A free text response box allowed for further comments on improving satisfaction with our publications. 16 responses were received. Of these, some were not related to the production and use of our statistical publications (e.g. parents looking for candidate level information - which is available via Subject Access Request; or researchers looking for access to underlying source data - which can be made available via appropriate data sharing protocols).

Of the comments which were related to the development of our publications, there were several requests by users for:

  • ‘visualisations’, ‘infographics’, ‘charts’, and ‘dashboards’ to allow for easier selection of data and understanding trends

  • further stage breakdowns for NQ attainment at subject level

  • centre level statistics

More specific requests covered these areas:

  • longer time series to be available instead of having to use multiple files

  • collection of further data on protected characteristics

  • an interest in commentary around gender bias in subject choice

  • assessment mark information to be widened beyond averages to indicate the spread of marks by subject

  • interest in groups of subjects combined together to assess A-C rates for groups of subjects

  • improvements to education authority statistics such as highlighting difficulty using the attainment workbook due to each authority having a separate tab

Other comments flagged issues with difficulties finding the same data year on year due to data changing or webpages moving.

Better communication was also suggested to highlight the publications we produce to users who may not be aware of the different publication types.

Additional comments

Further additional comments received covered:

  • general satisfaction with the publications

  • the improvement to timeliness of publications - with all provisional attainment statistics being available at the same time on Results Day from August 2023

  • SCQF level by SIMD being produced for Attainment Statistics, as well as further stage breakdowns

  • charts or dashboards being preferable to the tables currently produced

Response to the user consultation

The Data and Analytics team welcomes the feedback received from users via the survey. In determining the direction of our publications, the individual suggestions received have been balanced against a review of our regular user requests to our mailbox () in order to ensure that decisions made are justified by overall user demand.

Visualisations, charts and dashboards: There were several requests for better visualisations and better communication of the statistics available from survey respondents. Data and Analytics are considering using the R Shiny package to meet this identified user need. R Shiny applications could allow for interaction with our data in a point and click visual manner and allow for ‘dashboard’ style information to be made available via a more pleasing user interface. We will consider the feasibility of using R Shiny in order to meet this user requirement. If the use of R Shiny proceeds then the development of applications will be phased in at a suitable pace to allow for improvements and feedback from users in line with the Code of Practice for Statistics, as well as to allow staff to develop their use of R Shiny. We will continue to produce our tables for those that prefer to use our data for their own analysis as well as to meet accessibility guidelines for users that cannot interact with a visual interface.

Stage: There were several requests for further statistics on stage data. We currently produce stage data at a national level, by sex, and by centre type in our attainment tables. Producing further stage breakdowns for entries and attainment for National Qualifications is possible. However, given the small entries for some of the National Qualifications, it would not be possible to give further breakdowns for every subject. A review of our regular external requests and FOI did not find a sufficient demand to prioritise the production of further stage statistics. We will keep this suggestion under consideration.

SCQF level by SIMD: A request was made for the SCQF level analysis contained within Attainment Statistics to be widened to include SIMD. We currently produce a breakdown of results for National 5, Higher and Advanced Higher by SIMD, as well as by equalities characteristics, in the Equalities Monitoring Report. Appeals also contains SIMD analysis. Separately, we produce SCQF level by qualification type and stage at a national level, by sex, and by centre type in our August and December Attainment Statistics tables. A further breakdown of these tables by SIMD would be possible at a national level. A review of our regular external requests and FOI did not find a sufficient demand to prioritise the production of further SCQF level statistics. We will keep this suggestion under consideration.

However, this has raised the possibility of amalgamating some of the equalities analysis contained with the Equalities Monitoring Report into the Attainment Statistics publication and tables. We will explore this further in order to meet our requirements as Official Statistics producers.

Centre level statistics: Requests for centre level information are considered on a case-by-case basis currently. This will continue in the interim while we weigh up the potential benefits versus risks of producing statistics at a more granular level.

Webpage layout: Previously, provisional attainment statistics produced in August were subsequently replaced with final December figures. Feedback received via the user survey on the removal of data and movement of information on our statistical webpages was acted upon in December 2024, with provisional August figures being retained alongside December. This decision is supported by the Code of Practice for Statistics requirement that statistics should ‘continue to be publicly available’.

Longer time series availability: The coverage of our current workbooks attempts to balance giving a reasonable period for comparison with accessibility considerations. Currently, the size of our workbooks already requires scrolling in order to view the entire time series of data contained. Increasing the time series included in the workbooks is possible but this would mean that workbooks would increase in size and would require users to scroll further than currently. The feasibility of producing R Shiny applications is being assessed, which may allow for longer custom time series to be selected by users.

Protected characteristics data: Further data on protected characteristics may be collected by Qualifications Scotland.

Further commentary: It is possible to produce an automated summary of changes in entry patterns and attainment for individual subjects by sex, where entry numbers are sufficient. Given the number of individual subjects, a summary of changes at subject level would be lengthy and SQA would not be able to provide additional commentary on any changes identified. A review of our regular external requests and FOI has not identified a demand for additional commentary on gender bias in subject choice.

Assessment mark information: A review of regular requests via our external mailbox and FOI identified a user need for information about the highest mark achieved in an assessment and the number of candidates achieving these marks. In 2023, the Component Marks / Assessment Marks publication was expanded to include the maximum mark achieved by candidates and a count of how many candidates achieved this mark. A review of our regular external requests and FOI did not find a sufficient demand to justify the production of further statistics on the spread of marks currently.

Grouping subjects: A review of our regular external requests did not find a sufficient demand for grouped subject information. The production of grouped subject statistics would also require input from users as to which groups of subjects should be combined together. Users interested in grouped subject statistics are welcome to contact our mailbox.

The general satisfaction from users and noting of improvements to date are welcomed by the Data and Analytics team.

Further plans

We will be monitoring the impact of efficiencies we have introduced into the production process with a view to improving the timeliness of delivery of Official Statistics publications currently released in December. This will be reflected in the publication schedule for 2026 when it is anticipated publications will be produced by Qualifications Scotland following the planned transition of SQA in late 2025.

‘Progression Statistics 2023 to 2024’ will be published on 28th January 2025 as an Official Statistics in Development publication. The summary will include extended commentary on areas including Maths and English, the five highest number of progression routes per level, and the subjects with the highest proportion of candidates progressing.

A review of regular requests via our external mailbox and FOI identified the need for a new publication covering Candidate Achievement. This will be published following Results Day in August in 2025 and is expected to include a breakdown of the number of entries per candidate, number of A to C grades per candidate, number of A grades per candidate, and number of upper A grades per candidate. Coverage will include National 5 to Advanced Higher and include breakdowns by sex and centre type.

A review of regular requests via our external mailbox and FOI also identified the need for information to be published on a more timely basis on Higher National (HN) and Vocational Qualifications (VQ). To this end, we are exploring developing and publishing HNVQ attainment statistics on a more regular basis.

We are investigating routes to better meet the needs of our users through innovative and interactive visualisations. As part of this work, we are looking at the feasibility of using R Shiny with the aim of developing and releasing a R Shiny app for testing.

Lastly, as Official Statistics producers, SQA follow the Code of Practice for Statistics. This is currently under review and we will further consider our publications, communication and further user engagement in line with the refreshed edition (3.0).