SQA Statistics User Survey 2024
Official Statistics
Contents
Background
Scottish Qualifications Authority (SQA) became an Official Statistics producer in December 2019 and conducted its first statistics user consultation in May 2020.
Following this consultation, improvements to our publications were made which included improving the timeliness of publications, the publication of attainment for National 5, Higher and Advanced Higher by Scottish Index of Multiple Deprivation (SIMD) in the Equalities Monitoring Report, and a redesign of our statistical webpages.
In the years since the initial user consultation, further improvements to the production of our publications have taken place. These included the development of reproducible analytical pipelines for all our Official Statistics and Information publications, the production of our publications via R Markdown, a move towards the use of version control software, and further improvements to publication timings.
Given the extent of the changes since our last user consultation, SQA ran a further user consultation survey to understand how its statistical publications and information are being used and whether they are meeting users’ needs. The survey ran for six weeks, closing on 17th May 2024, and was advertised primarily via ScotStat.
Responses
105 valid responses were received, where at least one question was answered. Of these, 32 responses were full responses, with respondents answering all required questions in the survey and 73 were partial responses where at least one question was answered.
User demographics
The majority of respondents came from Education / Teaching, followed by Statistician / Data Analyst, and Research / Policy occupations.
Role | Responses |
---|---|
Education / Teaching | 12 |
Statistician / Data Analyst | 10 |
Research / Policy | 8 |
Parent | 3 |
Education Authority Director of Education (or equivalent) | 2 |
Student | 2 |
Other | 6 |
Of those who responded Education/Teaching, a summary of roles (Table 2) and centre types (Table 3) is below:
Occupation | Responses |
---|---|
Teacher / Lecturer | 9 |
Management / Leadership | 2 |
Centre Type | Responses |
---|---|
Education Authority School | 6 |
Independent School | 3 |
College | 1 |
Special School | 1 |
Publications
Respondents were asked which of our publications they used, for which they could select as many as applicable (Table 4).
Publication | Responses |
---|---|
Attainment Statistics | 39 |
Grade Boundaries | 27 |
Results Services / Appeals | 23 |
Component Marks | 18 |
Equalities Monitoring Report | 13 |
Assessment Arrangements | 11 |
Estimates | 11 |
Progression Tables | 11 |
None of the above | 10 |
Excluding responses where ‘None of the above’ were selected, the most selected publication used by respondents was Attainment Statistics with 25.5% of the total publications selected.
Follow up questions were asked for those selecting Attainment Statistics to understand the use of provisional August (Table 5) and final December post appeals (Table 6) publications.
Publication Type | Responses | Percentage (%) |
---|---|---|
Overall Summary | 29 | 34.1 |
Education Authority Breakdown | 21 | 24.7 |
Sex Breakdown | 20 | 23.5 |
Centre Type Breakdown | 15 | 17.6 |
Of those selecting August publication breakdowns, the Overall Summary was most selected by respondents. This contains the headline figures for attainment at a national level.
Publication Type | Responses | Percentage (%) |
---|---|---|
Overall Summary | 26 | 36.6 |
Sex Breakdown | 16 | 22.5 |
Education Authority Breakdown | 16 | 22.5 |
Centre Type Breakdown | 13 | 18.3 |
Similarly for those selecting December publication breakdowns, the Overall Summary was most selected by respondents. Education Authority (EA) and Sex breakdowns were selected at a similar level with Centre Type breakdowns selected less frequently.
Users selecting Progression Tables were asked a follow up question on their use of the publication to inform its transition to an Official Statistics in Development publication in 2025. Answers included using it for ‘comparative stats’ to compare schools versus a national picture; highlighting that ‘in its current format, it is satisfactory’; and is used to give ‘realistic expectations’ to pupils and their parents. Another comment mentioned a growing interest in ‘the best way to ascertain the path of a pupil from BGE to senior phase’.
Uses of our statistics
For those that selected one or more publication and responded to a follow up question on use, a summary of responses is below. Respondents could select one or more use per publication selected:
Publication | Monitoring Performance (%) | Personal Interest (%) | Policy Making (%) | Research Analysis (%) | Subject / Course Choice (%) |
---|---|---|---|---|---|
Attainment Statistics | 25.9 | 18.5 | 18.5 | 31.5 | 5.6 |
Grade Boundaries | 24.3 | 16.2 | 16.2 | 24.3 | 18.9 |
Results Services / Appeals | 19.2 | 15.4 | 19.2 | 34.6 | 11.5 |
Component Marks | 30.8 | 15.4 | 15.4 | 23.1 | 15.4 |
Equalities Monitoring Report | 22.2 | 16.7 | 27.8 | 27.8 | 5.6 |
Assessment Arrangements | 7.7 | 7.7 | 30.8 | 38.5 | 15.4 |
Estimates | 20.0 | 20.0 | 0.0 | 50.0 | 10.0 |
Progression Tables | 33.3 | 8.3 | 16.7 | 25.0 | 16.7 |
Satisfaction with publications
Of the 42 responses received on satisfaction with our publications, 71.4% were very satisfied or satisfied with our publications. Of the users that were very dissatisfied or dissatisfied, further information on how our publications could be improved to increase their satisfaction was not provided in the follow up questions.
Satisfaction level | Responses | Percentage (%) |
---|---|---|
Very satisfied | 9 | 21.4 |
Satisfied | 21 | 50.0 |
Neutral | 8 | 19.0 |
Dissatisfied | 2 | 4.8 |
Very dissatisfied | 2 | 4.8 |
Further comments on satisfaction
A free text response box allowed for further comments on improving satisfaction with our publications. 16 responses were received. Of these, some were not related to the production and use of our statistical publications (e.g. parents looking for candidate level information - which is available via Subject Access Request; or researchers looking for access to underlying source data - which can be made available via appropriate data sharing protocols).
Of the comments which were related to the development of our publications, there were several requests by users for:
-
‘visualisations’, ‘infographics’, ‘charts’, and ‘dashboards’ to allow for easier selection of data and understanding trends
-
further stage breakdowns for NQ attainment at subject level
-
centre level statistics
More specific requests covered these areas:
-
longer time series to be available instead of having to use multiple files
-
collection of further data on protected characteristics
-
an interest in commentary around gender bias in subject choice
-
assessment mark information to be widened beyond averages to indicate the spread of marks by subject
-
interest in groups of subjects combined together to assess A-C rates for groups of subjects
-
improvements to education authority statistics such as highlighting difficulty using the attainment workbook due to each authority having a separate tab
Other comments flagged issues with difficulties finding the same data year on year due to data changing or webpages moving.
Better communication was also suggested to highlight the publications we produce to users who may not be aware of the different publication types.
Additional comments
Further additional comments received covered:
-
general satisfaction with the publications
-
the improvement to timeliness of publications - with all provisional attainment statistics being available at the same time on Results Day from August 2023
-
SCQF level by SIMD being produced for Attainment Statistics, as well as further stage breakdowns
-
charts or dashboards being preferable to the tables currently produced
Response to the user consultation
The Data and Analytics team welcomes the feedback received from users via the survey. In determining the direction of our publications, the individual suggestions received have been balanced against a review of our regular user requests to our mailbox (data.analytics@sqa.org.uk) in order to ensure that decisions made are justified by overall user demand.
Visualisations, charts and dashboards: There were several requests for better visualisations and better communication of the statistics available from survey respondents. Data and Analytics are considering using the R Shiny package to meet this identified user need. R Shiny applications could allow for interaction with our data in a point and click visual manner and allow for ‘dashboard’ style information to be made available via a more pleasing user interface. We will consider the feasibility of using R Shiny in order to meet this user requirement. If the use of R Shiny proceeds then the development of applications will be phased in at a suitable pace to allow for improvements and feedback from users in line with the Code of Practice for Statistics, as well as to allow staff to develop their use of R Shiny. We will continue to produce our tables for those that prefer to use our data for their own analysis as well as to meet accessibility guidelines for users that cannot interact with a visual interface.
Stage: There were several requests for further statistics on stage data. We currently produce stage data at a national level, by sex, and by centre type in our attainment tables. Producing further stage breakdowns for entries and attainment for National Qualifications is possible. However, given the small entries for some of the National Qualifications, it would not be possible to give further breakdowns for every subject. A review of our regular external requests and FOI did not find a sufficient demand to prioritise the production of further stage statistics. We will keep this suggestion under consideration.
SCQF level by SIMD: A request was made for the SCQF level analysis contained within Attainment Statistics to be widened to include SIMD. We currently produce a breakdown of results for National 5, Higher and Advanced Higher by SIMD, as well as by equalities characteristics, in the Equalities Monitoring Report. Appeals also contains SIMD analysis. Separately, we produce SCQF level by qualification type and stage at a national level, by sex, and by centre type in our August and December Attainment Statistics tables. A further breakdown of these tables by SIMD would be possible at a national level. A review of our regular external requests and FOI did not find a sufficient demand to prioritise the production of further SCQF level statistics. We will keep this suggestion under consideration.
However, this has raised the possibility of amalgamating some of the equalities analysis contained with the Equalities Monitoring Report into the Attainment Statistics publication and tables. We will explore this further in order to meet our requirements as Official Statistics producers.
Centre level statistics: Requests for centre level information are considered on a case-by-case basis currently. This will continue in the interim while we weigh up the potential benefits versus risks of producing statistics at a more granular level.
Webpage layout: Previously, provisional attainment statistics produced in August were subsequently replaced with final December figures. Feedback received via the user survey on the removal of data and movement of information on our statistical webpages was acted upon in December 2024, with provisional August figures being retained alongside December. This decision is supported by the Code of Practice for Statistics requirement that statistics should ‘continue to be publicly available’.
Longer time series availability: The coverage of our current workbooks attempts to balance giving a reasonable period for comparison with accessibility considerations. Currently, the size of our workbooks already requires scrolling in order to view the entire time series of data contained. Increasing the time series included in the workbooks is possible but this would mean that workbooks would increase in size and would require users to scroll further than currently. The feasibility of producing R Shiny applications is being assessed, which may allow for longer custom time series to be selected by users.
Protected characteristics data: Further data on protected characteristics may be collected by Qualifications Scotland.
Further commentary: It is possible to produce an automated summary of changes in entry patterns and attainment for individual subjects by sex, where entry numbers are sufficient. Given the number of individual subjects, a summary of changes at subject level would be lengthy and SQA would not be able to provide additional commentary on any changes identified. A review of our regular external requests and FOI has not identified a demand for additional commentary on gender bias in subject choice.
Assessment mark information: A review of regular requests via our external mailbox and FOI identified a user need for information about the highest mark achieved in an assessment and the number of candidates achieving these marks. In 2023, the Component Marks / Assessment Marks publication was expanded to include the maximum mark achieved by candidates and a count of how many candidates achieved this mark. A review of our regular external requests and FOI did not find a sufficient demand to justify the production of further statistics on the spread of marks currently.
Grouping subjects: A review of our regular external requests did not find a sufficient demand for grouped subject information. The production of grouped subject statistics would also require input from users as to which groups of subjects should be combined together. Users interested in grouped subject statistics are welcome to contact our mailbox.
The general satisfaction from users and noting of improvements to date are welcomed by the Data and Analytics team.
Further plans
We will be monitoring the impact of efficiencies we have introduced into the production process with a view to improving the timeliness of delivery of Official Statistics publications currently released in December. This will be reflected in the publication schedule for 2026 when it is anticipated publications will be produced by Qualifications Scotland following the planned transition of SQA in late 2025.
‘Progression Statistics 2023 to 2024’ will be published on 28th January 2025 as an Official Statistics in Development publication. The summary will include extended commentary on areas including Maths and English, the five highest number of progression routes per level, and the subjects with the highest proportion of candidates progressing.
A review of regular requests via our external mailbox and FOI identified the need for a new publication covering Candidate Achievement. This will be published following Results Day in August in 2025 and is expected to include a breakdown of the number of entries per candidate, number of A to C grades per candidate, number of A grades per candidate, and number of upper A grades per candidate. Coverage will include National 5 to Advanced Higher and include breakdowns by sex and centre type.
A review of regular requests via our external mailbox and FOI also identified the need for information to be published on a more timely basis on Higher National (HN) and Vocational Qualifications (VQ). To this end, we are exploring developing and publishing HNVQ attainment statistics on a more regular basis.
We are investigating routes to better meet the needs of our users through innovative and interactive visualisations. As part of this work, we are looking at the feasibility of using R Shiny with the aim of developing and releasing a R Shiny app for testing.
Lastly, as Official Statistics producers, SQA follow the Code of Practice for Statistics. This is currently under review and we will further consider our publications, communication and further user engagement in line with the refreshed edition (3.0).