FOI22/23 105 Higher Drama Appeals 2022

Date published: 07/12/2022

Please can SQA provide the following information in relation to Higher Drama Appeals 2022:

1. on the clerical check, whether SQA confirmed in relation to each entry that the exam script is the relevant candidate’s script (i.e. that there is no possibility of a mix up over scripts) and information on the process SQA followed to ensure a summer exam script from the main examination diet is correctly attributed to the relevant learner;

2. where alternative assessment evidence (AAE) was supplied in relation to the Question Paper component of the exam only and such AAE was the most robust form of evidence as set out in SQA’s Appeals 2022:

General Information, May 2022 and as described in the Additional Information section below:

a. how SQA carried out the review of the AAE and, as the SQA has stated that all appeals consisted of a holistic review of all alternative evidence, what a holistic review consisted of, how this process was carried out and what notes and records were maintained of such reviews;

b. specifically, whether a review of the AAE involved remarking the AAE and whether a precise mark was given to the AAE in the SQA’ s review;

c. the process used to determine the success of the appeal and whether, following SQA’s review, a revised/appeal mark for the Question Paper component of the exam was then totalled, scaled and added to the marks achieved by a candidate for the Performance component of Higher Drama;

d. if not covered by a-c above, how decisions were arrived at by the SQA in terms of an appeal in the circumstances set out in the Additional Information or Comments? section of this FOI request;

e. what were SQA’s policies and/or practices as to how to maintain consistency between reviews (i.e. what was the policy to ensure one marker’s holistic review was consistent with another marker’s holistic review); and

f. where identical types of AAE were submitted for several learners from the same centre, what were SQA’s policies and procedures to ensure that there was consistency between the reviews of the identical types of AAE submitted for several learners from the same centre; and

3. in grading the AAE and/or the learner on appeal, whether the grade boundary for an A was the same as the summer exam (set at 65%) or if a different A grade boundary was set for review of the AAE. If a different A grade boundary was set for review of the AAE, what the A grade boundary was and the rationale for this decision.

The information requested is in relation to the SQA processes and procedures for Higher Drama Appeals in 2022 in circumstances where:

(i) an appeal was submitted in accordance with the appeals process set out in the SQA brochure Appeals 2022: What you need to know and all conditions for the appeal were met;

(ii) a Higher Drama prelim paper covering all (or substantially all) of the skills, knowledge and understanding assessed in the question paper component was submitted as the alternative assessment evidence for the Question Paper component of the Higher Drama exam and such prelim was sat under exam conditions with an external invigilator;

(iii) the prelim question paper was produced from a mix of questions from commercially produced question papers and past papers and the prelim was submitted with the accompanying marking scheme (again, according to SQA’s Appeals 2022: General Information, May 2022, the most robust form of alternative evidence that could be provided).

It is noted that:

(a) SQA’s Awarding Body Code of Practice (Code of Practice) dated March 2017 provides that, SQA will endeavour to ensure that processes for assessment and the maintenance of national standards are as open and transparent as possible;

(b) SQA’s Governing Principle for assessment set out in the Code of Practice provides that, SQA will ensure that all assessments used in its qualifications are valid and reliable. Validity being a measure of the accuracy of an assessment and reliability being concerned with the consistency of an assessment; and

(c) SQA’s document entitled Appeals 2022: General Information, May 2022, provides that, Appeals must be based on valid and reliable evidence of demonstrated attainment against national standards and Prelims are likely to be the most reliable indicator of performance in a question paper component

1. on the clerical check, whether SQA confirmed in relation to each entry that the exam script is the relevant candidate’s script (i.e. that there is no possibility of a mix up over scripts) and information on the process SQA followed to ensure a summer exam script from the main examination diet is correctly attributed to the relevant learner;

The clerical check process undertaken to support the 2022 Appeal Service confirmed that the examination scripts and all associated marks were credited to the correct candidate. 

During the main examination diet the majority of examination scripts/answer booklets are e-marked – this means that scanned images are made available to markers for marking. The following key processes are in place to ensure that scripts do not become mixed up.

For each subject/level SQA provides each candidate with a script booklet which has a barcode on the front page which aligns with the subject/level and has a set number of pages.
When taking the examination, on the front page of each booklet provided, candidates enter their centre ID (centre name and town), candidate ID (Scottish Candidate Number), candidate name, candidate Date Of Birth, and seat number.
Post examination, scripts are securely packed by SQA invigilators, collected from centres and delivered to the scanning house via a secure carrier.
At the scanning house, an indexing process is also in place to ensure that script images are associated with the correct candidate entry.
After indexing, the images of the scripts are made available to markers for marking by our subject experts.
The marking process also ensures that the integrity of a script has been maintained throughout.

 Where a script/answer booklet is not e-marked, the reconciliation, preparation for marking, marking and associated quality assurance processes are undertaken using the physical scripts by staff and appointees (subject experts). Any queries eg change in hand writing would be flagged and investigated.

2. where alternative assessment evidence (AAE) was supplied in relation to the Question Paper component of the exam only and such AAE was the most robust form of evidence as set out in SQA’s Appeals 2022:

General Information, May 2022 and as described in the Additional Information section below:

a. how SQA carried out the review of the AAE and, as the SQA has stated that all appeals consisted of a holistic review of all alternative evidence, what a holistic review consisted of, how this process was carried out and what notes and records were maintained of such reviews;

For each appeal candidate, SQA’s senior appointees reviewed all the AAE that the centre had presented for that candidate. The evidence was reviewed against the required skills, knowledge and understanding of the national course for that Question Paper component and judged against national standards.

The appointee considered AAE alongside candidate coursework component(s) (centres were not expected to provide alternative evidence of coursework components). Putting all of this evidence together, the coursework and the AAE, the appointee considered the weighting of the course components as stipulated in the course assessment specification and determined the grade that the evidence, altogether, represented.

In the context of a Higher Drama, if AEE was provided for the QP component only, the VA Performance mark stood, and this was considered in the holistic review of evidence ie QP AAE & VA Performance mark. Higher Drama QP component holds 40% weighting and Performing 60% weighting. A holistic review would have included a review of the QP assessment instrument (to ensure it was valid), the associated marking instructions, and the application of the marking instructions to the candidate’s evidence.

The appointee recorded a grade outcome only for each candidate - no further feedback was recorded.

b. specifically, whether a review of the AAE involved remarking the AAE and whether a precise mark was given to the AAE in the SQA’ s review;

The appointees needed to judge the AAE against the national standards for the course. This was not re-marking, however if there were issues with reliability in the marking of the AAE then appointees had to use their professional judgement as to what the evidence indicated, in terms of attainment against the national standard. Appointees didn’t record a separate mark for the AAE, as it is the totality of the evidence, including coursework attainment, and weighted appropriately, that determines the grade outcome

c. the process used to determine the success of the appeal and whether, following SQA’s review, a revised/appeal mark for the Question Paper component of the exam was then totalled, scaled and added to the marks achieved by a candidate for the Performance component of Higher Drama;

See above

d. if not covered by a-c above, how decisions were arrived at by the SQA in terms of an appeal in the circumstances set out in the Additional Information or Comments section of this FOI request;

e. what were SQA’s policies and/or practices as to how to maintain consistency between reviews (i.e. what was the policy to ensure one marker’s holistic review was consistent with another marker’s holistic review); and

All appointees involved in appeals were experienced appointees who are involved in developing SQA assessments and supporting markers throughout the year. They all received significant training prior to starting appeal reviews. Appointees were provided with guidance materials outlining principles and processes to be followed and were briefed by Qualifications Development staff to ensure consistency in approach when carrying out appeal reviews. Subject specific training included the use of exemplar material to exemplify national standards.

Quality assurance of decision making is built into the appeals process in the form of collaborative working, sampling and monitoring of decision making by the Principal Assessor with support from the Qualifications Manager

f. where identical types of AAE were submitted for several learners from the same centre, what were SQA’s policies and procedures to ensure that there was consistency between the reviews of the identical types of AAE submitted for several learners from the same centre; and

Appointees were allocated full centres to work on to ensure consistency of decision-making during appeals. Quality assurance of decision making is  built into the appeals process in the form of collaborative working, sampling and checking by the Principal Assessors with support from the Qualifications Manager

3. in grading the AAE and/or the learner on appeal, whether the grade boundary for an A was the same as the summer exam (set at 65%) or if a different A grade boundary was set for review of the AAE. If a different A grade boundary was set for review of the AAE, what the A grade boundary was and the rationale for this decision.

The focus of appeals reviews was on the grade-worthiness of the assessment evidence presented, irrespective of any grade boundaries or cut-offs – the 2022 SQA grade boundaries were not applied to the assessment evidence