These frequently asked questions (FAQs) relate to submissions to the REF 2021 and policy matters. They have been grouped by category.
New FAQs (August 2019)
If the journal issue is not denoted by the month of publication, you should enter your best possible estimate of the month of publication.
If it gives a season, and the month of publication is not otherwise known, please enter the following months:
• 1 = winter (beginning of year)
• 4 = spring
• 7 = summer
• 10 = autumn
• 12 = winter (end of year)
Tightly-grouped, short items with significant research in common are eligible to be submitted as a multi-component output (paragraph 271, Panel Criteria and Working Methods). In order to encourage the submission of outputs that fully represent the breadth and diversity of research in the arts and humanities, this provision is in place to support the submission of groups of items that would not individually be sufficiently substantial to be submitted as single items. For example, a cluster of related, short encyclopaedia articles, or a series of separately published poems that are part of a defined sequence, would fall into this category. We expect outputs submitted as ‘groups of short items’ to be accompanied by a clear statement explaining the rationale for their grouping.
Separately published papers are discrete outputs and cannot be grouped together as a single output. Where research has been split by a publisher and would only be considered a single coherent work when viewed together, these should be returned as one research output.
This provision is in place to support the presentation of a range of items that collectively and coherently represent the research dimensions of a single creative practice research output. Guidance on the submission of a multi-component output in relation to creative practice is provided in paragraph 265 and in Annex C of the Panel Criteria and Working Methods. Multi-component outputs are expected to include a range of item types which may include, for example images, video and/or audio files, a patent, which collectively enable the panel to access the research dimensions of the creative practice output. It is recommended that outputs submitted under this provision are accompanied by a clear statement explaining the research dimensions of the creative practice submitted as a research output.
Staff who use a different name on published research outputs should be returned to the REF with this name on REF1a. If staff data verification is required from the institution, we will require additional evidence to verify the individual’s identity.
Sub-panels 1-6 and 9 will only routinely refer to author contribution statements on outputs in cases where there are more than 15 authors and the submitted member of staff to whom the output is attributed is not identified as either the lead or corresponding author. If there are errors in an author contribution statement contained within the output, HEIs should flag this in the co-author contribution statement that they provide for the sub-panel to consider. If the sub-panel has any concerns about the information provided, the HEI may be asked to verify the co-author contribution through audit.
In all other cases (15 or less authors, or lead or corresponding author), if the sub-panel has any concerns surrounding the author contribution, we may seek to verify this through audit.
Yes. If the content of a repository is free-to-use (i.e. users are not required to pay to search, read or download the output) then it would meet open access requirements for the purposes of REF 2021.
Where elapsed periods are described in calendar months (for example for Open Access eligibility) the period will be determined by comparing the day of the month inclusively.
a. For example, when considering Open Access requirements as set out in paragraph 249a of the Guidance on Submissions, if an output was deposited on 28th March and met the access requirements on 28th April, it would be considered to be compliant with the Open Access requirements.
b. If an output was deposited on 28th March and met the access requirements on 29th April, it would be considered to be non-compliant.
When comparing calendar months with dates at the end of the month, the first day of the following month will be used where there is no day matching the previous month.
a. For example, if an output was deposited on 28th January and met the access requirements on 28th February, it would be considered to be compliant with the Open Access requirements.
b. If an output was deposited on 31st January and met the access requirements on 1st March, it would be considered to be compliant.
c. If an output was deposited on 31st January and met the access requirements on 2nd March, it would be considered to be non-compliant
These worked examples are using the access requirements but the same logic applies to other periods such as the three month deposit requirements.
No. HEIs may find it useful to adopt RIOXX to help manage open access in their institutions, but the use of RIOXX is not a requirement of the REF.
The REF team will use two metadata services, Unpaywall and CORE, to help with risk-based identification of HEIs that we may wish to select for possible open access audit review. These data tools will help us to identify where an audit should take place, but they will not be used to audit the OA status of outputs at HEIs.
It is not a requirement of the REF that HEIs use either of these metadata services, nor do we expect HEIs to integrate their CRIS or repository systems with these services. We will fully review the risk identification and ranking process for open access, to ensure that HEIs that are not integrated with these services are not put at a disadvantage when compared with those that are.
Some HEIs have nevertheless shown interest in having their outputs appear in these services in order to make available information about the open access status of their research. For HEIs wishing to do so, it is possible to register with both metadata services.
Both services require a URL in the form of an Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Many HEIs will be using a repository that exposes an OAI-PMH and these should be suitable for integration. HEIs can check whether they have a suitable OAI-PMH by searching for their repository on http://v2.sherpa.ac.uk/opendoar/ - the OAI-PMH URL should be listed on the details page of the HEI’s repository.
To register a repository with Unpaywall, you can fill in an online form that is available via the Unpaywall website, under ‘About’ (Data sources) - select the ‘this form’ link. The form requires an email address, the OAI-PMH URL and some information about the HEI and the repository.
To register a repository with CORE, HEIs should send an email to email@example.com stating the OAI-PMH URL, HEI name and repository name.
Submitting units are required to provide up to six key references that represent the body of research or a research project produced by the submitting unit that underpins the impact described in the case study. The sub-panels will not expect each referenced item to meet the quality threshold, but will wish to be satisfied that the research as a whole was of at least two-star quality. Where sub-panels identify within the referenced research at least one output of two-star quality or higher, and this is a key output underpinning the impact, this will normally be sufficient to demonstrate that the underpinning research as a whole meets the quality threshold.
Yes, we recognise that individual researchers may undertake research across multiple disciplines over time and that UOA boundaries are not rigid. Provided the underpinning research is within the scope of the UOA in which it is submitted, a case study may be submitted in a different UOA from the individual.
The HEI can claim the impact from the point at which research carried out by staff while working in the HEI made a distinct and material contribution to it. Any impact prior to that point can be described in the case study as context only (to help explain the background the impact), but cannot be claimed as part of the impact underpinned by the submitting HEI's research.
No. In addition to the maximum of 6 research references, grant information details may also be provided, where appropriate.
A maximum of 10 references to sources that can corroborate the impact may be included in each case study. This may include any number, within the maximum of 10, of factual statements already provided to the HEI. These must be submitted to the REF team by the deadline of 29 January 2021.
The details of a maximum of five individuals relating to these 10 sources may be entered for each case study, and these are to be submitted through the submission system. These five individuals may be contacted directly by the REF team to corroborate the information provided as part of the audit process. We do not envisage contacting more than five individuals for any particular case study, which is why we have set this limit. If a larger number of individuals could potentially provide such corroboration, then five should be selected that best represent this larger group. The corroborating sources listed should focus on the key claims made within the case study.
For further guidance on corroborating evidence, including the use of testimonials, please refer to paragraphs 310 and 311 of the ‘Panel criteria and working methods’.
We would normally expect staff to have been employed in the submitting unit during the eligibility period for underpinning research; that is, the period 1 January 2000 to 31 December 2020. However, we recognise there may be exceptional circumstances where this is not the case – where research that has made a distinct and material contribution to the impact was undertaken shortly before 1 January 2000, the direct output(s) of the research were published on or after 1 January 2000, and where the staff member(s) left the institution in the interval between these points.
Paragraph 65 describes two types of evidence. Firstly, there is the evidence and indicators of impact (provided in section 4 of the impact case study template) – these are normally looked at within the assessment process rather than audit. Corroborating sources to verify the impacts claimed should be provided in section 5 of the case study template. These sources will be made available to panels where requested via panel-instigated audit. We will also audit a proportion of case studies and will examine these sources during that process.
The second type of evidence relates to eligibility of the case study. The third sentence of this paragraph should be read as follows (italics indicate clarifying wording): ‘Evidence to verify the eligibility of the case study may include relevant extracts of staff contracts, records of research grants, or other appropriate evidence.’
During audit an HEI will need to state how it assured itself of a staff member’s ECR status where this is cited in REF6a/b forms, and what evidence it consulted. This may include, for example, evidence of the staff member’s career history (such as a CV) that was submitted by the member of staff when applying for the post at the submitting HEI. In line with the general principles set out in the audit guidance we recognise that HEIs may hold other evidence that is different from the example provided. We will consider all evidence on a fair and reasonable basis, the aim being to obtain sufficient evidence to verify the data that are being audited.
We do not expect to audit routinely supplementary information that has been published for submitted outputs. Audit queries may be raised by the panels about the relationship between the output and the supplementary information. As with all information provided by HEIs in submissions to REF 2021, this must be capable of verification.
When auditing staff details (REF1a/b), joint submissions may be included in one or more of the (up to 4) UOAs from which staff are randomly selected.
Note this means that we might only include staff from one of the submitting institutions within the joint submission, when sampling for audit.
Yes. Responsibility for mapping staff into UOAs will remain with institutions.
All Category A submitted staff (Category A eligible staff with significant responsibility for research) must be returned with a minimum of one output attributed to them in the submission. Where an individual’s circumstances have had an exceptional effect on their ability to work productively throughout the assessment period, so that the individual has not been able to produce an eligible output, a request may be made for the minimum of one requirement to be removed. Where a unit has not submitted a reduction request and is returned with fewer than 2.5 outputs per FTE, and/or has not attributed a minimum of one output to each Category A submitted staff member, any ‘missing’ outputs will be graded as ‘unclassified’.
No. The number of outputs for each submission will be calculated by multiplying the total FTE of ‘Category A submitted’ staff by 2.5.
Staff employed after the census date will not be eligible for submission.
No. The outputs of former staff optionally may be included in submissions, where the staff member was previously employed as Category A eligible when the output was demonstrably generated.
No. To be eligible for return, outputs must be authored by ‘Category A submitted’ staff or staff previously employed as ‘Category A eligible’ when the output was first made publicly available. Outputs co-authored by Category C staff may be submitted within the min. 1 and max. 5 limits of the Category A staff co-author.
Staff with significant responsibility for research are those for whom explicit time and resources are made available to engage actively in independent research, and that is an expectation of their job role. The Guidance on submissions provides a menu of suggested indicators of significant responsibility for research that institutions might use when developing their processes. This guidance does not prescribe a fixed set of criteria that all staff would be required to meet.
No. Evidence of research independence will only be required for staff on ‘research only’ contracts. However, staff on teaching and research contracts who are not independent researchers should be identified through the processes that the HEI has put in place to identify staff with significant responsibility for research.
The required total number of outputs will be applied at UOA level. The units involved in a joint submission are free to select their outputs from the total pool across the participating HEIs, regardless of the proportion of staff that each HEI is contributing (within the minimum and maximum limits per individual). HEIs will be allowed to specify how they would like the funding to be split – either by FTE or by a proportion they agree between themselves.
The Marie-Curie fellowships aren’t included in the list of independent fellowships because whether or not they have independence varies across the fellowships and the disciplines covered. HEIs should use the indicators in the guidance to establish whether individuals on these fellowships have independence.
While often related, HESA cost centres and UOAs are not the same. Responsibility for mapping staff into UOAs rests with the institution – using HESA data may be a logical way to do this, but this may vary across departments or institutions. As long as the mapping is logical, and rationale can be provided for the decisions to map staff into certain UOAs, it would be acceptable for HESA and UOAs to not align exactly. The REF team may choose to audit in certain cases (for example, where large numbers of staff have been submitted into UOAs which are unrelated to their HESA cost centre).
Staff circumstances and codes of practice
The funding bodies require HEIs to conduct equality impact assessments (EIAs) on the policies outlined in their COP. HEIs are encouraged to refer to these EIAs throughout the COP and may wish to include them as appendices. The funding bodies will require HEIs to submit the final version of their EIA, after the submission deadline in November 2020.
No. The removal of the requirement to submit a minimum of one can only be applied where the staff member has not been able to produce an eligible output in the REF period. However, the unit may still be able to apply for a reduction in the total number of outputs required for the unit based on the circumstances of this staff member.
The guidance on submissions sets out the data requirements for any staff circumstances requests. We have attempted to reduce the amount of information we request from HEIs and do not require HEIs to submit any evidence up-front. HEIs should ensure that they have sufficient evidence to satisfy themselves that the circumstances are as claimed. This may be requested through audit. Further details about audit requirements will be published in summer 2019.
Yes. Where there are changes to the Category A submitted staff employed in the unit after the request has been submitted, institutions will be able to make requests for further reductions at the point of submission by amending their REF 6a/b forms. Decisions in these instances will be taken during the assessment year – with respect to the new details only.
The identification of ECRs for the HESA return will require a different process to reporting ECRs with regard to requesting staff circumstances for REF 2021. The funding bodies and EDAP were keen to avoid a two-tier approach to staff circumstances, whereby some were identified automatically, while others would have to be declared. HEIs must therefore provide ECRs with the opportunity to come forward voluntarily to request that they wish the institution to adjust expectations because of their ECR status, or request the removal of the requirement of the minimum of 1.
Junior clinical academics will need to come forward voluntarily to request that they wish the institution to adjust expectations because of their status, or request the removal of the requirement of the minimum of 1. The template disclosure form contains a tick-box option where an individual can declare their status as a junior clinical academic.
Submitted data will be kept confidential to the REF team, the REF Equality and Diversity Advisory Panel, and main panel chairs. All these bodies are subject to confidentiality arrangements. The REF team will destroy the submitted data about individuals’ circumstances on completion of the assessment phase.
Yes, institutions within a joint submission, where one institution is submitting only to that UOA, may agree to use the code of practice of the lead institution. Explicit agreement will be required from all involved HEIs to acknowledge the submission of a single code of practice to cover all institutions.
The code of practice must detail the relevant governance and consultation structures within all HEIs for identification of significant responsibility for research (SRR) (if institutions are not submitting 100 per cent of Category A eligible staff), identification of research independence, and the selection of outputs.
The processes applied to identify SRR and research independence, and to select outputs for submission should be consistent across all institutions, though we acknowledge that structures may differ slightly between institutions. Joint processes must be articulated clearly and must not compromise adherence with other areas of the code of practice that apply to either institution’s submission in other UOAs and overall.
If this applies to your institution please contact firstname.lastname@example.org for more information.
Staff employed to cover family-related leave (and any other equality-related leave) should be treated the same as any other staff member. That means that:
- If they are employed by the HEI on the census date they are returned as a current member of staff (if they are eligible)
- If they are no longer employed, the HEI can submit their outputs from that period, the same as they would with any former member of staff.
Please note, this is different from arrangements for secondments. More information on secondments can be found at 120c of the Guidance on Submissions.
There is no obligation to use the staff circumstances template provided by the REF team, but we strongly suggest using this as the basis for collecting voluntary declarations of circumstances from your staff.
Yes, these outputs may be submitted. Where this is the case, institutions should amend their circumstances requests (REF 6a/6b) at the point of submission to adjust the overall number of outputs.
Rounding will be to the nearest whole number. Values ending in .5 should be rounded up.
Yes, where they are within the publication period and meet any other applicable eligibility criteria, these outputs may be included in submissions by the institution employing the staff member on the census date.
Yes. The minimum and maximum limits on the number of outputs will apply to the person, not their FTE.
Each missing output or case study will receive an ‘unclassified’ score.
Sub-panel 3 (Allied Health Professions, Dentistry, Nursing and Pharmacy) and Sub-panel 12 (Engineering) will provide discrete output sub-profiles against specified areas to the head of institution where requested. Details on how to request output sub-profiles in these UOAs can be found in the ‘Panel criteria’, Part 3, Section 1.
No. The REF is governed by a principle of equity and is committed to the fair and equal assessment of all types of research and forms of research output.
Yes. As was the case in REF 2014, each main panel has provided guidance on how outputs of extended scale and scope are characterised in their disciplines, and on the process for requesting an output to be double-weighted.
For Category A submitted staff, outputs that are within the publication period and meet any other applicable eligibility criteria (for example, open access requirements) are eligible.
No. An individual and their outputs can only be submitted to one unit of assessment. Where an individual holds a joint appointment across two or more submitting units within the same institution, the HEI must decide on one submission in which to return the individual.
It is our view that a ranked list would add greater complexity to the submission process for HEIs, in ensuring that the minimum and maximum boundaries are adhered to in the final set of assessed outputs. Institutions should therefore include a ‘reserve’ output for each output requested for double-weighting.
Yes. The decision whether to request double-weighting lies with the submitting unit.
Where the double-weighting request is accepted, the output will count as two items against the individual to whom it is attributed. (If it is a co-authored output, institutions may attribute the output to a maximum of two members of staff returned within the same submission, in which case it will count as one output for each of them). If the panel does not accept the request, and the output remains single-weighted, it will count as one item.
Double-weighting requests should be made in accordance with the minima and maxima requirements for attributing outputs to staff, so that in the event the request is accepted, or in the event that it is not and the reserve output is instead assessed, the minimum of one requirement is met for each Category A submitted staff member (unless individual circumstances apply), and no more than five outputs are attributed to any one member of current or former staff.
An output first published in its final form during the REF 2021 publication period that was ‘pre-published’ in the previous publication period – whether in full in a different form or as a preliminary version or working paper – is eligible for submission to the REF, provided that the ‘pre-published’ output was not submitted to REF 2014 by any institution.
Outputs made publicly available while a Category A submitted staff member was a research student are eligible for submission with the exception of a thesis or dissertation. For impact, the research underpinning an impact case study must have been undertaken while the person was working in the submitting HEI; any research undertaken while the individual was a research student could not be submitted as underpinning research for impact case studies.
The outputs from a PhD by publication can be submitted to REF 2021 by the HEI employing the staff member as Category A eligible on the census date. However, PhD theses and dissertations are not eligible for submission to REF 2021.
The tolerance band will be applied at UOA level.
If the staff member to whom the output is attributed was employed at a different UK HEI at the point of acceptance and the new HEI has not been able to determine compliance with the criteria, the output will not be required to meet the open access criteria.
Journal articles with an ISSN or conference contributions in conference proceedings with an ISSN produced by staff while they were at a former employer should not be returned as non-compliant with the REF 2021 OA policy requirements. These outputs may be out-of-scope if accepted for publication before 1st April 2016, or the submitting HEI may know that the output is compliant with the requirements. Otherwise the output should be returned with the exception outlined in paragraph 254a of the Guidance on Submissions.
In-scope outputs produced by former staff may not use the exemption in paragraph 254a of the Guidance on Submissions (aka. the institution is submitting something from a staff member that has moved on). The output may have to be returned as non-compliant if it is in-scope and compliance cannot be demonstrated and another exception is not suitable.
For the purposes of the REF, interdisciplinary research is understood to achieve outcomes (including new approaches) that could not be achieved within the framework of a single discipline. Interdisciplinary research features significant interaction between two or more disciplines and/or moves beyond established disciplinary foundations in applying or integrating research approaches from other disciplines.
All sub-panels will have at least two members appointed as interdisciplinary advisers. The interdisciplinary advisers will offer guidance to the sub-panels in their assessment of interdisciplinary outputs to enable their robust and valid assessment. This may include advising on the allocation of outputs and the calibration and moderation of scoring. Interdisciplinary advisers will not necessarily be expected to assess all interdisciplinary outputs submitted to their panel. The advisers will work in a network with their counterparts on other sub-panels. The network will meet at key points during the assessment phase and will provide a forum for reviewing joint working arrangements and identifying wider expertise requirements.
A case study should include references to up to six research outputs that represent the body of research or a research project that was carried out at the submitting institution. These should be key outputs that underpinned the impact, and that best demonstrate the quality of the body of work or project. The sub-panels will not expect each individual output to meet the quality threshold, but will wish to be satisfied that the listed work was predominantly of at least two-star quality.
Where more than one submitting unit made a distinct and material research contribution to an impact, each of those submitting units may submit a case study of the impact. Each submitting unit will need to show that its research made a distinct and material contribution to the impact. This applies whether an HEI wishes to submit the same impact in different submissions, or different HEIs.
Yes, we recognise that individual researchers may undertake research across multiple disciplines over time and that UOA boundaries are not rigid. Provided the underpinning research is within the scope of the UOA in which it is submitted, a case study may be submitted in a different UOA from the individual.
No. The underpinning research must be carried out by staff working in the submitting HEI and must be within the scope of the relevant UOA descriptor. It may include research undertaken by staff employed on non-Category A eligible contracts.
Units are not prohibited from submitting more than one case study based on the same body of research. However, they should take into account the extent to which this might reduce the reach and significance of the impact described.
The institution submitting a case study must have produced research which has made a distinct and material contribution to the impact described in the case study. Where a researcher has moved to a different institution during the period in which a body of research underpinning a case study was produced, the submitting institution should make clear that the research undertaken during the period the researcher spent at that institution made a material and distinct contribution to the impact claimed.
Yes. Underpinning research referenced in a case study may also be included in a submission as an output (listed in REF2), without disadvantage. In these situations, the assessment of the impact case study will have no bearing on the assessment of the quality of the output.
No. The panels recognise that the link between research and impact can be indirect and non-linear.
Yes. Data about research income and research doctoral degrees awarded must fall within the assessment period: 1 August 2013 to 31 July 2020.
Institutions can provide any data that they consider appropriate as evidence for claims made in the statement. A working group of the Forum for Responsible Research Metrics was established to consider the types of data that institutions might select to include, and the group provided guidance to the panels. Guidance on the inclusion of quantitative data can be found here.
As in 2014, the environment template includes a section for submitting units to outline the ‘unit context and structure, research and impact strategy’, including how research is structured across the unit. The panels set out their expectations for the environment statement in the ‘Panel criteria’, Part 3, Section 5.
The sub-panels will use the information provided in the institutional-level statement to inform and contextualise their assessment of the relevant sections of the unit-level template. The institutional-level statement will not be separately assessed or separately scored by the sub-panels.
Yes. The examples provided by the Forum are not intended to be prescriptive, or exhaustive. When including indicators, institutions should follow the eight principles set out in Annex A of the Forum’s guidance.
We intend to carry out a survey of institutional submission intentions during 2019 which will be used to inform the panel membership requirements for the assessment phase appointments. In early 2020, we will invite associations or organisations with a clear interest in the conduct, quality, funding or wider benefits of publicly funded research – except for mission groups, individual UK HEIs and groups within or subsidiaries of individual UK HEIs - to make nominations for the assessment phase panel membership. The nominations will be considered by the panel chairs and the funding bodies to ensure that appointments reflect the diversity of the UOA subject disciplines.
Assessors are appointed to contribute to the assessment of particular aspects of submissions (e.g. outputs or impact) as requested by the sub-panel. Sub-panel members retain responsibility for producing the draft assessment outcomes for each submission as a whole and for recommending these to the main panel.