Review of the Quality of Recorded Crime Statistics 2020

In 2018, the CSO published a Quality Improvement Proposal which identified the actions which the CSO saw as critical to improving the quality of PULSE data for statistical purposes. Within the proposal, the CSO outlined the need for a data quality management framework within AGS which “clearly outlines what constitutes good quality data, what data quality control mechanisms are in place, and how data quality is to be measured, monitored and independently audited to ensure fit-for-purpose crime data”. This recommendation still stands and is the primary focus of this chapter. 

As with all administrative data which is sourced and used by CSO for statistical purposes, the administrative data owner is primarily responsible for the data quality of their files, in this case AGS in respect of the PULSE crime incident dataset. The CSO is a downstream user of PULSE data, and is separate from the data recording and data quality assurance processes within AGS. Given the level of public interest served by Recorded Crime statistics and the nature of data quality issues which have arisen previously in respect of PULSE, the importance of strong, transparent quality assurance processes which ensure good quality data outputs cannot be overstated.

Recorded Crime statistics are the only statistics the CSO publishes which are categorised as “Under Reservation”. The process of building enhanced trust in PULSE data, such that the “Under Reservation” label would be lifted in the future, lies in AGS measuring and monitoring for quality, and ensuring that controls are in place to avoid systematic weaknesses in the end-to-end data recording process - from when an incident is first reported to AGS to the point where a record of the incident exists on PULSE which faithfully represents the current organisational position of AGS and can be used for statistical purposes. 

What does “good quality” mean in the context of crime statistics? The CSO notes the following from the Police Service of Northern Ireland (PSNI) User Guide to Police Recorded Crime Statistics in Northern Ireland: -

“the key objectives of the PSNI crime recording quality assurance process are to minimise the level of missed crime and to ensure that the correct crime type is recorded”. The main quality assurance processes and checks fall into two main groups;

1. Ensuring crimes are correctly identified and recorded in line with documented standards, including identifying potentially missed crimes from the PSNI’s Command and Control system, and
2. Undertaking “data quality checks on an ongoing basis in order to identify and correct inconsistencies within the data”, with “priority given to those that have a direct impact on the quality of published recorded crime figures”.

The PNSI have transparently reported on the quality of their data (see Section 4.2). A similarly transparent approach to reporting data quality by AGS, rooted in relevant quality metrics assessed regularly, will be important to build further trust in PULSE data.

“Good quality” means fitness for purpose. Data collection must be designed, in the first instance, to align with internal and external user needs, in terms of both the type and format of data that is recorded and the required precision or accuracy of the data. Further to this, the practical challenges of collecting accurate data should be considered, including understanding and mitigating for potential risks to data quality.

AGS have implemented quality assurance processes at the Garda Information Service Centre (GISC), which handles much of PULSE incident data entry and which reviews all PULSE crime incident records, and in Garda stations via the Performance and Accountability Framework (PAF). Between GISC and PAF there is a high level of review and supervision of incident records at a local level, including the classification, completeness and quality of PULSE data. It is a matter for AGS, perhaps via an internal audit function, to evaluate how effective these internal processes are as regards ensuring good quality data. From the perspective of CSO, in terms of CSO representing the wider user community, what is important is that:

• there is assurance that a rigorous risk management approach is being taken to identify and implement internal controls which prevent systematic weaknesses in the recording process
• the effectiveness of the internal controls is known

In addition to the thorough checking of PULSE records which is carried out by GISC and PAF, there is a need for a monitoring function within AGS which routinely checks the quality of PULSE data in terms of the identified risks to data quality. The purposes of such a function may include to: -

• identify and flag errors and inconsistencies which can be referred back to local districts for clarification or correction
• to systematically evaluate data quality and track progress over time
• to form the basis for transparent reporting of data quality to downstream data users

Micro-level checks can be used to check the individual compliance of records with recording policy and standards, for example:

• to test the logical sequence of date occurred, date reported, etc.
• to test the application of crime counting principles
• to check for the presence of a linked victim record in incident types where the recording of an injured party is anticipated
• to check the coherence of detection status fields against recorded sanctions
• to search free-text narrative fields for specific keywords to identify crime characteristics (e.g. use of weapons)

(Note that the above list is illustrative only. It is a matter for AGS to decide the types of checks that are needed to mitigate against identified risks to data quality)

The aggregated results of micro-level checks can also be used as quality indicators and metrics to report on data quality, to report progress over time, to test the effectiveness of policy intervention, and to provide assurance to downstream data users.

Macro-level checks, which take a ‘helicopter’ view of the data, are useful to assess stability and change, and to ensure comparability and coherence of data across time periods and across Garda management districts (for example to ensure uniformity of approach in the application of the stated crime counting rules).

When shared with downstream users, the checks outlined above represent the type of quality reporting which is necessary to build trust and confidence in the data. There is, at present, an absence of reporting on PULSE data quality by AGS themselves, and this limits the CSO’s ability to properly inform users about PULSE data quality. Accordingly, there is a need for AGS to monitor and evaluate data quality, and, ultimately, to report on data quality in a transparent way to users.  

Audit is a key process in providing independent, retrospective evaluation of products and processes. CSO have outlined the potential for internal audit to be used to evaluate whether data collection procedures are consistent with the operational guidance, and whether data outputs are fit for purpose for operational requirements.

CSO understands that the Garda Professional Standards Unit (GPSU) is the relevant internal body to perform such a function, and that data quality reviews have already been carried out by this unit. CSO would suggest that it would be important that an Garda internal audit (IA) of the crime reporting and recording process within AGS be scheduled. In so far as it would be possible given the particular sensitivity which can attach to internal audit reports, the findings of the IA should be communicated to key stakeholders with an outline of what recommendations, if any, are necessary in order to build further confidence in the crime reporting and recording process.

The CSO advises users that Recorded Crime statistics remain categorised as 'Statistics Under Reservation' for the present time, with the recently reported 999 calls issue unfortunately demonstrating why such a quality label remains necessary for the present.

The CSO is firmly of the view of the following: -

• There is clear evidence of further quality improvment in PULSE data since the last CSO quality review in 2018, as highlighted in this report (for example the improved data relating to detections and suspected offenders resulting from the implementation of internal controls on PULSE)
• That a suite of systematic, documented data quality checks, carried out by AGS and transparently communicated to users is important to build enhanced confidence in Recorded Crime statistics
• That, given the recently reported 999 calls issue particularly, a comprehensive end-to-end review of the incident reporting and data recording process is needed, with a particular focus on examining the risks of agreed AGS procedure for crime reporting and recording not being followed

Both the CSO and AGS have a shared interest in ensuring that, at some point, the 'Statistics Under Reservation' label is no longer applied to Recorded Crime statistics. At that point, users would be able to use the data without reference back to a significant caveat on its use. Such a point will be reached when there is greater confidence in the crime data and the underlying processes which generate the data, and the pathway to generating such confidence is through a trusted and transparent data quality verification process by AGS, rooted in a clear appreciation of the data quality risks involved in the crime reporting and recording process. Robust, trustworthy crime statistics are a vital tool for decisions makers and society, and the CSO remains committed to advising and assisting AGS in improving data for statistical purposes.