AGS data quality risk management and control

As part of this review, AGS completed an Internal Control Questionnaire (ICQ) for CSO (See Appendix 2). This questionnaire asked a series of control questions relating to the collection and quality assurance of PULSE data by AGS. In section A1, AGS were asked to outline the key risks to data quality, the controls in place to manage them, and what quality measures are in place to check that the risks are low/remain low.

The following lists the key areas of potential risk to data quality together with summary actions taken by AGS to mitigate against each risk.

      1. Crime incidents not being recorded on PULSE when they should have been

Relevant incidents (including all crime related incidents) are transferred from the Computer Aided Dispatch (CAD) system to PULSE upon closure in Command & Control Rooms. The new Garda SAFE system (CAD II) has meant that the CAD and PULSE systems are now integrated such that a PULSE incident is automatically created when a crime-related CAD incident is closed. This system was introduced in part as a quality assurance mechanism to prevent a scenario where a crime reported by a member of the public was not recorded on PULSE.

The Garda SAFE system has been deployed in three regions (Southern, Eastern and North Western) as of June 2023. The remaining Dublin Metropolitan region will be deployed in Q4 2023.

      2. PULSE Crime Incidents are not being recorded to the correct standard

There is a three stage process in the incident creation and validation:

1. The majority of crime Incidents are recorded by the Garda Information Services Centre (GISC). In the case of crime incidents being reported locally (i.e., at Garda station), the incident is recorded jointly by the GISC call taker and Garda member. GISC incident creation rates are reported monthly internally to Senior Management also published on the AGS website.
2. Every incident is reviewed, usually by a GISC Reviewer and any queries are sent back to the Garda member for clarification.
3. Incidents are reviewed locally at daily, weekly and monthly PAF meetings.

In addition to this, PULSE has a number of inherent controls built within the system that mandate the minimum level of data that is required for an incident to be recorded.

      3. PULSE Crime Incidents are not being reviewed or quality assured in a timely fashion

The review process for crime incidents is outlined in 2 above and the target is that each incident receives an initial review within 3 days. This target is monitored and reported at the Deputy Commissioner Policing and Security’s Monthly Performance Assurance Framework alongside reporting at GISC.

      4. PULSE Crime Incidents are incomplete or not accurate

In the event there are data quality issues with an incident, it is placed on Review Clarification. This generates an action for the Garda member responsible for the incident to correct it – directly or through GISC. It is then reviewed again.

Incidents on review clarification cannot be marked resolved and will continue to be flagged to the Garda until addressed. Each supervisor receives a report on the number of outstanding clarifications on their team as part of the PAF process.

      5. PULSE data is incoherent and conflicting

GISC have developed a number of coherency checks on PULSE data. The entire PULSE database is loaded into a Hadoop based big data analytics platform nightly. Automated checks are then carried out on a monthly basis on the entire database using the Microsoft Power BI based tools. Any errors are automatically flagged for review to be placed on Review Clarification. Metrics on error rates are published monthly on the Garda website. An example of this is the reported date versus created date representing an error rate 0.040%. 

The coherency checks (some of which mirror the CSO checks conducted for the CSO quality reviews) are currently carried out on a monthly basis and AGS envisage that GISC will in future move to carrying out the checks on a rolling basis. Data coherency checks focusing on 11 aspects of data quality have been developed using the Hadoop and Microsoft Power BI tools. These coherence checks ensure the data, when shared with the CSO, has been corrected and is coherent.

AGS Quality metrics

AGS has been publishing a series of monthly quality metrics on its website since May 2022. Please see statistics on AGS website.

These include among others the following:

• Percentage share and numbers of Crime Incidents created at GISC versus locally at Garda stations by Garda Members.
• Crime incident reclassification – overall share and that split between GISC and local.
• Crime incident invalidations – overall share and that split between GISC and local.
• The number of cases created at the Garda Information Services Centre (GISC) versus locally at Garda stations by Garda Members. Incidents are grouped together on PULSE or ‘cased’, to facilitate application of the Crime Counting Rules.
• Data quality check on application of crime counting rules for all incident types.
• Victim-Offender Relationship – data quality coherency check.
• Number of crime incidents where the reported data was later than incident creation date.

CSO understands that this series of metrics will be expanded over time.

Conclusion

The responses to the ICQ show that AGS have developed and implemented a more formal data quality management system for PULSE that ensures fit-for-purpose crime data. Risks to data quality are clearly identified and mitigations for those risks are clearly outlined.

Metrics on these quality checks are published on the AGS website since May 2022 and the CSO welcomes the recently published AGS review of the quality of 2022 PULSE data, the range of checks which very much replicate those carried out by the CSO for this review.