1982

Act

TRM Evaluation and
Monitoring Framework
EVIDENCE BASED POLICING CENTRE
Information
Official
the
under
1
Released

TRM Evaluation and Performance
Monitoring Framework
1982

Act
Project Name:
Evaluation and Monitoring of the Tactical Response Model
To:
Tactical Response Model Project Leaders
Date:
05/11/2021

Produced by:
Evidence Based Policing Centre (EBPC)

Reviewed by:
Dr Jess Dent, Lead Investigator TRM – EBPC
Released by:
Simon Williams, Director – EBPC
Endorsed by:
Sub Portfolio Committee on 11th November 2021.
Information
Official
the
under

2
Released

Summary
This document sets out the framework for monitoring and evaluating the impact and 1982
implementation of the Tactical Response Model (TRM). The below diagram, Tactical
Response Model Evaluation and Monitoring Framework, summarises the outcomes and
impacts of the TRM from a monitoring and evaluation perspective. The impact
Act
evaluation of TRM will be supported by a process evaluation that, through a range of
data sources and analysis methods, wil assess the extent to which the TRM has been
implemented and is operating as intended to achieve those outcomes. This document
also reflects a broad evaluation management plan that sets out how we propose to
implement the evaluation.
Information
Official
the
under

3
Released

TRM Draft Evaluation and Performance Monitoring Overview  1982

BE SAFER
FEEL SAFER
POLICING BY CONSENT
Act

E

M

FRONTLINE STAFF ARE
FRONTLINE STAFF &
O
COMMUNITIES
COMMUNITIES
COMMNITIES HAVE TRUST &
C
SAFER IN THEIR DAY-
THEIR WHANAU FEEL

ARE SAFER
FEEL SAFER
CONFIDENCE
TO-DAY DUTIES
SAFER

OUT

Increase staff confidence in
Increase public feelings of safety
Reduce likelihood of harm through risk identification & deployment
Increase public trust and confidence in New Zealand Police
decision-making
after high-risk events

S
Information
T
Reduce likelihood of harm during response
Increase staff and whānau feelings
Increase public feelings of safety
C
of safety
Increase feelings that police have an appropriate presence
overal
A
and response
Improve access to specialist

IMP
capability and intel igence

Official

THE IMPACT EVALUATION WILL HELP INFORM THE EXTENT TO WHICH WORKSTREAMS OF THE MODEL HAVE IMPACTED ON OUR OUTCOMES

ENHANCED FRONTLINE TRAINING
the

Introduction of four additional days of scenario-based training aiming to focus on appropriate tactical response and de-escalation
ESS

C

O

PR

IMPROVING FRONTLINE ACCESS TO SPECIALIST CAPABILITY
D
Deployment of staff with advanced tactical training as Tactical Prevention Teams and as Tactical Dog Teams (two-up dog teams)
under

AN
S

IE
STRENGTHENING RISK-BASED DEPLOYMENT & TECHNOLOGY

T
Addition of 28 new intel igence positions to support an intel igence-led, risk-based deployment model

IVI
Developing technology to provide more real-time risk information (including SSPOI and SearchX)

Providing 24/7 DCC coverage

ACT

Released
4

Background
The Policing environment in New Zealand is constantly evolving with direct impacts on  1982
frontline staff. To ensure frontline staff can respond safely to keep homes, roads, and
communities  free  from  harm,  capabilities  need  to  stay  ahead  of  this  changing
environment.  Within  this  context,  and  as  part  of  the  broader  Frontline  Safety
Act
Improvement  Programme,  the  Tactical  Response  Model  (TRM)  aims  to  improve
frontline  capability,  ensuring  that  it  is  fit  for  purpose  within  current  and  future
operating environments.
The TRM intends to achieve the following three specific outcomes:
1.  Frontline staff feel safer and more confident in their day-to-day duties
2.  Frontline staff are safer in their day-to-day duties and
3.  Communities are safer.
The model aims to make frontline safer and feel safer through increased capability and
capacity: enhanced tactical training, improved frontline access to specialist capability,
Information
and strengthened risk-based deployment, and technology.
We will be evaluating the TRM through three major workstreams. These are reflected in the
Table 1, below.

TRM Workstreams
Enhanced frontline
Improving frontline access  Strengthening risk-based
Official
training
to specialist capability
deployment and
technology
Components
the
Four added days of tactical
Tactical Dog Teams
New tactical intelligence
safety training
(handler accompanied by
staff and products
tactical operator)
New technologies (SSPOI
Tactical Prevention Teams
risk prediction tool;
SearchX information
search tool)
under
New processes (24/7 DCC
support for tactical
deployment; tactical
deployment model)
Table 1: Summary of the TRM workstreams and components

6
Released

The components of each workstream, outlined in Table 1, are intended to operate, to
some degree, both individually and holistically to achieve the TRM’s impacts and 1982
outcomes. The frontline training workstream equips frontline staff to respond to threats
to their safety as they go about their day to day jobs, to make them safer and feel safer.
The new specialist capability workstream intends to provide new tactical teams with
advanced training and skills to be deployed to high risk situations, implemented to keep
Act
frontline safe through resolving high risk situations in a way that keeps the community
safe. The risk-based deployment workstream intends to provide better tactical
intelligence to decision-makers in a timelier manner, enabling risks to be identified and
responded to in a way that maximises the safety of our people and our communities.
Information
Official
the
under

7
Released

Evaluation Objectives
1982
This evaluation comprises two objectives. These are (1) to evaluate the outcomes of the
TRM (impact evaluation) and (2) to assess the extent to which the TRM has been
implemented and operated as intended (process evaluation). In addressing these
objectives, the evaluation will:
Act
•  Evidence the extent to which the TRM has delivered its intended outcomes
•  Explain, where relevant the contribution of each workstream to TRM outcomes
•  Identify factors that impacted the delivery of intended outcomes through the
TRM
•  Identify ways in which the TRM could be improved to deliver
intended outcomes and
•  Identify any unintended effects of the TRM model
Evaluation Questions and Key Term Definitions
To address the objectives of the evaluation, the fol owing overarching evaluation
Information
questions have been developed, and are reflected in Table 2.

TRM Workstreams
Enhanced frontline
Improving frontline access to
Strengthening risk-
training
specialist capability
based deployment
and technology
Overarching Impact Evaluation Questions
Official
Do frontline staff feel safer and more confident in their day-to-day duties as a result of
the TRM?
Are frontline staff safer in their day-to-day duties as a result of the TRM?
the
Does the TRM improve the wellbeing of the whānau of frontline staff?
Are communities safer as a result of the TRM?
Overarching Process Evaluation Question
To what extent is the TRM model operating as intended?
Table 2: Evaluation questions
A more detailed description of the specific evaluation  designs for each evaluation
under
question, as well as their limitations and risks can be found in Appendix A.

9
Released

Within each of the above overarching questions are sub questions tailored to assess
the impact of individual components of the model.
1982
Training:
•  Does the tactical training make frontline officers safer compared to before
they had received the training and/or officers who are yet to receive the Act
TRM training?
•  Do frontline officers who have received the TRM Training make safer decisions
compared to before they had received the training and/or than officers who
are yet to receive the TRM Training?
•  Does the TRM training make frontline officers feel safer and more confident
than officers compared to before they had received the training and/or
officers who are yet to receive the TRM training?
Specialist Capability
•  Does the deployment of Tactical Dog Teams make frontline safer compared to
when they don’t have access to Tactical Dog Teams?
Information
•  Does the deployment of Tactical Dog Teams make frontline feel safer and
more confident compared to when they don’t have  access to Tactical
Dog Teams?
•  Does the deployment of Tactical Prevention Teams make frontline safer
compared to when they don’t have access to Tactical Prevention
Teams?
•  Does the deployment of Tactical Prevention Teams make frontline feel safer
and more confident compared to when they don’t have access to Tactical
Official
Prevention Teams?
  Definitions of Terms
The following terms from the above evaluation questions have been defined through a
the
series of workshops held with members of both the EPBC evaluation team and TRM
teams:
Frontline staff feel safer. Officer safety is defined as to whether frontline officers feel
that: 1) they can respond to situations as part of their job in a way that minimises the risk
of harm to themselves, their col eagues and others; and 2) they have the resources and feel
under
supported to do their job.
Frontline staff feel confident. Frontline officer confidence is defined as to whether
frontline officers feel that: Their skil s, knowledge, and training enable them to respond
effectively to situations as part of their job.
Are frontline staff safer. Safety is defined as: providing frontline staff with training,
equipment, and capability, allowing them to respond to situations in a way that minimises

10
Released

risk of harm to themselves, col eagues, and others.
Communities are safer. Safety is defined as: ensuring police use justifiable, appropriate,  1982
proportionate, and necessary, tactical responses that minimise the risk of physical and
emotional injuries to the community.

Act
  Overall Evaluation Approach
A mixed-methods approach will be used to answer the evaluation questions. The use of both
quantitative (e.g. surveys and police data) and qualitative methods (e.g. interviews, focus
groups and thematic analysis) for evaluation data col ection and analysis provides a more
holistic view and offsets the limitations of using either methodology on its own. The validity
and reliability  of evaluation findings are further reinforced through use of data triangulation,
which involves drawing on multiple methods, sources, and perspectives.

Impact Evaluation
Information
The  impact  evaluation  is  concerned  with  understanding  the  extent  to  which  the  TRM
delivered its intended outcomes.
The  components  of  the  TRM  in  the  risk-based  deployment  workstream  (i.e.,  tactical
intelligence staff, training, products, and tools, 24/7 DCC support and the Tactical Deployment
Model) operate collectively to support risk-based deployment of frontline and new specialist
teams. It is difficult to separate the effects of each individual component through individual
Official
impact  evaluations.  As  such,  we  wil   assess  the  impact  of  the  risk-based  deployment
workstream through the overal  impact of the TRM on feelings of frontline officer safety. We
wil  examine in greater detail the operation of each component of risk-based deployment
through the process evaluation, when considering the extent to which they have operated as
the
intended within the TRM system.
To help us understand additional impacts of each of the other TRM workstreams we have
designed additional impact evaluations. We plan to evaluate outcomes of TRM both as a
whole, as  well as  the  contributing  impacts of workstreams 1 (frontline training) and 2
(specialist capability). These contributing impacts include the effects of these workstreams on
under
safety  and feelings of safety and confidence of frontline, as well as improvements in
capability. We will evaluate each TRM workstream against questions that include but are not
limited to:

•  How and in what ways has decision-making changed under the TRM model? This
question  is  important  to  understand  how  the  TRM  is  making  our  people  and
communities safer by improving decision-making. This includes targeting high-risk

11
Released

offenders, de-escalating situations, deployment of specialist capability, and use of
tactical intelligence products.
1982
•  How and in what ways our people think that TRM  has affected them? This
question is important to understand whether people think that their safety or feelings
of safety can be attributed to the TRM model and which of its workstreams they think
are contributing to their safety/feelings of safety.
Act

A high-level summary of the measures, methodological approach and sources of information
are provided in Figure Tactical Response Model/ Evaluation Approach below. Further detail is
provided in Appendix A.

Process Evaluation
In addition to examining the impacts and outcomes of the TRM, we wil  also concurrently
explore how these impacts and outcomes were enabled through the operation of the TRM
in the Proof of Concept (PoC) districts and the extent to which the TRM was implemented
as  intended.  The  process  evaluation  is  important  because  it  wil   establish  what  factors
Information
facilitated  or  presented  barriers  to  implementation,  informing  improvements  to  TRM
processes as the model is implemented nationally.
Process evaluation questions wil  be answered using a mixed methods research design,
collating  as  much  information  as  possible  from  a  range  of  quantitative  and  qualitative
sources.     Where possible, we wil  triangulate between multiple sources of data, which wil
include our administrative data, documents (e.g. policies and procedures, debrief forms,
Official
intelligence products), interviews and focus groups with our people and communities, and
observations of the TRM in practice—though field observations wil  be limited to situations
that do not pose risk to observers, such as observing training sessions, DCC deployment
the
processes, and intelligence processes. Note that the interviews and focus groups aimed at
answering process   evaluation questions may also be used to address impact evaluation
questions (as described     above), rather than being conducted with the same participants on
separate occasions. We  expect the interviews and focus groups to be extensive, with
representatives from all police workgroups involved in the implementation of each TRM
workstream. We will evaluate each TRM workstream implementation against questions that
under
include but are not limited to:

•  Has  the  TRM  been  implemented  as  intended?  This question  is important  to
understand whether al  workstreams of the TRM were implemented, when and how
well in accordance to guidelines and principles. Within this broad question are more
specific questions such as:

12
Released

o  Have the workstreams and components  been  implemented in  the  expected
timeframes?
o  How wel  have the workstreams and components been implemented?
1982
o  To what extent have the workstreams and components been utilised?
By            whom, and in which districts?
o  How well have processes been supporting implementation of the
workstreams     and their components?
Act
o  Have the workstreams and components reached the people they were intended
to?
o  How adequate is capability and capacity for frontline staff?
(e.g.        workforce and training)
o  How adequate, accurate, and timely is communication around
the   workstreams and components?

•  Has the TRM been implemented differently in different districts? This
question is important to understand whether certain ways of implementing TRM
appear to be more effective, and in explaining any district differences observed in
the impact evaluation.

Information
•  What factors have facilitated or prevented implementation? This question is
important to understand what could be done to help the TRM to be more effective.
Within this broad question are more specific questions such as:
o  What barriers to implementing and delivering the
workstreams and components can be identified? How
have/can these been addressed or overcome?
o  Have any external factors influenced the delivery or implementation of
Official
the TRM?
o  Has the TRM been supported by the organisation in order to be effective?

the
Performance Monitoring
Performance monitoring wil  help ensure that we continue to track against outcomes and
inform continuous improvement efforts across our tactical response capabilities. This wil
form a part of our wider operational performance monitoring, including against our          three
goals of safe homes, roads and communities, and our vision of being the safest country.
under
The focus on the adequacy of our frontline capability and capacity wil  continue beyond the
phases of PoC across the country. Continuing to monitor   our tactical response and frontline
and community safety as part of performance is key to explaining and understanding any
observable changes, what might be driving them, and therefore, what strategic levers we
could pull to have a positive impact.
The deployment data col ection and analysis workstream of the evaluation will overlap and

13
Released

transition into ongoing performance monitoring across our tactical response capabilities.
See Appendix B for further detail of our initial timelines for deriving a shortlist of measures
1982
for development, primarily based on usefulness and feasibility. The shortlisting process is
also intended to inform future developments, for areas that are much harder to measure,
but vital for understanding the impacts of policing efforts on our desirable outcomes.
Act
Information
Official
the
under

14
Released

TACTICAL RESPONSE MODEL | EVALUATION APPROACH

   S
1982
E
FRONTLINE
COMMUNITIES

OM
  C
Act
FRONTLINE STAFF ARE
FRONTLINE STAFF & THEIR
COMMUNITIES
COMMUNITIES
COMMUNITIES HAVE
OUT
SAFER IN THEIR DAY-TO-
WHĀNAU FEEL SAFER
ARE SAFER
FEEL SAFER
TRUST & CONFIDENCE
DAY DUTIES

Measures
Measures

Reaction to training, perceived value, and applicability of
Self-reported Safety and Confidence

training, and impacts on decision making

Approach
Approach

Analysis of training forms, surveys, observation of
Comparative analysis of staff/whānau perceptions of

tline training
debriefs/reflections
safety and confidence related to enhanced frontline
Measures
Measures
Measures
POC/non-POC comparisons of performance
training in POC/non-POC districts before and/or after the
Public and Offender Injuries
Self-reported Feelings of Safety
Trust and confidence
fron
in decision- making scenario
introduction of TRM
During Deployment

Information
Sources

Reported Crime and

Training Schedules; QID and HR Data; Lesson Plans; End of Training
Sources
Victimizations

Forms; Survey; Training Assessments; Scenario-Based Assessments;
Surveys; Focus Groups and Interviews

Enhanced

Focus Groups; Interviews

Measures
Measures

Staff injuries, Deployment/event outcomes, Deployment decision-
s to
Self-reported Safety and Confidence

making, Deployment response times, Targeting high-risk offenders
y

Official

lit
Approach
Approach
Approach
Approach
acces bi
Approach
Comparative analysis of staff/whānau perceptions of
Matched comparison of
Analysis of community feelings
Comparative analysis of community
Comparing deployment data, decisions and outcomes, and injury
safety and confidence related to specialist capability in
deployment outcomes in
of safety and perception of
feelings of safety and perception of
t capa
rates in POC/non-POC districts before and after TRM. Deployment
POC/non-POC districts before and/or after the
POC/non-POC districts
police in POC districts
police in POC districts
lis
briefing and debriefing analysis within POC districts
introduction of TRM
before and/or after the

the

cia

introduction of TRM

Sources
spe
Sources

proving frontline
CARD; NIA; DAS; EOD Forms; AOS Deployment Forms; TOR; TPT

Surveys; Focus Groups and Interviews

Briefings and Intel Packs; Focus Groups and Interviews

Im

Measures

Measures
Sources
Sources
Sources
under
Self-reported Safety and Confidence
Focus Groups/Existing Community
Assaults on police, near misses, staff turnover and leave, use of
CARD; NIA; EOD Forms; TOR;
Focus Groups/Existing

Forums;
force
NZCVS and other surveys
Community Forums; NZCVS
nology
Approach

and other surveys
Approach
Comparative analysis of staff/whānau perceptions of
tech
safety and confidence related  to strengthened risk-based
Matched comparison of districts before and after introduction of
ing risk-based t &
deployment/tech in POC/non-POC districts before
TRM
en
and/or after the introduction of TRM

gthen
Sources
loym
Sources
Administrative and Workforce Data;
Surveys; Focus Groups and Interviews
Stren dep
Focus Groups and Interviews; Surveys
Released

Evaluation Management
1982
A high-level overview of the evaluation plan and data col ection periods is provided in
the flow diagram figure TRM Evaluation Plan below. Effective project management is
critical to a successful evaluation and specific details will be developed systematically
Act
on approval of the high-level proposal.
Project Resourcing
The evaluation project requires the design, coordination, undertaking, and analysis of
a number of tasks; as such, human resourcing needs are quite high.
Most of these positions wil  be held by staff at EBPC, however there is the option to
outsource some areas of the evaluation such as the training implementation and
process evaluation. Other roles such as transcription services will need to be recruited
from outside of EBPC. District coordinators will be required for communication and
continuity from each PoC district.
Information
A full break down of personnel, responsibilities and tasks wil  be developed once the
evaluation  plan  has  been  finalised.  Note:  Some  of  these  roles  may  be  held
simultaneously or periodically by a single individual as required by the project tasks.
Other requirements include a large amount of travel (TBC), and software needs for
analysis and project management.  Official
Milestone Reporting Periods
Once the expected start dates for the TRM are announced, more clarity can be provided
around interim reporting periods. There wil  be ongoing monitoring and regular
the
reporting milestones throughout the evaluation period. Specific workstreams may have
varying reporting frequencies.
Deliverables
Deliverables wil  be due in draft form at each milestone reporting period. At the
commencement of the evaluation period, work wil  commence on producing the final
under
evaluation document for the entire evaluation period.

Released

Risk register
Specific risks and limitations of evaluation designs have been outlined in the
1982
attached appendix. However, a risk register wil be developed and maintained
once the evaluation plan has been finalised, which wil also encompass
overarching project.
Act

Information

Official

the

under

17
Released

TRM | Evaluation
Plan
Community/Whānau
Frontline
1982
Training Workstream
Tactical and Intel Workstream
Act
Timeline
Community Workstream
Whānau
TRM Wide Workstream
Does the TRM training make frontline officers safer than when
they did not have the training?
Does the deployment of Tactical Dog Teams make
frontline and dog handlers safer?
Do frontline officers who have received the TRM Training make
safer decisions than officers who have not received the TRM
Does the deployment of Tactical Dog Teams make
Training?
frontline and dog handlers feel safer and more confident?
Does the TRM training make frontline officers feel safer and more
Does the deployment of Tactical Prevention Teams make
confident than officers who did not receive the TRM training?
staff safer?
Reaction survey Designed
Interviews and focus groups
Baseline period
Baseline Survey in field (PoC
Trainer focus group questions
(FGs) method and qestions
Deployment Data Collection
and non-PoC districts)
developed
designed
and Anlysis, (DDA)
Reaction survey
Month 1
Information
Observation/recording
DDA
Reaction survey
Month 2
Focus Groups with existing
Observation/recording
district advisory groups x2
Coaches focus group
DDA
Official
Initial case study period
Month 3
Reaction Survey
DDA
Observation/Recording
All districts (1-2 weeks per
dictrict)
the
Month 4
Focus Groups with existing
district advisory groups x2
DDA
Second case study period
Month 5
under
All district (1-2 weeks per
DDA
district)
Computer based training
scenarios with controls
Month 6
All district impact Focus
Whānau impact Focus
Trainees focus group
Groups
Groups/Interviews
DDA
Coaches focus group
Final case study period (impact
Post evaluation survey in field
focus)
Post PoC phase - Final
(PoC and non-PoC districts)
reporting
All district (1-2 weeks per district)
Released

The table below lists potential data sources/resources we will need for the process evaluation.  Note, it does not include any of the data sources the team currently has access to (such as CARD, NIA etc), nor the bespoke
1982
data sources to be created for the evaluation (such as surveys or interviews). It is limited to data the project has access to, and EBPC does not. Data sources highlighted are indicative, and it is anticipated that specific
requests will be sent when more detailed evaluation planning is undertaken.
  Data sources/resources
Training
Dogs &
Tactical intel
Tactical intelligence
Tactical intel
24/7 DCC
SSPOI
SearchX tool
Tactical deployment
Communication to the
TPT
staff
training
products
support
model
public
Act
QID’s of all staff in POC Districts,
X
X
X
X

X

along with role titles, whether
they wil  be trained, whether
they are part of a tactical dog
team, and whether they are part
of a TPT team.
QID’s for frontline staff in control
X

districts
Training schedules for each
X

X

district, including what is going
to be taught when where, and
who is attending
Detailed lesson plans
X

X

Staff call signs for TPTs and TDTs
X

Information
Observers may be used during
X
X
X
X
X
X
X
X
X

training, briefing/debriefing, field
observations.
Ful  TRM communication plan
X
X
X

X
Staff Roster Information for

X
X

X

selected roles
Deployment briefings/packs

X

X
X
X

X

including tactical intel igence
products
DAS data for all staff in district.

X

X

X
X
Official
End of deployment forms

X

X
X

X

AOS Deployment forms for POC

X
X
X

X

Districts and control districts
Tactical intel igence product

X

the
templates
Policy and process documents

X
X
X
X
X

Intel igence product feedback

X

forms
Table 3: potential data sources/resources needed for the process evaluation

under

15
Released

Appendix A: Detailed outline of evaluation
1982
Evaluation question: Are frontline staff safer in their day-to-day duties as a
result of the TRM?

Evaluation design
Details of evaluation design
Limitations and risks Act
Matched comparison
This option proposes to
We may not be able to control for al
districts pre and post
compare frontline safety in
possible causes of differences
introduction of TRM.
districts with the TRM and
between the TRM and non-TRM
This wil  be used to
districts without the TRM that
districts to be able to say for sure
that any difference between the TRM
identify and attribute
are matched as closely as
and non-TRM districts is attributable
differences in frontline
possible to the TRM districts, at  to the TRM.
safety to the TRM.
two time points: before and

after TRM is introduced.
Some aspects of TRM (intelligence

staff, intelligence products and
Potential measures: The
SSPOI) are being rol ed out to al
evaluation proposes to use a
districts. We will only be able to
attribute any difference between the
Information
range of proxy indicators of
TRM and non-TRM districts to the
safety, such as the number of
workstreams of TRM that are only in
assaults on police, number of
the POC districts and not in the
near misses, staff turnover and
non-TRM districts.
leave, and use of force reports.

Without establishing an effective
baseline, gauging changes resultant
Potential analytical approach:
from TRM components will be
Official
Statistical analysis will be
difficult, and may require a design
undertaken to see whether there  where we only survey after TRM is
is a significant difference
implemented and ask officers to
between TRM and non TRM
report their change in feelings of
the
districts’ change in the above
safety and confidence.
measures.
  Potential data sources:
Deployment data, administrative
data on assaults on police,
under injuries, staffing. Large scale
frontline staff survey on feelings
of safety and confidence. Focus
groups and interviews.

20
Released

Evaluation question: Do frontline staff feel safer and more confident in their
day-to-day duties as a result of the TRM?
1982

Evaluation design
Details of evaluation design
Limitations and risks
Matched comparison
This option proposes to compare  We may not be able to control for al
districts pre and post
frontline feelings of safety and
possible causes of differences
Act
introduction of TRM.
confidence in districts with the
between the TRM and non-TRM
This wil  be used to
TRM and districts without the
districts to be able to say for sure
identify and attribute
TRM that are matched as closely  that any difference between TRM
differences in frontline  as possible to the TRM districts,
and non-TRM districts is attributable
feelings of safety to
at two time points: before and
to the TRM.
the TRM.
after TRM is introduced. We then
compare the change in feelings
Some aspects of TRM (intelligence
of safety and confidence (from
staff, intelligence products and
before to after) between the TRM  SSPOI) are being rol ed out to al
districts and matched districts.
districts. We will only be able to

attribute any difference between the
Potential measures: The
TRM and non-TRM districts to the
Information
evaluation wil  measure self-
workstreams of TRM that are only in
reported feelings of safety and
the POC districts and not in the
confidence in relation to a range  non- TRM districts.
of tactical scenarios. Questions
about specific tactical scenarios
This design relies on a high response
wil  be designed to enable us to
rate to the survey. Having a high
understand which of the various  response rate is important so that we
Official
components of TRM are most
can reliably detect any difference
like to be contributing to any
between the groups, even if it is
difference.
small.

the
To help to understand how
This design relies on the survey
specific components of TRM are  having respondents are
contributing to officers’ feelings
representative of frontline. This is
of safety and confidence we wil
important so that we can generalise
include measures such as
the results of the survey to the
whether they have received the
frontline population as a whole.
under frontline training, whether they
were accompanied by TPT (in the  Without establishing an effective
scenarios considered above), and  baseline, gauging changes resultant
whether they received any of the  from the TRM will be difficult.
new intelligence products (in the
above scenarios). Frontline staff
will also be asked through focus

21
Released

groups and interviews as to the
reliability, timeliness,
1982
appropriateness, and whether
they feel safer as a result of the
intelligence they receive.

Act
Potential analytical approach:
Statistical analysis will be
undertaken to see whether there
is a significant difference
between TRM and non TRM
districts’ change in the above
measures, control ing for pre-
existing differences between the
TRM and non-TRM districts. The
statistical analysis will also
examine whether, control ing for
Information
district, officers feel safer and
more confident if they have
received the training, and been
accompanied by TPT or received
any of the new intelligence
products.
Potential data sources:
Official
Administrative data on assaults
on police, injuries, staffing. Large
scale frontline staff survey on
the
feelings of safety and confidence.
Focus groups and interviews.
under

22
Released

Evaluation question: Does the TRM improve the wellbeing of the whānau of frontline
staff?
1982

Evaluation design
Details of evaluation design
Limitations and risks
Matched comparison
This option proposes to gain
We may not be able to control for
districts post
insight into and compare
al  possible causes of differences
introduction of TRM.
whānau’s views, opinions and
between the TRM and non-TRM
Act

feelings around:
districts to be able to say for sure
Rationale: We need to
•  If the TRM has increased
that any difference between TRM
establish what would
the actual and/or
and non-TRM districts is
have happened but for
perceived safety of their
attributable to the TRM.
the TRM. To do this we
loved ones, and

need to compare two
•  if the TRM has improved
Some aspects of TRM (intelligence
groups where the only
the wellbeing of frontline  staff, intelligence products and
difference between
staff thereby improving
SSPOI) are being rol ed out to al
them is the intervention
the wellbeing of their
districts. We will only be able to
and to establish a
whānau.
attribute any difference between
baseline of how the two
the TRM and non-TRM districts to
groups already differed  Potential measures: The
the workstreams of TRM that are
before the TRM.
evaluation proposes to use
only in the POC districts and not
Information
However, with this
measures of whānau’s views,
in the non-TRM districts.
design we ask people
opinions, and feelings such as

to report their own
self-reported change in feelings
It would be preferable to have a 2,
baseline, which is less
of officer safety and perceptions
3- or 4-week lead time to ensure we
robust.
of the TRM (exact questions tbc).  can book enough people.
These wil  be created from
There is a risk of workshop fatigue,
synthesis and theming of focus
and we would potential y only hold
group outputs
workshops in the post-trial
Official

assessment phase, preferring
Views, opinions, and stories
interviews or group interviews
gathered in focus groups will be
while the trial is in-flight.
synthesised and themed.

the
Potential data sources:
•  Focus groups and
interviews with whānau
members
•  Interviews and visits with
whānau members
under

23
Released

Evaluation Question: Are communities safer as a result of the TRM?

Evaluation design
Details of evaluation design
Limitations and risks
1982
Matched comparison
This option proposes to compare  We may not be able to control for
districts pre and post
community safety in districts with  al  possible causes of differences
introduction of TRM.
the TRM and districts without the  between the TRM and non-TRM
Act

TRM that are matched as closely
districts to be able to say for sure
This wil  be used to
as possible to the TRM districts, at  that any difference between TRM
identify and attribute
two time points: before and after  and non-TRM districts are
differences in
TRM is introduced. We then
attributable to the TRM.
community safety to
compare the change in

the TRM.
community safety (from before to  Some aspects of TRM (intelligence
after) between the TRM districts
staff, intelligence products and
and matched districts.
SSPOI) are being rol ed out to al

districts. We will only be able to
Potential measures: The
attribute any difference between
evaluation proposes to use
the TRM and non-TRM districts to
measures of community safety
the workstreams of TRM that are
Information
such as the number of crime
only in the POC districts and not
types and crime harm committed  in the non-TRM districts.
by the high-risk offenders
targeted by the TRM and
measuring public and offender
injuries during deployment.

Official
Potential analytical approach:
Statistical analysis will be
undertaken to see whether there
is a significant difference between
the
TRM and non TRM districts’
change in community safety,
controlling for pre-existing
differences between the TRM and
non-TRM districts.

under Potential data sources: Reported
crime (NIA), NZCVS victimisations,
CARD, EOD Forms, and TORs.

24
Released

Appendix B: Performance
monitoring timeline of activities

1982

Week 0
Week 1
Week 2
Week 3
Week 4
Act
Week commencing 4 Oct
Week commencing 1 Nov
Week commencing 8 Nov
Week commencing 15 Nov
Week commencing 22 Nov

-  Agree the Evaluation and
-  Receive complete set of TRM
-  Complete review of documents
-  Prioritise long-list of measures using
-  Test alignment of the short-list
Performance Monitoring Framework
documents.
provided.
agreed prioritisation criteria.
measures to the OPF and develop a
(see Appendix One for a suggested

plan for integration.
draft)
-  Agree the prioritisation criteria.
-  Collate all existing measures.
-  Agree short-list of measures mapped

to the Evaluation and Performance
-  Agree workplan to develop short-list
-  Agree the TRM value chain.
-  Collate measures of performance
-  Agree drivers and additional measures
Monitoring Framework.
measures that do not currently exist.
derived from existing material to

of performance (if any).

-  Test short-list of measures with the
date(see Appendix Two).

-  Governance approval of TRM
TRM leads.
Information
-  Collate long-list of measures.
Performance Framework and short-list

measures.
-  Identify data sources, owners, and
frequency of collection for each
short-list measure.

Official
the
under
Released

Document Outline