State of the art metrics for
nursing: a rapid appraisal
Peter Griiths with Simon Jones, Jill Maben and Trevor Murrells
2008
State of the art metrics for nursing
Acknowledgements
This work was commissioned and supported by the Department of Health in England as
part of the Policy Research Programme which provides funding to the National Nursing
Research Unit. The views expressed are those of the authors, not of the Department of
Health. We thank the members of the Nursing Outcome Measures task and inish group
chaired by Professor Anne Marie Raferty and convened by Professor Dame Christine
Beasley, the Chief Nursing Oicer for England. These experts’ insights and experiences
informed us as we undertook this review to support their work. The work presented here
is our appraisal of the literature and is not intended to represent the full breadth of work
undertaken by that group. We particularly thank members of the group and others who
provided support and comments, and Janice Sigsworth, the Deputy Chief Nursing Oicer
who was directly responsible for commissioning this report.
Contact address for further information:
National Nursing Research Unit
King’s College London
James Clerk Maxwell Building
57 Waterloo Road
London SE1 8WA
nnru@kcl.ac.uk
http://www.kcl.ac.uk/schools/nursing/nnru
i
State of the art metrics for nursing
ii
State of the art metrics for nursing
Table of contents
Acknowledgments
i
Foreword
v
Overview
1
Introduction
2
Possible indicators
6
What makes a good indicator?
10
Evidence base: associations between nursing and outcomes
14
State of the art
17
Conclusion and recommendations
23
References
26
Appendix 1: Core data sources
31
Appendix 2: Indicators identiied
34
iii
State of the art metrics for nursing
iv
State of the art metrics for nursing
Foreword
The search for outcome measures for nursing has a long and distinguished history tracking
back to Florence Nightingale herself. Her epidemiological approach to outcomes research
and measurement has attracted admiration from statisticians and epidemiologists alike.
Despite this prestigious pedigree, progress on reining metrics for the outcomes of nursing
care has been slow. The prioritisation of quality within the Next Stage Review of the
National Health Service in England and the commitment to hold trusts accountable for
and to reward quality of care promises not only a renewed but relentless focus on quality of
care. This report reviews the status of the evidence base on nursing metrics and provides a
road map and set of recommendations to take nursing forward. I am grateful to members of
the task and inish group who helped to identify candidate metrics in key domains of care
and provided examples of good practice from their organisations. As Professor Griiths
and his team demonstrate, while there is much to be commended in such practice there
is still much more to do. Developing metrics is only the irst step in building a robust
infrastructure for implementation and fully integrating nursing into the governance and
management of the NHS at all levels. Such metrics can then enable the public to make
informed decisions about their care based on criteria which matter to them as well as to
managers and clinicians. We have a unique and unprecedented opportunity to make the
quality of nursing care count. This report is an important step in that process.
Professor Anne Marie Raferty
Dean and Head of the Florence Nightingale School of Nursing & Midwifery at King’s
College London
Chair of the Nursing Outcomes task and inish group
v
State of the art metrics for nursing
vi
State of the art metrics for nursing
Overview
•
The Next Stage Review commits to quality measurement that relects the compassion,
safety and efectiveness of nursing care. As such measurement provides both a
challenge and an opportunity for the profession, this report reviews ‘state of the art’
nursing quality measurement.
•
Many possible indicators and existing indicator sets measure nursing’s contributions.
Among the most widely used indicators are safety measures such as failure to rescue (death
among patients with treatable complications), falls, healthcare associated infection and
pressure ulcers. Neither efectiveness (positive contributions to well-being) nor compassion
(elements of patient experience) are strongly represented in the existing measures.
•
We did not set out to focus speciically on acute general inpatient nursing care, but
examples from this setting dominate both indicators and evidence reviewed because
there has been more development in these areas. Many themes apply equally to other
areas even if speciic indicators difer, and the lessons learned in acute care can assist in
developing indicators for all specialties and care settings.
•
To be useful, indicators must be measurable with available data at reasonable cost.
There must be evidence that the quality or quantity of nursing substantially contributes
to changes measured by the indicator. The indicator must be recognised as important
(by the public, managers and nurses) and nursing’s contribution must be recognised
(by nurses and others).
•
Nurses must have responsibility for actions leading to outcomes in terms of legitimate
authority, self-perception and sphere of practice. Measures should be chosen to
minimise the risk of gaming, where improving performance on speciic indicators
detracts from overall improvement. Measures focussing on the performance of care
(process) rather than outcome are most vulnerable to gaming.
•
Not all existing indicators meet these requirements, and considerable work will be
needed to develop practical, valid and useful indicators. Strong evidence supports an
association between nurse staing levels and mortality, but mortality is determined by
many causes and is not likely to be a useful quality measure for nursing.
•
This report identiies ‘best bets’ for indicator development, including measures of
safety, efectiveness and compassion. Health care providers should form a quality
coalition, facilitated by stakeholders includinf the Royal College of Nursing and The
NHS Confederation, to share best practices and to move toward standard measurement
of important nursing indicators. A programme of indicator development must include
development of technical speciications for indicators, research to validate them and,
crucially, patient involvement in identifying metrics for compassion.
•
The development of metrics and the establishment of a National Quality Board and
local quality observatories present a substantial opportunity for nursing to equip
itself with the tools needed to deliver excellent care, and these initiatives should be
welcomed within the profession.
1
State of the art metrics for nursing
Introduction
This work was commissioned to inform the Nursing Outcome Measures task and inish
group chaired by Professor Anne Marie Raferty and convened by the Chief Nursing
Oicer for England as part of the nursing contribution to Lord Darzi’s Next Stage Review
of the National Health Service. The group was tasked with identifying mechanisms for
giving nurses tools, training and support to improve quality of care across the country,
including:
•
Evidence-based metrics to measure nurse-delivered outcomes and patient experiences;
•
National publication of performance data to identify best-practice examples and help
nurses benchmark and improve their performance; and
•
“Ward-to-board” accountability for the quality of nursing care.
The Next Stage Review makes a commitment to develop an indicative set of metrics for
nursing1 that comprises of indicators of quality relecting the issues of safety, efectiveness
and compassion identiied by the task and inish group. This work will support a wider
NHS initiative to establish regional quality observatories and a National Quality Board
that will oversee the development of a quality measurement framework for all clinical
services.
Indicators serve to foster understanding of a system and how it can be improved, and
to monitor performance against agreed standards or benchmarks. Crucially, indicators
provide a mechanism by which care providers can be accountable for the quality of their
nursing services. Accountability for nursing quality exists at many levels, from the point of
care (where individual nurses are accountable to clinical managers and patients) to senior
management, commissioners and beyond. While “ward–to-board” accountability for care
is frequently referred to, a still wider view of accountability includes both the public and
policy-makers, including health service users, the general public and funders of care2.
A measuring system is needed, with a set of indicators that can:
•
quantify trends and characteristics;
•
describe performance in achieving health service goals (in this case, elements to which
nursing strongly contributes); and
•
provide information to improve nursing care.
Nursing-sensitive indicators
The group was tasked with inding measures of “nurse-delivered outcomes and patient
experience”. We take this to mean measures that directly relect nursing’s end results
in terms of impact upon patients. A large body of work describes quality measurement
systems that focus on multiple aspects of quality in the nursing process3 or on nursing
outcomes in terms of activities completed4. While occasionally the relationship between a
particular process and an outcome is so strongly established that a measure of the process
2
State of the art metrics for nursing
may suice or even be preferable as a proxy indicator, our reference point has been patient
outcome (including experience).
Patient outcomes and experiences vary for many reasons and relect the work of multiple
professions. Indeed, in many cases the greatest determinant of outcome is the patient,
whether because of underlying health status, behaviour or aspects of the wider social
environment. In considering nursing-sensitive outcomes and experience, we must identify
elements of variation that can be attributed largely to nursing care quality. To do this we
must seek evidence of correlation with nursing as well as evidence that this correlation
is a plausible consequence of variation in nursing rather than other factors. Determining
such plausibility requires professional knowledge of possible mechanisms and technical
knowledge, such as research evidence that adjusts for confounding factors.
These types of indicators can be used5:
•
to improve quality in local settings by monitoring and managing performance;
•
to support policy analysis and strategic decision-making, including commissioning,
reimbursement systems and accreditation; and
•
to research the role of nursing care in determining patient safety outcomes by
examining structure-outcome, process-outcome and structure-process-outcome
relationships.
The irst two uses signal that diferent information may be required and useful at
diferent levels. The third purpose of indicators is not the immediate concern of this report,
but such research is necessary to develop indicators it for the other uses.
Context
This initiative comes at a time of widespread public perception that nursing quality is
sometimes poor and lacking in essential elements6-8, as represented in the oft-repeated
accusation of nurses being ‘too posh to wash’. Recent reports on infection outbreaks in UK
hospitals have highlighted situations where underlying issues of nursing care quality were
given a low priority in the face of competing productivity targets9-11.
Concerns about nursing care are not limited to the general public or to the UK. There is
ongoing professional concern that nursing’s contribution to quality health care is underrecognised, leaving nursing services vulnerable to cost-reducing eforts12 ,13. Nursing’s
contribution to quality care is not consistently recognised; an Audit Commission report on
hospital staing variations14 concluded early in this decade that “Unless and until trusts that
spend more [on staing] can demonstrate a clear link with the quality of care that is delivered,
movement towards a more even allocation of resources seems reasonable both for patients and
staf.” (p15) This presumes there is no link to quality unless otherwise demonstrated, yet a
document from the same Audit Commission programme15 states: “It is diicult to … avoid
the conclusion that they [variations in staing] must result in diferences in the quality of care
available to patients in diferent trusts and on diferent wards.” (p3)
3
State of the art metrics for nursing
While varying nursing quality is clearly a concern, tangible ways to demonstrate
nursing’s contribution to quality care are less clear. In the UK, responses to healthcare
acquired infections may have been instrumental in reasserting nursing’s central and
fundamental role in providing a safe environment for care16. Yet this represents a narrow
view of nursing’s potential contribution to patients’ experiences, health and well-being.
A burgeoning number of recent studies explore the impact of variation in the quality and
quantity of nursing care on a wide range of outcomes and experiences17. An increasing
number of indicator sets identiied as nurse-sensitive5 are used by national and local
governance or quality improvement programmes. The task and inish group examined
difering systems used in a variety of NHS organisations.
By bringing the nursing contribution to the fore, the Next Stage Review and the
resulting commitment to measure the compassion, safety and efectiveness of nursing
care provide a challenge and an opportunity for the profession. In this light it is timely to
explore how nursing might demonstrate its contribution to quality outcomes and patient
experiences, as well as how quality indicators can hold all service providers accountable.
Developments must build on existing evidence and initiatives for consistency across
settings and to ensure that best practice is used.
This paper explores potential nursing-sensitive indicators identiied from published
literature and indicator sets currently in use. The requirements of a good set of indicators
for nursing are explored and evidence for indicators’ validity is considered through
an examination of evidence linking nursing contributions and patient outcomes. The
conclusion assesses the current state of the art in nursing-sensitive indicators.
We did not set out to focus on acute general inpatient nursing care, but both the
indicators and the evidence reviewed are dominated by examples from this setting because
there has been more indicator development in these areas. While the state of the art may
be more advanced in acute care, many themes apply equally to other areas even though
speciic indicators may difer. Certainly the lessons learned in acute care can usefully
underpin development of indicators for all specialties and care settings.
4
State of the art metrics for nursing
Box 1. Introduction: summary
•
Evidence abounds of public and professional concerns that nursing care quality is
variable and sometimes poor.
•
Nursing care quality has often failed to receive high level-attention in the face of
competing productivity targets.
•
There is a demand for measures whereby nursing can demonstrate and be held to
account for its contribution from point of care to the board room. Such measures
also are useful to all sectors of society beneiting from and making policy for health
care services, ranging from the public to politicians.
•
The Next Stage Review and resulting commitment to quality measurement
relecting nursing care’s efectiveness, safety and compassion provides both a
challenge and an opportunity for the profession.
•
This paper explores potential nurse-sensitive indicators identiied from published
literature and sets of indicators currently in use. This evidence base is used to
determine the current ‘state of the art’ of metrics for nursing.
5
State of the art metrics for nursing
Possible indicators
A wide range of potentially measurable indicators of nursing care quality can be
indentiied from nurse-sensitive outcomes. These outcomes are aspects of patient
experience, behaviour or health status (patient outcomes) that are determined in
whole or part by nursing care received and variations in its quality or quantity. The
precise construct is variously deined, and while most deinitions tend to focus upon
outcomes as results of speciic nursing interventions18 ,19, others have emphasised system
characteristics20 such as team functioning, staing levels and skill mix as important
determinants of outcome.
Quality indicators also can derive from known or widely presumed links between nursesensitive outcomes and nursing interventions or structural characteristics. For example, use
of a nutritional risk assessment might be identiied as an indicator of quality because it is
identiied as a nursing intervention leading to improved outcomes (improved nutrition).
Similarly, workforce variables such as staf satisfaction or skill mix might be used as
indicators because of such variables’ known or presumed relationships with important
patient outcomes.
Data sources used
A number of sources yielded possible indicators for nursing, including Doran’s review of
nurse-sensitive indicators21 and recent systematic reviews linking the ward environment,
nurse staing and patient outcomes22-25. Various indicator systems were selected on
the basis of their strong potential to give a high coverage of nurse-sensitive indicators,
and on the basis of their prominence and advocacy within high-proile bodies such as
the American Nurses Association, the UK’s NHS and the US Joint Commission for
the Accreditation of Healthcare Organisations (JCAHO). This coverage was generally
identiied because the indicator set was proposed as nurse–sensitive. Some relevant
indicator sets were developed for care settings where nursing takes a signiicant active lead
(e.g. community and home healthcare), even where the set was not proposed for nursing
per se. Appendix 1 lists key sources used.
To this set were added indicators identiied using database and web searches of terms
such as nurse-sensitive indicators and metrics. This yielded links to additional indicator
sets and locally developed ‘dashboards’ of indicators.
The intention was not to achieve a comprehensive list, but to give an accurate overview.
The authors explored these issues within the task and inish group and gleaned further
examples from members. We stopped searching when new searches failed to identify new
domains and no new individual indicators emerged that were substantively diferent from
those already identiied. Few indicators were identiied solely on our general web search,
suggesting that we have successfully identiied the bulk of the available content although
alternative speciications of the indicators we have identiied might exist. We have not
6
State of the art metrics for nursing
considered systems such as the Nursing Outcomes Classiication System26 because they
intend to provide a comprehensive taxonomy for recording the outcomes of speciic
interventions rather than to serve as general indicators of quality.
Indicators identiied
Appendix 2 lists the broad range of indicators identiied. Depending on the level of
precision of deinition, a list of nursing quality indicators could run into the hundreds. The
majority of sources identiied were speciic to acute general hospital care, although many
indicators could apply across other settings in adapted forms.
The level of deinition ofered varied considerably between sources. Reviews necessarily
were less precise in specifying outcomes and data-collection protocols. Variables generally
were clearly deined, although some (e.g. ICN*) deined variables in very general terms.
Existing indicator sets were generally the product of an extensive development process
and ofered precise speciications for data (e.g. relevant ICD codes, exclusions and riskadjustment models such as NQF). Other sources such as the AUKUH listed speciic
indicators, but ofered little in terms of data speciication or operational deinitions while
the ICN indicated only broad areas. Essence of Care benchmarks conversely ofered
considerable conceptual detail but little detail on measurement. The broad topics most
speciic to nursing are incorporated here.
A range of patient outcomes was identiied including aspects of knowledge, function
(including instrumental activities of daily living and continence), nutrition, experience
(including communication, satisfaction and complaints), preventative care such as
vaccinations, safety outcomes (e.g. failure to rescue, falls, infections, medication errors,
mortality, pressure ulcers), symptoms such as pain and dyspnoea and utilisation outcomes
such as hospital stay and unplanned admissions. Also identiied as possible indicators
were processes directly linked to these outcomes (e.g. pain assessment, risk assessment)
or relating to general aspects of quality such as planning and care coordination. Nursing
workforce characteristics featured heavily as well: possible indicators included staing
levels, skill mix (including qualiied nurses’ levels of educational preparation), team
expertise, staf turnover rates and indicators of team functioning, such as interprofessional
relations and perceptions of practice environment quality. Staf outcomes such as wellbeing and injury rates also were identiied.
Table 1 lists indicators that four or more sources identiied, and the assignment given
to them by the task and inish group in terms of safety, efectiveness and compassion**.
Similar indicators are grouped for brevity, even though sources’ precise deinitions
* See Appendix 1 for deinition of the acronyms used to refer to indicator sets
** Safety refers to adverse efects of care, efectiveness refers to positive beneits and compassion refers
to aspects of patient experience such as perceived dignity, respect and quality of communication.
Appendix 2 gives a fuller list of indicators identiied; these are classiied into topic areas using an earlier
schema focussing on content devised before the task and inish group devised its broad classiication
7
State of the art metrics for nursing
and speciications of indicators may difer. While the number of sources that identify a
particular indicator hint at its level of support, this should not be taken as a literal measure
because our sample of sources was somewhat arbitrary. In particular, we discuss below why
number of sources does not indicate evidence strength.
Table 1. Most frequently identiied indicators
Indicator
Area
Number of
sources
Pressure ulcer
Safety
11
Failure to rescue
Safety
9
Staing levels
Efectiveness
9
Falls
Safety
8
Health care associated infection: pneumonia
Safety
8
Staf satisfaction and well-being
Efectiveness
7
Health care associated infection: urinary tract infection
Safety
6
Staing, skill mix
Efectiveness
6
Medication administration errors
Safety
5
Mortality
Safety
5
Practice environment/perceived quality
Efectiveness
5
Satisfaction with (nursing) care
Compassion
5
Sickness rates
Efectiveness
5
Smoking advice
Efectiveness
5
Staing bank or agency use
Efectiveness
5
Communication
Compassion
4
Staf experience, knowledge, skills and expertise
Efectiveness
4
Health care associated infection: surgical wound
Safety
4
Instrumental activities of daily living and self–care
Efectiveness
4
Perception of adequate staing
Safety
4
Use of restraints
Compassion
4
Adverse events dominate the ield of nurse-sensitive indicators5. Despite our inclusion
of broader-based reviews such as Doran’s21 this remains the case here and is relected by
the high proportion of indicators of patient safety, particularly in relation to outcomes*.
Most efectiveness indicators stem from a decision to interpret structural staing outcomes
* The relative balance between the types of indicators is in part a product of the level at which they
are grouped. For example, the OASIS indicators include a detailed list of instrumental activities of
daily living which can be considered as several indicators or, as we did, one (IADL).
8
State of the art metrics for nursing
or other measures of workforce experience as indicating an ‘efective’ workforce under
this schema. Such indicators could equally well have been identiied as safety measures,
since much of the available evidence relates to the link between staing and patient safety.
Interpretation of items such as satisfaction under ‘compassion’ rests upon the potential
of the largely patient reported measures (including satisfaction measures) to incorporate
elements of patient experience that relect upon this dimension.
Even allowing for their difering natures and sources, the lack of overlap Savitz5 noted
remains. Indeed, agreement between sources is exaggerated by our grouping of similar
indicators. However, there does seem to be a degree of consistency in identifying as
indicators in acute-care settings:
•
failure to rescue (death among patients with treatable complications27);
•
falls (sometimes deined in terms of resulting injury);
•
health care associated infection (although the precise infections/speciication vary
considerably);
•
pressure ulcers; and
•
staing levels.
Nonetheless, the range of candidate indicators and lack of consistency in their content
and precise deinition make it essential to identify desirable indicator characteristics.
Box 2. Possible indicators: summary
•
Many possible indicators and indicator sets exist, mainly but not exclusively in
acute care. However, little consistency exists between indicator sets.
•
The most widely supported outcome indicators include failure to rescue (death
among patients with treatable complications), falls, healthcare associated infection
and pressure ulcers.
•
Other commonly advocated indicators were patient experience, including
communication with staf and support in activities in daily living; medication
administration errors; the practice environment’s perceived quality; and workforce
aspects including staing levels, skill mix and measures of well-being or satisfaction.
9
State of the art metrics for nursing
What makes a good indicator?
The National Quality Forum require indicators to be:important, scientiically sound
useable and feasible28.
Importance is deined in terms of
•
national goals
•
impact of outcomes on individual service users
•
the societal burden of disease
•
availability of strategies for improvement,
•
substantial variation in quality, or
•
quality which is consistently substandard.
Scientiic criteria include
•
precise speciicationof the indicator
•
reliability and validity of measures and
•
adequacy of risk adjustment.
Usability is the extent to which intended audiences can understand results and are
likely to ind them useful in decision-making while feasibility relates to the ability to
obtain quality data in a timely manner with a demand on resources that is proportionate
to beneits.These criteria are generally relected in the NHS Institute for Innovation and
Improvement’s Good Indicator Guide2, which speciies that indicators should describe “as
much about a system as possible in as few points as possible”. (p5)
So nursing indicators must be measurable with available data at reasonable cost, coding
and recording must be consistent and complete and measures must be valid. If an indicator
is to be used to represent the quality of nursing it must be attributable to nursing in a
number of senses, including:
•
evidence of sensitivity to nursing
•
recognition of the phenomenon’s importance
•
recognition as a nursing contribution (owned by nurses and acknowledged by others)
and
•
recognition as nurses’ responsibility in terms of legitimate authority, self-perception and
sphere of practice
Usability requires that there is suicient knowledge to inform action since identiication
of a particular level of achievement is not useful unless a strategy for improvement can be
identiied. Indicators must apply to many patients and types of clinical services if they
are used to represent general quality across specialities. Similarly, variation in a nursingattributable outcome must be substantial if the outcome is used to indicate nursing quality
as opposed to care in general. If an outcome is used to compare diverse populations and
10
State of the art metrics for nursing
clinical settings, it must be possible to adjust for difering baseline risks to ensure fair
comparison. The evidence behind any nursing process or structural indicator also must
establish that it is linked to outcomes29,30.
The choice of indicators must minimise the risk of gaming or perverse incentives.
Collecting data can be seen as unwarrantedly intrusive and burdensome at any
organisational level31 and proper organisational support and infrastructure for data
collection and feedback are essential. To maintain commitment, attention should focus
not just on the initial burden but also on the information’s perceived value. Timely and
informative data must be available to provide behavioural incentives and if necessary
allow remedial action to be taken. The challenge is to turn the data into “actionable
information” 31 so feedback mechanisms and formats such as the increasingly popular
graphics-based ‘dashboards’ and traic light systems are important31,32.
It is essential that actionable information be acted upon. As one member of the task and
inish group noted: “…outputs have to be used and the emerging issues addressed, otherwise
staf very quickly become cynical.”
Perhaps the most diicult balance to strike is between the data collection burden
and the need to ensure that performance on indicators represents broad achievement of
the goals of nursing or processes that deliver those goals. The more indicators that are
collected the greater the data collection burden but the danger of focussing on a few
narrowly deined indicators lies in the creation of perverse incentives33 where maximising
performance on indicators detracts from overall performance in other aspects or changes
performance in relation to the indicator that invalidates it by removing its relationship to
overall quality. Numerous examples of such gaming behaviours have been noted within the
English NHS34. One notable example was the 48-hour target for GP appointments that
led to practices refusing to ofer appointments more than 48 hours in advance34. Indicator
selection must therefore consider the potential for gaming and seek to minimise this.
Essentially the aim is to identify a relatively small number of indicators that still relate
clearly to the multifaceted and somewhat elusive concept of quality nursing care. All
indicator sets are incomplete; the challenge is to identify indicators that are important
in themselves and also strong indicators of overall quality. Bevan and Hood note that
indicators must be selected from a narrowly deined subset of target areas where both data
and measures of suiciently high quality are available34 (see Figure 1). The fundamental
issue that indicators are primarily decided by available measures cannot be avoided.
The limitations of available good measures may point towards using imperfect measures,
even though this may in efect “open a can of worms”34 because such measures provide an
incomplete and inaccurate picture. While pragmatic decisions may inluence the choice
of indicators and imperfect measures may be used in suiciently important areas, such
decisions may create additional opportunities for gaming and perverse incentives.
11
State of the art metrics for nursing
Figure 1. The relationship between the system and indicators (after Bevan & Hood 2006)
Total domain
Subset of interest
Subset with imperfect measures
Subset with good measures
This risk is illustrated by two examples relating to possible nursing indicators. One relates
to the use of routinely collected data for indicators such pressure ulcers or HCAI. Such
secondary diagnoses are notoriously poorly recorded35. While the chances of a patient
developing pressure ulcers may relate to the quality of nursing care, early detection and
proper documentation of pressure ulcers is also a marker of quality care which could lead to
higher rates of recording in good-quality settings than in lower-quality ones. Thus a perverse
incentive might be created for lower levels of surveillance and documentation among highperforming teams. It is certainly hard to see the incentive for poor-performing teams to
increase their reporting. Risk adjustment for pressure ulcers is further problematic and risk
assessments are not routinely conducted, centrally recorded or particularly discrmininating36
so comparisons between units (such as wards or hospitals) are diicult to interpret.
In the face of such problems, measures of process seem an appealing alternative. But
the apparent ease that many process measures ofer in resolving the diicult measurement
issues associated with outcomes may be outweighed by the choice of a process indicator
because it is measurable, but which has doubtful links to patient outcomes or experience.
Process data are rarely routinely available5 (which removes a potential advantage) and
it is hard to capture in routine audits the complex interpersonal care processes involved
in many nursing interventions. Thus indicators such as completed pressure ulcer risk
assessment tools may come to be adopted even though token compliance is possible (for
example, if completion of the form takes priority above identifying need). Furthermore,
the speciic process may be adopted because it is measurable rather than because it is
known to be efective. In the case of formal pressure ulcer risk assessment tools, accuracy
and efectiveness are both uncertain36.
Resolving these issues can never be straightforward and must be a matter of informed
judgement about potential beneits from deploying an imperfect indicator as opposed to no
12
State of the art metrics for nursing
indicator. Evidence of links between nursing and outcome must be further examined, since
without this knowledge any indicator has questionable utility.
Box 3. What makes a good indicator: summary
•
Indicators must be measurable with available data at reasonable cost.
•
There must be evidence of variability associated with nursing and this variability
must be substantial.
•
For process or structure measures, evidence must support links to important
outcomes.
•
The indicator must be recognised as important (by the public, managers and nurses)
and the contribution of nursing must also be recognised by nurses and others.
•
Nurses must have responsibility for actions that lead to the outcome in terms of
legitimate authority, self-perception and sphere of practice.
•
There must be suicient knowledge to inform remedial action.
•
Measures should be chosen to minimise the risk of gaming, where improving
performance on the indicators detracts from overall performance.
•
Measures, especially measures of outcome, generally need to be risk adjusted to
ensure comparability across settings.
13
State of the art metrics for nursing
Evidence base: associations between nursing and outcomes
Kane et al.22 provide the strongest single source of evidence for a link between nursing
and outcomes. This systematic review examines the impact of a general nursing variable,
the quantity of nursing care available, and assesses the extent to which nursing inluences
the indicators. The review included 96 studies linking nurse staing to patient outcomes.
Increased RN staing was associated with lower hospital-related mortality (per additional
full-time equivalent nurse per patient day) in:
•
intensive care units (odds ratio 0.91, 95% conidence interval 0.86–0.96)
•
surgical units (odds ratio 0.84; 95% conidence interval 0.80–0.89)
•
medical patients (odds ratio 0.94; 95% conidence interval 0.94–0.95)
Table 2. variance associated with outcomes (Kane 2007)
Indicator
% variance associated
with nurse staing
Surgical wound infection (surgery)
85%
Unplanned extubation (ICU)
51%
Hospital acquired bloodstream infection (surgery
36%
Cardiac arrest (all groups)
28%
Length of stay (ICU, surgery)
24%
Hospital acquired pneumonia (all groups)
19%
Failure to rescue (surgery)
16%
Respiratory failure (all groups)
6%
4.2%
Mortality
Hospital-wide, 4.2% of variation in mortality is attributable to nursing, assuming
causality. Unsurprisingly, other variables show stronger association with nursing (Table
2), the strongest association being with surgical wound infection. The variation associated
with nurse staing is 85%. This seems implausibly high and raises the question of the
extent to which the associations observed may be a result of confounding, where there is a
strong association between nurse staing and other patient or care quality variables which
are wholly or partly responsible for the outcome observed. There is certainly a possibility
that high nurse staing is associated with other hospital characteristics that are in turn
associated with quality of care (the ‘magnet’ phenomenon37).
Kane takes a cautious epidemiological approach to interpreting causation. Causality
is supported by evidence of a ‘dose-response relationship’ which appears curvilinear
as increased staing at the highest levels yields diminishing returns. The evidence is
consistent across study designs (including the use of risk adjustment) and while diferent
designs give some modiications in estimates, overall conclusions are unchanged. Evidence
14
State of the art metrics for nursing
supports a temporal association as some studies demonstrate that adverse outcomes occur
immediately after periods of low staing, although there is a lower estimate of efect on
failure to rescue in studies assessing this temporal association. There was no consistent
association between nurse staing and patient falls, pressure ulcers or urinary tract
infections, outcomes among the indicators most frequently identiied. Although there is
some evidence to support these21, their appearance on lists of indicators is clearly more
predicated upon a convincing theoretical proposition than the strength of the evidence.
Of the four most prominent outcome indicators identiied (failure to rescue, falls, HCAI
[pneumonia], pressure ulcers), only failure to rescue and HCAI pneumonia are supported
by Kane’s review.
This creates a dilemma. Kane’s evidence of cause would be more convincing if these
outcomes, which are presumed to be the most nurse-sensitive, were most closely associated
with nurse staing. Limitations of available data sets, and in particular poor coding of
secondary diagnoses in the administrative databases used in most of the larger studies,
provide a possible explanation for this inding. However, this is a presumption; even if it
is correct and the outcome is indeed linked to nursing, it appears from Kane’s study that
measures of these outcomes derived from administrative databases are not necessarily valid
indicators of nursing quality. Although most studies reviewed were from North America,
it is unlikely that coding of secondary diagnoses is better in the UK; indeed it is almost
certainly worse. Studies on UK nurse staing and patient outcomes have not utilised these
outcomes38 and in general coding of complications and secondary diagnoses, although
improving, is starting from a weaker baseline than in the US39.
This evidence of a link between nurse staing and patient outcomes supports using
staing and workforce variables as indicators. Evidence is stronger for levels of RN staing
and high skill mix than for total nurse staing23,40. Other evidence tends to support a range
of workforce variables such as job satisfaction and turnover21,22,24,38,41. Staf rated practice
environment quality is linked to patient outcomes in a number of studies in Katz’s review
and others24,42-45. Staing levels, staf satisfaction and staf perception of the quality of the
practice environment were among the most frequently identiied indicators in our sources
(see Table 1).
Box 4. Evidence base: summary
•
Strong evidence associates nurse staing levels and mortality.
•
Evidence linking variation in nurse staing to failure to rescue and hospital acquired
pneumonia suggests the potential signiicance of such indicators.
•
Indicators theoretically most closely associated with nursing, such as pressure
ulcers, are not clearly supported by strong or consistent evidence linking variation
to nurses’ work.
•
Evidence links other aspects of the workforce (job satisfaction, quality of the work
environment including leadership) with mortality.
15
State of the art metrics for nursing
Doran’s extensive review considers evidence for a wider range of possible indicators21.
However, while the review process is extensive and rigorous, the synthesis is narrative and
does not fully meet criteria for a systematic review46-48. The weight of evidence ofered
is considered in a qualitative fashion; crucially, much of it derives from studies in which
nursing interventions made a positive impact on patient outcomes. Rejecting this as a
core source of evidence might seem perverse but because speciic interventions generate
speciic outcomes, a focus on interventions is unlikely to yield valid overall indicators of
quality that apply to clinical nursing services as a whole. The identiication and delivery
of efective interventions and measurement of associated outcomes are important but
indicators must be able to broadly relect quality. Thus while Doran’s review may ofer
a guide to developing indicators for outcomes of speciic nursing interventions, it cannot
ofer the most authoritative overall source of evidence for indicators that relect the
broader care environment and characteristics. Future research might identify the variation
in outcome associated with quality nursing care from systematic reviews of nursing
interventions along the lines proposed by Mantz49, who assessed variation in outcome
associated with quality stroke care from a systematic review of stroke units. However,
searches of the Cochrane Library as part of this rapid appraisal did not reveal any
equivalent reference points for the most widely advocated potential indicators, including
falls, pressure ulcers and urinary tract infections.
16
State of the art metrics for nursing
State of the art
Best bets
While indicators cannot provide a complete picture or a complete solution, they can
provide a powerful mechanism to incentivise quality by making the contribution of nursing
more visible within the healthcare system. However, identifying and using indicators is
by no means straightforward. Table 4 illustrates the trade-ofs by considering the relative
merits of two possible indicators: mortality (for which the evidence base is strong) and
pressure ulcers (where there is broad consensus about the contribution of nursing).
Table 4: Comparison of mortality and pressure ulcers as outcome indicators for nursing
Criteria
Importance
Scientiic basis
Useability
Feasability
Impact
Mortality
High
Pressure ulcers
Medium
Variation in quality
High
Unsure
Evidence of sensitivity to nursing
Strong
Weak
Risk adjustment
Feasible
Problematic
Speciication/deinition of the outcome Clear
Problematic
Reliability of data collection
Good
Problematic
Variation attributable to nursing
Low
Unclear
Ownership by nursing
Unclear
High
Knowledge to inform action
Unclear
Clear
Wide applicability
Yes
Yes
Positive behavioural incentives
High
Mixed
Potential for gaming
Low
High
Timely availability of data
Potentially
Challenging
Routinely collected data
Yes
No
The importance of mortality as an issue is unambiguously high. There is ample evidence
that some unexplained variation probably relates to the quality of health services, and
evidence that some of this is attributable to variation in the quality of nursing. Generally
data collection is reliable because the outcome is unambiguous and easy to deine. Risk
adjustment is possible, if challenging. However, the actual variation attributable to nursing
is low, and direct ownership by the profession is also likely to be low since the contribution
of other professions to preventing (or causing) death is widely recognised. Similarly,
speciic actions required of nursing to rectify quality problems are often unclear. However
mortality is an outcome that applies to a wide range of settings, and while negative
behavioural incentives are possible (for example, refusing to admit sicker patients) they are
relatively unlikely at the level of a nursing service. Neither is mortality readily amenable to
17
State of the art metrics for nursing
gaming, and such information is potentially available from routinely collected and timely
administrative data.
By contrast, the importance of pressure ulcers is harder to asses. The problem is
prevalent but its social signiicance is harder to quantify, although economic impact is
potentially high50. While risk adjustment models do exist51, elements of coding for both the
outcome (grade of ulcer) and risk adjustment (for example mobility) are highly subjective
(introducing potential bias and unreliability) and are not generally available in UK data
sets. Determination of incidence is also problematic when pressure ulcer data are collected
in point prevalence surveys, and the amount of variation associated with quality nursing
is unclear. Clearly there is considerable ownership of the problem by the profession, as
evidenced by the sources we have reviewed, but the diference in outcomes between
those receiving high-quality nursing and poor-quality nursing is hard to estimate. If most
pressure ulcers are preventable it could be very high, but the evidence we reviewed is
neither consistent nor clear. Similarly, there is considerable scope for strategies to remedy
these problems to be developed and acted upon by nursing autonomously, although the
evidence base for action is not entirely clear. The outcome applies to a wide range of
patients in a range of settings, although it predominantly relates to people with protracted
(acute or long-term) institutional stays. Recording and coding problems potentially
incentivise negative behaviours (poor reporting) and a lack of routine outcome data
leaves open the potential for gaming with process measures. Because data are likely to
be available only from intermittent audit, timely feedback will be challenging (but not
impossible). It is hard to foresee fully valid data being derived from administrative data sets
in the near future, although this area holds potential52.
It is beyond the current paper’s scope to give a detailed consideration of all these issues
against all the possible indicators identiied, but the preceding discussion illustrates the
challenges. As part of this review we identiied a number of potential indicators and
identiied the strongest evidence for an association with variation in the quality of nursing.
Of the range of potential indicators, a number emerge as potential “front runners” either
because of strength of evidence or strength of opinion supporting the indicator (Box
5). Failure to rescue, hospital acquired infection, pressure ulcers and falls were the most
strongly advocated patient safety indicators among our sources. Of these, failure to rescue
and health care associated pneumonia are most clearly supported by evidence of variation
associated with nursing. Because failure to rescue is likely to be a rare outcome in most
settings, its utility for local quality monitoring is likely to be low. Staing levels (generally
registered nurse staing), staf satisfaction and perception of the practice environment are
also strongly supported workforce quality indicators generally supported by evidence.
While indicators of compassion were not strongly relected either in the evidence reviewed
or the sources identiied, they remain important. We classiied satisfaction as relecting
compassion but satisfaction is a complex construct. It is widely used and many measures
of satisfaction do contain items relating to patient experience of compassion but general
18
State of the art metrics for nursing
patient satisfaction surveys seem insensitive to important aspects of experience53. Elements
of communication should be regarded as intrinsic to quality and fundamental to both
compassionate and efective care, but speciic questions must be identiied to properly
relect patients’ experience of ‘compassionate’ nursing. Any indicator needs to be linked to
a nursing unit (for example a ward) and available in a timely fashion to provide maximum
impact on staf.
Box 5. ‘Front runners’ in the indicator stakes
Safety
•
Failure to rescue
•
Health care associated pneumonia
•
Health care associated infection
•
Pressure ulcers
•
Falls
Efectiveness
•
Staing levels and patterns
•
Staf satisfaction
•
Staf perception of the practice environment
Compassion
•
Experience of care (patient-reported)
•
Communication (patient-reported)
Key challenges
For many of the possible outcome metrics, the extent of the contribution of nursing relative
to other professions is questionable or controversial. For example, although functional
status/self-care/activities of daily living (ADL) have been cited as ‘nursing’ outcomes,
other professions might dispute this. In the case of mortality, despite strong evidence of an
association with nursing variables it is hard to ofer it as a measure of nursing quality per se
although the nursing contribution to it should not be disregarded. Of the widely advocated
outcomes we identiied, the most strongly supported by evidence from Kane’s systematic
review is failure to rescue. This is generally recognised as nurse-sensitive but staing levels
were associated with only 16% of the variation (and only for surgical patients). Hospital
acquired pneumonia (as opposed to HCAI in general) was associated with a larger
proportion of variation (31%) across all (acute) settings. While these proportions may
seem relatively small, this variation is related to staing levels only, not to other aspects of
quality nursing care. Although both indicators show strong potential in recent reports 52,
the feasibility of deriving them from UK administrative data sets is yet to be fully tested.
19
State of the art metrics for nursing
Inclusion of outcome indicators such as pressure ulcers and falls would need to be tied to
a strategy to incentivise surveillance, recording and coding. These speciic indicators are
unlikely to be suited to use of routinely collected administrative data sets, certainly in the
short term. Even US-based systems (with supposed better secondary coding) do not use
routine data for pressure ulcers and instead rely on intermittent surveys54.
While some outcomes may be controversial because of the relative contribution of other
professions to variation, all are subject to patient-level variation. In most cases the main
determinants of outcome are patients themselves, not care inputs. Adjustment for patientlevel variation in risk is likely to be a formidable challenge, particularly when comparing
across institutions. Existing indicator sets such as the NQF attempt to tackle this but
rely on US administrative data speciications and levels of secondary coding, so their
applicability to the UK is likely to be limited. Instead, these sets may provide a relatively
strong starting point for developing UK indicators to be derived from routinely collected
data. Risk adjustment does not present such a substantial obstacle for local quality
monitoring where the comparison is with a unit’s own past performance, provided that
there is no substantial change in the patient population to alter underlying risk.
It is appealing to consider care processes as useful alternatives to measuring outcomes.
For nursing this has often been represented by systems focussing on documentation and
recording of assessments. However, there is a potential problem of token compliance to
such process measures which then have no impact upon the care delivered, so such metrics
should be used with caution. In some cases process measures also need risk adjustment,
although it is generally less problematic55. The burden of the audit associated with process
measures will be relatively high where data are not routinely recorded in a useable format,
as is generally the case in the UK. Since an audit is likely to be required, patient outcome
audit may be a better investment although risk adjustment remains problematic.
Any indicator set must be suiciently diverse to indicate overall quality. While isolated
outcomes are poor relections of overall quality29, a purely process-based system is not well
suited to nursing. Such approaches do not work well in complex systems involving high
degrees of individual skill and multiple actions, and where there is great uncertainty about
speciic processes that contribute to success. In these cases, individual aspects of care that are
measurable do not relect the complex packages of care that represent best practice49, 55 , 56.
Workforce factors emerge as strong candidates because of their association with
important outcomes and because they are clearly nursing-related. However, the
considerable available evidence on staing levels and skill mix does little to guide the
benchmarking of staing levels for settings outside the US as contexts of care difer
signiicantly. Setting benchmarks based on historical workforce characteristics and ways
of working also may ‘freeze’ the current coniguration and stile innovation. Preliminary
economic modelling suggests that investment in highly skilled staf may deliver better
returns than investment in total staf40 ,41, indicating potential to change the current balance
between qualiied and unqualiied staf, providing qualiied staf are trained to a higher
20
State of the art metrics for nursing
level. Benchmarks used as measures of quality might create perverse incentives to employ
more junior and less qualiied staf at the expense of fewer more highly skilled but more
expensive ones and thus stile positive innovations in staing.
Although not directly supported by evidence, the JCAHO indicator of staing in
relation to the institution’s staing plan57 may be a more satisfactory indicator in the short
term, since it requires a staing plan and some justiication of staing levels. Measures
of staf satisfaction and perceived work environment quality are promising. Yet the
content of the NWI’s Practice Environment Scale58, while containing items of clear face
validity (relating to leadership and support, perceived standards and adequacy of care)
and wide applicability, has items with questionable support or that advocate practices
of questionable validity (e.g. use of nursing diagnoses). Staf-reported measures and
perceptions are less amenable to gaming and perverse incentives than staing quotas and
benchmarks although by no means immune.
Despite these challenges (Box 6), the motivation for developing indicators is compelling.
Even the harshest critics of such systems allow the potential for beneit and acknowledge
the lack of clear alternatives in the context of a publically funded health system34, although
there is much argument over the relative merits of process and outcome measures29,59.
Desires within the nursing profession for articulation of its important contribution
and among the public for improved standards point toward outcomes as a signiicant
component of any indicator system, as their importance is more clearly understood and
harder to contest.
Finally, our proposed indicators largely lack positive articulations of nursing’s impact.
Patient experience of care did not emerge prominently, although ‘satisfaction’ with nursing
is among the most widely used nursing ‘outcome’ measures60 and is a major determinant
of overall satisfaction scores61, 62. Nursing’s positive contributions do not easily translate
into speciic objective outcomes either because nursing makes a small contribution to the
overall outcome (such as functional status) or because the contribution is to the subjective
experience of the patient. As patient-reported outcome measures (PROMS) provide
patients opportunities to assess their treatment outcome63, they merit consideration
for evaluation of speciic nursing interventions or narrowly focussed nursing services.
However, PROMS’ utility as nursing indicators may be limited because the contribution
of nursing services to most speciic PROMS across a patient population is still likely to be
small in light of patients’ experiences with other health services personnel
Some elements of care processes, such as the quality of communication as assessed by
patients, are clearly on a pathway to efective care, because change in health behaviour
and treatment concordance require successful communication. The task and inish group
were also very clear about nurses’ contribution to patient dignity and experiences of care
as humane and compassionate. Concerns about a loss of compassion and expressions of
caring in the emotional sense underpin both professional and public concerns about the
state of nursing noted earlier6-8.
21
State of the art metrics for nursing
Therefore, in addition to key safety efectiveness and workforce indicators, any set of
measures that seeks to represent the nursing system as a whole must seek to represent
the experiences of people who are ‘nursed’. Our list of possible indicators touches on this
in items such as dignity, communication and satisfaction, which often are measured as
composite scores based on evaluations of aspects of experience. Such indicators have the
potential to be intrinsically important if properly surveyed (irrespective of evidence linking
them to objective outcome) and many aspects of experience, such as communication with
professionals, may have important objective consequences. Certainly, a lack of evidence
directly linking these indicators to other aspects of quality represents a challenge for future
research but not an obstacle to the use of patient experiences as indicators of quality. While
there are a number of possible candidates (such as items from the NHS patient survey) the
selection of indicators should not be decided by professionals alone. Challenges include
delivering results of such surveys in a timely fashion and attributing them to nursing units
such as wards rather than hospitals as a whole. Yet most potential indicators face these
challenges, and although diferent values and expectations complicate comparison across
diferent populations, providers’ opportunity for gaming is limited.
Box 6. Key challenges
•
Deining data and full speciication of indicators
•
Adjusting for risk
•
Improving the quality of clinical coding
•
Identifying indicators for nursing’s impacts in mental health, community, primary
care and paediatric settings
•
Identifying and deining indicators that cross care pathways and boundaries
•
Timely reporting at the nursing unit level
•
Delivering action to improve quality
22
State of the art metrics for nursing
Conclusion and recommendations
Developing indicators for nursing is challenging, but current circumstances provide
both an opportunity and an imperative for the profession to embrace tangible measures
of nursing’s contributions to patient care. In this report we have identiied candidate
indicators for further development. Some indicators have potential applications in a range
of settings, but indicators suitable for acute hospital settings have more evidence and
better speciication. We have been wary of process and structure indicators because of the
uncertain link of likely process measures to outcomes or, in the case of workforce measures,
the danger of ixing current workforce patterns and stiling innovation. Both processes and
structures are important, and staf-reported measures of work environment and patientreported measures of their care experiences are both potentially important indicators, yet
there are compelling reasons to ensure that nursing outcomes will be prominent in any
indicator set. Ideally these outcomes should relect all the dimensions of efectiveness,
safety and compassion.
The promise of ‘no more national targets’ and the Next Stage Review64 commitment
to reward quality are reassuring. We began by relecting on concerns that the quality of
nursing was being neglected in the face of performance targets focusing on productivity.
But performance and output of care remain important, and quality measures should
directly relect elements of patient health outcome and experience that beneit from highquality nursing care, lest these elements be neglected in favor of performance targets. The
Next Stage Review proposes rewarding quality to a much greater extent than at present.
If the quality of nursing can be properly relected in wider sets of metrics to be developed,
nursing’s contributions can be more fully recognised at all levels of NHS governance and
management. The amount of work needed to fully realise this opportunity should not be
underestimated, but without such metrics nursing faces increasing invisibility within a
performance-managed health service.
Most important, feedback on performance that is based on important measures of
nursing can improve nursing care quality by providing frontline staf with information on
trends, emerging problems and successes. Such feedback can help senior clinical managers
by giving overviews of performance and allowing them to both troubleshoot problems
and recognise success. Feedback can help managers articulate nursing’s contributions
in tangible terms when seeking resources to protect and enhance quality, and it allows
boards to ‘see’ these contributions and to properly support nursing. Finally, such nursing
metrics can empower the public to choose between care options using indicators based
on an aspect of health services which clearly matters to them as much as it matters to the
profession: the quality of nursing care.
23
State of the art metrics for nursing
Box 7. Conclusions
•
Of the range of potential indicators, a number emerge as potential “front-runners”.
Failure to rescue, health care associated infection, pressure ulcers and falls were the
most strongly advocated patient safety indicators.
•
Staing and skill mix are linked to patient outcome but their use as indicators would
stile change and create perverse incentives. Use of “staing matched to planned
staing” as part of a suite of indicators including outcomes has more potential.
•
Process indicators should be used with caution because of potential for gaming and
diiculty in linking speciic processes and patient outcomes.
•
Patient experience of compassionate care is an important outcome in its own right
and may provide the best measure of the nursing contributions to shared outcome
and evaluation of complex processes that are otherwise elusive.
•
Despite these and other challenges, the case for developing metrics is strong.
Recommendations
We believe the nursing profession should embrace development of metrics and establishment
of a National Quality Board and local quality observatories. These initiatives present a
substantial opportunity for the profession to equip itself with tools needed to deliver excellent
care into the future. Based on the evidence considered here, we recommend:
The new National Quality Board should ensure that nursing’s contribution is properly
represented in its programme of metrics development. In conjunction with care regulators
and other stakeholders, it should develop and support a national standard set of nursing
indicators including outcome measures for safety, efectiveness and compassion. As part of
this programme we recommend:
•
Existing data sources should be used where possible, but a process to develop and
adapt data collection systems to increase indicator validity also is needed.
•
Speciications for failure to rescue and HCA pneumonia should be developed as
outcome indicators derived from routinely collected data.
•
These ‘shadow’ indicators should be launched and any association with known
markers of quality (e.g. risk-adjusted mortality) should be tested.
•
Standards should be developed to collect and report data on falls and pressure ulcers, as
should risk adjustment models for use in regional and national benchmarking.
•
Staf surveys should include speciic nursing work environment questions.
•
Speciications are needed for a workforce planning indicator based on the JCAHO model.
•
Core nursing-related indicators in patient surveys should be identiied.
•
Patient groups and professionals can cooperate to identify key indicators of compassion.
24
State of the art metrics for nursing
The Department of Health should ensure that the Connecting for Health programme
provides a suitable infrastructure to support the collection of clinical outcome and process
data to provide nursing indicators and that a Minimum Dataset for Nursing (including
patient dependency and staing variables) can be derived from it.
The Royal College of Nursing, NHS Confederation and other key stakeholders should
consider facilitating a nursing quality coalition of organisations which currently use metrics
with the aim of sharing best practice, standardising data collection where this will be
useful, and sharing data to facilitate research on indicator validity. The California Nursing
Outcomes Coalition (CalNOC) may provide a model for this.
Professional bodies and senior members of the profession should reinforce and reiterate
the signiicance of nursing contribution in all settings to both nursing-sensitive and
shared outcomes, such as mortality and preventable admissions. Claims need to be
based on sound evidence and the nursing research community must focus more efort on
exploring outcomes.
Health care providers must identify mechanisms to ensure timely feedback of indicators at
the nursing unit level, and to ensure that positive action results from this information.
Health care commissioners and regulators should use nursing metrics as part of
their assessment of quality and the commissioning process, and should move toward
publication of indicators.
Further work is needed to develop indicator sets outside acute settings and for patient
pathways across settings.
25
State of the art metrics for nursing
References
1.
Department of Health. NHS Next Stage Review: A High Quality Workforce. London:
Department of Health, 2008.
2.
Pencheon D. The Good Indicators Guide: Understanding how to use and choose
indicators. Warwick: NHS Institute for Innovation and Improvement, 2008.
3.
Redfern SJ, Norman IJ. Quality assessment instruments in nursing: towards
validation. Int J Nurs Stud 1995;32(2):115.
4.
Higgins M, McCaughan D, Griiths M, Carr-Hill R. Assessing the outcomes of
nursing care. J Adv Nurs 1992;17:561.
5.
Savitz L, Jones C, Bernard S. Quality Indicators Sensitive to Nurse Staing in Acute
Care Settings. Advances in Patient Safety: From Research to Implementation (AHRQ
Publication No. 050021). Rockville: Agency for Healthcare Research and Quality,
2005:375-85.
6.
Corbin J. Is caring a lost art in nursing? Int J Nurs Stud 2008;45(2):163.
7.
Griiths P. Editorial: The art of losing...? A response to the question ‘is caring a lost
art?’ Int J Nurs Stud 2008;45(2):329.
8.
Raynor C. Why gossiping nurses are a symptom of an NHS in crisis. Daily Mail
2008 online http://www.dailymail.co.uk/pages/live/articles/health/healthmain.
html?in_article_id=561164&in_page_id=1774 (accessed April 20th 2008)
9.
Healthcare Commission. Investigation into outbreaks of Clostridium diicile at Stoke
Mandeville Hospital, Buckinghamshire Hospitals NHS Trust. London: Commission for
Healthcare Audit and Inspection, 2006.
10. Healthcare Commission. Investigation into outbreaks of Clostridium diicile at
Maidstone and Tunbridge Wells NHS Trust. London: Commission for Healthcare
Audit and Inspection, 2007.
11. Healthcare Commission. Learning from investigations. 2008.
12. Maben J. The art of caring: Invisible and subordinated: A response to Juliet Corbin:
‘Is caring a lost art in nursing’. Int J Nurs Stud 2008;45(3):335.
13. Clarke SJ. Research on Nurse Staing and its Outcomes. In: Nelson S, Gordon
S, editors. The Complexities of Care: Nursing Reconsidered. New York: Cornell
University Press, 2006.
14. Audit Commission. Review of National Finding: Ward Staing. Acute Hospital
Portfolio. London: Audit Commission, 2001:15.
15. Audit Commission. Ward Staing Management Tools: Practical Tips for Budget
Setting. London: Audit Commission, 2001:3.
16. Black N. Rise and demise of the hospital: a reappraisal of nursing. BMJ
2005;331(7529):1394.
26
State of the art metrics for nursing
17. Clarke SPP. Registered Nurse Staing and Patient Outcomes in Acute Care:
Looking Back, Pushing Forward. Med Care 2007;45(12):1126.
18. Maas ML, Johnson M, Moorhead S. Classifying Nursing-Sensitive Patient
Outcomes. J. Nurs. Sch. 1996;28(4):295.
19. Doran D. Preface. In: Doran D, editor. Nursing-sensitive outcomes: state of the science.
Sudbury / London: Jones and Bartlett Pub., 2003:vii-ix.
20. Mitchell PH, Ferketich S, Jennings BM. Quality Health Outcomes Model. J. Nurs.
Sch. 1998;30(1):43.
21. Doran D. Nursing-sensitive outcomes: state of the science. Sudbury / London: Jones
and Bartlett Pub., 2003.
22. Kane R, Shamliyan T, Mueller C, Duval S, Wilt T. The Association of Registered
Nurse Staing Levels and Patient Outcomes: Systematic Review and MetaAnalysis. Med Care 2007;45(12):1195.
23. Lankshear AJ, Sheldon TA, Maynard A. Nurse Staing and Healthcare Outcomes:
A Systematic Review of the International Research Evidence. Advances in Nursing
Science 2005;28(2):163.
24. Kazanjian A, Green C, Wong J, Reid R. Efect of the hospital nursing environment
on patient mortality: a systematic review. J Health Serv Res Policy 2005;10:111.
25. Lang TA, Hodge M, Olson V, Romano PS, Kravitz RL. Nurse-patient ratios: a
systematic review on the efects of nurse staing on patient, nurse employee, and
hospital outcomes. J Nurs Adm 2004;34(7-8):326.
26. Johnson M, Maas M. The Nursing Outcomes Classiication. J Nurs Care Qual
1998;12(5):9.
27. Silber JH, Romano PS, Rosen AK, Wang Y, Even-Shoshan O, Volpp KG.
Failure-to-Rescue: Comparing Deinitions to Measure Quality of Care. Med Care
2007;45(10):918.
28. National Quality Forum. A Comprehensive Framework for Hospital Care Performance
Evaluation: A consensus report. Washington: National Quality Forum, 2003.
29. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of
process and outcome data in managing performance of acute medical care: avoiding
institutional stigma. Lancet 2004;363(9415):1147.
30. Mainz J. Deining and classifying clinical indicators for quality improvement. Int J
Qual Health Care 2003;15(6):523.
31. Riehle AI, Hanold LS, Sprenger SL, Loeb JM. Specifying and Standardizing
Performance Measures for Use at a National Level: Implications for Nursingsensitive Care Performance Measures. Med Care Res Rev 2007;64(2_suppl):64S.
32. Donaldson N, Brown D, Aydin C, Bolton M, Rutledge D. Leveraging Nurse-related
Dashboard Benchmarks to Expedite Performance Improvement and Document
Excellence. J Nurs Adm 2005;35(4):163.
27
State of the art metrics for nursing
33. Boyne GA, Law J. Setting Public Service Outcome Targets: Lessons from Local
Public Service Agreements. Pub Money and Man 2005;25(4):253.
34. Bevan G, Hood C. What’s Measured Is What Matters: Targets and Gaming in the
English Public Health Care System. Pub Admin 2006;84(3):517.
35. Quan H, Parsons GA, Ghali WA. Assessing accuracy of diagnosis-type indicators for
lagging complications in administrative data. J Clin Epidemiol 2004;57(4):366.
36. Pancorbo-Hidalgo PL, Garcia-Fernandez FP, Lopez-Medina IM, Alvarez-Nieto C.
Risk assessment scales for pressure ulcer prevention: a systematic review. J Adv Nurs
2006;54(1):94.
37. Havens DS, Aiken LH. Shaping Systems to Promote Desired Outcomes: The
Magnet Hospital Model. JONA 1999;29(2):14.
38. Raferty AM, Clarke SP, Coles J, Ball J, James P, McKee M, et al. Outcomes of
variation in hospital nurse staing in English hospitals: Cross-sectional analysis of
survey data and discharge records. Int J Nurs Stud 2006.
39. McKee M, Coles J, James P. ‘Failure to rescue’ as a measure of quality of hospital
care: the limitations of secondary diagnosis coding in English hospital data. J Pub
Health 1999;21(4):453.
40. Needleman J, Buerhaus PI, Stewart M, Zelevinsky K, Mattke S. Nurse Staing
In Hospitals: Is there a Business Case for Quality? Health Af (Millwood)
2006;25(1):204.
41. Newbold D. The production economics of nursing: A discussion paper. Int J Nurs
Stud 2008;45(1):120.
42. Aiken LH, Clarke SP, Sloane DM, Sochalski J, Silber JH. Hospital Nurse
Staing and Patient Mortality, Nurse Burnout, and Job Dissatisfaction. JAMA
2002;288(16):1987.
43. West E, Maben J, Raferty A. Nursing and Patient Outcomes: How Can Employers
Provide the Right Environment for Nurses to Deliver High-Quality Care? Harvard
Health Policy Review 2006;7(1):64.
44. Stone PW, Mooney-Kane CM, Larson EL, Horan TM, Glance LG, Zwanziger
J, et al. Nurse Working Conditions and Patient Safety Outcomes. Med Care
2007;45(6):571.
45. Estabrooks CA, Midodzi WK, Cummings GG, Ricker KL, Giovannetti P.
The impact of hospital nursing characteristics on 30-day mortality. Nurs Res
2005;54(2):74.
46. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Metaanalysis of Observational Studies in Epidemiology: A Proposal for Reporting. JAMA
2000;283(15):2008.
28
State of the art metrics for nursing
47. Moher D, Cook D, Eastwood S, Olkin I, Rennie D, Stroup D. Improving the
quality of reports of meta-analyses and randomised controlled trials: the QUOROM
statement. Lancet 1999;354:1896
48. Center for Reviews and Dissemination. What are the criteria for the inclusion of
reviews on DARE? York: Center for Reviews and Dissemination online http://www.
york.ac.uk/inst/crd/darefaq.htm (accessed 01/05/2008)
49. Mant J. Process versus outcome indicators in the assessment of quality of health care.
Int J Qual Health Care 2001;13(6):475.
50. Bennett G, Dealey C, Posnett J. The cost of pressure ulcers in the UK. Age Ageing
2004;33(3):230.
51. Berlowitz DR, Brandeis GH, Morris JN, Ash AS, Anderson JJ, Kader B, et al.
Deriving a Risk-Adjustment Model for Pressure Ulcer Development Using the
Minimum Data Set. J Am Geriatr Soc 2001;49(7):866.
52. Bottle A, Aylin P. How NHS trusts could use patient safety indicators to help
improve care. Health Care Risk Report 2008;May 2008:12.
53. Jenkinson C, Coulter A, Bruster S, Richards N, Chandola T. Patients’ experiences
and satisfaction with health care: results of a questionnaire study of speciic aspects of
care. Qual Saf Health Care 2002;11(4):335.
54. National Quality Forum. National Voluntary Consensus Standards for NursingSensitive Care: An Initial Performance Measure Set. Washington: National Quality
Forum, 2004.
55. Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of processbased measures of health care quality. Int J Qual Health Care 2001;13(6):469.
56. Kendrick S. Using all the evidence: towards a truly intelligent National Health
Service. Health Bulletin 2001;59(2).
57.
JCAHO. Staing Efectiveness in Hospitals. Oak Brook: Joint Commission Resources, 2003.
58. Lake ET. Development of the practice environment scale of the nursing work index.
Res Nurs Health 2002;25(3):176.
59. Donabedian A. Evaluating the Quality of Medical Care. Milbank Quarterly
2005;83(4):691.
60. Avis M, Bond M, Arthur A. Satisfying solutions? A review of some unresolved issues
in the measurement of patient satisfaction. J Adv Nurs 1995;22(2):316.
61. Yellen E. The Inluence of Nurse-Sensitive Variables on Patient Satisfaction. AORN
2003;78(5):783.
62. Abramowitz S, Coté A, Berry E. Analyzing patient satisfaction: a multianalytic
approach. Quality Review Bulletin 1987;13(4):122.
63. Smith S, Cano S, Lamping D, Staniszewska S, Browne J, Lewsey J, et al.
Patient-Reported Outcome Measures (PROMs) for routine use in Treatment Centres:
recommendations based on a review of the scientiic evidence. London: London School
of Hygiene and Tropical Medicine, 2005.
29
State of the art metrics for nursing
64. Department of Health. High Quality Care for All: NHS Next Stage Review Final
Report. Norwich: Stationary Oice, 2008.
65. Healthcare Commission. Acute Hospital Portfolio Review: ward staing topic guide.
London: Healthcare Commission, 2004.
66. International Council of Nurses. Nursing Sensitive Outcome Indicators. Geneva:
International Council of Nurses online http://www.icn.ch/matters_indicators.htm
(accessed 01/05/2008)
67. Naylor MD. Advancing the Science in the Measurement of Health Care Quality
Inluenced by Nurses. Med Care Res Rev 2007;64(2_suppl):144S.
68. Van den Heede K, Clarke SP, Sermeus W, Vleugels A, Aiken LH. International
Experts’ Perspectives on the State of the Nurse Staing and Patient Outcomes
Literature. J Nurs Schol 2007;39(4):290.
30
State of the art metrics for nursing
Appendix 1: Core data sources
Name
Agency for
Healthcare
Quality and
Research
Patient safety
Indicators
(nursing subset)
American
Nurses
Association
Association of
UK University
Hospitals
nurse-sensitive
indicators
Doran 2007
(review)
Short ref Description
Setting/
clinical
group
Ref/link
AHRQ
Patient safety
indicators
Acute care* http://www.
qualityindicators.ahrq.gov/
psi_overview.htm 5
ANA
Nursesensitive
indicator set
Acute,
paediatric,
long term
care and
psychiatric
http://www.
mnursingworld.org/
ainMenuCategories/
professionalNursing/
PatientSafetyQuality/
NDNQI/NDNQI_1/
ngSensitiveIndicators.aspx
AUKUH Nursesensitive
indicator set
Acute care
http://www.aukuh.org.uk/
members/PCP.htm
Doran
Review of
Acute care
nurse-sensitive
outcomes
21
Essence of care
EoC
Benchmark
statements for
the quality of
nursing care
Not
speciied
http://www.dh.gov.uk/en/
Publicationsandstatistics/
Publications/
publicationsPolicy
AndGuidance/
DH_4005475
Healthcare
Commission
HC
Factors
assessed in
ward staing
review
Acute care
http://www.
healthcarecommission.
org.uk/erviceproviderin
formation/reviewsand
studies/servicereviews/
ahpmethodology/
wardstaing.cfm 65
31
State of the art metrics for nursing
Name
Short ref Description
Setting/
clinical
group
Ref/link
International
Council of
Nurses Brieing
ICN
Review of
Not
nurse-sensitive speciied
outcome
indicators
http://www.icn.ch/
matters_indicators.htm 66
Joint
Commission
staing
efectiveness
Indicators
JCA
Staing
Acute care
efectiveness
indicators
including NQF
indicators
31
Kane 2007
(review)
Kane
Review of
evidence
for staing
outcome
relationship
Acute care
22
Kazanjian 2005
(review)
Kaz
Review of
Acute care
evidence
linking ward
environment to
mortality
24
Lang 2004
(review)
Lang
Review of
evidence
for staing
outcome
relationship
Acute care
25
Lankshear 2005
(review)
Lank
Review of
evidence
for staing
outcome
relationship
Acute care
23
Nursing
Home Quality
Initiative
Medicare/
Medicaid
Quality
Compare
NHQI
Long-term
care/post–
acute care
indicator set
Long-term
care
http://www.medicare.gov/
NHCompare/
32
State of the art metrics for nursing
Name
Short ref Description
Setting/
clinical
group
Ref/link
NQF
Nursesensitive
indicator set
Acute care
http://www.qualityforum.
org/nursing/54 ,67
OASIS
Home health
care indicator
set
Home/
community
health care
http://www.cms.hhs.gov/
OASIS/
Other
Oth
Other clinical
indicator sets
/dashboards
identiied on
web searches
Generally
acute
N/A
Quality
Outcomes
Framework
(nursing items)
QOF
General
Primary
practice
care
quality
indicators with
speciic items
relating to
nursing
http://www.dh.gov.uk/en/
Healthcare/Primarycare/
Primarycarecontracting/
QOF/index.htm
Van den Heede
2007
VdH
Expert
Not
consensus
speciied
regarding
nurse-sensitive
outcomes
68
National
Quality Forum
Consensus
Standards
for Nursingsensitive Care
OASIS
(Medicare
reporting
subset)
33
Appendix 2: Indicators identiied
ANA
AUKUH
Doran
EoC
HC
ICN
JCA
Safety
√
√
√
√
√
√
√
√
Failure to rescue
Safety
√
√
√
√
√
√
Staing levels (nurses/hours per patient)
Workforce
√
√
√
√
Falls
Safety
√
√
√
HCAI (pneumonia)
Safety
√
√
Staf satisfaction and well-being
Workforce
HCAI (UTI)
Safety
Area
Pressure ulcer
34
Staing, skill mix
Workforce
Medication administration errors
Safety
Mortality
Safety
Practice environment/perceived quality
Workforce
Satisfaction with (nursing) care
Experience
Sickness rates
Workforce
Smoking advice
Preventative
Staing bank/agency
Workforce
Communication
Experience
Staf experience, knowledge, skills, expertise
HCAI (surgical wound)
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
Use of restraints
Function
Safety
Experience
√
√
√
√
√
11
√
9
√
√
9
√
√
8
√
√
8
√
√
7
√
√
6
√
6
√
5
√
5
√
√
√
√
√
Oth
QOF
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
√
5
√
5
√
5
√
5
4
√
4
4
√
√
√
√
√
√
√
√
5
√
√
√
√
√
Count
√
√
√
√
VdH
OASIS
√
√
Workforce
√
NQF
√
√
Perception of adequate staing
NHQI
√
√
√
√
Lank
√
Safety
Arrest/shock
√
√
Instrumental activities of daily living and self-care Function
Complaints
√
√
Workforce
Lang
√
√
√
Kaz
√
√
√
√
Kane
√
√
4
√
4
√
4
√
3
√
3
State of the art metrics for nursing
AHRQ
Indicator
Indicator
Area
Conidence and trust
AHRQ
AUKUH
EoC
HC
ICN
Experience
√
√
√
Continence
Function
√
Emergency equipement/drugs
Safety
Extubation, unplanned
Safety
Safety
√
Injuries to staf
Safety (staf)
Interprofessional relations
Workforce
Utilisation
35
Safety
NHQI
NQF
OASIS
Oth
QOF
√
√
√
3
√
√
RN vacancies
Workforce
Symptom control (other, e.g. nausea)
Symptom
√
Ability to talk to nurse
Experience
√
Care planning/assessment processes
Planning
√
Cleanliness
Experience
√
Confusion, delirium
Safety
Dignity/respect
Experience
√
3
√
3
√
3
√
3
√
3
3
√
√
√
√
√
√
√
3
3
√
√
3
3
√
√
Safety
3
√
√
Safety
3
√
3
√
√
√
√
√
√
3
√
√
√
Count
3
√
√
√
VdH
√
√
√
Respiratory failure
Utilisation
Lank
3
√
Record keeping/reporting systems
Emergency care
Lang
√
√
Pain assessment/assessment intervention cycles Symptom
PE/DVT
Kaz
√
Experience
Nutrition
Kane
√
√
√
2
2
2
√
√
√
√
√
2
2
2
State of the art metrics for nursing
Safety
HCAI (bloodstream)
Length of stay
JCA
√
HCAI (any)
Nutritional assessment/screening
Doran
√
√
Knowledge of condition and treatment
ANA
Indicator
Area
HCAI (c. dif.)
Safety
HCAI (central line)
Safety
HCAI (MRSA)
Safety
Utilisation
Workforce
Pain/pain control
Symptom
Patient involvement
Experience
ANA
AUKUH
Doran
EoC
HC
JCA
Kane
Kaz
√
36
RN turnover
Workforce
√
Staf intent to leave
Workforce
√
Symptom control (dyspnea)
Safety
√
Time spent in CPD
Workforce
√
Understaing (compared to staing plan)
Workforce
√
Appraisal
Workforce
Cannula iniltration
Safety
√
2
2
2
√
2
√
2
2
√
√
√
√
Count
2
√
√
2
2
√
2
√
√
2
√
√
2
2
√
2
√
√
2
2
√
Preventative
Safety
VdH
√
Preventative
Allergies recorded
QOF
√
Risk assessments/other health promotion
Safety
Oth
√
Nutrition
Accidents/incidents
OASIS
√
√
Protected meal times policy implemented
Safety
NQF
√
Safety
Workforce
NHQI
√
Safety
Vascular access device ‘incidents’
Lank
√
Post-operative complications
Workload
Lang
√
Pressure ulcer risk assessment/planning
Vaccination rates
ICN
√
√
2
√
2
√
√
√
2
1
√
1
√
1
√
1
State of the art metrics for nursing
Hospital admissions/readmissions
Leadership
AHRQ
Indicator
Area
Discharge from caseload
Utilisation
Discharge planning/case management processes
Planning
Safety
Induction
Workforce
Mortality in low–mortality groups
Safety
Patients absconding or lost
Safety
Personal learning plan
Workforce
37
Pressure ulcer monitoring systems
Safety
Psychiatric physical/sexual assault
Safety
Psychological well-being
Experience
Record of training and updating
Workforce
Restraint documentation
Self-harm risk assessment
ANA
AUKUH
Doran
EoC
HC
ICN
JCA
Kane
Kaz
Lang
Lank
NHQI
NQF
OASIS
Safety
Use of long-term catheter
Function
Violence and aggression toward staf
Safety (staf)
Weight loss
Nutrition
VdH
Count
1
√
1
√
1
√
1
√
1
√
1
√
1
√
1
√
1
√
1
√
1
√
1
√
1
√
Timely assessment/intervention for pneumonia Safety
Upper GI bleed
QOF
√
Function
Safety
Oth
√
1
1
√
1
√
√
A=acute care, C=home/community health care, L=long-term care, N=not speciied, P=paediatric, Pc=primary care, Ps=psychiatric
1
1
State of the art metrics for nursing
HCAI surveillance system
AHRQ
State of the art metrics for nursing
38
To order further copies
Please contact:
The National Nursing Research Unit at King’s College London
Waterloo Campus, 57 Waterloo Road
London SE1 8WA
Telephone 020 7848 3057
Fax 020 7848 3069
http://www.kcl.ac.uk/schools/nursing/nnru
PDF download: free
Hard copy: administration charge £15 (including UK p&p)
Cover picture: The Diagram [based on the “coxcomb” diagram used by Florence Nightingale to
depict rate and cause of mortaliry among British soldiers during the Crimean war]
Laura Potter 2007
Silver, red gold, yellow gold
Approx. 600cm chain
Photo: Tomás Valenzuela
Part of the Culture & Care Project
ISBN: 978-0-9555991-2-5
© King’s College London 2008