Age and Ageing Advance Access originally published online on July 23, 2007
Age and Ageing 2007 36(5):483-485; doi:10.1093/ageing/afm085
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Evaluating the impact of the National Service Framework for Older People; qualitative science or populist propaganda?
The ‘evidence-base’ for policy in United Kingdom healthcare is often conspicuous by its absence.But can we assemble evidence on the effectiveness of policy? If policy is about ways of organising services to deliver interventions of proven value, then evaluation could look at implementation, and crude outcomes. A policy to change stroke services to deliver thrombolysis, could be evaluated by the proportion of cases thrombolysed, safety data, and survival rates. Similarly, we might quantify the activity of falls services, or the rates of people going into care homes. Health policy is more than this, however, promoting structures, systems and procedures, which, whilst often attractive at face value, lack any rigorous empirical support. Moreover, policy is implemented within a social and political context of ideology, commercial interests, resource constraints, media reports, pressure groups, lobbyists and public expectations, all of which skew its effects.
The National Service Framework for Older People (NSFOP) was published in 2001 as a broad service specification for the way older people should be managed in health and social care. It included specific guidelines on stroke, falls and mental health, which were more or less uncontentious. However, there were also aspirational system-wide standards on avoiding age-based discrimination, promoting dignity and person-centred care, a Single Assessment Process, and rehabilitation in the form of intermediate care. This came without tightly defined operational targets, or additional funding. Professional ownership was uncertain; the policy was for older people in the health service everywhere, not just geriatric medicine or old age psychiatry.
In this issue we publish an evaluation of the impact of the NSFOP, commissioned by the UK healthcare regulator, the Healthcare Commission [1]. The paper sharply divided our reviewers.
The study reports qualitative methods—public meetings, focus groups, and individual interviews, involving a total of 3,500 older people from around the country. Older people were largely ignorant of the NSFOP, but perceived changes over time in health and social care, some positive, some negative, some in keeping with the NSFOP, others not. Expectations were generally low. However, many with experience of services were surprised at how good they were.
In healthcare, qualitative research has been used increasingly over the past 20 years, but is still viewed with suspicion by many doctors brought up in the quantitative tradition of measurement and hypothesis testing. Qualitative work seeks an in-depth understanding of behaviour and the reasons behind it—the ‘why’ and ‘how’ of actions and systems. Studies are based on observation or interviews. Participants are chosen to seek the widest possible range of views, and recruitment continues until no new views emerge. Data are analysed for ideas and opinions, and these are categorised into more generalisible themes. If done properly, this is not mere anecdote informed only by preconception and prejudice. There is rigor, and processes for checking objectivity and consistency, including searching for findings that refute emerging themes (to reduce investigator bias), discussion of emerging findings with informants (confirming validity), and examining results to ensure that they are consistent with each other and other published work.
Qualitative research can explore areas that are complex or not fully understood, allowing hypotheses to be generated. This includes revealing unanticipated effects of interventions, the conditions necessary for effects, deciding which quantitative outcomes should be studied, and explaining why quantitative results have emerged. Qualitative methods complement rather than displace the cross-sectional survey or randomised trial. Many a trial has had negative results because the wrong thing was measured. This is especially important when interventions (such as rehabilitation) are complex, and their value lies at several levels apart from changing health status (providing information or reassurance, building confidence, reducing stigma, being cared about, meeting cultural expectations). Without qualitative enquiry we may not even understand what an intervention does. Examples include changes in service culture, or person-centred aspects of services impacting on satisfaction.
Can qualitative methods help evaluate a policy? The NSFOP evaluation collected a lot of information. It was also very big, and geographically dispersed, in a way that qualitative studies rarely attempt. Four big questions remain: validity, representativeness, interpretation and causality.
Validity depends on whether opinions were adequately sampled, and analysed correctly. We must assume that these aspects were done properly. The study sought to include ‘hard to reach’ groups, although it is possible that care home residents, recent in-patients, the severely disabled or communication impaired, mentally ill or those with current intensive caring responsibilities will have been under-represented compared with the relatively fit. We are told that ethnic minorities, those with learning disabilities, rural residents or those in contact with drug and alcohol services were targeted for inclusion.
Obtaining a range of views is laudable, to avoid discrimination and marginalisation; equity is a cornerstone of health policy. But representativeness is also important. How frequent are different views in the population? This requires a parallel quantitative study. A qualitative study may be valid, and valuable, but can only be partial.
Interpretation is not always straightforward. Where does the responsibility of the health service begin and end in a field so inextricably intertwined with social attitudes as a whole? How do respondents distinguish use of services (which may help or hinder) from the circumstances that give rise to their use (illness, disability) when contemplating undesirable outcomes (death, dependency, care home placement)? Apparently participants did not separate services specific to older people (presumably aware of older peoples' and carers' problems and free of age-discrimination) from generic services used by older people (a mixed bag).
The most important question for ‘an evaluation’ is whether the NSFOP led to any changes? This is the question of causality. Unfortunately this is impossible to judge from this study. The NSFOP was introduced amid myriad other policies and changes. The study could not evaluate the NSFOP in isolation, or analyse its effects in any direct sense, but instead presents a snap shot of views and attitudes of various stakeholders about the NSFOP, and, (as many did not know much about it) health and social services in general. The approach taken here is that of perceived reality: if the NSFOP is perceived to be useful then it is. Whether it has brought about meaningful change is quite another matter. One reading of the findings would be that, from the point of view of older people at large, the NSFOP has had very little impact.
Moreover, the health service is highly political, and it can be manipulated for political advantage. For the politician, the implication of no effect would be unacceptable. This is where the risk of propagandising comes in. However, would an independent research team working on behalf of a quasi-autonomous regulator want to join a conspiracy of false attribution?
What can we tell from the study? The findings ring true. The health service has changed over recent years, both for better and for worse, and older people are aware of the changes. The study gives an interesting picture of how the health service is currently perceived by older people, itself a valuable thing.
Has the NSFOP been effective? We cannot tell, and this study cannot help us decide. It was perhaps optimistic to expect that it could.
Is it worth publishing? Qualitative work should describe and illuminate, and may be interpreted differently by different people. It may lead to new understandings (perhaps about expectations and older peoples' concern for their carers), may generate debate about the question itself (what was the NSFOP for?), methodological and philosophical questions (how do you evaluate policy?), or the formulation of new questions (how do we give older people their due if they expect so little?) In this the study has succeeded, although it was not its stated aim. We shall let you decide its worth.
Nottingham University Hospitals NHS Trust, City Hospital, Nottingham NG5 1PB, UK
Tel: +44 115 969 1169 ext 55657/56022 Fax: +44 115 960 8409 Email: rowan.harwood{at}nuh.nhs.uk
References
- Manthorpe J, Clough R, Cormes M, Bright L, Moriarty J, Iliffe S. Four years on: the impact of the National Service Framework for older people on the experiences, expectations and views of older people. In: Age Ageing. doi:10.1093/ageing/afm078.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  S. lliffe, J. Wilcock, J. Manthorpe, J. Moriarty, M. Cornes, R. Clough, L. Bright, and Older People Researching Social Issues (OPRSI) Can clinicians benefit from patient satisfaction surveys? Evaluating the NSF for Older People, 2005-2006 J R Soc Med, December 1, 2008; 101(12): 598 - 604. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. L. Powell Evaluating the impact of the National Service Framework for Older People; qualitative science or populist propaganda? Age Ageing, May 1, 2008; 37(3): 351 - 351. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||