Abstract

Objective: We aim to provide evidence for the feasibility and validity of a pragmatic approach to estimating the impact of long-term care (LTC). It is difficult to evaluate LTC interventions. The available methods all have limitations in terms of their ability to provide unbiased estimates of the true treatment effect, cost of implementation, and application to some evaluation contexts and questions (Heckman and Smith 1995; Byford and Sefton 2003). An alternative approach, described by Mueller et al (2014) as the ‘counterfactual self-estimation of program participants’ (CSEPP), resolves the evaluation problem by asking individuals receiving an intervention to imagine their own counterfactual in the absence of the intervention, and estimate what their situation would be like in that hypothetical counterfactual situation. We describe an implementation of this approach, developed independently by Netten et al. (2012) as part of the Adult Social Care Outcomes Toolkit (ASCOT), for use in the context of the evaluation of LTC interventions.

Data and methods: Our data comes from a survey of 748 adult social care users in 22 administrative areas across England, conducted during 2013 and 2014. The survey used the ASCOT-INT4 interview schedule and also included questions asking respondents and interviewers to reflect on the feasibility of the CSEPP method. These data allow us to: (i) assess the feasibility of the ASCOT CSEPP approach across different groups of social care users; (ii) assess the construct validity of the counterfactual ASCOT score by modelling the score as a function of social care need (using observed needs-related characteristics such as activities of daily living, ADLs); and (iii) assess whether the method produces biased results by comparing the results obtained from the ASCOT CSEPP approach to those obtained using a non-experimental instrumental variables method developed by Forder et al (2014), which we consider to be the best alternative method.

Results: In general respondents were able to answer the questions, but a sizeable proportion found it difficult to imagine no other help would step in. Certain groups, such as those with more ADL problems, found the questions and the instruction to assume no help would step in more difficult than others. Analysis of interviewer comments indicates that this may lead to underestimation of quality of life in the counterfactual situation. The ASCOT counterfactual score was associated with needs-related characteristics, providing some evidence for its construct validity. The estimate of the impact of social care is slightly higher than the best available alternative, but the estimates are significantly correlated at 0.45.

Policy implications: Compared against the best available alternative approach, this ASCOT CSEPP method provides reasonable estimates of the impact of English social care interventions and is relatively cheap and easy to implement. There may be some bias among people with high needs and more research into sources of bias is needed. Nevertheless, these findings indicate that it may be a useful tool for policy development, enabling researchers to establish quickly and fairly cheaply in small scale studies primary evidence about the potential effectiveness of long-term care interventions.