Oxford Science meeting 2

From TC Open Forum

Jump to: navigation, search

Back to the Introduction and First Two Sessions (1 of 2)

Contents

SESSION THREE: 9.15am Tuesday

SUMMARY OF SESSION

The morning opened with a brief recap of the previous day, in particular looking at the consensus statement produced at the end of the second session - this led to some small changes. There was discussion about the costs of doing an RCT along with the feasibility of funding. The time frame was debated with the idea of following a staged approach but it was thought that there was limited time and RCT evidence, whilst not guaranteeing funding, was probably a necessary pre-requite for the funding of TCs.

Mike Crawford then presented ‘The Oxford Plan’ (co-authored by Steve Pearce) which outlined a proposed RCT taking place in an Oxford day service. The history behind the plan was outlined with some evidence from the pilot project being used. The aims of the proposed study were detailed: the primary outcome being that the TC treatment leads to a reduction in cost associated with use of healthcare resources. The methodology, population and design were explained with additional detail regarding the rigorous randomisation process. The active treatment was to be the 4-day a week DTC in Oxford, with the control treatment being an ‘enhanced treatment as usual’. The follow-up procedures and outcome measures planned were considered, along with issues on sample size. The plan was reviewed to see if it “stood the test” of the criteria set out thin the previous session, and obstacles were discussed.

The open discussion then started by considering the importance of treatment fidelity issues and how they relate to using the Community of Community standards, with agreement that this is needed to provide external validity and as a precursor for moving onto multi-centre studies. The sample size was discussed in relation to the number of referrals the Oxfordshire service has. The ethical concern of randomising people with more severe PD to a less intensive (control) treatment was resolved in view of there being no evidence relating severity of PD to the intensity of treatment needed. The randomisation process and outcome measures were agreed: there was some discussion over whether this could be a 3-armed study with Control v Intensive 4-day TC v mini TC but this was thought too complex. There was much discussion over the primary outcome measure being cost: this is a key driver for policy makers as well as representing the ‘deranged attachment’ of patients. Cost and clinical outcomes will be published and since this is just planned as an exploratory study, the use of a Health Economist was not considered necessary.

The group then moved onto generating a consensus statement for the session, and finished by formulating ideas for forming a ‘project advisory group’ from volunteers.

Consensus on Oxford study proposal

  • Day TC treatment, as recently developed in several PD pilot sites, is a suitable

method for an exploratory RCT for PD as a two arm trial – of “options followed by TC” vs “TAU + crisis plan”

  • There is insufficient evidence that high severity of PD requires more intensive

treatment

  • Recruitment of approximately 80 participants could happen over 18 months; this

would be a suitable number for a pilot study.

  • Fidelity should be rigorously established; this could be with checklist of criteria

with an independent rater, possibly as adaptation of existing accreditation process

  • Qualitative analysis would be required for evaluation of the impact of

randomisation on the treatment programme, and quantitative analysis of numbers progressing from options group to TC before and after.

  • Baseline data which have already been collected should be included in analysis.
  • Primary outcome will be service utilisation costs, which needs to be supported by

hospital records (follow-up for up to five years agreed by REC)

  • Service utilisation is a clinical outcome in its own right
  • A project advisory group should be established from members of this workshop

DETAILED NOTES

Opening remarks

Chair: Peter Tyrer

Re-introduction from everyone, including Peter Tyrer (absent Monday)

RH: Summary of yesterday. Two sessions: ‘where we stood’ and ‘difficulties of experimental research in TCs’. We crystallised some points of agreement into a consensus statement (displayed on slide). I’ll go through this point by point for any comments/suggestions.

1: RCTs of therapeutic communities are an essential aspect of the enquiry – with policy makers,

funders and clinicians – for proper development of treatment programmes for PD, addictions and psychosis.

2: A modern democratic therapeutic community RCT has not been undertaken hitherto; potential

barriers have included: ‘Staff inability to hold a position of equipoise’ and ‘Funding bodies’ reluctance to support PD research’.

3: There is now sufficient data in the PD field to design a programme of randomised controlled

studies.

4: A small study to examine the effect of randomisation is the first step required in respect of PD: to

both examine whether randomisation is possible, and whether it has a deleterious effect on the treatment programme itself. 5: Reanalysing data using more sophisticated models may be helpful - in outcome and in population selection, including severity.

6: Matrix based on George’s Criteria.

Stage of evaluation PD community PD forensic Addictions Psychosis

Phase 0 (descriptive)
Phase 1 (quantitative)


Phase 2 (pilot trial)
Phase 3 (effectiveness trial)

RH: Any comments?

PT: Why the word ‘modern’ before RCT?

RH: Think that RCTs have been done before but not rigorously enough – I’ll take modern out.

PT: Key thing is to state “democratic TC” rather than just TC.

PT: Looks good that such harmony was achieved yesterday! Although scientific way is to decide what we need to do first and then get funding, I think in the present state it may be appropriate to consider what funding is available and then tailor research ideas to that funding? This is a key issue in view of previous frustrations. Does Malcolm know of any funding?

MR: The research funding in DSPD has evaporated – other sources will have to be found. Also, there is the question of settings: not sure that prison TCs are the best/easiest place to start with since it is harder to do an RCT. Maybe that day TCs are the best/easiest to start with.

MC: Would the demonstration of feasibility/results from an RCT in a day TC be relevant to a prison setting?

MR: Possibly/probably. Would have to commission feasibility sessions to judge relevance. Ongoing CRACKEMS project – now called RESETTLE. There is the Ashworth RCT currently underway for instance. These are all useful but not fully focused on TCs.

Best to start with the easiest option – a day TC. JG: What sort of costs are we looking at?

MC: Steve Pearce has put together some stuff on RCT on day TC. No fixed sum for research – fund things absolutely have to have, or cost more to fund things you would ‘like’. More money allows better quality resource.

SS: DO RCTs cost more than other designs?

MC: Not really. Some small extra costs re process of remote randomisation etc.

SS: So we will need to fund no matter what design

PT: Funding that is ‘kosher’ will provide resources (peer-review etc) to make sure there is quality research. Has to be a competitive tender. Is there really no money left in DSPD? MR: We may (not in this financial year) be able to release a relatively small sum to add to other funding.

GdeL: Consider using the step approach and fund in small parts. Cultivate interest for funding: stage 1 would be a feasibility study, implementing RCT activities such as randomisation etc? Then stage two would be ‘overlap’ looking at differences in retention/completion and impact of RCT in treatment. Would encourage funding for the next stages. Stage three would then just follow the sample. Might get bits of funding resulting in a position to bid for full RCT funding.

PT: I would be worried that we don’t have the time to do all that in that sort of depth. Time is running out in this territory.

MC: I will be presenting a proposal evaluating a day TC with Steve Pearce. I agree with the business idea of a ‘stepped approach’. However there are also ethical and logistical problems of randomising people without the sample being followed up – what would people be consenting to?

GdeL: What are the ethical issues?

MC: If I was a patient and I was going to participate in a study I would be happy to do so if they were going to discover something about whether the treatment was beneficial or not, but I would feel differently about it if I was being randomized on the basis of finding out whether it is possible to randomize people.

GdeL: Not sure there would be ethical issues – consider treatment benefits. Similar to dosage programme in cancer. Perfectly ethical for first level research gains.

SS: Maybe we should run a single case study of people in treatment? This could ensure they are making progress?

RH: I want to ask Peter what he means by “time is running out”. This is certainly the sensation we have in the field.

PT: The pace of RCTs is slow. If feasibility studies are done first, followed by a definitive RCT then we could be looking at an 8 years timescale. I would be hoping that something substantial could be done in the first stage.

EK: I think Peter is right, although the results of an RCT will not guarantee future funding of TCs – it’s not how things work, not how the decisions are made (the same in physical services).

JL: But is RCT evidence a pre-requisite of funding?

EK: No – consider the money given to the IAPT (Increasing Access to Psychological Therapies) where there is little evidence.

PT/JY: Disagree – IAPT does have strong evidence.

JY: The strong evidence base might not trigger funding but without it funding would not have been given?

EK: It is one of the variables. An RCT will not be sufficient to get funding but it may be necessary. However, it is not the only way of ‘saving’ TCs.

PT: If we don’t do it soon there will be no TCs to evaluate! Brief aside discussion about DSPD funding, competitive sources of funding, etc

Can Oxford design a satisfactory RCT?

Initial ideas from the British TC-RN

Mike Crawford – Presentation “The Oxford Plan”

Review of the matrix from George’s presentation the previous day:

  • Community and addictions TCs are probably ready to do an RCT. Prisons and

Psychosis TCs are not ready yet.

  • This presentation will focus on PD Community Services and is very much ‘work in

progress’. ===A pilot randomised trial of day-therapeutic community intervention for people with PD===

  • Steve Pearce; Consultant Psychiatrist in Psychotherapy Complex Needs Service Thames

Valley Initiative

  • Mike Crawford; Imperial College London
  • Others.

BIGSPD Nottingham 2006

Another meeting discussing whether it is possible to do a TC trial and how/ where to get funding……. A lot of the barriers to doing a RCT of a day-based treatment are solvable. The research done by Bateman & Fonagy (1999) showed that a randomised trial of complex day-based treatment for people with PD is possible Speaking to Steve he raised interesting point in that the study had little or no funding – so why don’t we try and do the same in a democratic day TC? I will go through the Context (quickly) and Evidence base to date (quickly) since these were discussed yesterday. Then will look at our proposed design and methodology before requesting questions & comments.

Therapeutic communities

  • The history of TCs (Rapoport, 1960)
  • Democratic TC – the model has been examined and described (Haigh,1999)
  • Observational studies have demonstrated improved health and economic outcomes

(Chiesa et al. 2002; Davies & Campling 2003)

  • Randomised controlled trials (RCT) of DTCs have not been conducted (Lees et al. 1999)
    • none of this is new and was discussed yesterday.

Residential Democratic TC for adults with PD

We have Observational data and testimony of service users BUT

  • Take on less than 30% of those who are referred (Rutter & Tyrer, 2003)
  • Costs are substantial
  • Drop-out rate is high (Chiesa & Fonagy, 2000)
  • Step-down model as or more effective than residential alone (Chiesa & Fonagy, 2000)
  • Challenges of RCT – motivation and ‘standard care’
    • likewise, all this was discussed yesterday.

When the focus for TC treatment for PD was first considered it was in residential settings where the logistical barriers to doing RCT were much greater (although I don’t think they were insurmountable).

Community based DTC

More recent is the development of community based day TC which try and replicate a residential model in an outpatient setting. There was a boost to day TCs as part of the expansion of PD services after the ‘No Longer a Diagnosis of Exclusion’ document in 2002 – 11 pilot services scattered around England were funded of which 6 included TC principles and day TCs were established in Cumbria, Nottingham and the Thames Valley.

  • Replicate key principles of

deinstitutionalised approach to staff-patient interaction, emphasis on personal responsibility, while optimising opportunities for sharing and participation in communal activity within the confines of an out-patient programme

  • Developed through pressures on

residential TCs and expansion of communitybased treatment following ‘No longer a diagnosis of exclusion’ (NIMHE, 2002)

  • More amenable to experimental evaluation?

Qualitative findings

The evaluation of these services (mostly a qualitative piece of research) suggests that the TC treatments were amongst the hardest for people to us but gave some of the greatest benefits. It was the hardest to use (role of staff, use of medication)

  • ‘you know, we need some kind of response and, if it was made clear initially that

those responses don’t exist, it would be easier to deal with. But it is not and it is so frustrating.’ (FGSU29)

  • ‘I am just not happy about the medication thing … I mean, I am living alone. I can’t

not have meds … What if I start getting aggressive, all sorts of things that some of the medication might dampen down and I have got to go home on my own.’ (SU49) But gave the greatest benefits

  • ‘I did feel quite low on Monday and then left quite cheerful because the one thing that

I am feeling is a belonging with some of the others.’ (SU49)

  • ‘I know it sounds strange, but we are like one big family, like when we’re all together
    • everybody helps everybody else.’ (SU47)

Quantitative findings

  • Large number of referrals – 2 to 15 per month with the greatest number at TVI who had

enormous numbers of referrals (so Oxford may be a good place to do a study!)

  • Levels of uptake (60%) and retention (85%) greater in Day TC than those reported by

residential services

  • Impact on mental health and service utilisation: not collected centrally – but local data

supports view that levels of contact with A&E and in-patient mental health services decrease

  • Following completion of the pilot projects, data still being collected across 5 sites,

using CORE, SFQ, service utilisation (TCRN)

Developments

Since the end of the pilot projects there have been some further developments. One issue we considered was that of outcome measures: we conducted a Delphi exercise to get a consensus amongst experts, providers, clinicians, service users and we got people to say what they thought should be the outcome measures for PD services. There was a consensus on the top 3 items from all groups being outcome measures on the quality of life, social functioning, and mental distress (although the prioritisation of these 3 differed between these groups).

Another important development is that even during the course of the project some of the services being provided were becoming less intense and there was pressure from Service Providers to expand the catchment areas of services – necessitating some reduction on intensity of service being provided. This development towards less-intensive treatment is also being seen in other areas, such as mentalisation based therapy, where treatment is moving from an intensive 5 day-based treatment to maybe once a week group or therapy. Also move towards “evidence-based treatments” such as DBT; CBT; CAT; MBT which are less intensive rather than more intensive (Crawford et al, 2007; Karterud, 2007).

These developments are great in respect that less intensive treatments are easier to evaluate but they are damaging in relation to one aspect which is Community. TC principles seem to be those that are least easy to sustain in this changing service environment.

So, onto the actual study itself which is: ==A pilot randomised trial of day-therapeutic community intervention for people with PD or... ‘TACIT’ – TherApeutic Community Intervention Trial Aims==

  • To examine whether treatment in a day therapeutic community among people with

personality disorder leads to reductions in costs associated with use of other healthcare resources

    • reason why that is the primary outcome measure is that this is the area where

greatest differences have been shown

  • To determine whether treatment in a day therapeutic community leads to improved

mental health, increased social functioning and improved quality of life

    • important secondary measures

Method

  • Design: Two-arm, parallel, randomised controlled trial
  • Population: people aged 16 to 65 who are registered with a GP in the Oxfordshire area

and have a personality disorder (assessed by SCID-II)

  • Recruitment: From people in contact with mental health services and living in

Oxfordshire

  • Exclusion criteria:
i) A primary diagnosis of a psychotic disorder, alcohol or drug dependence
ii) A degree of learning disability, or intellectual impairment which prevents use of

DTC services;

iii) Unwilling or unable to provide written informed consent to participate in the

trial.

Active treatment

Active treatment = Day-DTC provided by the Oxfordshire service which has 3 main components:

  • an entry component
    • this is the weekly options group and individual sessions; visitors slot at the TC
  • a main component
    • 4 day TC (for up to 18 months) including creative group, psychodrama, small groups,

objectives groups and large groups. Cooking, shopping, eating, working and playing together PLUS ‘out of hours’ telephone support

  • an end component
    • being access to weekly peer support group

Control treatment

A key criteria for a study like this. If there was any money there may be a different kind of control treatment… Plan that the control group would receive ‘Enhanced Treatment as Usual’ (TAU) which would be on an individual basis and using a service user derived ‘care plan’ (much like the SUN project) identifying crisis plans etc. There would be follow-up reviews, on a monthly basis.

Randomisation

  • Telephone randomisation would be remote (for higher quality) undertaken

independently by the Department of Psychological Medicine, Imperial College London. The clinical team would have nothing to do with the randomisation.

  • Randomisation ratio of 1:1 and a minimisation scheme in order to balance potential

confounding variables that we know to be important: age, sex and baseline service utilisation and particularly severity across intervention groups.

  • Once a study subject who meets inclusion criteria provides consent to take part in the

study, a trial coordinator will randomise the patient and then contact the local treatment team to inform them of the patient’s randomisation status

Follow-up

Follow-up is initially planned at 6, 12 and 18 months following randomisation. It will be key that the outcome measures used will be the same as in the pilot and currently in the TC Research Network. The battery of tests is relatively short (taking 30-40 minutes). It is planned that the member of staff carrying out these assessments would be blind as to the participant’s allocation status (more cost but better quality and avoids criticism of unmasked ratings).

Outcome measures

These will be the same as those used in an ongoing cohort study among 5 day TCs

  • Primary outcome: Direct healthcare costs inpatient, outpatient, A&E

PT - how are these going to be measured? By asking the patients themselves to give an account of their contact with A&E with inpatient services being verified by electronic record of inpatient treatment.

PT - What about societal costs (benefits etc)? This is the difference between a cheap trial/feasibility pilot and a more costly trial. We think people do end up using less social care services and may end up in employment but need to employ a researcher to get those cost savings measured.

  • Other measures:
    • A measure of mental health: the CORE-OM
    • A measure of Social function: SFQ 8 item questionnaire
    • Self harm questionnaire: modified Davidson measure (2000)
    • Quality of Life measure: EQ-5HD

Sample size

In this kind of exploratory study, sample size is less important. Two factors worth bearing in mind:

1 – in the RCT of mentalisation based treatments in partial hospitalisation day hospital

study statistically significant differences were found on a sample of 42 people and

2 - if the treatment effects of day TCs are equivalent to those of inpatient TCs you don’t

need a large sample size in order to find significantly significant differences in treatment outcomes.

  • Previous studies have shown that patients treated in DTCs have a mean number of 45

(SD = 71) days of inpatient psychiatric treatment in the year prior to referral and 12 (SD = 22) days in the year following referral. Davies S, Campling P. (2003) Therapeutic community treatment of personality disorder. Br.J.Psych. 44, S24-S27. We think about 70-80 people randomised may be sufficient to get a really good indication of the effect size of the intervention.

  • A sample of 76 patients (38 DTC and 38 TAU) would be required to have 80% power and

5% level of statistical significance to demonstrate a reduction in the mean number of inpatient days of this magnitude.

  • As our primary outcome is based on examination of routine records we estimate that

the proportion lost to follow-up will be small (10%). We aim to randomise 85 participants over the course of 24 months.

Data Analysis

All of this is ‘standard stuff’ for an RCT.

  • All primary statistical analysis will use the intention-to-treat principle.
  • The difference between the arms of the trial will be estimated using appropriate

parametric and non-parametric tests.

  • Analysis of covariance will be conducted in to adjust for baseline variables considered

of prognostic importance (with small sample cannot rely on randomisation to balance the treatment arms).

  • The effects of randomisation status on secondary outcome measures will then be

examined

  • Cost and cost-effectiveness analyses will be undertaken. The number of patients

dropping out before the assessments at 12 months will be compared between the arms of the trial, and sensitivity analyses undertaken to consider their potential effect on the overall results of the trial.

Does this plan stand the test?

General discussion:

Yesterday George presented his criteria needed to run an RCT:

1. Sufficient evidence so far
2. Knowledge of active ingredients
3. Assessment of treatment fidelity
4. Appropriate control group
5. Minimise overlap (between active

intervention and control treatment)

6. Appropriate assessment
7. Consider the importance of self selection
8. Have appropriate after care

My sense is that this proposed study does ‘tick’ all these boxes. We do have sufficient knowledge/evidence so far, we do know a good deal about the active ingredients (a real strength in an RCT), that it is possible to assess treatment fidelity (although there is a cost associated with that since it will require recording of groups etc).

I feel happy that our proposed control of enhanced treatment as usual is a sensible one: it is achievable and is something that when you are trying to recruit patients to a study and say ‘would you be willing to have a crisis plan and regular follow up compared to a weekly Options group’ is a reasonable initial choice to ask people to make.

Assessment is deliberately kept short to make study feasible. Regarding the self-selection criteria, we know that we will not be working with everyone with a personality disorder – only a population of people who have some degree of motivation to engage in treatment, and that is reasonable clinical decision to make.

GdeL - why not include a self-report measure to assess level of ‘self-selection’ This would be helpful to inform clinical knowledge as well – could help address the question of ‘at what stage of motivation is it appropriate to make this referral’. May also explain difference between those who stay and those who drop of TC. The issue of aftercare: those in control group will have less aftercare but will have continuing access to services.

GdeL - Aftercare is absolutely essential to have sustained recovery so need to pay a lot of attention to this

GdeL - May have difficulties equating control condition and treatment condition in terms of intensity

Yes – the study will be open to that criticism that we are comparing a higher intensity treatment to a lower intensity treatment but it is that intensity which is a key ingredient of the TC treatment. I cannot see a way we can construct an alternative control treatment of the same intensity as TC without substantial investment in an extra service.

GdeL - It is a very vulnerable research position to be in since you are talking about number of client contacts: clients who are willing to go to a 4 day a week service will have an enhanced motivational element. Initially the contact in the treatment group will be a once-a-week Options group and not all of these will go onto the intensive 4 day a week TC, although this is accessible to them. It goes back to the incremental approach: we need to establish a treatment effect at this earlier ‘pilot’ stage in order to get investment to fund a more intensive control treatment. But it is a real change for an RCT in any intensive treatment.

RH - You have already mentioned that services are becoming less intensive and there is quite a movement for setting up ‘mini TCs’ which are 1 or 2 day a week. I wonder whether that would be possible to use for control?

PT - But that would give overlap of the TC elements.

PT - ‘Treatment as usual’ can vary widely from intensive or nothing so it is good to have this set enhanced TAU to control.

MY – Re the data – there be an analysis of contacts so stratification of data should work.

PT – Problem in that variation of costs is so high and unfortunately leads to significance being quite low If we randomise 70-80 people and you see clinically significant but not statistically significant differences in costs but you fail to demonstrate changes in other outcomes: given we have studies already which do show differences in outcomes, this study will not do the day TC model any favours!

EB – How exactly does the randomization operate? The member of the clinical team in the Oxford service will assess someone as suitable, checking diagnosis, exclusion criteria etc and then pass on details either by telephone or email to a trial co-ordinator based in London and then that person will dial-up an automated randomisation service. They will input the criteria used for the minimisation process and will get a number out saying 1 or 2. They will then go back to the clinical team to say which intervention the person will receive.

VJ – If the longest follow-up is 18 months, this won’t cover people all the way through from starting Options to after TC? That’s true. What we would really like is 2 or 3 years follow-up but considering this as a staged approach and increasing follow-up time costs more (something would be good to do in a future trial).

VJ: Does open up criticism that use of services, medication, A& E etc will increase after the patient leaves TC Agree – in staged approach at the end of the first year can see drop-out rates and treatment effects and then apply for extra funding to follow these people up.

GdeL: I would like to ask you to reconsider Rex’s idea of using ‘mini-TCs’ - could we compare the different intensity of day TC model, high intensity versus low intensity. At a second stage, yes, but for the initial trial it is more important to demonstrate treatment effects per se.

EB – How exactly does the randomization operate?

PT – Could use a 3-armed trial: High intensity TC v Low intensity TC v Control Much harder logistically!

Obstacles

Funding :

  • There was a proposal to MRC some time ago which didn’t get funded.
  • In last 3 or 4 years been involved in proposing for funding for large PD project – didn’t

get funded

  • Possible sources MRC, trial platform, HTA (suggestion), NIHR programme grant (no

longer research money embedded in clinical funding)

  • Leaves open the ‘Bateman and Fonagy DIY approach’ to start with little funding.
  • Around 100K would definitely get something going and around £50K could probably just

about get it on the road.

Conclusions

  • Feasibility of a trial of day TC can be tested
  • Small scale funding would help and enable the process to be started.
  • It would not provide the evidence for a massive investment in day TCs but would add

to evidence base for effectiveness of treatments for people with PD, putting it level with services such as DBT.

  • Support search for funding for a proper pragmatic trial (which would require proper

funding)

  • May help to establish the effectiveness of an alternative mode of intervention for

people with PD

What treatment would people get if they refuse randomisation?

It would have to be the enhanced treatment as usual – or nobody would agree to be randomised.

Will it work anywhere else?

Discussion of generalisability:

Peter Tyrer (Chair) opens up discussion for thoughts and questions.

RH: Let’s start with Mike’s conclusion. Can we agree those statements?

PT: Not sure about using the word feasibility. We could use “pilot” or “exploratory” instead.

MC: Exploratory is good – pilot may get confused with ‘pilot services’

PT: Is the day TC model relatively well defined now?

RH: Yes it is and has standards from the Community of Communities to ensure fidelity of program.

GdeL: In the US we call ‘day TC’ = ‘modified TC’ to signify you are not doing certain things that you would in the residential model: puts us in a better position to support the model as a 24/7 model.

MC: Has the service development reached a level where it is sensible or desirable in terms of treatment fidelity to record, say, committee meetings and have those listened to by someone who has been involved in setting standards to rate them in terms of their adherence to a model?.

RH: They are already recorded by written notes as part of the audit for the Communities of Communities.

MC: How are judgements made about whether the notes from the meeting adheres to a TC model? Are they TC experts? Who reads those accounts?

RH: They are reviewed by a researcher expert in TCs, and is done as part of the accreditation review every 3 years.

MC: Is it possible/logistical/desirable to have more consistent judgement of fidelity by assessing recordings?

GdeL: Yes – in the TCs for substance abuse in the US recording are assessed against 10 item scale. Not just whether meetings take place but also the way they are done – whether the TC elements are included.

RH: Those things sound very similar to the CofC standards. But I feel that listening to recordings doesn’t necessarily show something has the ‘TC Culture’

MC: As long as this was taking place at somewhere we all have confidence in the model, then it’s not about convincing the researcher but is all about external validity,

GdeL: I think someone needs to be physically present to assess ‘community’ (and this can be good clinical training) but they need a checklist and to be a good observer.

MC: I agree – these are all aspects of rigour that make high standard research that people will accept. What do you think would be the response to the request to tape record community meetings for the purpose of evaluation?

VJ: I think observation would be preferable to audio recordings partly because audio tapes won’t purvey the full image – but I don’t think there will be any problem getting agreement for this observation.

EB: It is also possible to quantify tapes by using various methods.

MC: One of the aims of an exploratory study should be to refine the methods for assessing treatment fidelity.

GdeL: What are the key elements of the day treatment TC model for PD?

RH: There are various elements – mostly encapsulated in the C of C documents, and a recent article in the ATC journal.

JY: Given that the TC you are evaluating has already been assessed on these criteria, are we gaining anything my doing an additional check on fidelity?

MC: I think we would be confident – but need to answer external concerns.

JY: But there is some accreditation?

MC: External

BS: In the prison service there is accreditation – need to be physically present.

MC: Also a strong argument to doing this would be for when we move onto next phase multicentre studies – fidelity is critical when evaluating other centres.

JY: How were the criteria established?

RH: They are based on a 6 year audit cycle, refined and modified each year, and represent a consensus between patients, staff etc.

MC: For the pilot study: an observer can be a member of staff.

VJ: How are you going to look at whether the RCT process affects treatment, as Steve was discussing yesterday?

MC: For the really really cheap study… Need additional resource to collect the qualitative data. The main point would be to see if the randomisation process affects the treatment process which can be done by interviews. The data of qualitative data is time consuming (so expensive). We could do the study without doing the process evaluation although it would be a shame.

GdeL: Has the study been launched yet?

MC: No, we are in the process of getting ethics approval and R&D approval and Steve is talking with the staff.

PT: Are we happy with the numbers - n=75? This would mean recruitment over 18 months and a total study length of 2 ½ years.

FW: Priority to do a pilot to assess impact on treatment. Need to do qualitative

assessment now for comparison position. GdeL: In the next 12 months we could get baseline data. Will be a retrospective impact? Assess key indicators now.

MC: We are in the fortunate position of having baseline data both qualitative and quantitative data on what service users feel has been achieved by using the service, also what service providers believe has worked well an less well and some quantitative data on characteristics of people using the service.

GdeL: Lay out some impact on patients now to give rigorous basis for analysis – substantial usage for comparison.

MC: We have that baseline data already for the 80 (approx) people using the PD services in Oxfordshire.

FW: Steve Pearce yesterday was talking about his concern using ‘intent to treat’ data.

MC: Steve has belief and some evidence that attendance at the weekly Options group has some positive beneficial effects. Already have data on number of people who are assessed and offered the options group who attend – that figure is in the region of 75-80% of people.

FW: So what is Steve’s concern?

MC: Not sure! A minority of those who attend options go on to attend TC

FW: So is this study just an assessment of the Options group then?

MC: No, since the Options group contains elements of TC and includes a visit to the TC to see what it is like – so different

RH: It’s a trial of the whole of the Thames Valley model really – which is why I wonder whether it is replicable elsewhere since the engagement phase is an integral part of the model. What you are doing is avoiding randomising people when they are in a group together and may have made significant changes already

MC: No – generalisability should not be a problem - so many services are adopting the approach of having a less intensive initial phase of intervention where people can find out what the treatment process is like and make an informed decision about whether to use the intensive intervention.

PT: But this is an important question – particularly the pragmatic questions as to whether is it generalisable to other parts of the country.

RH/MC: A preparation group is standard for the other day TCs in the country e.g. Nottingham, FDL.

GdeL: Need a profile of severity of patients and consider where we would want to clinically refer?

MC: We have a tiered approach, whatever the severity is. We start with less intensive treatments for all and then look at the intensity of treatment needed.

GdeL: There are ethical consideration regarding randomising people with severe PD to less intensive treatments.

MC: But whilst there may be evidence in substance abuse that higher severity needs more intensive treatments, we don’t have any evidence in PD to say that if severe PD we need more intensive treatment.

GdeL: There may still be some struggle with internal contradictions i.e. severity & intensity.

MC: And that will be tested. Possible those in the weekly control treatment may do better – the intensity of 4 day a week day TC may not be right for some people.

PT: It seems like there might be a case for a 3- armed trial: Control, Intensive Day TC and mini (less intensive day TC).

VJ: There are also practical differences (e.g. location) in randomising a population between Day TC and mini-TC.

EK: Mike is right to keep it simple.

JY: But would be better to keep intensity the same both in active intervention and control treatment

MC: There is just no service available to us to do this.

PT: We can’t control for intensity without introducing some overlap. We really need to make the control treatment as less likely to produce effect.

MC: Ultimately ‘project match’ is where you want to go- but we are a long way away from asking those questions as to ‘who does best from what treatment for PD?’

COFFEE BREAK

PT: We are considering a 2-armed study: options group & control group

FW: Why not do the randomisation in the Options groups?

MC: Yes, but that would be much harder to reach equipoise over.

PT: Is everyone happy with the suggested outcome?

MC: The major outcome is cost savings. This is also a key driver for policy makers and purchasers

RH: I see the cost being equal to the deranged attachment

MC: Need length of follow up to really assess costs.

JY: Surely the primary outcome is treatment efficacy not cost?

PT: Demonstrating treatment A is better than treatment B has little impact on accountants unless it is cheaper

EK: The Layard money came purely through the Treasury – not the Department of Health – so purely economic resources.

GdeL: Endorse the importance of cost. There is a convergence between clinical improvement and cost.

MC: There may be ethical difficulties examining records for 5years follow-up

VJ: Self-report or from records?

RH: Needs to be supported from hospital records

JY: For the research to be published, will cost outcomes be enough?

MC: We would publish both primary and secondary outcomes.

JY: Is there not a danger that finance will overshadow clinical effectiveness?

PT: As a publisher, the two arms (cost and clinical) can be combined

MC: Use of services such as A&E are clinical outcomes as well as cost

Gdel: Will we have costs data client by client? Costs as primary outcome purely a policy approach. There is a correlation between clinical improvement of clients which results in less costs.

MC: This is just an exploratory study

JY: Do you need an expert for Health Economics?

MC: Not for such a small scale study with published costs.

JY/RH: It might not cost any more- you could get someone interested and enthusiastic

EK: The finical cost is easy to determine – economist not needed

PT: As a publisher, a combination of finance and clinical data is needed

FW: Need to consider length of time needed and people’s attitude changes by publication date. There are four components to ‘tick’ in the health economics box:

1. less expensive, slightly less effective
2. more expensive, slightly less effective
3. less expensive, slightly more effective
4. more expensive, slightly more effective

1 and 3 would be very good, 2 very bad. Question whether component 4 would be OK for policy makes?

Consensus statement discussed.

GdeL: Can this group follow what is happening re the Oxford Study? Could we be voluntary members of a project advisory group?

MC: Are there other volunteers? Governance of the research would certainly benefit from an advisory group

PT: Also data management and ethics would benefit

MC: But ‘expert’ panel comprised of members of this workshop?

GdeL: Is there a hidden agenda? The origins of the meeting re universal difficulties of TCs. This is an example of the commitment of science in TC research – bringing together a ‘skilful’ group.

Consensus reached.

SESSION FOUR: 11.30am Tuesday

SUMMARY OF SESSION FOUR

The final session examined methodologies, alternative to the ‘science’ of randomised control trials, which could produce findings to influence treatment, policy and science. Different statistical approaches were examined, including multi-level analysis which was though particularly appropriate for TCs, which led to discussion as to what data we already had (particularly the large cohort data from forensic settings) and whether it would be useful to re-examine or re-analyse it.

Qualitative approaches were thought particularly relevant to try and capture the hermeneutic aspects of ‘what goes on’ in a TC and to record and possible quantify the emerging phenomena. There was debate over whether we ‘already know’ what the essential ingredients of a TC are or whether the area is complex and further work is needed to identify those key elements: qualitative data from the pilot projects indicated the importance of aspects such as ‘peer-support’. Despite the ‘unstable and chaotic’ nature of TCs which constantly change themselves, it was thought that although complex, elements could be defined and investigated: “if it exists, it can be measured and evaluated”. There was also feeling that service-user led research could be particularly valuable although it does require time for training and support.

It was recognised that there wasn’t time to address all the factors around alternative methodologies and that this could possibly be the subject of another meeting. There were also issues such as different levels of treatment intensity (from mini-TCs and micro-TCs) and TCs for psychosis that it was felt were not fully discussed due to a lack of time. After further debate about the position of RCTs with policy makers, it was agree that RCTs were probably ‘necessary’ but certainly not ‘sufficient’ and additional research and other work should take place to complement the RCT. The consensus statement was reached before the meeting was finally closed.

Consensus on alternative designs

  • RCTs are necessary but not sufficient for the policy-makers and clinicians and

service users

  • Different approaches are required to explore the active ingredients.
  • Re-examining the data to assess changes at different levels: time series, multilevel

modelling etc.

    • Econometric research
    • Service user research
    • Qualitative research
    • Action research

Questions on alternative designs

  • Can we measure change? Behaviour, yes - internal change?
  • How important are other methodologies for influencing treatment, policy and

science? What is the ‘killer fact’? Follow ups

  • The work of this group will be presented to EWODOR
  • The process of this workshop can be used elsewhere; G De L will attempt to set up

a similar one with leading researchers and policy-makers in USA

DETAILED NOTES

Chair: Jenny Yeind

What else will produce useful findings?

Methodology to influence treatment, policy and science.

JY: This final session follows our looking at details of the proposed RTC. What other (non-RCT) methodology could be used to influence treatment, policy and science?

EB: There exists a body of methodology and researchers in this field. Maybe we should have an international meeting?

MC: Could we piggyback onto an existing conference?

EB: There are more scientists in the TC field than is recognised. Questions are lurking everywhere.

JY: When looking at other methodologies, what other qualitative approaches are we considering?

GdeL: As well as other methodologies, I’m particularly interested in hearing about different statistical approaches.

MY: We have looked at large scale data in a medium secure unit, to evaluate effects of help in changing behaviour. We took trends over time and age demographics: if there had been no effect, the ‘post care’ trend should be the same as the ‘at admission’ trend.

JY: That type of analysis has advantages: you don’t have to see differences between groups but it is possible to see the change in selfcomparison, comparing slopes at different stages.

MY: We used multi-level statistical tests.

RH: And this multi-level analysis is very appropriate for TC set ups.

PT: I would say that it’s not superior to RCT modelling but it is easier to use.

JY: Could we broaden this discussion into a more general session about different methodologies

MY: I don’t know where this is the hierarchy of evidence?

MC: Multi-level modelling is very useful for day TCs. But there is a challenge in a day TC in that there are not enough levels with sufficient numbers to analyse. How many clusters at the level of service do you need? MY: That depended on heterogeneity: you want to be able to ‘disaggregate’ homogenous groups. In terms of meta-analysis, the random effects model (used where samples are not homogenous) overlaps with the multi-level analysis.

EK: Is the data around now to apply different new statistical modelling to?

MC: TCRN is getting there in terms of quantitative data. ‘Learning the lessons’ only has data re retention/attrition in services.

MR: In the prison service we now have sufficient data for cohort studies, and psychometric measures now routinely used. Is there a

possibility of tracking quite large cohorts? JY: I think a key point can be made as to the appreciation of what data we already have and what we can get out of it, possibly by different analysis.

MC: A serious feasibility study is what is needed.

KH: Another way of thinking about it is to consider what are the issues about improving the science of TCs and putting together a cumulative body of knowledge. How broad are the foundations of the science of TCs at present? How can that be built upon? Methods should match questions. If you already have good data sets that raises opportunities for you too find out more. Is there a sense of ‘this is what we need to know next’ in TCs?

EB: I would like to know how far are we on the quantification of qualitative hermeneutic data since the TC is existentialist and traditionally used qualitative hermeneutic analysis. I never see studies on this in the journals but would be an interesting thing to look at since it is so close to the essence of our work.

JG: This might be something picked upon if observation of, say, community meetings goes ahead.

SS: We can look at transcripts but also need to look at non-verbal as well. It’s the ethnographers question – what’s going on here? If you look at that across a range of setting you can see what is common to settings and possibly link to outcomes?

PT: What I find difficult in this field is that if you look at the sort of data you are talking about here – “what’s going on here? What’s important?” - you would have thought that we’ve had more than 50 years of experience with TCs and surely any really important ingredient or component in the effectiveness of treatment that is there would have been identified by now. If it is something different e.g. “what’s going on in DSPD unit” I can accept that question but we’ve surely had long enough to show this in TCs?

RH: I think not! It is something very difficult to define and it’s about the quality and nature of relationships which more to do with values and hermeneutics than behaviour.

JG: Interpretation is a rough translation of hermeneutics!

MC: I don’t come from a TC background but it strikes me that peer-support certainly seems vital from the pilot projects. There is a lot more to TCs than peer support though and some other treatment modality have peer support.

RH: But behind that there is a quality of relationships where peer want to support each other and see it as part if the ‘treatment as method’.

GdeL: We do have an idea of the critical elements of TC and it did take us 50 years! We have good hypothesis now of the active elements – operationalised in ‘community as method’. It took a long time to understand a very complex approach. We now have a better understanding of what in a high fidelity programme that produces change. My biggest difficulty is that in the earliest days we were criticised for not doing RCTs. We didn’t know then what the active ingredients of the TC was. Had to break down the questions one-by-one we had to ask questions like:

1. Who are these people coming into

treatment?

2. How do they do? Do they ‘improve’?

Asking again: do we have a TC-PD data research history? Do we have good outcome variables? In terms of alternative methodologies need to go back to the data that exists and then go ahead with ‘little exploratory RCT’

JY: There seems to be two things here: 1. issue of a lower level, pre RCT investigation 2. issue that’s ‘what’s going in’ – an understanding of what matters

EB: How can we quantify the ‘phenomena’ so that it becomes statistically manipulative and acceptable at the highest level.

RH: I would says that its quality is more unstable and chaotic: TCs do not stay the same, a community will change itself over the months and years so the procedures and policies that you see today will be different from those there 3 or 4 years ago. The field is doing the same (e.g. 1960s residential TCs, now day units and mini-TCs). How do we cope with that innovation - how do we research that change?

GdeL: Different for substance/addiction TCs. Synergistic analysis is complex but very useful. Is there ‘treatment’ or not (consider AA who didn’t term their work as ‘treatment’ but as support, avoiding criticisms of treatment without research evidence). If we get into the concept of ‘community as method’ then break that down in methodology terms then I think that the analysis is complex but feasible.

JY: Are you saying we should be investigating the active ingredients of ‘Community as method’: ingredients or do we know them already?

GdeL: I say we have good hypothesis – but Rex says it’s too elusive?

EB: This methodology cannot go enough because we have phenomena, not only behaviours. Maybe one day we can explain the phenomena by neuro-scanning to see what is going on in the brain?

JG: The change we seem to be talking about seems to be changing people’s behaviour. But I’m interesting in the changes in peoples understanding – the meaning they are making out of their experiences. This may be more significant in dealing with people with psychosis: they may have behaviours it would be useful to change but it may also be that behaviours may not need changing but their understanding of themselves may need changing. Of course, for TCs in prison and addictions perhaps the emphasis on behaviour changing is much more to the fore but I suspect that their sense of themselves and their identity and meaning of their experiences will also have some key part to play.

JY: From a psychologist’s viewpoint, it is likely that understanding will drive behaviour changes

JG: This may not be the right place for it, but we need to address questions such as ‘what we mean by knowledge’ and whether we can know about everything – these are perhaps more fundamental and underlay the assumptions being made in these discussions.

MC: There is some evidence. Qualitative data pilot from SU indicates main factor is peersupport.

PT: What bothers me here is what I regard as the ‘abandonment of enquiry because something is too complex’. George’s example of the AA model – I think it is disgraceful that they push this 12 step model without any proof all these elements are needed. Complex behaviours such specific inputs or peersupport being a constant feature - these can be defined and investigated. If it exists, it can be measured and evaluated. I can’t believe that it isn’t possible for all the important elements of TC to be dissected and measured.

GdeL: We now have to deal with the complexity which includes measuring all the essential ingredients. The AA stance was strategic – science could interfere with what it is doing. There had to be evolution and maturation for substance addiction TCs before research - no money in USA for funding for process research.

Jenny Yeind: start to formulate the consensus statement.

MC: One thing: we had that matrix and I think we’ve talked a little about observational methods in a prison context, but I’m aware we haven’t really thought about the role of TCs and psychosis – I don’t know whether we need to do that?

JY: We haven’t – but I think we need to focus on the brief for this session which is ‘methodology beyond RCTs’ and maybe psychosis can come into that.

JG: Bringing together points Mike and George made about funding: we have a number of member organisations represented here: should we not take a more business like perspective on research and budgeting for it and generating research funds ourselves rather than relying on statutory funding and making applications there. Shouldn’t we be trying to target the commercial sector for sponsorship and so on. Shouldn’t we have a strategy that goes across organisations like ATC, CofC, European Federation etc and have some research fund strategy that comes from this meeting?

MC: The funding that was embedded within NHS services for research has been taken away. It may be different in the voluntary sectors but my sense is that within the NHS we can’t take some of the money provided for the clinical service and use that to evaluate it – paid to deliver it only.

RH: But who could we get apart from drug companies? Does anyone have friends in the city….

PT: One of the problems generally now is that if the funding agency is seen to have a declaration of interest, then it subtracts from the quality of the research. There would be an obvious interest in publishing positive results. To get research that is listened to has to be independent. The organisation can highlight the issues to the potential funders. There is something in the NHS called Research for Patient Benefit but we have to make a very good case for it.

JY: I’m not clear whether we have agreed as to whether we need to be exploring the active ingredients or not. Have we reached any consensus on whether we think it is worth doing more work to explore the processes, or whether we have that work already?

EK: Outside of Mike’s process we have mentioned other methods that can be used to explore the interaction/ process.

JY: But I’m not sure whether we have agreed that it is necessary or not?

EK: If you think that from recent literature, it is only 6-10% of change from any psycho-social intervention is specific to the intervention (with some exceptions like DBT).

JY: So 90% of the change is non-specific to the intervention? What’s there reference for that?

MC: I’ve heard this so many times and often wondered where it came from – I don’t believe it…

EK: I will send on references! JY: What else do we what to say about methodology? Are there any issues arising from other approaches? We did raise the question as to whether we have data that’s worth re-analysing? Is that important. MY: We were talking about methodology for assessing changes at different levels. When I talk about models it is about changes based on suitable large data and reasonably wellformed outcomes – that’s why I was thinking maybe forensic setting would be more suitable for this analysis. George has talked about more basic changes at the very bottom level – individual changes and changes on psychological basis as well as behavioural changes. There are large projects in education where we have applied multiple methods to evaluate different levels of the project – both quantitative and qualitative. Then there is action research as another method which we could investigate – the participants in treatments are part of the research.

RH: I think every TC is an action research project that never writes itself up…

MY: Then you need to get some structure into it – we can investigate this

EK: There are other methods for evaluating interaction – e.g. there is a coding structure to evaluate motivational interviewing and there are other methods?

MY: I think what is very important is the experience of being in TCs so best if researcher is experiencing what goes on in the TC since that information does not come across in statistical techniques.

BS: Researchers need to be aware of their own ‘interpretations’. Within the field of psychosis can train the service users to do research – it takes a lot of intensive work and support. They can decided what they want to research – user-led work which is quite powerful and it is difficult but possible to make it scientific.

RH: User led work important for grant applications. TCRN is ‘jointly owned’.

JY: Service users and qualitative research is now important in all projects.

EB: Qualitative research can give us more on phenomena. Evidence is broad – a phenomena can be part of the evidence and can be use as a quantitative unit. We need to bring TCs into evidence-based treatment.

GdeL: I agree. One concern - Peter made an important comment when we were discussing alternative methodologies. Where would all these multi-level analysis, complex time series, etc., stand in the hierarchy of evidence? If we didn’t have an RCT we would be down at the bottom no matter how sophisticated or logical the techniques were. It seems like we are back to the ‘gold standard’. We have to have an RCT otherwise we only have lower value evidence.

RH: Can we get a consensus on that? Do we need to do any of this work – it is that it might be interesting but not necessary?

JY: If we look at methodologies to influence treatment, science and policy, we are just talking about the research methodologies needed to influence the academic/scientific community and there are other domains for other methodologies.

EK: RCTs do not influence policy.

MC: We could have a debate on this but I’ve seen it happen. It puts people in a strange position: I spend a lot of time saying we need to do trials but then I spend a little bit of time speaking to policy makers who are saying there are no trials so we do not have the evidence. I know of specific areas where treatments have been cut because bodies like NICE don’t recommend the treatment.

EK: That’s a different point – use is made of trials to justify taking up spending. The decision will often have already been made. NICE has had some effect in that it will use that ‘box model’ and does make selections based on that. You have to get above a tolerance level on quality before they would recommend something. Obviously RCTs are in a lot of those – but not all (e.g. recent recommendations in substance abuse).

MC: I’ve come across more than one example where a clinical service has stopped providing an intervention on the basis that it wasn’t in the NICE guidance – and it wasn’t in the NICE guidance because there wasn’t a trial.

EK: Yes, I think that ‘negative’ effect is true – but you don’t get the other side of it. JY: So we have got an asymmetrical relationship between research finding and policy implementation – 2 different communities basically. Is there any methodology that would impact on policy more directly?

EK: There is a big divide but well done economic research can influence policy – econometrics.

JG: But I think you are saying there are other factors considered apart from research?

EK: Politicians work on ‘killer facts’!

JG: So for us it just means that as well as research we have to do other work – influencing people, whatever. So I think it is a good idea to do an RCT plus other work, trying to change the climate.

EK: RCT probably necessary but not often sufficient

KH: I think there is a role for evidence. A lot of people who aren’t involved in TCs find them very ‘mysterious’ and both quantitative and qualitative research could help to dispel this ‘mysteriousness’ but not reduce the complexity. So when talking about it to new people in the field you have a better bridge to make the ideas ‘less worrying’. That’s where qualitative research could help articulate what we are doing.

EK: But I’ll give you an example: when the policy reviewers looked at a residential TC the measures they considered were occupancy, turnover and the consequent cost – not once in any argument, any report, any presentation, was the efficacy of TCs discussed at all. No other.

JY: So in our first bullet point we maybe need to distinguish that RCTs are necessary for the scientific community but are not sufficient for the policy makers

MC: Not ‘science’ – it’s evaluative social research.

JG: I like Eric’s phrase that “Evidence is more than just facts”

MC: I think RCTs are necessary but not sufficient for everybody. One thing that interests me is the issue of intensity of treatment, which we haven’t talked a lot about but I think it is a pressing issue – Rex was talking about mini TCs, perhaps micro TCs? I’m very interested in how small the intervention might be and still be judged to have a beneficial effect – I don’t see how that is addressed by this RCT. If there is going to be one RCT of TCs in the next 10 years we will have done well – there won’t be 10 RCTs – and it seems there are other pressing questions to answer.

EK: In the next few years with this RCT – maybe with DIY funding – there are some other methodologies that could be additionally applied. People working in the TC could be trained up to implement them. There would be some time/cost to analyse that data but it wouldn’t be huge. You would then have these additional data sets to make available to people around the TC and can be added to the writing up with the RCT. It would yield very interesting supplementary data for fields which don’t necessarily regard RCTs as the ‘gold standard’

Consensus statement discussed.

FW: What is the follow on from this meeting? Is there energy to have another meeting – now we are rushing lots of ideas about where to go next?

EK: Maybe there is an equivalent meeting just on other methodologies not about RCTs. SS: Would it be helpful to have agreed outcomes we can use across settings?

RH: To finish off, I will just say thank you all very much and hope it has been useful. The tangible outcomes of this meeting will be circulation of consensus statements, notes, presentations. A written summary, and maybe an article, and recording kept in archives.


PETT & archives, Article, Circulation of notes etc,

Presentations

Personal tools
radioTC international
what else is here?