International Comparative ACT Study Process and Data

How ACT teams compare in Toronto, Birmingham, Nashville and Auckland

Key words leadership; IIMHL; exchange visits; collaborative working relationships; assertive community treatment teams; assertive outreach teams

Neil Deuchar and Katie Saunders
Birmingham and Solihull Mental Health NHS Trust, Birmingham, UK
Jane Vanderpyl
Te Pou, The National Centre of Mental Health Research, Information and Workforce Development,
Auckland, New Zealand
Thomas W Doub and Jules M Marquart
Centerstone CMHC, Inc., Nashville, Tennessee, US
Steve Lurie, Angela Da Silva and Heather McKee
Canadian Mental Health Association, Toronto Branch, Toronto, Ontario, Canada
Verity Humberstone
Intensive Community Treatment Team, Counties Manukau District Health Board, Auckland, New Zealand
Stuart Moyle
Mental Health Services, Counties Manukau District Health Board, Auckland, New Zealand

Assertive community treatment teams provide treatment and support to people living with severe and persistent mental health problems through a team of professionals from a variety of disciplines, including: psychiatry, nursing, social work and other health professions and paraprofessionals. ACT teams have been shown to reduce hospitalisation for people living with schizophrenia who experience significant disability and co-occurring disorders such as substance abuse (Ontario Ministry of Health and Long Term Care, 2006). We review the development and implementation of the study, some of the findings and implications for future collaborative work in this and other areas.

One of the features of the IIMHL site matches is that sites invariably have similarities and differences, given the different policy and funding frameworks in the various IIMHL member countries. One of the benefits of these exchanges is the opportunity to examine a particular service delivery issue and find out what the similarities and differences are across sites.

The four sites in this study all provide services to people with serious and persistent mental health problems. The size and range of services offered by each organisation differ although some services, such as assertive outreach, were thought to be similar. The partner sites are: Centerstone in Nashville, Tennessee, United States, Birmingham Solihull Mental Health Trust, United Kingdom, Counties Manakau District Health Board Mental Health Service, Auckland, New Zealand, and the Toronto Branch of the Canadian Mental Health Association in Ontario, Canada. Two sites provide both inpatient and community mental health services (Auckland and Birmingham) and two sites provide community mental health services only (Nashville and Toronto).

The lens for this analysis is bifocal: the story of a multi-site collaborative research study, and the comparison of assertive outreach (ACT) teams, client characteristics and outcomes across sites. Dewa and colleagues (2002) note that while the fundamental goal of the mental health system is to provide integrated community-based services, there is limited empirical evidence for planning and a lack of standard evaluation methods. They argue that a multi-site study can address this gap by using the same study design and instruments to measure characteristics and outcomes. While this concept appears straightforward, successful implementation is dependent on the active participation of all sites, valuing informal participation, communication, trust, flexibility and commitment. In a subsequent article (Dewa et al, 2004) they note that learning to communicate in a common language is critical. Our partnership experienced many of the same dynamics.

There is an emerging literature regarding multi-site comparisons of assertive community treatment. Salyers and colleagues (2003) evaluated the fidelity of 51 ACT programs and 36 case management programs in the United States using the 28 item Dartmouth Assertive Community Treatment Scale to measure structure and process of assertive community treatment teams. They were able to establish normative measures.

Gehrs and colleagues (2004) note that fidelity measurement is an evolving field and that the structure and process of programs can be shaped by the contexts in which they operate.

During the Nashville site visit, Centerstone leaders described their continuous treatment teams (CTT) which seemed similar to assertive outreach. Discussions began on the feasibility of comparing CTT to assertive outreach, and a proposal was agreed to following the Auckland visit in 2005.

It was agreed that each site would identify team participants who would represent their site in the study. Toronto was represented by the CMHA Executive Director, one of the agency’s program directors, and the research and evaluation coordinator. The Toronto site was also able to recruit a researcher from the Centre for Addiction and Mental Health, who had experience conducting multi-site studies. Nashville was represented by the Vice President of Research. Birmingham was represented by the Chief of Psychiatry of the Birmingham Solihull Trust and Auckland was represented by the Manager of the County Manakau’s mental health research and evaluation unit.

It was agreed that Toronto and the IIMHL administration would co-ordinate monthly teleconferences to develop and implement the study. Initial teleconferences focused on options for gathering comparative information and scope. It was decided that the study should describe system characteristics, team characteristics, client characteristics and outcomes.

A program comparison questionnaire was developed to examine system characteristics such as the broader community setting, team age and target population.

The Dartmouth Assertive Community Treatment Scale (DACTS) was chosen to measure team characteristics (Teague et al, 1998). DACTS is able to distinguish between assertive outreach and other types of case management and fidelity measures similar to DACTS have been shown to correlate positively with improved outcomes (Salyers et al, 2003). Data was entered by the teams and sent via email to Toronto for analysis in the fall (autumn) of 2005.

A prospective study was ruled out given the length of time it would take to enroll new clients in the study. A snapshot study using a point in time was felt to be most feasible. Each site agreed to extract information on clients enrolled in services as of 31 March 2006, and clients discharged from 1 April 2005 and 31 March 2006 from their clinical data bases.

Initially Birmingham proposed using the HoNOS survey which is used in the UK (Wing et al, 1996). The HoNOS set of scales measures a range of psychiatric symptoms, physical health, and social functioning problems associated with mental illness and are designed for routine use in clinical settings. After a dry run attempt at scoring, it was agreed that use of the HoNOS would require training at each site, and since there were varying degrees of familiarity with the instrument, the study team began to search for a simpler option. Fifty-five ACT teams in Ontario had been using the CDS (common data set), a variation on the PSR toolkit since 2001 (Lurie et al, 2007) to submit aggregate data on client characteristics and outcomes. After reviewing sample reports from Ontario, it was decided that it would be possible to use the CDS to extract client information from clinical databases, but the study team would also have to develop common definitions which mapped to the CDS domains, as each site/jurisdiction had different ways of reporting on domains such as hospitalisation, justice system involvement, school attainment, and ethnicity. Client data was entered into a SPSS spreadsheet developed by the Auckland site and sent via email to Auckland for analysis in the spring of 2006. All identifying client information was removed from the data files before transmission. Local ethics approval was obtained for each site.

System and program characteristics

Auckland: Intensive Community Treatment Teams at Counties Manukau District Health (CMDHB) Board Mental Health Service

CMDHB’s target area comprises a large urban area in South Auckland and extends south over a large rural catchment. The total adult CMDHB population numbers approximately 300,000. The population served is ethnically diverse with high levels of deprivation. CMDHB provides a range of services including: adult and older adult inpatient and community mental health services; secure inpatient rehabilitation services; and community child and youth mental health services. The two CMDHB ACT teams are government funded, serve approximately 200 clients (at any one time) with an Axis 1 diagnosis (typically a psychotic illness) or a severe personality disorder. Clients reside in the Counties Manukau DHB catchment area, and are between the ages of 16 and 65 years old. Core care is provided by nurse case-managers (CMs), each caring for 20 clients, shared with community support workers (CSWs). Case-managers see their clients approximately weekly, varied according to need. The team is multidisciplinary and also includes a daily medication dispensing nurse, forensic liaison nurse, consultant psychiatrist, junior doctor, occupational therapist and a clinical psychologist. Medical and other multidisciplinary review is carried out in the community. At times of crisis, the team can increase consumer support, and will organise any additional resources such as respite or hospitalisation, thus providing continuity of care.

Birmingham: Assertive Outreach Team at Birmingham and Solihull Mental Health Trust

Birmingham and Solihull comprise two populations. The former (circa 1 million people) is richly diverse and contains some of the highest areas of psychosocial morbidity in the UK. The latter (circa 200,000 people) is a more suburban, middle class environment. The Trust currently deploys eight assertive outreach teams, concentrated in the areas of most need. Three of these teams participated in the current study. The Trust serves a major black and ethnic minority population. The principles of the teams centres around the prevention of admission of ‘revolving door’ patients (that is those who have had several admissions under the Mental Health Act in the past) and the management of enhanced risk (particularly as this may relate to risk to others). In addition, caseloads almost exclusively comprise people with psychosis, although dual diagnosis (psychosis with substance misuse) and triple diagnosis (the former plus personality disorder) are disproportionately represented among the teams’ caseloads.

Nashville: Homeless Outreach Partnership and Empowerment (HOPE) at Centerstone

HOPE serves Montgomery County, Tennessee, a metropolitan area positioned adjacent to the Fort Campbell Army Post and saturated with active duty and retired military individuals and families. This area is culturally diverse and contains pockets of high and generational poverty, as well as a significant population that is homeless or at-risk of homelessness. In partnership with a local homeless shelter, HOPE utilises a modified ACT model to conduct outreach and engage individuals in behavioural health treatment. The HOPE program entry criteria include: age (adults 18 years and older); diagnosis of serious mental illness; presence of a co-occurring substance use disorder; homelessness (or at risk of homelessness). In addition to their behavioural health and housing problems, this population frequently has serious challenges in the domains of physical health, employment, social/family supports, and criminal justice.

HOPE focuses on outreach, engagement and screening, assessment, treatment, and recovery/ maintenance. The program offers: low staff to client ratio; flexible service delivery; comprehensive services; case management; psychoeducational groups; 24/7/365 crisis assistance. The HOPE team helps clients find housing, jobs, psychiatric treatment, medication management and obtain money for day-to-day expenses. Rental assistance/housing support funds would likely refer to rent down payments such as first and last month’s rent and household supplies. Since 2004, the HOPE has enrolled 111 people into services.

Toronto: West Metro, East Metro and New Dimensions Assertive Community Treatment Teams at the Canadian Mental Health Association Toronto Branch

The three Toronto ACT teams, part of the Canadian Mental Health Association, serve an urban population in the central and east parts of the city of Toronto. The city is ethno-culturally diverse, with immigrants representing almost half (49.5%) of its total 2.5 million population. Toronto has a low unemployment rate (7%) and nearly two-thirds of the population have at least a high school education. The mental health service environment is changing as regional health funding authorities are developed. There are two tertiary hospitals and 18 general hospitals with psychiatric departments serving Toronto, along with 80 community mental health and addiction programs and 13 ACT teams currently operating in Toronto.

ACT is a voluntary health service for people who have: persistent symptoms of serious mental illness which include: a diagnosis of a chronic psychotic illness; both frequent and long-term hospitalisations; serious functional impairments affecting activities of daily living, work, and housing; inability to meet survival needs from community sources, including housing, social and healthcare services. These services also target people with co-existing substance abuse disorder and a history of involvement in the criminal justice system (or are at high risk of involvement). ACT teams in Ontario are governed by provincial standards that have been developed with the service expertise of ACT practitioners and provincial system stakeholders. The standards specify client characteristics, team composition, client to staff ratios (1:8), requirements for service plans, intake process and other operational issues pertaining to team functioning. The standards do not prescribe the nature of services to be offered, beyond setting staffing from specified health professions and the high frequency/intensity (>2 sessions of 1 hour each/weekly, up to multiple daily contacts) in community venues. Service plans are flexible and individualised to meet the assessed needs of clients on an on-going basis. Teams offer 24/7/365 service availability.

Team characteristics

The DACT Scale was used to measure and compare team characteristics across the sites. This scale is a 28-item tool which measures the fidelity of ACT programmes to the ACT model (Salyers et al, 2003). It has been successfully used to compare teams internationally (Gehrs et al, 2004). It assesses the function and structure of teams in three subscales – human resources (HR), organisational boundaries (OB) and service intensity (SI). The higher the score, the more concordant the teams’ structure and function are to the original ACT model. HR refers to client/clinician ratio, the numbers of various professional staff in the team, the clinical role of the team leader and specialist roles. OB refers to the ability of the ACT to deal with crises, take full responsibility for all the treatment needs of their clientele, tendency to initiate hospital admissions and intake and discharge rates. SI includes data around frequency of contact, ability of the team to manage concurrent disorders and inclusion of clientele in case management. High fidelity is a score of 4.0 or more, medium fidelity is a score of 3.0–3.9 and low fidelity is a score of 2.9 or below.

Table 1: Overview of program characteristics

click on the image to enlarge

There was some variation in the length of time each team had been established, ranging from 1997 (ICT-T and Birmingham) to 2004 (Nashville).

The ratio of clinician to clientele averaged 1:10, there being a slightly lighter caseload for each clinician in Toronto and a heavier caseload per clinician in Nashville. The average caseload per team was 85, noting that the Birmingham data relates to three teams. Birmingham, therefore, had more teams per capita than some of the other sites but the teams were smaller (less staff and less clientele). A total of 765 ACT clients were included in the study from nine teams in four countries. The largest group were from the three ACT teams based in Toronto (35%), another 28% were from Counties Manukau two ACT teams and 25% were drawn from three ACT teams based in Birmingham. Only 11% of the group were from Nashville’s ACT team (see Figure 1).

Patients were less likely to be discharged from Birmingham teams than any other site, the percentage of clientele having been discharged from their teams over the study period being 3% compared to an average of 12%. Nashville discharged most frequently but it is noted that their target clientele were different from that of the others. Even if Nashville is excluded from this metric, however, Birmingham appears to discharge significantly less frequently than the other two sites.

Accordingly, Birmingham retained more clientele in assertive outreach than any of the other sites, some 60% of patients having been in receipt of services for five years or more and a further 15% for three to five years.

Time registered with services varied across the services (see Figure 2). The four sites showed wide variation with Birmingham having the largest proportion of clients in the service for more than four years (60%). At the opposite end of the spectrum was the Nashville team where most clients had been with the service less than one year (88%). This may be due to Nashville having only been established in 2004. However, Nashville also had the highest proportion of discharges from the service (see Figure 2). Nashville had discharged 26% of their clients in the previous year, while Birmingham had only discharged 3%. Of the 21 clients discharged by Nashville ACT team, 19 had been with the service less than a year.

DACTS scores were compared within and across teams in the four sites. As can be seen from Table 2, the lowest concordance across the teams was in the service intensity domain, the highest concordance being in the organisation boundaries domain. The greatest variability was in the human resources domain and the lowest in service intensity. This suggests that lower adherence to the service intensity domain of the DACTS is a commonality across the four sites, although most sites report medium fidelity.

Commonalities across the teams in terms of HR included having small clinician to clientele ratios,having nurses on the clinical staff and programme size. The greatest variabilities related to the team leader having a clinical role and the presence of substance misuse and vocational specialties roles.

Commonalities across the teams in terms of organisation boundaries included a generally high level of fidelity with the exception of crisis services and full responsibility for treatment, Nashville representing something of an outlier in respect of demonstrating a much higher turnover of clientele than the other sites.

Commonalities across the teams in terms of service intensity included a generally lower level of fidelity with only a third of teams offering individualised and Value organisation HOPE group substance misuse treatments. Few teams included consumers in case management roles.

Client group characteristics

Males averaged 66% of ACT clients across all four sites, and there were only small variations (see Table 3). The average age across the group was 41 years (SD 11 years). Each of the sites varied in their racial/ethnic distribution. Whites made up the largest racial/ethnic group at all sites (43%), ranging between 34% at Counties Manukau and 54% at Nashville (see Table 3). Toronto, Birmingham and Nashville ACT teams all had a high proportion of clients identifying as black. In contrast, 57% of Counties Manukau ACT clients identified as either Mãori or Pacific peoples.

Major diagnostic categories consisted of schizophrenia 77% (n = 583), and mood disorders 17% (n = 128). Nashville ACT team differed significantly from the other three siteswith nearly three quarters of ACT clients having a diagnosis of mood disorder (74%). Counties Manukau also had a small group without any Axis I diagnosis (2%). Of this group, six had a diagnosis of borderline personality. Across the four sites, 31% of ACT clients had a concurrent diagnosis of substance misuse. However, Nashville ACT clients (89%) and Counties Manukau ACT clients (52%) were much more likely to have a concurrent substance misuse diagnosis than the other two sites. Also included in the study were a subset of 267 continuing treatment team (CCT) clients from three teams in the Nashville mental health service. The CTT clients showed some key differences from the ACT clients. The majority of the Nashville CTT clients were female (64%) compared to only 34% of the ACT clients. A very high proportion of the CTT clients were white (84%) compared to only 43% of ACT clients. The majority of CTT clients had a diagnosis of mood disorder (54% compared to 17% of ACT clients). However, over 30% of both CTT and ACT clients had a diagnosis of substance misuse disorder (39% CTT and 31% ACT clients). Patterns of service use and discharge varied dramatically for the two groups. Most CTT clients had only been in the service for a year (63% CTT clients compared to 19% of ACT clients). Almost all CTT clients were discharged within two years of entry into the service (92%).

Education, employment and living status on entry to ACT teams

A high proportion of ACT clients had not completed high school (51%, n = 387). However, compared to the other sites Birmingham had the highest proportion of clients who had completed high school (71%) and a university degree (13%) (see Table 4). All sites reported high levels of unemployment among their clients on entry to the ACT team (81%) and low levels of permanent employment (7%).

Birmingham and Toronto both had a high proportion of clients living alone on entry to the ACT team. In contrast Nashville and Counties Manukau differed, Nashville had a high proportion of clients who were homeless (see Table 4), while Counties Manukau had a high proportion of clients living with family members.

Client and service outcomes

ACT client outcomes were examined in the areas of employment status and hospital bed use by length of time in service for each participating site.

Few clients who were unemployed at the time of entry into the ACT team became employed, irrespective of length of time in the ACT service (see Figure 6). Only Counties Manukau ACT teams showed a consistent increase in the proportion of clients obtaining employment by length of time in service. Of the group who had been in the service more than four years and unemployed at entry to ACT team, 23% were now in some type of paid employment. Both Toronto and Counties Manukau ACT clients showed large reductions in acute psychiatric hospital bed use from the year prior to entry into the ACT team. Both sites showed similar patterns of continuing decreases over a number of years post entry into the ACT programme (see Table 5).

Discussion

The DACTS data shows that assertive outreach teams are similar in terms of fidelity measures across jurisdictions, although levels of funding may explain some differences and systems contexts explain others. The fact that none of the teams were providing group programming for concurrent disorders may be illustrative of the difficulties mental health systems are having developing integrated treatment strategies for persons with concurrent disorders.

While Auckland and Nashville had higher rates of substance abuse than Toronto or Birmingham, none of the sites were using group based interventions with this population, even though it has been identified as an evidence-based practice. This same phenomenon is being identified in a study of 70 ACT teams in Ontario currently under way.

Similar to the Salyers study (2003), ours found a range of scores for DACTS items among study sites. The average DACTS score for the teams across sites was 3.8, suggesting medium fidelity to the model. Our study showed that it is possible to compare team characteristics internationally, using the DACTS.

The client data shows that service system contexts and community contexts influence demographics and outcomes. Ethnicity varied across sites as did educational attainment, employment and housing. For example, 45% of Auckland clients live with their families while over 50% of the clients in Toronto and Birmingham live on their own. Auckland was the only site to show consistent improvement in employment outcomes over time.

The teams in Toronto, Auckland and Birmingham served a majority of clients living with schizophrenia, while Nashville clients were more likely to have mood disorders and a high rate of substance abuse. Despite the differences, all sites were serving client populations with serious and persistent mental health problems.

Length of stay in the program was similar in Toronto, Auckland and Birmingham, while Nashville had shorter stays that may be driven by the funding available. The question of how long clients should stay in ACT merits further exploration and is to some degree dependent on the ability of service systems and teams to flex levels of support. Since the ACT model was developed to provide longterm community support to clients outside of hospital, it is not unreasonable to assume that some clients may require this level of support for a long, or indefinite period.

While outcome data on hospitalisation was limited to two sites, the declines in hospitalisations are consistent with the ACT literature. Nashville’s use of an ACT-like model for their continuous treatment teams is consistent with international interest in developing variations on the ACT model (van Veldhuizen, 2007).

With regard to client data collection itself, the project team was able to generate common data definitions, which allow for meaningful comparisons across jurisdictions. The sharing of comparative data among teams can facilitate meaningful reflection about the program model in all jurisdictions.

Summary of project learnings

It is possible to do a comparative, exploratory study of mental health programs across jurisdictions without funding. Key ingredients include leadership, sharing of the labour, multiple means of communication, and regular communication. Keeping the study design relatively simple and leveraging data capture instruments already in use, can minimise complexity. All sites found that the study exposed limitations in their data collection systems. This reinforces the need to continue to focus on data quality as well as data capture in our various systems.

The experience gained in the comparative ACT study may lead to the development of a similar project on early psychosis intervention. Finally, using the IIMHL meetings and other conferences to present the findings and study as it developed over a two-year period imposed deadlines, which resulted in things being done, as well as opportunities to reflect on the data and the collection process itself. As Dewa and colleagues note, a multi-site study builds relationships (2004). This reinforces the mission of IIMHL itself and can only lead to a richer relationship among participants.

please click on any of the following links to view the associated document in .pdf format

• ACT Data Summary 2006-07
• ACT Report Final 2006-07
• ACT International Comparative Study Process
  

references

1. Bond GR & Drake RE (2007) Should we adopt the Dutch version of ACT? Commentary on ‘FACT: A Dutch version of ACT’. Community Mental Health Journal 43 (4) 435–438.
2. Dewa CS, Durbin J, Wasylenki D, Ochocka J, Eastabrook S, Boydell KM & Goering P (2002) Considering a multisite study? How to take the leap and have a soft landing. Journal of Community Psychology 30 (2) 173–187.
3. Dewa CS, Butterill D, Durbin J & Goering P (2004) No matter how you land: Challenges of a longitudinal multi-site evaluation. The Canadian Journal of Program Evaluation 19 (3) 1–28.
4. Gehrs M, Smith Fowler H, Rouke SB, Smith M & Bradley Cousins J (2004) Inside the black box: Challenges in implementation evaluation of community mental health case management programs. The Canadian Journal of Evaluation 19 (3) 109–133.
5. Lurie S, Kirsh, B & Hodge S (2007) Can ACT lead to more work? The Ontario experience. Canadian Journal Community Mental Health 26 (1) 161–171.
6. Ontario Ministry of Health and Long Term Care (MOHLTC) ACT Technical Advisory Panel (2006) 2004/05 Ontario ACT Data Outcome Monitoring Report. Toronto: Author
7. Salyers MP, Bond GR, Teague GB, Cox JF, Smith ME, Hicks ML & Koop JI (2003) Is it ACT yet? Real-world examples of evaluating the degree of implementation for assertive community treatment. The Journal of Behavioral Health Services and Research 30 (3) 304–320.
8. Teague GB, Bond GR & Drake RE (1998) Program fidelity in assertive community treatment: Development and use of a measure. American Journal of Orthopsychiatry 68 216–232.
9. van Veldhuizen JR (2007) FACT: A Dutch version of ACT. Community Mental Health Journal 43 (4) 421–433.
10. Wing JK, Curtis RH & Beevor AS (1996) HoNOS: Health of the Nation Outcome Scales: Report on Research and Development July 1993–December 1995.London: Royal College of Psychiatrists.

We thank the authors and publisher of this report as well as its sponsors for allowing to reproduce it here in its entirety for the greater benefit of all Conference participants.

2006/07 Ontario ACT Data Outcome Monitoring Report
Ontario Technical Advisory Panel
Ministry of Health and Long-Term Care
February 2008