Wilfrid Laurier University
Wilfrid Laurier University
Scholars Commons @ Laurier
Scholars Commons @ Laurier
Theses and Dissertations (Comprehensive)
2020
A program implementation fidelity assessment of a Housing First
A program implementation fidelity assessment of a Housing First
program in Ontario
program in Ontario
Steven Bigioni
bigi9310@mylaurier.ca
Follow this and additional works at: https://scholars.wlu.ca/etd
Part of the Community Psychology Commons
Recommended Citation
Recommended Citation
Bigioni, Steven, “A program implementation fidelity assessment of a Housing First program in Ontario”
(2020). Theses and Dissertations (Comprehensive). 2274.
https://scholars.wlu.ca/etd/2274
This Thesis is brought to you for free and open access by Scholars Commons @ Laurier. It has been accepted for
inclusion in Theses and Dissertations (Comprehensive) by an authorized administrator of Scholars Commons @
Laurier. For more information, please contact scholarscommons@wlu.ca.
Running head: FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
1
A program implementation fidelity assessment of a Housing First
program in Ontario
by
Steven Bigioni
Honors BA Kinesiology, Western University, 2011
THESIS
Submitted to the department of Psychology in partial fulfillment of the requirements for Master
of Arts in Community Psychology
Wilfrid Laurier University
2020
Steven Bigioni 2020 ©
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
2
Abstract:
This research sought to assess the degree of fidelity to the Housing First model achieved
by a new Housing First program in a mid-sized Canadian municipal region, and the
factors that promoted or hindered fidelity therein. The program was delivering an
adaptation to the Housing First model that prioritized access to housing and support
services, which was assessed simultaneously. Fidelity ratings were gathered by a team of
researchers during a site visit that included observation of a staff meeting, seven
interviews with program leaders and staff, two focus groups with program participants,
and 10 chart reviews. Overall, the findings show a high degree of fidelity with an average
score of 3.55 on a 4-point scale, across 44 fidelity domain items. Results revealed high
fidelity in the domains for service philosophy, separation of housing and services and the
newly created domain of support and skills development used to assess the home-based
support adaptation. Lower scores were found for housing choice and structure, service
array, and program design. Challenges to program fidelity were found in housing
availability and affordability, service continuation through housing loss, linking with
employment and educational services, 24-hour coverage, and participant representation in
the program. Factors that could account for these challenges include the low vacancy
rates in the jurisdiction, prescriptive policy frameworks, and a slower pace of
implementation than anticipated. This study demonstrates the use of a fidelity assessment
to provide direct, actionable feedback for program improvement.
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
3
Contents
Abstract ……………………………………………………………………………………………………………………………………… 2
Introduction
………………………………………………………………………………………………………………………………… 4
Literature Review ……………………………………………………………………………………………………………………… 6
Background …………………………………………………………………………………………………………………………….. 6
Housing First …………………………………………………………………………………………………………………………. 10
The Program
………………………………………………………………………………………………………………………….. 13
Fidelity Assessment
………………………………………………………………………………………………………………… 17
Research Aims
……………………………………………………………………………………………………………………….. 21
Method ……………………………………………………………………………………………………………………………………. 22
Community Partners ………………………………………………………………………………………………………………. 23
Participant Recruitment and Data Collection …………………………………………………………………………….. 24
Measures
………………………………………………………………………………………………………………………………. 26
Research Design …………………………………………………………………………………………………………………….. 27
Ethical Considerations
…………………………………………………………………………………………………………….. 29
Data Analysis …………………………………………………………………………………………………………………………. 30
Positionality ………………………………………………………………………………………………………………………….. 32
Knowledge Translation Strategies ……………………………………………………………………………………………. 35
Results …………………………………………………………………………………………………………………………………….. 36
Items Promoting Fidelity
…………………………………………………………………………………………………………. 40
Items Hindering Fidelity ………………………………………………………………………………………………………….. 43
Discussion ……………………………………………………………………………………………………………………………….. 45
Limitations ……………………………………………………………………………………………………………………………. 55
Conclusion and Recommendations……………………………………………………………………………………………. 57
References
………………………………………………………………………………………………………………………………… 59
Appendix A: TCPS-2 Certificate …………………………………………………………………………………………………. 72
Appendix B: Adapted fidelity scale for evaluation of The Program………………………………………………….. 73
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
4
Introduction
In Canada, housing costs have skyrocketed in the past decade with a recent report by a
major financial institution finding housing affordability to be at historic lows (Royal Bank of
Canada, 2019). Those affected by the affordable housing crisis tend to be young people and/or
those with lower incomes (Gaetz, Donaldson, Richter, & Gulliver, 2013). For many, the rising
costs of living mean they are at a greater risk of experiencing homelessness, with 1 in 5
households experiencing housing affordability issues (Canada Without Poverty, 2020). The
Canadian federal government has recognized this as a priority and committed to a significant
investment in housing over the next 10 years (Canada Mortgage and Housing Corporation
[CMHC], 2018).
One strategy that has come to prominence in North America in recent years is Housing
First (HF), a model that provides housing without prerequisites for sobriety or psychiatric
treatment (Tsemberis, Gulcur, & Nakae, 2004) to individuals experiencing homelessness. HF has
proven to be a successful model to help people experiencing homelessness and mental illness
find and retain housing (Goering et al., 2014; Gulcur, Stefancic, Shinn, Tsemberis, & Fischer,
2003; Tsemberis et al., 2004). Many Canadian cities are now actively working to implement HF
solutions with an aim to eliminate chronic homelessness (i.e., greater than 180 days without a
home in a year) (‘Region’, 2018). The HF Program (‘The Program’ hereafter) in this medium-
sized Canadian municipal region (‘The Region’ hereafter) is one such solution which builds new
elements onto the existing HF model by introducing skill building and home-based supports after
people move-in to housing in order to help residents successfully retain housing. The Program
was preceded by a pilot program in The Region that realized success with 95% of participants
able to gain and retain housing over two years (‘Region’, 2018). The goal of this research is to
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
5
determine the extent to which The Program has been implemented in accordance with HF
principles and initial program goals.
In order to properly assess this new program, it is important to understand the context
surrounding housing and homelessness. Exploring the prevalence of homelessness in Canada,
some of the root causes of homelessness and the many adverse effects homelessness can have on
individuals and society will help define the Canadian context. Identifying strategies that have
been employed to solve the issue helps to inform the history of homelessness policy. Finally,
literature is presented on the role of program and fidelity evaluations in ensuring successful
implementation and outcomes for programs and their application to Housing First protocols.
The available literature shows homelessness to be a significant issue in Canada at present,
stemming from a wide range of intersecting causes and having a number of individual and
societal-level effects (Gaetz, Dej, Richter & Redman, 2016; Echenberg & Jensen, 2012; Rech,
2019). Traditionally, the response to homelessness has been to manage the problem without
addressing the root cause, through emergency shelters and programs that require abstinence from
substance and psychiatric treatment, an approach that has yielded limited success in re-housing
people (Gulcur et al., 2003; Rech, 2019). More recent strategies have focused on the Housing
First (HF) model after successful trials have shown it to be a viable and effective strategy in
Canada (Goering et al., 2014, Gaetz, Scott & Gulliver, 2013).
Fidelity assessments serve an important role in determining how faithfully a program is
being implemented according to a set of standards (Centers for Disease Control and Prevention
[CDC], 2012). Programs implemented with high fidelity to the HF model can demonstrate better
participant outcomes (Durlak & DuPre, 2008). The specific context in which a program is being
implemented can also affect participant outcomes (Durlak & DuPre, 2008), and for the purposes
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
6
of this research, context will be used as a lens through which results are interpreted. The
increasing prevalence of fidelity assessments in Housing First evaluations is a result of the wide
adoption of the model and reflects the importance of accurate implementation to program
outcomes (Pleace, 2016; Polvere et al., 2014). We are conducting a process evaluation of The
Program, that will measure to what degree it is adhering to HF principles and assess how the
program’s unique goal of delivering home-based supports are being met. This thesis focuses on
the fidelity assessment as part of the larger process evaluation of The Program.
Literature Review
Background
Exploring the current state of homelessness in Canada reveals a significant problem that
affects a diverse population. On a given night, there are approximately 35,000 people
experiencing homelessness in Canada, which, over the course of a year, rises to 235,000 people
(Gaetz et al., 2016). The demographics of homelessness have traditionally been single adult men,
however since the mid-2000s, the population of people experiencing homelessness has become
much more diverse (Gaetz et al., 2016). There is now a higher proportion of women, youth and
families experiencing homelessness as well as people who identify as Indigenous, as newcomers
to Canada or as LGBTQ2S (Gaetz et al., 2016). Additionally, the population of people
experiencing homelessness is one that is disproportionately affected by mental illness
(Echenberg & Jensen, 2012). In The Region in 2017-2018, 2,652 people stayed in a shelter bed
and though that is a 3% decrease from the year previous, the length of stay for individuals in
shelter increased by 24%, from 24 days to 42 days on average (‘Region’, 2018). This
corresponds to 40% increase in individuals experiencing chronic homelessness and highlights the
difficulty people have recovering from homelessness.
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
7
Addressing the underlying causes of homelessness can be difficult as there are many
factors, both systemic and related to individual circumstances, that are responsible for people
experiencing homelessness (Office of the High Commissioner for Human Rights, 2015). To
understand some of the reasons homelessness exists, it is relevant to consider how government
funding cuts and resource allocation has affected housing stability to create the problem that
exists today.
For a period of around 20 years, beginning in the early 1980s, the federal government of
Canada began withdrawing funding from affordable housing organizations and programs
(Cohen, Morrison & Smith, 1995). In that period of time, the number of social housing units
built annually through funding by all levels of government in Canada dropped precipitously,
from 20,450 in 1982 to 1,000 in 1995 (Gaetz, Gulliver & Richter, 2014). It is estimated that these
funds that were cut could have created up to 100,000 new affordable housing units in that time
frame (Gaetz et al., 2014). Though funding for social housing has increased in the years since,
including a commitment of $2.2 billion in affordable housing spending over two years in the
2016 Canadian federal budget, a significant lack of affordable housing units now exists, limiting
housing options for people at-risk of or currently experiencing homelessness (Gaetz et al., 2016;
Gaetz et al., 2014; MBNCanada, 2017).
A weakened social welfare support system also contributes to why people may
experience homelessness. As funding for housing programs was being cut by federal
governments in the 1980s, so too was funding for social welfare programs (Cohen et al., 1995;
Gaetz et al., 2014). A 2012 review of risk factors for homelessness in Canada identified a
significant gap between the level of social assistance benefits people receive and the high cost of
rent (Echenberg & Jensen, 2012). Consequently, those who rely on social assistance programs
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
8
either cannot afford to enter the rental market or spend a disproportionate amount of their income
on housing rent (Gaetz et al., 2013).
Economically, it is in society’s best interest to find a solution to homelessness, rather than
to manage the problem. The costs to society associated with services most often used by people
experiencing homelessness (shelters, health care, policing) are exorbitant when compared with
the cost to provide housing and support. A 2005 article by Pomeroy looked at different costs
associated with homelessness across four Canadian cities (Toronto, Montreal, Vancouver and
Halifax). They estimated that costs from institutional responses (correctional facilities and
psychiatric hospitals) could range anywhere from $66,000 to $120,000 annually and costs
associated with emergency shelters could be up to $42,000 annually, per person. For comparison,
costs for supportive and transitional housing were found to be from $13,000 to $18,000 and
affordable housing (without supports) to be up to $8,000 annually, per person. These kinds of
economic results are echoed in a study by Goering et al. (2014), which found a significant cost
savings for people in a HF trial compared to treatment as usual. A more recent analysis estimated
the cost of homelessness to the Canadian economy at $7.05 billion (Gaetz, 2012). These studies
demonstrate the financial burden society’s traditional responses have incurred and make it clear
that a new strategy is needed.
Problems finding, obtaining and retaining housing often arise for people leaving
institutional care. As mental health institutions were closed down in favour of community care in
the latter half of the 20th century (known as deinstitutionalization), proper support services were
not in place to ensure adequate care for this vulnerable population (Belcher & Toomey, 1988;
Canadian Population Health Initiative of the Canadian Institute for Health Information [CHPI],
2009; Martin, 1990; Niles, 2013). Former patients were often discharged into tenuous living
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
9
situations and without proper treatment and living support, many became homeless (CHPI, 2009;
Niles, 2013). A high proportion of people experiencing homelessness have one or more mental
health problems, reflecting a need for more specialized supports (Echenberg & Jensen, 2012;
Goering, Tolomiczenko, Sheldon, Boydell, & Wasylenki, 2002; Lamb & Bacharach, 2001).
The effect of homelessness on individuals manifests in many ways. People experiencing
homelessness regularly experience stigmatization and discrimination, which is often
characterized by punitive government responses (Office of the High Commissioner for Human
Rights, 2015; O’Sullivan, 2012; Parnaby, 2003). The criminalization of homelessness is common
in North America, with the widespread use of laws that are designed to specifically target people
living outdoors in a city (National Law Center on Homelessness and Poverty, 2014). These laws
make simple acts of living difficult or illegal for those without a home and lead to social
isolation and separation (O’Sullivan, 2012). Culturally, social narratives promoted by neoliberal
ideas of individual responsibility (Taylor-Gooby & Leruth, 2018), say that people experiencing
homelessness are inferior or somehow inherently different from the general population, which
could also contribute to the stigmatization of an individual experiencing homelessness (Belcher
& Deforge, 2012).
Individuals experiencing homelessness also have a much greater risk of physical health
problems than the general population (Gaetz et al., 2013). This population has significantly
higher rates of mortality, higher incidences of problems like seizures and chronic obstructive
pulmonary disease and poor detection and/or inadequate care of existing health problems
(Hwang, 2001). These problems arise, or can be exacerbated, by living conditions outdoors
(inability to maintain adequate personal hygiene) or in shelters (overcrowding) or through
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
10
systemic factors like a lack of proper identification, which can impede one’s ability to receive
healthcare (Hwang, 2001).
The solutions that were originally created in the mid-1980s offered support in the form of
emergency shelter programs and supportive and transitional housing for people experiencing
homelessness (Rech, 2019). These responses however did not sufficiently respond to the more
complex needs of many individuals who experience homelessness (e.g., substance use, mental
illness) as their housing problems persisted and, in some cases, worsened (Gaetz et al., 2016).
This is, in part, because of a requirement in many traditional shelter and treatment programs that
people receiving housing and services achieve and maintain their sobriety and seek treatment for
any addictions or mental health problems they have (Tsemberis, Moran, Shinn, Asmussen, &
Shern, 2003). If a person were to breach these conditions, they could be evicted from their
housing and removed from the program, returning to homelessness and shelter living.
Housing First
A new strategy was developed in the early 2000s called Housing First (HF) which takes a
new approach. In this program, housing is the baseline of support given to people experiencing
homelessness and mental illness, without requirements tied to sobriety or psychiatric treatment
(Gaetz et al., 2013). Treatment supports to help people with substance use or mental health
problems were offered and used as needed and as desired by participants, with no outcomes tied
to housing support. Five core principles guide the delivery of HF programs: housing choice and
structure; separation of housing and services; service philosophy (e.g., utilizing a harm reduction
approach); service array (i.e., extent of community support services available); and eliminating
barriers to housing access and retention (Nelson et al., 2014; Stefancic, Tsemberis, Messeri,
Drake & Goering, 2013).
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
11
The HF model employs methods that promote participant empowerment. Empowerment,
as defined by Zimmerman and Eisman (2017), is made up of a sense of control, a critical
awareness of one’s environment and the ability to pursue goals and affect outcomes. These three
components are reflected in the HF core principles. In providing housing without any readiness
requirements and stressing participant-directed program development and recovery, people in HF
programs can regain control over their lives, and begin to make positive changes (Davidson et
al., 2014; Tsemberis et al., 2004). Through direct and responsible engagement in program
services and progress, participants can build a critical sense of the factors that have led them to
experience homelessness and an ability to pursue positive outcomes (Kirst, Zerger, Harris,
Plenert & Stergiopoulos, 2014).
Popularized in New York, NY, initial studies of HF found significantly better outcomes
for those in Housing First programs compared to traditional supports (Gulcur et al., 2003;
Tsemberis et al., 2004). Researchers found that a program that offered choice to the individual
about the location and type of housing and which services they would like to use, and when, was
preferred and led to better outcomes compared to working on a continuum where housing and
services were prescribed (Stefancic et al., 2013; Tsemberis et al., 2004; Tsemberis et al., 2003).
In a trial conducted with homeless individuals with mental illness and a history of substance use,
the Housing First model was shown to significantly increase participants’ perception of choice,
their time spent stably housed and their use of substance-use treatment services compared to the
standard of care (Tsemberis et al., 2004). Participants in that study showed an 80% housing
retention rate over the 2-year study, demonstrating that a person’s mental health is not indicative
of their ability to remain housed.
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
12
The strength of the model was tested in the Canadian context in a large, multi-site,
randomized control trial of HF called At-Home/Chez Soi. Conducted in five major cities, of
different sizes and with different resources, the five-year study compared the HF approach
against treatment as usual (TAU; using existing housing and support services in the community)
for over 2,000 individuals experiencing homelessness and mental illness (Goering et al., 2014).
For those individuals in the HF group, support was provided in the form of coordinated
professional service teams to help individuals with mental illness and complex needs minimize
hospitalization and enhance positive outcomes (Goering et al., 2014). The study reported better
results for housing stability, participants’ health and many other measures than achieved by the
TAU participants (Goering et al., 2012; Goering et al., 2014).
An outcome evaluation of the five-year At Home/Chez Soi project found 62% of
participants in the HF treatment group were housed all of the time compared to 31% for the TAU
group, and only 16% of the HF group were housed none of the time compared to 46% for the
TAU group (Goering et al., 2014). In concert with more stable housing outcomes, participants in
the HF group spent less time in temporary housing, emergency shelter, institutions and on the
street than did the TAU participants (Goering et al., 2014). Another finding of note from At
Home/Chez Soi trial, mentioned briefly earlier, was the cost associated with implementing the
HF model compared to TAU. Though support staff are expensive to provide, the cost savings for
supplemental services (shelters, physician visits, police responses, etc.) decreased by over
$21,000 per person for the highest needs participants (Goering et al., 2014). These positive
findings and many others were echoed by participants in qualitative interviews conducted as part
of the study (Kirst et al., 2014).
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
13
The Program
The Program began in 2018 and is being delivered by a local multi-service organization
in The Region. The Program is designed to combine Housing First (HF) principles with greater
home-based supports. This differentiates The Program from other HF trials by extending
supports to aid participants in adjusting to their new housing (e.g., taking care of a home,
financial planning, living independently). The Program has a specialized team that initially
works with their participants to ensure necessary documentation and finances are in order to
prepare individuals for moving into a home. Once a participant is deemed document ready, the
team works to find suitable housing with the individual’s input about the location and type of
housing they would prefer. Once moved-in to their new housing, participants begin receiving
support from the home-based support team who ensure a smooth transition into housing through
ongoing support with housing retention, skill building (e.g. how to maintain a home and cook
healthy meals) and linking participants to community services.
Support team members work with the participants through five essential and sequential
stages of recovery from homelessness: stabilizing housing, individualized housing support
coordination planning, promoting self-awareness, recognizing self-management, and reframe and
rebuild (Housing Services, 2017c). All work with participants is meant to be trauma-informed
and person-centered (Housing Services, 2017a). The five stages structure a home-based support
worker’s case management to create an individualized case plan, and help participants set
recovery goals and critically reflect on their progress toward housing stability. Home-based
support workers provide support for up to 18 months, depending on depth of need, with support
scaling down as a participant progresses toward recovery (Housing Services, 2017c). After a
participant is deemed able to maintain their housing stability and transition to independence, the
supports from their home-based support worker scale down significantly but they remain ready
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
14
to re-engage supports if necessary (Housing Services, 2017c). Coordination between the initial
intake system and the home-based support team, including how the stages are identified and
actions to take with participants, are laid out in foundational frameworks created by the regional
government (Housing Services, 2017a, 2017b).
Program participants are drawn from a Regionally-held list of individuals with high needs
who are experiencing chronic homelessness. Level of need is determined using a standardized
assessment measure, the SPDAT (Service Prioritization Decision Assistance Tool) (OrgCode,
2016), which is delivered to everyone who enters the housing system in the region. This measure
assesses a variety of factors to determine level of need, including physical and mental health,
substance use history, housing stability and self-management skills. Those who score highest on
the SPDAT are deemed to be at the highest level of need and are prioritized for service on the
intake list (Housing Services, 2017b), which is a central registry of people who have been
deemed chronically homeless (>180 days spent homeless in the past year or 18 months over the
past three years).
As with other HF programs, The Program is guided by a program theory that is a
foundation for how it is meant to be delivered in The Region and how it will achieve the desired
effect. Program theory is used to determine what a program needs to do to meet its desired goals
and what additional impacts may arise from actions taken (Chen, 1990). Chen updated this
definition (2005) to emphasize the role of the context and setting the program is occurring in and
the implicit and explicit assumptions being made by the program. Beyond simply looking at the
actions taken by a program, this theory incorporates the underlying factors that can affect a
program’s success. For The Program this includes the principles of HF, the added component of
home-based support, the local organizations that contribute to the housing system in The Region,
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
15
as well as the region’s community and governmental priorities. Broader contextual factors that
affect this program and population include the increasing cost of living and as well as housing
and social policies being pursued at each level of government. Considering this holistic approach
will lead to a more complete and nuanced initial understanding of The Program’s design and
what factors, both within and outside of the program, may influence its ability to be implemented
appropriately.
In its design, it can be said that The Program also incorporates theories of social support,
community integration, and empowerment. Social support refers to the presence and content of
personal relationships and the associated benefits to people that result from having those
relationships (Turner & Turner, 2016). The presence of personal relationships refers to the social
ties and network a person has and the content is the functional support one gets, emotionally,
materially or through guidance (Saegert & Capriano, 2017). These concepts have been studied at
length and have been demonstrated to provide many health benefits, including to both physical
and mental health through mechanisms like stress-buffering (Chang, Heller, Pickett, & Chen,
2013; Kerman, Sylvestre, Aubry, Distasio, & Schutz, 2019; Saegert & Capriano, 2017). The
Program makes social interaction and networking an essential component of the program’s
delivery, practices which are supported by this evidence. Participants are in regular contact with
their support team and are connected with services throughout the community.
Community integration is a concept intimately tied to many housing programs as it
stresses the building and maintaining of physical, social and psychological connections to the
community (Wong & Solomon, 2002). These connections manifest through spending time in a
community, having social interactions and building a social network, and by an individual
perceiving membership in a community and having emotional connections with other
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
16
community members (Wong & Solomon, 2002). These authors propose that these facets of
community integration are contingent upon the personal and local contexts within which the
housing program is being delivered; that is the housing, behavioural, and support environments.
These environments include the accessibility of community resources, the normalization of
housing (housing that is located among the mainstream population), the degree of participant
independence and the level of active support participants receive (Wong & Solomon, 2002). To
help facilitate community integration for its residents, The Program aims to house people directly
in community settings, stresses community interaction and works to help residents build life
skills to further their independence (Housing Services, 2017a).
The Program emphasizes empowerment in the way it is designed, from person-centered
recovery to community and social change. Power exists at various ecological levels, and hence
empowerment can occur at the individual, organizational, community and societal levels (Keys,
McConnell, Motley, Liao, & McAuliff, 2017). As power is gained at the individual level it may
lead to the empowerment of organizations or community groups those individuals belong to
(Keys et al., 2017). In this way, The Program not only aims to build empowerment for its
participants, but through their increased power, could help foster empowerment of communities,
organizations and the larger society. By providing a stable base of support (housing) and
developing personal skills, communities can re-integrate formerly marginalized individuals who
can contribute to organizations within the community for the betterment of the society.
The Program may focus on the individual but its impacts have the potential to be felt
outside of the realm of their housing supports. The Program also promotes empowerment
through the inclusion of peer support workers as housing support coaches on the home-based
support teams (Housing Services, 2017a). These roles are filled by people with a lived
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
17
experience of homelessness who understand the challenges of becoming housed and navigating
recovery from a personal perspective. Participating in peer support as either a provider or
recipient increases people’s empowerment, sense of independence and self-confidence by
exploring new ways of thinking and engaging in a process of mutually developing solutions
(Repper & Carter, 2011).
Fidelity Assessment
The demonstrated strength of the HF model in the U.S. and Canada has led to broad
international adoption. Projects that use the HF model exist in many European countries, as well
as New Zealand, and Australia (Australian Housing and Urban Research Institute, 2018; Housing
First Auckland, 2019; Pleace, 2016). With different local histories, government priorities, and
social welfare and housing systems, the implementation of HF in these new locations can take
different forms. As HF is adopted in the new locales, implementation concerns arise that can lead
to questions of program drift from core principles for implementation – that is, are the new
programs faithfully implemented to the HF model or is there deviation which could affect results
(Gaetz et al., 2013; Johnson, Parkinson, & Parcell, 2012; Stefancic et al., 2013)?
As HF targets a historically marginalized population with unique and complex needs,
proper training of staff and adequate implementation are further complicated when adapting a
model that originated in the U.S. (O’Campo, Zerger, Gozdzik, Jeyaratnam, & Stergiopoulos,
2015). With significantly different social and health care contexts as well as population and
geographic differences, the adaptation of HF to The Region may encounter unique challenges
(e.g. coordinating care and access between the cities and municipalities in the region). While
there is concern about implementation in different regions, studies have shown that programs can
adapt the HF model to specific contexts and populations while maintaining adherence to core
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
18
principles (Johnson et al., 2012; Nelson et al., 2014; Nelson et al., 2017; Stergiopoulos et al.,
2012).
As a means of ensuring appropriate implementation, it is common to use implementation
evaluations and fidelity assessments in order to assess how well a program is working and
whether there is any deviation from the accepted model (CDC, 2012). An implementation
evaluation is used in the early stages of a program, when information about program processes
can help improve how it operates (CDC, 2012). The Program is currently in the early
implementation phase and, as part of a fulsome implementation evaluation that will also assess
other aspects of program functioning and stakeholder views, this work will focus on a fidelity
assessment to determine how closely it is aligning with HF principles.
A fidelity assessment assesses the degree to which a program is implemented in
accordance with a program model or set of standards (Bond, Evans, Salyers, Williams, & Kim,
2000), and tends to be one part of a larger implementation evaluation. The use of fidelity
assessments in the implementation stages of a program helps to ensure consistency and correct
errors in implementation at an early stage (Macnaughton et al., 2015). This enables programs to
monitor implementation and adjust, as needed, in order to maintain theoretical integrity to the
model and overall program quality (Saunders, Evans, & Joshi, 2005). Fidelity assessments can
also help programs determine whether results of a program are due to the program model or
some other confounding factor (Moncher & Prinz, 1991). They provide a rich source of
information about strengths and weaknesses for specific design procedures and participant cases,
beyond simple checks of whether a protocol was followed or not (Hogue, Liddle, Singer, &
Leckrone, 2005).
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
19
For HF projects, the Pathways Housing First Fidelity Scale was created (Stefancic et al.,
2013) to score program fidelity to the HF model principles on a scale of one to four (four being
highest fidelity) (Nelson et al., 2014). The scale is comprised of 38 items categorized under five
overarching domains: (1) housing choice and structure (reflecting choice in type and location of
housing); (2) separation of housing and services (reflecting housing rights and responsibilities for
program participants); (3) service philosophy (used to reflect underlying HF philosophy); (4)
service array (used to assess the extent and availability of community support services); and (5)
program structure (reflecting other good programming practices, e.g., low participant/staff ratio)
(Nelson et al., 2014). Questions in each domain are specifically defined to ensure accuracy and
consistency in scoring. This scale allows evaluators to assess all aspects of a program and
provide specific feedback about the degree to which HF principles are being followed, rather
than a dichotomous yes or no. The Pathways HF Fidelity Scale has been used in many fidelity
evaluations, including in assessing programs of a similar geographical size (Tsemberis, Howard,
& Vandelinde, 2016) and during early and later implementation evaluations of the At
Home/Chez Soi study (Macnaughton et al., 2015; Nelson et al., 2014).
Another benefit to assessing program fidelity is the demonstrated link between fidelity in
implementation and participant outcomes. Durlak and DuPre (2008) conducted a systematic
review of nearly 500 studies examining the relationship between participant outcomes and
program implementation fidelity in a variety of program types (e.g., drug prevention, mental and
physical health promotion). The authors found extensive evidence that carefully implemented
programs achieve better outcomes for their participants. Programs that achieved high fidelity
tended to score well in areas of provider self-efficacy, program adaptability and organizational
capacity. Effective programs were able to successfully negotiate model adherence to local
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
20
contexts, had a high-level of provider and staff buy-in to program philosophy and were able to
provide a wide range of services to participants (Durlak and DuPre, 2008). These and other
metrics are all captured in the Pathways Housing First Fidelity Scale that was used to assess The
Program in this current research.
Evaluations by Macnaughton et al. (2015) and Nelson et al. (2014), used the Pathways
Housing First Fidelity Scale to evaluate fidelity in the five sites of the At Home/Chez Soi study.
Similar to the findings of Durlak and DuPre (2008), factors that contributed to good fidelity in
the At Home/Chez Soi study included the growing expertise of staff and their comfort with the
HF model and values, organizational capacity, and community partnerships (Macnaughton et al.,
2015). These factors influenced the programs’ ability to meet the needs of their participants on
every level – from staff support, to organizational and community resources. With strengths in
these areas, the At Home/Chez Soi sites maintained high-fidelity programs that worked to the
benefit of participants (Macnaughton et al., 2015). Factors that were found to impede fidelity in
the study sites included lack of support services for participants (e.g., mental health services),
staff turnover, participant isolation and an inability for some participants to successfully adjust to
being housed (Macnaughton et al., 2015; Nelson et al., 2014). In identifying these factors, the
programs can develop strategies to address these deficiencies, which may involve better supports
for staff and participants or developing more community connections.
A follow-up evaluation of each At Home/Chez Soi program site two years after the end
of the study reported 75% of sites still active in providing treatment and maintaining a high level
of fidelity (Nelson et al., 2017). Three of the five program locations had expanded their HF
services, demonstrating the commitment to, and success of, the model. Nelson et al. (2017)
identified several factors that influence sustainability including the amount of knowledge
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
21
dissemination the projects engaged in and the alignment between the HF model and government
policy and funding. The outcomes of the fidelity evaluations are used to improve the program by
identifying areas of implementation strength and weakness, and subsequently suggesting
adjustments.
Results from previous fidelity assessments indicate common successes and challenges
faced by other HF programs. Fidelity assessments of HF programs in different parts of the world
have found that the domains of separation of housing and services, and service philosophy tend
to score highly (Greenwood, Stefancic, Tsemberis, & Busch-Geertsema, 2013; Manning,
Greenwood, & Kirby, 2018; Nelson et al., 2014; Samosh et al., 2018; Tsemberis et al., 2016).
This indicates that many programs are developed with a strong foundation of Housing First
principles and are cognizant of the importance of helping participants normalize and maintain
their housing. Many programs experienced lower scores in the program structure and service
array domains, with problems related to having participant representation in the program,
adequate service coverage or providing employment and education services (Manning et al.,
2018; Nelson et al., 2014; Samosh et al., 2018; Tsemberis et al., 2016). Though tools for fidelity
assessments are being updated and adapted and the methods and measures between these
assessments may have been slightly different, these common themes should be noted and
examined for their applicability to The Program.
Research Aims
A fidelity assessment will provide a complete perspective of how The Program is
operating according to the HF model and where it can be improved to better serve its
participants. By understanding the theories that underlie The Program, recommendations for
program improvement can be developed to address both technical and theoretical components.
As such, this research strives to understand two main questions:
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
22
1) How is The Program being implemented with fidelity to the HF model and in
accordance to relevant theories?
2) How is The Program being implemented with fidelity to the intended goals of the
home-based support component of the program?
The final aim of this research is to provide concrete and actionable feedback to the
community partner. As a community psychologist, it is vital that research conducted in
community can be used to better those communities. As the evaluation process was highly
community-engaged, key partners on the evaluation, such as the organization that is delivering
The Program, will be provided with the fidelity scores, rationales for those scores, and consulted
to determine the appropriate steps to improve their program as it develops with the goal of
achieving the best outcomes for their participants.
Method
This research took place in a medium-sized Canadian municipal region in coordination
with community partners involved in The Program. Mixed methods were used to develop an
understanding of the adherence of The Program to the Housing First (HF) model in
implementation and to intended program goals of home-based support. The previously developed
Pathways HF Fidelity Scale (Stefancic et al., 2013) was used and adapted to include questions
about the unique home-based support aspect of The Program. The result was the addition of a
new domain, support and skills development, and six new items intended to capture the unique
and critical elements of the home-based support aspects of The Program. The adaptation was
done by myself, a master’s student, and my PhD supervisor, and developed items were
forwarded to the community partner for review and approval prior to use. This study has been
approved by the Research Ethics Board (REB) at our host University.
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
23
Community Partners
Throughout the course of the research, we worked with local partners from The Region.
This included the host organization for The Program, and the housing division of the regional
government that provides oversight for The Program.
Initially, we developed a relationship with community partners to build trust and an
understanding of common goals for the research project through the formation of an advisory
group. The advisory group was comprised of the four members of the research team, two
program leaders from the host organization as well as the program liaison form the regional
government and met semi-regularly to discuss research timelines, progress and method. Having
an advisory group ensures the incorporation of community partners’ experience, perspectives and
input throughout the research process (Newman et al., 2011). We then began familiarizing the
advisory group with the protocols of the previously created fidelity assessment procedure and the
associated materials (Aubry & Nelson, 2019), and gathered feedback on these materials.
Continuous consultation with community partners occurred throughout the project and outputs
were created that were specifically tailored to be useful for their organization (i.e. an executive
report that outlined key learnings in addition to a full report that further detailed the research
process).
FIDELITY EVALUATION OF A HOUSING FIRST PROGRAM
24
Participant Recruitment and Data Collection
The bulk of data collection took place during a one-day site visit in September 2019 to
the host organization in The Region. The research team for the site visit included a combination
of faculty members and students, including myself. In accordance with the fidelity protocol
designed by Nelson and Aubry (2019), a single site visit is all that is needed to gather the
required information and helps minimize the burden the research may have on the host
organization. Research team members took part in a training session prior to the site visit to
review the process for the site visit day; how to gather information during the team meeting
observation; how to deliver the scale in an interview setting; and how to score the scale. This
training was delivered by Dr. Geoff Nelson (part of the research team) as he has taken part in this
type of fidelity evaluation visit prior to this. The following figure (Figure 1) represents a
graphical representation of the site visit protocol.
Figure 1. Fidelity site visit protocol.
All participants were recruited using convenience sampling from various levels of the
program, including program leaders, staff and participants. Program leaders include a manager
and team supervisors of The Program, of which we interviewed 3 during the site visit. Program
staff are service providers involved with different aspects of The Program delivery and were
RESEARCH TEAM
ARRIVES ON SITE
OBSERVATION OF
STAFF MEETING
1ST ROUND OF
INTERVIEWS
WITH PROGRAM
LEADERS AND
STAFF
FOCUS GROUPS
WITH PROGRAM
PARTICIPANTS
2ND ROUND OF
INTERVIEWS
WITH PROGRAM
LEADERS AND
STAFF
PARTICIPANT
CHART REVIEW
RESEARCH TEAM
MEETING TO
DETERMINE FINAL
SCORES
PRELIMINARY
FEEDBACK TO
PROGRAM
