Minnesota State University, Mankato
Minnesota State University, Mankato
Cornerstone: A Collection of Scholarly
Cornerstone: A Collection of Scholarly
and Creative Works for Minnesota
and Creative Works for Minnesota
State University, Mankato
State University, Mankato
All Theses, Dissertations, and Other Capstone
Projects
Theses, Dissertations, and Other Capstone
Projects
2019
Assessing Facilitator Adherence for the Delivery of Cognitive
Assessing Facilitator Adherence for the Delivery of Cognitive
Training Programs to Older Adults
Training Programs to Older Adults
Lydia Fry
Minnesota State University, Mankato
Follow this and additional works at: https://cornerstone.lib.mnsu.edu/etds
Part of the Clinical Psychology Commons
Recommended Citation
Recommended Citation
Fry, L. (2019). Assessing facilitator adherence for the delivery of cognitive training programs to older
adults [Master’s thesis, Minnesota State University, Mankato]. Cornerstone: A Collection of Scholarly and
Creative Works for Minnesota State University, Mankato. https://cornerstone.lib.mnsu.edu/etds/904/
This Thesis is brought to you for free and open access by the Theses, Dissertations, and Other Capstone Projects
at Cornerstone: A Collection of Scholarly and Creative Works for Minnesota State University, Mankato. It has been
accepted for inclusion in All Theses, Dissertations, and Other Capstone Projects by an authorized administrator of
Cornerstone: A Collection of Scholarly and Creative Works for Minnesota State University, Mankato.
Assessing Facilitator Adherence for the Delivery of Cognitive Training Programs to Older
Adults
Lydia Fry
A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of
Masters of Arts in Clinical Psychology
Minnesota State University, Mankato
Mankato, Minnesota
May 2019
i
April 12, 2019
Assessing Facilitator Adherence for the Delivery of Cognitive Training Programs to Older
Adults
Lydia Fry
This thesis has been examined and approved by the following members of the student’s
committee.
_______Dr. Jeff Buchanan______________________
Advisor
_______Dr. Bradley Arsznov____________________
Committee Member
__________Dr. Don Ebel_______________________
Committee Member
ii
Table of Contents
Abstract……………………………………………………………………………………………1
Introduction………………………………………………………………………………………..2
Methods……………………………………………………………………………………………6
Results……………………………………………………………………………………………11
Discussion………………………………………………………………………………………..14
References………………………………………………………………………………………..18
Tables…………………………………………………………………………………………….20
Appendix A………………………………………………………………………………………22
Appendix B………………………………………………………………………………………24
Appendix C………………………………………………………………………………………25
Appendix D………………………………………………………………………………………26
Appendix E………………………………………………………………………………………29
Appendix F……………………………………………………………………………………….33
Appendix G………………………………………………………………………………………37
Appendix H………………………………………………………………………………………40
Appendix I……………………………………………………………………………………….41
1
Abstract
The majority of psychological research has focused on outcomes of intervention, while
there has been relatively little focus on measuring adherence to treatment protocol and how
competently interventions are delivered. Issues of adherence and competence apply not only to
psychotherapy research, but also in the field of cognitive training. The New England Cognitive
Center (NECC) has disseminated manualized cognitive training programs for older adults who
may be experiencing varying levels of cognitive decline. The purpose of the current study was to
develop measures of adherence and competence for the Active Mind cognitive training program
created by the NECC as well as to refine the content of the instruments and examine their
psychometric properties. The current study found that the Active Mind Adherence Instrument
and Global Adherence/Competence Instrument have fair to substantial inter-observer reliability.
These data suggest that program facilitators are completing the required tasks at a high level, and
are therefore, appropriately adhering to the treatment manual. Reliability data for the measure of
competence was somewhat lower, suggesting that assessing competence requires more
subjective judgement and improvements to the exiting instrument are needed. Results suggest
that the adherence measure can be used to provide detailed, constructive feedback to ensure the
proper delivery of the Active Mind cognitive training program, while the competency measure
requires modifications before it can be used to evaluate program facilitators.
2
Assessing Facilitator Adherence for the Delivery of Cognitive Training Programs to Older
Adults
Treatment integrity, also known as procedural integrity, is defined as the “degree to
which treatments are implemented as planned, designed, or intended and is concerned with the
accuracy and consistency with which interventions are implemented” (McIntyre et al., 2007).
Treatment integrity consists of two components; adherence and competence. Treatment
adherence is defined as the “extent to which a therapist used interventions and approaches
prescribed by the treatment manual and avoided the use of intervention procedures proscribed by
the manual” (Waltz et al., 1993). Adherence can be seen as an objective measure of whether a
person did or did not perform specific tasks during an intervention. Competence is defined as the
“extent to which the therapists conducting the interventions took the relevant aspects of the
therapeutic context into account and responded to these contextual variables appropriately”
(Waltz et al., 1993). In contrast to adherence, competence can be seen as a more subjective
measure of a persons’ knowledge of when to perform specific tasks related to an intervention.
Procedural integrity is an important construct to measure because without proper
adherence and competence a causal relationship between the independent variable (e.g., an
intervention) and dependent variables is more difficult to establish (Hagermoser, Sanetti, &
Kratochwill, 2008). In other words, if it is unclear how well an intervention was implemented,
variability in outcomes within an intervention group could be due to variations in treatment
adherence or competence. Utilization of structured treatment manuals represent one way by
which to improve treatment integrity. A manual provides operationally defined behavioral
expectations with simple and easily understood directions for someone to follow in order to
properly implement an intervention (Lane et al., 2004). Manuals have made the delivery of
interventions more consistent across facilitators and settings.
3
Although the importance of treatment integrity is known, the majority of the literature in
the area of psychotherapy outcome research has neglected the assessment of treatment integrity
at the expense of measuring treatment outcomes such as symptom reduction. Perepletchikova,
Treat, and Kazdin (2007) reported that within the field of psychotherapy only 3.5% of published
articles had adequate implementation of treatment integrity measures. The same researchers also
reported that 8.9% of published articles adequately implemented adherence procedures and only
1.5% implemented competency measures. With the established causal relationship between
treatment integrity and treatment outcome, measuring adherence and competence levels is
critical and requires greater attention in the psychotherapy outcome literature.
Treatment Integrity and Cognitive Training
The failure to measure treatment integrity is not endemic within the field of
psychotherapy outcome research. In recent years, there has been a growing interest in the
development of cognitive training (CT) programs for older adults. This increase in interest is
evident when Googling “cognitive training”, which produces links to over 43.5 million website
pages. CT is the “non-pharmacological method that aims to help people with early-stage
dementia make the most of their memory and cognitive functioning despite the difficulties they
are experiencing” with the goal to improve or maintain overall cognitive functioning or
functioning in a given cognitive domain such as memory, attention, or problem solving (Bahaer-
Fuchs, Clare, & Woods, 2013). CT programs implement standardized and structured tasks that
target differing cognitive domains. Tasks or “exercises” increase in difficulty as the individual
progresses through the training program. CT can be delivered in a variety of ways, however
training has primarily been delivered via computer or mobile devices (Finn & McDonald, 2014;
Goghari & Lawlor-Savage, 2018; Hard et al., 2015; Lu, Lin, & Yueh, 2017). Finn and McDonald
(2014) as well as Hard et al., (2015) used the highly popular Lumosity program as a CT
4
intervention. Lu, Lin, and Yueh (2017) created a CT program for mobile devices that mirrored
Lumosity.
On the other hand, some researchers have developed cognitive training that are delivered
via in-person classes, a situation that ideally involves the utilization of a structured or
manualized intervention as well as the measurement of treatment adherence and competence
(Cheng et al., 2012; Kuper, Gajewski, Frieg, & Falkenstein 2017). For example, Cheng et al.,
(2012) had participants complete 24 one-hour sessions that included a 15-minute lecture, 30
minutes of learning a specific cognitive technique or strategy, and 15 minutes devoted to
consolidating the newly practiced skill. However, no manual was prepared for this in-person CT
program and there was no mention of procedural integrity.
Similarly, Kuper, Gajewski, Frieg, and Falkenstein (2017) had participants complete four
weeks of CT programming that consisted of pencil-based exercises such as Sudokus. Facilitators
of these exercises also lectured participants on how these exercises are relevant to their cognitive
functioning. Similar to Cheng et al., (2012) there was no creation of a program manual and no
mention of treatment integrity.
The lack of manualized treatment extends beyond these two studies. In fact, there are
currently no CT studies that mention the measurement of treatment adherence or competence
(Barah-Fuchs, Clare, & Woods, 2013). Also, the failure to measure treatment integrity is not just
occurring within the field of psychotherapy research, but is also present within the research of
CT. Without manualized treatments and known treatment integrity of facilitators it is not
possible to definitely determine the effect that CT programs have on outcome measures.
The New England Cognitive Center CT Programs
In the absence of manualized training programs in the literature, the New England
Cognitive Center (NECC) has created multiple manualized CT programs that are designed for
5
individuals with varying degrees of cognitive impairment. Each program consists of 24 one-hour
sessions facilitated over the course of eight to twelve weeks. Each session has multiple guided
activities that focus on cognitive domains such as language, visual-spatial, memory (primarily
verbal memory), attention/concentration, and problem solving which gradually get more difficult
over the course of the program. Included with each class is a highly structured and detailed
sourcebook that serves as a treatment manual. The sourcebook provides instructions and scripts
for introducing exercises, example exercises, and instructions for properly assisting participants
in completing exercises.
Trained facilitators that work for the organization where programs are implemented
deliver the NECC programs. Facilitators help guide participants through each class so
individuals understand exercises and complete them to the best of their ability. Before a person is
able to facilitate a NECC cognitive training program, one must undergo extensive training.
Prospective facilitators are required to attend an in-person training course specific to the
cognitive training program that is to be implemented. This in-person training course is taught by
a NECC master trainer and lasts one to two full days depending on the number of programs that
facilitators are wanting to be trained in. After the in-person training, prospective facilitators then
observe the master trainer conduct a cognitive training session.
Purpose of the Study
Given this extensive training and availability of structured treatment manuals, it is
important to assess the degree to which facilitators adhere to the manual and how competently
they deliver the programs. The purpose of the current study is to develop instruments that
measure treatment adherence and competence for a suite of five manualized cognitive training
programs developed by the NECC. In addition, the study will examine the psychometric
6
properties of these instruments, with an emphasis on assessing inter-rater reliability and content
validity.
Methods
Participants
The participants were two facilitators of CT programs. The first participant was a 24-
year-old female. This participant has been employed as a Life Enrichment Coordinator for one
and a half years at a community center for older adults (“Facilitator A”). This participant
received and completed NECC cognitive program training in May of 2017. Participant A
facilitated three CT classes prior to the current study.
The second participant was a 58-year-old female. This participant has been employed as
an Activity Assistant for 17 years at a convent that provides a variety of housing options for
retired nuns (“Facilitator B”). Facilitator B has facilitated ten CT classes prior to the current
study. Facilitator B also completed the required NECC training. Following training, both
facilitators were provided a detailed instruction manual that described the content of each class
and instructions for how to complete each activity.
Settings
The organization that Facilitator A was employed at is an adult day services community
center for older adults with disabilities. In addition to providing Mind Aerobics courses, the
organization also offers health services, fitness services, and many clubs and activities for older
adults including hiking club and photography club. The Mind Aerobic sessions themselves were
conducted in an activity room with a large table in the center and the older adults participating in
the mind aerobics coursed seated around the table.
Facilitator B is employed at a Midwestern convent and living community for Catholic
sisters. In addition to providing Mind Aerobics courses, the convent also provides educational
7
and ministry services to the surrounding community. Similar to the settings of Facilitator A,
Facilitator B conducted the Mind Aerobics course in an activity room with a large table in the
center with the participants of the mind aerobics course seated around the table.
Materials
The NECC is “a non-profit organization… that develops and disseminates innovative,
research-based cognitive fitness programs. NECC combines the latest advances in
neuropsychological research with sound educational principles to create effective interventions
that enhance brain health, independence and quality of life. The organization’s focus is on older
adults who wish to maximize mental functioning and individuals with Alzheimer’s disease and
dementia”(New England Cogntive Center, n.d.). In line with their mission of helping older adults
with cognitive decline, the NECC created the Mind Aerobics suite of cognitive training
programs. The programs are designed to systematically stimulate six cognitive domains in order
to maintain or improve cognitive abilities. The suite of Mind Aerobics courses are designed for
individuals with varying degrees of cognitive impairment. These programs include Mind
Sharpener, Mind Works, Lively Mind, Active Mind, and Ready Mind. Mind Sharpener has the
target population of those with normal to forgetful cognitive decline, Mind Works targets those
with mild to moderate cognitive decline, Lively Mind targets those with moderate cognitive
decline, and Ready Mind targets individuals with severe cognitive decline.
Active Mind was specifically made for individuals with moderately severe cognitive
decline and for those who may have been diagnosed with moderate dementia. This program
consists of 24 one-hour sessions over the course of eight to 12 weeks. The program consists of
multiple guided exercises with manipulatives to support hands-on learning. For example, all
Active Mind sessions start with the exercise Rapid Response. This is an exercise in which
participants first use their left hand, then their right hand, and finally both hands to point at
8
colored numbers called out by the facilitator who progressively call out numbers faster and
faster. Other exercises target the following cognitive skills: attention/concentration, language,
short-term memory, visuospatial skills, and problem solving. Course content gets progressively
more difficult as the course continues such that later classes are more challenging compared ot
earlier classes.
Measures
All measures used were created by Dr. Jeff Buchanan under the guidance of Patti Said,
Executive Director of NECC and developer of the cognitive programs. During the consent
process, participants of the current study filled out the Facilitator Demographic Form (appendix
B). This form gathered demographic and background information of the participants including
age, years of experience in working with the mind aerobics population, how many Mind
Aerobics courses they have facilitated, and when they had completed the NECC Mind Aerobics
training.
The Active Mind Adherence Instrument (Appendix D) was created to measure if a
facilitator performed all proscribed instructions described in the cognitive training manuals
provided by the NECC. This instrument includes sections for all activities and exercises included
in the classes. The items in the instrument were developed based on instructions provided in the
program “sourcebook”, which is a manual that provides detailed instructions concerning how to
deliver the programs. When using the instrument, a “+” is recorded if the facilitator properly
performed the item in the given activity. However, if the facilitator did not perform a proscribed
instruction, a “-“ is recorded on the instrument for that specific item. This instrument is scored by
calculating the percentage of tasks completed (i.e., number of tasks successfully completed
divided by the total number of tasks required for the class).
9
The Global Adherence and Competence instrument (“GACI”; Appendix C) was also
created to measure how well facilitators conducted the classes. While the content of the
adherence instruments are specifically designed for each program, the GACI is a global measure
of facilitator competence that can be used regardless of which cognitive training program is
delivered. The original content of this measure was based on the experience and knowledge
researchers had with the Mind Aerobics programs. Content was then validated by having the
master NECC facilitator (Ms. Said) review the content of the instrument and make necessary
adjustments. The GACI uses a rating scale from 0-2 to evaluate how well the program was
delivered by the facilitator, with a rating of two being “excellent”, a rating of 1 being
“satisfactory”, and a rating of zero being an “unsatisfactory” performance. The items on the
measure correspond to the universal instructions for facilitators across the five Mind Aerobic
instructional manuals. Examples of the universal instructions and items in the GACI are
“facilitator regularly used the names of the participants”, “facilitator delivered praise and
encouragement to participants,” and “facilitator enunciated clearly.”
After video recordings were completed, participants completed the Post-Observation
Facilitator Follow-Up Questionnaire (Appendix H). This questionnaire includes four open-ended
questions designed to gather information from the facilitator concerning their opinions about the
Minds Aerobics course they facilitated. Example questions include, “Overall, how well did you
feel you facilitated the class?”, and “Are there any exercises in this class that you believe are
more difficult for the participants?”
After observers had completed watching the recorded sessions, they completed the Post-
Observation Observer Follow-Up Questionnaire (Appendix I). This questionnaire includes three
open-ended questions designed to gather information from the observer concerning their
experience using the adherence and competence instruments. Example questions include, “Were
10
there any programs/activities that were especially difficult to code?” and “Are there any
suggestions on improving the assessments?”
Procedure
Prior to data collection, participants received, reviewed, and signed a consent form
(Appendix A). After informed consent had be given, participants were given the Facilitator
Demographic Form (appendix B) to complete. Also prior to data collection, the researchers
mailed the participants GoPro Hero 3 cameras to place in the rooms in which the Mind Aerobic
classes were being held. Facilitators then filmed three of the 24 sessions of the class; one of the
first three sessions, one of the middle three sessions, and one of the last three sessions in order to
get a representative sample of the facilitator conducting different sessions throughout the entire
class. Facilitator A directed two Mind Aerobic classes two times a week for a total of 12 weeks
in the Spring of 2018 and the Fall of 2018. Facilitator B directed the Mind Aerobic sessions two
times a week for a total of 12 weeks in the Fall of 2018. Upon completion of the class,
facilitators mailed the GoPro’s back to the researchers. Facilitator A completed two classes,
which included six videos, and Facilitator B completed one class that included three videos.
Therefore, a total of nine video recordings were analyzed for the current study. The facilitators
also completed the Post-Observation Facilitator Follow-Up Questionnaire at the end of the Active
Mind course.
After video recordings were received by the researchers, two observers watched the
videos and scored the videos using the Active Mind adherence and competence instrument and
the GACI. The author of this paper served as the primary observer. Secondary observers were
undergraduate research assistants that were trained to use the adherence and competence
measures prior to video coding. This training first consisted of an explanation of the operational
definitions of the task items in the instrument. Next, the observers were required to watch
11
training videos and received feedback regarding their performance. The training videos consisted
of two 30-minute segments provided by Facilitator A for the current study. The first training
video consisted Facilitator A delivering the first half of the first Active Mind session. The second
training video consisted of the second half of the second Active Mind session provided by
Facilitator A. This allowed observers to be trained on the Active Mind adherence and
competence instrument for all exercises in the Active Mind course. Finally, training also
involved recalibration. Recalibration involved adjusting item content when there were
disagreements between the primary and secondary observers. When disagreements occurred, the
disagreement was discussed to determine what changes should be made to the instrument. The
training videos were then viewed again in order to come to agreement on the given task items.
Training and recalibration continued until a kappa of at least .80 was obtained. Additionally, Ms.
Said watched four of the nine video recordings using both the Active Mind adherence measure
and the GACI. Ms. Said served as a “master observer” given her intimate knowledge of the
programs and her involvement in the development of the adherence and competence instruments.
After video coding had ended for all nine videos, all observers completed the Post-Observation
Observer Follow-Up Questionnaire.
For data analysis, IBM’s Statistical Package for the Social Sciences (SPSS) was used to
calculate Kappa between the primary observer, secondary observers, and the master observer to
establish IOA reliability and validity values.
Results
Revisions to the Instruments
As mentioned previously, both instruments were reviewed by the NECC Executive
Director who developed the cognitive training programs. This was done as an informal way to
establish content validity of the instruments. This review resulted in some changes to the
12
instruments. In regards to GACI, Ms. Said formatted the page layout of the instrument and re-
worded the items in order for them to better correspond to the wording of the NECC Mind
Aerobics sourcebooks. In addition, Ms. Said added items 12 and 15. Item 12 states, “Facilitator
reviewed correct answers and solicited feedback in a timely manner prior to moving on to the
next activity.” Item 15 states, “Facilitator reminded participants as necessary that while correct
answers are nice, the purpose of the program is to strengthen cognition through challenge.
Attempting to get the correct answer is more important than the correct answer.”
In regards to the Active Mind Adherence Instrument, Ms. Said again formatted the page
layout of the instrument and as well as re-worded the instrument to be identical to the wording of
the Active Mind sourcebook. For example, Ms. Said changed the word “class” to “session.” She
also re-worded task items. For example, Ms. Said changed the task item, “The rationale for the
activity was provided (i.e., a short explanation and/or example of why is the skill important or
necessary in everyday life)” to “Rationale for activity was provided (i.e., short explanation and
examples of why the skill is important in everyday life).” Additionally, Ms. Said changed the
order of the task items to better correspond to the order in which the items occur within the
session.
During the course of training the undergraduate secondary observers, additional changes
were made to the Active Mind Adherence Instrument and the GACI. Regarding the Active Mind
Adherence Instrument, the number of trials required for each activity were added given that the
secondary observers were less familiar with the program. For example, the task item of,
“Facilitator delivered appropriate number trials” for the “Cube It! Adherence Criteria” the
addition of, “(sessions 1-7 require 2 trials, 8-12 require 1 trial)” was made. In regards to the
GACI, the anchors corresponding to each rating were changed. Originally, a rating of 0
corresponded to “unsatisfactory”, 1 corresponded to “satisfactory”, and 2 corresponded to
13
“excellent”. To better anchor these items for those who are not master observers, a more
descriptive anchor was added. This addition consists of 0 corresponding to the anchor, “item was
completed almost never”, 1 corresponding to the anchor, “item was completed most of the time”,
and 2 corresponding to the anchor, “item was always completed.”
Inter-Observer Agreement
Inter-observer agreement (IOA) was calculated by using Cohen’s kappa. Cohen’s kappa
was used as it takes chance agreement into account. As shown in Table 1, reliability between the
primary observer and secondary observers for the Active Mind Adherence Instrument
averaged .77 and had a range of .455-.954. Reliability between the primary observer and the
NECC master observer averaged .63, with a range of .38-.875. In general, these Kappa values
represent substantial agreement between observers.
As shown in Table 2, reliability between the primary observer and secondary observers
for the GACI averaged .3 and had a range of .103-.694. Reliability between the primary observer
and the NECC master observer averaged .391 with a range of .25-.667. In general, the Kappa
values represent fair agreement between observers.
Post-Observation Qualitative Data
In the Post-Observation Observer Follow-Up Questionnaire, all observers suggested that
overall the observations went well and that the instruments were easy to use. For example, the
master trainer (Ms. Said) stated, “observations went well and forms seem to be appropriate and
easy to use”. Observers also suggested that there were no specific Active Mind
activities/exercises that were more difficult to code compared to others. Observers offered no
suggestions on how to improve the assessments, only that it is recommended to watch a
maximum of two videos in one sitting to prevent fatigue.
14
In the Post-Observation Facilitator Follow-Up Questionnaire, Facilitator A and B felt that they
had facilitated the classes well. In the questionnaire, both facilitators pointed out that the
activities involving adding numbers were the most difficult to facilitate because participants
sometimes became confused during these activities. Facilitator B stated, “The add’em exercise,
even though it’s rather simple, was a bit difficult to get them to understand the concept of them
having to add up vertically and horizontally, for some reason that was difficult for them to
understand.” In addition, the facilitators believe that the activities requiring participants to add
numbers, answer questions after listening to a story, and activities requiring participants to
sequence events were most difficult to perform. In regards to suggestions the facilitators had to
improve training for new facilitators, Facilitator A suggested having better explanations of why
they are teaching the class, whereas Facilitator B suggested having strategies to keep the courses
light hearted, fun, and encouraging.
Discussion
Real World Implications
The results of this study suggest that the Active Mind Adherence Instrument has
excellent reliability and the GACI has fair/adequate reliability. In addition, content validity of the
instruments was established through expert review completed by an NECC master trainer and
developer of the Active Mind program. The results of the current study have implications for the
NECC, particularly with regard to how they train and evaluate facilitators of the Active Mind
program. Specifically, NECC staff will be able to use these instruments while training new
facilitators to ensure that they are performing all required tasks before they are able to facilitate
the Active Mind course. This allows the NECC master trainer to ensure that new facilitators are
truly adhering to the Active Mind protocol, as well as ensure that the new facilitators are
competent in the delivering protocols in the intended manner.
15
The NECC will also be able to use these instruments in follow up booster training
sessions. It is common for the NECC master trainer to travel to the facilities that are providing
CT programs to older individuals in order to conduct refresher trainings. The instruments
examined in this study will provide a structured method by which to evaluate facilitators and
provide constructive feedback to ensure the NECC courses are being properly administered.
With the reassurance that courses are being facilitated properly, the NECC will better be able to
state with confidence that the CT outcomes for individuals are due to the activities and exercises
of the CT programs and not confounding variables due to the delivery of the programs. For
example, the level of attention a facilitator gives to participants of a CT program may have an
effect on the outcomes participants’ experience (Bahar-Fuchs, Clare, & Woods, 2013). If a
facilitator does not follow the proscribed level attention as described in the NECC sourcebooks
either by increasing or decreasing the level of attention paid to the participants, the facilitator is
now acting as a confounding variable to CT program outcomes.
The potential increase in adherence and competence of facilitators may also increase the positive
outcomes of individuals participating in the cognitive training programs. It has been shown that a
reduction in adherence to an intervention protocol reduces the positive outcomes experienced by
those involved (McIntyre et al., 2007). Positive outcomes experienced by those who have
participated in CT programs include maintaining and improving cognitive function in many
domains such as memory, attention, and problem solving (Barha-Fuchs, 2013). Having
facilitators who are properly trained to comprehend and adhere to the Mind Aerobics
sourcebooks and who receive feedback during follow up booster training may improve outcomes
for participants.
16
Limitations
There are several limitations associated with the current study. The first is reactivity.
Reactivity is the change in behaviors of participants due to being observed (Lipinski, & Nelson,
1974). The facilitators who served as participants in the current study may have had reactive
behaviors, as well as the individuals participating in the Active Mind courses may also have had
reactive behaviors as all involved were aware of the observations taking place. For example,
facilitators may have been more adherent or behaved in different ways knowing that they were
being videotaped and that these video recordings would be observed by research staff.
Additionally, the potential change in behaviors of the individuals participating in the Active Mind
program may have also affected the facilitators’ ability in performing the prescribed task items of
the class. Prior to starting data collection, reactivity was taken into account and minimized.
Reactivity was reduced by using small GoPro cameras placed in the corners of the room so as to
be as unobtrusive and unnoticeable as possible.
Another limitation to the current study was the lack of consistent and coordinated training
between the master observer and the primary and secondary observers. Although, the primary
observer did train the secondary observers in the use of the Active Mind Adherence Instrument
and GACI, the primary observer did not receive any training by the master observer. In addition,
there was no communication with the master observer in terms of defining different components
of competence and providing examples and non-examples of the task items in the measures. This
lack of training between the master and primary observers may have resulted in lower IOA for
the measure of competence than what would have been possible.
Future Research
For future research, it would be beneficial to address and increase the agreement between
observers using the GACI. Most of the disagreement between observers was due to task items 16
17
through 18 that are only scored “if applicable.” The clarity of the task items are not in question,
as none of the observers had suggestions for clarifying and increasing the understanding of the
measures. Thus, it is assumed that determining when a task item is applicable or not is the
determining factor in the disagreements that occurred. Agreement can be increased between
these three task items by receiving training by the master observer as to when these items are
most applicable and providing more examples and non-examples of when the task items would
and would not be scored.
It is also imperative that future research determines the IOA levels of the Lively Mind,
Mind Sharpener, and Ready Mind adherence assessments (Appendix E-G, respectively). Similar
to the Active Mind Adherence Instrument, these instruments were revised by the NECC master
trainer in order to informally establish content validity, so their content mirrors the Active Mind
Adherence Assessment. Given that the current study found excellent IOA for the Active Mind
Adherence Assessment and due to the likeness of the additional assessments, it is reasonable to
suggest that these other adherence instruments will have similarly high IOA. However, it cannot
be assumed that these other instruments with have equally high IOA, so they must be
systematically evaluated in future research. Determining the IOA reliability of all of the
adherence instruments will allow the NECC to train, follow up, provide feedback, and assess
adherence/competence for facilitators who are not just facilitating the Active Mind class but to
those who are also facilitating the other NECC programs.
18
References
Bahar-Fuchs, A., Clare, L., & Woods, B. (2013). Cognitive training and cognitive rehabilitation
for mild to moderate Alzehimer’s disease and vascular dementia (review). The Cochran
Library 6, 1-103.
Hagermoser Sanetti, L.M., & Kratochwill, T.R. (2008). Treatment integrity in behavioral
consultation: Measurement, promotion, and outcomes. International Journal of
Behavioral Consultation and Therapy 4(1), 95-114.
Hsu, L.M., & Field, R. (2003). Interrater agreement measures: Comments on kappan, Cohen’s
kappa, Scott’s , and Aickin’s . Understanding Statistics, 2(3), 205-219.
Lane, K.L., Bocian, K.M., MacMillan, D.L., & Gresham, F.M. (2004). Treatment integrity: An
essential-but often forgotten-component of school-based interventions. Preventing School
Failure, 48, 36-43.
Lipinski, D., & Nelson, R. (1974). Problems in the use of naturalistic observation as a means of
behavior assessment. Behavior Therapy, 5, 341-351.
Luborksy, L., & DeRubeis, R.J. (1984). The use of psychotherapy treatment manuals: A small
revolution in psychotherapy research style. Clinical Psychology Review, 4, 5-14.
McIntyre, L.L., Gresham, F.M., DiGennaro, F.D., & Reed, D.D. (2007). Treatment integrity of
school-based interventions with children in the journal of applied behavior analysis 1991-
2005. Journal of Applical Behavior Analysis 40(4), 659-672.
New England Cognitive Center (n.d.) Who we are. Retrieved from
http://cognitivecenter.org/who-we-are
19
Newman, C.F. (2010). Competency in conducting cognitive-behavioral therapy: Foundational,
functional and supervisory aspects. Psychotherapy Theory, Research, Practice, Training
47(1), 12-19.
Oltra-Cucarella, J., Perez-Elvira, R., Espert., R., & McCormick, A.S. (2016). Are cognitive
interventions effective in Alzheimer’s disease a controlled meta-analysis of the effects of
bias. Neuropsychology 30(5), 631-652.
Perepletchikova, F., Treat, T.A., & Kazdin, A.E. (2007). Treatment integrity in psychotherapy
research: Analysis of the studies examination of the associated factors. Journal of
Consulting and Clinical Psychology 75(6), 829-841.
Stauffer, M.D., & Pehrsson, D.E. (2012). Mindfulness competencies for counselors and
psychotherapists. Journal of Mental Health Counseling 34(3), 227-239.
Waltz, J., Addis, M.E., Koerner, K., & Jacobson, N.S. (1993). Testing the integrity of a
psychotherapy protocol: Assessment of adherence and competence. Journal of
Consulting and Clinical Psychology 61(4), 620-630.
Weeasekera, P., Antony, M.M., Bellissimo, A., Bieling, P., Shurina-Egan, J., Spencer, A.,
Whyte, R., & Wolper-Zur, A. (2003). Competency assessment in the McMaster
psychotherapy program. Academic Psychiatry 27(3), 166-173.
20
Tables
Table 1
Calculated IOA for the Active Mind Adherence instrument
Observers
Kappa Value
Range
Primary
observer and
undergraduate
secondary
observers
.77**
.455-.954
Primary
observer and
master observer,
Ms. Said
.63**
.38-.875
Note: ** denotes substantial agreement, * denotes fair agreement
21
Table 2
Calculated IOA for the GACI
Observers
Kappa Value
Range
Primary
observer and
undergraduate
secondary
observers
.30*
. 103-.694
Primary
observer and
master observer,
Ms. Said
.391*
.25-.667
Note: ** denotes substantial agreement, * denotes fair agreement
