9834_External Validation Of An Electronic Phenotyping Algorithm To Detect Attention To Elevated Bmi And Weight-Related

Yale University
Yale University
EliScholar – A Digital Platform for Scholarly Publishing at Yale
EliScholar – A Digital Platform for Scholarly Publishing at Yale
Yale Medicine Thesis Digital Library
School of Medicine
January 2020
External Validation Of An Electronic Phenotyping Algorithm To
External Validation Of An Electronic Phenotyping Algorithm To
Detect Attention To Elevated Bmi And Weight-Related
Detect Attention To Elevated Bmi And Weight-Related
Comorbidities In Pediatric Primary Care.
Comorbidities In Pediatric Primary Care.
Anya Golkowski Barron
Follow this and additional works at: https://elischolar.library.yale.edu/ymtdl
Recommended Citation
Recommended Citation
Golkowski Barron, Anya, “External Validation Of An Electronic Phenotyping Algorithm To Detect Attention
To Elevated Bmi And Weight-Related Comorbidities In Pediatric Primary Care.” (2020). Yale Medicine
Thesis Digital Library. 3905.
https://elischolar.library.yale.edu/ymtdl/3905
This Open Access Thesis is brought to you for free and open access by the School of Medicine at EliScholar – A
Digital Platform for Scholarly Publishing at Yale. It has been accepted for inclusion in Yale Medicine Thesis Digital
Library by an authorized administrator of EliScholar – A Digital Platform for Scholarly Publishing at Yale. For more
information, please contact elischolar@yale.edu.

External Validation of an Electronic Phenotyping Algorithm to Detect Attention to
Elevated BMI and Weight-Related Comorbidities in Pediatric Primary Care

A Thesis Submitted to the
Yale University School of Medicine
in Partial Fulfillment of the Requirements for the
Degree of Doctor of Medicine

By
Anya Golkowski Barron
2020

ii
ABSTRACT

External Validation of an Electronic Phenotyping Algorithm to Detect Attention to
Elevated BMI and Weight-Related Comorbidities in Pediatric Primary Care.
Anya Golkowski Barron1, Christy Turer 2, Ada Fenick1, Kaitlin Maciejewski1, and Mona
Sharifi1.
1Department of Pediatrics, Yale University, School of Medicine, New Haven, CT.
2Department of Pediatrics, University of Texas Southwestern Medical Center and Children’s Health, Dallas, TX.

Pediatric obesity is a growing national and global concern with nearly 1 in 5
children in the U.S. affected [1].The American Academy of Pediatrics endorsed expert
committee recommendations in 2007 to assist clinicians in pediatric weight management;
however, adherence to these recommendations among primary care providers is
suboptimal, and measuring adherence in feasible and pragmatic ways is challenging[2-4].
Commonly used quality measures that rely on billing data alone are an inadequate
measure of provider attention to weight status in pediatric populations as they do not
capture whether providers communicate about elevated body mass index (BMI) and
associated medical risks with families. Electronic phenotyping is a unique tool that has
the ability to use multiple areas of stored clinical data to group individuals according to
pre-defined characteristics such as diagnostic codes, laboratory values or medications.
We examined the external validity of a phenotyping algorithm, developed previously by
Turer et al and validated in a single health system in Texas, that assesses pediatric
providers’ attention to obesity and overweight using structured data from the electronic
health record (EHR), to three pediatric primary care practices affiliated with Yale New
Haven Health. Well child visit encounters were labeled either “no attention”, “attention to
BMI only”, “attention to comorbidity only,” or “attention to BMI and comorbidity”. The
performance of the algorithm was evaluated on the ability to predict “no attention”, using

iii
chart review as the reference standard. The application of the minimally altered algorithm
yielded a sensitivity of 94.0% and a specificity of 79.2% for predicting “no attention”,
compared to a sensitivity of 97.9% and a specificity of 94.8% in the original study. Our
findings suggest that while electronic phenotyping using structured EHR inputs provides
a better evaluation of clinic encounters than use of diagnostic codes alone, methods that
incorporate information in unstructured (“free text”) clinical notes may yield better
results.

iv

Acknowledgements

To Julian and Brian for being the salt and light of my world.

To my parents for being the giants whose shoulders I stand on.

To Dr. Mona Sharifi the office of student research and the department of Pediatrics
without whom this work would not be possible.

v
TABLE OF CONTENTS

1. Abstract…………………………………………………………………….. ii
2. Acknowledgements………………………………………………………….iv
3. Introduction…………………………………………………………………..1
a. Definitions of Pediatric Overweight and Obesity………………………… 1
b. What does Pediatric Overweight and Obesity look like in the US………. 2
c. Current Guidelines on Addressing Pediatric Obesity……………………. 3
d. Current Practice vs. Guidelines…………………………………………….. 5
e. Methods of Assessing Provider Attention to Pediatric Weight Status…… 6
4. Statement of Purpose……………………………………………………….. 8
5. Methods………………………………………………………………………10
6. Results……………………………………………………………………….18
7. Discussion……………………………………………………………………23
8. References……………………………………………………………………26

INTRODUCTION

1. Definitions of Pediatric Overweight and Obesity
Overweight and obesity are clinical terms used to denote excess body weight, most
frequently thought of in the form of adipose tissue. A commonly used measure for
estimating body fat percentages in medicine is body mass index (BMI). BMI provides a
measure of body weight adjusted for height, and although it does not provide a direct
measure of body fat, levels do correlate with and are predictive of future adiposity [5].
BMI is also clinically useful as it can easily be assessed in the primary care setting with
routine measurements of height and weight as opposed to more precise but less feasible
methods such as dual-energy x-ray absorptiometry. Given the nature of the calculation,
BMI may overestimate adiposity in children who have shorter statures or higher muscle
mass and may underestimate adiposity in children with very low muscle mass. However,
given its low cost, clinical utility and practicality, it is broadly used in clinical
environments. It is therefore applied as an initial screen in assessing a patient’s risk for
obesity and obesity-related comorbidities. Due to the fact that children’s BMI
measurements change dramatically with age and differ with sex, age-and sex-specific
BMI percentiles based on the Center for Disease Control (CDC) growth charts are used in
place of raw BMI values[6]. Cutoff points for increased health risks are defined
according to the 2007 expert committee recommendations convened by the department of
Health and Human Services[5]. These guidelines suggest that a BMI of less than the 85th
percentile is unlikely to pose health risk, whereas a BMI greater than or equal to the 95th
percentile would confer significant risk. The terms “overweight” are therefore applied to
a BMI ³ 85th percentile and “obesity” to a BMI ³ the 95th percentile. While the CDC

2
growth charts are useful for a large percentage of patients with overweight and obesity,
BMI percentiles beyond the 97th percentile are not clinically useful, as large changes in
BMI result in small percentile changes at the extreme. Therefore an additional metric,
percentage of BMI at the 95th percentile (%BMIp95), is used to better assess and follow
patients with severe obesity, defined as a BMI greater than or equal to 120% of the 95th
percentile for age and sex[7].
2. What Does Pediatric Obesity and Overweight look like in the US?

On a population level, obesity disproportionately affects children from
racial/ethnic minority backgrounds. African-American and Latino children display higher
BMI scores from a young age and maintain a higher BMI growth trajectory compared to
their non-Hispanic White counterparts [8]. According to some studies looking at
disparities in obesity prevalence, obesity seems to emerge and is sustained earlier in
Hispanic children relative to African Americans, but both groups experience higher BMIs
by the 8th grade relative to non-Hispanic White children[9]. The morbidities associated
obesity, such as hypertension and type II diabetes, are also disproportionately diagnosed
in minority children and tend to be seen more in boys [10]. Having diseases such as
elevated blood pressure or diabetes in childhood confers further risk of these diseases
carrying on into adulthood and increases overall risk of mortality from cardiovascular or
metabolic diseases [10, 11].
The risk factors associated with obesity are complex and intertwined. In general,
poverty is positively associated with obesity prevalence[5]. There is evidence that genes
play a role in obesity risk, and having one or both parents with obesity, increases the risk
of a child developing obesity significantly [12]. However, the rapid increase in

3
prevalence at a population level suggests that environmental factors play a greater role
than genetic shifts in the population[11]. Many associations with obesity risk such as
infant birth weight, increased screen time, sleep patterns, and neighborhood-level factors
have been described, but their interdependence and individual contribution to a patient’s
risk are largely undefined, making prediction, and prevention particularly difficult[6, 13-
16].
Childhood obesity and overweight have shown to be predictors of future obesity,
putting patients at risk for the eventual development of obesity-related comorbidities.[17] The medical complications of obesity are far reaching and include a range of life altering
disorders including hypertension, diabetes mellitus, non-alcoholic fatty liver disease,
dyslipidemia, asthma, and sleep apnea[18]. Managing these co-morbidities incur
significant cost to individual patients and healthcare systems. One study estimates the
lifetime cost for elementary students aged 6-11 with obesity to be $31,869 for boys and
$39,815 for girls due mainly to the care required for comorbidity management [19].
3. Current Guidelines on Addressing Pediatric Obesity
In 2007, an expert committee was formed to revise the 1998 recommendations on
childhood obesity. The recommendations were rooted in the latest evidence-based data
and the experience of clinical experts to address prevention, assessment, and treatment of
childhood overweight and obesity. The guidelines suggest that all children ages 2 years
and older be screened with initial BMI measurements, family history of obesity and
obesity-related disorders, and current diet and lifestyle practices. If a patient has a BMI
that is ≥85th percentile, the first steps a provider should take are to assess the medical and
behavioral risks of the individual patient. Medical risk assessment includes screening for

4
common comorbid conditions such as hypertension, type 2 diabetes, hyperlipidemia, and
non-alcoholic fatty liver disease. It was recommended by the committee that laboratory
tests to screen for and diagnose such conditions be conducted every 2 years for children
ages 10 years and older with obesity (or with overweight if they have associated risk
factors) [5]. Behavioral assessment includes identifying obesogenic behaviors such as
elevated screen time, fast food consumption, sugar-sweetened beverage intake, and
sedentary lifestyle. Providers should then take steps to address overweight and obesity,
and the guidelines make suggestions of four different treatment stages. These stages are:
stage 1 prevention plus, stage 2 structured weight management, stage 3 comprehensive
multidisciplinary approach and stage 4 tertiary care intervention. Each stage builds from
office-based counseling for lifestyle and family recommendations (stage 1) to nutrition
and psychological counseling (stages 2 and 3). Stage 4 uses interventions such as
medications, very low calorie diets, and bariatric surgery[5]. In cases of a child not
reaching a desired weight goal or in the presence of significant comorbidities,
pharmacotherapy can be considered. Orlistat is the only FDA approved medication for
the treatment of overweight and obesity in adolescents. Moderate improvements in BMI
have been associated with the use of Orlistat however, unlike in adult counterparts,
improvement in lipids or insulin sensitivity have not been consistently shown. Metformin
has also shown some ability to improve BMI in some short-term obesity studies when
used in conjunction with lifestyle modifications. Reported results on lipid and insulin
sensitivity have been variable and Metformin is not FDA approved for weight reduction
in pediatric patients [20].
4. Current Practice vs. Guidelines

5
Pediatric primary care providers (PCPs) are the cornerstone of addressing
pediatric obesity as many successful interventions rely on PCPs to screen for and manage
children with elevated BMI [21, 22]. Studies report that patients and families see their
primary care provider as a reliable source of information and their recommendations have
positive impacts on weight management [23, 24]. However, suboptimal rates of diagnosis
of overweight and obesity based on BMI percentile in pediatric primary care persist [25,
26]. One 2011 study based on self-reported practice, found that less than 50% of primary
care providers assessed BMI regularly in children and 58% reported rarely, or only
sometimes using BMI percentiles to track weight [27]. Another 2011 study, found that
pediatric providers reported unfamiliarity with the 2007 practice guidelines and
diagnostic criteria for overweight and obesity suggesting that uptake of new practices has
been slow[28]. Use of the Electronic Health Record (EHR) imparts the ability to auto-
calculate BMI percentiles theoretically improving provider attention and diagnosis. Yet,
despite some improvement with broad implementations of the EHR, children with
overweight and obesity are still underdiagnosed[25, 29]. Counseling behaviors amongst
providers have also been shown to be variable depending on factors such as sex, personal
beliefs and attitudes[25]. In particular, younger children (2-5 years old) and children
with overweight are more likely to be underdiagnosed, not receive diet and exercise
counseling and have an absence of screening studies [2, 25, 30]. Perceived barriers to
providing adequate care are often reported to be the sensitivity of the topic, clinic time
constraints, and feelings of futility [3, 31, 32]. These inconsistencies across providers
present missed opportunities to engage with families early, influence BMI trajectories,
and provide high-quality care.

6
5. Methods of Assessing Provider Attention to Pediatric Weight Status
Given the growing need for PCP attention to childhood obesity and the
suboptimal rates of diagnosis and screening, it is important to identify methods to support
clinicians in this task. Broad use of the EHR puts researchers in a position to easily
collect large amounts of data regarding physician practice. While manual chart review is
still widely done, the process is laborious and may often limit sample sizes. Electronic
phenotyping involves automated identification of subjects based on exclusion and
inclusion criteria present in stored clinical data. Electronic phenotyping is typically used
to identify patients with certain characteristics for a given purpose i.e.; a clinical trial, or
retrospective study. In a study published in 2018 titled, “Algorithm to detect pediatric
provider attention to high BMI and associated medical risk,” Dr. Christy Turer and
colleagues developed an electronic phenotyping algorithm using extractable EHR
variables to indicate adherence to the 2007 expert committee guidelines on childhood
obesity. Using diagnostic codes, laboratory studies, referrals, medications and procedures
they categorized provider behavior in response to elevated BMI measurements into one
of three phenotypes: “no attention”, attention to “BMI Alone”, and attention to
“BMI/Medical Risk”. Validation of the performance of the electronic phenotypes using
manual chart review showed excellent sensitivity and specificity to detect provider
attention types in pediatric clinics in Dallas, Texas[33]. By employing an algorithm to
evaluate clinician behavior, Turer et al created a tool that went beyond identifying
patients with disease characteristics to identifying encounters that follow guideline-based
care. Furthermore, a follow up study published in 2019 by Turer et. al, demonstrated that
children categorized by the algorithm as having primary care visits with attention to

7
elevated BMI and/or obesity-related medical risk were more likely to have improvement
in weight status at follow-up visits [34]. Based on the results of these studies out of Texas
and the anticipated benefits of using electronic phenotypes to augment provider practices,
we sought to replicate and externally validate the Turer algorithm[33] in the Yale New
Haven pediatric primary care setting.

8

STATEMENT OF PURPOSE

Hypothesis: We anticipate that the algorithm developed by Turer et. al 2018 to identify
attention to elevated BMI and weight-related comorbidities among pediatric primary care
clinicians would be applicable to 6-12 year-old children with overweight/obesity, defined
as a BMI ≥85th percentile, seen for well child visits in Yale New Haven Health pediatric
primary care practices. Specifically, we hypothesize that implementation of this
algorithm among patients at Yale-affiliated practices will yield a sensitivity and
specificity for predicting attention (per manual review of EHR documentation) that are
similar to the Turer study at a health system in Dallas, Texas. We also anticipate, based
on data from previous research, that children with obesity or severe obesity will be more
likely to be assigned an attention category in comparison to children with overweight,
and non-Hispanic Black and Hispanic children will be more likely to be assigned an
attention category than their non-Hispanic White counterparts [2, 25, 30, 35]. We expect
to see variations on clinical practice based on trainee level as has been documented
previously and therefore predict that children with encounters in the summer months
(July- September), when new physicians begin their residency training, will be more
likely to receive “no attention” than children seen later in other months [28, 36, 37].
Lastly, we predict that children with public versus private health care payors will be more
likely to receive higher levels of attention.
Specific Aim 1: To externally validate the algorithm described by Turer et. al 2018
among 6-12 year old children with overweight/obesity seen at the Yale New Haven
Hospital-affiliated pediatric primary care practices.

9
Specific Aim 2: To examine associations between pediatric provider attention and 1)
weight category (overweight, obesity, severe obesity), 2) insurance type, 3) race/ethnicity
and 4) season of encounter.

10

METHODS

Data Source:
We examined the records of 300 randomly selected patients ages 6-12 with two or
more measurements of elevated BMI percentiles (³85th percentile) for age and sex who
were seen for well child visits on two or more occasions at any one of three pediatric
primary care practices in the Yale New Haven Health system: the Yale Pediatric Primary
Care Clinic (PCC), Yale Health Center (YHC), and Saint Raphael Campus Primary Care
Clinic (SRC) from June 1, 2018 to May 31, 2019. If multiple encounters for the same
patient occurred within that time period, we examined the encounter from the first
chronological date. We categorized children’s weight status based on the Center for
Disease Control (CDC) growth charts from 2000 which classifies BMI-for-age into the
following categories stratified by sex: overweight ³ 85th to < 95th percentile, and obesity ³95th percentile[38]. The intention of this study was to only examine patients with overweight or obesity. Exclusion Criteria: Patients were excluded from the study if they had less than two recorded BMI measurements above the 85th percentile to ensure that we were not examining visits with aberrant or incorrect BMI recordings. We also excluded children that were taking medications or had conditions that impact growth and nutrition (e.g., pregnancy, thyroid dysfunction, growth hormone abnormalities and sex hormone abnormalities). Measures and Data Collection: We extracted the following variables from the medical records of eligible patients: 11 a. Visit and problem list diagnosis codes entered on the date of the encounter b. Referrals entered on the date of the encounter c. Procedures/ lab orders entered on the date of the encounter d. Medication lists queried for prescriptions written on day of the encounter e. Age calculated in months based off of patient’s birthdate and age at visit f. BMI calculated using height and weight on the date of the visit g. BMI categorization defined as overweight (≥85th – < 95th percentile), obese (≥95th - <120% of the 95th percentile) and severely obese (>120% of the
95th percentile) using CDC growth charts BMI for age.
h. Sex (Male or Female)
i. Race/ethnicity defined as non-Hispanic Black, non- Hispanic White,
Hispanic and Asian.
j. Insurance type defined as public (Medicaid), private (Blue Cross Blue
Shield, Managed, or other commercial insurance), and Uninsured (self-pay or
missing)
k. Provider type: defined as Nurse Practitioner, Physician Assistant,
Physician (Attendings and Fellows), and Resident
Construction and Implementation of Algorithm to Detect Provider Attention:

The algorithm was modeled after the electronic phenotype described by Turer et.
al[33]. After collecting the diagnostic codes, laboratory studies, medications, procedures
and referrals used by our cohort, patient visits were classified into the following broad
attention types: No Attention, Attention to BMI alone, Attention to Comorbidities alone,

12
and Attention to BMI and Comorbidities. The cohort was then sub-classified into
comorbidity subtypes (Attention to Diabetes, Attention to Fatty Liver Disease, Attention
to Hyperlipidemia and Attention to Vitamin D Deficiency) based on criteria listed by
Turer et al (Figure 1). Criteria for classifying visits into attention types are listed in Table
1 for broad categories and Table 2 for comorbidity sub-types. The criteria were defined
by reviewing the diagnostic codes, laboratory studies, medications, referrals and
procedures used by Turer’s team and comparing them to the corresponding values used in
our population. Given that the original study was conducted using ICD-9 diagnostic
codes, we first converted all codes into ICD-10 using the following website:
http://www.icd10codesearch.com/.

13
Table 1. Diagnostic codes, laboratory studies, referrals and procedures that were used to qualify for each attention type
based on the data from Yale pediatric primary care practices

Attention Type

Diagnosis Code (ICD-10)

Medicines and
Laboratory Studies

Referrals
and
Procedures

BMI alone
E66.09 Obesity due to excess calories, unspecified
obesity severity
E66.09, Z68.54 Obesity due to excess calories
without serious comorbidity with body mass index
(BMI) in 95th to 98th percentile for age in
pediatric patient
E66.3 Overweight
E66.3, Z68.53 Overweight peds (BMI 85-94.9
percentile)
E66.3, Z68.54 Body mass index (BMI) of 95th to
99th percentile for age in pediatric patient
E66.9 Obesity (BMI 30-39.9)/Obesity, unspecified
classification, unspecified obesity type,
unspecified whether serious comorbidity present
E66.9, Z68.53 Obesity, pediatric, BMI 85th to less
than 95th percentile for age
E66.9, Z68.54 Obesity peds (BMI >=95
percentile)/BMI (body mass index), pediatric 95-
99% for age, obese child structured weight
management/multidisciplinary intervention
category
R63.3 Feeding difficulties
R63.5 Weight gain/abnormal weight gain
Z68.41BMI 40.0-44.9, adult (HC Code)
Z68.53 BMI (body mass index), pediatric, 85% to
less than 95% for age
Z68.54 BMI pediatric, greater than or equal to
95% for age
Z71.3 Nutritional counseling
Z72.4 Inappropriate diet and eating habits

Orlistat
DNA methylation analysis for
Angelman or Prader Willi
Syndrome
Mutation analysis for Angelman or
Prader Willi Syndrome
*No patients in our population were
taking medicines to treat obesity
*No patients in our population had
lab studies for Angelman, Prader
Willi syndrome targeted gene
mutation analysis or DNA
methylation analysis DNA
methylation analysis
Nutrition
counseling
Exercise
counseling
Nutrition Referral

Attention to BMI and
Comorbidity
³1 diagnosis code for BMI and ³1 diagnosis code
for:
•
Elevated blood pressure/hypertension
•
Acanthosis, prediabetes, diabetes type 2
•
Lipid disorders
•
Fatty liver disease
•
Vitamin D deficiency
*See table 2 for codes broken up by comorbidity

Labs to screen for diabetes, lipid
disorders, fatty liver disease, or
vitamin D deficiency (2 lab orders
required, because increased
likelihood lab ordered to screen
for comorbidities related
to overweight/obesity)
• Medicines to treat hypertension,
diabetes, lipid disorders, or
vitamin D deficiency

*See table 2 for laboratory studies
broken up by comorbidity

Referral to
tertiary weight
management
clinic, other
medical
specialist
(endocrinologist,
GI, Cardiologist,
etc)

*See table 2 for
referrals broken
up by
comorbidity
Attention to
Comorbidity Alone
³1 diagnosis code for:
•
Elevated blood pressure/hypertension
•
Acanthosis, prediabetes, diabetes type 2
•
Lipid disorders
•
Fatty liver disease
•
Vitamin D deficiency
*See table 2 for codes broken up by comorbidity

Labs to screen for diabetes, lipid
disorders, fatty liver disease, or
vitamin D deficiency (2 lab orders
required, because increased
likelihood lab ordered to screen
for comorbidities related
to overweight/obesity)
• Medicines to treat hypertension,
diabetes, lipid disorders, or
vitamin D deficiency

*See table 2 for laboratory studies
broken up by comorbidity
Referral to
tertiary weight
management
clinic, other
medical
specialist
(endocrinologist,
GI, Cardiologist,
etc)

*See table 2 for
referrals broken
up by
comorbidity

No attention

None of the above
None of the above
None of the
above

14

Figure 1: Overview of how attention types are assigned based on EHR collected variables. Please see Table 1 for
specific diagnostic codes, laboratory orders, medication prescriptions and referrals for each attention type.

Comorbidity
Attention Type
Diagnosis Codes
Referrals
Laboratory Studies
Medicines

Attention to Hypertension
R03.0: Elevated BP without
diagnosis of hypertension/single
episode of elevated blood
pressure reading/ borderline
hypertension

Pediatric
Nephrology
*No lab orders were used in
original study for this
comorbidity
acetazolamide
amlodipine
atenolol
hydralazine
propranolol

Attention to Diabetes
E11.65: Uncontrolled type 2
diabetes mellitus without
complication, without long-term
current use of insulin
L83: Acanthosis Nigricans
R73.03: Attention to Diabetes
R73.09: Elevated hemoglobin
A1c
Z13.1: Diabetes mellitus
screening

Endocrinology,
Diabetes &
Metabolism
Glucose
Glucose, fasting
Glucose, gray top
HEMOGLOBINA1C
Insulin, total (BH GH LMW Q
YH)
Insulin, total (BH GH Q YH)
insulin aspar prot-
insulin aspart
insulin aspart
insulin degludec
insulin detemir
insulin glargine
insulin lispro
insulin lispro
protamine-lispro
metformin
metformin ER

Attention to
Hyperlipidemia
Z13.220: Screening for
cholesterol level

Pediatric
cardiology

Cardiovascular
Disease

Cholesterol, Total
HDL cholesterol
LDL CHOLESTEROL,
DIRECT
LIPID PANEL
LIPID PANEL WITH
LDL/HDL RATIO (L)
atorvastatin

15
Lipid profile w/ non-HDL
cholesterol (L)

Attention to Fatty Liver
Disease
R74.0: Elevated ALT
measurement
R74.8: Elevated liver enzymes
R94.5: Elevated LFTs

Pediatric
Hepatology
ALT
ALT+AST
AST
Hepatic function panel
Hepatic function panel (LFT)
LIVER FUNCTION TESTS
(YH)
Comprehensive metabolic panel
Comprehensive metabolic panel
without glucose

Attention to Vitamin D
Deficiency
E55.9: Vitamin D deficiency

QuestAssureD 25-OH vitamin
D, (D2,D3), LC/MS/MS (LMW
Q)
QuestAssureD 25-OH vitamin
D, (D2,D3), LC/MS/MS (BH
LMW Q)
VITAMIN D, 25-HYDROXY,
TOTAL (GH L)
Vitamin D 25 hydroxy (BH
LMW)
Vitamin D, 25-hydroxy,
LC/MS/MS (Q YH)
cholecalciferol
(vitamin D3)
ergocalciferol

Table 2: Values used to qualify for attention to each comorbidity subtype based on data from our Yale associated
practices.

Reference Standard:

To evaluate the performance of the electronic phenotype, an independent chart
review process was done to validate the algorithm’s ability to detect “No Attention”. The
dual purpose of the review was to manually examine the values used by the algorithm
(diagnostic codes, labs, etc.) and to inquiry the visit encounter for other evidence of
attention for which the algorithm was not designed to detect such as written text, or
media entries. This process was done with substantial input from Dr. Christy Turer and
attempts were made to maintain fidelity between the chart review she conducted and
ours.

We began by examining the chart review/abstraction guide and questionnaire used
in the Turer et al 2018 study to review 300 charts. We first converted her questionnaire
from a paper copy to an online version in Qualtrics™ survey software (Qualtrics, Provo,
UT). Several questions in the original questionnaire were intended to collect data for

16
other projects and were thus eliminated from our survey. Given that there are institutional
differences in EPIC layout and note templates, our chart review guide had to be adapted
to give clear directions for how to locate the desired information. In conjunction with Dr.
Turer and in effort to replicate her team’s process as much as possible, we also modified
and added questions to the chart abstraction questionnaire to improve clarity and
completeness. Examples of this include looking for laboratory orders in addition to
laboratory results and adding a question to manually look at visit and problem list
diagnostic codes associated with an encounter. Chart reviews included reviewing the
growth chart, problem list, medication list, laboratory studies, family history, externally
uploaded media, and visit notes associated with the visit date for each patient. Each chart
was reviewed systematically using the Qualtrics survey. We used two separate reviewers
(AG, AF) to examine 30 charts (10% of total) and compared responses to each Qualtrics
survey question. Discrepancies in the responses were resolved with either a third-party
reviewer (MS) or direct discussion between the reviewers. Interrater reliability was
measured using the kappa statistic and interpreted using the guidelines outlined by Koch
and Landis[39].

A difference in our chart review process in comparison to the original project was
the selection of charts. Dr. Turer’s team randomly selected 100 charts from each attention
type (No Attention, Attention to BMI, Attention to Comorbidity). To control for bias, we
chose to blindly review 300 charts from the cohort of 6-12 year olds and compare
assigned attention types after completion of the review.
Statistical Analysis:
Our primary outcome was the algorithm’s sensitivity and specificity of predicting
no attention versus any attention compared to the reference standard.

17
The secondary analysis looked at demographic differences between attention
types (no attention, attention to BMI alone, attention to Comorbidities alone, and
attention to BMI and Comorbidities), for both chart review and algorithm, using Chi-
squared tests of association.
Kappa statistics were computed by hand for interrater reliability (n = 2) of the
chart review. All other analyses was completed using SAS version 9.4 (SAS Institute,
Cary, NC).
Responsibilities:
The thesis primary author (AGB) was responsible for IRB writing and approval
(with oversight from Dr. Sharifi), data randomization, creation of the chart review tool
and review of clinic encounters. Kaitlin Maciejewski, MS (biostatistician) developed the
SAS code to implement the algorithm and to conduct the statistical analysis. Additional
support clarifying which variables were included in the original algorithm in Texas and
general guidance was provided by Christy Boling Turer MD, MHS. Ada Fenick, MD
assisted with the duplicate review of 10% of clinical encounters and helped refine the
chart review tool together with Drs. Turer and Sharifi.

18
RESULTS
Demographics:
We reviewed 329 charts to identify 300 encounters that met our inclusion criteria
and excluded 29 charts due to BMI measurements not meeting inclusion criteria. The
mean±SD age of the sample was 10±1.87 years and 58.3% of children were male. Table
3 displays the demographics and encounter characteristics of the sample. In terms of
weight categorization, 15.3% met criteria for severe obesity defined as ≥120% of 95th
percentile, 41.3% met criteria for obesity, and 43.3% met criteria for overweight. The
most prevalent race/ethnicity was Hispanic/Latino, compromising 42.7% of the cohort,
Non-Hispanic Black was the next most prevalent 31% followed by Non-Hispanic White
(15%) and Asian/Other (11.3%). Of the clinics included in our cohort, the majority (63.7
%) of encounters were conducted at the PCC, 24.7% at YHC, and 11.7% at SRC. The
majority of patients in our sample had a public insurance payor type (64.7%) and the
remainder had a private payor (14.3%) such as a managed healthcare or “Blue-cross
Blue-shield,” or other means (20.7%). Of the visit encounters examined, 49% were
conducted by resident physicians with attending supervision, 25% by nurse Practitioners,
17.3% by attendings and 8.3% by physician assistants.
Table 5 displays the prevalence of attention to BMI and/or obesity-related
comorbidities stratified by demographic and encounter characteristics. We observed
statistically significant differences in the likelihood of classification as “no attention” by
BMI category (p<0.001 for chart review and p<0.001 for algorithm) and by clinician type (p<0.001 for chart review and p<0.001 for algorithm). We did not observe statistically significant differences by race/ethnicity, season of encounter or insurance type. Validation of the Algorithm: 19 Of the 30 charts comprehensively reviewed by two reviewers (AG and AF), kappa scores suggested substantial inter-observer agreement: 0.697 (95% confidence interval 0.482-0.912). Of the 300 charts reviewed in total, 50 were assigned no attention by chart review compared to 99 assigned no attention by the algorithm. Chart review identified 102 charts as attention to BMI, 4 as attention to comorbidity alone, and 144 as attention to BMI and Comorbidity. The algorithm correctly identified 66 and 82 as Attention to BMI and Attention to BMI and Comorbidity, respectively. This yielded a sensitivity of 94.0%, specificity of 79.2%, positive predictive value 47.5%, and negative predictive value 98.5% (Table 4). A review of the charts for which the algorithm incorrectly labeled the encounter as “no attention,” the discordance between the algorithm and chart review was due to evidence of attention in the form of free text within the progress note. Of the 52 encounters incorrectly labeled as no attention” by the algorithm, 83% had evidence of documentation in the assessment and plan and 49% had evidence of attention in the subjective sections of the progress note. A few encounters were incorrectly labeled as attention by the algorithm due to the use of qualifying laboratory studies, diagnosis codes, or referrals but not in the context of weight management. For example, in one encounter the provider ordered screening lipids as part of routine care during a 10 year- old’s well child visit in addition to vitamin D screening as part of deficiency screening in refugee clinic; these two lab orders satisfy the algorithm’s categorization of “attention to comorbidity” but were appropriately not identified as attention during chart review. These findings are summarized in Figure 2.