Scholars’ Mine
Scholars’ Mine
Masters Theses
Student Theses and Dissertations
Fall 2016
Using eye-tracking to understand user behavior in deception
Using eye-tracking to understand user behavior in deception
detection system interaction
detection system interaction
Prashanth Kumar Lakkapragada
Follow this and additional works at: https://scholarsmine.mst.edu/masters_theses
Part of the Technology and Innovation Commons
Department:
Department:
Recommended Citation
Recommended Citation
Lakkapragada, Prashanth Kumar, “Using eye-tracking to understand user behavior in deception detection
system interaction” (2016). Masters Theses. 7605.
https://scholarsmine.mst.edu/masters_theses/7605
This thesis is brought to you by Scholars’ Mine, a service of the Missouri S&T Library and Learning Resources. This
work is protected by U. S. Copyright Law. Unauthorized use including reproduction for redistribution requires the
permission of the copyright holder. For more information, please contact scholarsmine@mst.edu.
USING EYE-TRACKING TO UNDERSTAND USER BEHAVIOR IN
DECEPTION DETECTION SYSTEM INTERACTION
by
PRASHANTH KUMAR LAKKAPRAGADA
A THESIS
Presented to the Faculty of the Graduate School of the
MISSOURI UNIVERSITY OF SCIENCE AND TECHNOLOGY
In Partial Fulfillment of the Requirements for the Degree
MASTER OF SCIENCE IN INFORMATION SCIENCE & TECHNOLOGY
2016
Approved by
Dr. Fiona Fui-Hoon Nah
Dr. Keng Siau
Dr. Nathan Twyman
Dr. Michael Hilgers
ii
2016
Prashanth Kumar Lakkapragada
All Rights Reserved
iii
ABSTRACT
This research presents the analysis of data collected using eye-tracking devices on
user interaction with a deception detection system. The differences between two groups
of subjects, namely Innocent and Guilty, were compared, where Innocent subjects did not
carry any explosive and hence, had nothing to hide in declaring objects that they were
carrying whereas Guilty subjects had to lie to deceive the system. The results indicate
that there is no significant difference in pupil dilation between the Innocent and Guilty
subjects. However, the amount of fixations on the empty spaces of slides containing an
explosive image can be used to identify Innocent versus Guilty subjects where subjects in
the Guilty condition were more likely than subjects in the Innocent condition to focus on
the empty spaces between the images of objects on those slides.
Keywords: Eye tracking, cognition, deception detection, visual behavior, data
mining, iMotion attention tool.
iv
ACKNOWLEDGMENTS
I would like to thank my advisor, Dr. Fiona Fui-Hoon Nah, for the continuous
support, enormous knowledge, and motivation. Her support has been outstanding right
from the beginning and gave me knowledge on how to write a research oriented paper
and also taught me the IBM SPSS tool for data analysis.
Besides my advisor, I would like to thank Dr. Nathan Twyman for permitting me
to use his research data for my research. I am grateful to him for providing me all the
resources needed for my thesis. His guidance in analyzing the data and writing the paper
helped me to complete my research successfully.
I would like to thank the rest of my thesis committee members, Dr. Keng Siau and
Dr. Michael Hilgers, for their insightful encouragement and comments on my initial
research proposal which helped me to steer in the right direction.
Finally, I would like to thank my parents and all my friends for supporting me and
encouraging me with their blessings throughout my master’s degree program.
v
TABLE OF CONTENTS
Page
ABSTRACT
……………………………………………………………………………………………………….. iii
ACKNOWLEDGMENTS ……………………………………………………………………………………. iv
LIST OF FIGURES …………………………………………………………………………………………….. vi
LIST OF TABLES
……………………………………………………………………………………………… vii
SECTION
1. INTRODUCTION
………………………………………………………………………………………. 1
2. LITERATURE REVIEW
…………………………………………………………………………….. 3
3. RESEARCH METHODOLOGY
………………………………………………………………….. 8
3.1. EXPERIMENTAL DESIGN …………………………………………………………….. 8
3.2. EXPERIMENT PROCEDURES ……………………………………………………….. 8
4. ANALYSIS METHOD ……………………………………………………………………………… 12
5. ANALYSIS AND RESULTS …………………………………………………………………….. 14
6. CONCLUSION AND LIMITATIONS
………………………………………………………… 25
APPENDIX
………………………………………………………………………………………………. .…… 27
BIBLIOGRAPHY…………………………………………………………………………..35
VITA………………………………………………………………………………………………………………….38
vi
LIST OF FIGURES
Page
Figure 3.1. Screening room layout
………………………………………………………………………….. 9
Figure 3.2. Sample screen
……………………………………………………………………………………. 11
Figure 5.1. Fixation plot for critical slide in the first sequence (G vs I)
……………………… 15
Figure 5.2. Fixation plot for critical slide in the second sequence (G vs I) …………………. 16
Figure 5.3. Fixation plot for critical slide in the third sequence (G vs I)…………………….. 17
Figure 5.4. Fixation plot for critical slide in the fourth sequence (G vs I) ………………….. 17
Figure 5.5. Slide showing the center of the screen ………………………………………………….. 18
Figure 5.6. Heat maps of critical slides for Guilty participants …………………………………. 19
Figure 5.7. Heat maps of critical slides for Innocent participants ……………………………… 19
Figure 5.8. Slide showing the empty space between the images on the screen
……………. 22
vii
LIST OF TABLES
Page
Table 2.1. Summary of techniques used for deception detection ………………………………… 3
Table 2.2. Drawbacks of techniques used for deception detection
………………………………. 4
Table 5.1. Number of participants who focused at the center vs explosive
…………………. 20
Table 5.2. Summary of participant attention (center region)
…………………………………….. 21
Table 5.3. Number of fixations at the center of the screen ……………………………………….. 21
Table 5.4. Number of fixations in the empty space of the screen
………………………………. 22
Table 5.5. Fixation percentage on the explosive (G vs I) …………………………………………. 24
1. INTRODUCTION
Making accurate judgements is an important aspect of investigative interviewing
(Raskin, Honts, & Kircher, 2013). Detection of deception is an important aspect for
national and personal security (Deokar & Madhusudan, 2005). The recent Paris attack
and shooting attack at a nightclub in Orlando, Florida show the importance of national
and personal security. Are there ways to stop these attacks?
As Benjamin Franklin said, “An ounce of prevention is worth a pound of cure.” It
is important to identify the threats in advance rather than waiting for attacks to happen.
Hidden information by individuals is the most important cue and also the most difficult
information to retrieve or detect as individuals may try to hide information intentionally
(Twyman, Lowry, Burgoon, & Nunamaker Jr, 2014b). The lack of skill and control on
procedures being followed as well as human errors are potential causes that make the
retrieval of such information complex (Twyman, Elkins, Burgoon, & Nunamaker,
2014a).
Facial analysis, eye tracking, and concealed information online tests are a few of
the technologies which can be used to detect deception (Twyman et al., 2014a). Eye gaze
movements can be used to analyze user behavior in online environments (Klami, 2010).
Visual attention depends on the task being performed by an individual (Gidlöf, Wallin,
Dewhurst, & Holmqvist, 2013). The data collected by eye tracking devices can be used to
analyze the visual behavior or characteristics of individuals in different conditions.
The objective of this research is to analyze the data collected by eye tracking to
identify potential threats. In this research, the eye tracking data for a deception detection
system collected by Twyman et al. (2014b) is used to analyze the visual behavior of
2
individuals in different conditions (i.e., Innocent and Guilty). This exploratory research
summarizes the analysis performed on the data.
This paper is organized in the following manner. A literature review is presented
on research in eye tracking and the psychology of eye gaze. Different types of analysis
were carried out on the data and the results are reported. The theoretical explanations
underlying the results of the analysis are also provided. The thesis concludes with
limitations and future scope for research.
3
2. LITERATURE REVIEW
Safeguarding of national security and personal tasks is a highly challenging task
(Deokar & Madhusudan, 2005). Recent attacks in various countries, including the United
States, show that there is not enough security in place and they warrant more research in
the security field. Most commonly used techniques are behavioral analysis interviews
(BAI), comparison question tests (CQT), and concealed information tests (CIT) (Vrij,
2008). Changes in the electric waves on the skin are used as measurement for CIT
(Ambach, Bursch, Stark, & Vaitl, 2010). New tools to assist humans are developed
continuously based on the research in this field (Vrij, 2008).
Various techniques are summarized in Table 2.1 along with their descriptions
(Ambach et al., 2010; Masip, Herrero, Garrido, & Barba, 2011; Twyman et al., 2014b).
Table 2.1. Summary of techniques used for deception detection
Technique
Measurement
Comparison
question test (CQT)
Changes in electric signals are measured to detect deception
Behavioral analysis
interview (BAI)
Using behavior provoking questions to observe differences in
verbal and non-verbal responses
Concealed
information
test(CIT)
Comparisons of response for relevant and irrelevant items
4
Although the above mentioned techniques are widely used, they are believed to
lack accuracy because all the above techniques need human interventions (Masip et al.,
2011). Some of the drawbacks are listed in the Table 2.2.
Table 2.2. Drawbacks of techniques used for deception detection
Technique
Drawbacks
Comparison question test (CQT)
Time consuming and low validity
Behavioral analysis interview (BAI)
Time consuming
Concealed information test (CIT)
Impact of counter-measures
The CQT theory states that guilty persons tend to react more to relevant questions
whereas innocent persons tend to react to comparison questions (MacNeill, Bradley,
Cullen, & Arsenault, 2014). A research study has shown that CQT is 90% accurate in
identifying guilty and innocent persons but it is very time consuming to interview every
person (Offe & Offe, 2007). However, the main criticism faced by CQT is the absence of
relevant theory on individuals’ behaviors (Ben-Shakhar, Gamer, Iacono, Meijer, &
Verschuere, 2015).
The behavior of individuals based on their intention will change and in most
of the cases, the guilty person or person in guilt tends to manipulate his or her behavior
(Masip & Herrero, 2013). This can be identified through the use of behavioral analysis
5
interviews (BAI). However, research shows that BAI may not be accurate (Vrij, Mann, &
Fisher, 2006). The study by Vrij et al. (2006) shows us that guilty persons are more
helpful than innocent persons which is opposite or contradictory to the BAI theory. In the
study conducted by Masip et al. (2013), both Guilty and Innocent groups would tend to
look innocent and the Guilty group even used countermeasures and were successful in
convincing the interviewer that they were innocent. Much research is needed in this area
to validate the BAI theory (Horvath, Blair, & Buckley, 2008) and better methods are
needed for identification and evaluation.
Both CQT and BAI techniques depend heavily on the capability of the
interviewer in identifying the culprit (Twyman et al., 2014b). There is a need for a
technique which is less dependent on the interviewer and the concealed information test
(CIT) is a possible solution for it (Twyman et al., 2014b). The CIT technique is
considered a more valid approach when compared to CQT and BAI (Ben-Shakhar &
Elaad, 2003; Iacono & Lykken, 1997). Japan uses the CIT approach widely in crime
investigations (Ogawa, Matsuda, & Tsuneoka, 2015). This technique takes minimal time
to complete the process and can be effective when used with invasive sensors (Twyman
et al., 2014b). However, research on non-invasive sensors is also warranted. Twyman et
al. (2014a, 2014b) conducted experiments combining CIT, eye tracking and facial
analysis to analyze the behavior of guilty and innocent participants in a mock crime
scenario.
Research shows that taking cognition into account will improve the accuracy of
lie and truth detection (Granhag, Vrij, & Verschuere, 2015). One of the recent trending
non-invasive technology is eye tracking. Eye tracking is one of numerous
6
psychophysiological techniques (Pak & Zhou, 2013). The movement of an eye can be
used to understand the cognitive process of an individual (Just & Carpenter, 1976).
Visual attention depends on the task being performed by an individual (Gidlöf et al.,
2013). Researchers advocate that there is a relationship between the cognitive process of
what we see and our eye gaze movements (Fleisher & Gordon, 2010; Zulawski,
Wicklander, Sturman, & Hoover, 2001). Measurements like pupil dilation, revisit time,
response time etc. can be used in understanding the cognitive response of an individual.
Using infra-red camera on the eye tracking devices, the pupil dilation and gaze
movements can be tracked (Bhuvaneswari & Satheesh Kumar, 2015). Stimuli is first
processed by the peripheral attention (Twyman et al., 2014b). Eyes tend to move towards
the stimuli if it is significant to an individual (Twyman et al., 2014b). Lying increases the
cognitive load since it involves making up a story and remembering it through the test
(Granhag et al., 2015). Innocents do not have to hide their inner feeling whereas guilty
suspects have to hide their inner feelings (Granhag et al., 2015). According to the
defensive responsive theory, guilty behavior tends to escape or avoid the situation (Gray,
1987).
Analysis of eye movements can uncover cognition in humans while performing
any task (Merkley & Ansari, 2010). Study conducted by Twyman et al. (2014b) used CIT
with eye tracking to assess the individuals’ conditions. The results show that individuals,
in the Guilty condition, tend to focus on a safety point when the stimuli contained
relevant objects (Twyman et al., 2014b). The study also reported that defensive behavior
is not affected by time (Twyman et al., 2014b). However, this could change with constant
exposure to the stimuli. Lying takes a little more time when compared to truth telling
7
(Walczyk, Roper, Seemann, & Humphrey, 2003). The results of a study show that the
Guilty participants responded quickly to the statements about a theft in which they were
involved than to the neutral statements (Raskin, Honts, Kircher, & ebrary, 2014).
However, the Innocent participants responded more quickly to all the statements when
compared to the Guilty participants (Raskin et al., 2014).
8
3. RESEARCH METHODOLOGY
The experiment was conducted by Twyman et al. (2014b) and is described as
follows.
3.1. EXPERIMENTAL DESIGN
The deception detection system was evaluated using a lab experiment. A building
security screening context was chosen to simulate the experimental conditions. Students
were recruited as participants for this study as finding individuals who have the intention
to commit a crime was challenging.
3.2. EXPERIMENT PROCEDURES
The participants were divided into two conditions, namely Guilty and Innocent
(I), with greater number of participants in the Guilty condition. The intention of having a
greater number of participants in the Guilty condition is to further classify them into four
groups with equal participants in each sub-group. Also, the number of participants in
each sub-group is equal to the number of participants in the Innocent condition. The four
sub-groups under the Guilty condition are: 1. Guilty with no training on countermeasure
(G), 2. Guilty with training on mental countermeasure (CM), 3. Guilty with training on
physical countermeasure (CP), and 4. Guilty with training on all, i.e., both mental and
physical, countermeasures (CA). The participants were instructed to pack a bag with
standard items such as books, clothes etc. and deliver it to another person, who is inside
the building. The building is secured and the participant must pass through the security
checkpoint to deliver it, since the contact is located in the safe part of the building. The
9
participants in the Guilty condition constructed and packed a mock improvised explosive
device (IED) in the bag along with the standard items which were common to all the
participants. A room, as shown in the Figure 3.1, was chosen only for screening the
participants and arrangements were made in such a way that there is no distraction for the
participant.
Figure 3.1. Screening room layout
10
To enter the safe part of the building, the participants must answer the questions
verbally, with Yes or No, asked by the second-generation Automated Screening Kiosk
(ASK). A brief introduction about the ASK is given to the participant during sign-up. The
participants were also informed that if they pass through the screening room successfully
without being caught, an amount of $20 is awarded. After the participants complete the
screening process, they were instructed to answer the post-experiment survey. They were
later explained that the whole task is just an experiment and they did not do any illegal
activity during the experiment.
At the ASK, the interview proceeded as mentioned: When the participant stepped
near the ASK system, voice instructions were given by the system and the individual was
guided to undergo a 15-second process of calibration. Following this, an avatar would
appear on the screen and asked “Are you carrying any of these items?” Immediately after
the question, four banned items were displayed on the screen for a duration of five
seconds. An example of the items displayed on the screen can be seen in Figure 3.2.
The questioning process was repeated five times and an IED was displayed only
once on the screen. This sequence was repeated three more times with changes in the
position of the images between different quadrants. On the whole, the question was asked
20 times and it took 4 minutes to complete the process. At the end, the participant was
instructed to fill out the post experiment questionnaire and was told that this was just an
experiment.
11
Figure 3.2. Sample screen
12
4. ANALYSIS METHOD
This research aims to analyze the data collected using eye tracking tools and
various methods such as gaze patterns and heat maps. The study is limited to the analysis
of the participants under the Guilty (G) and Innocent (I) conditions. Different measures
based on the heat maps were used to analyze attention, respondent count, time spent on
the area of interest, and the fixation sequence.
The data was saved in four data frames which were eye tracking data, participant
mapping information, slide mapping information, and image mapping information. Eye
tracking data contains the X and Y coordinates of each participant’s gaze for each slide
along with pupil diameters of the left and right eyes. The participant data frame stores
information about the participant’s condition (i.e., G or I), and the date of participation
along with the rotation of slide sequence for each participant. The participant data frame
also has information on whether the participant was disqualified and the reason for
disqualification if so. Participant and eye tracking data can be mapped using the
participant ID. Information about slide sequence for each rotation and critical slide (i.e.,
the slide that includes an explosive) is stored in the slide mapping data frame. The slide
mapping data frame also stores information about the quadrant in which the image of an
explosive is displayed in the critical slide. Slide number column maps the eye tracking
data with slide mapping data frame. Image mapping data frame stores the information
about the objects displayed on each slide. It also stores the quadrant in which the object
was displayed in each slide. SlideID is used to map the image mapping data frame with
the slide mapping data frame.
13
R was used to read the data and generate suitable subsets after the data was
cleansed. The subsets were analyzed using the iMotions attention tool, IBM SPSS
statistics, and tableau. The data was fed into the iMotion attention tool, a tool to analyze
eye tracking data, and different measures like fixation points, area of interest, heat maps,
respondent count, and time spent were compared and assessed. The raw data was taken
and formatted into meaningful subsets using R programming. The subset data was then
loaded into iMotions to generate metrics like fixation points, area of interest, heat maps,
respondent count, and time spent.
14
5. ANALYSIS AND RESULTS
The raw data was loaded into the R Studio and suitable subsets were made. The
outlying data points, which were lying outside the screen were filtered out. The next step
was to remove the participants who were disqualified in the main study. 6 out of 71
participants in Guilty and Innocent conditions were disqualified. Some of the reasons for
disqualifying participants include failure of eye tracking calibration, problems in
following the experimental procedures, answering Yes when the bomb was displayed.
The final data set contains 32 participants in the Guilty condition and 33 in the Innocent
condition. The data comprising the disqualified participants was filtered out of the data
set. In each rotation, the images displayed were the same but were placed in different
quadrants. The data points were rotated in such a way that the placement of the images is
the same in all rotations. The next step involves sub-setting the data based on the
condition, rotation and sequence. Scatter plots were plotted using the X and Y
coordinates of the eye gaze for each critical slide in each sequence. For each condition,
four graphs are plotted as the critical slide was displayed four times. The images shown
in this study were generated with the help of the slide and image mapping information
from the secondary data. Figure 5.1 shows the scatter plots comparing data points on
critical slide 1 for Innocent and Guilty conditions.
The scatter plots show that the number of data points between the images is
higher for participants in the Guilty condition than for participants in the Innocent
condition for all four critical slides, which refer to slides that display the image of an
explosive. According to the spotlight theory of attention, objects on the screen can be
recognized using the peripheral or covert attention i.e., through the corner of the eyes.
15
Previous research shows that an object or changes in an object can be detected with the
help of our peripheral vision (Schall & Bergstrom, 2014; Vater, Kredel, & Hossner,
2016). Emotional information can be recognized with peripheral vision (Calvo, Avero, &
Nummenmaa, 2011). The guilty participants identified the explosive displayed on the
kiosk screen with their peripheral vision and hence, there are more data points near the
images of the objects on the screen. This observation can be observed in all scatter plots
comparing the Guilty condition and the Innocent condition.
Figure 5.1. Fixation plot for critical slide in the first sequence (G vs I)
16
Another reason for having more data points between the images could be due to
the saccades. With a saccadic movement, a person can make easy recognition of an object
(Dandekar, Ding, Privitera, Carney, & Klein, 2012). Higher number of data points
between the images for participants in the Guilty condition could be due to saccadic
movement of eyes. Research shows that the initial saccadic movement was not affected
by the condition of the participant (Twyman et al., 2014b). The study also shows that
after detecting the critical item in the foil, participants avoided looking at the object
(Twyman et al., 2014b). However, scatter plots show that saccadic movements were
made by the participants not only near the object of detection but also on the entire
screen. Figures 5.2, 5.3 and 5.4 shows the scatter plots for critical slide 2, critical slide 3
and critical slide 4 respectively.
Figure 5.2. Fixation plot for critical slide in the second sequence (G vs I)
17
Figure 5.3. Fixation plot for critical slide in the third sequence (G vs I)
Figure 5.4. Fixation plot for critical slide in the fourth sequence (G vs I)
