Scholars’ Mine
Scholars’ Mine
Masters Theses
Student Theses and Dissertations
Summer 2016
Effect of timing and source of online product recommendations:
Effect of timing and source of online product recommendations:
An eye-tracking study
An eye-tracking study
Qing Zeng
Follow this and additional works at: https://scholarsmine.mst.edu/masters_theses
Part of the Technology and Innovation Commons
Department:
Department:
Recommended Citation
Recommended Citation
Zeng, Qing, “Effect of timing and source of online product recommendations: An eye-tracking study”
(2016). Masters Theses. 7576.
https://scholarsmine.mst.edu/masters_theses/7576
This thesis is brought to you by Scholars’ Mine, a service of the Missouri S&T Library and Learning Resources. This
work is protected by U. S. Copyright Law. Unauthorized use including reproduction for redistribution requires the
permission of the copyright holder. For more information, please contact scholarsmine@mst.edu.
EFFECT OF TIMING AND SOURCE OF ONLINE PRODUCT
RECOMMENDATIONS: AN EYE-TRACKING STUDY
by
Qing Zeng
A THESIS
Presented to the Faculty of the Graduate School of the
MISSOURI UNIVERSITY OF SCIENCE AND TECHNOLOGY
In Partial Fulfillment of the Requirements for the Degree
MASTER OF SCIENCE IN INFORMATION SCIENCE AND TECHNOLOGY
2016
Approved by
Dr. Fiona Nah, Advisor
Dr. Keng Siau
Dr. Richard Hall
2016
Qing Zeng
All Rights Reserved
iii
ABSTRACT
Online retail business has become an emerging market for almost all business
owners. Online recommender systems provide better services to the consumers as well as
assist consumers with their decision making processes. In this study, a controlled lab
experiment was conducted to assess the effect of recommendation timing (early, mid, and
late) and recommendation source (expert reviews vs. consumer reviews) on e-commerce
users’ interest and attention. Eye-tracking data was extracted from the experiment and
analyzed. The results suggest that users show more interest in recommendation based on
consumer reviews than recommendation based on expert reviews. Earlier
recommendations do not receive greater user attention than later recommendations.
iv
ACKNOWLEDGMENTS
First, I want to thank Dr. Fiona Nah from Missouri University of Science and
Technology for mentoring me on various research projects including this research. I
would also like to thank my committee members, Dr. Keng Siau and Dr. Richard Hall,
for your advice and suggestions on this thesis.
Next, I want to thank Dr. Amy Shi, Dr. Chuan-Hoo Tan, and Dr. Choon Lin Sia
from City University of Hong Kong for giving me the opportunity to collaborate with
them on this experimental research study. I would also like to thank Samuel Smith for
helping to proofread the thesis.
Last but not least, I want to thank all members from the Laboratory for
Information Technology Evaluation for helping to coordinate and conduct these
experiments.
v
TABLE OF CONTENTS
Page
ABSTRACT
……………………………………………………………………………………………………….. iii
ACKNOWLEDGMENTS ……………………………………………………………………………………. iv
LIST OF FIGURES …………………………………………………………………………………………….. vi
LIST OF TABLES
……………………………………………………………………………………………… vii
SECTION
1.
INTRODUCTION
…………………………………………………………………………………. 1
2.
LITERATURE REVIEW
……………………………………………………………………….. 2
2.1. ONLINE PRODUCT RECOMMENDER SYSTEMS ……………………………….. 2
2.2. EYE-TRACKING …………………………………………………………………………………. 5
3.
THEORETICAL BACKGROUND AND HYPOTHESES ……………………….. 11
4.
METHODOLOGY
………………………………………………………………………………. 14
5.
DATA ANALYSIS AND RESULTS …………………………………………………….. 16
5.1. DATA ANALYSIS ON PUPIL DILATION …………………………………………… 17
5.2. DATA ANALYSIS ON FIXATION DURATION PER SECOND ……………. 18
6.
DISCUSSION AND CONCLUSION …………………………………………………….. 22
7.
CONTRIBUTIONS AND IMPLICATIONS
…………………………………………… 23
8.
LIMITATIONS AND FUTURE RESEARCH ………………………………………… 24
REFERENCES ………………………………………………………………………………………………….. 25
VITA …………………………………………………………………………………….29
vi
LIST OF FIGURES
Page
Figure 5.1: Interaction effect of recommendation timing and product type
………………….20
vii
LIST OF TABLES
Page
Table 2.1: Summary of literature review on online recommender systems ……………………3
Table 2.2: Summary of literature review on eye-tracking research
……………………………….7
Table 5.1: Descriptive statistics for pupil dilation
…………………………………………………….17
Table 5.2: Descriptive statistics for FDPS
……………………………………………………………….19
Table 5.3: Mean values of FDPS for recommendation timing and product type …………..20
1. INTRODUCTION
Based on data from the U.S. Census Bureau, U.S. retail e-commerce sales for the
first quarter of 2016 has reached $92.8 billion, which accounts for 7.8 percent of total
retail sales (DeNale & Weidenhamer, 2016). Over the past decade, sales of retail e-
commerce have a yearly growth of more than 15%. In order to boost sales, more and
more retailers are implementing online recommender systems, or, recommendation
agents (RAs) which can provide better services and help customers with the decision
making process. The algorithms underlying online recommender systems (Hostler et al.,
2012) as well as the effects of online recommender systems (Adomavicius et al., 2013)
have been studied in the past decade but there is little research to assess its efficacy and
user interest.
Although online product recommender systems have been influential in boosting
sales as well as user satisfaction, there are still some recommender systems that are
poorly designed or ineffectively implemented. The goal of this research is to study some
of the key characteristics of online product recommender systems and their effects on
users.
In this research, the researcher is interested to examine the effects of an online
product recommender system on users’ attention and interest in terms of the display
timing (i.e., early, mid, and late) of the recommendation and the sources of
recommendation content (i.e., expert vs. consumer).
We expect the outcome of this research to be helpful to online retailers in
improving their online recommender systems.
2
2. LITERATURE REVIEW
2.1. ONLINE PRODUCT RECOMMENDER SYSTEMS
Online product recommender systems are widely used to provide consumers with
alternatives that they might be interested in. Current product recommender systems are
using various filtering systems including content-based filtering, collaborative filtering,
and hybrid methods to provide consumers with the right products (Aciar et al., 2007).
Online retailers rely on recommender systems as a decision aid to the customers in order
to provide better service and to boost sales. According to research conducted by Forrester
Research, product recommender systems accounted for 10 to 30 percent of total sales by
a retailer (Schonfeld, 2007).
Prior studies on product recommender systems are mainly focused on the
optimization of algorithms to provide more accurate predictions and suggestions to the
customers (Hostler et al., 2012). According to Adomavicius et al. (2013), most
recommender systems take into account consumers’ ratings of the products experienced
and used them to calculate ratings for the products and to predict customer preferences.
One type of recommender systems that is widely used is called the collaborative
recommendation system. Such type of systems does not recommend items based on
similarities with the users’ past preferences, but on what similar users like. Another
popular type of recommender systems is called content-based recommendation system. It
provides recommendations by comparing products to users’ profiles. Based on the match
of product features and user preferences, the item with the highest rating will be
recommended to the user. Some recommender systems implement a hybrid approach to
combine both content-based and collaborative systems to avoid the weaknesses of either
systems (Balabanovic, 1997). Although most recommender systems have limitations such
as the requirement to have a large amount of prior customer data (Ansari, Essegaier, &
Kohli, 2000), the impact of recommendation systems on consumers’ decision making
process has been effective. Lu et al. (2015) evaluated recommender systems in different
business settings to provide suggestions on building an effective recommender system.
In general, online user reviews can influence consumers through awareness
effects or persuasive effects (Duan et al., 2008). Awareness effects can create exposure of
3
the product to the customer so that he or she is more likely to include the recommended
product in his or her choice set. Persuasive effects can improve customers’ attitudes
during the evaluation process, which in turn affects their decision making. Online reviews
can also affect customers’ decision making through word of mouth.
Table 2.1 provides a summary of research studies on online recommender
systems.
Table 2.1: Summary of literature review on online recommender systems
Author(s)
Focus
Key findings
Adomavicius et al.,
2013
Influence
People’s preference ratings can be significantly
influenced by recommendations. Recommender
systems have an anchoring effect that is continuous
and linear.
Ansari et al., 2000
Algorithm
Simple and flexible models are described to
incorporate revealed preferences on the basis of
explicit and implicit data. Procedure for ratings data
were also developed.
Balabanović &
Shoham, 1997
Algorithm
A hybrid online recommender system called Fab was
tested against other sources which resulted in higher
performance.
Cosley et al., 2003
Influence
In movie ratings, a recommender’s prediction can
influence users’ opinion.
Hostler et al., 2012
Influence
Recommendation systems can increase consumers’
perceived attractiveness of products
Guan et al., 2014
Algorithm
A recommender system was proposed that takes into
account the item quality and user rating preferences
that decreases the computing complexity was
proposed. A significant higher accuracy was
observed compared to three other benchmark
systems.
4
Table 2.1: Summary of literature review on online recommender systems (cont.)
Iacobucci et al.,
2000
Algorithm
Based on an extensive review process and cluster
analysis, simple Jaccard coefficient was found to be a
reliable index of similarity. Content-based similarity
among products and the affirmative customer
network should be taken into account in forming the
recommendations.
Kim et al., 2005
Algorithm
A recommender system using consumer navigational
and behavioral pattern to estimate the consumer
preference levels was examined to outperform
benchmarking systems.
Lee & Kwon, 2008
Algorithm,
Influence
Casual maps based recommendation mechanism was
recommended to enhance consumers’ decision
satisfaction, attitude towards the recommended
products, positive purchase intentions, and actual
purchase.
Senecal & Nantel,
2004
Features,
Influence
Product recommendations result in higher purchase
rate than without recommendations.
Recommendation systems labeled “recommender
system” were more influential than labeled as
“human experts” and “other consumers”.
Shi et al., 2013
Features,
Influence
The timing and basis of the recommendations
resulted in significant differences in consumers’
decision satisfaction and decision difficulty.
Wang & Benbasat,
2007
Algorithm,
Influence
The study found that the use of explanation facilities
enhanced consumers’ initial trusting beliefs.
Furthermore, different explanation types could
influence different trusting beliefs.
5
2.2. EYE-TRACKING
The use of eye-tracking devices on information processing tasks has been around
for more than a century. In order to track the location of eye fixations, invasive methods
were first implemented involving direct mechanical contact with participants’ cornea. As
research on eye-tracking evolves, non-invasive methods such as using light on the cornea
and recording the reflection to obtain fixation data became more common (Jacob & Karn,
2003). Eye-tracking studies became more popular as equipment became more accurate
and psychological theories became more advanced. Researchers were able to use eye-
tracking data to study cognitive processes. However, using eye-tracking to study usability
issues was more scarce due to issues regarding data collection and data analysis methods
(Jacob & Karn, 2003). In the 1980s, as personal computers became more popular,
researchers started using eye-tracking devices to study and solve problems in human-
computer interaction.
Klin et al. (2002) utilized eye-tracking and discovered that reduced eye region
fixation durations can predict autism. Also, increased fixation duration on mouths
indicates improved social adjustment from autism. By studying the gaze shift data from
eye-tracking devices, Mason et al. (2013) found that illustrations with text result in higher
participants’ effort to integrate verbal and pictorial information. Tsai et al. (2012) used
eye-tracking devices to track participants’ eye movement during problem-solving tasks to
study their ability to find relevant information and problem-solving outcomes. They
found that successful problem solvers focused more on relevant factors.
People’s attention will only focus on the things they need and will ignore others
that are presumed to be irrelevant (Triesch et al., 2003). The results from the study by
Orquin & Loose (2013) also indicate that decision makers direct their attention to goal-
related stimuli. According to Rayner (1978), cognitive processing during a fixation
affects the fixation duration. In other words, a longer fixation duration on a certain piece
of information implies a higher intensity of cognitive processing.
Based on a study by Shimojo et al. (2003), longer fixation durations reflect higher
preference upon choices. As shown in their research, fixations increased exposure to the
stimuli which transitioned into preferences. Preferences can reinforce people’s fixations
and enhance their perceptions of attractiveness which in turn influence decision making.
6
The study by Krajbich et al. (2010) indicates that visual fixation process could have a
causal effect on people’s value comparison process. People’s perceived value on choices
influences fixations. Also, fixation durations increase as the difficulty of choices
increases.
Based on the eye-mind assumption highlighted by Just & Carpenter (1980),
people’s eyes will remain fixated as long as information is being processed. Rayner (1998)
states that although people can move their attention without moving their eyes, it is more
efficient to move the eyes than to move attention while fixated on complex stimuli.
During an online shopping process, there are many stimuli on the screen. Eye fixation is a
well-developed predictor of attention.
Human pupils react not only to change of environmental luminance, but also to
change in cognitive processing (Brisson et al., 2013). Pupil dilation was found to be a
consequence of attentional effort (Hoeks & Levelt, 1993). According to Laeng et al.
(2012), pupil diameter, which is also called “pupillometry”, has been used to estimate the
intensity of mental activities, change of emotions, change of mental states, and change of
attention for more than 50 years. Pupil dilation not only reflects emotional stimuli but
also indicates some cognitive mechanisms. In standard light conditions, the diameter of
human pupils averages at about 3 mm while the size can reach an average of 7 mm in dim
light condition (MacLachlan & Howland, 2002). Pupil diameter is very difficult to
control voluntarily. It has been shown that pupil diameter can only be controlled
indirectly by mentally imaging a stimuli (Laeng et al., 2012). This characteristic of pupil
dilation makes it a good objective measure. In the study by Einhäuser et al (2010), pupil
response was defined as the consolidation of mental states related to arousal and mental
activities. Pupil dilation was also found to indicate interest to tasks such that an increase
in pupil diameter reflects increased interest in the stimuli (Hess & Polt, 1960). Positive
pupil dilation indicates an increased task engagement. Negative pupil dilation indicates
disengagement from the task.
Greater pupil dilation indicates higher decision threshold in difficult decision
making (Cavanagh et al., 2014). The dilation of pupils can represent the need to increase
cognitive control when conflicts exist during decision making. Pupil dilation can
7
represents both the mental activity involved as well as the difficulty of the task (Hess &
Polt, 1964).
An aggregated review of select prior studies using the eye-tracking technique is
listed in table 2.2.
Table 2.2: Summary of literature review on eye-tracking research
Author(s)
Metrics
Key Findings
Brisson et al.,
2013
Pupil diameter
Eye-tracking systems have some systematic error
estimating pupil size. When reading language
such as English, subjects were most aroused
when started and became least aroused when
reaching the end.
Cavanagh et al.,
2014
Gaze dwell time,
Pupil dilation
Higher gaze dwell time predicted higher drift
rate toward the fixation option. Higher pupil
dilation predicted higher decision threshold
during difficult decisions.
Einhäuser et al.,
2010
Pupil dilation
Pupil dilation reflects post-decisional
consolidation of the choice but not the pre-
decisional consolidation of the choice.
Gilzenrat et al.,
2010
Pupil dilation
Pupil diameter can be measured to reflect locus
coeruleus activities. Increases in pupil diameter
baseline indicated decrease in task utility and
disengagement from the task. Decreases in pupil
diameter baseline but increases in task-related
dilations indicated higher task engagement.
Hess & Polt,
1964
Pupil diameter
Pupil diameter reflected the total mental activity.
Hoeks & Levelt,
1993
Pupil dilation
Pupil dilation is a consequence of attentional
effort. Attentional input and pupil responses are
found to have a linear relationship.
8
Table 2.2: Summary of literature review on eye-tracking research (cont.)
Jacob & Karn,
2003
Fixation, gaze
duration, scan
path
In HCI, the most widely used eye-tracking
metrics are number of fixations, proportion of
gaze on each area of interest, fixation duration
mean, number of fixations, gaze duration mean,
and fixations per second.
Johnson &
Mayer, 2012
Area of interest,
Scan path
Spatial contiguity resulted in more attempts to
integrate words and pictures and had more
successful integration of words and pictures
during learning which resulted in more
meaningful learning outcomes.
Just & Carpenter,
1980
Gaze duration,
fixation duration
Readers pause longer on the last word in a
sentence when reading. Readers also pause
longer when processing loads are greater like
accessing infrequent word etc.
Kang &
Wheatley, 2015
Pupil diameter
Real-time changes in stimulus salience motivate
pupil dilation. Fluctuations of pupil size reflect
what is being attended.
Kliegl et al.,
2006
Fixation duration
Most of the time, people’s mind processes
several words in parallel at different cognitive
levels.
Klin et al., 2002
Fixation duration
People with autism exhibit abnormal pattern of
social visual pursuit with reduced eye fixations
and increased fixations on mouths, bodies, and
objects.
Krajbich et al.,
2010
Fixation duration
The visual fixation process has a causal effect on
the value comparison process. People’s choice
can be biased by manipulated relative fixation
durations.
9
Table 2.2: Summary of literature review on eye-tracking research (cont.)
Laeng et al.,
2012
Pupil diameter
By reviewing prior studies in pupillometry, pupil
response was found to be helpful in studying
preverbal or nonverbal participants.
Mason et al.,
2013
Fixation count,
fixation duration,
first-pass fixation
time on an Area
of Interest, look-
back fixation time
Eye-tracking revealed that abstract illustration
was more efficient than text alone. Readers made
greater effort to integrate verbal and pictorial
information when the information was presented
with text and illustrations.
Naber et al.,
2013
Pupil dilation,
pupil oscillation
Pupil dilations and constrictions are both
enhanced by available attentional resources.
O’Regan, 1980
Fixation duration,
saccade size
When processing linguistic information within
six letters distance from current fixation point,
the duration of current fixation and the size of
the next saccade to be made would be affected.
Beyond six words, the linguistic processing
became very slow and the current fixation
duration and size of next saccade would not be
affected.
Orquin & Loose,
2013
Fixation location
Attention processes plays a significant role in
making decisions. When making decisions,
people make trade-offs between working
memories and fixations.
Rayner, 1978
Fixation duration,
saccade length,
scan path
Eye-tracking data can tell us the processing
activities involved in a task. Using eye-tracking
data as the dependent variable is a valid
approach when studying information processing
tasks.
10
Table 2.2: Summary of literature review on eye-tracking research (cont.)
Shimojo et al.,
2003
Gaze location,
gaze duration
Manipulated gaze duration results in significant
preference biases. People’s own gaze bias can be
interpreted as a subconscious level preference.
Triesch et al.,
2003
Gaze direction
Human vision has a highly task specific nature
that people only focus on the information that
lead to the solution of a certain task.
Tsai et al., 2012
Fixation duration,
fixation heat map,
scan path
Participants pay more attention to chosen options
rather than to rejected options when making a
decision. Successful problem solvers and
unsuccessful problem solvers have significantly
difference in the scan sequences.
Wierda et al.,
2012
Pupil dilation
Pupil dilation provides important information
regarding the occurrence of attentional
processes.
Hess & Polt,
1960
Pupil dilation
Pupil dilation reflected people’s increased
interest in visual stimuli.
Krugman, 1964
Pupil dilation
Pupil dilation indicated usefulness on
interpreting user interest on visual stimuli.
Stass & Willis,
1967
Pupil dilation
By measuring the pupil dilation, the study found
out both women and men were attracted by
others who appear to be interested in them.
11
3. THEORETICAL BACKGROUND AND HYPOTHESES
Jacobs et al. (2015) examined participants’ involvement in, and evaluation of
motion pictures. The results indicate that consumer reviews significantly influence
participants’ evaluation while expert reviews do not. However, expert reviews have no
effect on participants’ involvement with the content compared to consumer reviews. In
the context of online shopping, Purnawirawan et al. (2014) found that for positive
reviews, expert review did not result in higher purchase intention compared to consumer
reviews. However, according to Senecal and Nantel (2004) , consumer recommendations
were found to be significantly more trustworthy than expert recommendations. The
results from the study by Utz et al. (2012) also indicated that consumer reviews were the
key factor to judge the online store trustworthiness. Chiou et al. (2014) found out that in
the context of culture offerings, online expert culture reviews have a significantly higher
credibility than consumer reviews.
Despite the contradicted findings from previous studies, consumer reviews, in
many cases, were found to be more trustworthy than expert reviews in e-commerce.
Trustworthiness can lead to higher user attention on the reviews. Bettman et al. (1998)
also explained that due to the limited processing capability of consumers, they generally
cannot process all available information and they would only direct their interest on
information that are perceived to be relevant to their current goals.
Although expert opinions have higher authority in certain contexts, as a shopping
aid, recommender systems can be more useful if consumers can be attracted to the
content. The similarity-attraction paradigm (Byrne, 1971) can be used to explain the
reason consumer reviews won over expert reviews in a number of prior studies in the
context of online retailing. The similarity-attraction paradigm posits that people like and
are attracted to people who are similar to them (Byrne, 1971). Byrne and Griffitt (1973)
found that attraction was found to be positively affected by people with similarities. Also,
economic status, simple behavioral acts, and task performance were also found to
positively influence perceived attraction among people.
Consumer reviews were written by former consumers who were previously likely
to be in or who were facing similar situations as the current customer or shopper. Hence,
12
customers can empathize and relate well with consumers who likely had more similarity
in goals, experiences, and/or attitudes. Experts, On the other hand, though considered to
be a higher authority in certain fields, may not share similarities with current customer or
shopper. Consequently, consumer recommendations are expected to attract greater
interest than expert recommendations.
Therefore, based on the similarity-attraction paradigm, the following hypothesis is
proposed:
H1: Consumer recommendations will attract greater user interest than expert
recommendations.
According to Bettman et al. (1998), people often have no well-defined
preferences until they start to build the “preference pool” when they need to make a
choice. As indicated from the study by Shi et al. (2013), in the initial phase of online
shopping, people tend to increase the “preference pool” in order to not miss any
potentially good alternatives. When the “preference pool” reaches saturation, people tend
to narrow down the alternatives by rejecting new products and reduce the size of the
“preference pool” in order to reach the task goals.
Galinsky and Mussweiler (2001) found that the first offers served as anchors and
were a strong predictor of the final deal in a seller-buyer context. During the buyer’s
decision making process, his or her judgements rely heavily on the initial anchor.
People’s judgement are severely biased by uncertainty and anchoring bias can occur
(Tversky & Kahneman, 1974). As consumers work on shopping tasks, their uncertainty
about the outcome will be lower as they carry out the evaluation process. Hence, the
initial anchors on specific products that have gone through the evaluation process can
deter attention on subsequent product recommendations offered by the online
recommender system.
Based on the anchoring effect and bias, decision makers tend to make decisions
toward the initial anchor (Adomavicius et al., 2013). In an e-commerce context, after a
decision maker has anchored on specific products of interest to them, they are less likely
13
to attend to other recommendations offered by online recommender systems. Hence, the
following hypothesis is proposed:
H2: The earlier a recommendation is offered by online recommender systems, the greater
the user attention toward the recommendation.
14
4. METHODOLOGY
A 2 X 3 X 2 mixed experimental design was used for his research and the
experiment was conducted in the Laboratory for Information Evaluation on the Missouri
University of Science and Technology campus. The first factor refers to the source of
recommendations provided by an e-commerce website: expert vs. consumer
recommendations. The second factor refers to timing of recommendations, i.e., when the
recommendations are offered, i.e., early, middle, or late in the e-commerce shopping
process. The third factor refers to two product types, laptop and cell phone, used in this
experiment. The first and second factors are between-subject factors whereas the third
factor is a within-subject factor. Hence, there are 6 (i.e., 2 X 3) experimental conditions
in this study. Subjects were randomly assigned to one of the 6 conditions, and the two
products or levels associated with the within-subjects factor were counterbalanced to
address any potential ordering effect.
76 subjects were recruited from Missouri University of Science and Technology.
All subjects were pursuing their bachelor’s degree. All subjects have normal eye-sight
before or after adjustment. The average size in each experimental condition is 12. A
consent form was provided to each subject upon arrival. Subjects were asked to go
through a training session on the shopping website that was designed for the experiment.
They were then asked to carry out two shopping tasks including two types of products:
cell phones and laptops. As mentioned earlier, the sequence of these two tasks involving
two different products was counterbalanced. Each subject was given extra credit for their
class and was provided with a souvenir after the experiment.
Three Tobii T60 eye-trackers, i.e., one of them in each of three separate lab
rooms, were used as the computer monitor displays for the experiment. The resolution of
the display is 1280 * 1024. The use of three eye-trackers allowed us to conduct three
concurrent experimental sessions with the subjects. The moderator (or experimenter) at
each of the three stations was given a standardized moderator script to following in
conducting the experiment to avoid moderator biases. The luminance of all lab rooms
were controlled to be at the same level.
15
The recommendation source was manipulated in two categories: expert vs.
consumer. In the experiment, the recommendation source was highlighted on the
recommendation pages. The heading used for the recommendation page was either
“Other consumers recommend this product to you” or “Experts recommend this product
to you”. Several product reviews were provided on each product recommendation page
and they were extracted from existing e-commerce websites. On each recommendation
page, an image of the recommended product along with specifications of the
recommended product were displayed.
The recommendation timing was manipulated in three categories: right after
entering the website (i.e., early recommendation), after clicking “Add to shopping cart”
for the first chosen product (i.e., mid recommendation), and after clicking “Purchase”
button (i.e., late recommendation). Early recommendation appeared when the subject first
entered the shopping website and before any other activities were conducted, i.e., no
alternatives were gathered by this time. Mid recommendation popped up right after the
subject has added the first item into the shopping cart as alternatives were being
collected. Late recommendation appeared when the subject clicked on the purchase
button as preliminary purchase decision has been made.
The subjects were asked to complete two shopping tasks: (i) purchase a laptop,
and (ii) purchase a cell phone. Both products were chosen because of their popularity
among the pilot test subjects. The laptops had higher average prices than the cell phones.
The task sequence was counterbalanced such that some subjects shopped for a cell phone
first while others shopped for a laptop first.
The shopping website allowed subjects to search using various combination of
search criteria to browse product details from the search results. The subjects were
allowed to conduct search activities within the product database until decisions were
made. Single criteria searches and multiple criteria searches were both supported. There
was no time limit given to complete each task.
16
5. DATA ANALYSIS AND RESULTS
Due to eye-tracking recording failure, 5 out of the 76 data points were excluded
from the data set. All data were recorded by Tobii Studio software on Tobii T60 eye-
trackers. The corneal reflection based devices computed and recorded the data including
time, coordinates of eye movement activities, eye movement activities, and pupil
diameter at a sample rate of 60 per second. Several variables were computed by using the
video recordings of all subjects.
A data reduction procedure was conducted to convert raw data into cleansed
fixation data on the recommendation pages. All data were exported from Tobii Studio in
the format of xlsx. Five Excel VBAs were implemented to achieve the following goals:
calculating pupil diameter baseline, cleansing data by time, cleansing data by gaze type,
removing duplicate fixation entries, and calculating targeted pupil diameters. The pupil
diameter baseline was calculated based on the first 100 seconds of recording during
which all subjects were going through the instructions for the experiment.
Fixation durations on the recommendation pages for each subject were calculated.
As the total browsing time varied across subjects, we calculated fixation duration per
second by dividing total fixation duration by total recommendation browsing time. Pupil
dilation was calculated as the percentage of pupil diameter change when browsing the
product recommendation page versus the baseline condition (i.e., when reading
instructions). By reviewing the recording footages, we observed that all subjects fixated
on the recommendation title which indicated their awareness of the recommendation
source.
Outlier tests were conducted to detect and remove potential outliers for both
dependent variables. 4 outliers were detected and removed for data analysis on pupil
dilation. 10 outliers were detected and removed for data analysis on fixation duration per
second.
Order effects were tested for both dependent variables and no order effects for
tasks (i.e., order of product types) were found for pupil dilation or fixation duration per
second as dependent variables.
17
Statistical analysis were performed using SPSS 21 to conduct three-way ANOVA
for each of the dependent variables for the two between-subjects factors:
recommendation source and recommendation timing, and one within-subjects factor:
product type.
5.1. DATA ANALYSIS ON PUPIL DILATION
The pupil diameter for each task was calculated by averaging the left and right
pupil diameters. The average of the pupil diameters was then calculated based on the time
stamp of product recommendation page to reveal the target pupil diameter (target PD):
diameter of the pupil when looking at the product recommendation page. Pupil dilation
was then computed relative to the pupil diameter baseline (PDBL) using following
equation.
𝑃𝑢𝑝𝑖𝑙 𝑑𝑖𝑙𝑎𝑡𝑖𝑜𝑛= (𝑡𝑎𝑟𝑔𝑒𝑡 𝑃𝐷−𝑃𝐷𝐵𝐿) ÷ 𝑃𝐷𝐵𝐿
Pupil dilation reveals the percentage of change on pupil diameter at a given period
of time as compared to the baseline.
Excluding the outliers, 67 sets of data for both tasks were used for the analysis.
We have an average sample size of 11 for each of the experimental conditions. The
descriptive statistics for pupil dilation was shown in table 5.1.
Table 5.1: Descriptive statistics for pupil dilation
Timing
Source
Mean
# of Subjects
Pupil dilation
_cell phone
Early
Expert
-4.04%
12
Consumer
-0.76%
10
Total
-2.57%
22
Mid
Expert
-3.68%
12
Consumer
-0.29%
11
Total
-1.78%
23
Late
Expert
-1.50%
11
Consumer
0.00%
11
Total
-0.75
22
Total
Expert
-3.13%
35
Consumer
-0.14%
32
Total
-1.70%
67
