AI-Enabled Post-Purchase Customer Experience December, 2024

Background
With the advancement of AI, e-commerce is set to revolutionize post-purchase experiences by leveraging technologies such as
advanced large language models, collaborative multi-agent systems, and voice-enabled solutions. These innovations have the
potential to transform routine interactions into seamless, personalized, and proactive engagements that exceed customer
expectations. This research focuses on three key objectives: understanding evolving customer preferences for AI-driven
post-purchase interactions, evaluating the impact of next-generation AI technologies, and projecting how these advancements will
shape the customer journey in the near and mid-term.

Executive Summary
The rise of artificial intelligence is set to transform the post-purchase customer experience in e-commerce. Our comprehensive
research, involving 448 survey respondents and in-depth interviews, reveals that consumers are increasingly seeking personalized,
intuitive digital interactions while maintaining control over their experience.

We identified four distinct customer personas—from tech-savvy enthusiasts to more traditional website users—who share common
desires for AI-powered solutions that are helpful without being intrusive. Customers want hyper-personalized experiences that treat
them as individuals, with 71% expecting tailored content and 84% valuing personalized interactions. However, they are equally
concerned about data privacy and want transparent, controllable AI systems.

The research uncovered key opportunities for AI integration, including personalized landing pages, intelligent shopping assistants,
and features like "Shopping Recap" that provide comprehensive insights into purchasing habits. While users are excited about AI's
potential to streamline tasks like order tracking and provide personalized recommendations, they prefer AI that advises rather than
completely takes over decision-making. Notably, 50% of respondents find current AI solutions too generic, highlighting the need for
more sophisticated, context-aware technologies.

However, the implementation of AI is not without significant risks. Key concerns include potential data privacy breaches, AI bias that
could unfairly impact customer experiences, and the danger of over-reliance on AI systems that lack human emotional intelligence.
There are also implementation challenges, such as high computational costs and the need for sophisticated ethical guardrails.
Companies must be vigilant about transparency, regularly audit AI systems for bias, and provide clear mechanisms for users to
understand and override AI decisions.

To successfully implement AI in e-commerce, businesses must prioritize user trust, data protection, and intuitive design.
Recommended strategies include conducting iterative feature testing, deploying machine learning models on customer devices,
exploring partnerships with AI startups, and investing in research on AI ethics and personalization. By balancing technological
innovation with user-centric design, companies can create more engaging, efficient, and personalized online shopping experiences
that meet evolving consumer expectations.

Key Insights and Emerging Opportunities

The team’s research consisted of, but not limited to, consumer surveys and interviews. The survey engaged 448 respondents.
Roughly half of the respondents were Carnegie Mellon University students and the other half were from across diverse
demographics. The aim of the survey was to understand user’s preferences of possible AI features in the post-purchase space.
Qualitative in-person interviews with 20 consumers provided deeper insights, supplemented by expert contributions from AI experts
and human-centered design experts. The survey revealed four distinct customer personas. Tech-Savvy Enthusiasts (28.6%) are
proficient in both website usability and AI interaction, appreciating seamless experiences that blend traditional and AI-driven
elements. Passive Browsers (25.4%), score high on usability but vary in AI engagement and prefer traditional website features and
take a more exploratory approach to AI. AI Experimenters (18.1%), despite facing challenges with website usability, show a strong
interest in AI and are open to innovative tools if usability improves. Website-Focused Users (27.9%) prefer straightforward,
conventional website interactions and are less engaged with AI, favoring traditional user experiences.

Our insights can be classified into three broad categories - Hyper-personalization, Intuitive AI Interactions and Agentic AI. In the
following sections, we dive deeper into each of these.

Insight 1: Hyper-personalisation
Greater demand for hyper-personalized customer engagement: Consumers now expect businesses to offer more personalized
interactions as a basic requirement in e-commerce. A significant 71% expect tailored content, while 84% value being treated as
individuals rather than transactions, highlighting the importance of personalized engagement in winning customer loyalty14. For
example, generative AI can meet these expectations by delivering creative, hyper-personalized notifications that have been shown to
increase retention by 60% and order frequency by 45%16. However, this shift comes with challenges, as only 21% of consumers
strongly agree that businesses are adequately protecting their identity and information11. Despite this caution, 66% of customers

Privileged & Confidential Page 1

AI-Enabled Post-Purchase Customer Experience December, 2024
want personalized messaging, and 58% value follow-ups after purchases14, indicating a willingness to share data when it enhances
their experience.

Appreciation for personalized product usage guides: 47% of respondents rated "personalized product usage guides and tutorials" as
"moderately" to "extremely" important after placing an order37 which suggests that users value personalized guidance that helps
them effectively utilize purchased products. This is also an opportunity to integrate multimodal AI as it is moving towards improving
visual, & audio based real time input & feedback—such as identifying issues via images or video shared by customers to diagnose
problems or suggest product upgrades. For example, AI could interpret a customer-uploaded photo of a broken part to provide exact
replacement recommendations or video tutorials for at-home repairs.

Openness to predictive reordering: 30% of respondents found "predictive/automated reordering of items" to be at least
"moderately" important to “very” important, indicating a willingness to embrace AI for simplifying repeat purchases. This aligns with
the desire expressed in previous conversations for AI to handle routine tasks and streamline the shopping process. However, 41% of
the respondents did not find it important “at all”, further driving the point that there has to be a phased rollout of AI features
depending on the personas37.

Privacy is a major concern; control and transparency are key: Many interviewees expressed hesitation about sharing personal
information for hyper-personalization due to concerns about data breaches and misuse. One person stated, "Our biggest concern is
breach of personal data and that data being used without my consent". This indicates that companies must prioritize data security
and transparency to gain user trust. Individuals are more receptive to hyper-personalization if they have control over their data and
understand how it's being used. One interviewee said, "I would like to have it based on my approval or my discretion. I wouldn't want
the shopping itself to be automated without my consent". This implies that providing clear choices to opt in or out, along with
straightforward explanations of how AI helps in their shopping, could help reduce user concerns4.

Opportunities :
Recent advancements in reinforcement learning (RL) have greatly improved personalization in things like content curation and user
experience. Technologies like deep reinforcement learning help RL handle complex tasks, and adding human feedback has made AI
systems more user-friendly and effective. These developments make it possible for systems to offer more dynamic and context-aware
personalization in areas such as e-commerce and marketing21,22. Some areas of opportunities are as follows:

Personalized landing pages: As generative AI and reinforcement learning get better, websites and apps of the future could be
hyper-personalized, changing to fit each person who visits. Everything on a website—what it displays, how it’s arranged, the deals it
offers, and the way it looks—can be customized for the consumer based on what they do on the site, what they like, and their past
visits18, 19. There are startups that leverage generative AI to create different versions of content and learn how to make the website
better for when the person visits again, which can change how we shop online, especially because we like websites that feel personal
to us20. To show you what this means, in the appendix are two different app screens made for two types of people. One page is for
someone who loves new technology, has high income, and often uses a digital assistant. The other page is for someone who doesn't
care much about tech and mostly wants to check their spending and order details, and they don't often buy things just because they
are recommended.

“Shopping Recap” (Inspired by Spotify Wrapped): Implementing an “Shopping Recap” feature, similar to Spotify’s “Wrapped”, offers
ecommerce companies a great chance to boost user engagement, strengthen customer loyalty. It is a great way to make the
post-purchase journey more enjoyable, as it will act as a one-stop shop dashboard for all of the customer’s past shopping habits. The
website gains increased engagement, deeper customer insights, and potential growth in sales. Through interviews and surveys,
consumers are looking for a personalized summary of their past shopping habits and how to improve future shopping experiences. In
the appendix are some examples of these interfaces; more can be found here.

AI powered Credit Card Optimization: Experian reports that the average American holds four credit cards, indicating a significant
user base for this feature. A JD Power study found that 36% of consumers select credit cards based on reward programs, showing a
demand for optimization tools. Incorporating an AI-powered credit card optimization feature can improve user satisfaction by
automatically selecting the best credit card, maximizing customer’s benefits to get the most points, cashback, or discounts for every
transaction. It eliminates confusion by providing clear, real-time recommendations on which card to use. A startup working in this
space is MAXREWARDS that helps manage credit cards and suggest users the card to use that will help them maximize rewards,
cashbacks and benefits (snapshot).

Risks :

Data privacy: Personalized experiences must strike a balance between exploration and exploitation. Building trust not only requires
transparency in data usage, strict compliance with regulations, but the user experience must also clearly communicate the working

Privileged & Confidential Page 2

AI-Enabled Post-Purchase Customer Experience December, 2024
of AI systems & data usage40. Giving customers control over their data and providing clear explanations of AI algorithms can foster
trust and acceptance.

AI bias: A key limitation of current AI models is the difficulty in ensuring fairness and avoiding bias, as noted by AI experts43, which
could be magnified in highly personal experiences, like ecommerce. Bias in AI-driven personalization can severely impact consumer
trust and brand reputation41. AI experts emphasize ethical design principles as a non negotiable43, such as guardrails to prevent
harmful behavior and frameworks like Constitutional AI39. Regular audits, ongoing evaluations, and adherence to emerging
compliance requirements are critical to mitigating these risks.

Implementation Costs: While 1:1 personalized web pages are technically possible, they pose a computation and scalability
challenge44. To address these hurdles, businesses can adopt a phased approach, starting with smaller pilot projects to test the
technology, gather user feedback, and refine the system before scaling.

Insight 2: Intuitive AI Interaction

People seek user-friendly, conversational artificial intelligence: People want AI that feels natural, responds to natural language and
is easy to use, whether that is through voice commands or text-based chat. Generative AI is a game changer, powering chatbots and
automated voice systems. However, one of the biggest challenges with AI-based support is that many customers find it lacking
customer-friendly qualities. In the survey, 50.2% of respondents said AI often “gives generic solutions; I need specificity,” while 40.4%
felt that “it’s not human and not intelligent enough37.” When asked about their experiences with customer service, people mostly
had negative feedback about talking to automated bots or agents over the phone. Over half (53%) said they would not prefer this
type of interaction, while 23% were somewhat comfortable with it. On the other hand, speaking to a human agent received much
more positive responses, with 30% marking it as their top preference and 24% as a very preferred method.

Importance of customer support and its growing expectations: The customer care AI market is projected to value $6 billion by 2032.
Customers are increasingly concerned about the use of their time, with a 63% increase in the expected speed of response, and a 57%
increase in the expected speed of resolution12. A user emphasized that interacting with a real agent via phone was the fastest way to
resolve customer service issues, highlighting the importance of time-saving solutions. “A lot of the time chatbots respond with ‘I
don’t understand what you’re saying’, or they give you the wrong information”, said a consumer in a research report conducted by
Zendesk11. 74% of customers also expect for AI to be able to handle takeover and improve transition from a chatbot to a human
agent; catching the human agent upto speed and giving them an understanding of past interactions. 78% of consumers want AI to
summarise their questions or issues so the person helping them can quickly resolve the matter when being transferred from a
chatbot to a human agent11.

Text-based chat is popular; graphical interfaces can feel creepy: Text-based chat interfaces are also seen as highly convenient,
particularly when using a desktop computer. One interviewee described interacting with ChatGPT as "like I'm texting my mentor,"
highlighting the familiar and comfortable nature of this interaction style. Another user stated a preference for typing it on the phone
but did not prefer it for a desktop4. This suggests a potential shift in preference based on the device being used. Interestingly, there's
some resistance to AI interfaces that try to be too lifelike or human-like. One interviewee specifically said they would find a
"graphical bot speaking to me or like making it look lifelike" to be "creepy," suggesting that a more minimalist and functional
approach is generally preferred4.

Opportunities :

Improved natural language processing (NLP) techniques help AI systems better understand user intentions and context for more
advanced dialogues. Large language models, like GPT-4, enhance conversational AI by handling complex conversations and various
tasks. Additionally, these systems can now process multiple types of inputs like images, text, and audio, and improved speech
recognition technologies make voice interactions more accurate. AI can also detect emotions, allowing for responses that are more
empathetic and personalized23, 24, 25, 26, 27. Some areas of opportunities are as follows:

Revolutionize customer service with advanced AI voice agents and bots: Enhance automated customer service by utilizing
advanced, human-like voice agents and bots capable of addressing customer queries with natural, conversational interactions. AI
startups such as Simple AI are focusing on developing sophisticated AI assistants. Their beta product features a voice assistant that
replicates human-like speech and responses so convincingly that customers may perceive they are interacting with a human agent.
The innovative technology developed by such startups has the potential to be seamlessly integrated into the daily operations of
e-commerce businesses, significantly improving customer satisfaction levels while maintaining cost efficiency24, 26, 27.

Personalize mundane communications: Companies like Zomato and Sephora have successfully used AI to create personalized, witty,
and timely notifications that connect with users by referencing their preferences, habits, and relatable moments. This strategy has
proven highly effective in boosting engagement, loyalty, and even social sharing. With the advantage of first-party user data and

Privileged & Confidential Page 3

AI-Enabled Post-Purchase Customer Experience December, 2024
cohesive ecosystems of smart devices and subscription services (like Alexa, Echo, and Prime), e-commerce platforms can craft
highly-personalized communication styles to enhance user engagement across every modality - SMS, text, email or voice15.

Risks :

Lack of autonomy & transparency: Customers may feel misled when interacting with AI instead of humans, eroding trust and risking
legal issues. While intuitive AI is appealing, overly human-like behavior unsettles users and can breed distrust42.To address this,
customer education should be a key part of GTM strategies. Explaining how algorithms work, especially for personalized
recommendations, and providing insights into data usage could foster trust40. UX design must regularly remind users they are
interacting with AI.

Over-Reliance on AI: AI experts warn that over-reliance on AI in customer management could harm customer relationships. Some
aspects of being human such as emotional intelligence and creative expression cannot be replicated43. Users also expressed concerns
about an over reliance on AI systems while questioning their reliability in outputs. Hence, Human-driven feedback mechanisms
alongside AI outputs must be incorporated to ensure as a responsible practice to ensure oversight and safety.

Insight 2: Agentic AI Advisory

Specific tasks are more acceptable; limited autonomy is key: People are more willing to assign specific, clearly outlined tasks to AI
“agents”. Managing order tracking, providing delivery updates, and scheduling reminders seem to be areas where users are more
comfortable giving up control. However, tasks involving financial transactions or major purchase decisions remain areas where
human intervention is preferred. They like features like automated refunds, but only if they can approve the action first. This shows
that users are willing to let AI assist them, but they don't want it to take over completely. When asked if survey respondents would
be okay with an AI agent automatically cancelling and replacing a delayed order, 40.6% said they’d be open to it but would want to
review the replacement before placing the order. As one person put it, they would rather have AI "advise me" than "do it for me4, 37.”

The brand will have to earn the consumer’s trust, it is not given: If and when AI agents are implemented in e-commerce platforms,
users are more receptive to AI agents that demonstrate reliability, transparency, and a track record of success. Many are willing to try
out AI-powered features so it is advised to have a gradual implementation, phased roll-outs and the option to override AI decisions.
The sentiment seems to be "let's start small and see how it goes" rather than "hand over the reins completely." One person, for
example, stated they would be comfortable with an AI agent if "it's going to be really accurate with all the recommendations". This
suggests that trust in AI's capabilities is a prerequisite for acceptance of more autonomous behavior4.

Opportunities :

Autonomous AI agents and on-device machine learning (ML) are rapidly transforming how AI integrates into daily life and work.
Apple’s "Apple Intelligence" initiative and Google’s advancements with Gemini Nano are paving the way for powerful, on-device ML28,
29, 30, 31, 32
. These technologies enable ML models to run locally on consumer devices, reducing reliance on cloud processing while
maintaining efficiency and privacy. Perplexity has introduced an advanced AI shopping assistant that improves online shopping by
using chat and image inputs to let users research products, compare them in simple language, view detailed product cards, and buy
with one click. Rakuten and Agora are developing similar tools34, 35. Given below are areas of opportunities:

Transform chatbots to next-generation intelligent shopping assistants: Customers' desire for more reliable automation of simple
tasks shows a shift from basic chatbots to smarter, context-aware agents. For instance, Amazon’s Rufus’ ability can be improved to
handle complex queries about products, spending, manage multi-instruction tasks for order management, and make proactive and
context-aware recommendations for delivery, tracking & upselling. In the survey, when asked what would you like a personal
shopping assistant to do, the respondents said “look to optimize purchases”, “simply buy something when I run out of it”, “remind
me of refunds, discounts, and track the package without using another app” and “give me a dashboard with stats37.” For example,
one person desired a shopping assistant who would "understand my style from my past purchases" and alert them about new items
from their preferred brands. Nearly 53% selected “automated refunds and smart cancellations” (for example, automatically
cancelling a delayed order) as “very” or “extremely” important in their post-purchase journey37. Wells Fargo’s AI assistant, Fargo, also
caters to a similar use case by providing customers the shopping experience, provides personalized support, and offers financial
insights related to customers' purchasing behaviors. Given in the appendix are the snapshots of Fargo’s functionalities.

Risks :

Controllability: As expressed from customer interviews, users expect AI to assist, not replace, human decision-making, with
mechanisms for oversight and intervention. Experts also noted that AI should not be framed as "humans vs. AI" to avoid promoting a
perception of AI as an unreliable threat43. Hence, explainable AI systems should be designed with safety in mind, incorporating

Privileged & Confidential Page 4

AI-Enabled Post-Purchase Customer Experience December, 2024
built-in guardrails to prevent harm. User experiences with shopping assistants must be designed with mechanisms that allow users to
review or override decisions easily.

A novel approach to collecting survey responses: LLM Survey Responses vs Human Survey Responses
The primary aim of this experiment was to evaluate the capability of Language Learning Models (LLMs) to replicate and empathize
with human behavior, specifically in the context of online shopping experiences, and to analyse how an LLM would respond to the
survey. The study involved comparing genuine customer survey responses with those generated by LLMs, which were provided with
predefined personas mimicking real survey participants. This approach sought to determine how well LLMs could understand and
mimic actual consumer behavior in a meaningful way.

A key observation was that the genuine customer responses exhibited balanced clustering, with moderate overlaps reflecting the
natural diversity of consumer preferences and behaviors. In contrast, the LLM-generated responses formed tighter groupings with
less spread-out centroids. This suggests that the LLMs' clustering was more systematic or synthetic, lacking the variability observed in
genuine human responses.

In conclusion, the clusters formed by LLM responses differ significantly from those based on genuine customer data. This indicates
that LLMs, as of now, are limited in their ability to fully emulate the nuanced and diverse experiences of real consumers. While LLMs
can generate structured and consistent responses, they currently fall short of capturing the depth of actual human behavior in the
context of e-commerce surveys.

Next Steps
We recommend the following next steps to further the ongoing research efforts into the future of AI and post purchase ecommerce:

Product features and rollouts: Conduct iterative testing of proposed features with diverse demographics to refine their design and
functionality, and use real-world scenarios to stress-test AI systems for scalability. Additionally, expand trials to customers of other
e-commerce platforms to gauge feature adoption across different e-commerce platforms, and collect competitive insights to
strategically position ahead in the market.

Upcoming AI technologies: Focus on deploying machine learning (ML) models directly onto customer devices to enhance privacy and
personalization, while also integrating multimodal inputs such as text, voice, and image to improve user interaction and engagement,
especially in predictive and diagnostic scenarios. Additionally, introduce AI-powered financial tools that recommend the most
suitable credit cards for purchases and highlight missed rewards, similar to MaxRewards, and expand the integration of loyalty
programs with predictive financial insights. Moreover, evaluate which AI capabilities the company could develop in-house and where
it may be more efficient to acquire startups or form external partnerships. It is crucial to stay updated on advancements in
reinforcement learning with human feedback (RLHF) and multimodal capabilities, and incorporate these into the company’s strategic
roadmap.

Innovation through partnerships and research: Focus on startups specializing in hyper-personalization, multimodal AI, or
conversational AI to accelerate development cycles and gain niche expertise. Create a team to monitor startups working on real-time
personalization, reinforcement learning for UX, and AI customer support innovations. Commission research at Carnegie Mellon
University or other institutions focusing on topics such as AI ethics, real-time AI decision-making, or on-device ML for
e-commerce.Implement pilot programs for AI technologies, such as shopping recaps or autonomous agents, to gather customer
feedback and fine-tune solutions.

Appendix

Research Scope
To explore how AI can enhance post-purchase experiences, the research employed a comprehensive approach combining surveys,
interviews, and desk research. Surveys engaged 448 respondents across diverse demographics to identify preferences, expectations,
and pain points in e-commerce. Qualitative interviews with 20 consumers provided deeper insights, supplemented by expert
contributions from AI experts and human-centered design specialists. Desk research contextualized findings by analyzing competitive
strategies and market trends in e-commerce. The project employs a multi-faceted approach:

Methods:
● Surveys: Engaged 448 respondents across diverse demographics to quantify preferences, expectations, and pain points in
post-purchase experiences.
● Interviews: Conducted with 20 Amazon customer archetypes for qualitative insights, complemented by discussions with AI
experts, including CMU professors Anand Rao, a seasoned strategist in applied AI, and Sherry Tongshuang Wu, an expert in

Privileged & Confidential Page 5

AI-Enabled Post-Purchase Customer Experience December, 2024
human-centered AI design. Their contributions offered deep insights into AI’s current and future capabilities in transforming
post-purchase experiences.
● Desk Research: Analyzed competitive strategies, market trends, and literature to contextualize findings within the broader
e-commerce landscape.

Target Audience: Focused on e-commerce customers across diverse personas to ensure broad relevance.
The user personas were derived from a principal component analysis (PCA) mapping customer interactions along two dimensions:
Website Usability (PCA1) : Refers to the ease with which users navigate, interact with, and achieve their goals on a website, including
features like intuitive design, responsiveness, and efficient task completion.
AI Interaction (PCA2) : Encompasses user engagement with AI-driven tools, such as chatbots, personalized recommendations, and
predictive features, reflecting their comfort and willingness to adopt these technologies in their shopping journey.

Opportunities Interface Samples

I. Personalized landing pages

Fig 1. Tech optimist’s page Fig 2. Tech sceptic’s page

II. “Shopping Recap”

Privileged & Confidential Page 6

AI-Enabled Post-Purchase Customer Experience December, 2024

Fig 3. “Shopping Recap” interface samples

III. AI-powered credit card optimization

Fig 4. Maxrewards

IV. Wells Fargo Virtual Assistant

Privileged & Confidential Page 7

AI-Enabled Post-Purchase Customer Experience December, 2024

Fig 5. Wells Fargo’s virtual assistant, Fargo

A novel approach to collecting survey responses: LLM Survey Responses vs Human Survey Responses
Large Language Models (LLMs) have emerged as a powerful tool for generating synthetic survey responses, offering researchers and
analysts new possibilities in data collection and analysis. This document explores the application of LLMs in survey research,
comparing their outputs to human responses and examining the characteristics and differences observed.
LLM Application in Surveys
LLMs can be utilized in survey research in several ways:
1. Generating Synthetic Responses: LLMs can create a large volume of survey responses based on predefined parameters,
helping researchers simulate diverse participant pools.
2. Pilot Testing: Before launching a full-scale survey, LLMs can generate test responses to identify potential issues with
question wording or survey structure.
3. Data Augmentation: LLMs can supplement existing human responses, especially in cases where certain demographic groups
are underrepresented.
4. Scenario Analysis: Researchers can use LLMs to generate responses for various hypothetical scenarios, aiding in predictive
analysis.
Comparability to Human Responses
The comparability of LLM-generated responses to human responses varies depending on several factors:
Similarities:
● Linguistic Diversity: LLMs can produce responses with varied language patterns, similar to a diverse human population.
● Consistency: LLMs maintain consistent persona characteristics across multiple questions, mimicking individual human
respondents.
Differences:
● Nuance and Emotion: Human responses often contain subtle emotional cues that LLMs may struggle to replicate
authentically.
● Personal Experiences: While LLMs can generate plausible scenarios, they lack genuine personal experiences that often
inform human responses.
● Inconsistency and Errors: Humans may provide inconsistent or erroneous responses due to fatigue or misunderstanding,
which LLMs typically avoid unless specifically programmed to simulate such behavior.
Characteristics of LLM-Generated Responses
LLM-generated survey responses often exhibit the following characteristics:
1. Coherence: Responses are generally well-structured and logically consistent across related questions.
2. Variability: LLMs can produce a wide range of responses, reflecting diverse viewpoints and experiences.
3. Scalability: Large volumes of responses can be generated quickly, allowing for extensive data analysis.
4. Customizability: Responses can be tailored to specific demographic profiles or scenarios as needed.
Observed Differences
When comparing LLM-generated responses to human responses, several key differences emerge:
1. Depth of Personal Context: Human responses often include rich, personal anecdotes that LLMs may struggle to replicate
authentically.
2. Unexpected Insights: Humans occasionally provide surprising or innovative responses that fall outside the typical patterns
LLMs are trained on.

Privileged & Confidential Page 8

AI-Enabled Post-Purchase Customer Experience December, 2024
3. Bias Patterns: While LLMs can be programmed to avoid certain biases, they may introduce new, subtle biases based on their
training data. Human responses reflect societal biases in different ways.
4. Response Time Variations: Human response times vary based on question complexity and individual characteristics. LLMs
typically generate responses at a consistent rate.
5. Outlier Responses: Humans may occasionally provide extreme or outlier responses due to unique personal experiences or
misunderstanding. LLMs tend to generate responses within expected parameters unless specifically instructed otherwise.
LLM vs Human Survey Insights: A Comparative Analysis
The process involved analyzing two datasets: genuine responses from actual e-commerce customers and synthetic responses
generated by the LLMs. Before conducting the analysis, both datasets underwent thorough cleaning and preprocessing to ensure
they were suitable for clustering. Using the elbow method, the optimal number of clusters was determined to be four for both
datasets. Principal Component Analysis (PCA) was applied to reduce dimensionality, and the resulting clusters were visualized for
comparison.
1. Skew Towards Majority Opinion
● LLM responses show a significant bias towards the majority opinion.
● Example: Q1_Viz.1,5 demonstrates LLM skew towards "very important" or "extremely important" categories.
● Human responses, while also skewed (~50%) towards these categories, show more distribution in "Unsure" or "not
important" categories.
● Similar patterns are observed in Q2_Viz, Q4_Viz, Q5_Viz, Q8_Viz, and Q10_Viz.
● LLMs struggle to capture relative ranking among top reasons (e.g., in Q2_Viz for reasons people dislike LLMs).
2. Avoidance of Extreme Opinions
● LLMs tend to avoid selecting extreme response options.
● Q1_Viz.2: Where humans chose "Not at all important" or "Extremely important", LLMs favored "Slightly Important"
or "Very important".
● Q3_Viz.1: LLMs selected "very important" for human preference of tapping buttons to check previous order details,
avoiding extreme choices.
3. Misalignment with Human Opinion
● Some questions show significant divergence between LLM and human responses.
● Q1_Viz.3,4: LLMs showed neutrality ("Moderately Important") where humans indicated low importance for
personal shopping assistants and personalized product usage guides.
● Q3_Viz.2,3: LLM responses were opposite to human responses, particularly regarding the importance of voice
assistants.
4. Impact of Scale on LLM Responses
● Increasing the number of LLM-generated responses improved the distribution range.
● 50 personas: Responses mostly restricted to majority opinion.
● 300 personas: Greater distribution across the response range observed.
5. Generic Responses Across Personas
● LLM-generated reasons for scores were similar across different personas (e.g., age 50 vs 25).
● Lack of effective capture of social cues and nuances associated with demographic factors.
● Possible intentional bias avoidance or statistical bias in LLM responses.
6. Effectiveness in Relation to Question Complexity
● LLMs perform better with simpler question formats.
● More effective for "Yes/No/Unsure" questions compared to scale-based questions (e.g., "Score between 1-5").
● Q6 Sheets: Closer distribution between human and GPT responses for Yes/No/Unsure options.
7. Technical Insights
● Temperature > 1.5: GPT produced illegible responses.
● Temperature > 1: Perplexity models produced illegible responses.
● Experiments with changing temperature, source, or model yielded varying results.
● Frequency penalty for Llama model created very illegible responses.

Privileged & Confidential Page 9

AI-Enabled Post-Purchase Customer Experience December, 2024

Build vs. Buy:

Evaluate Core Competencies: Identify which AI capabilities Amazon could develop in-house versus areas where acquiring startups or
external partnerships would be more efficient.Stay updated on AI advancements in RLHF (Reinforcement Learning with Human
Feedback) and multimodal capabilities, integrating them into Amazon's roadmap.

Acquire or Partner with Startups: Focus on startups specializing in hyper-personalization, multimodal AI, or conversational AI to
accelerate development cycles and gain niche expertise.Create a team to monitor startups working on real-time personalization,
reinforcement learning for UX, an in d AI customer support innovations.

AI Advancements:
Proactive Research Investments: Commission research at CMU or other institutions focusing on topics such as AI ethics, real-time AI
decision-making, or on-device ML for e-commerce.Implement pilot programs for AI technologies, such as shopping recaps or
autonomous agents, to gather customer feedback and fine-tune solutions.

Enhance Customer-Centric Features:

Focus on Hyper-Personalization:
Launch predictive reordering and personalized product guides incrementally based on personas.
Develop opt-in/opt-out features to address privacy concerns.
Use Generative AI to create highly adaptable prototypes for dynamic landing pages and personalized user experiences.
Test these prototypes across different regions and demographics to refine functionality.

AI-Powered Financial Tools:

Expand into areas like AI-driven credit card optimization to provide additional value to customers.

Engage with Consumer Behavior Insights:

Persona-Specific Pilots: Tailor initiatives to the four personas identified in your research to maximize user satisfaction and adoption
rates.
Improve Data Transparency: Develop clear communication strategies around data usage to build trust and foster user acceptance of
AI features.

Expand Research Scope:

Explore Related Domains: Investigate areas like AI ethics in customer interactions, sustainability-driven e-commerce AI features, or
real-time emotional sentiment analysis.
International Adaptability: Study how different cultural contexts impact AI feature adoption and tailor solutions accordingly.

Immediate Actionable Steps

1. Pilot a "Shopping Recap" Feature
○ Collaborate with the personalization team to design and prototype a "Shopping Recap" inspired by Spotify
Wrapped.

Privileged & Confidential Page 10

AI-Enabled Post-Purchase Customer Experience December, 2024
○ Test the feature with Amazon's Prime members to gauge its impact on customer loyalty and engagement.
○ Use metrics like retention rates and engagement with personalized recommendations to evaluate its effectiveness.
2. Enhance AI-Driven Hyper-Personalization
○ Expand on existing recommendation engines by incorporating GenAI to deliver dynamic product usage guides and
predictive reordering systems.
○ Prioritize transparency and user control by integrating clear data-sharing preferences within customer settings.
○ Roll out in phases tailored to customer personas identified in the study (e.g., Tech-Savvy Enthusiasts first).
3. Develop Advanced Conversational AI Assistants
○ Prototype an AI assistant modeled on Wells Fargo's Fargo to handle customer service queries, purchase
recommendations, and financial insights.
○ Integrate features such as voice and text-based conversational capabilities, ensuring customer preferences are
respected.
○ Perform A/B testing to identify the best modalities (voice vs. text).
4. Focus on Agentic AI Advisory
○ Launch a limited beta version of an AI assistant that advises customers on financial or shopping decisions, such as
credit card optimization.
○ Ensure users can retain control by approving AI-recommended actions before execution.
○ Use customer feedback to fine-tune trust-building measures and improve agent performance.
Opportunities for Long-Term Implementation
1. On-Device Machine Learning and Multimodal Capabilities
○ Work on deploying ML models directly on customer devices for privacy-enhanced personalization.
○ Integrate multimodal inputs (e.g., text, voice, image) to improve user interaction and engagement, particularly for
predictive and diagnostic use cases.
2. AI-Enhanced Post-Purchase Support
○ Implement image and video-based troubleshooting features in customer support to provide instant solutions for
common issues, such as product repair guides or parts replacement suggestions.
○ Expand the use of RLHF (Reinforcement Learning with Human Feedback) to refine customer interactions.
3. Refinement of Personas for Broader Applications
○ Leverage survey and interview data to refine and expand customer personas, ensuring applicability across different
Amazon services like Alexa, Kindle, and Prime.
○ Tailor hyper-personalized features to these personas to drive adoption and satisfaction.
4. Integration of Financial Tools
○ Introduce AI-powered financial tools that recommend optimal credit cards for purchases or highlight missed
rewards, similar to MaxRewards.
○ Expand into integrating loyalty programs with predictive financial insights.
5. Ethics and Privacy Framework Development
○ Establish a framework for ethical AI use, prioritizing transparency, fairness, and user consent.
○ Collaborate with cross-functional teams to address customer concerns about data breaches and misuse.

If You Had Another Semester

1. Comprehensive Usability Testing
a. Conduct iterative testing of proposed features with diverse demographics to refine their design and functionality.
b. Use real-world scenarios to stress-test AI systems and evaluate scalability.
2. Explore Academic-Industry Collaborations
a. Partner with AI research groups or universities to explore cutting-edge advancements in GenAI, RLHF, and
multimodal AI applications.
b. Run joint projects to address specific challenges like bias mitigation and real-time personalization.
3. Scale Experimentation Across Market Segments
a. Expand trials to non-Amazon customers to understand feature adoption across different e-commerce platforms.
b. Collect competitive insights to position Amazon ahead in the market.

References
1. Cottier, Ben, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen. “The Rising Costs of Training Frontier AI
Models.” arXiv, May 31, 2024. https://doi.org/10.48550/arXiv.2405.21015.
2. Lee, Kai-Fu. “How Do You Get to Artificial General Intelligence? Think Lighter.” Wired. Accessed November 29, 2024.
https://www.wired.com/story/how-do-you-get-to-artificial-general-intelligence-think-lighter/.
3. Diaz, Adriana. “Apple’s New AI Could Broach Sensitive Data, Experts Warn — Take These Steps to Protect Your Privacy,”
November 26, 2024.

Privileged & Confidential Page 11

AI-Enabled Post-Purchase Customer Experience December, 2024
https://nypost.com/2024/11/26/tech/apples-new-ai-could-broach-sensitive-data-experts-warn-take-these-steps-to-protect-
your-privacy/.
4. Customer Interviews Insights [Notebook]
5. Survey Analysis
6. Insights + Research Brainstorming - Figma Board
7. https://www.businesswire.com/news/home/20241008559209/en/America-Speaks-81-of-Consumers-Use-Voice-Tech-Daily-
or-Weekly%E2%80%94Is-Your-Brand-Listening
8. What are your biggest concerns while using AI? Amazon Survey Dashboard
9. How comfortable are you with AI taking proactive shopping decisions on your behalf? Survey Dashboard
10. Expert Interviews
11. https://www.zendesk.com/in/blog/consumer-perceptions-ai/
12. https://www.aiprm.com/ai-in-customer-service-statistics/
13.
14. https://www.mckinsey.com/capabilities/growth-marketing-and-sales/our-insights/the-value-of-getting-personalization-right
-or-wrong-is-multiplying
15. https://blog.zomato.com/introducing-zomato-ai-your-foodie-buddy
16. https://chroniclexpress.com/blogView/507
17. https://www.forbes.com/councils/forbesbusinesscouncil/2024/06/07/generative-ai-and-personalization-at-scale-crafting-cu
stomer-experiences/
18. https://www.producthunt.com/stories/this-ml-concept-will-bring-hyper-personalization-to-your-screen
19. https://venturebeat.com/automation/offerfit-gets-25m-to-kill-a-b-testing-for-marketing-with-machine-learning-personalizat
ion/
20. https://butternut.ai/
21. https://www.youtube.com/watch?v=R6BSme-9Zr8&t=1s
22. https://ojs.aaai.org/index.php/AAAI/article/view/30296
23. https://www.producthunt.com/stories/what-is-conversational-ai-how-it-works
24. https://www.gartner.com/en/newsroom/press-releases/2022-08-31-gartner-predicts-conversational-ai-will-reduce-contac
25. https://tech-stack.com/blog/future-of-chatbots/
26. https://www.sprinklr.com/products/customer-service/voice-bots/
27. https://www.ycombinator.com/companies/simple-ai
28. https://news.bloomberglaw.com/private-equity/openai-nears-launch-of-ai-agent-tool-to-automate-tasks-for-users
29. https://www.theverge.com/2024/11/13/24295879/openai-agent-operator-autonomous-aiv
30. https://www.microsoft.com/en-us/research/blog/research-at-microsoft-2023-a-year-of-groundbreaking-ai-advances-and-di
scoveries/
31. https://arxiv.org/html/2407.21075v1
32. https://deepmind.google/technologies/gemini/nano/
33. https://www.reddit.com/r/perplexity_ai/comments/1gxa6m4/did_you_know_perplexity_is_diving_into_shopping/
34. https://www.zdnet.com/article/is-perplexity-pro-worth-the-subscription-this-free-shipping-perk-just-might-convince-me/
35. https://www.theverge.com/2024/11/18/24299574/perplexity-ai-search-engine-buy-products
36. Amazon Wrapped Sample Visualizations
37. https://cmu.yul1.qualtrics.com/reports/RC/public/Y211LTY3MWMxZjhlNmNhZjAzMDAwOGU0YjM0ZC1VUl8wb0VDRjJDVUd
PZVRFUDQ=
38. https://public.tableau.com/app/profile/kshitij.dhara/viz/AmazonSurveyDashboard/Q2_Viz?publish=yes
39. https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
40. https://pair.withgoogle.com/chapter/explainability-trust/
41. ​https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-id
USKCN1MK0AG/
42. https://www.citizen.org/article/chatbots-are-not-people-dangerous-human-like-anthropomorphic-ai-report/
43. AI Expert Interviews | Insights & Data
44. https://research.google/blog/reducing-the-computational-cost-of-deep-reinforcement-learning-research/

Privileged & Confidential Page 12

AI-Enabled Post-Purchase Customer Experience December, 2024

Privileged & Confidential Page 13