2016 Outbrain + CNN chatbot
Goal: Understand real-world context around the usage of news delivery chatbots, to rapidly improve our early-stage product
Note: This experience continues to guide my approach to contemporary AI projects
Summary
- Proactively led UX research to shape a ground-breaking chatbot experience developed in partnership with CNN and hosted on Facebook Messenger
- Defined the real-world user problem through mixed-method research: survey; interviews; diary studies; and prototype testing, while guiding partners toward evidence-based decisions
- Challenged assumptions around user behavior and clarified key drivers of engagement, such as personalization, notifications, and task specificity
- Rapidly influenced design and copy, directly resulting in a 5x increase in onboarding and adoption after a single iteration
- Shifted communication strategy from tech novelty to user relevance
Key outcomes
500%
Fivefold increase to onboarding and adoption
The problem space
We had the infrastructure to build a chatbot, but we needed to make sure we were actually solving user needs, and solving them in the right way.
We assumed consumers would value communicating with a chatbot about news headlines, but we needed to know more about this problem space.
My role and my partners
I rapidly led research to refine the definition of the user problem, and refine the solution.
I teamed up with product stakeholders who had been leveraging the ChatFuel development platform.
Research
- I ran a survey to determine the popularity of comparitive experiences, including messenger apps and news chatbots
- I interviewed some of the survey participants, while observing their usage of news chatbots
- I led a diary study with a different set of the survey participants, who provided real-time details of their actual usage
- Later, I led asynchronous prototype research with folks from across the US
Usability testing sample (with audio)
Findings
- My research challenged some of the assumptions of my product, tech, and UX partners
- Right away, research informed critical changes to call-to-action and onboarding copy
- We were reminded that not everyone used the chat app; many would use the mobile browser
- Push notifications and personalization were critical for user engagement
- When people searched, it was almost always for a specific name, team, or event
- I communicated some of the findings as personas
Outcomes
- Research directly informed changes to copy and initial CTA treatment, leading to a 5x increase to onboarding following a single round of edits
- Contextual knowledge was key to our marketing communications around the feature
My takeaways
Looking back at this experience, I still find it remarkable that such rapid research helped reveal so much.
It may seem obvious now, but the hyper-specific nature of our users' inquiries, tied closely to the importance of personalization, highlighted the potential value of this chatbot unambiguously in the minds of our users.
This guided us away from communicating the feature as "the world at your fingertips" or some other internal or tech-focused messsaging. Instead, my research allowed us to tailor our onboarding messaging and CTAs, and based on the metrics, it clearly worked.
Despite this taking place in 2016, my experience provides direct guidance on an approach to contemporary AI interfaces.
User quote
It's pretty cool to have the news set up [this] way, figuring out what your likes are and providing stories that work for you.
Content Consumer