Yankee Global Auto-Ship Messaging 📝 Learner (ABT-112 cxp.012)

This wiki page is designed to summarize and capture our AB experiment goals, screen shot previews, and results after a test concludes.

Experiment Name:
Auto-Ship Copy Enhancement (include Subscribe to Save globally for Yankee) [New.22.012]

Hypothesis:

By updating the Auto-Ship language, the number of Auto-Ship toggle clicks and transactions with Auto-Ship will increase.

Results & Recs:

Results: after running for 22 days variant v1 (Subscribe to Save) shows stat sig increase in purchases.
- Engagement:
  - V1 results do not show measurable impacts for users opting into Auto-Ship.
  - Radio button clicks on PDP were down (nearly at stat sig).
  - PLP Toggles showed indications that users interacted with the PLP toggles more, but this toggle was largely ignored in both the Control & V1 variants.
Conversions:
- Despite V1’s lack of engagement, there was a 0.85% lift in Transaction Rate (with 90% stat sig.). Pinpointing the reason why users were more likely to convert under V1 is unclear.
- It does appear that v1 helped to decrease user uncertainty and increased confidence to purchase.
Note: During the course of this test, the copy below PLP Auto-Ship filter toggle was hidden behind a tool tip. Although not likely to color results greatly, this is an unintended variable.

Recommendation (CXperts):

Recommendation: Conclude test. With the macro goals showing improvement but with no impact on Auto-Ship uptake, implementation of v1 should be discussed within the Yankee biz team.

By The Numbers: data from ABTasty

Subscribe to Save Radio Clicks

Auto-Ship Nav Image Clicks

Find Out More Nav Clicks

Filter Learn More Clicks

Filter Toggle Clicks

Add to Bag Clicks

Transaction Rate

Experiment Design

Findings:

FullStory & UserTesting research shows that users are confused by term/word “Auto-Ship”

Goals:

↑ Auto-Ship Toggle Click
↑ Transactions with Auto-Ship

Target:

Yankee DTC website ( YankeeCandle.com )
All Pages / All Users
PLPs: Original Jar Candles
PDPs: Large Jar Candles
Global Navigation
About Auto-Ship page
Cart

AB Experiment Previews:
Screen Shots, Video Capture, & Preview links

Video Capture Preview:

See below test preview video with voiceover to understand the experiment’s design and areas affected by the copy change.

_2022-09-23 New.22.012 Global Auto-Ship Messaging AB Test.mp4

Note: This screen capture video is helpful in providing a quick preview to understand the nature and purpose of the test, but may not be as up-to-date as the below preview links.

Screenshot Preview:

Test Variation(s): V1 (Subscribe to Save)

Global Nav
PLPs
PDPs
Auto-Ship Page (Sub-headline)
Cart Page

AB Tasty Preview Links: