Yankee Global Auto-Ship Messaging 📝 Learner (ABT-112 cxp.012)

This wiki page is designed to summarize and capture our AB experiment goals, screen shot previews, and results after a test concludes.

Experiment Name:
Auto-Ship Copy Enhancement (include Subscribe to Save globally for Yankee) [New.22.012]  

Hypothesis:  

By updating the Auto-Ship language, the number of Auto-Ship toggle clicks and transactions with Auto-Ship will increase.

Results & Recs:

• Results: after running for 22 days variant v1 (Subscribe to Save) shows stat sig increase in purchases.

  • Engagement:

    • V1 results do not show measurable impacts for users opting into Auto-Ship.

    • Radio button clicks on PDP were down (nearly at stat sig).

    • PLP Toggles showed indications that users interacted with the PLP toggles more, but this toggle was largely ignored in both the Control & V1 variants.  

       

• Conversions:

  • Despite V1’s lack of engagement, there was a 0.85% lift in Transaction Rate (with 90% stat sig.). Pinpointing the reason why users were more likely to convert under V1 is unclear.

  • It does appear that v1 helped to decrease user uncertainty and increased confidence to purchase.

• Note: During the course of this test, the copy below PLP Auto-Ship filter toggle was hidden behind a tool tip. Although not likely to color results greatly, this is an unintended variable.

Recommendation (CXperts):

• Recommendation: Conclude test. With the macro goals showing improvement but with no impact on Auto-Ship uptake, implementation of v1 should be discussed within the Yankee biz team.

By The Numbers: data from ABTasty

Subscribe to Save Radio Clicks

Open

Auto-Ship Nav Image Clicks

Open

Find Out More Nav Clicks

Filter Learn More Clicks

Filter Toggle Clicks

Add to Bag Clicks 

Transaction Rate

Experiment Design

Findings:

• FullStory & UserTesting research shows that users are confused by term/word “Auto-Ship”

Goals:

• ↑ Auto-Ship Toggle Click

• ↑ Transactions with Auto-Ship

Target:  

• Yankee DTC website ( YankeeCandle.com )

• All Pages / All Users
  
  

• PLPs: Original Jar Candles

• PDPs: Large Jar Candles

• Global Navigation

• About Auto-Ship page

• Cart

AB Experiment Previews:
Screen Shots, Video Capture, & Preview links

Video Capture Preview:

See below test preview video with voiceover to understand the experiment’s design and areas affected by the copy change.

• Note: This screen capture video is helpful in providing a quick preview to understand the nature and purpose of the test, but may not be as up-to-date as the below preview links.
  
  

Screenshot Preview: 

Test Variation(s): V1 (Subscribe to Save)

1. Global Nav

    

    

2. PLPs

    

    

3. PDPs

4. Auto-Ship Page (Sub-headline)

    

5. Cart Page

    

AB Tasty Preview Links: 

Test Variation(s): V1 (Subscribe to Save)

1. Global Nav
   https://protect-us.mimecast.com/s/TS60CW61oWhjlEwVEfx0F0U?domain=app.abtasty.com

2. PLPs
   https://protect-us.mimecast.com/s/KLyiCXDPpYunpQ7WQuDpoAP?domain=app.abtasty.com

3. PDPs
   https://protect-us.mimecast.com/s/Ri0LCYEZqVc3G7ow7sVydYe?domain=app.abtasty.com

4. Autoship Page Subheadline
   https://protect-us.mimecast.com/s/WFSDCZ6grZhMO6n06hxoHje?domain=app.abtasty.com

5. Cart Page
   https://protect-us.mimecast.com/s/NgCsC1wM3VHpv4wl4I1x-FD?domain=app.abtasty.com

Preview Instructions:

1. Close other tabs that are the main site domain

2. Click the link above, which will take you to the test preview landing page (so not in private browser to allow cookies)

3. Click the test preview link and verify you are getting the variation

4. Verify on desktop and mobile breakpoints (collapsed desktop browser is fine)

Appendix: AB Test Variants & original CXPerts slides

AB Test Variant: v0 control

AB Test Variant: v1 (relocated to top of PLP)