...
Experiment Name:
Auto-Ship Copy Enhancement (include Subscribe to Save globally for Yankee) [New.22.012] Â
Hypothesis: Â
By updating the Auto-Ship language, the number of Auto-Ship toggle clicks and transactions with Auto-Ship will increase.
...
Results & Recs:
Results: after running for 22 days variant v1 (Subscribe to Save) shows stat sig increase in purchases.
Engagement:
V1 results do not show measurable impacts for users opting into Auto-Ship.
Radio button clicks on PDP were down (nearly at stat sig).
PLP Toggles showed indications that users interacted with the PLP toggles more, but this toggle was largely ignored in both the Control & V1 variants. Â
Conversions:
Despite V1’s lack of engagement, there was a 0.85% lift in Transaction Rate (with 90% stat sig.). Pinpointing the reason why users were more likely to convert under V1 is unclear.
It does appear that v1 helped to decrease user uncertainty and increased confidence to purchase.
Note: During the course of this test, the copy below PLP Auto-Ship filter toggle was hidden behind a tool tip. Although not likely to color results greatly, this is an unintended variable.
Recommendation (CXperts):
Recommendation: Conclude test. With the macro goals showing improvement but with no impact on Auto-Ship uptake, implementation of v1 should be discussed within the Yankee biz team.
...
By The Numbers: data from ABTasty
Subscribe to Save Radio Clicks
...
Auto-Ship Nav Image Clicks
...
Find Out More Nav Clicks
...
Filter Learn More Clicks
...
Filter Toggle Clicks
...
Add to Bag ClicksÂ
...
Transaction Rate
...
...
Experiment Design
Findings:
FullStory & UserTesting research shows that users are confused by term/word “Auto-Ship”
...
Close other tabs that are the main site domain
Click the link above, which will take you to the test preview landing page (so not in private browser to allow cookies)
Click the test preview link and verify you are getting the variation
Verify on desktop and mobile breakpoints (collapsed desktop browser is fine)
Results:
( pending )
Recommendation (CXperts):
( pending )
By The Numbers: data from ABTasty
...
)
...