AB Update & Recap Summary email 2022-08-19

Learner   

20-Jul to 5-Aug 

[22.006]

Cart Page Minimized NAV Bar (Yankee)   

Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting. 

↑ [Cont-to-Checkout] CTA
↑ Transaction   

• Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates.

• RECs & Next Steps: Checkout to utilize a minimized nav is under consideration for Odyssey.

• For more details, visit this test’s wiki page.  

Winner ⭐️   

26-Jul to 18-Aug  

[22.005]

PLP Filter Order (Yankee)  

Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions.

↑ PDP Visits
↑ Add to Cart
↑ Conversions  

• Results: This test ran for 22 days, and v1-Filter-Reorder is a Win! Filter Interactions: v1 Filter Interactions had a stat sig uplift, increasing by ~15% with 100% stat sig. Although Filter Toggle Interactions may have decreased (not stat sig), all other filters received uplift of at least 6%, most notably Fragrance Type & Color which increased by ~20% with 100% stat sig.

• RECs & Next Steps: Declare v1 a winner and implement; continue to iterate on tests around filter treatments and locations, specifically the toggle filters.
  

• For more details, visit this test’s wiki page.  

Winner ⭐️  

22-Jul to 5-Aug  

[22.004]

PLP & HP Add-to-Bag CTA Color (Yankee)  

Hypothesis: By changing from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions.

↑ [Add-to-Cart] Clicks
↑ Conversions  

• Results: Green wins! The v1-Green [Add-to-Bag] CTA variant outperformed v2-Blue variant and the control. Stat Sig of 90% was achieved showing a positive impact to Order Confirmation pages by ~1.5% uplift compared to the control. This is a direct result of the color change, which increased clicks to the [Add-to-Bag] CTA by 2.5% at 99% stat sig.

• RECs & Next Steps: Updating the [Add-to-Bag] CTA color to Green for PLP & HP is under discussion with Product/UX.

• For more details, visit this test’s wiki page.

Learner   

22-Jul to 15-Aug  

[22.003]

Checkout Guest CTA (Yankee)

Hypothesis: By changing the color of the primary Sign-In CTA for guests in the checkout flow, we will increase conversions via Guest checkout.

↑ [Cont-as-Guest] Clicks
↑ Conversions   

• Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views).

• RECs & Next Steps:
 End the test. Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points.
 
(This test is listed in our experimentation IDEAS area and is under consideration for future testing).

• For more details, visit this test’s wiki page.   

Learner   

28-Jul to 11-Aug  

[22.002]

Global Bag to Cart (Yankee)   

Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion.

↑ Conversions
↑ Engagement
↑ Cart Clicks  

• Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users.

• RECs: End the test and consider experimenting with mini-cart per the learnings of this test. CXperts’ revised hypothesis; mixed results of this test may be a result of users intending to engage the mini-cart (available with add-to-cart clicks), thus the increase in clicks to the Cart icon, but were potentially frustrated in that lack of behavior.

• Next Steps: Consider prioritizing tests around Mini-Cart functionality to validate this hypothesis.

• For more details, visit this test’s wiki page.

Learner   

27-Jun to 1-Aug

[22.001]

PDP Add-to-Bag CTA Color Swap (Yankee) 

Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions.

↑ [Add-to-Cart] Clicks
↑ Conversions
↑ Conversion Funnel (cart + checkout visits)  

• Results: Original CTA color outperformed both variants with 98% stat sig for [Add-to-Bag] CTA clicks.

• RECs & Next Steps: CTA Color changes under consideration per Test004’s results which revealed Green Add-To-Cart CTAs to outperform both variants and the control.

• For more details, visit this test’s wiki page.  

In-Flight  

Aug Launched  

[22.007]  

PLP & HP Quantity Selector removal (Yankee)   

Hypothesis: By removing the Quantity Selector on the PLP, we will reduce user friction and hesitation on the PLP, increasing PDP visits, add to bag clicks, and conversions.

↑ PDP Tile Clicks
↑ Conversions  

For more details, visit this test’s wiki page.   

In-Flight  

Aug Launched  

[22.009]  

PLP Star-Rating # Reviews presentation (Yankee)   

Hypothesis: By adding the number of reviews underneath the stars on products within the PLP, we will illustrate product popularity accurately, decreasing user hesitations, increasing “Add to Bag” CTA clicks and conversions.

↑ Add to Bag CTA Clicks
↑ Conversions  

For more details, visit this test’s wiki page.

Coming Soon  

Aug Launch planned  

[22.008]

Checkout UI simplification (padlock icon removal)   

Hypothesis: By updating or removing the padlock icon we will decrease confusion within checkout, increasing conversions.

↑ Conversions
↑ Engagement

Coming Soon  

Sept Launch planned  

[22.010]  

Global NAV | Bag Icon Functionality (mini-cart toggle)

↑ Conversions
↑ Engagement

Coming Soon  

Sept Launch planned  

[22.011]  

PDP Return & Exchange Copy   

↑ Conversions
↑ Engagement

Coming Soon  

Sept Launch planned  

[22.012]  

Global | Auto-Ship Messaging   

↑ Conversions
↑ Engagement

Coming Soon  

The future  

(Your Experiment Here!) 

See details below and submit your hypothesis today!