Winner ⭐️   

[22.005]

PLP Filter Order (Yankee)  

Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions.

↑ PDP Visits
↑ Add to Cart
↑ Conversions  

Test Timing:
26-Jul to 18-Aug  

Results: This test ran for 22 days, and v1-Filter-Reorder is a Win! Filter Interactions: v1 Filter Interactions had a stat sig uplift, increasing by ~15% with 100% stat sig. Although Filter Toggle Interactions may have decreased (not stat sig), all other filters received uplift of at least 6%, most notably Fragrance Type & Color which increased by ~20% with 100% stat sig.

RECs & Next Steps: Declare v1 a winner and implement; continue to iterate on tests around filter treatments and locations, specifically the toggle filters.

For more details, visit this test’s wiki page.  

Winner ⭐️  

[22.004]

PLP & HP Add-to-Bag CTA Color (Yankee)  

Hypothesis: By changing from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions.

↑ [Add-to-Cart] Clicks
↑ Conversions  

Test Timing:
22-Jul to 5-Aug  

Results: Green wins! The v1-Green [Add-to-Bag] CTA variant outperformed v2-Blue variant and the control. Stat Sig of 90% was achieved showing a positive impact to Order Confirmation pages by ~1.5% uplift compared to the control. This is a direct result of the color change, which increased clicks to the [Add-to-Bag] CTA by 2.5% at 99% stat sig.

RECs & Next Steps: Updating the [Add-to-Bag] CTA color to Green for PLP & HP is under discussion with Product/UX.

For more details, visit this test’s wiki page.

Learner   

[22.006]

Cart Page Minimized NAV Bar (Yankee)   

Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting. 

↑ [Cont-to-Checkout] CTA
↑ Transaction   

Test Timing:
20-Jul to 5-Aug 

Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates.

RECs & Next Steps: Checkout to utilize a minimized nav is under consideration for Odyssey.

For more details, visit this test’s wiki page.  

Learner   

[22.003]

Checkout Guest CTA (Yankee)

Hypothesis: By changing the color of the primary Sign-In CTA for guests in the checkout flow, we will increase conversions via Guest checkout.

↑ [Cont-as-Guest] Clicks
↑ Conversions   

Test Timing:
22-Jul to 15-Aug  

Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views).

(No immediate follow up for Odyssey—test was inconclusive)

Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points.
 
(This test is listed in our experimentation IDEAS area and is under consideration for future testing).

For more details, visit this test’s wiki page.   

Learner   

[22.002]

Global Bag to Cart (Yankee)   

Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion.

↑ Conversions
↑ Engagement
↑ Cart Clicks  

Test Timing:
28-Jul to 11-Aug  

Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users.

(No immediate follow up for Odyssey—test was inconclusive)

Next Steps: Consider prioritizing tests around Mini-Cart functionality to validate learnings of this test and our new/revised hypothesis;

• … mixed results may be a result of users intending to engage the mini-cart (available with add-to-cart clicks), thus the increase in clicks to the Cart icon, but were potentially frustrated in that lack of behavior.

For more details, visit this test’s wiki page.

Learner   

[22.001]

PDP Add-to-Bag CTA Color Swap (Yankee) 

Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions.

↑ [Add-to-Cart] Clicks
↑ Conversions
↑ Conversion Funnel (cart + checkout visits)  

Test Timing:
27-Jun to 1-Aug

Results: Original CTA color outperformed both variants with 98% stat sig for [Add-to-Bag] CTA clicks.

(No immediate follow up for Odyssey—test was inconclusive)

RECs & Next Steps:
[x] Merged related learning regarding ATC [Add-to-Cart] CTA Color with Winning test 004. Learnings merged into voc ticket:

For more details, visit this test’s wiki page.