About: This wiki page captures the latest AB Experimentation learnings and captures what we have logged via VOC to ensure Productization occurs as we conclude and learn from our AB Tests.
Recently Completed AB Experiments
& Next Steps logged in VOC
Experiment | Experiment Name | Test Details | Results & Experiment | Next Steps & Action Items: |
|---|---|---|---|---|
Winner ⭐️ | [22.005] PLP Filter Order (Yankee) Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions. | ↑ PDP Visits Test Timing: | Results: This test ran for 22 days, and v1-Filter-Reorder is a Win! Filter Interactions: v1 Filter Interactions had a stat sig uplift, increasing by ~15% with 100% stat sig. Although Filter Toggle Interactions may have decreased (not stat sig), all other filters received uplift of at least 6%, most notably Fragrance Type & Color which increased by ~20% with 100% stat sig. | ODYI-1504 - Getting issue details... STATUS RECs & Next Steps: Declare v1 a winner and implement; continue to iterate on tests around filter treatments and locations, specifically the toggle filters. |
Winner ⭐️ | [22.004] PLP & HP Add-to-Bag CTA Color (Yankee) Hypothesis: By changing from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions. | ↑ [Add-to-Cart] Clicks Test Timing:
| Results: Green wins! The v1-Green [Add-to-Bag] CTA variant outperformed v2-Blue variant and the control. Stat Sig of 90% was achieved showing a positive impact to Order Confirmation pages by ~1.5% uplift compared to the control. This is a direct result of the color change, which increased clicks to the [Add-to-Bag] CTA by 2.5% at 99% stat sig. | ODYI-1503 - Getting issue details... STATUS RECs & Next Steps: Updating the [Add-to-Bag] CTA color to Green for PLP & HP is under discussion with Product/UX. |
Learner | [22.006] Cart Page Minimized NAV Bar (Yankee) Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting. | ↑ [Cont-to-Checkout] CTA Test Timing: | Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates. | ODYI-1505 - Getting issue details... STATUS RECs & Next Steps: Checkout to utilize a minimized nav is under consideration for Odyssey. |
Learner | [22.003] Checkout Guest CTA (Yankee) | ↑ [Cont-as-Guest] Clicks Test Timing: | Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views). | (No immediate follow up for Odyssey—test was inconclusive) Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points. (This test is listed in our experimentation IDEAS area and is under consideration for future testing). |
Learner | [22.002] Global Bag to Cart (Yankee) Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion. | ↑ Conversions Test Timing: | Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users. | (No immediate follow up for Odyssey—test was inconclusive) Next Steps: Consider prioritizing tests around Mini-Cart functionality to validate learnings of this test and our new/revised hypothesis;
|
Learner | [22.001] PDP Add-to-Bag CTA Color Swap (Yankee) Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions. | ↑ [Add-to-Cart] Clicks Test Timing: | Results: Original CTA color outperformed both variants with 98% stat sig for [Add-to-Bag] CTA clicks. | (No immediate follow up for Odyssey—test was inconclusive) RECs & Next Steps: |
In-Flight + Planned Experiments
Experiment | Experiment Name | Test | Test Details |
|---|---|---|---|
In-Flight | [22.007] PLP & HP Quantity Selector removal (Yankee) Hypothesis: By removing the Quantity Selector on the PLP, we will reduce user friction and hesitation on the PLP, increasing PDP visits, add to bag clicks, and conversions. | Aug Launched | ↑ PDP Tile Clicks For more details, visit this test’s wiki page.
|
In-Flight | [22.009] PLP Star-Rating # Reviews presentation (Yankee) Hypothesis: By adding the number of reviews underneath the stars on products within the PLP, we will illustrate product popularity accurately, decreasing user hesitations, increasing “Add to Bag” CTA clicks and conversions. | Aug Launched | ↑ Add to Bag CTA Clicks |
In-Flight | [22.008] Checkout UI simplification (padlock icon removal) Hypothesis: By updating or removing the padlock icon we will decrease confusion within checkout, increasing conversions. | Aug Launched | ↑ Conversions |
Coming Soon | [ODYI-1513] Checkout | Skip Login Screen by relocating login option to appear within Shipping Step Hypothesis: Removing the separate Login Screen (within the Checkout flow) will create an overall easier shopping-to-checkout experience for our customers and improve conversion rates. | Sept Launch planned | ↑ Conversions |
Coming Soon | [ODYI-1510] Mini Cart | add Checkout CTA Button Hypothesis: Giving users the option to bypass the (full) Cart will help conversion rates and create an easier shopping-to-checkout experience. | Sept Launch planned | ↑ Conversions |
Coming Soon | [22.0xx] Global NAV | Bag Icon functionality update (expand Mini-Cart instead of current switching pages to full cart) Hypothesis: Making the bag icon open a mini cart will allow users to view their cart without leaving product pages which will increase the average number of items users add to their cart before checkout. Findings: Once a user adds a product to bag they are presented with a mini cart; but there is no other way to produce said mini cart later. | Sept Launch planned | ↑ Average Order Revenue |
Coming Soon | [22.011] PDP Return & Exchange Copy Hypothesis: By adding copy about returns and exchanges in the PDP product detail section, we can boost user confidence and increase conversions. | Sept Launch planned | ↑ Add to Cart Clicks |
Coming Soon | [22.012] Global | Auto-Ship Messaging Hypothesis: By updating the auto-ship language, we will increase the number of auto-ship toggle clicks and transactions with auto-ship. | Sept Launch planned | ↑ Auto-Ship Toggle Click |
Coming Soon | [22.010] Checkout | Shipping Address Changes Message Hypothesis: By adding Shipping Address Changes messaging to checkout, we will reduce friction in the checkout flow, increasing continuation and conversions. | Sept | ↑ Conversions |
Coming Soon | (Your Experiment Here!) | The future | See details below and submit your hypothesis today! |
Do you have an idea for an experiment?
Submit your experiment idea & hypothesis here.
We hope you found this update to be informative and helpful in understanding where we’re headed as we develop our experimentation program. We’re actively working to extend our testing capabilities to include Marmot in the coming weeks and will be connecting with a few of you shortly. Again, a big thank you to everyone for their interest and support!
Please reach out with any questions via email or slack—we’re happy to chat.
-Thanks, Alex Alesio (Deactivated)
Add Comment