About: This wiki page captures the latest AB Experimentaiton updates sent via email on 2022-Aug-19.
Subject: AB Testing & Experimentation Update
Sent: Friday, August 19, 2022 8:00 am
From: Alesio, Alexander <Alexander.Alesio@Newellco.com>
Hello Team,
We're excited to share our latest AB experimentation updates as we continue to test and learn on Yankee’s website. So far, we’ve made solid progress with 8 tests launched to date, 6 of which have completed and provided results which I have summarized below. Also, a big thank you to our many eComm partners who have made this possible!
Recently Completed AB Experiments
...
Experiment
Status
...
Test
Timing
...
Experiment Name
& Hypothesis
...
Metrics
(Primary, Secondary, & Tertiary)
...
Results &
Next Steps
...
Jira ticket(s) logged
...
Learner
...
20-Jul to 5-Aug
...
[22.006]
Cart Page Minimized NAV Bar (Yankee)
Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting.
...
↑ [Cont-to-Checkout] CTA
↑ Transaction
...
Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates.
...
Experimentation learnings and captures what we have logged via VOC to ensure Productization occurs as we conclude and learn from our AB Tests.
Recently Completed AB Experiments
& Next Steps logged in VOC
Experiment | Experiment Name | Test Details | Results & Experiment | Next Steps & Action Items: | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Winner ⭐️ | [22.005] PLP Filter Order (Yankee) Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions. | ↑ PDP Visits Test Timing: | Results: This test ran for 22 days, and v1-Filter-Reorder is a Win! Filter Interactions: v1 Filter Interactions had a stat sig uplift, increasing by ~15% with 100% stat sig. Although Filter Toggle Interactions may have decreased (not stat sig), all other filters received uplift of at least 6%, most notably Fragrance Type & Color which increased by ~20% with 100% stat sig. |
RECs & Next Steps: Declare v1 a winner and implement; continue to iterate on tests around filter treatments and locations, specifically the toggle filters. | | ||||||||||||||
Winner ⭐️ | 26-Jul to 18-Aug ⭐️ | [22.005004] PLP Filter Order & HP Add-to-Bag CTA Color (Yankee) Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions. | ↑ PDP Visits |
| Winner ⭐️ | [22.004] PLP & HP Add-to-Bag CTA Color (Yankee) Hypothesis: By changing from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions. | ↑ [Add-to-Cart] Clicks | Results: Green wins! The v1-Green [Add-to-Bag] CTA variant outperformed v2-Blue variant and the control. Stat Sig of 90% was achieved showing a positive impact to Order Confirmation pages by ~1.5% uplift compared to the control. This is a direct result of the color change, which increased clicks to the [Add-to-Bag] CTA by 2.5% at 99% stat sig. from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions. | ↑ [Add-to-Cart] Clicks Test Timing:
| Results: Green wins! The v1-Green [Add-to-Bag] CTA variant outperformed v2-Blue variant and the control. Stat Sig of 90% was achieved showing a positive impact to Order Confirmation pages by ~1.5% uplift compared to the control. This is a direct result of the color change, which increased clicks to the [Add-to-Bag] CTA by 2.5% at 99% stat sig. |
RECs & Next Steps: Updating the [Add-to-Bag] CTA color to Green for PLP & HP is under discussion with Product/UX. | ||||||||
Learner | [22.006] Cart Page Minimized NAV Bar (Yankee) Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting. | ↑ [Cont-to-Checkout] CTA Test Timing: | Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates. |
RECs & Next Steps: Checkout to utilize a minimized nav is under consideration for Odyssey. | |||||||||||||||
Learner | 22-Jul to 15-Aug [22.003] Checkout Guest CTA (Yankee) | ↑ [Cont-as-Guest] Clicks Test Timing: | Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views). RECs & Next Steps: End the test.(Order Confirmation Page Views). | (No immediate follow up for Odyssey—test was inconclusive) Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points. (This test is listed in our experimentation IDEAS area and is under consideration for future testing). | |||||||||||||||
Learner | 28-Jul to 11-Aug [22.002] Global Bag to Cart (Yankee) Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion. | ↑ Conversions Test Timing: | Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users. RECs: End the test and consider experimenting with mini-cart per the | (No immediate follow up for Odyssey—test was inconclusive) Next Steps: Consider prioritizing tests around Mini-Cart functionality to validate learnings of this test | . CXperts’ and our new/revised hypothesis; | mixed results of this test
| . Next Steps: Consider prioritizing tests around Mini-Cart functionality to validate this hypothesis
| ||||||||||||
Learner | 27-Jun to 1-Aug | [22.001] PDP Add-to-Bag CTA Color Swap (Yankee) Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions. | ↑ [Add-to-Cart] Clicks Test Timing: | Results: Original CTA color outperformed both variants with 98% stat sig for [Add-to-Bag] CTA clicks. | (No immediate follow up for Odyssey—test was inconclusive) RECs & Next Steps | : CTA Color changes under consideration per Test004’s results which revealed Green Add-To-Cart CTAs to outperform both variants and the control. :
|
In-Flight + Planned Experiments
Experiment | Test | Experiment Name | Metrics |
|---|---|---|---|
In-Flight | Aug Launched | [22.007] PLP & HP Quantity Selector removal (Yankee) Hypothesis: By removing the Quantity Selector on the PLP, we will reduce user friction and hesitation on the PLP, increasing PDP visits, add to bag clicks, and conversions. | ↑ PDP Tile Clicks For more details, visit this test’s wiki page.
|
In-Flight | Aug Launched | [22.009] PLP Star-Rating # Reviews presentation (Yankee) Hypothesis: By adding the number of reviews underneath the stars on products within the PLP, we will illustrate product popularity accurately, decreasing user hesitations, increasing “Add to Bag” CTA clicks and conversions. | ↑ Add to Bag CTA Clicks |
Coming Soon | Aug Launch planned | [22.008] Checkout UI simplification (padlock icon removal) Hypothesis: By updating or removing the padlock icon we will decrease confusion within checkout, increasing conversions. | ↑ Conversions |
Coming Soon | Sept Launch planned | [22.010] Global NAV | Bag Icon Functionality (mini-cart toggle) | ↑ Conversions |
Coming Soon | Sept Launch planned | [22.011] PDP Return & Exchange Copy | ↑ Conversions |
Coming Soon | Sept Launch planned | [22.012] Global | Auto-Ship Messaging | ↑ Conversions |
Coming Soon | The future | (Your Experiment Here!) | See details below and submit your hypothesis today! |
Do you have an idea for an experiment?
Submit your experiment idea & hypothesis here.
...
-Thanks, Alex Alesio (Deactivated)
...
...
[x] Logged in VOC ( will clear out later)
ATC Add-to-Cart CTA Color update (HP PLP PDP) per AB experiment results
AB Test [22.004]: PLP & HP Add-to-Bag CTA Color (Yankee) Winner ⭐️
Hypothesis: By changing from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions.
...
Recommendations / Next Steps / Test Results
RECs & Next Steps:
Update the [Add-to-Bag] CTA color to Green on PLP & HP,
as (v1) variant was a clear winner.
Test Results:
Results: We have a ⭐️ Winner! (v1) Green Wins!
The v1-Green [Add-to-Bag] CTA variant outperformed v2-Blue variant and the control.
We hit stat sig of 90% that we are driving a positive impact to order confirmation pages by ~1.5% uplift compared to the control (refer to first image below).
This is a direct result of the color change, which increased clicks to the [Add-to-Bag] CTA by 2.5% at 99% stat sig (refer to second image below).
...
Test Details:
Metrics (Primary, Secondary, & Tertiary):
↑ [Add-to-Cart] Clicks
↑ Conversions
...
For more details, visit this test’s wiki page.
...
PLP Filters | update order of filters (per AB experiment results)
AB Test [22.005]: PLP Filter Order (Yankee) Winner ⭐️
Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions.
...
Recommendations / Next Steps / Test Results
RECs & Next Steps:
Implement (v1) Filter Order on PLP/Search for Yankee as v1 was declared a winner that outperformed current state.
Revised filter order shown below:
...
Test Results:
Results: This test ran for 22 days, and v1-Filter-Reorder is a Win! Filter Interactions: v1 Filter Interactions had a stat sig uplift, increasing by ~15% with 100% stat sig. Although Filter Toggle Interactions may have decreased (not stat sig), all other filters received uplift of at least 6%, most notably Fragrance Type & Color which increased by ~20% with 100% stat sig.
Optional (for future test consideration): Continue to iterate on tests around filter treatments and locations, specifically the toggle filters.
...
Test Details:
Metrics (Primary, Secondary, & Tertiary):
↑ PDP Visits
↑ Add to Cart
↑ Conversions
For more details, visit this test’s wiki page.
...
Cart Page | Implement Minimized NAV Bar (per AB experiment results)
AB Test [22.006]: Cart Page Minimized NAV Bar (Yankee)
Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting.
...
Recommendations / Next Steps / Test Results
RECs & Next Steps:
Update Checkout to utilize a minimized navigation as it will keep users focused within the checkout funnel, and should results in increased CVR and a reduced bounce rate.
See below screen shot & video for implementation guidelines
...
Video Capture
...
...
Test Results:
Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates.
...
Test Details:
Metrics (Primary, Secondary, & Tertiary):
↑ [Cont-to-Checkout] CTA
↑ Transaction
For more details, visit this test’s wiki page.
...
[x] test results / action merged
Note: Test 001 results & action items merged into Test-004 as results relate to the same [Add-to-Cart] CTA color testing.
AB Test [22.001]: PDP Add-to-Bag CTA Color Swap (Yankee)
Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions.
...
Recommendations / Next Steps / Test Results
RECs & Next Steps:
Update [add-to-cart] cta color on PDP to Green per Test004’s winning results and for consistency (if we update [add-to-cart] on PLP & HP, we should do it for PDP as well for UI consistency.
Test Results:
When testing [add-to-cart] on PLP/HP, Test004 revealed Green as the winning CTA color (outperforming both variants and the control).
...
Test Details:
Metrics (Primary, Secondary, & Tertiary):
↑ [Add-to-Cart] Clicks
↑ Conversions
↑ Conversion Funnel (cart + checkout visits)
For more details, visit this test’s wiki page.
...
[x] inconclusive / do not log / no action required
AB Test [22.003]: Checkout Guest CTA (Yankee) inconclusive
(Inconclusive test results—no action required for Odyssey)
Hypothesis: By changing the color of the primary Sign-In CTA for guests in the checkout flow, we will increase conversions via Guest checkout.
...
Recommendations / Next Steps / Test Results
RECs & Next Steps:
No ODY changes recommended, as this test was inconclusive.
Future ab testing steps to consider:
additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points. (This test is listed in our experimentation IDEAS area and is under consideration for future testing).
Test Results:
Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views).
...
Test Details:
Metrics (Primary, Secondary, & Tertiary):
↑ [Cont-as-Guest] Clicks
↑ Conversions
...
[22.002] Global Bag to Cart (Yankee)
(Inconclusive test results—no action required for Odyssey)
Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion.
...
Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users.
RECs: End the test and consider experimenting with mini-cart per the learnings of this test. CXperts’ revised hypothesis; mixed results of this test may be a result of users intending to engage the mini-cart (available with add-to-cart clicks), thus the increase in clicks to the Cart icon, but were potentially frustrated in that lack of behavior.
Next Steps: Consider prioritizing tests around Mini-Cart functionality to validate this hypothesis.
Metrics (Primary, Secondary, & Tertiary):
↑ Conversions
↑ Engagement
↑ Cart Clicks
...