temp update sept7
- Alex Alesio (Deactivated)
About: This wiki page captures the latest AB Experimentation learnings and captures what we have logged via VOC to ensure Productization occurs as we conclude and learn from our AB Tests.
Recently Completed AB Experiments & Next Steps
Winner ⭐️ Quantity Selector Removal on ProdTiles [22.007] [ODYI-1529]
Results (the verdict): After 20 days of test runtime, v1(Quantity Selector Removed from Product Tiles) is a Winner!
RECs & Next Steps: Action items logged to remove qty selector from product tiles on Yankee: https://newellbrands.atlassian.net/browse/ODYI-1560
Additional Info: Experiment Wiki Page with test summary & results details
Learner | PLP Star-Rating # Reviews presentation [22.009] [ODYI-1530]
Results: Test ran for 14 days and the results are inconclusive. No stat sig impact on order confirmation page visits or other checkout page visits between either variation. No differences between mobile or desktop users.
No next steps required—learner experiment.
Recommendation; conclude the test and move on to more fruitful tests. This is a good reminder that users may say one thing but do another, in that reviews are important to them, but maybe not on the PLP.Additional Info: Experiment Wiki Page with test summary & results details
Winner ⭐️ PLP Filter Order [22.005] [ODYI-1535]
Results: the test ran for 22 days and v1 (Filters Reordered) is a Winner!
RECs & Next Steps: Recommendation: Declare v1 a winner and implement via ODCD-14118: https://newellbrands.atlassian.net/browse/ODYI-1504
Additional Info: Experiment Wiki Page with test summary & results details
Winner ⭐️ Add-to-Bag CTA Color update across Yankee [22.004] [ODYI-1534]
Results: Green wins!
RECs & Next Steps: Update Yankee per winning variant. Modify [Add-to-Bag] CTA Color to Green via ODCD-14119: https://newellbrands.atlassian.net/browse/ODYI-1503
Additional Info: Experiment Wiki Page with test summary & results details
Learner | Cart Page Minimized NAV Bar [22.006] [ODYI-1536]
Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates.
RECs & Next Steps: Recommendation: Update Checkout to utilize a minimized NAV. Action item logged via ODCD-14117: https://newellbrands.atlassian.net/browse/ODYI-1505
Additional Info: Experiment Wiki Page with test summary & results details
old
Experiment | Experiment Name | Test Details | Results & Experiment | Next Steps & Action Items: |
|---|---|---|---|---|
| Winner ⭐️ Quantity Selector Removal on ProdTiles [22.007] [ODYI-1529]
|
|
|
|
| Learner | PLP Star-Rating # Reviews presentation [22.009] [ODYI-1530]
|
|
|
|
Winner ⭐️ | Winner ⭐️ PLP Filter Order (Yankee) |
|
|
|
| Winner ⭐️ Add-to-Bag CTA Color update across Yankee [22.004] [ODYI-1534]
|
|
|
|
Learner | Learner | Cart Page Minimized NAV Bar [22.006] [ODYI-1536]
|
|
|
|
Learner | Checkout Guest CTA (Yankee) Hypothesis: By changing the color of the primary Sign-In CTA for guests in the checkout flow, we will increase conversions via Guest checkout. | ↑ [Cont-as-Guest] Clicks Test Timing:
| Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views). | (No immediate follow up—test results were mixed) Potential next steps: Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points. (This test is listed in our experimentation IDEAS area and is under consideration for future testing). |
Learner | Global Bag to Cart (Yankee) Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion. | ↑ Conversions Test Timing:
| Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users. | (No immediate follow up—test results were mixed) Potential next steps: Consider prioritizing tests around Mini-Cart functionality to validate learnings of this test and revised hypothesis: … mixed results may be a result of users intending to engage the mini-cart (available with add-to-cart clicks), thus the increase in clicks to the Cart icon, but were potentially frustrated in that lack of behavior. |
Learner | PDP Add-to-Bag CTA Color Swap (Yankee) Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions. | ↑ [Add-to-Cart] Clicks Test Timing: | Results: Original CTA color outperformed both variants with 98% stat sig for [Add-to-Bag] CTA clicks. | (No immediate follow up.) CTA color updates from this test were merged into separate ticket. RECs & Next Steps: Merged related learning regarding ATC [Add-to-Cart] CTA Color with Winning test 004. Learnings merged into voc ticket: https://newellbrands.atlassian.net/browse/ODYI-1503
|
In-Flight & Upcoming Experiments
Experiment | Experiment Name | Test | Test Details |
|---|---|---|---|
In-Flight | Checkout UI simplification (padlock icon removal) Hypothesis: By updating or removing the padlock icon we will decrease confusion within checkout, increasing conversions. | Aug Launched | ↑ Conversions |
In-Flight | (updating in progress) | (updating in progress) | (updating in progress) |
Coming Soon | Checkout | Skip Login Hypothesis: By removing the Login step in checkout, we will create an easier checkout exp. thus increasing conversions. | September | More Info here: |
Coming Soon | PDP Return & Exchange Copy Hypothesis: By adding copy about returns and exchanges in the PDP product detail section, we can boost user confidence and increase conversions.
| September | More Info here: |
Coming Soon | Global | Auto-Ship Messaging Hypothesis: By updating the auto-ship language, we will increase the number of auto-ship toggle clicks and transactions with auto-ship.
| September | More Info here: |
|
|
|
|
Coming Soon | Global Nav | Bag Icon Functionality Hypothesis: Making the bag icon open a mini cart will allow users to view their cart without leaving product pages which will increase the average number of items users add to their cart before checkout. | October | More Info here: |
Coming Soon | PDP | Product Style Options Hypothesis: By focusing recommended Styles to only show specific types of products, this will minimize the number of options, decrease users being overwhelmed, and thus increasing site conversions. | October | More info here: |
Coming Soon | PLP | Auto Ship Toggle Location Hypothesis: By moving the Auto-Ship toggle next to the sort drop down, we will increase toggle interaction and increase purchases with subscriptions. | October | More info here: |
Backlog | Mini Cart | add Checkout CTA Button Hypothesis: Giving users the option to bypass the (full) Cart will help conversion rates and create an easier shopping-to-checkout experience. | backlog | More Info here: |
Backlog | Checkout | Shipping Address Changes Message Hypothesis: By adding Shipping Address Changes messaging to checkout, we will reduce friction in the checkout flow, increasing continuation and conversions. | backlog | More Info here: |
Coming Soon | (Your Experiment Here!) | The Future | See details below and submit your hypothesis today! |
Do you have an idea for an experiment?
Submit your experiment idea & hypothesis here.
We hope you found this update to be informative and helpful in understanding where we’re headed as we develop our experimentation program. We’re actively working to extend our testing capabilities to include Marmot in the coming weeks and will be connecting with a few of you shortly. Again, a big thank you to everyone for their interest and support!
Please reach out with any questions via email or slack—we’re happy to chat.
-Thanks, @Alex Alesio (Deactivated)