Hello Team—Before the Thanksgiving holiday kicks off, we wanted to provide a quick update regarding the latest AB Experimentation results for November's tests.
Summary of November’s AB Tests—
(1) Winner:
Yankee Checkout CC Expiration Text (MMYY)
(2) Tests Trending Positive:
Yankee Global NAV Carousel Stripe Banner
Yankee NAV L1 “Shop By” to “New Products”
(3) Learners:
Graco NAV Shop All redirect to Landing Page (vs Journey Page)
Yankee Auto-Ship Toggle Location (global change)
Yankee PDP Fragrance Notes
Results & Details—
Winner: Yankee Checkout CC Expiration Text
Hypothesis: By adding allowed helper text near credit card expiration, we will better convey to users allowed format, which will decrease friction and increase conversions.
Results: after 18 days, variant V1 (with MMYY Helper Text) is a Winner!
This minor edit goes to show that even small tweaks within checkout can have huge impacts!
Engagement: V1 saw a 0.75% improvement in users clicking on the Submit Order button with 96% stat sig.
Conversions: V1’s Transaction Rate improved 0.48% (with 94% stat sig) and was at parity with control for Mobile. This could be due to the fact that autocomplete on mobile phones is likely more prevalent than on Desktop browses.
Recommendation: End test and implement the winning V1 variant.
Trending Positive: Yankee Global Carousel Stripe
Hypothesis: By making the Global Carousel Stripe static with one consistent message or by removing it entirely, we will reduce friction, thus increasing carousel engagement and purchases.
Results: after 7 days of run time, both V1 (Static) & V2 (banner removed) are Winning.
Engagement: Control saw more clicks to the carousel with stat sig. V1 saw 22% fewer clicks and V2 saw virtually no clicks due to the carousel element being removed. PDP page views are seeing stat sig improvements under the two variations with a 1.16% lift for V1 and 0.75% increase for V2.
Conversions: Transaction rate is showing great improvements for V1 and V2. V1 shows a 1.52% lift in transaction rate at 95% stat sig and V2 is showing a 1.88% lift with 97% stat sig.
In just 7 days, this test has generated $25k in revenue above the control.
Recommendation: Early results are very promising, but it is still too early to tell. Sustain test for another week for more definitive results.
Trending Positive: Yankee NAV L1 “Shop By” to “New Products”
Hypothesis: By redirecting users from the “Shop By” section to the “New” landing page we will be resonating more with users, increasing engagement and conversions.
Results: after 13 days, V2 (“New Products”) is Winning for both L1 Click Engagement & Transaction Rate metrics.
Engagement: V2 generated the largest lift in L1 nav clicks with 92% more clicks on L1 at 100% stat sig. V1 ("New”) & V3 ("Shop New") also saw increases in L1 Clicks.
V1 saw a 39% increase and V2 41% increase (both stat sig). L2 NAV Clicks went down for all variations but with the increase in Clicks to L1 element, users are finding what they need through the variation without having to drill down to L2.
Conversions: V2 saw a 10% lift in transaction rates with 89% confidence, nearly stat sig. V1 and V3 saw stat sig improvements in AOV, whereas V2 showed improvement but without stat sig. However, when looking at potential revenue lift across the three variations, V2 shows the most potential to increase revenue. Within the 13 days that this test has run, V2 generated $38k in revenue uplift over the control.
Recommendation: implement V2 after running for another 1-2 days to verify test results.
Learner: Graco Global Shop All Nav
Hypothesis: By redirecting Car-Seat and Stroller navigation clicks to Shop-All-pages instead of Journey pages, we will allow users to start browsing and shopping earlier in the process, thus increasing PDP pageviews, Add to Carts, and Purchases.
Results: after running for 17 days, this test’s results are inconclusive.
Engagement: V1 (Shop All) Links pageviews increased 0.35% with 94% stat sig. Clicks on the Stroller-L1 & Car-Seats-L1 links showed some improvements but lacked stat sig.(which isn’t entirely surprising as there are no visible differences between V1 and the control that would change user behavior).
Conversions: Transactions Rates & AOV saw some uptick but lack stat sig.
Recommendation: End test and consider implementation based on preference.
Learner: Yankee Global Auto-Ship Toggle Location
Note: Final test results may change as this test was recently restarted—expect final results by the end of November.
Hypothesis: By moving the Auto-Ship toggle next to the sort drop down, we will increase toggle interaction and increase purchases with subscriptions.
Results: after 13 days of run time, the test results are inconclusive.
Engagement: Auto-ship toggle clicks in V1 (Toggle Top) were down 21% with stat sig. However, interestingly auto-ship radio clicks on PDP were up 15% with 91% stat sig. Potential explanation for this is that since the toggle is not in the sidebar filter, users who sought auto-ship and clicked the toggle had a higher intent to transact with auto-ship whereas users may hit the toggle in the sidebar filter regardless of intent.
Conversions: Transaction rate is down for V1 with control winning nearly at stat sig, but there is indication that AOV could increase under V1 with a 0.1% increase in Average Order Quantity (however, not at stat sig).
Recommendation: Conclude test and move on to more fruitful AB tests.
Learner: Yankee PDP Fragrance Notes
Hypothesis: By making the fragrance notes more visible, we can provide more information to the users, increase add to bag clicks and conversions.
Results: this ran for 14 days, and results are inconclusive.
Engagement: V1 (Expanded) saw a 1.26% increase in Add to Bag button clicks on PDP with 99% stat sig. However, progression through the checkout funnel saw an overall decrease. Cart page visits saw a 0.69% decrease with stat sig and subsequent funnel step page views saw a decrease.
Conversions: V1 transaction rates saw a 1.32% decline with stat sig. AOV saw a slight uptick but is essentially at parity with control. It is unknown why ATB clicks went up, but users did not follow through with their purchases. Filtering down to different user demographics such as device or new/returning visitors did not yield additional insights.
Recommendation: Conclude test and iterate on new test ideas around fragrance notes.
Coming Soon… December’s upcoming AB Tests
We’ve worked with Yankee, Graco, and our Tech teams to arrive at some low-risk tests as we close out this year’s testing in December.
Yankee Rewards in Guest Carts
Yankee Fragrance by Room
Graco Display of Applied Discounts
Graco PLP hide Retail Exclusives
...Visit our AB Experimentation Wiki for the full download and to learn more about our AB Experimentation program.
Special thanks to our AB Partners…
We’re already seeing some impressive numbers including a $429.17K uplift resulting from Yankee & Graco’s experiments which we kicked off in July of this year. As you can see your support goes a long way—keep those ideas coming!
Have a test idea? Submit your experiment hypothesis today @ Bit.ly/abRequest or Slack us!
We’re happy to discuss test ideas and address any questions you may have!
Have a great Thanksgiving everyone! 🦃
-Alex
//// previous update below
About: This page is designed to summarize our active AB experimentation tests, planned/roadmapped work, and the results & next steps coming out of each experiment to productionize our learnings.
Recently Completed Experiments & Next Steps
Status | Experiment Name | Test Goals | Results & Experiment | Next Steps & Action Items: |
|---|---|---|---|---|
Winner ⭐️ Test Duration: | Quantity Selector removal on HP PLP Hypothesis: By removing the Quantity Selector on the PLP, we will reduce user friction and hesitation on the PLP, increasing PDP visits, add to bag clicks, and conversions. | ↑ PDP Visits | Results (the verdict): After 20 days of test runtime, v1(Quantity Selector Removed from Product Tiles) is a Winner! | RECs & Next Steps
Additional Info: |
Learner | PLP Star-Rating # Reviews presentation Hypothesis: By adding the number of reviews underneath the stars on products within the PLP, we will illustrate product popularity accurately, decreasing user hesitations, increasing “Add to Bag” CTA clicks and conversions. | ↑ Add to Bag CTA Clicks | Results: Test ran for 14 days and the results are inconclusive. No stat sig impact on order confirmation page visits or other checkout page visits between either variation. No differences between mobile or desktop users. | No next steps required—learner experiment. Recommendation; conclude the test and move on to more fruitful tests. This is a good reminder that users may say one thing but do another, in that reviews are important to them, but maybe not on the PLP. Additional Info: |
Winner ⭐️ | PLP Filter Order (Yankee) Hypothesis: By changing the order of the filters in the sidebar, we can help users find their preferred product more easily, increasing PDP visits, add to bag CTA clicks, and conversions. | ↑ PDP Visits | Results: the test ran for 22 days and v1 (Filters Reordered) is a Winner! Result Details: v1 Filter Interactions had a stat sig uplift, increasing by ~15% with 100% stat sig. Although Filter Toggle Interactions may have decreased (not stat sig), all other filters received uplift of at least 6%, most notably Fragrance Type & Color which increased by ~20% with 100% stat sig. | RECs & Next Steps:
Additional Info: |
Winner ⭐️ | Add-to-Bag CTA Color update across Yankee Hypothesis: By changing from a “ghost” button style to a filled button style with a primary color, we can increase Add to Bag CTA clicks on the Homepage / PLP and thus conversions. | ↑ [Add-to-Cart] Clicks
| Results: Green wins! | RECs & Next Steps:
Additional Info: |
Learner | Cart Page Minimized NAV Bar (Yankee) Hypothesis: By minimizing the nav bar by default, we minimize distraction and focus users to the checkout process, thus increasing the number of users continuing to checkout and transacting. | ↑ [Cont-to-Checkout] CTA | Results: Minimized Nav (v1) showed increased CTA Clicks [Continue to Checkout] and slightly reduced bounce rates. | RECs & Next Steps:
Additional Info: |
Learner | Checkout Guest CTA (Yankee) Hypothesis: By changing the color of the primary Sign-In CTA for guests in the checkout flow, we will increase conversions via Guest checkout. | ↑ [Cont-as-Guest] Clicks | Results: After 21 days of runtime, this test is inconclusive. Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views). | (No immediate follow up—test results were mixed) Potential next steps: Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points. (This test is listed in our experimentation IDEAS area and is under consideration for future testing). |
Learner Test Duration: | Global Bag to Cart (Yankee) Hypothesis: By changing the bag icon and copy “Bag” to a cart icon and copy “Cart” within the global nav header, we can increase cart clicks, engagement, and conversion. | ↑ Conversions | Results: After 14 days of runtime, the results are mixed. Engagement: we see v1-Cart driving a 100% stat sig increase in global cart icon clicks by about 3.2%, leading us to believe it may be more visible or interesting to users. | (No immediate follow up—test results were mixed) Potential next steps: Consider prioritizing tests around Mini-Cart functionality to validate learnings of this test and revised hypothesis: … mixed results may be a result of users intending to engage the mini-cart (available with add-to-cart clicks), thus the increase in clicks to the Cart icon, but were potentially frustrated in that lack of behavior. |
Learner Test Duration: | PDP Add-to-Bag CTA Color Swap (Yankee) Hypothesis: By changing the add-to-cart button color, we can increase the number of add-to-cart clicks and conversions. | ↑ [Add-to-Cart] Clicks | Results: Original CTA color outperformed both variants with 98% stat sig for [Add-to-Bag] CTA clicks. | (No immediate follow up.) CTA color updates from this test were merged into separate ticket. RECs & Next Steps: Merged related learning regarding ATC [Add-to-Cart] CTA Color with Winning test 004. Learnings merged into voc ticket: ODYI-1503 - Getting issue details... STATUS |
In-Flight Tests & Upcoming Experiment Roadmap
Experiment | Experiment Name | Test Details |
|---|---|---|
In-Flight Test Launched | Checkout UI simplification (padlock icon removal) Hypothesis: By updating or removing the padlock icon we will decrease confusion within checkout, increasing conversions. | ↑ Conversions |
Coming Soon September | Checkout | Skip Login Hypothesis: By removing the Login step in checkout, we will create an easier checkout exp. thus increasing conversions. | More Info here: ODYI-1513 - Getting issue details... STATUS |
Coming Soon September | PDP Return & Exchange Copy Hypothesis: By adding copy about returns and exchanges in the PDP product detail section, we can boost user confidence and increase conversions. | More Info here: ODYI-1526 - Getting issue details... STATUS |
Coming Soon September | Global | Auto-Ship Messaging Hypothesis: By updating the auto-ship language, we will increase the number of auto-ship toggle clicks and transactions with auto-ship. | More Info here: ODYI-1527 - Getting issue details... STATUS |
Coming Soon October | Global Nav | Bag Icon Functionality Hypothesis: Making the bag icon open a mini cart will allow users to view their cart without leaving product pages which will increase the average number of items users add to their cart before checkout. | More Info here: ODYI-1521 - Getting issue details... STATUS |
Coming Soon October | PDP | Product Style Options Hypothesis: By focusing recommended Styles to only show specific types of products, this will minimize the number of options, decrease users being overwhelmed, and thus increasing site conversions. | More info here: ODYI-1555 - Getting issue details... STATUS |
Coming Soon October | PLP | Auto Ship Toggle Location Hypothesis: By moving the Auto-Ship toggle next to the sort drop down, we will increase toggle interaction and increase purchases with subscriptions. | More info here: ODYI-1554 - Getting issue details... STATUS |
Backlog | Mini Cart | add Checkout CTA Button Hypothesis: Giving users the option to bypass the (full) Cart will help conversion rates and create an easier shopping-to-checkout experience. | More Info here: |
Backlog | Checkout | Shipping Address Changes Message Hypothesis: By adding Shipping Address Changes messaging to checkout, we will reduce friction in the checkout flow, increasing continuation and conversions. | More Info here: |
Coming Soon | (Your Experiment Here!) | Do you have an idea for an experiment?Submit your experiment idea & hypothesis here. We hope you found this update to be informative and helpful in understanding where we’re headed as we develop our experimentation program. We’re actively working to extend our testing capabilities to include Marmot in the coming weeks and will be connecting with a few of you shortly. Again, a big thank you to everyone for their interest and support! Please reach out with any questions via email or slack—we’re happy to chat. -Thanks, Alex Alesio (Deactivated) |
Site Ramp Up | Scaling AB experimentation across key sites:
Below is a recap of our plans to scale our AB Experimentation capabilities across our key Newell sites.
(8) sites are planned to receive AB Tasty experimentation functionality. Yankee is live and fully connected with Graco soon to follow. This leaves (6) remaining sites to be activated. See below for details.
AB Tasty site activation plans & timing:
Completed:
Yankee (2022-June-1)
In Progress:
Graco (2022-October-1)
john.morrison (Unlicensed) & Doug working towards AB Tasty site configuration.
Upcoming (2022):
Marmot (2022-Nov-1) Targeting a November launch pending ELF/PWA dependencies which @Robinson, Doug (cc john.morrison (Unlicensed)) are actively working through.
FoodSaver (2022-Nov-1)
Upcoming (2023):
Coleman (2023-Feb-1)
BabyJogger (2023-Feb-1)
Calphalon (2023-Feb-1)
ExOfficio (2023-Feb-1)
Technical Work to configure AB Tasty: Alex Alesio (Deactivated) to continue working closely with john.morrison (Unlicensed) and @Doug who have been instrumental in executing the technical work required to have AB Tasty configured, tested, and implemented for our sites. @John has agreed to create and own the JIRA tickets required for AB Tasty implementation and will drive them forward with the team’s help.
Backlog build out per brand: On a separate but related track, Alex Alesio (Deactivated) will continue to coordinate with Brand / DIV business teams to start generating hypotheses ensuring our experimentation roadmap continues to be build out. If you have a hypothesis or test idea, you can submit it today Bit.ly/abRequest.
Please reach out to Alex Alesio (Deactivated) with any questions via Email or Slack. Happy to provide clarity where needed. Thanks! -Alex
Add Comment