Versions Compared

Version Old Version 8 New Version 9
Changes made by

Alex Alesio (Deactivated)

Alex Alesio (Deactivated)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

new-22-003 Checkout | Guest CTA (Yankee)

Test Results Update (Aug 11)

Subject: Re: Test Update: new.22.003 _ Checkout | Guest CTA | A/B Test
From: Steven Shyne <steven@cxperts.io>Sent: Thursday, August 11, 2022 19:59

Hi Alex,

When we looked at this test at 2 weeks, we were inconclusive but still had some volatility in the test data; usually we like to go one more week (21 days total) in that case. 

I'm looking at this now (21 days of runtime) and I'm going to call this test Inconclusive. 
This is after checking both Mobile and Desktop and there is no stat sig difference in behavior for primary goal (clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views). 

Interesting though that this Color CTA test won on Homepage / PLP but not PDP and Checkout. My hypothesis though isn't about location, it's more so that the other test had ghost buttons, making me believe we should avoid ghost buttons site wide. I'll turn the test off now and update our records accordingly. -Thanks, Steven

After 21 days of runtime, this test is Inconclusive. Both Mobile and Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views).  

...

Experiment Design:  

...

...

Hypothesis:  

  • By changing the color of the primary Sign-In CTA for guests in the checkout flow,
    we will increase conversions via Guest checkout.

...

...

Results (added Aug 11):

  • Results: After 21 days of runtime, this test is inconclusive.

  • ( pending )

Recommendation (CXperts):

  • ( pending )

By The Numbers: data from ABTasty

  • ( pending )Both Mobile & Desktop do not have stat sig difference in behavior for primary goal (Clicks to Checkout as a Guest CTA) or macro goal (Order Confirmation Page Views).

Recommendation & Next Steps:


  • End the test. Consider additional testing on Checkout to understand how Loyalty Messaging on Checkout can incentivize users to Sign-In to generate Rewards Points.
 

    (This test is listed in our experimentation IDEAS area and is under consideration for future testing)

  • Note: Interesting though that this Color CTA test won on Homepage / PLP (but not PDP and Checkout). CXPerts' hypothesis though isn't about location—it's more so that the other test had “ghost” buttons, making me believe we should avoid ghost buttons site wide.

...

...

Video Capture Preview:

AB Test 003 Preview Video ( 2022-08-11 RecordIt ywDtAtnhjB ).mp4

Note: This screen capture video is helpful in providing a quick preview to understand the nature and purpose of the test, but may not be as up-to-date as the below preview links.

...

Screenshot Preview: 

...

...

AB Tasty Preview Links: 

...