Starting August 12, 11am EST, I’ll be posting live updates directly to this post as we A/B test our startup’s homepage (and learn along the way).
First Hypothesis: Increase signups for Teachable by implementing an uncomfortably minimal homepage using Welcome Mat.
Background on the Welcome Mat experiment:
About 2 years ago, my friend Mattan Griffel first posted about the concept of the minimal homepage.
Mattan’s post picked up a ton of traction and was reposted by Andrew Chen under the title The highest ROI way to increase signups: The Minimal Homepage.
The premise of the minimal homepage is increasing your signups by keeping your homepage stupid simple.
Check out the hompages of some of the fastest growing startups around like LinkedIn, Spotify, DropBox and Pinterest?:
Notice how each of these pages have almost NO information outside of a main heading, some social proof and call to action. Would you ever consider such a minimalistic design?
The minimal design worked NOT because these are well-known startups. Most of them STARTED with this approach and kept it because it worked.
Check out DropBox’s first minimal homepage:
DropBox today maintains a similar approach, despite its estimated $10 billion valuation.
To-date, I’ve seen hundreds of A/B tests that confirm a minimal homepage improves signups.
Here’s why keeping your homepage minimal works:
- Reduces information overload
- Focuses your audience on a single, clear value proposition
- Let’s you test what actually converts someone
Testing your homepage is high leverage
Your homepage is the place where all your users start coming in. Improve that and everything else gets easier.
Improve your conversion rate and you don’t have to bring in as many people to your site for the same amount of signups. Second, once you have their email, you have more chances to reengage people who are interested, but not immediately bought into being your user or customer.
The challenge with the typical minimal homepage
Despite that evidence, at Teachable, we’ve stuck to the thinking that users need information before signing up..so far.
For a B2B/B2C hybrid startup, we have to think about presenting information for someone evaluating and comparing us to other option.
While we’ve tried to be minimal and keep as much information off of the homepage as possible, we’ve found many customers demand this information. This is common even if they’re signing up for our free account.
Because of this, you’ll see links to examples, features and pricing. You’re also able to scroll down the page to see user testimonials and SEO keywords we’re targeting.
Starting the A/B test: Say Welcome…to the Welcome Mat
You might have recently seen the Sumo Welcome Mat app pop up all over the interwebs.
“The Welcome Mat” is basically an animation that temporarily covers your homepage.
When someone lands on your new site, the welcome mat animation takes over a page with a single call to action. It then disappears if a user scrolls down.
Looks a lot like the minimal homepage, right?
Last month, Welcome Mat’s launch earned nearly 500 upvotes on ProductHunt with comments like this one from Andrew Warner:
And it’s for good reason. We all know the benefits of collecting email addresses: building your email list is essential for growth because it gives you a way to contact your audience about your product. I like the way Kevan Lee of Buffer puts it when he says: What better than the privilege of contacting someone directly? It’s Seth Godin’s permission marketing realized.
But at Fedora, we’re already focused on simplicity and our design is relatively simple. Can the welcome mat improve our signups?
It’s a low risk experiment. What I love about the welcome mat is that even if you’re totally resistant to the idea, it gives you the best of both worlds.
You get the effect of increased signups from the minimal homepage, but can still add more information below the fold. You’re also not forcing anyone to signup, you’re just making it easier to do so.
Background on implementing the Fedora Homepage A/B Test
The Fedora homepage can get thousands of hits daily, and hundreds of signups, but what would happen if we added the welcome mat to it?
Over at least the next 24 hours, we’re going to test taking over our homepage using the welcome mat, and give as much data as we can about the results.
Objective: Increase the number of webinar signups to Fedora
To get there, we’re going to A/B test our current approach to adding a simple welcome mat that’s focused on getting someone to sign up for our webinar.
For the second-half of the test, the Without Welcome Mat Test, we’re going to collect signups as we normally do:
- Direct Signup
- Modal Signup
- Pop-Up Signup (this is triggered based on exit intent)
For the With Welcome Mat test, we’re going to have all the same elements, but add in this Welcome Mat:
How we’re implementing the A/B test
To accomplish the experiment, we’ll be using Optimizely, Sumo and MixPanel to measure the results.
Optimizely will send 50% of our traffic to both the A and B test. We’ve decided against using Optimizely to measure to the sign ups, given front end implementations can be a bit inaccurate.
With Sumo, we’ve hacked together two accounts that are placed on the same page–one of these will be for the A test, and the other for the B test.
We’ve also created super properties with MixPanel, which we’ve implemented on the backend for more accuracy. This tags users from each variation, so we can see how they interact after signing up.
I’ll then be totaling these up in an Google spreadsheet and testing for statistical significance.
To keep it interesting, below I’ll be live blogging this thing.
**FIRST UPDATE: August 12th, 11:00AM - August 12, 6:00PM**
One slip up, that I didn’t realize when starting, was that our webinar pop up was set to display after 5 seconds, not on exit intent.
What this means is that for anyone who saw a Welcome Mat never got to see our pop up. A missed opportunity that would improve our overall conversion rate. This is no big deal because:
For far we’ve brought in 3X more webinar sign ups with the Welcome Mat than without.
Digging deeper, here’s how the results have looked:
Here, you’ll notice, the conversion rate for a webinar sign up is actually only slightly higher with the Welcome Mat. The conversion rate being the same, makes sense given the copy and the offer are almost identical.
However, we still have a 3X improvement in sign ups with the Welcome Mat. What appears to be happening is we have a lot more people seeing the Welcome Mat than the pop up over a similar period of time.
Although we’re still on our way to being statistically significant as you can see here:
Statistical significance only tells us which conversion rate is better when we compare with and without the Welcome Mat.
It’s fairly plain to see, that by starting off strong with a Welcome Mat, we’re able to reach more people. In this case, reach and convert 3X more.
Tomorrow, I’ll dig in a bit deeper showing how this has affected our total sign ups for the day.
**SECOND UPDATE: August 12th, 11:00AM - August 13, 11:00AM**
After 24 hours the results of the experiment appear very promising. We’re seeing a 2.6X improvement in webinar sign ups, and 40% lift in conversion rate with the Welcome Mat.
I shot a quick video going through some additional detail below as well as next steps. I also dig a little into our set up to run the quick and dirty A/B tests and statistical significance.
Right now, I want YOUR thoughts: Improving signups is one of the highest leverage things you can possibly do to grow YOUR product as well.
Is there anything we should test that we can learn from together? Let me know what you think in the comments below.