Last year at FINCON, I gave a well-received talk about how I split tested a page on Bargaineering.
It was the #1 money making page on the site and generated hundreds of thousands of dollars a year.
It was so successful for a variety of reasons but the number one reason it did well is an idea that is ingrained my brain forever.
One post, one goal.
When someone visited that page, I had one goal.
Get them to apply for a credit card.
Nothing else mattered. I didn’t want them to subscribe to the email list (I killed the popup), I didn’t want them to comment (I turned off comments), and I didn’t want them to go to another page on the side (I removed the sidebar).
All my testing was focused on one goal – Get. The. Click.
When I started, about 10% of visitors would click on the Apply button. When I was “done,” about 35% of visitors would click on the Apply button. All because I had one goal and I played with the page until I maximized the one goal.
The key was understanding that my blog post had one goal. Not two. Not three.
You can’t test against three goals or even two goals. It’s like trying to herd cats, one cat goes this way, another goes that way, and before you know it the group of loosely associated felines has scattered and you’ve wasted your time.
Have you ever written an email to ask three questions and the person replied with an answer to just the first one? It happens waaaay too much. ISN’T IT INFURIATING!?!?!?!
If so, then you know the perils of having more than one goal in an email. One email, one goal.
The first mistake people make is to try to do too much. You think you’re being efficient but you’re being inefficient. The key to winning is simplifying.
The second mistake people make is to not have a goal in the first place. Why does that post exist? What do you want out of it?
(if the answer is – because I always post on Mondays… well that’s a terrible reason)
What’s the point of this post?
Please tell me what you think. 🙂
Last year, around this time, we did a private beta launch of $5 Meal Plan, the meal plan service I started with Erin Chase. You might remember Erin Chase, founder of the amazingly successful $5 Dinners, author of cookbooks, and our guest for the 12th episode of the podcast.
We officially launched the service on August 1st to the general public but we actually created a Beta program back in May. Both launches exceeded all of our targets (which says something about our ability to set targets huh?) and it was the first time I ever launched a product, let alone a membership site, and I thought I’d share a little of the behind the scenes.
Today, one year after beta launch and ten months after official launch, membership numbers are strong and growing. It was all based on a solid beta test foundation that helped shaped our service, from how the plans looked to when they were emailed, and this article discusses our beta strategy and execution.
Our Overall Strategy
The overall strategy for $5 Meal Plan was to come up with the core offering and then fine tune it based on the feedback from an initial beta group of testers in May. After a few months, getting both the product and the process down, we’d launch in August just in time for the rush of back to school, with a product that was honed and ready for prime time.
Step 1. Growing Our Email List
As you know, your email list is gold.
For $5 Meal Plan, our email list consisted of exactly 1 subscriber (me… even Erin wasn’t on it!).
We would need more. 🙂
Our strategy for populating the list was through surveys sent to Erin’s readers, of which there were tens of thousands, and signing up the respondents to our Prelaunch email list.
The first survey we sent out was a beta test for our survey! I wanted to learn the language of the people interested in $5 Meal Plan but also write the survey in a language used by the people filling it out. The first survey was sent out just to her followers on Facebook. The survey was 9 questions and all open ended.
The goal was to:
- Collect enough responses so we could create the same survey but with multiple choice answers.
- Understand the language so we could integrate it into our copy.
- Collect email address so we could add them to our list and…
- Ask them more specific questions based on their responses.
Here were the questions for the first survey:
- Are you responsible for meal planning? If yes, continue.
- How do you meal plan and how long does it take you each week?
- What do you like about meal planning?
- What do you dislike about meal planning?
- What is the most challenging part of meal planning?
- Are you currently using any meal planning services? If so, which ones. If not, what do you feel about them?
- If you do use a meal planning subscription service, what do you wish it did differently? Does it offer all that you need and expect?
- Do you have any special food preferences? (Food allergies, intolerances, dietary restrictions or choice to remove ingredients from diet.)
- Do you own any of The $5 Dinner Mom Cookbooks? (Original dinner cookbook, breakfast and lunch cookbook or the one dish dinners cookbook.)
Nine questions in a survey is a lot of questions. Especially so many open ended “fill in the blank” questions. We knew response rates were going to be (relatively) low but we only needed enough to create a survey populated with multiple choice questions. (I like to send surveys that can be finished in < 30 seconds)
We collected ~150 responses, which was enough for me to create a second version, consisting of questions with multiple choice answers (which are easier to quantitatively analyze and also faster to fill out). Erin sent that out several times to her email list and we collected 2,130 responses for the second survey. We made some minor adjustments to the questions (for clarity).
The most common reason why people meal plan is because they don't want to stress about what to eat for dinner each night (78%) and so they can save money (64%) and time (52%). I would use common phrases and words in those free form answers to create the survey but also for use in the copy on the landing page.
It's not a coincidence that the home page for $5 Meal Plan calls out the top three reasons.
Feeding the List
We collected our responses about a month before we intended to offer the private beta. As a bribe, and as a way to keep the list active for that month, we took the ten most popular recipes on $5 Dinners, by visits according to Google Analytics, and crafted a autoresponder series that sent out emails over the course of the month. People who signed up received several emails over the next month with great recipes, a gift that was inline with what we were offering.
The series was a huge hit with an average open rate of 58% and many many email replies telling us what they loved about the recipe, what they tweaked, what they disliked, etc. Replies are great.
One side benefit of sending out this autoresponder drip is that we would learn the optimal time to send out emails to this group – 7 PM Eastern. I would send out all of our launch emails at that time.
Quick Alpha Test
In emailing the survey respondents, I started collected about a dozen emails that I felt a good relationship with. These would be the folks I’d invite to an Alpha test where they’d be testing the checkout page and all the steps involved in signing up.
All of them signed up without any problems, which was great. Unfortunately we’d still have different problems during the beta launch! 🙂
Beta Launch Emails
First things first, you need to establish goals. Our goal for the beta launch was to get ~300 members beta testing the menu plan, shopping list, and our forums and membership site. We arbitrarily picked 300, it “felt” right.
We wanted a large enough group to capture all the different things that could go wrong and have enough people to give our forums a lift.
The launch itself was going to be pretty simple – four emails:
- A longer email about 36 hours before the close – A
- A shorter one about 12 hours before the close – B
- A brief reminder about an hour before the close – C
- A brief email a day after it closes for a waitlist – W
The timing and number of emails varies but the basic strategy is the same. Frequent emails that get shorter and shorter. A waitlist email after it closes.
Some metrics from our emails:
Email A: When we sent the initial beta test invitation, which was approximately 36 hours before we would close the beta, it went to ~2,100 subscribers.
Nearly half opened the email (48.9%, of the entire list) and half of them click on the beta (24.2%, of the entire list). Of those who clicked through, 40% signed up for our free trial beta that required them to enter in a credit card (more on this later).
We would lose 24 subscribers, or a little under 1%, to bounces or unsubscription.
Email B: When we sent the next email, which was 12 hours before the beta closed, it went out to ~1,900 subscribers.
This time only 35% opened the email and 11.1% clicked, of those 25% would sign up to the private beta. We’d lose another under 1%.
Email C: This is last email we’d send before the close of the beta. We extended the beta by five hours because of technical problems we ran into (more on that below) but our final numbers were ~1800 email sent, 27% opens and 9.3% clicks. 50% of the people who clicked through would eventually sign up. We only lost 9 email addresses with this send.
Since we delayed the close by 5 hours, we had an opportunity to add another truly “last minute” email. I decided against it since we were going to hit our goal of 300 members.
Signups by email:
- Email A – 202
- Email B – 51
- Email C – 81
Email W: This is the post-beta survey & waitlist email. My real goal for this email was to collect information from the people who didn’t join. These will eventually become objections I will want to kill on any copy I write.
This also gives folks who missed the beta a chance to get in later, if people drop out. You might be surprised to hear this but there already have been folks who cancelled their accounts, a few did it the same day they signed up (we would call them instaquits and 0.05% of signups instaquit, a rule that even applied during our launch).
Waitlists are often seen as a marketing ploy because you collect the waitlist and then let everyone in. We would eventually allow everyone on the waitlist to join because our 300 target was a target and not a limit. If someone wanted to try it out, we let them. It might seem like a marketing ploy after the fact, but we were giving away six months of the service for free so it seemed like they were getting the better end of the deal. 🙂
Beta Launch Landing Page
The landing page is about as sparse as you can get. I won’t even link to it here because there really isn’t much to it. It had a picture of Erin, since she’s the menu planner, with some text that said:
We’re super excited to have you to join us and help us make this the best experience possible!
What we’re offering is an absolutely free 6-month private beta trial of our service, after which you’ll be joining our monthly membership program where you will be billed just $5 a month. All we ask is that you let us know what you think of the service, what could be improved, and help us make it even better!
Please act quickly, we’re only accepting sign ups until May 13th, 12:00 PM (noon) ET. (due to some bugs, some people were having trouble signing up so we extended it until noon)
click to join now!
You can cancel at anytime.
Why so brief? All of our selling was done in the email and we didn’t want to go overboard. We’re not selling a $500 or $5,000 course, this is a $5 per month subscription for a product that is easily understood with few objections we could overcome in copy.
Key Lessons from the Beta
We learned two crucial things during the beta.
First, we used to send out meal plans on Sunday night. We figured most meal planning was done on a Sunday so why not send it out then?
Wrong. The planning might be done on a Sunday but the shopping happens much earlier. We did a survey and the bulk of respondents (like 85%+) requested that we send out meal plans on a Friday, Saturday, or Sunday — because that’s when they shop! So we send it out at 11am on Fridays.
Next, we used to have forums on the site. I knew that community was a big deal when it comes to any type of membership site and forums are a great way to build that community… unless no one uses them. We had 300+ members and virtually zero engagement on the forum.
Why? Because they weren’t used to going to our forums. We were trying to influence too many habits at once, without making it clear what the benefits was, and this was one battle I didn’t feel like overcoming (getting them to use the meal plans was the battle I wanted to fight).
So we created a closed Facebook group (members were already on Facebook all the time) and as I write this, it’s almost 12,000 members. There are pluses and minuses to Facebook but the majority are pluses – more on that in a future post.
About those Real Technical Problems
We extended the beta from 7am to noon because of technical problems. These were real technical problems, not “technical problems so we can get more signups.”
For one, our checkout page wasn’t mobile responsive – clearly a mistake because 31% of visitors use a mobile device and 15% use a tablet. This wasn’t just an aesthetic issue either, the way the page broke made it impossible for visitors to enter in their billing information… which is bad for a checkout page! 🙂
Thankfully, people emailed me and we were able to fix it.
Another “problem” was our host WPEngine. They use very aggressive caching and so there would be some unexpected and unpredictable behavior that made signing up very difficult (you don’t want to ever cache a checkout page!). It’s a known issue and there is a workaround involving cache exceptions, so going forward it’s not a problem.
Things I Would Do Differently
We were thrilled with the results. We hit our targets, got 15%+ of the email list to join the beta, and we have a vibrant community from which to grow.
But, here’s what I’d do differently… I would not close a beta at 7 AM. I don’t know what I was thinking closing it in the morning (4AM PT!), but I’d just shut it down at midnight PT and update the pages whenever I woke up the next day.
One thing I’d debate more is requiring a credit card for the trial. This is a debate that has raged on for years. (more on this in a future post, now that it’s a year later, we have data on the credit card vs. no credit card thing) I don’t know the answer other than the most common reason people didn’t sign up was because they didn’t want to enter in a credit card.
That said, I would like to believe that the members who did are more committed… which is crucial for a beta test. We don’t want tire kickers with no buy in to be the source of our direction!
That’s it for now, hope you enjoyed it. If you’ve done launches, do you see anything I could’ve done differently? Or otherwise have questions on my logic? Let me know in the comments.