Today, Amy is challenging Emily Emmer to break down some preconceived notions around conversion rate optimization. Emily is talking about achieving organizational goals, understanding and using your data, and much more.

 

Highlights & Takeaways:

• Conversion rate optimization often goes hand-in-hand with analytics
• The job of a conversion rate optimization strategist is simply to make a website work really well for the end users
• In order to act on the data you collect, you must be able to understand and interpret the analytics
• It’s crucial to understand the mindset of the visitor plus the organizational objectives and goals, and align those into a conversion strategy that can be tested
• Move users forward with micro conversions and get them saying yes
• Sending more traffic is not the solution if your users are unprepared or under-prepared to make a conversion when they arrive; increased numbers won’t solve your conversion issue

 

 

Show Notes:

Amy: Today we’ve got Emily Emmer on the show. Emily, it’s great to have you.

Emily Emmer: Thank you so much. I love Paid Search Magic, it’s one of my favorites.

Amy: Oh, well thank you. I appreciate that. Before we get started, I just wanted to queue this up a little bit because Paid Search Magic is primarily a podcast for a paid search audience but right now on the podcast we’re on a streak about conversion rate optimization, which is a different discipline than PPC or paid search.

The reason that I think it’s so important and worthy of discussion is that for someone who does paid search you have to know the basics of conversion rate optimization or CRO. You have to be able to tell a good landing page experience from a bad one even if you don’t specialize in testing, copywriting or design or what have you. You need to be able to identify some opportunities of improvement if a page is underperforming. This is a skill that can be learned and it really goes hand in hand with paid search.
I think sometimes when something feels new or we’re just not familiar with it, we tune it out and we imagine we don’t know anything about it or we can’t weigh in on it or be helpful and it just feels inaccessible.

So Emily, your mission today if you choose to accept it is to help us tear down some of our preconceived ideas about conversion rate optimization just so we can figure out what we need to know and maybe even how we can get our hands dirty with it so we can be better at our jobs in paid search. Are you up for the task?

Emily Emmer: Absolutely. I’m looking forward to it.

Amy: Cool, well I was hoping that maybe we could kick things off by you just giving a little bit of a background of your story, how you got into conversion rate optimization.

 

Emily’s Background

Emily Emmer: Sure so it was years and years ago that I wanted to get into digital marketing. I had been in marketing analytics at general marketing for about eight years and I really had wanted to get into digital marketing. I wound up having the opportunity to work for a small startup, which is a great way of getting into anything that you want to get into to start with. Someone who’s brand new. You can figure it out together.

 

I wound up having the opportunity to work for a small startup, which is a great way of getting into anything that you want to get into to start with.

I wound up working at that startup, did a little bit of everything there and it was a great opportunity in many ways. One of the reasons was because I realized how little I knew, how much there was to know about digital marketing and how many questions I had, how many different types of disciplines there were. It did give me an opportunity to dabble with a lot of different things. The great thing about that opportunity was that it enabled me to get some experience and then parlay that into a digital marketing position with a corporation.

I worked for a storage company based out of Utah and I was hired as their first digital marketing employee. It was a digital marketing generalist position. As our digital group grew, I had to choose a discipline. I really loved that conversion mindset so that was really what I wound up specializing in. I did analytics with that two pieces that are often paired together. That’s currently what my role is. I’m a director of conversion analytics for Western Governors University. It’s a national university that’s based in Salt Lake. Suddenly that’s how I got into it. My path was very much generalist, digital marketing for several years before finally specializing in CRO.

Amy: Awesome, yeah, thanks for that background. I’m wondering if you could explain a little bit more. You said that conversion rate optimization often goes hand in hand with analytics could you maybe give us some context for why that might be?

 

How come conversion rate optimization and analytics fit together so well?

Emily Emmer: Yeah, absolutely. When you’re working with paid search, CROs often limited to landing page experiences where you’re trying to give a visitor, a specific type of prospect, certain information about a company or organization to help them move forward to some specific conversion.

Landing page optimization is a really clean microcosm of larger conversion rate optimization. The job of a conversion rate optimization strategist is simply to make a website work really well for the people using it and to have it work really well for the organization. Ultimately, what a conversion strategist does is understand the goals of the organization and understand the goals of the visitor and bring those two things in as close alignment as possible. As the organization is better able to help that prospect find what they need, the more that prospect’s able to move forward.

 

Ultimately, what a conversion strategist does is understand the goals of the organization and understand the goals of the visitor and bring those two things in as close alignment as possible.

You know when you say conversion rate optimization to different people in business they’ll be like, “Oh, okay so your job is to scientifically get more people to do whatever you want them to do, right? Kind of manipulate the visitors, right?” There is that element but I think like I was saying a little earlier, it’s more about getting the organization more of what they’re hoping to achieve and giving the visitors more of what they’re hoping to achieve. Some of that is specifically done in terms of incremental testing like, “Okay so does the CTA button … Does it work better if it’s red or if it’s orange?” That’s your basic level of conversion rate optimization.

Why analytics is so important is that if you can’t read the data very well then you’re not able to understand what the test results are telling you. Are they totally wonky? Are they in line with what’s happening currently on your site? You have to be able to use and interpret data. What I’m trying to get at is I think there’s a two-fold reason that analytics and CRO goes so hand in hand.
The first is data has to inform the strategy for conversion. Otherwise, you’re really just shooting arrows at an undefined target and two you have to be able to do the more technical part of the testing, which includes the readout of the test.

Amy: Yeah, that makes a lot of sense and I think what you said the first part of it to be able to interpret what’s going on on the website in the first place to inform what test would be appropriate. The example that you gave about the color of the CTA, is that an example that you find comes up a lot, someone’s going to get a lot of bang for their buck testing a button call or is that something that … are there other things that are, say low hanging fruit that someone can get a lot of value out of testing if they don’t really know where to start and maybe they either don’t have access to the data or they just need an entry point?

Emily Emmer: Yeah so I would say that example’s one that the experience conversion strategist would look down on. They would be like, “Oh, don’t waste your time optimizing to a button color, color schemes, but they actually can make a really big difference. You could see a lift from say an orange button versus a red button and that’s a really easy test to set up. You’re going to be a little better served that something’s that connected to the thought process of the visitors to your site.
The clarity always trumps cleverness and persuasion. One of the most effective places you can start with is the action because that often gets you back to the offer. What action do you as a paid search strategist do you want the people that you’re driving to that page to take? If you want them to take that action, how can you more clearly label that? Then once you’ve clearly identified and labeled it-

Amy: Is that something that you would test then? That’s where you would start the test is to, or are you just saying that there’s where you would start figuring out what your page should be about?

 

Understanding both the mindset of the visitor and the organizational goals

Emily Emmer: I think the first thing to think about is just the mindset of the visitor. A lot of times, especially working in an agency environment, I would receive a specific test or a page. Okay, people are coming to this page, will you start fixing the page before you start testing the page? I would often feel “Well, sure but you’ve only given me half the story, not even half the story, you’ve just given me the page. I know what the conversion point is and I know what the page works right now but I have no idea what the mindset of the visitor is without looking at the ads.”
In your case, I think for you and your audience, you’re thinking about people that are clicking on some kind of paid search ad. What was the offer in the paid search ad trying to solve for and why would someone have clicked on that? Then try to connect that up with, “Okay, now looking at the page with that in mind, how does that connect with what we were offering in the ad? What problem that visitor was trying to solve by clicking on that ad?”

Amy: I think that’s a really important thing to start with. What is our goal? Then we would want to make sure that there’s a lot of strong message match between the landing page in the website and the ad. I do want to see if we can close the gap on something. I don’t know if you have any opinion on this but I’m curious to know if you do.
You’ve been talking about the visitor mindset, and the psychology of what drove someone to get somewhere. I’m curious personally if that’s the primary thing is what’s motivating someone, I don’t quite understand if my motivation has been identified, my mindset, everything is going the way it should be, why would the color of a button have anything to do with how I would respond to an offer? If it’s green or blue, that seems, as long as I can see the button, as long as I understand what the offer is why do you think that something like that is … It’s so often brought up. I know it’s an easy example because frankly I think it doesn’t require necessarily an understanding of the mindset that got you there, which is like, “Oh, let’s see, which color works best,” but why would it have any influence on if I’m properly prepped for the offer and ready for it, why would that change whether I clicked on it or not?

 

What do you think converts better?

“Apply free this week” or “Apply now”?

 

Emily Emmer: You know I can talk about color but I think easier would be to almost talk about the call to action offer words. At my university, we look at “apply free this week” versus “apply now“, which would you predict of those would test better, Amy?

Amy: I would say the one that has the most urgency in it, so I would guess that apply free this week is going to make it feel like it would go away by next week. That’s my uninformed guess, having no idea of what actually works.

Emily Emmer: Right, that actually is true but wouldn’t you think apply now would have just as much urgency?

Amy: Okay, sure, yes. “Apply now“, that’s my guess.

Emily Emmer: No, no. Your first guess is right and we’ve tested it. That’s accurate so apply free this week has both the urgency and the deadline so that works a lot better, whereas “apply now” is something that we’ve used for years and years. It’s still on many places on our website as opposed to “apply free” this week but we tested all three and “apply free” and “apply free this week” both tested better than say “apply now“. Visitors tourists don’t know that that’s something that they can pretty much take advantage of at any point so-

Amy: I think that’s a really good example of something that can be seemingly small but really does speak to that mindset, right? Because there’s plenty of places where of course an application is free but there’s plenty of places where an application can also cost you money and a university is probably a situation that there might you know, I’m not sure if I start this application am I going to get to the end and it’s pay $300 and you can hit send. That probably helps to reduce some of that friction that just helps encourage someone to take that next step.

Emily Emmer: That’s another thing that I’ve found is sometimes you can change something that helps a lot of people move from one micro-conversion to the next, but it won’t necessarily help those same people complete your macro conversion for this site. An example that might be if I can get more people to my check out I can get more people to complete my check out. Well that’s true and it’s not true. If you can get more people to the checkout who found the product they want or who had all of the questions that they needed answered by the time they get to the cart then your standard completion rate on that checkout page is going to apply.

If you do something on the site that then leads more people to that checkout page who are unprepared or underprepared, it won’t help. You’ll have a higher abandonment rate and you’ll probably still just have most of the people who are going to be completing the form anyway who are ready at that stage to move forward.

 

If you do something on the site that then leads more people to that checkout page who are unprepared or underprepared, it won’t help.

 

Amy: Emily I feel like this is such an important concept especially as it relates back to paid search. Just in the last couple of weeks I’ve had this issue come up with several clients where I feel like their understanding of in air quotes here, “the funnel” or the customer journey is that it’s like a magic slip ‘n slide, that you pour in a lot of volume, a lot of traffic and then you get a lot of sales. It’s just like whish until more traffic, more volume because we want our revenue to go up. Okay but more traffic with the wrong audience, with the wrong targeting or going to a page, again if they’re not prepared, if they’re not likely to take action, doesn’t solve that problem because you don’t just get to keep those same rates as you do if someone is highly qualified.
I feel like it’s something that not everyone has it. It almost takes a certain level of maybe maturity, digital maturity or awareness to get that to some extent it’s a numbers game but to another extent it’s almost a psychology game. You have to get people prepared if you expect them to take action. They don’t just do it because you took them to a certain page or game them a certain CTA.

Emily Emmer: Yeah, I love what you said there about it really being a psychology game. It ultimately, when you’re in the conversion game it’s all a psychology game, which is really the opposite of a magic slip ‘n slide. They’re the two extremes. One, you can just pour a volume in and you can get conversions and two a psychology game is all about what is happening in the minds of all the prospects that are coming and are they the right prospects?
It becomes a very, very different mentality. I think as business owners, it’s hard for business owners not to think in terms of well if I just pay more, I can get more. If I just pay 20% more in paid search, I can have at least 20% more in conversions. That might be true but it might not be true with that and that’s where the conversion rate optimization I think can really help. Especially as you put those elements together, as you’re getting the right people to the right pages, the right experiences on a site.

Amy: Yeah, absolutely. I know that a lot of times people think that they have a traffic problem when it turns out they actually have an offer problem. That could be the pricing of it. It could just be what they’re selling or it could be even the way it’s the offer’s conveyed on a page. If those things aren’t correct, then more traffic isn’t going to fix that for them. You said earlier that it’s more important to be … clarity trumps cleverness. It’s something we hear sometimes. Why do you think that it’s important to reiterate that? What’s causing people to try to be clever in the first place and how do we overcome that if maybe it’s the client that’s trying to be clever? What keeps them from wanting to be clear?

Emily Emmer: I think the cleverness is often a way of trying to take a shortcut to persuading people whereas clarity, helps people self-select to making a decision or moving forward.

Amy: That’s a really good way to look at it. Yeah. I think sometimes, coming up with your unique value proposition, what separates you from other people can be hard but coming up with, “Oh, what’s a pun or some sort of something that we think is catchy?” There’s a little bit less work that’s involved in understanding our audience. Maybe it is a faster path to take that we think is going to help but ultimately if it’s not obvious what you’re talking about, if you can’t self-select then I mean I don’t know that in the digital world we live in that our clever headlines tend to sell or convert a lot of people over to trying their product.

 

Getting the customer to say, “Yes, yes, yes!“

Emily Emmer: Right, that’s absolutely true. Cleverness is often just a crutch. We have to be more clever if our offer is less clear or if we have less distinction between our competitors, so we need to be more urgent for the visitors who come here. I think to your point there’s a lot of work that has to be done in terms of really defining what the offer is, really defining what the incentive is, helping the prospect, your ideal prospect understand why your ideal for them. Where are the micro yeses?
Mike Robs, MarketingSherpa, they talk about an inverted conversion funnel. Instead of people falling out of your funnel, having to climb up to your funnel with yes. Yes, I want to do this. Yes, I want to do this. Yes, I want to do this. Then the more yeses that they can build through your site, your experience, your page, your ad, your offer each one of those builds on the subsequent yes and every yes that the prospect can confidently say yes to gives them more confidence in taking the further action. There’s a ton of smaller conversions that are all really important and those small conversions are the building block of that bigger conversion.

Amy: Yeah, I think that’s a really good point and I’ve been in the process of thinking through traffic and volume and how it does is in itself is not the answer. I’ve returned to the whole inverted funnel myself several times again within the last couple of weeks just, “Hey, it’s an uphill process.”

 

What is a micro yes or conversion?

Maybe you could explain a little bit more about the idea of a micro yes or micro conversion because understanding that we can’t necessarily take someone immediately from being cold traffic on a page to saying yes to a really big ask. How are you creating a series of micro yeses and what does that look like? Are these small call to action buttons like are you having a good day? Yes or no. Are we asking questions and sounding like infomercials or how do we elicit that yes or are we actually asking someone to be taking action or just nodding their heads to the copy? What does that process look like?

Emily Emmer: You know on some sites it might be getting someone to participate in a quiz like you’re saying. If I’m looking for a product and the microsite or a page offers me a quick quiz to help me figure out my needs or figure out what product, if I can present something like that, getting that visitor to participate in the quiz is micro yes. Does that make sense?

Amy: Mm-hmm (affirmative)

Emily Emmer: Another micro yes that has worked really well for me is subscriptions. Having some kind of offer that pops up that relates to something the visitor wants where they are then part of an email subscription or you’re sending them some smaller offer. It’s not the ultimate conversion getting someone to subscribe enables the organization to continue to contact that person, it gives the visitor something that they want either continued conversation with the organization with a very low threshold of risk. All they’re giving out is their email. That’s an example of a micro yes.

 

All they’re giving out is their email. That’s an example of a micro yes.

 

I think it could be even smaller than it can be the right headline that relates to the mindset of the person and helps them want to move forward. Do you have other examples that you can think of?

Amy: Yeah, I do. I feel it’s an interesting question because they’re like the copywriting optimization, which would be the same as I was just reading a book recently where someone was talking about just you getting in the mind of your prospects and getting them to again be nodding along with you and agreeing with what you were saying. To some extent I think that mindset comes into play just in the way you write copy. The person can see themselves in your copy. That’s an example of not necessarily a micro conversion, there’s no actual conversion action but they’re buying into it and converting themselves, if I dare to use that term as they read through the copy.

I think that for a lot of websites there will always be this questionable, should we be gating the content or not? Often with paid search we do. It’s not so much a question. We’re not going to pay to drive people to a page where there’s no action to take. It’s usually a pretty answer there of yes, we’re going to ask for some exchange of information in exchange for the value so we have to make sure this is high value.

To your point lead gen and collecting an email so you can continue to nurture that lead or use a drip campaign to continue to communicate can be a really good micro conversion before someone’s ready to take that bigger step. I know I’ve got a client right now where they’ve got basically, trying to get someone to buy a sample of their product before they go on and buy the much larger product.

I think it’s really important to make sure that the micro conversion relates to the bigger picture because a lot of times people don’t give it a lot of thought and they’ll just say, “Well we’ll do a white paper.” Okay, but does the white paper or whatever you’ve gated have anything to do with the ultimate yes because if it doesn’t you might want to reconsider what you’re offering people in exchange for their information because they may or may not ultimately be your prospects in the first place.

Emily Emmer: Yeah, I think that’s a really good nuance to this discussion is the micro yeses, the smaller conversions they really need to relate to that ultimate conversion. I was trying to think through some examples of different industries, companies that I worked with, another one would be when I worked in self-storage. One of the questions that’s really difficult for prospects is what size of self-storage unit do I need? It’s very difficult to answer that with copy so what we did was we created videos of different sizes of storage units and put stuff into it. How much stuff can fit into say a five by five versus a 10 by 10. We had those videos on each product page. In that case a micro yes is getting people to watch a video to figure out which of these products is right for me.

Amy: I think that’s a really good example of where being in-house is going to give you a huge advantage. As an agency where you don’t necessarily have access, you can’t convince your client, “Hey, you have to make these videos,” or “Hey, you have to really invest in this copy in order to get to where you need to be”. To link that back to paid search, most of the time our paid ads are based on what’s on the landing page. We’re not going out and interviewing potential clients and customers and writing an ad for them. We’re taking what’s on the landing page and making a teaser trailer for it in an ad-

Emily Emmer: Yeah, I think the bigger point there though is trying to understand what are the blockers and what can I do to help with smaller steps someone move forward? If ultimately my goal is to have someone rent a storage unit, I have to understand a lot of the difficulties along the way. It might be putting their zip code into a zip code finder to find what is near me. It can be any number of things but I think to your point the micro conversions need to relate to whatever conversion paid search is being really optimized toward.

 

Closing the gap with missing information

Amy: Right, and I guess what I’m wondering is as someone who’s doing paid search who’s not specializing in conversion rate optimization and who doesn’t have access, who’s not that internal resource, I only have access to the information on the page in order to write the ad and in order to get it to convert. I don’t have anything beyond that. How do you close that gap if you’re not in-house? What are some tips for how we can expand? Like “Ugh, these bullet points are terrible but I don’t know what the actual bullet points should be. I don’t have access to all the information.” What are some ideas of how I can build that out and get people to be saying, “Yes,” if I don’t have any additional resources to use?

Emily Emmer: I think there are some different approaches to it. One thing that might help you figure out what should be on that page is to talk to the client about what are the top five questions of people that are the type of prospects we’re bringing in with this campaign? Then you can design the page to specifically answer those questions, so then they’re ready to take a step.
The other thing that you can do is try and understand the high converting pages on the site. Again, this gets back to some data and analytics. What’s working well now and why? With the assets we already have, what’s really working well in terms of getting people to make that conversion with the site is built around?

If nothing’s working well then you probably need to go back and really work with that client, reduce some people’s anxieties, answering their questions so that they have more confidence, increasing the credibility on the site. I don’t know. Do you feel like those are fair suggestions for limited content?

As a paid strategist your job isn’t really to come up with all of the CRO type work that someone totally specialized could do. Like, “Oh, let’s create some new videos,” but a lot of times a bit more of an investigative, a curious mindset can help you uncover some assets that you might not have seen at first or information that is going to really help with the structure of the page or the presentation of the information that is on that landing page.

Amy: I think those are some really great answers and some really good ideas to get people moving forward. Absolutely so thanks for bearing with me and sharing some of those. Yeah, I think at the end of the day it’s not like with paid search there’s only so much we can do to get the right traffic but if it’s the wrong offer, if it’s not compelling to people, obviously this isn’t a quest of can we force them to take action? Because, no. We can’t force anyone to take action. No one can but when we’re being evaluated for really what was our cost per lead? What was our conversion rate? What was our ROI?

There’s some expectation of your ability to move the needle for a client and so if they are the stuck point because, “Oh, well we’re just driving someone to the page that says, “Contact us.”” If we can give them a little bit of value and say, “Hey, let’s use your keyword and write a compelling headline and make it more appropriate for the visit,” then we can start to see more success and lean into it and grow. Whereas, we wouldn’t be ale to grow if we just, “Well, it is what it is and it’s your fault.” That doesn’t get anyone anywhere. Those are some really good tips there.

Emily Emmer: Yeah and I would say too, what makes a good conversion strategist is pretty much like a good behavioral mindset, a behavioral scientist mindset. Just thinking about the pages at some point with a behavioral scientist mindset is going to help a lot. Why are these people coming to this page and what are they hoping to accomplish? What are their fears? What are their concerns? Why would they not move forward? What is going to help with reducing that friction? What’s going to help reduce those concerns, mitigate those concerns, increase our credibility, increase the desire?

A lot of times even just creating an offer around things that are already present like I was talking about earlier “apply free” versus “apply now“. Those are really different and they have very different success rates. Even without something like “apply free this week“, well “apply free this week“, which is most successful for us was really ultimately just about pushing ourselves to be able to say, “Is it true that we can offer our prospects a free application this week?” Well, yes we can do that and we can do that a little earlier in the journey as opposed to later in the journey. Is it important for us to have a conversation with the prospect before we tell them that we are willing to waive their application fee or do we actually need to have that conversation before we’re willing to have that?

Those are all the more strategic questions but I think as paid strategists you put on that behavioral scientist mindset. It will help you have some of those conversations with the client that might push them because I think to your point you’re often limited or feel limited by what the client presents to you but most clients will really appreciate some instruction of what they’re just presenting you to like, “Oh, you know. Yeah, I think we’d be willing to extend the offer a little more because especially with paid search. The traffic is already more expensive so then aren’t you as a client, aren’t we a little more willing to try and help those people convert than we are about someone who’s come in from a less expensive source?”

Amy: Yeah, I think that’s a really good point and that leads to another challenge when I’m a client and someone’s like, “Oh, well you know your paid search isn’t working so well because your landing page is all wrong.” Then I can just say, “Oh, well fix it for me. Thank you. I’m open to that,” but without really knowing what needs to be fixed.

I know that there’s a lot of debate around should I just use a dedicated landing page kind of an on-bound page where there’s just the call to action on it and there’s no distracting links even a primary navigation and maybe footers have been stripped or should I be taking someone to a fully functional site that also has a lead gen or call to action form on it? What are the questions that you would ask in order to make that decision of how much, what kind of page to use or how much to link off of or distract someone with?

Emily Emmer: I think that’s all going to be dependent on the industry and dependent on the company. It’s a perfect thing to test. I don’t think anyone can really answer that without testing it. I would definitely recommend testing the stripped down microsite or page versus something like the page with some navigation versus maybe a more internal page that still would drive that person to the conversion and just test, which of these works better? A more informational page, a landing page, a landing page without any links versus with? Those are great things to test.

Amy: Awesome, let’s wrap this up with one more question about testing. Within paid search, within the tools that we most easily have on hand, AdWords itself is not a testing platform and we can try to get closer to statistical significance but at the end of the day the variation that wins out is just going to be the one ultimately that made the most money, just in general. There’s not the rigorous testing that there would be as if you were using an actual testing tool such as Optimizely.

 

What exactly is variance testing, and why does it matter?

My question around testing is I know you’ve spoken about and written about variance testing. I was wondering if you could just explain that briefly what it is and why that’s important and how it could affect the results of tests?

Emily Emmer: Yeah so variance testing, I was trying to think of a really quick and simple analogy to share with your group here. I think the simplest way to talk about it is just that variance testing helps to control for randomness within the population. Instead of testing your control say landing page A versus landing page B that has changes, maybe a different call to action, maybe a little bit of a different offer. If we test those two against each other we might see version B has 10, 20% lift in conversion or lift in conversion rate or both. You’re seeing a positive effect in B but if you don’t … Doing that in isolation just an A/B test, it doesn’t by itself control for the randomness in a population.

If we have 200 people and we split those 200 people in half and say, “How many in each group of 100 need a haircut today?” Well of the 200 let’s say, 10% need a haircut today so 20 of those people are going to need that and if we split that into a hundred and a hundred, we might get 15 in one group that needed a haircut and five in another group that needed a haircut. That’s before we say, “Haircuts are half priced today.” Once we say, “Haircuts are half priced today,” to one group we might get a certain amount more but by itself those two groups, they don’t necessarily split out to 10 people and 10 people need a haircut, right?

Amy: Right.

Emily Emmer: So what variance testing does is it says, “Okay, instead of A or your control versus B the variation. B is a half-priced haircut. We’re going to have two groups of A.” Let’s say now we have 300 people in the set just to keep these numbers simple, so we’ve we’ve got a group of a hundred, a group of a hundred, and a group of a hundred. Group A is haircuts are two dollars, and the other group A which is our next hundred people, haircuts are two dollars and group B, haircuts are a dollar.
In that particular group we might have eight in the first group A that need a haircut and 12 in the second group A that need a haircut and then 10 that need it in B. If we do that every day for 30 days, we’re going to wind up with pretty much the same numbers in A and B. In A, the two versions of A, which become our two control groups they’re going to eventually wind up with 10% for each of them. Then we can compare that to B and see if, “Oh, if half-priced haircuts wound up in 20% of people that were willing to get a haircut or is that an additional 5% of that population or does it doubles and now 20%.

Does that make sense? I was trying to think of a really simple way. Ultimately, variance testing is just instead of creating only two experiences to compare against each other, your control will split again to give you an indication of when to conclude the test because you want to say, “Okay, we can conclude this test when capital A versus lowercase a are both showing about the same conversion rate.” If I’m seeing that my control, my variance control is say 8% down in conversions but my test experience B is 8% up in conversions then I know that ultimately my test experience B is still really flat because I have at least an 8% variance across the experiences that is just the natural randomness of the population.

Amy: Will that be something different than achieving statistical significance with a level of confidence? Does this add to that or is it just a different way of accomplishing that?

Emily Emmer: It is a different way of accomplishing it. It is and ultimately achieving the same thing, but I think what happens is for most clients and most of us as marketers we don’t always have the discipline to actually achieve statistical significance. It’s hard to always let a test run as long as it’s needed to. It’s very easy to get excited about something and say, “Oh, okay this is done. We’ve had enough.” Especially with websites that you don’t work with as much or populations that you’re not as familiar with, it can be pretty hard to know with what type of randomness you’re working with.
It’s just another tool of working to achieve statistical significance but it’s also another tool that can help give your client more confidence in a test result.

Amy: Awesome, thank you for explaining that. I do want to add a little caveat for anyone who’s listening and hearing about this for the first time that this may not work very well for AdWords, Ad testing or any testing that relies on anything other than randomness because AdWords is not necessarily committed to rotate evenly. AdWords is incetivized to make the most money it can and it’s going to get money from the clicks. If it starts to get a certain learning about how an audience responds to that ad, it’s going to continue to apply that learning so you may never get back to … it’s not just a split. There’s a lot of inferences that are involved there and so how something is split through it a true testing platform is going to perform differently.

You could try to split your audience and try to accomplish variance testing and you may or may not ever be able to get there with ads. Just wanting to throw that out there as a little caveat for where this could or you may not see results of it being applied if you’re just trying to do it that way.

Emily Emmer: Yeah, I think it’s definitely a lot easier to do this type of testing with a testing tool and with a landing page. I think that’s where the most applicable for any paid search strategist but I think the principle is just probably a nice tool to be able to think about particularly if you’re being really challenged on two types of different variations of an ad that might be very high performing or if you just really are struggling to find another high performing ad or offer. It’s just probably a tool to think about but particularly when you have more high volume or a way to control for it. I think the variance testing is definitely a harder tool to apply to paid search testing.

Amy: No, but I mean for the landing pages though it’s perfect, right? It’s really important to be able to say, “Hey, this one looks like a winner but let’s make sure.” Because there is a natural variance between some performance to some extent is by chance and is random and so you do need to let something run long enough to know that it is because of a change to variables and not just randomness anymore. It’s a really important principle for sure.

Emily Emmer: I will also say when it comes to tests where you’re doing, any kind of test if the lower the traffic you have, the more big and bold your test should be because you’re going to see less change in the results with smaller test changes. If you have lower traffic to a landing page, to anything, if you have lower traffic I would recommend making bolder changes from the control.

A variance testing is also really helpful for sites with smaller traffic so that you can have a lot more confidence in your results. You might have to run the test longer because you’re splitting that variance up but it’s another way that I’ve done it is instead of just having just two versions of the control and one of the test is to have two versions of the control and two of the test. Really, what you’re trying to do there is to create some discipline for yourself and also to not have to panic if you’re seeing that your test experience is down 20%. When okay your variance is also down 15% so there’s probably just a little bit of randomness that’s happening.

Amy: No, that makes a lot of sense. Well thank you so much for explaining that and for coming on and helping us better understand conversion rate optimization. I really appreciated having you on.

Emily Emmer: Oh, it’s my pleasure thank you so much for inviting me.

Amy: Yeah, and-

Emily Emmer: It’s always great to talk with you.

Amy: You, too and if anyone wants to learn more about what you do or contact you link up into the show notes where it’d be a good place to find you.

Emily Emmer: LinkedIn is probably the best place to find me I feel like I look at that probably the most frequently of my social platforms for that type of question. Yeah, feel free to add me on LinkedIn or to just send me a message. I’d be happy to answer any questions or add any of your listeners to my network.

Amy: Okay, awesome so that’s linkedin.com/in/emilyemmer and we will link that up in the show notes as well.

Emily Emmer: Thanks so much, Amy.

Amy: All right, thanks Emily.

Emily Emmer: Take care.

 

 

Resources:

Find Emily on LinkedIn here.

Podcast: Play in new window | Download (Duration: 47:23 — 86.8MB) | Embed

Subscribe: RSS