CC 066: A/B Split Testing For Content Marketers With Nick Disabato

Show Transcript

[0:00:09.0] ANNOUNCER: Welcome to the Content Champion Podcast. We provide the training and tools to help you become a content marketing champion in your online business. Introducing your host, the content champion himself, Loz James.

[INTRODUCTION]

[0:00:28.7] LJ: Welcome to the Content Champion Podcast. Thanks for listening. On today’s show, I’m delighted to welcome Nick Disabato, the writer, designer and AB testing expert, and author of The AB Testing Manual. Nick’s draft design consultancy has a proven track record of using research driven AB testing to increase revenue for his E-Commerce clients. I’m hoping to tap into this specialist knowledge today so we can make meaningful conversion decisions on our own business websites.

As ever, you can download all the show notes and transcripts of this podcast at contentchampion.com. So let’s dive in.

[INTERVIEW]

[0:01:09.6] LJ: Thanks for coming on, Nick.

[0:01:10.6] ND: Happy to be here.

[0:01:12.5] LJ: Now, before we get the lowdown on AB testing, we’d love to hear your backstory first please.

[0:01:18.1] ND: Sure, I come from an interaction design user experience type background. I don’t have that much of a marketing background. I know a lot about design research methodology and about, for a lack of better term, product I guess it’s called this days and got into AB testing because I actually kind of fell into it by accident.

I was thinking about things that I could do on retainer basis and things that could have involved design activity in some capacity. AB testing seems like a really good fit, fell into that about three and a half years ago now. I started Draft Revise, which is a now quarterly AB testing service. I’ve been doing that ever since.

[0:01:59.8] LJ: Just on that, with the work you do through Draft, what are some of the mistakes that people are making when you go to them to work on their AB testing?

[0:02:08.4] ND: Yeah, a lot of people who come to Draft have tried AB testing and hasn’t worked out well for them and they’re trying to figure out why. The most common mistake is jumping direction into AB testing.

The most common scenario I see is somebody hears, “AB testing is the new hotness and here is some AB test that you can run,” and it’s like a call to action color test or like a headline test or something like that. They’re just like, “Change the headline,” and so you change the headline and it turns out inconclusive and either you don’t have enough traffic to get AB tests to statistical significance, that’s not uncommon.

More importantly, you haven’t really bothered thinking through exactly what headline to be using, or what call action to be doing, or what layout is working well. These incremental changes don’t really work very well as AB tests. I recommend kind of a rethink of the process. Obviously because I’m a designer, when you have a hammer, everything looks like a nail, right?

I recommend design research methodology for this and so I say, “Okay, well, let’s try and figure out what to test and what might be the most high impact place to test.” I find that that results in a much higher success rate for my testing efforts and it still provides a solid ROI. Because the goal is to make more back than you spend on this entire process, right?

People feel discouraged when it’s like, “Well, not only did I have to pay for an AB testing framework, not only do I have a consultant but now this consultants wants to go out and do research and that’s crazy because it doesn’t seem like that has anything to do with AB testing.” I spend a lot of time saying, “No, actually it does.”

It’s fundamentally a question of measuring the economic impact of a design decision, that’s what AB testing is. That can be anything, right? You can change the color of your call to action and determine that it’s not actually providing significant lift. But I would rather use that as a very powerful tool to either do more ambitious changes or more focused and intentional changes that actually match what your customers are telling you.

[0:04:07.6] LJ: Okay, so it’s a research based, strategic process and just winding it back to the definition you gave there in that answer, AB testing at its heart is just looking at one variable at a time to try and increase the conversion on that variable. How do you see that as a definition for people that don’t know?

[0:04:29.3] ND: Yeah, if you know nothing about AB testing, it’s the scientific method applied to making changes on your website. You have a control, it’s your existing website, and then you have a change, that’s the variation. They are A and B, hence, AB testing. You send 50% of your traffic to each variation and track your customers to see what their behavior is. That one goal is usually sales or revenue, average revenue per user. Those are the most common that I use.

But you can track anything and I track usually multiple things at once. So if you’re like a SAS business or something like that, I’ll say like clicks to pricing, clicks to signup, actual signups, conversions from trial, all of those things are different goals. You’re focusing — usually there’s one primary goal that you’re using to calculate out whether or not it’s worked well.

It’s dangerous to use kind of secondary metrics for that for a lot of reasons and we can’t get into details about that later but yeah.

[0:05:25.1] LJ: Okay, when you start working with clients, a lot of them are maybe weary about going from a research point of view first because they want to get sort of into it, using all the tools that you can use online now and small businesses can use them to do AB testing quite cheaply. But you have to sort of reel them back and say, “No, this is a scientific research based process.” Tell us about that, tell us about your approach to AB testing?

[0:05:52.7] ND: Yeah, sure. It begins with research, obviously. So if you aren’t doing a lot of research, I’d set some things up so that can involve either quantitative research which you might already be passively collecting like Google analytics or like browser device now, something like that. To heat maps, scroll maps, behavior recordings, those sorts of things.

I’m taking a look at all of those to see how people are actually behaving in practice. In addition to that is more qualitative methods of AB testing of research that include user interviews, conducting usability tests, conducting surveys and asking for more free form responses. Then in the middle it’s kind of more a quantitative and qualitative hybrid where you have things like, you can take a look at behavior recording, you can try and understand how long it takes somebody to go through a funnel but you’re actually seeing what their behavior actually looks like.

I’m setting all of those things up, I might put together a screener and get people on the phone for interviews and I’m definitely getting heat maps and scroll maps because those are very cheap and easy ways to get high impact results and then determining, “Okay, well, it seems like people are dropping off here. What can we do about that? It seems like people have a reservation about buying the product because of this reason and that’s not articulated anywhere on the website. Can we change any of the copy to address that objection or that concern?”

You’re trying to find ways to capture leaky revenue no matter where you see it. If your checkout form is too long, pair back the number of fields on it, run a test on that, and determine what the actual lift is. If people are coming in from an unexpected demographic and they have a different set of priorities and they’re buying the product for reasons you didn’t necessarily understand, put together a landing page for it.

Rework the home page if you get less traffic, or direct an ad campaign that addresses that sort of objection, that sort of thing. You basically listen and then try and get people what they want right? Do it in a way that still resonates with your business goals.

[0:07:56.6] LJ: Okay, we’ll go through some of those methodologies in more detail and then go and look at how we can set up some of those tests and use the research information we’ve got. Just quickly, as an aside here, there’s a bit of a chicken and egg thing going on, the chicken and egg equation here. Because of course you’re going to get your research information but you won’t know necessarily what questions to ask of that until you can see what’s going on.

[0:08:20.5] ND: Yeah. As it applies to an interview, I usually come in with five or six questions about using the product that come from what’s called a “job to be done interview” and I can cite a leg that provides a lot of incite around that in the show notes if you desire. Where you’re basically asking, “Okay, well, why did you come to the website today?” Because I have to capture you through the websites. There’s that. “What do you do? What motivates you? What were the objections that you had before you came in and bought this product? What other competitors did you look at? What was the last thing that held you back on it? What are the things that excited you about this?”

Any of those questions, usually they bring back a response that is interesting in some capacity and then I sit back and I ask as many open ended questions as humanly possible before I move onto the next one. I’ll be like, “Tell me more about that. You said this, can you expand on that a little bit?” Then I just shut up and listen and that’s the time when I usually get really interesting stories, right? Yeah, there is definitely, I agree with you, there’s kind of an unknown-unknown situation right? We don’t know what we don’t know.

When a client comes in and they’re like, “Why are you getting on the phone with these customers?” I’m like, “Well, I don’t know.” They’re like, “Well, why are we doing it?” And I’m like, “Well, precisely because we don’t know and because we’ll be surprised by what they tell us.” That’s usually the case, especially if you’ve never done it before, it’s very much like you get a ton of quick wins from the first interview. You get a few more from the second interview.

The less frequently you do it, the more surprised you are by the first round of it, I found in my experience. Yeah, I agree with you that it feels risky to be doing that especially when all you want to do is get to the tests and get the results from the tests. It’s way more exciting, usually they result in heat maps also and they’re pretty pictures and people just kind of love those. But I agree with you that you definitely have to do a little bit of selling to convince people of it. But once they see the power of it then in my experience, they’re willing to go deeper on it and continue doing it.

[0:10:24.6] LJ: Okay, well let’s dig down into some of this quantity and qualitative research methods that you mentioned earlier. You’ve talked about Google Analytics, there’s browser and device analysis, you got your heat and scroll maps, we talked about behavior recordings and heuristic evaluations as well. Talk us through some of those and just touch upon how this help and then we’ll look at the qualitative stuff afterwards.

[0:10:46.8] ND: Yeah, Google Analytics is one of those things that’s just ungodly powerful but it’s also such a poor user experience that you probably don’t actually use it very well. You’re making sure that there’s no huge fires or that you use it to check your overall traffic numbers, but that’s really it.

This happens in businesses of all sizes. I see them not fine tuning their Google Analytics and install wells. There’s kind of two things that I’m doing with that. One of them is I’m preparing it so I can track your conversion rates effectively. So I’m setting up goals like somebody makes a purchase, somebody goes to this page, somebody views your introductory video.

Then I’m picking out funnels. So if you’re a SAS, there’s pretty consistent funnel from like home pricing signup onboarding. I create those and if you’re an e-commerce site, it’s home, landing, product, cart, confirmation and then you’ve successfully placed an order. So that sort of thing. I’m putting those together and those are extremely powerful utilities you get for free.

The other side of Google Analytics is more for like one off tweaks that have everything to do with optimization and very little to do with actually testing. I will go through your browser analysis and your device analysis and your site speed and do all of the boring quotidian work of fixing your sites so that it loads properly across browsers and across devices because every single client that I work with, it’s like, your site is converting 35% worse on [insert browser name here]. Why is that? Your site is dismal on mobile. Why is that?

Sometimes there are easy answers to that. Like there is one that I work with that’s a B2B SAS, that’s an email provider right? They don’t get a whole lot of trial signups on mobile because they’re an email provider and so you need to migrate your list over and do a bunch of fiddly wonky things. But the call to action on mobile for them is “start a trial”. I actually set it up so that you can request a demo from their sales team instead and demo requests spike, demos convert to trial signups. Something like 85% for them, it’s something crazy high.

So you’re capturing a lot of interest and funneling it in, in a way that makes sense for the person. If I’m on the train, in a subway tunnel and I can’t actually sign up for your service but I can throw my email address in a forum, I’m going to go ahead and do that. It’s like the path of the least resistance. That’s one example of it influencing an AB test really effectively. Most of the time, it’s like your responsive doesn’t work well, we need to fine tune your mobile experience in some capacity, and those are design improvements. Like, I’m looking at it and trying to figure out design improvements.

That’s it for Google analytics, and a little bit about browser and device analysis as well. Heat and scroll maps are more — those are the real quantitative to AB testing that I do, I look at where people are clicking and scrolling and most importantly, where they are not clicking or scrolling. If you have three calls to action on the page and everybody is B lining for one of them, does that one fulfill your business goals? Does it map to revenue generation? Are people bouncing off of the page and going in really weird directions in your funnel?

I’ve seen so many SAS business have like the demo app and then there’s no call to action from the demo app. I’ve had them do like the features page and then it’s like, you have to back track. I’ve had them not have sticky navigation. It helps you be intentional about your navigation on every page of the site. It may not necessarily mean it’s consistent navigation on every page. But it’s the navigation that makes the most sense to move the person along towards conversion.

[0:14:18.6] LJ: Just as an interjection here, because as a content marketer, I’m a nerd for all of this stuff. Actually, when you get into that position when you’re evaluating a SAS piece of software and you go in and say do the organic search on Google, you land on a long form piece of content like a blog post with a video in and you end up signing up for that demo and then you end up buying the product.

It’s actually quite masterful when it works properly and it goes through and you could see where they’ve obviously refined that process and tested it so me as the customer for that interaction, it’s a very seamless streamline process through their content marketing funnel to the end goal of actually me, making a purchase and when it’s done properly, it’s fantastic.

[0:15:06.7] ND: Yeah, absolutely. You can even — when it’s done properly it’s fantastic and then the initial things that you get from it are also quite scary, right? If you have a huge, long piece of content and the call to action is all the way at the end, you run a scroll map and find that only 20% of your readers are getting to the end then you have like maybe issues with the headline of a content or the initial pitch in the content or it’s not grabbing people.

Or they’re just instapapering it and coming back later. Is there an opportunity to put the call to action earlier, after the first few paragraphs? Is there an opportunity to rewrite that content and approach it in a different more rigorous strategy around it? You have to be as logical and scientific about it as possible but there’s also very much this like emotional response to getting this data back and you’re like, “Oh god, this wasn’t actually working like I expected and what now?” It’s not enough to just write the content. You have to understand its ramifications on your readers and your perspective customers.

[0:16:04.0] LJ: It’s fascinating, we’ve just started using mouse flow for several client projects and actually looking at what you think people are doing and then what they are actually and reality doing with exactly the sort of interactions you’re talking about there. It’s really interest resting and you can instantly say my goodness, if only they have the option to do that there or to do this there, that certain point of scrolling or whatever and there are various tools as well that we can use as visual website optimizers isn’t there and a few others.

Talk us through a few of those because there’s plenty out there that are affordable for small businesses aren’t there?

[0:16:39.2] ND: Yeah. Visual Website Optimizer is one of the two big AB testing frameworks out there, it’s done Optimizely. They’re kind of the canon and icon of the AB testing world, they’re an eternal feature shoot out and they’re both good businesses, go with whichever you want but VWO is interesting because they also include heat maps and scroll maps and behavior recordings.

You can sign up for, I think it’s like a $49 a month plan with VWO or it might even be less than that. You get this basically free. So what I do with my clients is I sign up for a VWO, I have a huge agency plan, I get them on there and then I just run behavior recordings and heat maps on all the pages of their site while I’m doing interviews and all the other bits of my research.

Then I come back in five or six days, take a look and I’m like, “Okay, well, here are the heat maps, here are the scroll maps, here are some interesting things that we might be able to test out of it.” There’s another one called Hot Jar that’s a competitor to Mouseflow, they do behavior recording and I think there is actually a free plan for Hot Jar. There’s Crazy Egg, which has a pretty generous trial period.

There are a lot of ways to test on little or no budget. In the United states, usually corporate petty cash cards have a $500 limit. You can easily come in under this limit and get a couple of usability tests even from a site called usertesting.com, you can run a heuristic evaluation in your spare time.

I have a whole check list for that in one of my courses and there are ton of online. ConversionXL has one, I think Copy Hackers has one. You can run Hot Jar, you can get a VWO account and do this pretty easily and with very little managerial support even, where you’re just gathering a ton of data and saying, “Okay, well this is what it tells us. What now? What do we do with that?” So many customers just don’t bother talking with their users that a lot of people end up shocked that the data says this because they thought something different.

[0:18:40.1] LJ: Just actually mention ConversionXL, I spoke with Peep Laja of ConversionXL a few years ago on the podcast. As well as you could self, he really knows his stuff when it comes to conversion testing and everything related to it. Yeah, let’s move on as well. Okay then, to the behavior recordings, we’ve touched on that, and then heuristic evaluation after that if you would?

[0:19:00.9] ND: Sure, behavior recordings are, I’ve been talking about it a lot, haven’t described it. It’s essentially you are getting a video back of somebody’s cursor going around the screen, seeing where they hesitate, when they scroll down, whether they scroll back up, what they’re highlighting, that sort of stuff. If you think that’s only for a desktop, you get little dot to show where somebody’s fingers are, two dots if they’re pinch zooming, that sort of stuff.

These are just insanely powerful for understanding more real world behavior about what people are doing, it’s especially good from a content perspective for your listeners because, “Okay, well how far are they going down? Are they reading only three paragraphs and clicking?” There’s a tech blog I used to work with, called The Wire Cutter, for about a year and a half.

They do like recommendations and deep dives of tons of different products in a specific category. They find the best exercise headphones for instance. The behavior recordings— I’ll probably share this bit. It basically showed exactly what I was thinking because it’s exactly how I browse the wire cutter. I load the page on the wire cutter and I see, “Wow, that’s a really tiny scroll bar.”

I scroll down and back up and I’m like, “Wow, they really did their research, that’s a ton of research, what’s the pick. Okay, that’s the pick, okay I’m done. Nobody ever reads, I mean, some people do because they’re nerds but they read the entire thing on Wire Cutter right? You just go, you look like this person is trustworthy, they clearly know what they’re talking about, it’s pretty well written, here’s the recommendation.

I’m going to go and click it. Your interaction is actually relatively tiny. From a content standpoint, it’s doing the right work like it’s not BS content right? It’s still valuable but what it needs to do is signify trustworthy in that scenario. It does a very handsome job of that. It functions really well. Are people reading every single word of the content? No. Is it fulfilling the business goals of the wire cutter? Absolutely it is right?”

That’s an example of behavior recordings working in a way that may not have been terribly expected. You put in 300 hours of research to a 30,000 word article and then you put it up and nobody reads it, that must feel really deflating. But in fact, it actually is doing the right job. People just don’t have an attention span.

[BREAK]

[0:22:20.3] ANNOUNCER: You’re listening to the Content Champion podcast. Showcasing the training and tools you need to become a content marketing champion in your online business.

[INTERVIEW CONTINUED]

[0:21:41.2] LJ: Again, a question that comes out of that, when you’re working with a specific client and it’s down to context I guess and the psychology of the context of conversion. When you’re looking through those scroll maps and behavior recordings. When people reach that balance of that sort of see saw of information and value and then they sort of gain that level of trust that they want.

Do you find that’s the sort of roughly the same place all the time and all the content or is that then related to how they found the site and whether they came in from a certain search term and the intent that goes with that? Do you see what I’m getting at?

[0:22:20.3] ND: You’re getting at a really great point and this is something that could probably be a three hour response, but it’s very much a fine art to come up with the right next question to be asking yourself about the data, right? Because if I look at somebody just scrolling and then they click and like, “Okay, well is it converting?” Well then I would go to Google Analytics and see if it’s converting because I’ve configured the goals over there properly hopefully.

Okay, well, that’s interesting. So it is working really well, what can we do to convey the trustworthy ness of it better? Is that the right question to be asking? Okay, well we’ll get some people on the phone and then we ask them, “What are your motivations going through this page?” Or run a usability test, which we’ll talk about in a moment where somebody is vocalizing their inner monolog as they’re going through.

Because I need to validate those hunches. Those are a little bit more expensive and time consuming to put together but they also, by this point, you might see I know what questions to ask right? So I’m saying, “Okay, find the right exercise headphones on the site. Plunk you in the front page of the Wire Cutter, right?

Okay, well I didn’t know that, that that was the right question to be asking. Once you get to the exercise headphones page, here are some more questions. “What is your impression of this page? Is it easy to understand what exercise headphones to be getting? Most importantly, is there anything holding you back from buying those exercise headphones? Is there anything that we can do to improve the credibility of this page overall and the trustworthiness of this page?”

Those are all pretty open ended questions, right? But they’re specific enough because we’re still trying to drill into our hunches. By that point, hopefully if we’ve done our job right and if we’ve got a decent enough recruit for their usability test, we’re going to get high quality insights from them that will allow us to rework the layout of this page or present it in a way that converts better and conveys trustworthiness a little bit more effectively. That’s when you run the AB test. You’re running that right there.

[0:24:26.1] LJ: It’s really interesting. I’m going to tell my dad about this because he is a retired maths and computers teacher and he’s going to love looking into the ramifications of this. Because it occurs to me that from a content marketing perspective, you can make blanket assumptions about, “Oh okay, look, I’m going to check these keywords, I’m going to do my research on the content and the audience and come up with a long form piece of content.”

What we’re saying, strategically, if you actually drill down to this on a statistical, scientific level, the piece of content then converts best for that audience with that particular way into the site could be a video with two lines of text under it.

[0:25:04.0] ND: It could be yeah or it could be a 40,000 word marketing page and there’s no way to know until we actually get in front of them and ask them. Historically, you can look at other blogs such as Copy Hackers or ConversionXL and figure out what circumstances along form pages more likely to convert and that helps. It’s best practices and it’s familiar to a lot of people but you can go one better. They call it optimization because you start with the best practices and then figure out where those breakdown for your business.

[0:25:36.8] LJ: For sure, okay and let’s finish up the quantitive research section with the evaluation of everything we’ve just talked about.

[0:25:43.4] ND: Yeah, this is the one that you can probably just do by yourself for zero dollars. Heuristic evaluation is an ancient usability technique going back to time and memorial. You have a checklist of heuristics and you are determining if the site actually fulfills those heuristics if they are applicable. So, for example, is it easy to check out on mobile? Are there any extraneous form fields? Does it ask for billing and shipping and if so am I delivering a digital product?

Pay well, get rid of the shipping address in that situation. Does it offer — I continued harping on mobile forms, but does it offer the right keyboard? So when I am entering the credit card does it switch the numeric keypad? Does it force autocorrect on addresses, which it really shouldn’t be doing because addresses are usually proper nouns? Like in Chicago, we have Diversey Avenue and if it autocorrects to “Diversity Avenue” then you’re doing it wrong, right? That is a bad address and you’re not going to get your thing.

Those are basically just you going through trying to perform transactions, trying to find things, pretending like you’re a user and trying to convert into a customer and these are things that a shockingly small amount of people actually do. One thing I love doing is going through the exercise where I make all of the developers at a site, I give them all dummy credit cards and make them sign up for the service and I’m like, “Tell me how frustrating that was? Just write it up.”

And they all rebel against it, they all completely hate doing this because they have better things to do, right? But then they go at through and actually do it and they’re like, “Oh my god, oh no,” and these are the things that are taking your money. That’s actually what’s funding your paycheck. The thing that is funding your paycheck is broken and so that’s another flip side of it but a heuristic evaluation, you’re going through yourself and just cleared eyed saying, “Does it meet this mark?” Checkbox, yes or no.

In e-commerce for example, are there related products, are they sensible to the actual thing that you’re providing, are there recommendation for products for that are particularly popular? If I am going through from a content marketing standpoint, is there a call to action at the end of the post? Is there a right rail that provides a similar call to action? Is the navigation sticky? Does extraneous meta data for it like by line and date and stuff like that does it fulfill the overall goals of the page or can we get rid of them for things that are more like evergreen content?

Because if you have evergreen content from 2011, people are less likely to trust it. Those are things that I would be asking in any sort of heuristic evaluation of a site. For my clients, I developed these bespoke because I’ve been doing it for a number of years but you can find these from basically any usability testing site just to make sure that it’s ticking the right boxes. It’s pretty easy to put together, you can do it probably in an afternoon.

[0:28:43.3] LJ: The two things that come out of that, a couple of quick stories, I’ve got a friend who works for a large government department here in the UK their website and he does exactly these type of testing but over reams, and reams, and reams of pages and paperwork and checkpoints to do and he said that and — he describes what he did is absolutely incredible, the amount of minute evaluation they do and things you wouldn’t think would be broken and assumptions again that you make are broken and this link doesn’t go there.

And you find out that people are going down Warren’s rabbit holes of usability that they shouldn’t be so you have to reel that all back in. That was really interesting talking about that but the other thing that reminded me of when you were talking was another friend and I can’t name this company because they’re a big UK brand, they didn’t do any of the research and testing we’re talking about and just thought they’d get their whole website, a massive project redesigned because it would increase conversions and then of course it didn’t because they weren’t basing on any evaluation.

[0:29:45.4] ND: But it looks nicer, right?

[0:29:46.3] LJ: It looks nicer and then everyone twiddling pencils and twiddling their fingers saying, “Yeah but it doesn’t convert as well as the one that didn’t look quite as nice. Who did the research?” And that’s a very good example of why you have to start with this. So let’s move onto the qualitative research methods, we’ve got user interviews, usability test and surveys and you’ve touched upon some of these already.

[0:30:07.9] ND: Yeah, so user interviews, I’ve already described those. You are getting somebody on the phone. One thing I didn’t really talk about is recruitment. There is a utility called Ethn.io that basically puts a screener on your site and it asks for customers like contact information, whether they’d be cool with an interview. It’s very focused in that way. But then, you can add in qualifying questions. You want to generally cut out about 50% of the people at least that are coming in the door and to recruit from a variety of different used cases and demographics. So you might get like a power user or a total novice, somebody who’s coming to the site for the first time.

You basically send them a link, I use a website called Calendly, it was how we organized this call, and it connects with your Google Calendar to find available time. You provide a little bit of air gap before and after the call so that you could prepare and decompress psychologically after it because interviews wipe you. You are not a hero enough that you can just bang-bang-bang, do a bunch of interviews and you get people on the phone and ask a bunch of questions, listen to their story and try not to push back because the customer’s right in this situation. It is often a surprising insight that you can drill down into a little bit more.

Oh yeah, and you compensate them for their time. I give people usually an Amazon gift card or like a freebie or a free service or something like that.

[0:31:36.1] LJ: Okay and then this dove tails into those usability test as well because then you are starting to be informed by what you’ve discovered.

[0:31:44.9] ND: Yeah, for sure. Yeah so the usability test are, you get a recording of somebody on a computer. Usually they’ll turn on like the eye sight camera on the top and record the screen and you will get a picture and picture thing and you are asking them to complete various tasks, and vocalize your internal monologue as much as humanly possible the entire time.

Then you get backup video and you just watch the video determine where things are breaking down, what surprised you, was it easy for people to find, were they pleased, were they frustrated, what emotions were they feeling during it? It’s a combination of qualitative and quantitative because if somebody converts but they were angry doing it, that’s not good either right? That’s an opportunity to begin the customer relationship on a little bit better footing.

[0:32:30.3] LJ: And I can remember just looking at surveys as well because I want to get into what we do with all of these information at how we set up meaningful AB tests in a minute but the last qualitative research method is surveys really and I can remember doing one of these as a teenager and it is about the questions you ask because of course, if you asked closed questions you get nothing. I can remember just saying, “Yes, no, yes, no, yes, yes, no,” and then I was allowed to go. I thought, “Well that’s not really very helpful to either of us is it?”

[0:33:00.0] ND: Yeah so there are a ton of conversion focus surveys that you can do. Typeform and Wufoo are both good, cheap and free utilities. I think they both have free plans and you are getting a combination of qualitative and quantitative insights from this too. There is something called the net promoter score, which if you haven’t heard, you probably filled it out at some point. “On a scale from one to ten, how likely is it that you would recommend this product or service to a friend or a colleague?”

You know that one, right? And there’s an automatic calculation for it that came up from this one fancy pants consulting firm and whatever have you. That’s very quantitative, right? It’s literally a one to ten but then on forms, you can add in free text fields and I love asking what are called the brainautic questions, which are usually for testimonials. So I will send these after you get a product and I forgot the six, I’ll list them on the show notes but it’s like, “What was the thing that held you back before you bought this? List the number one benefit. List three more benefits. Is there anything else you would like to add?”

There are six questions, I think I just listed four of them, but you get the idea. It’s trying to get a sense of somebody’s thought process as they went through the process of conversation. Maybe they came back to the site three or four times and hesitated before they actually bought it, what held them back? Was that valid it’s like a $5,000 service. People tend to balk on those things for a little bit or get approval from the rest of their organization for B2B products, that sort of thing. That’s a good starting point for putting together a survey that offers a lot of qualitative insights.

You know what I do with those surveys? I will send them out automatically after somebody purchases, or I will throw it on the front page of the site as an annual survey and get as many people as humanly possible. Each of those are very different research techniques that offer different insights. One of them you get in basically continuously and then sometimes, I’ll get back a survey that is particularly interesting and I’d be like, “What’s your phone number? Can I call you? Here’s my Calendly link,” and then we’ll have more interviews because somebody said something that was particularly interesting and now they get a free year of the service or an Amazon gift card or whatever have you, and then we have a little bit more to go on as far as AB testing insights.

So just to sum up like through this, I’m never just running a test and then sitting back and waiting three weeks for the results. I’m always thinking about the next test to be running and there’s got to be a sort of a pipeline that comes along with this. So there is research, I’m always going research in some capacity and it can be looking at heat maps, it can be conducting interviews, whatever. I’m coordinating with developers and designers in the client’s team usually or I am hiring them of my own to build these out as prototypes because we have to make them into something that’s testable and sometimes that actually requires a significant development effort.

[0:35:49.2] LJ: Okay and this is the key pivot though, this is the key pivot for me as a small business owner or content marketer listening to this, you’ve got a ton of experience then informing you when you’ve got all of these research, ongoing research, what questions you ask of that research and this is crucial because I guess this is where you can get it wrong. So how do we do this? How do we go about asking the right questions informed by that research?

[0:36:14.2] ND: That’s another three hour answer there. The very broad stroke answer is, are there opportunities to capture more revenue in whatever we’re looking at and if so, what shape does that take? So sometimes it’s a very clear cut like your form to checkout has 58 fields and it needs three. Okay, great get rid of the 55 fields, solved. Sometimes it’s much more squishy and I think that’s why you’re asking this question.

From a content standpoint, you’re appealing to specific desires from the customer and so let’s do something that is just very theoretical but not unreasonable that involves both quantitative and qualitative methodology. I post something to my blog and I think it’s completely amazing and super evergreen and really, really helpful for people. Very few people go to it, very few people link back to it and very few people are scrolling when they actually get there.

All of those are quantitative things that I am pulling off of Google Analytics, heat maps, and scroll maps and so I’m taking a look at that. Okay, well if that’s the case maybe I didn’t have correct what the motivations where the people were coming in, or maybe they were already fulfilling those needs somewhere else or maybe this wasn’t ranking well on Google and we need to find some way to handle the SEO of it better.

There are a lot of different potential explanations for this, and this is the moment right here that having 10 years of experience in this industry is much more valuable. I might search on Blog Post to be like why isn’t my content performing well. If you are particularly expert in this, you might have other ideas of your own around it. Talking to colleagues and brainstorming those ideas, I will be the very first person to admit that I do not have all the answers to absolutely everything that I do and so a lot of the time, I’m like, “This isn’t working and I don’t know, can we get on a call and brainstorm some ideas why?”

Usually my client and other team members they come in a little bit frustrated because there is an emotional impact of, “This isn’t working.” But then they also wonder, “Okay, well what are the potential explanations for it?” None of those are right. We don’t know because we haven’t followed the hunch through to figure out what those are but now that we have the questions and the hunch is on them, we can start conducting interviews or usability tests that ask open ended questions that lead us to confirm or deny whether those hunches are the case.

Then we start to validate those ideas and that’s when we can actually take action. I think the hardest step within this is that brainstorming, because you can also come up with an idea that might be the wrong idea and then you run into this Dunning–Kruger unknown-unknown thing where you’re very, very sure of an explanation that turns out to be wrong and then you run an AB test and it tanks and there’s no way around it, it’s very demoralizing.

Dealing with the impact of that, you’ve done a lot of work and then you have a failing test. But much of the time, most of the time for me at least, I end up with a test that moves a little bit more in the right direction and shows that I have hit on something and, “Well that’s interesting, okay let’s explore that a little bit further.”

[0:39:33.5] LJ: We could talk for three hours on this because it’s fascinating, but wouldn’t it be great if you’ve written a book about it and you have.

[0:39:40.4] ND: Shocking.

[0:39:41.4] LJ: It’s called The AB Testing Manual and I’ve had an advance copy and it’s fascinating. So tell us about the book if we want to learn more about this and where can we get it from.

[0:39:51.4] ND: Yeah, so the book is a part of a whole course actually. So if you go to abtestingmanual.com, it’s a very creative domain name, you can take a look. There are a bunch of components to it. There is five and a half hours of videos that I recorded with my friend, Patrick McKenzie. He developed the first ever AB testing framework for Ruby on Rails and now does content for Atlas at Stripe.

I flew him in from Tokyo and we banged out five and a half hours of amazing actionable video around kind of business and procedural things, kind of things that I am talking about here and then the book itself is I think went 230 pages split between research, actual how to run the AB test, and then what to do with it operationally in your organization and then it comes with all of the checklists and tool kits and everything that I use with my clients.

So if you’re wondering how do I run here a heuristic, buy the AB Testing Manual. How do I set up a project management system so that I can come up with AB Testing ideas and shepherd them through the process, well, I have a Trello board available for you in the AB Testing Manual. You are getting the idea. So I wrote this whole thing, it came out as of this recording like two or three weeks ago and about 400 people have bought it, really liked it so far. It’s been going really well.

[0:41:08.7] LJ: Fantastic, well look just before we go onto the PS question, where else can we find you online, Nick, aside from that?

[0:41:15.1] ND: Draft.nu is my business and abtestingmanual.com is the book. If you want to sign up for — I write a weekly letter to people. I draft.nu/letters and a lot of it is about AB Testing, some of it is other random things such as the best deep dish pizza in Chicago. So if you ever find yourself lying here but yeah, that’s pretty much it.

[0:41:20.3] ANNOUNCER: Wait for it listeners, here comes the PS Question.

[0:41:55.3] LJ: Okay, this is what I call the PS Question. Before you go, could you please share one advanced AB testing tactic that we can use right after the show?

[0:42:04.5] ND: Yes, advanced AB testing tactic? So my recommendation would be if you have a ton of traffic coming into your site, so this is definitely more advanced. If you are getting more than probably like I would say 2,000 transactions every month, run what is called — and this is intermediate to advance. Run what’s called a bandit experiment and it basically uses an algorithm to determine whether or not a variation is converting better or worse and automatically ramps up or down the proportion of traffic that is going in.

So remember when I said that AB testing is 50-50? Not so with a bandit experiment. If your variant performs really, really well all of a sudden it’s going to ramp up to a hundred. Now here’s where it gets advanced, what are the criteria that proposes that ramp up? Like when do we turn off a given variation and give up on it? Bandit algorithms have a varying degree of greediness that you can ascribe to them. So they can be like, “Well we might be sure about this, we’re going to let it go for a little bit longer.” Or they’re just like, “Nope, we’re done”.

If you can take a look at varying degrees of greedy bandit algorithms for your business, setting up a bandit algorithm is easy. Fine tuning it is the much harder thing. You can do this within a week if you have a really wonky data science background. Or you can just put together a bandit test within a day using your normally AB testing framework. There’s usually like a checkbox that they have that’s like wrap it up or down and it’s like, “Okay great. Are you greedy or not? Okay, great there’s your pull down.” That’s something you can do immediately after this show and begin thinking about more advanced intelligent, like artificially intelligent ways to optimize your site.

[0:43:51.3] LJ: And just as a P.P.S. Question if you would, how far wide are we from having an algorithmic AB testing system where we can just literary plug everything in, press a button and it will spit out, you need X, Y, Z to increase your conversions, or is that a long way off?

[0:44:09.6] ND: I bet those already exists in Fortune 500 companies. I bet thinking of very data driven companies like Amazon, Netflix, Facebook, eBay, they all run AB tests all the time and are ramping up or down variations based on this. I bet they’re probably using something with some antecedent machine learning already. I bet it exists.

I think that it’s a matter of that is going to start at the biggest of organizations that have the most amount of traffic. The real question is when it trickles down to me and you listening to this. You probably aren’t in that bucket and I think we’re probably three or four years off from that happening, but I think it’s there and you probably use it day to day. I think it exists.

[0:44:56.9] LJ: Fantastic, well that’s a fascinating interview. Great answers to the P.S. and the P.P.S. question and all that remains for me to say Nick is, thanks very much for your time and I wish you all the best with everything in future.

[0:45:08.9] NB: Thank you so much. It’s an honor. I hope this is helpful.

[OUTRO]

[0:45:14.0] ANNOUNCER: You’ve been listening to The Content Champion Podcast, available at contentchampion.com and on iTunes. Until next time, thanks for listening.

[END]