Hi, let's see, in this set of videos, we're going to talk a little bit about A/B testing. It's a commonly used technique in marketing, and it's become even more so with the online marketing capabilities. So why don't we just dive right in and begin. So first, let me talk a little bit about what is A/B testing. That's really just the process of comparing two groups. You might have heard a test group and a control group. This is often used in science or the scientific literature. A control group would be random selection, random sample where you do nothing, and then on another half of the group, the test group, you would perform some treatment. So one group will not get the drug and one group will get the drug, and that'll be the treatment group. Often in the marketing analytics world, this is referred to as A/B testing, more than testing control group. It's really a matter of vocabulary. Two groups usually means two versions of some market asset. So what do I mean by that? A market asset is something you want to research such as, if it's a webpage, a title, a layout of a design, perhaps you're testing gets a new marketing slogan or a phrase, a feature like the color of a website button, a font, all these things are compared. So you might run the webpage, your original webpage that you've been using for awhile, and then you might tweak it to test for something. The only difference between the two groups is that the test group has the treatment, and the control group does not. This is usually referred to as A/B testing. So here's an example that I alluded to earlier in digital marketing. This is an example of a webpage experiment. Then say you have these two webpages, and you're trying to decide whether or not to use green button which you can see down here, there we go, a green button or a blue button, and you're trying to understand which one attracts more clicks. Which one actually gets the user involved. At some level, this might seem like a superficial change, but these are the types of things that you will want to consider if you're working on a digital marketing campaign. So the overall procedure for A/B testing is, you have some product design. So if you're designing some widget, you might add a feature or not add a feature. Performing A/B test, do the analysis and see if what you expected it to happen happens, or doesn't happen. Then if it works out okay, you can launch the product. Within A/B testing, you design your experiment, you conduct your data analysis, develop your hypothesis, and if your changes support your hypothesis, you can go into product review and product launch. This slide just basically tells you what I just said. You want to select the test treatment. In A/B testing, you're only testing one feature at a time. You want to make sure you have a clear goal in mind. What is your objective? What are you trying to do? Then associated with that is, what is your metric? How are you going to measure the change? You want us create a test group, and a control group? Generally speaking, you want to split your groups evenly and randomly, and you want to determine your sample size so that you have enough data, and then you collect the data and you perform your analysis including hypothesis testing. So step one, let's drill down into these in detail. Select one independent variable that you want to test. In the blue green example, the color of the "Learn more" button is the only variable. You can test multiple variables, but you want to do this one at a time. There are a lot of companies that just exploit this especially in the digital marketing error. They will test one feature to one set of customers, another feature to another set of customers compare, and they just iterate through this process continuously. Step two is to identify the goal and the metric. So this sounds easy, but in fact, it is probably one of the more challenging aspects of a good analytics program, and you want to identify the goal along with how you're going to measure that test. Sometimes in the blue green button example, is just the number of clicks. So the metric is very obvious, but sometimes it's not so obvious. If you change your metric, you might want to set up a different testing procedure. So that in itself is not too hard to understand, but you want to keep that in mind and have good records when you compare across tests. So you have older A/B tests that you're looking at. You're looking at their data, and then somehow you switch objective or maybe your metric and you're not going to be able to compare across tests as easily. So the next step in the process is to create a test group, and a control group. Ideally, in a true experiment, we want to do this using random assignment. We're assessing the proposition if x then y. So in the blue green button example, if the button is green, then clicks will increase. If not x, then not y. If the button is not green, then the clicks will not increase. This proves the cause of the effectiveness. Generally speaking, you want two equivalent groups, and we randomly assign subjects into the control group, or the test group. Generally speaking, by convention, the control group is A and the treatment group is B. The fundamental idea here is randomization, which randomly allocates the subjects into the two groups. This is the most ideal form of an experiment, and it's called the true experiment. Mostly because it controls for many other extraneous factors. Another thing you're going to want to consider is the sample size. How much data are you going to collect? A/B testing uses this notion of statistical significance to determine whether or not a treatment works or doesn't work. So sample size is a key variable. Basically, the larger the sample size, the higher your confidence level is in your experiment. So before the test, you should think about the optimal sample size, as well as the statistical significance level. Generally speaking, we use a significance level of five percent and a confidence interval of 95 percent. The things to consider here are that with sample size, remember you're in a corporate environment and collecting data is not necessarily free. So if you're doing a field experiment, you are going out and testing markets in a retail setting. If you want a lot of data, that might cost a lot of money to collect. So trying to find the maximum amount of statistical power and using the minimal sample size to get that power is something that you're going to want to consider. Then one last point about statistical significance level of five percent, these are scientific conventions. But if you see something maybe 5.01 percent, or even six percent, it's worth looking at that a little further to see if that's something that's worth investigating. So how do you get the sample size? It's right here. The formulas, let's talk through this formula real quickly. So here we have Alpha. We'll start there, and that's your significance level. Your 0.05, for example, and this is a normal distribution and it's going to be a two-tail test so that's why I divide over two. So if you have a normal curve here, standard normal. Z is distributed, standard normal 0,1. What is that value out here? So that's the z, and then you square that term as a sum estimate of your variance. If you can actually calculate the variance that's ideal. Sometimes people use this rule of thumb estimate, which is the range. The maximum value minus the minimum value, divided by four. If you can't get anything better, so that's what this term is. Then you divide that by your effect size, and your effect size is how far apart your testing. So if you think you're going to get your test is a 100 more clicks with the green button, your effect size will be that value there. Then if you just plug in the formula, you will get the n value here or the number of your sample size. Collect and analyze data is the fourth step in your analysis. Make sure you have enough time to obtain a solid sample size. Avoid conflicts between your test and other team's test, project, or company business plans. So there might be other data analytics efforts going on in your company, and you don't want to clash with their data, if you're doing field experiments, or trial runs, you want to make sure that you're free of outside influences in your experiment. You can also have a comprehensive data collection plan, a backup plan to ensure reproducibility. Reproducibility is something that's really important. You want to be able to make sure that if someone come back to you in a month and says, "Can you rerun that analysis?" You can do it, and should have the same results. Then they might ask you to extend your statistical analysis using different tests and different techniques. So that's what your ideal form is. So now that I've talked about the overall general process of running an A/B test, in the next video I'm going to talk about specific statistical test under different scenarios.
