{"id":2360,"date":"2016-04-02T17:30:25","date_gmt":"2016-04-02T12:00:25","guid":{"rendered":"http:\/\/codetheory.in\/?p=2360"},"modified":"2016-04-09T14:14:41","modified_gmt":"2016-04-09T08:44:41","slug":"split-testing","status":"publish","type":"post","link":"https:\/\/codetheory.in\/split-testing\/","title":{"rendered":"Guide (Introduction, Implementation and How it Works) to A\/B Testing and Split URL Testing (VWO, Optimizely, Google Analytics)"},"content":{"rendered":"<p>Recently I\u2019ve been investing a lot of time into A\/B (split) testing\/experiments and analysing their results. Slowly I&#8217;ve started to love this way of rolling out changes on the web (can be surely used for mobile apps as well), as it is a complete data-driven approach and can produce quite surprising (or shocking, how you see it) results at times. With this approach you&#8217;ll always know what works (instead of speculating) with firm evidence and eventually drive more sales or signups or whatever that matters to you. That said, I would recommend not A\/B testing just <em>anything and everything<\/em> but always find the right situations or cases that would make sense to experiment. If you&#8217;ve enough bandwidth (extra time) then use this approach to keep on analysing your visitor behaviour and <em>draw learnings<\/em> that help you make more prospects achieve the objectives that you&#8217;ve set for them, eventually turning them into customers.<\/p>\n<p><!--more--><\/p>\n<p>I&#8217;ll start off with basic (yet comprehensive) introductions to the different types of website testings (surely these techniques can be exercised in a mobile app too) and then cover how you can start doing testing right today using various third-party easy-to-use softwares\/tools. Finally I&#8217;ll also explain how it all works which is going to be more of a technical piece (for the technically-interested folks).<\/p>\n<p><strong>Heads up:<\/strong> The article might seem lengthy but I ensure it&#8217;s not going to take much time to complete. The content flow should be nice and easy and the overall complexity level is fairly low. So without further ado, let\u2019s begin!<\/p>\n<h2>The Tools<\/h2>\n<p>Before we actually kick off with all sorts of information, let me tell you that I\u2019ve used these three tools for A\/B testing:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.optimizely.com\/\" target=\"_blank\" rel=\"noopener\">Optimizely<\/a><\/li>\n<li><a href=\"https:\/\/vwo.com\/\" target=\"_blank\" rel=\"noopener\">VWO<\/a><\/li>\n<li><a href=\"https:\/\/support.google.com\/analytics\/answer\/1745147?hl=en&#038;ref_topic=1745207&#038;rd=1\" target=\"_blank\" rel=\"noopener\">Google Analytics Content Experiments<\/a><\/li>\n<\/ul>\n<p>We started off with Optimizely for our A\/B testing needs but eventually moved on to VWO as we found it more intuitive and well designed in terms of UI\/UX. It is easier to use (IMHO) as well as has some extra nifty features here and there. I\u2019ve also used Google Analytics Content Experiments in the past, though it is very limited feature-wise. Surely it can get your job done though if you&#8217;re a technical person who can modify HTML\/CSS\/JS (and server-side) code several times to adjust the test. Other tools will let you create multiple goals and assign them to your experiment and then help visualize the data\/reports in a much intuitive way. Along with heatmaps, user segmentation and lots of other engagement data, VWO or Optimizely will prove to be better than GA Content Experiments.<\/p>\n<p>My objective is not to evoke a sense of bias in you with all these information but more to give you a fair idea about the various website\/app testing tools out there in the market. Also having used all these tools, you could say I&#8217;ve a fair idea of how A\/B testing is done and the final data is analysed practically.<\/p>\n<h2>A\/B (Split) Testing<\/h2>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing.png\" rel=\"attachment wp-att-2442\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing.png\" alt=\"a\/b (split) testing\" width=\"1100\" height=\"500\" class=\"aligncenter size-full wp-image-2442\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing.png 1100w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing-300x136.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing-768x349.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing-1024x465.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Testing-50x23.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>Let me explain A\/B or Split testing first. A\/B testing basically means that you have two different versions of your website (V1 and V2), where you show V1 to 50% of your traffic and V2 to the other 50%. Whichever performs better in terms of conversion is your winning version (or variation) that you finally start showing to 100% of your traffic. It\u2019s that simple!<\/p>\n<p>Do note that it is not necessary that the two variations have to be radically different, like two completely different web designs altogether. Instead of two entirely different websites (multiple pages), the two variations can just be a single web page with a small\/medium difference in the content. For instance imagine an e-commerce product page where in V1 (or original version) the product image shows on the left and the product title\/description is on the right where as in V2 (new variation) the placement is quite the opposite (content on the left and image on the right). Instead of changing the entire layout, you might consider doing minor changes like changing the heading and an image. That is also perfectly valid.<\/p>\n<p>So basically you can either test macro changes like an entire website re-design or even a micro change like a change in title of a particular web page.<\/p>\n<p>Of course, factors like the number of variations and traffic split are variable:<\/p>\n<ul>\n<li>You can have multiple versions\/variations &#8211; V1, V2, V3 \u2026 Vn<\/li>\n<li>You can split traffic in any percentage &#8211; 50\/50, 40\/60, 80\/20. With 3 variations the split could be &#8211; 33\/33\/33, 25\/25\/50, 55\/25\/20 and so on.<\/li>\n<\/ul>\n<p>If you\u2019re using a nice software (like one of those tools I listed above) you can even play with the percentage of your traffic you want to include in your experiment. You could include only 20% of your traffic in the experiment or even 40% (or whatever number). So if you have an experiment with 3 variations (V1, V2, V3) where the traffic split is (25\/25\/50) and the traffic under consideration for the test is 40% then you effectively end up sending 25% of 40% of-entire-traffic to V1 and V2 where as 50% of that same chunk to V3. Good one!<\/p>\n<h3>Controls and Variations<\/h3>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations.png\" rel=\"attachment wp-att-2441\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations.png\" alt=\"control and variations\" width=\"1100\" height=\"500\" class=\"aligncenter size-full wp-image-2441\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations.png 1100w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations-300x136.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations-768x349.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations-1024x465.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Control-and-Variations-50x23.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>It is very important to get these two terms right with regards to A\/B\/n (more than one variation) testing. When experimenting you&#8217;ll have your default version which is called <strong>control<\/strong> and one or more other versions which you want to test against the control. These one-or-more-other-versions are called <strong>variations<\/strong> technically. So you&#8217;ve your <em>control<\/em> version of the website where you make small\/medium\/large changes to one or more pages (tiny headline changes to complete web re-designs) to create one or more variations.<\/p>\n<p>So A and B are different versions of a property (website\/webpage or app\/screens) where you can call A as the control and B as a variation. Although <em>A\/B testing<\/em> is also used to refer to experiments where you have multiple variations (control, V1, V2, V3), but some also refer to such setups as <em>A\/B\/n testing<\/em> which is technically more appropriate.<\/p>\n<h2>Multivariate Testing<\/h2>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing.png\" rel=\"attachment wp-att-2444\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing.png\" alt=\"multi variate testing\" width=\"1100\" height=\"500\" class=\"aligncenter size-full wp-image-2444\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing.png 1100w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing-300x136.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing-768x349.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing-1024x465.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Multi-variate-Testing-50x23.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>There&#8217;s another type of split testing called Multivariate testing. Understanding this piece is fairly simple. Imagine the pricing page of any SAAS business. The page will have a heading, a sub-heading probably and then a couple of pricing plans. There will obviously be more information below the pricing section but we&#8217;ll constrict our focus on this part of the fold only.<\/p>\n<p>You could definitely test different headings on the pricing page like (imaginary examples):<\/p>\n<ul>\n<li>Start your 14-day Free Trial<\/li>\n<li>Signup for [Business\/Product Name]<\/li>\n<li>Amazing Plans Just For You<\/li>\n<\/ul>\n<p>Send a 33\/33\/33 split of your traffic to each version\/variation and you&#8217;re A\/B testing already. But imagine a case where you had 3 different types of headings, 3 sub-headings and 3 different designs (maybe placement-wise or UI-wise) for the pricing plans. Now you want to test a possible combination of all these elements to see which will perform better. In that case, you might end up having 27 (3 x 3 x 3) variations or versions to test where each will receive a 1\/27 split of your traffic. This type of testing where you have different types of elements of which you want to see which combination performs best in terms of conversions is called Multivariate testing.<\/p>\n<p>Bonus: See how Highrise had <a href=\"https:\/\/signalvnoise.com\/posts\/1525-writing-decisions-headline-tests-on-the-highrise-signup-page\" target=\"_blank\" rel=\"noopener\">tried different headings<\/a> (ages ago) to increase signups on their pricing page.<\/p>\n<p>So as a rule of thumb, whenever you have different versions of different elements that might form multiple combinations that you want to test among your users, Multivariate is the way to go! Don&#8217;t confuse this case with one where you have 2 different versions of your heading (H1, H2) and 2 different versions of an image (I1, I2) but you just want to test the combinations &#8211; (H1, I1) and (H2, I2) &#8211; this is a normal A\/B testing candidate, not a Multivariate one.<\/p>\n<h2>Split URL Testing<\/h2>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing.png\" rel=\"attachment wp-att-2445\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing.png\" alt=\"split url testing\" width=\"1100\" height=\"500\" class=\"aligncenter size-full wp-image-2445\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing.png 1100w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing-300x136.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing-768x349.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing-1024x465.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Split-Url-Testing-50x23.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>What the heck is this now ? Split URL testing ? It&#8217;s all like A\/B testing only but just that the versions (variations) exist on different URLs either on the same domain or different domains. In A\/B testing if you have two different versions of a particular component on the home page, you can drive 50% traffic to each version right on to the same URL. But in this case, you can have the different versions hosted on different URLs and drive the traffic split to the various URLs. Here are some URL examples for two variations:<\/p>\n<ul>\n<li>http:\/\/example.com and http:\/\/example.com\/variation1<\/li>\n<li>http:\/\/example.com\/variation1 and http:\/\/example.com\/variation2<\/li>\n<li>http:\/\/example.com and http:\/\/anotherexample.com<\/li>\n<li>http:\/\/example.com\/variation2 and http:\/\/anotherexample.com\/variation2<\/li>\n<\/ul>\n<p>You&#8217;d generally want to do this type of testing when you&#8217;re testing two majorly different versions of the same page. Basically imagine a new design of your home page altogether. There have been times where for major re-designs like a new website design altogether or a new mobile responsive version of the current website, I used the normal A\/B testing to set custom cookies and then reload the page basis the variation cookie that was set. Not only was this method weird but it also skewed the results at times when the refresh would happen too late or just go wrong. It&#8217;d also impact other tracking tools which just didn&#8217;t make any sense. Split URL is a a great candidate for this kind of testing and I wish I had gone this route back then.<\/p>\n<p>Bonus: There&#8217;s a Highrise case study on how they tried totally radially different variations to boost their conversions. Here are <a href=\"https:\/\/signalvnoise.com\/posts\/2977-behind-the-scenes-highrise-marketing-site-ab-testing-part-1\" target=\"_blank\" rel=\"noopener\">Part 1<\/a>, <a href=\"https:\/\/signalvnoise.com\/posts\/2983-behind-the-scenes-ab-testing-part-2-how-we-test\" target=\"_blank\" rel=\"noopener\">Part 2<\/a> and <a href=\"https:\/\/signalvnoise.com\/posts\/2991-behind-the-scenes-ab-testing-part-3-final\" target=\"_blank\" rel=\"noopener\">Part 3<\/a>. You&#8217;ll learn loads from them.<\/p>\n<h2>What are Conversions and Goals ?<\/h2>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions.png\" rel=\"attachment wp-att-2443\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions.png\" alt=\"goals and conversions\" width=\"1100\" height=\"500\" class=\"aligncenter size-full wp-image-2443\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions.png 1100w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions-300x136.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions-768x349.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions-1024x465.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Goals-and-Conversions-50x23.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>I&#8217;ve been mentioning this keyword all around but what exactly is it ? Conversions is basically what you care about. It is the success rate of an expected Goal that you&#8217;d have or set in your A\/B testing software or process. Let&#8217;s understand with a couple of examples.<\/p>\n<p>Let&#8217;s say you re-designed your homepage altogether. First version had a &#8220;Signup&#8221; button for your product that lead to a signup page or opened up a signup modal where as the second version has a signup form right there above the fold (in the initial viewport), right when the page loads. Seems easier to signup in the second version right ?<\/p>\n<p>In this case you clearly &#8220;care&#8221; about signups. So that is going to be one of your goals. Goals are what you think is going to be a goal or an achievement of the visitor\/user on your website. Different tools let you setup this goal for an experiment in different ways. You can either:<\/p>\n<ul>\n<li>Setup a goal to track page visits on the successful page that you see right after the signup.<\/li>\n<li>Setup a goal to track the signup submit button click but this is not entirely fool-proof as there can be validation errors and the clicks will increment the total count in your reports\/results, hence increasing the conversions.<\/li>\n<li>Setup a goal to track form submits to a particular URL (your signup URL in this case).<\/li>\n<li>Setup a goal to track something else like triggering an event.<\/li>\n<\/ul>\n<p>Effectively what happens is that, in the results you will see the total number of people landing for each variation on the home page and then if you are have setup a Goal that tracks page visits on the successful page then you will see the total number of people who reached there for each variation which gives you the conversion number and hence the conversion rate. Here&#8217;s an example:<\/p>\n<ul>\n<li>V1 (50\/1050) &#8211; 50 people out of 1050 who landed on the homepage in V1 converted (4.76%).<\/li>\n<li>V2 (200\/1100) &#8211; 200 people out of 1100 who landed on V2&#8217;s homepage converted (18.18%).<\/li>\n<\/ul>\n<p>Clearly the second variation has a way higher conversion number\/rate for your Goal and hence you will want to update your design with that same version.<\/p>\n<h2>Split Testing Scenarios<\/h2>\n<p>Let\u2019s explore a couple of different scenarios where you will probably want to A\/B test:<\/p>\n<ul>\n<li>You started your company six months back and launched a brand new website for it. Now you find that design quite antiquated and not so intuitive because your business model has changed a lot in the last six months. You come up with a new design of the website but it <em>might be<\/em> a bad idea to just go ahead and launch it. Instead you should use a third-party software (one of the above) or have an internal implementation to A\/B test to split your traffic between the old and the new designs and see which performs better in terms of conversions.<\/li>\n<li>You&#8217;ve decided to try different design for a particular component. The component could be as simple as a button or as complex as an entire search widget or a header or that signup form on your home page right above the fold. I&#8217;d say definitely A\/B test that. Of course don&#8217;t start A\/B testing very small things like useless buttons or tooltips here or there but do test what could improve your entire funnel and what matters for your business. A button is not worth A\/B testing unless it leads to a signup or purchase conversion and that&#8217;s what you care about.<\/li>\n<li>You want to build a mobile optimized\/responsive version of you desktop site.<\/li>\n<\/ul>\n<p>Be really really smart with what you A\/B test. Don&#8217;t waste a lot of time making silly variations that wastes your time and energy in first thinking about them, then making the changes and then analysing the results. I&#8217;ve seen a lot of people make that mistake and it&#8217;s not worth it, trust me! It could also possibly piss off other people with whom you work in tandem.<\/p>\n<p>Bonus: <a href=\"http:\/\/blog.wishpond.com\/post\/98235786280\/50-a-b-split-test-conversion-optimization-case-studies\" target=\"_blank\" rel=\"noopener\">Here<\/a> are <a href=\"https:\/\/vwo.com\/blog\/10-kickass-ab-testing-case-studies\/\" target=\"_blank\" rel=\"noopener\">some<\/a> case <a href=\"https:\/\/blog.optimizely.com\/tag\/case-studies\/\" target=\"_blank\" rel=\"noopener\">studies<\/a> for <a href=\"https:\/\/vwo.com\/resources\/casestudy\/\" target=\"_blank\" rel=\"noopener\">fun<\/a> with surprising uplifts.<\/p>\n<p><strong>Protip:<\/strong> Although these case studies might evoke that urge to start A\/B testing right away, don&#8217;t be random with it. What worked for those brands may not work for you. Don&#8217;t start making small changes like changing the color of a CTA or modifying the headline expecting major uplifts like those case studies. You&#8217;ll actually end up wasting weeks or months in time. Feel free to take some of those numbers with a pinch of salt and use the content as an inspiration to start researching and analysing how you could bring about a radical change on your website leading to a radical uplift in the conversion and hence revenue\/engagement.<\/p>\n<h3>A\/B, Multivariate, Split URL &#8211; When to Use Which ?<\/h3>\n<p>It&#8217;ll be good to have some thumb rules or at least basic ideas around which type of testing to use in which scenario. You probably know the answer by now but I thought a devoted section will reinforce the mindset.<\/p>\n<p>Use the <strong>normal A\/B test<\/strong> for almost any sort of new feature launch or changes on any webpage. Following are some cases that are good candidates for plain simple A\/B tests:<\/p>\n<ul>\n<li>Redesign of an entire component like a header, footer or search widget.<\/li>\n<li>Modifying few elements on a particular page or even multiple pages like heading, buttons, tables, forms.<\/li>\n<li>A new UI change like some form of competition comparison indicator, videos or images addition\/removal, layout changes, new UI-intensive feature launch, etc.<\/li>\n<\/ul>\n<p>Note: Do use your intelligence to experiment only when one is required. If something will impact your conversion funnel that matters, then only go the A\/B testing route. If it is about some FAQ design change where you or your designer are pretty sure that the UI is getting a decent revamp, don&#8217;t go the experimentation route as it might not be fruitful at all.<\/p>\n<p>If you ever want to play around with the combinations of multiple versions of multiple elements, go ahead with <strong>Multivariate testing<\/strong>. Although keep in mind that a multivariate experiment will simply require a lot of traffic to produce a reasonable winning result\/combination. Why so ? Because the number of combinations will be very high. It will increase massively with the addition of a single variation or a single element. This is the formula that you&#8217;ll always have to remember:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nTotal # of Variations = &#x5B;# of E1 Variations] x &#x5B;# of E2 Variations] x &#x5B;# of En Variations]\r\n<\/pre>\n<p>&#8230; where the total # of variations is actually all the combinations up for testing and E1, E2 &#8230; En are the number of elements with their multiple variations.<\/p>\n<p>Finally, <strong>Split URL testing<\/strong> should probably be only used when testing major re-design. Like a completely different homepage (or product, checkout, payment page) redesign or a website redesign. So out of 100% traffic coming to yoursite.com, 50% will see V1 design of homepage (and\/or further pages) whereas 50% users will be redirected to yoursite.com\/variation2 to see V2 design of homepage (and\/or further steps\/pages). If you&#8217;re wondering how this is technically implemented then we&#8217;ll be covering that in a bit.<\/p>\n<p>Another case for Split URL is a brand new mobile version\/design of your current site, which is sort of similar to the previous case.<\/p>\n<p>For me, 9 out of 10 tests have been A\/B, 1 Split URL and probably none Multivariate. YMMV!<\/p>\n<h2>Split Testing in Action<\/h2>\n<p>We&#8217;ve spoken a hell lot about the concepts but how is it all put into action ? Well, for that you can (and probably should) use third-party tools like VWO, Optimizely or Google Analytics content experiments as I already mentioned earlier. I&#8217;ll talk about the implementation keeping VWO in mind (and using it as examples) but it is not much different from Optimizely (except the UI\/UX) as well as conceptually much similar to content experiments. Surely you can build something in-house too or have a plain simple internal A\/B testing framework coded, but the implementation process won&#8217;t be much different from using something like VWO, hence whatever content follows won&#8217;t be irrelevant.<\/p>\n<p>I&#8217;ll list the integral steps that are generally required to setup and run an experiment on VWO as well as GA Content Experiments.<\/p>\n<h4>1) Create a Campaign<\/h4>\n<p>In VWO go to the create section, you&#8217;ll find multiple options (A\/B test, Split URL, Multivariate, etc.). We&#8217;ll just go ahead with A\/B Test type as that&#8217;ll be enough and you should easily be able to figure out the others once you know this one.<\/p>\n<p>Creating an A\/B Test type will ask you for your website URL so just punch in that data. Not only will this URL be used to preview your website in their <a href=\"https:\/\/vwo.com\/knowledge\/using-vwo-code-editor\/\" target=\"_blank\" rel=\"noopener\">live visual editor<\/a> (which is pretty kickass), it will also be the URL where the experiment will run for the users once they land on this page either directly or through some internal\/external referrer. So this will be the URL basically where VWO will drop cookies to attach a particular visitor to your experiment (with one of the variations). So for your visitors, the experiment begins from here.<\/p>\n<p>Surely once the campaign <em>is created<\/em>, you can go to &#8220;Settings > URLs&#8221; to add multiple URLs on which the experiment should start off for the visitors. Let&#8217;s take two cases for some more clarity:<\/p>\n<ul>\n<li>If you want that the experiment to start on all the pages on your website, wherever the user lands, then punch in this pattern &#8211; <code>http:\/\/yoursite.com*<\/code>.<\/li>\n<li>If you&#8217;ve city specific home pages for users, and you want the experiment to only start on the homepage, then use a regexp pattern like this &#8211; <code>http(.*?):\/\/yoursite.com\/(city1|city2|city3)<\/code>.<\/li>\n<\/ul>\n<p>This is how the &#8220;Settings > URLs&#8221; section looks on VWO:<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url.png\" rel=\"attachment wp-att-2408\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url.png\" alt=\"vwo campaign setting urls\" width=\"1726\" height=\"970\" class=\"aligncenter size-full wp-image-2408\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url.png 1726w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url-300x169.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url-768x432.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url-1024x575.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url-1200x674.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-campaign-settings-url-50x28.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<h4>2) Create Variations<\/h4>\n<p>During the creation process, once you enter your URLs, you&#8217;ll be able to create variations and make appropriate changes for those variations right there in their live visual editor. You can move components\/elements around, make textual (as well as image\/video based) changes, remove\/add elements and if you know coding then you can also add\/edit HTML, CSS and JavaScript code.<\/p>\n<p><!-- image: better version --><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/1030961524\/original\/blob1443088084183.png?1443088081\" alt=\"VWO live visual editor\"><\/p>\n<p>But what if the changes are too complicated to be able to make via the live visual editor ? Sure, some technical guy will need to be involved but what if it is too messy to make all the complicated changes via the editor itself ? Let me tell you what <em>I do<\/em> in such cases.<\/p>\n<p>In the editor you see a list of variations which on clicking will show you a small option called &#8220;Edit code&#8221;. Clicking on this will open up a code editing section, where I usually set a custom cookie using JavaScript. So when the user lands on to the specified URLs where the experiment starts for them, the JS code is executed which sets the custom cookie that can be accessed both in the frontend as well as backend (in the subsequent requests). Basis this new cookie that is set, I make corresponding changes in the UI using JS or even spit different HTML code directly from the the backend when\/if required. I can now also track data differently from the frontend\/backend if required. Of course VWO will also set its own cookie to track the experiment as well as the variation of which the visitor is a part but there&#8217;s no point in using their cookie as you&#8217;ll either not know their names and even if they have some sort of standard VWO object that stores such information (which they do, try <code>_vwo_exp<\/code> in console) that you&#8217;d be able to use, I think you&#8217;ll just be better off with a simple custom cookie that you set yourself and work with that.<\/p>\n<pre class=\"brush: jscript; title: ; notranslate\" title=\"\">\r\n\/\/ in Control's code editor\r\nvwo_$(function () {\r\n  document.cookie = 'custom_control_cookie=1; path=\/; domain=codetheory.in'; \/\/ can set the expiry too if required!\r\n});\r\n\r\n\/\/ in Variation1's code editor\r\nvwo_$(function () {\r\n  document.cookie = 'custom_variation1_cookie=1; path=\/; domain=codetheory.in'; \/\/ can set the expiry too if required!\r\n});\r\n<\/pre>\n<h4>3) Goals and Traffic Distribution<\/h4>\n<p>Once you&#8217;re done with creating your variations and assigning changes to them via code or the editor itself, you&#8217;ll be prompted to create your Goals in the creation process. We&#8217;ve already discussed goals before. Goals are what you want to track whether the users actually accomplish or not as they matter to your business. You&#8217;ve certain objectives for your visitors that you want them to achieve. This is how you&#8217;d create Goals on VWO (from the creation process or &#8220;Settings > Goals&#8221; after creation):<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals.png\" rel=\"attachment wp-att-2410\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals.png\" alt=\"vwo create goals section\" width=\"1742\" height=\"994\" class=\"aligncenter size-full wp-image-2410\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals.png 1742w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals-300x171.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals-768x438.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals-1024x584.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals-1200x685.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-create-goals-50x29.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>Very simple, yet powerful way of tracking different types of conversions and later based on that data in the reports, announcing the winning variation.<\/p>\n<p>Also don&#8217;t forget to specify your traffic distribution. You&#8217;ll find it as a small option below (I think they could have made it a lot more prominent). Here&#8217;s a screenshot (from the last step of the creation process or &#8220;Settings > Others&#8221; after creation):<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split.png\" rel=\"attachment wp-att-2411\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split.png\" alt=\"vwo traffic split\" width=\"1754\" height=\"878\" class=\"aligncenter size-full wp-image-2411\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split.png 1754w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split-300x150.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split-768x384.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split-1024x513.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split-1200x601.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-traffic-split-50x25.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<h4>4) Start Campaign<\/h4>\n<p>Once you&#8217;re done, go ahead and start your campaign. Hooray! You just learnt how to create A\/B experiments which will run fantastically and you can always find your way back to the campaign settings and change them as you deem necessary.<\/p>\n<h4>5) Analyse Results\/Data<\/h4>\n<p>As your campaign runs for some time and visitors accomplish the goals, you&#8217;ll be able to see the reports in the &#8220;Detailed Report&#8221; section and compare them either by Goals vs Variations or the other way round. Here&#8217;s a screenshot of one of my campaign reports:<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report.png\" rel=\"attachment wp-att-2412\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report.png\" alt=\"vwo detailed report\" width=\"2442\" height=\"984\" class=\"aligncenter size-full wp-image-2412\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report.png 2442w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report-300x121.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report-768x309.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report-1024x413.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report-1200x484.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/vwo-details-report-50x20.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>(They also have a nice graph that I&#8217;ve deleted from the screenshot in order to accommodate the report.)<\/p>\n<p>The reports are simple to understand, but for those who&#8217;d want to be sure about everything by exploring a little more (although not required), feel free to go through these two small VWO knowledge base links:<\/p>\n<ul>\n<li><a href=\"https:\/\/vwo.com\/knowledge\/view-detailed-reports\/\" target=\"_blank\" rel=\"noopener\">https:\/\/vwo.com\/knowledge\/view-detailed-reports\/<\/a><\/li>\n<li><a href=\"https:\/\/vwo.com\/knowledge\/interprete-vwo-reports\/\" target=\"_blank\" rel=\"noopener\">https:\/\/vwo.com\/knowledge\/interprete-vwo-reports\/<\/a><\/li>\n<\/ul>\n<p>That&#8217;s it! That&#8217;s all there is to implementing A\/B testing on your website.<\/p>\n<h3>Further Notes on GA and Optimizely<\/h3>\n<p>If you know how to setup an experiment in VWO, then you should pretty much be able to do the same in Optimizely. It&#8217;s not much different. Conceptually and module-wise it&#8217;s all the same.<\/p>\n<p>Note: Although VWO only does website A\/B (split) testing as of now, Optimizely is also into mobile apps split testing, which is cool! So yeah, not only can you experiment multiple variations on your website, but you can also do the same in mobile apps as well.<\/p>\n<p>In Google Analytics, you&#8217;ve got <a href=\"https:\/\/support.google.com\/analytics\/answer\/1745147?hl=en&#038;ref_topic=1745207&#038;rd=1\" target=\"_blank\" rel=\"noopener\">Content<\/a> <a href=\"https:\/\/developers.google.com\/analytics\/solutions\/experiments\" target=\"_blank\" rel=\"noopener\">Experiments<\/a> that can be used to do split testing. If you just go to your website profile\/view in analytics and click on &#8220;Behaviour > Experiments&#8221;, you&#8217;ll find that section. This is where you&#8217;ll have to create and view reports for your experiments. Of course it lacks <strong>a lot<\/strong> of the features you&#8217;ll find in VWO or Optimizely.<\/p>\n<p>The creation process is fairly simple, it has about 4 steps:<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment.png\" rel=\"attachment wp-att-2414\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment.png\" alt=\"ga new experiment\" width=\"1536\" height=\"1238\" class=\"aligncenter size-full wp-image-2414\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment.png 1536w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment-300x242.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment-768x619.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment-1024x825.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment-1200x967.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-new-experiment-50x40.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>The <em>first step<\/em> asks you to name the experiment and specify your primary Goal (or objective) for the experiment. You can set other important variables like what <strong>percentage of traffic<\/strong> do you want to allocate to your experiment, for <strong>how long<\/strong> should it run (2 weeks minimum for instance), set a <strong>statistical confidence level<\/strong> (95% at least) and whether to <strong>disable multi-armed bandit optimization<\/strong> or not (more on this in a bit).<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment.png\" rel=\"attachment wp-att-2415\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment.png\" alt=\"ga configure experiment\" width=\"1646\" height=\"1108\" class=\"aligncenter size-full wp-image-2415\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment.png 1646w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment-300x202.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment-768x517.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment-1024x689.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment-1200x808.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-configure-experiment-50x34.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>If you look at the image above (the <em>second step<\/em>), it&#8217;s basically ask you to specify original (control) and variations details which&#8217;ll mostly be the names and URLs for them. If you&#8217;re <em>relating this<\/em> to a split URL test, then you&#8217;re right! The easiest (or default) way to setup experiments on GA is actually the split URL way. You can surely setup a non-split URL testing (without redirection) by writing some JS code (yes! you should know JS) in your origin\/control page, which we will discuss in a bit.<\/p>\n<p>In the <em>third step<\/em> you&#8217;ll be presented with a small JavaScript snippet that you&#8217;ll have to copy paste right after the opening of your <code>&lt;head&gt;<\/code> tag. This is different from your normal GA tracking snippet that you had pasted before the <code>&lt;\/body&gt;<\/code> closing tag. You&#8217;ll also be able to see your <em>Experiment ID<\/em> and <em>Experiment Key<\/em> here.<\/p>\n<p>The <em>final (fourth) step<\/em> will allow you to validate the presence of this snippet (from the previous step) on your control and variations and finally start the experiment.<\/p>\n<p>By default the way experiments work in GA is that once the user lands on the original page, GA will either redirect the user to one of the variations or just refresh the original page. With the refresh or redirect, GA also appends a GET param to store the experiment key, ID and variation number information. So the URL will look something like this:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n# Refresh (original)\r\nhttp:\/\/codetheory.in\/?utm_expid=65042302-0.g3WqZjLOQB6vC19Dpq_mwA.0\r\n# Redirect (variation)\r\nhttp:\/\/codetheory.in\/variation1\/?utm_expid=65042302-0.g3WqZjLOQB6vC19Dpq_mwA.0\r\n<\/pre>\n<p>You can surely avoid these params in your URLs and also the split URL testing by spending some time integrating the Experiments API into your JS code. It&#8217;s very simple, go ahead and <a href=\"https:\/\/developers.google.com\/analytics\/solutions\/experiments-client-side\" target=\"_blank\" rel=\"noopener\">read this client-side experiments without redirection<\/a> document. In essence, what you do is punch in random data for your variations while setting up the experiment and then make a call to a method available (<code>cxApi.chooseVariation()<\/code> currently) from their JS based API SDK to select the chosen variation for the visitor. Basis the variation index (0 if control, 1 if V1, 2 if V2), you can make changes to your UI and the rest of the data tracking will be automatically handled by GA.<\/p>\n<p>You can also have a server-side implementation. Let&#8217;s see what all will be involved in such an implementation:<\/p>\n<ul>\n<li>When the user visits the URL where the experimentation should start, you have to decide whether the visitor should be included in the experiment or not based on the URL as well as the traffic percentage you are including in your experiment. It has been explained well with sample codes <a href=\"https:\/\/developers.google.com\/analytics\/solutions\/experiments-server-side#choose\" target=\"_blank\" rel=\"noopener\">here<\/a>. You&#8217;ll also have to decide which variation to show (again sample code is there) and ensure to track all these data using cookies.<\/li>\n<li>Once the decisions have been made, you&#8217;ll have to <a href=\"https:\/\/developers.google.com\/analytics\/devguides\/collection\/analyticsjs\/experiments#pro-server\" target=\"_blank\" rel=\"noopener\">let GA know<\/a> about the experiment ID and variation index so that it can store this data and also show reports.<\/li>\n<li>Retrieve the experiment data for visitors periodically and <a href=\"https:\/\/developers.google.com\/analytics\/solutions\/experiments-server-side#store\" target=\"_blank\" rel=\"noopener\">store them<\/a> in your own database.<\/li>\n<\/ul>\n<p>The number of things you can\/should do sounds cool but I highly doubt you&#8217;ll ever need to go this route. But it&#8217;s good to know!<\/p>\n<p>In the &#8220;Behaviour > Experiments&#8221; section, you should be able to compare your variations by going through the reports that GA generates by tracking all the data. Here&#8217;s a sample report (fairly easy to understand, no need to read some sort of doc for it):<\/p>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report.png\" rel=\"attachment wp-att-2417\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report.png\" alt=\"ga experiment report\" width=\"2144\" height=\"1142\" class=\"aligncenter size-full wp-image-2417\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report.png 2144w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report-300x160.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report-768x409.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report-1024x545.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report-1200x639.png 1200w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/ga-experiment-report-50x27.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<h4>Multi-Armed Bandit Experiments<\/h4>\n<p>Note: The idea is to just scratch the surface of this concept but not explain it from top to bottom as that&#8217;s an advanced subject and will become a different guide in itself.<\/p>\n<p>By default GA employs <a href=\"https:\/\/en.wikipedia.org\/wiki\/Multi-armed_bandit\" target=\"_blank\" rel=\"noopener\">multi-armed bandit<\/a> optimization. You can surely disable it while creating the experiment as mentioned earlier. What this optimization basically does is, it checks the different metrics (goals are an important one) of the experiment&#8217;s variations twice in a day and starts sending more traffic to the variation that performs better out of all. This way you loose less conversions as the poorly performing variations start loosing traffic. Pretty cool, right! This is currently not there in the other tools.<\/p>\n<p>Although you wouldn&#8217;t want to care about it at this stage (unless you already do a lot of A\/B tests in life and are looking for ways to optimize the methodology), but just incase if you do then here are some interesting links around multi-armed bandit approach (aka epsilon greedy) &#8211; <a href=\"https:\/\/support.google.com\/analytics\/answer\/2844870?hl=en&#038;ref_topic=1745207\" target=\"_blank\" rel=\"noopener\">Google support<\/a> <a href=\"https:\/\/news.ycombinator.com\/item?id=4040022\" target=\"_blank\" rel=\"noopener\">and<\/a> <a href=\"https:\/\/news.ycombinator.com\/item?id=4052997\" target=\"_blank\" rel=\"noopener\">some<\/a> <a href=\"https:\/\/news.ycombinator.com\/item?id=4060658\" target=\"_blank\" rel=\"noopener\">HN<\/a> <a href=\"https:\/\/news.ycombinator.com\/item?id=4532578\" target=\"_blank\" rel=\"noopener\">discussions<\/a>. If you do care then not only will you enjoy the articles in those hacker news links, but also the whole discussions. There are a few interesting comments around how multi-armed bandit is different from A\/B testing and how you can use both of them together. Hence, they shouldn&#8217;t be compared. A\/B testing is more geared towards cases where the objective is to find the best performing version of your change whereas the goal of bandit is to optimize the given\/current content. What this means in practise is that what types of video suggestions should youtube be showing on the right hand side of the main video that you&#8217;re watching or what products should rank above others on ecommerce sites and made more discoverable and hence purchasable, are all part of bandit optimization.<\/p>\n<p>Also <a href=\"https:\/\/news.ycombinator.com\/item?id=4040616\" target=\"_blank\" rel=\"noopener\">this comment<\/a> is a good read that states some significant issues with the multi-bandit approach from the perspective of a site owner\/company. Although it also has its own debate and oh such is life!<\/p>\n<h2>How it Works (Technical Details) ?<\/h2>\n<p>Finally we&#8217;re on to the technical aspects of how A\/B testing works in the browser at least. It&#8217;s quite simple actually. The A\/B testing softwares will have to track the users by assigning them the experiment ID and the variation ID of that experiment. This data has to be &#8220;remembered&#8221; and will require a client-side storage system. Softwares typically and heavily make use of cookies for this purpose. In this case, such cookies (set by a script loaded from vwo.com for instance) are known as third party cookies. Just fyi, first party cookies are those that you&#8217;d have set yourself (a.com setting cookies for a.com on a.com).<\/p>\n<p>So when you land on example.com which is running a split test using one of the tools we&#8217;ve discussed before, the tools set multiple cookies to keep track of the user and maintain all sorts of experiment related information about him. From generating a unique client-id against which the data will be updated on their servers to assigning the experiment IDs and variation data, cookies are just a simple storage mechanism that can help track the user&#8217;s engagement.<\/p>\n<p>Note: At this point you might wonder what if you have two experiments with different variations whose impact on the UI are conflicting ? Since different cookies for the different experiments will be set, you&#8217;ll have UI conflicts if the different experiment variations are supposed to affect the same element\/component. So before starting a new experiment, always make sure that such case won&#8217;t arise by reflecting upon all the previous experiments that you had started and are still running. If there are other team members who also run experiments on the same property then make sure to talk to them and discuss any possible conflicting scenario. This is very important as the data that you gather eventually will all be useless because they would have been gathered over conflicting set of UI elements\/components that the visitors ended up seeing.<\/p>\n<p>In case of a split URL test too (on the same domain), cookies are dropped and the visitors are tracked further. But what happens when the control and variation are on different domains ? How would a.com (control\/original) set a cookie for b.com (variation) when it redirects a visitor to that version ? What I mean is let&#8217;s say the user lands on a.com and the tool decides that he should belong to V1 which is b.com and hence he&#8217;ll be redirected. But after the redirection how will b.com have the details regarding the experiment that was started on a.com and remember that the user belongs to that same experiment. Well, you cannot:<\/p>\n<ul>\n<li>Directly execute a <code>document.cookie = 'test=1; domain=b.com'<\/code> from a.com to set all sorts of cookie on b.com for user tracking. Browsers won&#8217;t allow that and you know why (it wouldn&#8217;t make sense if a third-party domain could set n number of useless cookies for your domain).<\/li>\n<li>Create an iframe with b.com as source and then access its <code>document<\/code> object to set a cookie. Accessing the DOM of a cross-origin frame is not permitted by browsers.<\/li>\n<\/ul>\n<p>But you can:<\/p>\n<ul>\n<li>Use the <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/API\/Window\/postMessage\" target=\"_blank\" rel=\"noopener\"><code>postMessage()<\/code><\/a> API to enable cross-origin communication. So VWO&#8217;s script on a.com could create an iframe with b.com as the <code>src<\/code> and then set a cookie for b.com using the <code>postMessage()<\/code> API &#8211; basically <code>postMessage()<\/code> has to be called from a.com and in the <a href=\"https:\/\/developer.mozilla.org\/en-US\/docs\/Web\/API\/Window\/postMessage#The_dispatched_event\" target=\"_blank\" rel=\"noopener\"><code>message<\/code><\/a> event listener on b.com&#8217;s iframe window, the cookies have to be set. But what if the iframe takes a long time to load and by that time VWO redirects the user to the variation (b.com) ? The browser won&#8217;t get the chance to set the cookie inside the iframe and that exactly is the issue with this method my friend!<\/li>\n<\/ul>\n<p>So what to do ? A third-party server can be used (can be yours or someone else&#8217;s). In this case we&#8217;ll just discuss how VWO does it (in a simplified manner though). That&#8217;ll give you a good idea of how to do it in general too. When you land on a.com, VWO&#8217;s script will make a call to say dev.vwo.com\/tpc to set a third party cookie (on a.com) to contain the experiment and variation ID. Imagine a request like this:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n# POST request to\r\nhttp:\/\/dev.vwo.com\/tpc?name=vwo_exp_30_split&value=2&days=100\r\n# Response will contain 'Set-Cookie: vwo_exp_30_split=2; expires=Mon, 11-Jul-16 19:03:59 GMT'\r\n<\/pre>\n<p>This cookie gets set on dev.vwo.com. Then a redirection is initiated to b.com (variation) on which the vwo snippet assigns a unique client ID (generated server-side) to a JS variable &#8211; <code>_vwo_uuid_30=03EEC64F4AC572B73E5BFC966D94B379<\/code>. Then it makes another third party cookie call to dev.vwo.com\/tpc with this unique ID as the new parameter. So something like this happens:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n# POST request to\r\nhttp:\/\/dev.vwo.com\/tpc?name=_vwo_uuid_30&value=03EEC64F4AC572B73E5BFC966D94B379&days=365\r\n# Response headers will contain 'Set-Cookie: _vwo_uuid_30=03EEC64F4AC572B73E5BFC966D94B379; expires=Sun, 02-Apr-17 19:06:33 GMT'\r\n<\/pre>\n<p>Now imagine the user goes back to a.com, since that is the control URL, the user will again get re-directed to b.com (because it will have the <code>vwo_exp_30_split=2<\/codE> cookie set earlier) and the same uuid (<code>03EEC64F4AC572B73E5BFC966D94B379<\/code> from b.com) will be set on a.com too because this time when the vwo snippet tries to load from dev.vwo.com\/snippet.php (dynamic) for instance, it'll get the uuid cookie (in request headers) and set the same client ID cookie for a.com too. Had the user not gone to just a.com\/ but a.com\/some-page, then the same process would have repeated of the vwo snippet getting the uuid cookie in the request headers and then setting that for a.com. If you notice the experiment ID is already available from the <code>uuid<\/code> cookie name which is <code>_vwo_uuid_30<\/code> (30 in this case) and the variation was already there in the <code>vwo_exp_30_split<\/code> cookie value (2 in our case). Do keep in mind that when the user landed on b.com after a redirection, the tpc call created a cookie to identify the user uniquely and the variation number\/index was also known because the snippet.php got that in the request and could now set the same cookie for b.com. The snippet.php is basically an endpoint that'd spit dynamic JS code (response content-type is <code>text\/javascript<\/code>) with the help of a dynamic programming language (PHP in this case).<\/p>\n<p>That's all that is required to track a user to show him the right changes on the webpage\/site. Clever right?! Surely the chances of you doing cross-domain URL testing would be rare but it is just a good piece of information to know.<\/p>\n<h2>Bonus<\/h2>\n<h3>On SEO<\/h3>\n<p>Some people sweat the fact that different variations in an experiment either in a split or a split URL test with different content will adversely impact their ranks in google search results. Well, if you follow certain (very simple) guidelines, they you really don't have to fret about it. Don't believe me ? <a href=\"https:\/\/webmasters.googleblog.com\/2012\/08\/website-testing-google-search.html\" target=\"_blank\" rel=\"noopener\">Believe Google<\/a>!<\/p>\n<p>So basically (summarizing that articke), if your variation is hosted on a different URL, use the <code>noindex<\/code> meta tags there or modify your <code>robots.txt<\/code> to not index that page in Google search results. Also the <a href=\"https:\/\/support.google.com\/webmasters\/answer\/139066?rd=1\" target=\"_blank\" rel=\"noopener\"><code>rel=\"cannonical\"<\/code><\/a> helps when you can't use <code>noindex<\/code> meta tags. Also keep a check on your redirects, they should be 302 (temporary) redirects or javascript ones. The last portion of that article is important which says most of the changes that you do in a split testing will probably not affect your search engine results as the content won't change but mostly the user interface which is generally fine.<\/p>\n<h3>Statistical Significance\/Confidence<\/h3>\n<p><a href=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence.png\" rel=\"attachment wp-att-2439\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence.png\" alt=\"statistical significance\" width=\"1100\" height=\"500\" class=\"aligncenter size-full wp-image-2439\" srcset=\"https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence.png 1100w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence-300x136.png 300w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence-768x349.png 768w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence-1024x465.png 1024w, https:\/\/codetheory.in\/wp-content\/uploads\/2016\/04\/Statistical-Confidence-50x23.png 50w\" sizes=\"(max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/a><\/p>\n<p>Most of the tools you use will have this feature called statistical significance indicator. This value will basically tell you whether the uplift in a variation over the others is actually due to their differences and definitely cannot be attributed to co-incidence\/chance\/accident. The convention is to call a variation statistically significant or confident if the value is 95% or above. Not sure if there's a rule behind that, but it's just a convention that is followed. My suggestion would be to not make your variation live as the updated version unless your test reaches statistical significance. At times this might not happen, then you'll have to use your intellect or experience to decide whether you should actually make your variation live or just scrap it. For instance, a variation will not lead to an uplift but might have an aesthetically better design according to you, your designer and the other people in the team. Then definitely go for it!<\/p>\n<p>There will be times when your tool will recommend you to run your tests for 8 weeks or basically a lot of time to reach that statistical threshold (95%). This might happen if your traffic is too low. Low traffic takes more time to statistically prove that the uplift or dip is not due to some randomness. High traffic takes lesser time to prove that the uplift\/dip is due to the actually differences and not just randomness. So the two important factors deriving statistical significance is traffic count and time.<\/p>\n<p>VWO has an <a href=\"https:\/\/vwo.com\/ab-split-test-significance-calculator\/\" target=\"_blank\" rel=\"noopener\">online calculator<\/a> that you can use to calculate statistical significance basis your data. They've also well-written how they <a href=\"https:\/\/vwo.com\/knowledge\/calculating-ctbo\/\" target=\"_blank\" rel=\"noopener\">calculate it<\/a> and made an <a href=\"https:\/\/vwo.com\/blog\/ab-testing-significance-calculator-spreadsheet-in-excel\/\" target=\"_blank\" rel=\"noopener\">excel sheet available<\/a> containing another calculator with those formulas. Some mathematical nerds might also <a href=\"http:\/\/www.evanmiller.org\/how-not-to-run-an-ab-test.html\" target=\"_blank\" rel=\"noopener\">like this article<\/a>.<\/p>\n<h3>A\/B\/B Testings<\/h3>\n<p>A\/B\/B seems weird, so what is it ? At times I've done a type of testing where there's a control (A) and a variant (B) and what I do is setup another variation in the campaign which is just a replica of the variant. There's no difference between V1 and V2, just that I create two different variations that are the same. Such tests are called A\/B\/B (self-explanatory now) tests. But why would one do so ?<\/p>\n<p>This is done more from an experimentation (with data) point of view and add another layer of certainty to the winning variation. When we moved from Optimizely to VWO, we wanted to make sure VWO's tracks visitors properly and reports sane numbers. We wanted to ensure that there's no vague differences (or mis-tracking) between the control and variation and what it reports is actually very close to the actual difference. A\/B\/B is helpful in such cases.<\/p>\n<p>If B\/B are almost close then you can be sure that VWO is doing a good job at tracking and the difference between the control and variation in terms of numbers can be believed. Where as if B\/B have a good deal of difference (imagine 30%) even after you've collected two weeks of data (might happen rarely) then you know that such a difference might be applicable between A (control) and the worse performing B (variant). So in that case you'll have a range by which the data <em>can<\/em> be off and act accordingly in your analysis. It might be hard to imagine such cases at the moment but it does happen at times and that's when you end up spending more and more time analysing data here and there and dig further into other tools like Google Analytics to cross check different metrics.<\/p>\n<p>For similar reasons some people even do A\/A\/B\/B testing. Of course that'll require more traffic, once you've high traffic you can be lucky and have as many variations as you want!<\/p>\n<h2>Summary<\/h2>\n<p>That was quite a bit but fairly easy to digest :). Now you exactly know what A\/B (split) testing is and how it can help you setup your goals (objectives for visitors) on which you'd always want to increase conversions with various changes on your websites or apps. You also know how it all works technically including cross-domain split URL testing (which is sort of what the initial motivation was to write this article, but I ended up writing all about split testing). You learnt about a couple of different tools that you can start using right away to kick off split testing on your platforms. So what are you waiting for ? Start testing right away and if you face any issues feel free to ask questions in the comments section below.<\/p>\n<p>Note: Setting up and running tests are all good but it is also very important to go through your result reports properly and verify which one is the winning variation. Lots of people make mistakes by choosing the wrong variation as the winner because the reports can be misleading at times (which gets rectified if you follow certain rules). A good example is that, in the first 2 days V2 might convert better than V1 but over the next 2 days which constitute a weekend, V1 might perform overall better than V2. Yes that happens a lot! I've spent quite some time going through various resources by people who have a fair amount of experience in A\/B\/n testing and will summarize my learnings and thoughts around analyzing data in the next article.<\/p>\n<p>If you need help with A\/B testing on your website, <a href=\"http:\/\/codetheory.in\/contact\/\">contact me<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently I\u2019ve been investing a lot of time into A\/B (split) testing\/experiments and analysing their results. Slowly I&#8217;ve started to love this way of rolling out changes on the web (can be surely used for mobile apps as well), as it is a complete data-driven approach and can produce quite surprising (or shocking, how you &hellip; <a href=\"https:\/\/codetheory.in\/split-testing\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Guide (Introduction, Implementation and How it Works) to A\/B Testing and Split URL Testing (VWO, Optimizely, Google Analytics)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[40],"tags":[173,176,175,172,87,174],"_links":{"self":[{"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/posts\/2360"}],"collection":[{"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/comments?post=2360"}],"version-history":[{"count":63,"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/posts\/2360\/revisions"}],"predecessor-version":[{"id":2447,"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/posts\/2360\/revisions\/2447"}],"wp:attachment":[{"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/media?parent=2360"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/categories?post=2360"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codetheory.in\/wp-json\/wp\/v2\/tags?post=2360"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}