Georgiy Mogelashvili blog

Azure Websites Testing in Production to win

Azure Websites Testing in Production to win

*This service has been recently renamed to Traffic Routing. Don’t be confused, article still applies to it :)

Hi!

I’d like to tell you about very new service in the Microsoft Azure family called Azure Websites Testing in Production (or just TiP). It has not been publicly announced yet but it’s already available to play with. This service will help you manage and test your website deployments more reliably. Also you can implement A/B testing with it. So before we start I’d like to say some words about what it is and why you might need it.

A/B testing

If we think about the name itself then we can suggest that A/B testing is about testing A against of B. And that’s 100% correct. This is also called split-testing or testing in production. It helps you analyze users’ behavior based on some variable parameters or features. Imagine you want to implement something completely new for your website. Let’s say – change background color from red to blue. How would you then know that people like that? What if new color will annoy your visitors and they will convert less (by conversion you may consider everything – payment, ads click, subscription etc)? To prevent yourself from assumptions based on your only feelings (which unfortunately might be wrong) you can run some A/B tests against your hypothesis. This means that you have two deployments running in parallel: one of them has old design and functionality (base) and the second has new features deployed (variant). Then if you split your incoming traffic 50/50 and monitor users’ behavior you will be able to make you decisions based on statistics and real data. If variant converts better than make it your new base and direct 100% traffic there. But if not then keep base and think about new features to implement.

Sounds simple, isn’t it? In fact A/B testing is much more complicated. There is lots of statistics there. You can’t just take A, compare it to B and be 100% sure in results. You have to consider every errors in numbers you might have just because your users can behave in tons of different ways depending on day, time, weather etc. As an example imagine, that you have an online store which sells toys. It’s Christmas time now and you decide to paint your website into red and green and put Santa Claus next to the logo. You make it as an A/B test and run it from 20th till 25th of December. After that time you may have positive results for your variant (the one with Santa). But what happens if you make your decision based on that data? You put your variant as a base forever from now and will be very surprised that you start earning less money in January, February etc. That happened because you weren’t running your test for a long enough time to have your results conclusive. Of course this example is very naive but gives you a picture of what may happen if you ignore statistics.

You can read more about A/B testing somewhere in the Internet and now let’s talk about how you can run your own A/B tests with Microsoft Azure.

Tools

Let’s think that you are now good at theory behind A/B testing and know how to manipulate use statistics when making decisions. But another problem we have now is how to implement this technique? How should your website understand which variant should be exposed to this or that visitor? And who shall dispatch users – application itself or low level web server? There are more questions there. What should we do with new visitor? Website have to divide users between buckets (base and variant) randomly but evenly. Every returning visitor we can dispatch as a newcomer or put him in the bucket he was put before. If you decide to implement A/B testing by yourself there a lots of thing to do.

Luckily there are some tools which provide A/B out of the box. For example, MailChimp can send two (or more) different versions of emails to your customers so that you can choose one which performs better. There are some tools for website testing as well, for example OptimizelyVisual Website Optimizer and Google Analytics. But they only allow you to test content changes, not functionality of your website (new features, performance improvements etc). All those tools are for marketing purposes only and not suitable for technical stuff. I hope that you can point me out better tools for A/B testing if any in the comments. From my side I will tell you about how Microsoft Azure can handle all the infrastructure part for you.

Azure Websites Testing in Production

As I said earlier, Azure Websites Testing in Production has not been publicly announced yet, but you already have access to all it’s power in new Azure Portal (aka Ibiza).

In fact TiP is more than just A/B testing. You can use it as a complex solution for validation your business ideas and deployments. There are several scenarios of usage:

  • Canary Testing – test your new, probably unstable, feature on a small amount of your loyal visitors by providing them a link to the updated version of your website.
  • Ramp-Up testing (aka Canary Deployment) – test your prototype by slowly increasing number of visitors seeing new feature. You can start with 5%, monitor performance/errors for a while and add another 5%. Do this until you find some critical errors or until you reach 100% (then you’ll have your new base).
  • Recovery Testing – having stable slot operating while introducing something new will let you fallback to working solution very fast in case of unexpected errors. You may store more than one deployments of a product and switch them if necessary.
  • A/B Testing – the one I was talking about. Consider it as a generic scenario, which includes previous three.

Hopefully, you don’t need to read tons of documentation in order to implement all those scenarios. You can do this in one place in Azure by tuning some parameters. Below I’ll finally explain what to do.

Prepare

Well, obviously, in order to be able to use Azure Websites Testing in Production your web application should be running in Azure. More of that it has to be set up as Standard pricing tier (you need Slots be available and they are only in Standard).

Create Website For Azure Websites Testing In Production

Choose which website you will be testing

Open website blade and go to its bottom. There you will find ‘Testing in production’ tile:

Azure Websites Testing in Production Part

Create new Testing in Production

Add Azure Websites Testing in Production Slot

Create testing in production

You will be asked to create a new deployment slot. Feel free to choose as many slots as you need for testing purposes. By the way, you might have been familiar with deployment slots already. They were introduced a year ago as a cool feature for staging your deployments. Read more in this documentation article. So, this exactly feature is used for testing in production purposes.

To create a new slot you only have to specify its name and configuration source if you want to re-use some config details from your running website. Please keep in mind that ‘production‘ name is system reserved for main application version so please don’t be surprised that you can’t select it.

Add new slot for Azure Websites Testing in Production

Add new slot for testing

Let’s create three new slots and give them some sensible names:

Azure Websites Testing in Production Slots

Slots for testing in production

Each slot is by fact a standalone website with its own files and config and can be easily accessed by a URL like <main_website_name>-<slot_name>.azurewebsites.net. You can deploy your code to a slot with Publish Profile or FTP or whatever you do for regular website deployment.

After you set up and configure all your slots you are ready for testing.

Set up testing

In order to begin testing (A/B or Canary or whatever) you have to specify percentage of your visitors to be routed into each slot you have. Go back to Testing in production blade. When slots are created you will be able to adjust traffic for each of them:

Azure Websites Testing in Production slots configuration

Configure slots in Azure Websites Testing in Production

While editing your slots you always will see how many of your visitors will be routed to your base. It’s obvious when you have only one-two variants but can be handy if you have dozens of them (good luck in monitoring all these).

By the way, as each slot is a regular website, you can then create more complex scenarios with testing inside testing (did you watch Inception movie?).

Complex tests in Azure Websites Testing in Production

Complex tests inside tests

Please note the name of a website on a previous picture. It’s no more a glamcoder, it’s glamcoder-tip-test-1, which is a testing slot with 10% of traffic (set few steps before). But now inside the testing slot we have one more slot (test-1) which gets 50% of traffic. So, we have 50% out of 10%, which results into 5% of all traffic.

Why do you need this? Well, if you are OK with exposing new feature to 50% of your visitors, then you can just have one level of slots. You will then get a normal distribution and even amount of visitors in each deployment, which allows you make you decision statistically correct. But what if you want only 10% of your visitors to see the changes? You can’t compare 10% to 90% then because it’s obviously can’t be correct. That’s where second level of testing slots might help. If you have a slot with variant, and then inside this slot you again split your visitors to variant and base, then you can compare them between each other as if it’s a regular 50/50 distribution. You only need to measure and track them correctly.

How it works

There is no information about how it works in particular. But what I can tell you is that you as a developer don’t have to do anything specific in order to make it work. All the distribution is made under the hood and completely obscure to your visitor. He just types a URL of your website as always but then somewhere inside Azure Websites he will land on the corresponding slot automatically. The decision is based on some statistical algorithms but as far as I know you don’t have to worry about that and be sure that each slot will get only that amount of visitors it was configured for.

On the very first visit a cookie will be created for a user. It will have a slot ID as a value. Whenever this user returns to your website he will be automatically routed to the same slot he was before. That means that user’s experience will remain the same for all the time you run an experiment.

Azure Websites Testing in Production Cookie

Cookie with testing in production slot ID

Monitoring

For now there is no out-of-the-box built-in solutions for monitoring your Azure Websites Testing in Production. You can pick whatever you like most. You can use Azure Analytics, Google Analytics or any other service you like. It’s only you who knows how to track conversion rate and what is good for your site. I hope that Microsoft will bring some advanced tools for monitoring A/B tests but for now you can create your own (which might be better or not).

What’s next

You can ask now – what am I supposed do with this new and unfinished service? Well, that’s quite a reasonable question. I wrote this article wittingly before this service is publicly announced because. I wanted to tell you about great opportunity for each website owner to become more confident in decision making. You don’t have to be blind now when releasing new feature. Please, test it on small amount of your visitors first and then release it to everyone or to none. I truly believe that every website should do that because only your users can tell you what they like and what they hate. So let them test your website while you staying aside and monitoring what’s going on.

There is also a great introductionary video about Azure Websites Testing in Production on Channel 9.

P.S.

If you decided to start using Azure Websites Testing in Production for your website and faced some issues, bugs or have any other feedback please contact me. I will forward your message to a development team so that they can make the product even better.

6 thoughts on “Azure Websites Testing in Production to win

  1. sami

    First, Thank you for your article It was very helpful for me.

    But I have one question
    You said “On the very first visit a cookie will be created for a user. It will have a slot ID as a value. Whenever this user returns to your website he will be automatically routed to the same slot he was before. That means that user’s experience will remain the same for all the time you run an experiment.”

    My question is there any expiration for that cookie. Cause I have an issue one of our users Visited our vesion B website. After one day she refreshed the same window but azure routed her to Version A.

    How to enforce routing to be to the same slot all the time (I mean for same user)

    Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>