Selecting Users For A/B (Champion/Challenger) Testing
- by Gordon Guthrie
We have a framework that offers A/B split testing.
You have a 'champion' version of a page and you develop a 'challenger' version of it.
Then you run the website and allocate some of your users the champion and some the challenger and measure their different responses.
If the challenger is better than the champion at achieving your metrics then you dethrone the champion and develop a new challenger...
My question is what mechanisms should I consider to allocate the versions?
A number of options spring to mind:
odd or even IP address (or sub-segments) ****.****.****.123 gets champion but .124 gets challenger
cookie push - check for a champion/challenger cookie, if it doesn't exist randomly allocate the user to one and push the cookie
Best practice? suggestions? comments? experience?