At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection. (...)
An experiment in web search diverts some subset of the incoming queries to an alternate processing path and potentially changes what is served to the user. (...) In addition to specifying how serving is changed via alternate parameter values, experiments must also specify what subset of traffic is diverted. One easy way to do experiment diversion is random traffic, which is effectively flipping a coin on every incoming query. One issue with random traffic experiment diversion is that if the experiment is a user-visible change (e.g., changing the background color), the queries from a single user may pop in and pop out of the experiment (e.g., toggle between yellow and pink), which can be disorienting. Thus, a common mechanism used in web experimentation is to use the cookie as the basis of diversion; cookies are used by web sites to track unique users. In reality, cookies are machine/browser specific and easily cleared; thus, while a cookie does not correspond to a user, it can be used to provide a consistent user experience over successive queries. For experiment diversion, we do not divert on individual cookies, but rather a cookie mod: given a numeric representation of a cookie, take that number modulo 1000, and all cookies whose mod equals 42, for example, would be grouped together for experiment diversion. Assuming cookie assignment is random, any cookie mod should be equivalent to any other cookie mod.
