Urnings: A new method for tracking dynamically changing parameters in paired comparison systems

Abstract

We introduce a new rating system for tracking the development of parameters based on a stream of observations. Rating systems are applied in competitive games, adaptive learning systems, and platforms for product and service ratings. We model each observation as an outcome of a game of chance that depends on the parameters of interest (e.g., the outcome of a chess game depends on the abilities of the two players). Determining the probabilities of the different game outcomes is conceptualized as an urn problem, where a rating is represented by a proportion of colored balls in an urn. This setup allows for evaluating the standard errors of the ratings and performing statistical inferences about the development of and relations between parameters. Theoretical properties of the system in terms of the invariant distributions of the ratings and their convergence are derived. The properties of the rating system are illustrated with simulated examples and its potential for answering research questions is illustrated using data from competitive chess.