abstract = "The utility of current metrics used in genetic
programming (GP) systems, such as computational effort
and mean-best-fitness, varies depending upon the
problem and the resource that needs to be optimized.
Inferences about the underlying system can only be made
when a sufficient number of runs are performed to
estimate the relevant metric within some confidence
interval. This paper proposes a new algorithm for
determining the minimum number of independent runs
needed to make inferences about a GP system. As such,
we view our algorithm as a meta-metric that should be
satisfied before any inferences about a system are
made. We call this metric COSMOS, as it estimates the
number of independent runs needed to achieve the
Convergence Of Sample Means Of the Order Statistics. It
is agnostic to the underlying GP system and can be used
to evaluate extant performance metrics, as well as
problem difficulty. We suggest ways for which COSMOS
may be used to identify problems for which GP may be
uniquely qualified to solve.",
notes = "Also known as \cite{2330848} Distributed at
GECCO-2012.