Optimizing CMS build infrastructure via Apache Mesos
release_j4t3amm2hfclllqcc24oek33ge
by
David Abdurachmanov, Alessandro Degano, Peter Elmer, Giulio Eulisse,
David Mendez, Shahzad Muzaffar
2015
Abstract
The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC)
at CERN consists of 6M lines of in-house code, developed over a decade by
nearly 1000 physicists, as well as a comparable amount of general use
open-source code. A critical ingredient to the success of the construction and
early operation of the WLCG was the convergence, around the year 2000, on the
use of a homogeneous environment of commodity x86-64 processors and Linux.
Apache Mesos is a cluster manager that provides efficient resource isolation
and sharing across distributed applications, or frameworks. It can run Hadoop,
Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of
nodes. We present how we migrated our continuos integration system to schedule
jobs on a relatively small Apache Mesos enabled cluster and how this resulted
in better resource usage, higher peak performance and lower latency thanks to
the dynamic scheduling capabilities of Mesos.
In text/plain
format
Archived Files and Locations
application/pdf 94.7 kB
file_un44qjikl5bthek4q6jnetdsie
|
arxiv.org (repository) web.archive.org (webarchive) |
1507.07429v2
access all versions, variants, and formats of this works (eg, pre-prints)