Optimizing CMS build infrastructure via Apache Mesos release_j4t3amm2hfclllqcc24oek33ge

by David Abdurachmanov, Alessandro Degano, Peter Elmer, Giulio Eulisse, David Mendez, Shahzad Muzaffar

Released as a article .

2015  

Abstract

The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneous environment of commodity x86-64 processors and Linux. Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of nodes. We present how we migrated our continuos integration system to schedule jobs on a relatively small Apache Mesos enabled cluster and how this resulted in better resource usage, higher peak performance and lower latency thanks to the dynamic scheduling capabilities of Mesos.
In text/plain format

Archived Files and Locations

application/pdf  94.7 kB
file_un44qjikl5bthek4q6jnetdsie
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   accepted
Date   2015-07-28
Version   v2
Language   en ?
arXiv  1507.07429v2
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 8e2d7075-7120-4148-a0de-f6ff0929b073
API URL: JSON