How Much Higher Can HTCondor Fly?
- UC, San Diego
- Fermilab
- Wisconsin U., Madison
- Nebraska U.
The HTCondor high throughput computing system is heavily used in the high energy physics (HEP) community as the batch system for several Worldwide LHC Computing Grid (WLCG) resources. Moreover, it is the backbone of GlidelnWMS, the pilot system used by the computing organization of the Compact Muon Solenoid (CMS) experiment. To prepare for LHC Run 2, we probed the scalability limits of new versions and configurations of HTCondor with a goal of reaching 200,000 simultaneous running jobs in a single internationally distributed dynamic pool.In this paper, we first describe how we created an opportunistic distributed testbed capable of exercising runs with 200,000 simultaneous jobs without impacting production. This testbed methodology is appropriate not only for scale testing HTCondor, but potentially for many other services. In addition to the test conditions and the testbed topology, we include the suggested configuration options used to obtain the scaling results, and describe some of the changes to HTCondor inspired by our testing that enabled sustained operations at scales well beyond previous limits.
- Research Organization:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP)
- DOE Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1247507
- Report Number(s):
- FERMILAB-CONF-15-613-CD; 1413936
- Journal Information:
- J.Phys.Conf.Ser., Vol. 664, Issue 6; Conference: 21st International Conference on Computing in High Energy and Nuclear Physics, Okinawa, Japan, 04/13-04/17/2015
- Country of Publication:
- United States
- Language:
- English
Similar Records
Use of DAGMan in CRAB3 to improve the splitting of CMS user jobs
Pushing HTCondor and glideinWMS to 200K+ Jobs in a Global Pool for CMS before Run 2