The 'last mile' of data handling: Fermilab's IFDH tools
- Fermilab
IFDH (Intensity Frontier Data Handling), is a suite of tools for data movement tasks for Fermilab experiments and is an important part of the FIFE[2] (Fabric for Intensity Frontier [1] Experiments) initiative described at this conference. IFDH encompasses moving input data from caches or storage elements to compute nodes (the 'last mile' of data movement) and moving output data potentially to those caches as part of the journey back to the user. IFDH also involves throttling and locking to ensure that large numbers of jobs do not cause data movement bottlenecks. IFDH is realized as an easy to use layer that users call in their job scripts (e.g. 'ifdh cp'), hiding the low level data movement tools. One advantage of this layer is that the underlying low level tools can be selected or changed without the need for the user to alter their scripts. Logging and performance monitoring can also be added easily. This system will be presented in detail as well as its impact on the ease of data handling at Fermilab experiments.
- Research Organization:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP)
- DOE Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1295714
- Report Number(s):
- FERMILAB-CONF-13-481-CD; 1302037
- Journal Information:
- J.Phys.Conf.Ser., Vol. 513; Conference: 20th International Conference on Computing in High Energy and Nuclear Physics, Amsterdam, The Netherlands, 10/14-10/18/2013
- Country of Publication:
- United States
- Language:
- English
Similar Records
The Fabric for Frontier Experiments Project at Fermilab
Distributed processing and analysis of physics data in the D0 SAM system at Fermilab