Lessons Learned from Managing a Petabyte
- SLAC
The amount of data collected and stored by the average business doubles each year. Many commercial databases are already approaching hundreds of terabytes, and at this rate, will soon be managing petabytes. More data enables new functionality and capability, but the larger scale reveals new problems and issues hidden in ''smaller'' terascale environments. This paper presents some of these new problems along with implemented solutions in the framework of a petabyte dataset for a large High Energy Physics experiment. Through experience with two persistence technologies, a commercial database and a file-based approach, we expose format-independent concepts and issues prevalent at this new scale of computing.
- Research Organization:
- Stanford Linear Accelerator Center (SLAC), Menlo Park, CA
- Sponsoring Organization:
- SC
- DOE Contract Number:
- AC02-76SF00515
- OSTI ID:
- 839755
- Report Number(s):
- SLAC-PUB-10963
- Country of Publication:
- United States
- Language:
- English
Similar Records
Fermilab's multi-petabyte scalable mass storage system
Designing a Multi-Petabyte Database for LSST
Image processing tools for petabyte-scale light sheet microscopy data
Conference
·
Fri Dec 31 23:00:00 EST 2004
·
OSTI ID:879085
Designing a Multi-Petabyte Database for LSST
Conference
·
Tue Jan 09 23:00:00 EST 2007
·
OSTI ID:897453
Image processing tools for petabyte-scale light sheet microscopy data
Journal Article
·
Thu Oct 17 00:00:00 EDT 2024
· Nature Methods
·
OSTI ID:2477940