skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: catsHTM: A Tool for Fast Accessing and Cross-matching Large Astronomical Catalogs

Abstract

Fast access to large catalogs is required for some astronomical applications. Here we introduce the catsHTM tool, consisting of several large catalogs reformatted into HDF5-based file format, which can be downloaded and used locally. To allow fast access, the catalogs are partitioned into hierarchical triangular meshes and stored in HDF5 files. Several tools are provided to perform efficient cone searches at resolutions spanning from a few arc-seconds to degrees, within a few milliseconds time. The first released version includes the following catalogs (by alphabetical order): 2MASS, 2MASS extended sources, AKARI, APASS, Cosmos, DECaLS/DR5, FIRST, GAIA/DR1, GAIA/DR2, GALEX/DR6Plus7, HSC/v2, IPHAS/DR2, NED redshifts, NVSS, Pan-STARRS1/DR1, PTF photometric catalog, ROSAT faint source, SDSS sources, SDSS/DR14 spectroscopy, SkyMapper, Spitzer/SAGE, Spitzer/IRAC galactic center, UCAC4, UKIDSS/DR10, VST/ATLAS/DR3, VST/KiDS/DR3, WISE and XMM. We provide Python code that allows to perform cone searches, as well as MATLAB code for performing cone searches, catalog cross-matching, general searches, as well as load and create these catalogs.

Authors:
 [1];  [1]
  1. Weizmann Inst. of Science, Rehovot (Israel)
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
Sponsoring Org.:
USDOE
OSTI Identifier:
1543942
Resource Type:
Accepted Manuscript
Journal Name:
Publications of the Astronomical Society of the Pacific
Additional Journal Information:
Journal Volume: 130; Journal Issue: 989; Journal ID: ISSN 0004-6280
Publisher:
Astronomical Society of the Pacific (ASP)
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; Astronomy & Astrophysics

Citation Formats

Soumagnac, Maayane T., and Ofek, Eran O. catsHTM: A Tool for Fast Accessing and Cross-matching Large Astronomical Catalogs. United States: N. p., 2018. Web. doi:10.1088/1538-3873/aac410.
Soumagnac, Maayane T., & Ofek, Eran O. catsHTM: A Tool for Fast Accessing and Cross-matching Large Astronomical Catalogs. United States. doi:10.1088/1538-3873/aac410.
Soumagnac, Maayane T., and Ofek, Eran O. Tue . "catsHTM: A Tool for Fast Accessing and Cross-matching Large Astronomical Catalogs". United States. doi:10.1088/1538-3873/aac410. https://www.osti.gov/servlets/purl/1543942.
@article{osti_1543942,
title = {catsHTM: A Tool for Fast Accessing and Cross-matching Large Astronomical Catalogs},
author = {Soumagnac, Maayane T. and Ofek, Eran O.},
abstractNote = {Fast access to large catalogs is required for some astronomical applications. Here we introduce the catsHTM tool, consisting of several large catalogs reformatted into HDF5-based file format, which can be downloaded and used locally. To allow fast access, the catalogs are partitioned into hierarchical triangular meshes and stored in HDF5 files. Several tools are provided to perform efficient cone searches at resolutions spanning from a few arc-seconds to degrees, within a few milliseconds time. The first released version includes the following catalogs (by alphabetical order): 2MASS, 2MASS extended sources, AKARI, APASS, Cosmos, DECaLS/DR5, FIRST, GAIA/DR1, GAIA/DR2, GALEX/DR6Plus7, HSC/v2, IPHAS/DR2, NED redshifts, NVSS, Pan-STARRS1/DR1, PTF photometric catalog, ROSAT faint source, SDSS sources, SDSS/DR14 spectroscopy, SkyMapper, Spitzer/SAGE, Spitzer/IRAC galactic center, UCAC4, UKIDSS/DR10, VST/ATLAS/DR3, VST/KiDS/DR3, WISE and XMM. We provide Python code that allows to perform cone searches, as well as MATLAB code for performing cone searches, catalog cross-matching, general searches, as well as load and create these catalogs.},
doi = {10.1088/1538-3873/aac410},
journal = {Publications of the Astronomical Society of the Pacific},
number = 989,
volume = 130,
place = {United States},
year = {2018},
month = {6}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 9 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

The Tenth data Release of the Sloan Digital sky Survey: First Spectroscopic data from the Sdss-Iii Apache Point Observatory Galactic Evolution Experiment
journal, March 2014

  • Ahn, Christopher P.; Alexandroff, Rachael; Allende Prieto, Carlos
  • The Astrophysical Journal Supplement Series, Vol. 211, Issue 2
  • DOI: 10.1088/0067-0049/211/2/17

SkyQuery: Federating Astronomy Archives
journal, May 2013

  • Budavari, Tamas; Dobos, Laszlo; Szalay, Alexander S.
  • Computing in Science & Engineering, Vol. 15, Issue 3
  • DOI: 10.1109/MCSE.2013.41

The Fourth us Naval Observatory ccd Astrograph Catalog (Ucac4)
journal, January 2013


Spitzer Survey of the Large Magellanic Cloud: Surveying the Agents of a Galaxy?s Evolution (SAGE). I. Overview and Initial Results
journal, January 2006

  • Meixner, Margaret; Gordon, Karl D.; Indebetouw, Remy
  • The Astronomical Journal, Vol. 132, Issue 6
  • DOI: 10.1086/508185

The FIRST Survey: Faint Images of the Radio Sky at Twenty Centimeters
journal, September 1995

  • Becker, Robert H.; White, Richard L.; Helfand, David J.
  • The Astrophysical Journal, Vol. 450
  • DOI: 10.1086/176166

Probabilistic multi-catalogue positional cross-match
journal, January 2017


The SIMBAD astronomical database: The CDS reference database for astronomical objects
journal, April 2000

  • Wenger, M.; Ochsenbein, F.; Egret, D.
  • Astronomy and Astrophysics Supplement Series, Vol. 143, Issue 1
  • DOI: 10.1051/aas:2000332

C 3 , A Command-line Catalog Cross-match Tool for Large Astrophysical Catalogs
journal, December 2016

  • Riccio, Giuseppe; Brescia, Massimo; Cavuoti, Stefano
  • Publications of the Astronomical Society of the Pacific, Vol. 129, Issue 972
  • DOI: 10.1088/1538-3873/129/972/024005

2MASS Extended Source Catalog: Overview and Algorithms
journal, May 2000

  • Jarrett, T. H.; Chester, T.; Cutri, R.
  • The Astronomical Journal, Vol. 119, Issue 5
  • DOI: 10.1086/301330

SkyMapper Southern Survey: First Data Release (DR1)
journal, January 2018

  • Wolf, Christian; Onken, Christopher A.; Luvaul, Lance C.
  • Publications of the Astronomical Society of Australia, Vol. 35
  • DOI: 10.1017/pasa.2018.5

The XMM-Newton serendipitous survey : VII. The third
journal, April 2016


The Two Micron All Sky Survey (2MASS)
journal, February 2006

  • Skrutskie, M. F.; Cutri, R. M.; Stiening, R.
  • The Astronomical Journal, Vol. 131, Issue 2
  • DOI: 10.1086/498708

Searchable Sky Coverage of Astronomical Observations: Footprints and Exposures
journal, November 2010

  • Budavári, Tamás; Szalay, Alexander S.; Fekete, György
  • Publications of the Astronomical Society of the Pacific, Vol. 122, Issue 897
  • DOI: 10.1086/657302

The AKARI/IRC mid-infrared all-sky survey
journal, May 2010


Point Sources from a Spitzer IRAC Survey of the Galactic Center
journal, March 2008

  • Ramírez, Solange V.; Arendt, Richard G.; Sellgren, Kris
  • The Astrophysical Journal Supplement Series, Vol. 175, Issue 1
  • DOI: 10.1086/524015

The Palomar Transient Factory photometric catalog 1.0
journal, August 2012

  • Ofek, E. O.; Laher, R.; Surace, J.
  • Publications of the Astronomical Society of the Pacific, Vol. 124, Issue 918
  • DOI: 10.1086/666978

Gaia Data Release 1 : Summary of the astrometric, photometric, and survey properties
journal, November 2016


The Galaxy Evolution Explorer : A Space Ultraviolet Survey Mission
journal, January 2005

  • Martin, D. Christopher; Fanson, James; Schiminovich, David
  • The Astrophysical Journal, Vol. 619, Issue 1
  • DOI: 10.1086/426387

Probabilistic Cross‐Identification of Astronomical Sources
journal, May 2008

  • Budavári, Tamás; Szalay, Alexander S.
  • The Astrophysical Journal, Vol. 679, Issue 1
  • DOI: 10.1086/587156

The NRAO VLA Sky Survey
journal, May 1998

  • Condon, J. J.; Cotton, W. D.; Greisen, E. W.
  • The Astronomical Journal, Vol. 115, Issue 5
  • DOI: 10.1086/300337

The VizieR database of astronomical catalogues
journal, April 2000

  • Ochsenbein, F.; Bauer, P.; Marcout, J.
  • Astronomy and Astrophysics Supplement Series, Vol. 143, Issue 1
  • DOI: 10.1051/aas:2000169

The Palomar Transient Factory: System Overview, Performance, and First Results
journal, December 2009

  • Law, Nicholas M.; Kulkarni, Shrinivas R.; Dekany, Richard G.
  • Publications of the Astronomical Society of the Pacific, Vol. 121, Issue 886
  • DOI: 10.1086/648598

The third data release of the Kilo-Degree Survey and associated data products
journal, August 2017


Finding counterparts for all-sky X-ray surveys with Nway: a Bayesian algorithm for cross-matching multiple catalogues
journal, October 2017

  • Salvato, M.; Buchner, J.; Budavári, T.
  • Monthly Notices of the Royal Astronomical Society, Vol. 473, Issue 4
  • DOI: 10.1093/mnras/stx2651

The VLT Survey Telescope ATLAS
journal, June 2015

  • Shanks, T.; Metcalfe, N.; Chehade, B.
  • Monthly Notices of the Royal Astronomical Society, Vol. 451, Issue 4
  • DOI: 10.1093/mnras/stv1130

Version 1 of the Hubble Source Catalog
journal, May 2016

  • Whitmore, Bradley C.; Allam, Sahar S.; Budavári, Tamás
  • The Astronomical Journal, Vol. 151, Issue 6
  • DOI: 10.3847/0004-6256/151/6/134

The First Release COSMOS Optical and Near‐IR Data and Catalog
journal, September 2007

  • Capak, P.; Aussel, H.; Ajiki, M.
  • The Astrophysical Journal Supplement Series, Vol. 172, Issue 1
  • DOI: 10.1086/519081

The Wide-Field Infrared Survey Explorer (Wise): Mission Description and Initial On-Orbit Performance
journal, November 2010


The UKIRT Infrared Deep Sky Survey (UKIDSS)
journal, August 2007


The second data release of the INT Photometric Hα Survey of the Northern Galactic Plane (IPHAS DR2)
journal, September 2014

  • Barentsen, Geert; Farnhill, H. J.; Drew, J. E.
  • Monthly Notices of the Royal Astronomical Society, Vol. 444, Issue 4
  • DOI: 10.1093/mnras/stu1651

Cross Catalogue Matching with Virtual Observatory and Parameterization of Stars
journal, January 2012


    Works referencing / citing this record:

    Astronomical data fusion: recent progress and future prospects — a survey
    journal, June 2019


    Transient processing and analysis using AMPEL: alert management, photometry, and evaluation of light curves
    journal, November 2019