Trace: a high-throughput tomographic reconstruction engine for large-scale datasets

Bicer, Tekin; Gürsoy, Doğa; Andrade, Vincent De; Kettimuthu, Rajkumar; Scullin, William; Carlo, Francesco De; Foster, Ian T.

doi:10.1186/s40679-017-0040-7

Trace: a high-throughput tomographic reconstruction engine for large-scale datasets

Journal Article · Sat Jan 28 00:00:00 EST 2017 · Advanced Structural and Chemical Imaging

DOI:https://doi.org/10.1186/s40679-017-0040-7· OSTI ID:1344024

; Gürsoy, Doğa; Andrade, Vincent De; Kettimuthu, Rajkumar; Scullin, William; Carlo, Francesco De; Foster, Ian T.

Abstract Background

Modern synchrotron light sources and detectors produce data at such scale and complexity that large-scale computation is required to unleash their full power. One of the widely used imaging techniques that generates data at tens of gigabytes per second is computed tomography (CT). Although CT experiments result in rapid data generation, the analysis and reconstruction of the collected data may require hours or even days of computation time with a medium-sized workstation, which hinders the scientific progress that relies on the results of analysis.

Methods

We present Trace, a data-intensive computing engine that we have developed to enable high-performance implementation of iterative tomographic reconstruction algorithms for parallel computers. Trace provides fine-grained reconstruction of tomography datasets using both (thread-level) shared memory and (process-level) distributed memory parallelization. Trace utilizes a special data structure called replicated reconstruction object to maximize application performance. We also present the optimizations that we apply to the replicated reconstruction objects and evaluate them using tomography datasets collected at the Advanced Photon Source.

Results

Our experimental evaluations show that our optimizations and parallelization techniques can provide 158× speedup using 32 compute nodes (384 cores) over a single-core configuration and decrease the end-to-end processing time of a large sinogram (with 4501 × 1 × 22,400 dimensions) from 12.5 h to <5 min per iteration.

Conclusion

The proposed tomographic reconstruction engine can efficiently process large-scale tomographic data using many compute nodes and minimize reconstruction times.

View Journal Article

Research Organization:: Argonne National Lab. (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 1344024

Alternate ID(s):: OSTI ID: 1375811

Journal Information:: Advanced Structural and Chemical Imaging, Journal Name: Advanced Structural and Chemical Imaging Journal Issue: 1 Vol. 3; ISSN 2198-0926

Publisher:: Springer Science + Business MediaCopyright Statement

Country of Publication:: Germany

Language:: English

References (32)

IDEAL: Images Across Domains, Experiments, Algorithms and Learning Ushizima, Daniela M.; Bale, Hrishikesh A.; Bethel, E. Wes JOM, Vol. 68, Issue 11 https://doi.org/10.1007/s11837-016-2098-4	journal	September 2016
CUDA optimization strategies for compute- and memory-bound neuroimaging algorithms Lee, Daren; Dinov, Ivo; Dong, Bin Computer Methods and Programs in Biomedicine, Vol. 106, Issue 3 https://doi.org/10.1016/j.cmpb.2010.10.013	journal	June 2012
Accelerating advanced MRI reconstructions on GPUs Stone, S. S.; Haldar, J. P.; Tsao, S. C. Journal of Parallel and Distributed Computing, Vol. 68, Issue 10 https://doi.org/10.1016/j.jpdc.2008.05.013	journal	October 2008
Electron tomography based on a total variation minimization reconstruction technique Goris, B.; Van den Broek, W.; Batenburg, K. J. Ultramicroscopy, Vol. 113 https://doi.org/10.1016/j.ultramic.2011.11.004	journal	February 2012
The ASTRA Toolbox: A platform for advanced algorithm development in electron tomography van Aarle, Wim; Palenstijn, Willem Jan; De Beenhouwer, Jan Ultramicroscopy, Vol. 157 https://doi.org/10.1016/j.ultramic.2015.05.002	journal	October 2015
Real-time data-intensive computing Parkinson, Dilworth Y.; Beattie, Keith; Chen, Xian PROCEEDINGS OF THE 12TH INTERNATIONAL CONFERENCE ON SYNCHROTRON RADIATION INSTRUMENTATION – SRI2015, AIP Conference Proceedings https://doi.org/10.1063/1.4952921	conference	January 2016
CAMERA: The Center for Advanced Mathematics for Energy Research Applications Donatelli, J.; Haranczyk, M.; Hexemer, A. Synchrotron Radiation News, Vol. 28, Issue 2 https://doi.org/10.1080/08940886.2015.1013413	journal	March 2015
Fast tomographic reconstruction on multicore computers Agulleiro, J. I.; Fernandez, J. J. Bioinformatics, Vol. 27, Issue 4 https://doi.org/10.1093/bioinformatics/btq692	journal	December 2010
Maximum a posteriori estimation of crystallographic phases in X-ray diffraction tomography Gürsoy, Doĝa; Biçer, Tekin; Almer, Jonathan D. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 373, Issue 2043 https://doi.org/10.1098/rsta.2014.0392	journal	June 2015
Regridding reconstruction algorithm for real-time tomographic imaging Marone, F.; Stampanoni, M. Journal of Synchrotron Radiation, Vol. 19, Issue 6 https://doi.org/10.1107/S0909049512032864	journal	September 2012
A comparative study of X-ray tomographic microscopy on shales at different synchrotron facilities: ALS, APS and SLS Kanitpanyacharoen, Waruntorn; Parkinson, Dilworth Y.; De Carlo, Francesco Journal of Synchrotron Radiation, Vol. 20, Issue 1 https://doi.org/10.1107/S0909049512044354	journal	November 2012
TomoPy: a framework for the analysis of synchrotron tomographic data Gürsoy, Dogˇa; De Carlo, Francesco; Xiao, Xianghui Journal of Synchrotron Radiation, Vol. 21, Issue 5 https://doi.org/10.1107/S1600577514013939	journal	August 2014
Data Analysis WorkbeNch ( DAWN ) Basham, Mark; Filik, Jacob; Wharmby, Michael T. Journal of Synchrotron Radiation, Vol. 22, Issue 3 https://doi.org/10.1107/S1600577515002283	journal	April 2015
Integration of TomoPy and the ASTRA toolbox for advanced processing and reconstruction of tomographic synchrotron data Pelt, Daniël M.; Gürsoy, Dogˇa; Palenstijn, Willem Jan Journal of Synchrotron Radiation, Vol. 23, Issue 3 https://doi.org/10.1107/S1600577516005658	journal	April 2016
Optimization of tomographic reconstruction workflows on geographically distributed resources Bicer, Tekin; Gürsoy, Dogˇa; Kettimuthu, Rajkumar Journal of Synchrotron Radiation, Vol. 23, Issue 4 https://doi.org/10.1107/S1600577516007980	journal	June 2016
A Map-Reduce System with an Alternate API for Multi-core Environments Jiang, Wei; Ravi, Vignesh T.; Agrawal, Gagan 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing https://doi.org/10.1109/CCGRID.2010.10	conference	May 2010
A data-parallel algorithm for iterative tomographic image reconstruction Johnson, C. A.; Sofer, A. Proceedings. Frontiers '99. Seventh Symposium on the Frontiers of Massively Parallel Computation https://doi.org/10.1109/FMPC.1999.750592	conference	January 1999
Spade: Decentralized orchestration of data movement and warehousing for physics experiments Patton, Simon; Samak, Taghrid; Tull, Craig E. 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM) https://doi.org/10.1109/INM.2015.7140427	conference	May 2015
TIMBIR: A Method for Time-Space Reconstruction From Interlaced Views Aditya Mohan, K.; Venkatakrishnan, S. V.; Gibbs, John W. IEEE Transactions on Computational Imaging, Vol. 1, Issue 2 https://doi.org/10.1109/TCI.2015.2431913	journal	June 2015
DART: A Practical Reconstruction Algorithm for Discrete Tomography Batenburg, K. J.; Sijbers, J. IEEE Transactions on Image Processing, Vol. 20, Issue 9 https://doi.org/10.1109/TIP.2011.2131661	journal	September 2011
Shared memory parallelization of data mining algorithms: techniques, programming interface, and performance Jin, Ruoming; Yang, Ge; Agrawal, G. IEEE Transactions on Knowledge and Data Engineering, Vol. 17, Issue 1, p. 71-89 https://doi.org/10.1109/TKDE.2005.18	journal	January 2005
Fast, Accurate and Shift-Varying Line Projections for Iterative Reconstruction Using the GPU Pratx, G.; Chinn, G.; Olcott, P. D. IEEE Transactions on Medical Imaging, Vol. 28, Issue 3 https://doi.org/10.1109/TMI.2008.2006518	journal	March 2009
Hybrid MPI-OpenMP Programming for Parallel OSEM PET Reconstruction Jones, M. D.; Yao, R.; Bhole, C. P. IEEE Transactions on Nuclear Science, Vol. 53, Issue 5 https://doi.org/10.1109/TNS.2006.882295	journal	October 2006
Workflow Management for Real-Time Analysis of Lightsource Experiments Deslippe, Jack; Essiari, Abdelilah; Patton, Simon J. 2014 9th Workshop on Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS.2014.9	conference	November 2014
A new workflow for x-ray fluorescence tomography: MAPStoTomoPy Hong, Young Pyo; Chen, Si; Jacobsen, Chris SPIE Optical Engineering + Applications, SPIE Proceedings https://doi.org/10.1117/12.2194162	conference	September 2015
A fast forward projection using multithreads for multirays on GPUs in medical image reconstruction: Fast forward projection on GPUs Chou, Cheng-Ying; Chuo, Yi-Yen; Hung, Yukai Medical Physics, Vol. 38, Issue 7 https://doi.org/10.1118/1.3591994	journal	June 2011
Fast calculation of the exact radiological path for a three-dimensional CT array: Technical Reports: 3D CT array path calculation Siddon, Robert L. Medical Physics, Vol. 12, Issue 2 https://doi.org/10.1118/1.595715	journal	March 1985
High performance model based image reconstruction Wang, Xiao; Sabne, Amit; Kisner, Sherman Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming - PPoPP '16 https://doi.org/10.1145/2851141.2851163	conference	January 2016
A Fast CT Reconstruction Scheme for a General Multi-Core PC Zeng, Kai; Bai, Erwei; Wang, Ge International Journal of Biomedical Imaging, Vol. 2007 https://doi.org/10.1155/2007/29160	journal	January 2007
Pushing the limits for medical image reconstruction on recent standard multicore processors Treibig, Jan; Hager, Georg; Hofmann, Hannes G. The International Journal of High Performance Computing Applications, Vol. 27, Issue 2 https://doi.org/10.1177/1094342012442424	journal	June 2012
Hyperspectral image reconstruction for x-ray fluorescence tomography Gürsoy, Doǧa; Biçer, Tekin; Lanzirotti, Antonio Optics Express, Vol. 23, Issue 7 https://doi.org/10.1364/OE.23.009014	journal	January 2015
Time-resolved X-ray Tomography of Gasoline Direct Injection Sprays Duke, Daniel J.; Swantek, Andrew B.; Sovis, Nicolas M. SAE International Journal of Engines, Vol. 9, Issue 1 https://doi.org/10.4271/2015-01-1873	journal	September 2015

Similar Records

Physics‐based iterative reconstruction for dual‐source and flying focal spot computed tomography

Journal Article · Tue Jun 22 20:00:00 EDT 2021 · Medical Physics · OSTI ID:1798379

Accelerating error correction in tomographic reconstruction

Journal Article · Sun Jul 10 20:00:00 EDT 2022 · Communications Materials · OSTI ID:1875723

Related Subjects

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND TECHNOLOGY
97 MATHEMATICS AND COMPUTING
Big Data
High-Throughput
Reconstruction
Tomography

Trace: a high-throughput tomographic reconstruction engine for large-scale datasets

Citation Formats

References (32)

Similar Records

Related Subjects