Massively Scalable Near Duplicate Detection in Streams of Documents using MDSH
- ORNL
In a world where large-scale text collections are not only becoming ubiquitous but also are growing at increasing rates, near duplicate documents are becoming a growing concern that has the potential to hinder many different information filtering tasks. While others have tried to address this problem, prior techniques have only been used on limited collection sizes and static cases. We will briefly describe the problem in the context of Open Source Intelligence (OSINT) along with our additional constraints for performance. In this work we propose two variations on Multi-dimensional Spectral Hash (MDSH) tailored for working on extremely large, growing sets of text documents. We analyze the memory and runtime characteristics of our techniques and provide an informal analysis of the quality of the near-duplicate clusters produced by our techniques.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- DE-AC05-00OR22725
- OSTI ID:
- 1096305
- Resource Relation:
- Conference: IEEE International Conference on Big Data 2013, Santa Clara, CA, USA, 20131006, 20131009
- Country of Publication:
- United States
- Language:
- English
Similar Records
ParaText : scalable solutions for processing and searching very large document collections : final LDRD report.
Mining for Emerging Technologies within Text Streams and Documents