Hashes are not suitable to verify fixity of the public archived web

Journal Article · Fri Jun 09 00:00:00 EDT 2023 · PLoS ONE

DOI:https://doi.org/10.1371/journal.pone.0286879· OSTI ID:2406553

Aturban, Mohamed ^[1]; Klein, Martin ^[2]; Van de Sompel, Herbert ^[3]; Alam, Sawood ^[4]; Nelson, Michael L. ^[5]; Weigle, Michele C. ^[5]

---

1. Columbia College, Columbia, MO (United States)
2. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
3. Data Archiving and Networked Services, The Hague (Netherlands)
4. Internet Archive, San Francisco, CA (United States)
5. Old Dominion Univ., Norfolk, VA (United States)

Web archives, such as the Internet Archive, preserve the web and allow access to prior states of web pages. We implicitly trust their versions of archived pages, but as their role moves from preserving curios of the past to facilitating present day adjudication, we are concerned with verifying the fixity of archived web pages, or mementos, to ensure they have always remained unaltered. A widely used technique in digital preservation to verify the fixity of an archived resource is to periodically compute a cryptographic hash value on a resource and then compare it with a previous hash value. If the hash values generated on the same resource are identical, then the fixity of the resource is verified. We tested this process by conducting a study on 16,627 mementos from 17 public web archives. We replayed and downloaded the mementos 39 times using a headless browser over a period of 442 days and generated a hash for each memento after each download, resulting in 39 hashes per memento. The hash is calculated by including not only the content of the base HTML of a memento but also all embedded resources, such as images and style sheets. We expected to always observe the same hash for a memento regardless of the number of downloads. However, our results indicate that 88.45% of mementos produce more than one unique hash value, and about 16% (or one in six) of those mementos always produce different hash values. We identify and quantify the types of changes that cause the same memento to produce different hashes. These results point to the need for defining an archive-aware hashing function, as conventional hashing functions are not suitable for replayed archived web pages.

• Research Organization: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
• Sponsoring Organization: USDOE
• Grant/Contract Number: 89233218CNA000001
• OSTI ID: 2406553
• Report Number(s): LA-UR--23-25690
• Journal Information: PLoS ONE, Journal Name: PLoS ONE Journal Issue: 6 Vol. 18; ISSN 1932-6203
• Publisher: Public Library of Science
• Country of Publication: United States
• Language: English

Related Subjects

96 KNOWLEDGE MANAGEMENT AND PRESERVATION
Digital preservation
Fixity of archived resources
Hashes
Information Science
Web archiving