What kind of facilities does the Internet Archive have and how is it operated?
The Long Now of the Web: Inside the Internet Archive's Fight Against Forgetting | HackerNoon
https://hackernoon.com/the-long-now-of-the-web-inside-the-internet-archives-fight-against-forgetting
The Internet Archive is headquartered in Richmond , California, USA, in a former Christian Science church. Surrounded by neoclassical columns, the imposing building houses 212 petabytes of data stored by the Wayback Machine.
by
◆Storage
The heart of the Internet Archive is its dedicated, gigantic storage rack, PetaBox . In the early 2000s, enterprise storage offered by major companies was designed for banks and stock exchanges, which required high-speed data communication, and required prohibitive prices and power. However, the Internet Archive's requirements were high density, low cost, and low power consumption, so they decided to create their own storage system to suit their needs.
Brewster Kale , founder of the Internet Archive and an early engineer at supercomputer company Thinking Machines , built PetaBox using consumer-grade products rather than high-performance RAID arrays. HackerNoon points out that his design philosophy of handling data redundancy in software rather than hardware, without using expensive RAID controllers, was revolutionary at the time.
In 2004, PetaBox had a capacity of about 100TB per rack, but by 2010 this had increased to 480TB per rack, reaching 1.4PB in the generation used between 2024 and 2025. Despite this, power consumption per rack has remained stable at around 6-8kW since the beginning, and because the capacity per drive has increased dramatically, the number of drives managed by the Internet Archive as a whole has remained roughly constant.
One of the unique features of the Internet Archive's infrastructure is a unique thermal management system that harnesses Richmond's cool climate for cooling and reuses excess heat to heat the building. This design choice significantly reduces the facility's power consumption, allowing the organization to allocate limited funds to things like purchasing drives rather than paying for electricity.
The Internet Archive also operates 28,000 drives at any given time, meaning that drives often fail. While a typical company would need to immediately replace a failed drive, the Internet Archive mirrors data across multiple machines and geographically distant data centers to maintain redundancy, preventing problems even if a certain number of drives fail. This low-maintenance design allows a very small team to manage storage on a scale comparable to that of major technology companies.
Crawler
Archiving web pages is not a passive process; it requires software called a crawler to crawl through various web pages and copy what it finds. In the case of the Internet Archive, this has largely been done using
While crawlers like those used by Google focus on extracting text, Heritrix captures the exact state of a web page, including images, stylesheets, embedded objects, etc. Heritrix records not only the page content but also the HTTP headers exchanged between the server and the browser. This metadata contains important information such as when the web page was captured and which server delivered it.
However, while the majority of web pages were static HTML files and hyperlinks when Heritrix was built, the rise of dynamic web pages, including social media feeds and JavaScript, has led to the Internet Archive using tools like Brozzler , which captures web pages exactly as they appear to the user by running JavaScript and opening menus before capturing, and Umbra , which uses browser automation to load content.
The Internet Archive also has a feature called 'Save Page Now,' which allows you to enter the URL of any web page and save the archive. HackerNoon says that this feature democratizes crawling and has become an essential tool for journalists, researchers, and fact-checkers.
◆Financing
The Internet Archive operates one of the most visited websites in the world, yet operates on a budget that is incomparable to that of major technology companies like Google and Meta: annual revenues of $26.8 million in 2024 and expenses of $23.5 million.
The Internet Archive does not rely on advertising or subscriptions, and its main sources of revenue are user donations of $5 to $10 (approximately ¥800 to ¥1600) and grants from various charities. It also offers paid archiving and digitization services, including:
Archive-It: A service that allows libraries and universities to build their own web archives. Subscriptions start at $2,400 per year for 100GB of data, expanding to 1TB for $12,000 per year. This service generates millions of dollars in annual revenue and supports the Internet Archive.
Digitization Services: The Internet Archive operates a digitization center that scans books and other media. The dedicated book scanner, Scribe, can scan books non-destructively, and digitization of bound books starts at $0.15 per page.
Vault: A relatively new service,
Legal disputes
The Internet Archive's mission is 'universal access to all knowledge,' but as its archive expands beyond web pages to include books, music, and software, it has increasingly become embroiled in legal disputes over copyright. For example, in 2020, the Internet Archive expanded its e-book lending service, but a major publishing group sued the organization, claiming that this was copyright infringement, and the Internet Archive lost the case.
Internet Archive loses again in lawsuit claiming e-book lending is fair use - GIGAZINE
The Internet Archive has also been sued by major record companies for its 'Great 78' project, which preserves and makes available records created between 1898 and the 1950s. A settlement was reached in September 2025, but this resulted in the removal of access to many copyrighted recordings.
Internet Archive vs. Record Company Lawsuit Expands to Approximately 100 Billion Yen in Damages, Meanwhile, Moves to Settle - GIGAZINE
Meanwhile, the Internet Archive was designated a Federal Depository Library in July 2025, which holds U.S. government publications. This gives the Internet Archive the legal authority to collect and preserve U.S. government publications and provide access to them. HackerNoon points out that being designated a Federal Depository Library provides a crucial layer of legal protection for parts of the collection.
What will change with the Internet Archive being designated as the US Federal Government's Depository Library? - GIGAZINE
◆ Recent efforts and future prospects
The Internet Archive, which has been under legal threat since 2020, has revealed a serious vulnerability: centralization. To protect its data from court orders or disasters that could strike its headquarters, the Internet Archive is promoting the DWeb movement, which aims to build a decentralized web. Technically, the organization is working to integrate with the InterPlanetary File System (IPFS) , which identifies content by its cryptographic hash rather than its location, and Filecoin , a blockchain-based decentralized storage platform.
The Internet Archive is also working to crawl numerous government websites following the change of US presidency. The 2025 crawl, the year President Donald Trump took office, was the largest ever, collecting over 500TB of government data. This project sees the Internet Archive acting as a 'guardian of history,' ensuring that previously published climate data, census reports, and policy documents are not lost when the new administration takes office.
HackerNoon wrote, 'In the 21st century, the Internet Archive is a contradiction: operating on a scale comparable to a Silicon Valley giant, housed in a church, and run by librarians. It's a fragile institution, subject to lawsuits and budgetary constraints, yet it's also the most robust memory bank humanity has ever built.'
Related Posts:
in Web Service, Posted by log1h_ik