Long-Duration Storage Needs an Active Archive

The Library of Congress contains more than 25 million cataloged books, 74.5 million manuscripts, 5.6 million maps, 8.2 million items of sheet music, 4.2 million audio materials, and 17.3 million visual materials – the oldest item dates from 2040 BC. An archive’s point is storing material for a very long time yet having it readily accessible.

This presents a problem for digital archives. Material on HDD or SSD might only last five years – perhaps ten at most. Tape is much better, with up to 30 years for LTO. But that isn’t enough when attempting to safeguard a society’s knowledge and culture for the next generations that is based entirely on digital records.

Martin Kunze, CMO and Founder of Cerabyte, advocates the use of glass-ceramic storage media. His company has performed aging tests at very high and very low temperatures on this kind of media. These tests indicated a potential storage lifetime of more than a millennium. In addition, the data is not corrupted when exposed to electromagnetic pulses, UV, or gamma rays.

“We need a new accessible permanent storage tier made from inorganic material and with physical bits such as glass or ceramic media that acts as an immutable and easily accessible ledger,” said Kunze. “Cerabyte media is not prone to bit rot, offers high data throughput, and is scalable.”

John Monroe, Chief Analyst at Furthur Market Research, believes that the complex global needs of zettabyte-scale archival storage have been poorly served with expensive solutions that consume an inordinate share of the world’s available energy. “The storage industry is ripe for transformative disruption. In concert and conjunction with tape, new technologies such as Cerabyte’s will be required to provide viable and cost-effective solutions to enterprise customers’ crucial challenges with the security, immutability, and sustainability (SIS) of their vital data,” said Monroe.

Ceramic media is undoubtedly one possible solution. Alternatively, optical storage could offer a solution to data longevity. Currently, it provides a similar longevity time frame as tape, though innovation seeks to change that.

Horst Schellong, Technical Director at Savartus, made the case for optical storage at a recent Active Archive Alliance Video Conference, AI Needs Active Archive. He laid out a near-term storage density roadmap for archive-grade discs – a 12-disc cartridge with a similar form factor to LTO tape.

These multi-layer discs are readily accessible by newly developed drives with more powerful lasers and a multi-layer focus length. Their concurrent drive technology and erasure code options have generated faster data transfer rates.

“Between now and 2030, we will see a 20-fold increase in optical capacity,” said Schellong. “It will achieve the lowest media cost per TB and be backward read/write compatible through all generations.”

Optical storage is a physical write-once-read-many (WORM) format. This prevents data alteration and makes it ideal for archiving. Additionally, it is EMP- and EMR-proof and impervious to most disasters, viruses, or malware. The lasers it uses are a mature technology that has been used for decades. Schellong noted that these discs can last at least 50 to 100 years according to NIST-accelerated life tests.

“The goal is to achieve limited or no migration cycles throughout the life of the data,” he said.

When used in the data center, costs would also be low. This is estimated to be similar to the TCO of LTO when you consider the media cost/TB and a system power consumption of less than 250W/PB. Thus, the cooling requirements within the data center would be very low.

Use cases for this kind of optical storage include:

*Long-term archival storage to meet compliance and retention mandates
*Seamless system integration via an S3 Glacier front end
*A managed offline storage class for air gap-based security

Optical and AI

Such developments are timely when you consider the consequences of AI-based compute. AI is likely to bring about exponential growth in data. This, in turn, will lead to a leap in storage and energy costs. Active archives utilizing new-generation ceramic or optical storage will play a part in relieving the data storage and cost burden of the AI revolution.

AI needs a great many more storage classes with automatic tiering of cold data. It must be integrated with sustainable and long-lasting storage media. This kind of WORM-based storage is a good fit for those needing to meet compliance and regulatory requirements. Such systems can also take advantage of AI-enabled centralized data access and management.