How Storage Must Evolve to Serve the Metaverse
During the Active Archive virtual conference, John Monroe of Furthur Market Research treated the audience to an in-depth briefing on the future of storage. Here are some of the highlights:
The evolving virtual world of the metaverse is ramping up the need for enterprise storage once again. After all, the metaverse must be underpinned by a vast pool of storage, which might be termed the dataverse. And the dataverse will be one of the hottest areas of IT innovation in the coming years.
Managing a multi-million petabyte (PB) dataverse over increasingly lengthy time periods is giving rise to new use cases for traditional storage technologies while acting as a catalyst for creating new, more cost-effective, and more power-efficient storage. These technologies must be capable of storing data for 10 years, 20 years, 50 years, or even longer periods of time.
We now have billions and billions of people, systems, and sensors connected in a global network. They have generated and will continue to create immense quantities of data. The data lakes we discussed in 2010 were only data pools at the turn of the century. They became data oceans in recent years, and now it is evolving into a huge multi-million PB dataverse.
How big can it grow? The immensity of the dataverse might exceed 30 zettabytes by 2030. That’s up from less than 5 TB in 2022, almost double what it was in 2020. About 60% of those ZBs will be required for archiving.
Flash Expansion Stalls
Enterprise flash will continue to expand in the coming years, but less than predicted. It will exceed 350 exabytes (EB) by 2025 and more than 1,250 EB by 2030. Viewed in terms of percentage of the total annual storage shipments, enterprise flash will still only amount to 15.6% by 2030.
Meanwhile, hard disk drives (HDDs) will grow to more than 1,750 EB by 2025 and surge to more than 5,000 EB by 2030. By that point, however, HDDs will actually be in decline at 62.3% of total shipments in 2030, down from a peak of 72% in 2020. This drop-off accelerates through 2035, when HDD capacity shipments will fall to less than 2,0000 EB, accounting for only 15.6% of the market. By that point, 3,500 or more EBs of flash will ship (29.1%). The rest will comprise tape, photonic, and other forms of storage.
Tape usage peaked around 2010 at 40% of total PB shipped. It fell to 14.4% in 2020. Tape usage is expected to remain relatively flat till 2030 and then recover through 2035 to account for more than 20% of the market. This trajectory is due to the need for active archives that can store huge quantities of cold data yet make it available relatively quickly.
The big unknown factor in all this is photonic (primarily particle and optical storage) and other evolving forms of storage. By 2025, they should make up 2.4% of annual capacity shipments, almost 10% by 2030 and perhaps more than 50% by 2035. But this is the big unknown factor.
In whatever way the market shakes out, the quantity of storage is exploding and will continue to do so. From 91 EB of storage in 2010, we could see a dataverse of more than 50,000 EB by 2035.
Manufacturing Limitations
Some have predicted that flash will take over everything and its price point will continue to fall. These predictions are overly optimistic and are likely to run aground due to manufacturing limitations impacting both flash and hard drives.
HDD prices are already down to around a cent per gigabyte and will probably fall to as little as 0.6 cents. Solid State Drives (SSDs) will never get anywhere near that price point. Even if they did, predictions of flash dominance run aground due to manufacturing realities. There are only so many fabs out there and only so many planned. Further, they cater to a huge consumer and cellular markets. The maximum production capability of NAND fabs is probably going to be around 3 ZB by 2035. 3 ZB is woefully inadequate to service the needs of the enterprise.
Yes, there will be increasingly more flash as a proportion of the total. But the proportion increasing at the greatest rate will be the active archive technologies that demand a whole lot less in terms of cost per GB and energy usage. It is a little-known fact that flash consumes even more power than hard drives. Sustainability, then, is yet another area that will inhibit flash growth.
HDDs will suffer by 2035. But even then, they still will represent 1.7 ZB. That’s considerably more than shipped last year. Just as tape didn’t die, neither will HDDs.
Our dataverse, therefore, will comprise all forms of storage: disk, tape, flash, optical, and others we have yet to conceive.
You can hear John’s keynote talk and learn more about emerging use cases for active archives by listening to the recordings of the 2023 Active Archive Virtual Conference.
