Active Archiving for the Edge – What Are Your Options?
The “Edge” has increasingly become a hot topic as employees are often remote, IoT proliferates and Remote Office / Branch Offices increasingly need to send and retrieve archived data. On top of this are the “distant edge” applications, places on Earth that have little or no direct connection to the outside world such as oil rigs, research or underwater vessels, weather or seismic sensors, or even some sites using video surveillance.
There are three main methods of archiving from the edge.
1. Wired / Satellite Connection – previously WAN / VPN increasingly S3
2.Disk-based storage – NAS RAID or S3-based Object Storage
3.Removable media based – software creates media sets typically using tape
Wired / Satellite Connection
Wired or Satellite transmission of data through dedicated or shared connections has rapidly expanded as infrastructure has increased and improved. In the past this involved VPNs (Virtual Private Networks) and/or WAN (Wide Area Networks) which added complexity and another layer to fail. Today, many organizations can store remote data using a “compatible” S3 API. Free or low-cost S3 clients allow content to be manually or automatically archived to Private or Public Clouds and even s3-based tape libraries, allowing access to all who need it. Data is written using SSL and perhaps file level encryption as well. The downside is that connection speeds are still relatively slow or, if they are high speed then they are expensive. Shared internet connections often offer high download speeds but slower upload so archiving from remote sites can be much slower than expected. A dedicated 1Gbit Ethernet upload connection can transmit a maximum of 8TB per day (assumes no downloads on same link) and costs about $20k per year. This is the easiest method to connect remote sites with a central archive.
Disk-based Storage
For distant edge sites, another method of archiving data from the Edge is to physically transport a disk-based system (either NAS RAID or S3 Object Storage). Data is stored to a large disk system and periodically shipped to the required location (Cloud or Primary Data Center) and copied there – before the unit is shipped back and potentially re-used for new content. Typically, this requires at least two units as one is always needed at the Edge to store data produced while the other unit is in transit. Units use 8 to 12 x Hard Drives in each enclosure – with 20TB hard drives now available that is 240TB raw capacity and from 200TB useable. The latest hard drives (Seagate Exos 2X18 – dual actuator) has a performance of 550MB per sec, using RAID6 with 12 drives this provides total performance of 1.1GB per sec which means about 90TB per day. Standard, single actuator hard disk systems would have a performance half this – so 45TB per day maximum. Data is secured using RAID levels or Erasure Coding (for Object Storage) and encryption is also used for increased security. Units are transported in shock resistant cases for transportation and systems weigh about 40lbs (18Kg) or about 4TB per lb. Disk-based Systems can be purchased from around $16,500 or $82.50 per TB.
Removable Media-based Storage
The final alternative is to use removable media which today mainly means using LTO tape. With latest generation LTO9, that equates to 18TB per media and a transfer rate of 400MB/sec (uncompressed) or the ability to write 32TB per day (uncompressed) per tape drive. As only media is transported the weight (and therefore cost of transport) is significantly reduced. LTO cartridges weigh 0.4 lbs. – so giving 45TB per lb. (10 times that of the disk system). LTO media is more secure than disks (10,000 more reliable based on Bit Error Rate). Encryption and compression can be handled by the tape drive. With a software application, multiple tape media can be spanned as a complete set of data and multiple drives can be aggregated to write a large set of files, allowing a similar top performance (1.2GB per sec) as the disk systems above using only 3 x LTO9 drives. Media can then be mounted at the destination site, it can be immediately accessed via SMB, NFS and / or S3 if this data is its own set, or copied to the organizations central archive. Tape can use the industry standard LTFS file system (where different tape applications are used at each site) or for additional security, an optimized proprietary tape file system can be used (if the same application is used at the Edge and Central locations). LTO9 media retails for about $110 per cartridge or $6.11 per TB. Of course, the solution also requires a tape drive(s) or library at both locations. A single LTO9 drive costs about $4,000 and a library starts at approximately $10,000.
| Wired (if available) | Disk-Based | Tape-Based | |
| Maximum Capacity archived per day | 8TB per day – dedicated 1Gbit with balanced upload / download speeds) | 45TB per day – standard disk drives
90TB per day – new drives with dual actuators |
32TB per day – 1 x LTO9 drive
90TB per day – 3 x LTO9 drive library |
| Cost | $20,000 per year – 1Gbit Dedicated Ethernet (USA) | $16,500 for 200TB – two systems required if data collection is constant (one in transit and one in place) | $4,000 for tape drive
$110 per 18TB tape media – double media required if data collection is constant (one set in transit and one in place) |
All three options are valid for archiving data from the edge, the choice should be made based on overall cost and capacity created at the edge. If available, wired transmission is certainly the easiest and allows edge users to also use the central archive. S3 is not only cloud but can also be used with centralized object storage or even tape-based archives. Transporting tape cartridges is a very low-cost method of migrating large capacities, but does require hardware at both locations. This is probably used most where there is a steady / constant need to move data to a central location, whereas a disk system is used where periodic or one-off migrations are needed, for example an on-location video shoot.
