AI is storage’s biggest opportunity

An AI revolution is sweeping through the IT storage world, providing a massively beneficial environment requiring more data to be stored and delivered to AI models and agents, more data to be protected, more data access governance and much better storage operating environments. The downside is that AI can run amok, with data mishaps and deliberate agent-enhanced attacks. We are using AI to make things better and need AI to prevent AI itself from making things worse. Forty-four months ago, when ChatGPT was released, the storage world began an irreversible migration into the AI era. The technology developments needed to provide fast data access to the favorite AI processor, the GPU, revolutionized the NAND and SSD suppliers, the flash array hardware and software vendors and the HPC/supercomputing world. The old and relatively steady, pre-ChatGPT era Dell, HPE, and NetApp-dominated enterprise storage array business met a set of new vendors growing fast, coming from the all-flash array and HPC worlds. Vendors such as DDN, Pure Storage, VAST Data, and WEKA grew rapidly as parallel data access became a key software technology, alongside disaggregated storage array designs, increased use of unstructured data, the rapid growth of analytics, and the emergence of AI-focused data lakes such as Databricks and Snowflake. A whole new public cloud sector emerged: the GPU-as-a-Service neoclouds, such as CoreWeave and Lambda. AI training dominated the early AI storage days but is now being overtaken by AI inference; production AI, with enterprises deploying AI factories to build, tune, deploy, and run their own and third-party AI agents. Model Context Protocol (MCP) and graph technology enable digital employees to act and reason, access and change data, both structured and unstructured. They can make mistakes, meaning that their activities have to be recorded so that they can be reversed if they take a wrong track. These digital employees have to be governed with Agent Identity Access Management. Stepping back for a moment, we can see storage and AI meet in five places: Storage providing data for AI Protecting AI data and actions Storage cyber-resilience extended to govern AI data access Storage using AI in its own operations Storage being protected from AI-driven attacks. Data for AI The dominant GPU vendor, Nvidia, has eagerly supported storage delivery technologies to keep its GPUs running and not being IO-bound. Feeding them data from parallel file systems using disk drive arrays was not good enough. The disks were replaced by SSDs, with NVMe and PCIe interconnects replacing the disk drive era’s SAS and SATA protocols. The transfer of data from a storage array’s x86 controller and its DRAM to the GPU server’s CPU+DRAM subsystem and then to the GPU’s capacity-limited high-bandwidth memory (HBM) was too slow. GPUDirect cut out the storage array controller and its memory from the data flow, with RDMA access from the SSDs. This was applied to files first and then to objects with S3 over RDMA-type technologies. Cloudian, MinIO and Scality have been active here. KV caching schemes have been set up to logically extend a GPU’s HBM to SSDs, reduce HBM data load waits and avoid token recomputation. Nvidia has partnered with the main enterprise storage array vendors to make this happen. Disaggregated storage array (DASE) technology, pioneered by VAST Data, has been adopted by Dell, Everpure, HPE and NetApp. Monolithic, high-end storage arrays from Hitachi Vantara, IBM, and Lenovo-acquired Infinidat have not adopted DASE, nor GPUDirect and not KV caching. Their architecture precludes them from joining in and they are becoming the storage equivalent of mainframes, a left-behind but still necessary niche. AI models process tokens, which are turned into vector embedding data that needs storing and searching. Dedicated vector database suppliers have sprung up, such as Pinecone, Qdrant, Weaviate, and Zilliz, while multi-model OLAP and OLTP databases have added vector support, SingleStore being an example. AI data pipeline technologies are being developed to enable an organization’s entire data estate to be used as an AI data source, but without copying it to a single repository. Apache Iceberg is being used to give data lakes access to external storage and logically bring it into the data lake’s namespace. The monolithic arrays are, of course, data sources for AI pipelines and will contribute. Data management suppliers doubled down on initiatives to map organizations’ data and make it available, via classification, selection, filtering and metadata access so as to protect privileged data, reduce bulk data movements, and accelerate targeted data movement; think Arcitecta, Datadobi, Hammerspace and Komprise. An advantage of DASE array architecture is that storage is decoupled from compute, and the compute can be scaled up to run AI technologies directly on the array. This means that array vendors’ software stacks can be extended upwards into AI data pipelines. They can have the capability to provide services for AI agents and become, in effect, AI operating systems. Indeed VAST Data explicitly calls its extended SW stack its AIOS. Protecting AI data and actions Backup and cyber-resilience vendors, such as Cohesity, Commvault, Druva, Rubrik, and Veeam, have recognized that their backups represent a great data source for AI models and agents. They developed internal AI agents, such as Cohesity’s Gaia, its Gen AI search assistant, to build on this idea. The backup vendors see that their backups contain a temporal record of data changes but they, the changes, have been made by different entities in an organization’s IT estate and are not correlated. AI agents can monitor the backups and find links between events that indicate an existing or developing cyber attack. They can identify the start of an attack, the affected data, the last known good copy of that data, and restore it, helping with cyber-attack response and re...

Indexed and credited by AIPROPX. Originating outlet: The Register. Open at source →