What Alluxio is
Alluxio is a data layer between compute and storage, abstracting the files or objects in underlying persistent storage systems and providing a shared data access layer for compute applications. Underlying storage can be on-premise or in the cloud as indicated in the diagram. Benefits include:
Memory-Speed I/OAlluxio is used as a distributed shared caching service so that compute applications can transparently cache frequently accessed data, especially data from remote locations, to provide in-memory I/O throughput.
Enabling Cloud or Object StorageEmerging cloud or object storage provide different semantics and have performance implications compared to traditional file systems. Applications accessing a cloud storage get no node-level locality or cross-application caching, and see huge overhead in common filesystem operations like "ls" or "mv". Deploying Alluxio connected to cloud or object storage closes this gap and achieves a significant performance gain.
Easy Data ManagementAlluxio creates a single point of access to multiple data sources. For example, if you have multiple versions of HDFS in your enterprise or multiple cloud storage vendors, Alluxio gives an application the ability to talk to different versions of the same storage, without complex system configuration and management.
Easy Application DeploymentAlluxio manages communication between applications and file or object storage, translating data access requests from applications to any persistent underlying storage interface. When migrating data from one persistent store to another, no application changes are required.
Case StudiesAlluxio is one of the fastest growing open source projects in the big data ecosystem with over 800 contributors from over 100 organizations contributing 29,000+ commits and earning 3,600+ stars and counting. It is used in production in some of the world’s largest deployments from companies like NVIDIA, Wells Fargo, Tencent, Baidu, Didi, and more. Click below to learn more.
New York Meetup Presentations (Sep 17th, 2018)