Alluxio 1.7.0 Release

We are excited to announce the release of Alluxio v1.7.0. This release provides significant architecture and usability improvements. Highlights for the release are listed below.

Downloads for this release can be found here: Downloads

Asynchronous caching

Alluxio 1.7.0 improves the performance of cold-reading a block partially or non-sequentially with the default read type. Previously clients used a flag to force full reads of blocks in order to store them into Alluxio to speed up follow up reads on the same blocks. Now these reads for caching purpose will be handled by the Alluxio in the background, vastly decreasing the latency of partial read requests in many workloads. For example, reading the first 10MB of a 512MB block with partial caching on required a read of the entire block (512MB); now the client reads 10MB and continues processing while an Alluxio worker loads the 512MB block in the background.

A tunable configuration for this feature is the alluxio.worker.network.netty.async.cache.manager.threads.max property. This determines the maximum number of concurrent blocks a worker will attempt to cache. By default the value is high (512), but in cases where large amounts of data are expected to be asynchronously cached, lowering the value can reduce resource contention.

Kubernetes integration

Kubernetes is an emerging cluster orchestration framework which supports long-running containerized applications. Alluxio 1.7.0 includes documentation and configurability improvements when deploying on Kubernetes. This release includes recommended practices when deploying Alluxio services as docker containers managed by Kubernetes. Further information about the integration can be found here.

Tiered Locality

The tiered locality feature allows users to take full advantage of locality in their clusters. Alluxio has always supported node-level locality, but now users may specify arbitrary locality levels such as rack, region, or availability zone. This way, clients can prefer to read from workers in the same rack, taking advantage of higher network transfer speeds and reducing cost of data transfer. Further information about tiered locality can be found here.

UFS Synchronization

When applications interact with Alluxio, Alluxio will perform actions to the UFS in order to maintain synchronization. However, if operations are performed out-of-band to the UFS, it is possible that the Alluxio metadata and the UFS metadata are out-of-sync. The existing feature of loading metadata allowed discovery of new UFS files, but did not consider deleted files in UFS. With the UFS synchronization feature, Alluxio will transparently interact with the UFS to sync the metadata before performing an Alluxio operation. Users can configure to enable this transparent syncing on a per-operation basis, by using the user parameter alluxio.user.file.metadata.sync.interval. Information about the configuration parameter can be found here.

Improved FUSE Integration

Alluxio FUSE is moved off the experimental features list and is by default bundled with the distribution tarball. Alluxio FUSE 1.7.0 improves the error handling and adds user/group support. In addition, it supports multiple mounting points on a local machine so users can mount Alluxio to different local directories. Lastly, Alluxio FUSE now supports MacOS. Further information about Alluxio FUSE can be found here.

Acknowledgements

We would like to thank the following members of the community for their contributions to Alluxio 1.7.0. The release would not have been possible without your efforts!

Abhijeet Dhumal, Adit Madan, Aman Singh, Andrew Audibert, Bian Xingyu, Bin Fan, Bin Feng, Binbin Pan, Calvin Jia, Chaomin Yu, Charles, Cheng Chang, Cheng Wei, ChengCheng Mai, Chunyang Tan, CristianoJason, DVampire, Di Yin, Dongqi Tang, Fan Zhifang, Fanvel Chen, Feng Han, Frank Zhang, Gene Pang, Gu Huwang, Gurmeet S Budhraja, Hao Li, Haoyuan Li, Hong Li, Hua Yu, Huiyun Yang, Jiang Xi, Jiang Yuan, Jianrong Chen, Jimmy Song, Johan Andersson, Jun Shi, Kaixuan Huang, Kathy Liu, Ketao Wang, Li Chongjie, Li Lun, Li Pengcheng, Li Yang, Liangliang Yin, Liu Dun, Lu Chenyue, Lu Yi-Jun, Lucas, Luo Kexue, Mao Baolong, Min Chen, Mingtao Ji, Mohamed Aboelhassan, Pang Bo, Pavel Simó, Pei Junyu, Peize Yu, Qi Yang, Qianhui Lu, Qingwei Sun, Qiu Hu, Qiwei Tang, Rangxing Gan, Ren Jie, Shaohua Gao, Shaoqing Tan, Shawy Geng, Shen Hao, Shen Yao, Shenhuan Lv, Shuxin Li, Simeng Wu, Tiancheng Zhang, Tianshuo Zhou, Tianzuo Wang, Tony Tan, Wang Zhen, Wei Sun, Weiwei Hu, Weiwei Zhang, William Zapata, Wu Tianheng, Xiang-Rong Sheng, Xiangyang Luo, Xiaoxu Li, Yanan Jiang, Yang Yao, Yanqin Jin, Yanya Zhang, Yao Xiao, Yawen Pan, Yifan Liu, Yifan Xia, Yijia Huang, Yindong Zheng, Yishun Dou, Yuanyuan Pan, Yupeng Fu, Yuting Chen, Zehan Shen, Zewei Sun, Zhang Huiming, Zhaoxin Huan, Zhaoyang Fu, Zhaoyang Liu, Zhen Jia, Zhixiang Zhang, Zhuokai Shen, Zou Minqiang, FirstKayleinNJU, FujiZ, Jasminest, Mitisky, NJUZengC, Narcissuscyn, WrRan, XINDQ, Zetary, Zhouyangh, chaos, demeiyan, ganjun, imrainkeeper, jixiangyibao, lebronran, qgxiaozhan, roosteracham, sshooting, theOtherWorld, wandoucao, wangl, weanl, xiaoshankeji, yimi1021, yunlaix, zdchow, zhou07, zhtchen