Building Alluxio Master Branch
- Unit Tests
- Compute Framework Support
- Distro Support
- System Settings
This guide describes how to compile Alluxio from the beginning.
Checkout the Alluxio master branch from Github and build the source code:
$ git clone git://github.com/alluxio/alluxio.git $ cd alluxio $ mvn install -DskipTests
If you are seeing
java.lang.OutOfMemoryError: Java heap space, please execute:
$ export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
If you want to build a particular version of Alluxio, for example 1.7.1, please do
git checkout v1.7.1 after
The Maven build system fetches its dependencies, compiles source code, runs unit tests, and packages the system. If this is the first time you are building the project, it can take a while to download all the dependencies. Subsequent builds, however, will be much faster.
Once Alluxio is built, you can start it with:
$ ./bin/alluxio format $ ./bin/alluxio-start.sh local
To verify that Alluxio is running, you can visit http://localhost:19999 or check the log in the
alluxio/logs directory. You can also run a simple program:
$ ./bin/alluxio runTests
You should be able to see results similar to the following:
/default_tests_files/BasicFile_STORE_SYNC_PERSIST has been removed 2015-10-20 23:02:54,403 INFO (ClientBase.java:connect) - Alluxio client (version 1.0.0) is trying to connect with FileSystemMaster master @ localhost/127.0.0.1:19998 2015-10-20 23:02:54,422 INFO (ClientBase.java:connect) - Client registered with FileSystemMaster master @ localhost/127.0.0.1:19998 2015-10-20 23:02:54,460 INFO (BasicOperations.java:createFile) - createFile with fileId 1476395007 took 65 ms. 2015-10-20 23:02:54,557 INFO (ClientBase.java:connect) - Alluxio client (version 1.0.0) is trying to connect with BlockMaster master @ localhost/127.0.0.1:19998 2015-10-20 23:02:54,558 INFO (ClientBase.java:connect) - Client registered with BlockMaster master @ localhost/127.0.0.1:19998 2015-10-20 23:02:54,590 INFO (WorkerClient.java:connect) - Connecting local worker @ /192.168.31.242:29998 2015-10-20 23:02:54,654 INFO (FileUtils.java:createStorageDirPath) - Folder /Volumes/ramdisk/alluxioworker/6601007274872912185 was created! 2015-10-20 23:02:54,657 INFO (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /Volumes/ramdisk/alluxioworker/6601007274872912185/1459617792 2015-10-20 23:02:54,658 INFO (WorkerClient.java:connect) - Connecting local worker @ /192.168.31.242:29998 2015-10-20 23:02:54,754 INFO (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_STORE_SYNC_PERSIST took 294 ms. 2015-10-20 23:02:54,803 INFO (BasicOperations.java:readFile) - readFile file /default_tests_files/BasicFile_STORE_SYNC_PERSIST took 47 ms. Passed the test!
You can also stop the system by using:
$ ./bin/alluxio-stop.sh local
To run all unit tests:
$ mvn test
To run all the unit tests with under storage other than local filesystem:
$ mvn test [ -Dhadoop.version=x.x.x ] [ -P<under-storage-profile> ]
Currently supported values for
Not Specified # [Default] Tests against local file system swiftTest # Tests against a simulated Swift cluster hdfsTest # Tests against HDFS minicluster glusterfsTest # Tests against GlusterFS s3Test # Tests against Amazon S3 (requires a real S3 bucket) ossTest # Tests against Aliyun OSS (requires a real OSS bucket) gcsTest # Tests against Google Cloud Storage (requires a real GCS bucket)
To have the logs output to STDOUT, append the following to the
Compute Framework Support
To build Alluxio against different compute frameworks, you can run Maven build with different compute profiles. The generated Alluxio client is located at
You can run the following command to compile Alluxio with Hadoop.
$ mvn install -P<HADOOP_PROFILE> -DskipTests
Available Hadoop profiles include
different distributions of Hadoop. You can further set a specific Hadoop release version to
compile by checking out this section.
Spark/Flink/Presto and etc
You can run the following command to compile Alluxio server and client jars that work with different compute frameworks.
$ mvn install -DskipTests
To build Alluxio against one of the different distros of hadoop, you only need to change the
hadoop.version. You can run the following command:
$ mvn install -P<HADOOP_PROFILE> -Dhadoop.version=<HADOOP_VERSION> -DskipTests
<HADOOP_VERSION> can be set for different distributions.
All main builds are from Apache so all Apache releases can be used directly
-Dhadoop.version=2.2.0 -Dhadoop.version=2.3.0 -Dhadoop.version=2.4.0
To build against Cloudera’s releases, just use a version like
To build against a MapR release
-Dhadoop.version=2.7.0-mapr-1607 -Dhadoop.version=2.7.0-mapr-1602 -Dhadoop.version=2.7.0-mapr-1506 -Dhadoop.version=2.3.0-mapr-4.0.0-FCS
To build against a Pivotal release, just use a version like
To build against a Hortonworks release, just use a version like
-Dhadoop.version=184.108.40.206.0.5.0-67 -Dhadoop.version=220.127.116.11.1.0.0-92 -Dhadoop.version=18.104.22.168.1.3.0-563
Sometimes you will need to play with a few system settings in order to have the unit tests pass locally. A common setting that may need to be set is ulimit.
In order to increase the number of files and processes allowed, run the following
$ sudo launchctl limit maxfiles 32768 32768 $ sudo launchctl limit maxproc 32768 32768
It is also recommended to exclude your local clone of Alluxio from Spotlight indexing. Otherwise, your Mac may hang constantly trying to re-index the file system during the unit tests. To do this, go to
System Preferences > Spotlight > Privacy, click the
+ button, browse to the directory containing your local clone of Alluxio, and click
Choose to add it to the exclusions list.