Building Alluxio Master Branch

This guide describes how to compile Alluxio from the beginning.

The prerequisite for this guide is that you have Java 7 (or above), Maven, and Thrift 0.9.3 (Optional) installed.

Checkout the Alluxio master branch from Github and package:

$ git clone git://github.com/alluxio/alluxio.git
$ cd alluxio
$ mvn install -DskipTests

If you are seeing java.lang.OutOfMemoryError: Java heap space, please execute:

$ export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

If you want to build a particular version of Alluxio, for example 1.6.1, please do git checkout v1.6.1 after cd alluxio.

The Maven build system fetches its dependencies, compiles source code, runs unit tests, and packages the system. If this is the first time you are building the project, it can take a while to download all the dependencies. Subsequent builds, however, will be much faster.

Once Alluxio is built, you can start it with:

$ ./bin/alluxio format
$ ./bin/alluxio-start.sh local

To verify that Alluxio is running, you can visit http://localhost:19999 or check the log in the alluxio/logs directory. You can also run a simple program:

$ ./bin/alluxio runTests

You should be able to see results similar to the following:

/default_tests_files/BasicFile_STORE_SYNC_PERSIST has been removed
2015-10-20 23:02:54,403 INFO   (ClientBase.java:connect) - Alluxio client (version 1.0.0) is trying to connect with FileSystemMaster master @ localhost/127.0.0.1:19998
2015-10-20 23:02:54,422 INFO   (ClientBase.java:connect) - Client registered with FileSystemMaster master @ localhost/127.0.0.1:19998
2015-10-20 23:02:54,460 INFO   (BasicOperations.java:createFile) - createFile with fileId 1476395007 took 65 ms.
2015-10-20 23:02:54,557 INFO   (ClientBase.java:connect) - Alluxio client (version 1.0.0) is trying to connect with BlockMaster master @ localhost/127.0.0.1:19998
2015-10-20 23:02:54,558 INFO   (ClientBase.java:connect) - Client registered with BlockMaster master @ localhost/127.0.0.1:19998
2015-10-20 23:02:54,590 INFO   (WorkerClient.java:connect) - Connecting local worker @ /192.168.31.242:29998
2015-10-20 23:02:54,654 INFO   (FileUtils.java:createStorageDirPath) - Folder /Volumes/ramdisk/alluxioworker/6601007274872912185 was created!
2015-10-20 23:02:54,657 INFO   (LocalBlockOutStream.java:<init>) - LocalBlockOutStream created new file block, block path: /Volumes/ramdisk/alluxioworker/6601007274872912185/1459617792
2015-10-20 23:02:54,658 INFO   (WorkerClient.java:connect) - Connecting local worker @ /192.168.31.242:29998
2015-10-20 23:02:54,754 INFO   (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/BasicFile_STORE_SYNC_PERSIST took 294 ms.
2015-10-20 23:02:54,803 INFO   (BasicOperations.java:readFile) - readFile file /default_tests_files/BasicFile_STORE_SYNC_PERSIST took 47 ms.
Passed the test!

You can also stop the system by using:

$ ./bin/alluxio-stop.sh local

Unit Tests

To run all unit tests:

$ mvn test

To run all the unit tests with under storage other than local filesystem:

$ mvn test [ -Dhadoop.version=x.x.x ] [ -P<under-storage-profile> ]

Currently supported values for <under-storage-profile> are:

Not Specified # [Default] Tests against local file system
swiftTest     # Tests against a simulated Swift cluster
hdfsTest      # Tests against HDFS minicluster
glusterfsTest # Tests against GlusterFS
s3Test        # Tests against Amazon S3 (requires a real S3 bucket)
ossTest       # Tests against Aliyun OSS (requires a real OSS bucket)
gcsTest       # Tests against Google Cloud Storage (requires a real GCS bucket)

To have the logs output to STDOUT, append the following to the mvn command

-Dtest.output.redirect=false -Dalluxio.root.logger=DEBUG,CONSOLE

Compute Framework Support

To build Alluxio against different compute frameworks, you can run Maven build with different compute profiles. The generated Alluxio client is located at /<PATH_TO_ALLUXIO>/client/alluxio-1.7.0-SNAPSHOT-client.jar.

Hadoop

You can run the following command to compile Alluxio with Hadoop.

$ mvn install -P<HADOOP_PROFILE> -DskipTests

Available Hadoop profiles include hadoop-1, hadoop-2.2, hadoop-2.3hadoop-2.8 for different distributions of Hadoop. You can further set a specific Hadoop release version to compile by checking out this section.

Spark/Flink/Presto and etc

You can run the following command to compile Alluxio server and client jars that work with different compute frameworks.

$ mvn install -DskipTests

Distro Support

To build Alluxio against one of the different distros of hadoop, you only need to change the hadoop.version. You can run the following command:

$ mvn install -P<HADOOP_PROFILE> -Dhadoop.version=<HADOOP_VERSION> -DskipTests

where <HADOOP_VERSION> can be set for different distributions.

Apache

All main builds are from Apache so all Apache releases can be used directly

-Dhadoop.version=2.2.0
-Dhadoop.version=2.3.0
-Dhadoop.version=2.4.0

Cloudera

To build against Cloudera’s releases, just use a version like $apacheRelease-cdh$cdhRelease

-Dhadoop.version=2.3.0-cdh5.1.0
-Dhadoop.version=2.0.0-cdh4.7.0

MapR

To build against a MapR release

-Dhadoop.version=2.7.0-mapr-1607
-Dhadoop.version=2.7.0-mapr-1602
-Dhadoop.version=2.7.0-mapr-1506
-Dhadoop.version=2.3.0-mapr-4.0.0-FCS

Pivotal

To build against a Pivotal release, just use a version like $apacheRelease-gphd-$pivotalRelease

-Dhadoop.version=2.0.5-alpha-gphd-2.1.1.0
-Dhadoop.version=2.2.0-gphd-3.0.1.0

Hortonworks

To build against a Hortonworks release, just use a version like $apacheRelease.$hortonRelease

-Dhadoop.version=2.1.0.2.0.5.0-67
-Dhadoop.version=2.2.0.2.1.0.0-92
-Dhadoop.version=2.4.0.2.1.3.0-563

System Settings

Sometimes you will need to play with a few system settings in order to have the unit tests pass locally. A common setting that may need to be set is ulimit.

Mac

In order to increase the number of files and processes allowed, run the following

$ sudo launchctl limit maxfiles 32768 32768
$ sudo launchctl limit maxproc 32768 32768

It is also recommended to exclude your local clone of Alluxio from Spotlight indexing. Otherwise, your Mac may hang constantly trying to re-index the file system during the unit tests. To do this, go to System Preferences > Spotlight > Privacy, click the + button, browse to the directory containing your local clone of Alluxio, and click Choose to add it to the exclusions list.

Need help? Ask a Question