(Experimental) Key Value System Client API

Overview

In addition to Filesystem API which allows applications to read, write or manage files, Alluxio also serves key-value system on top of Alluxio filesystem. Like files in Alluxio filesystem, the semantics of key-value system are also write-once:

  • Users can create a key-value store and insert key-value pairs into the store. A store becomes immutable after it is complete.
  • Users can open key-value stores after they are complete.

Each single key-value store is denoted by an AlluxioURI like alluxio://path/my-kvstore. Depending on the total size and block size specified by the user, a single key-value store may consist of one or multiple partitions, but the internal is managed by Alluxio and thus transparent to users.

Configuration Parameters For Key-Value System

Key-Value support in Alluxio is disabled by default, and it can be enabled in Alluxio by setting alluxio.keyvalue.enabled to true (see configuration parameters)

Property NameDefaultMeaning
alluxio.keyvalue.enabled false Whether the keyvalue interface is enabled.
alluxio.keyvalue.partition.size.bytes.max 512MB Maximum size of each partition.

Quick Test

After enabling Key-Value support in Alluxio, you can run ./bin/alluxio runKVTest to test whether the Key-Value system is running. You should see Passed the test! at the bottom of the output if the Key-Value system is correctly started.

Accessing Key-Value System in Java Application

Getting a Key-Value System Client

To obtain an Alluxio key-value system client in Java code, use:

KeyValueSystem kvs = KeyValueSystem.Factory.create();

Creating a new key-value store

To create a new key-value store, use KeyValueSystem#createStore(AlluxioURI), which returns a writer to add key-value pairs. For example:

KeyValueStoreWriter writer = kvs.createStore(new AlluxioURI("alluxio://path/my-kvstore"));
// Insert key-value pair ("100", "foo")
writer.put("100", "foo");
// Insert key-value pair ("200", "bar")
writer.put("200", "bar");
// Close and complete the store
writer.close();

Note that,

  • Before the writer closes, the store is not complete and can not be read;
  • It is possible that the store is larger than the maximum allowed size of one partition, and in this case, the writer will save key-value pairs into multiple partitions. But the switch is transparent.
  • The keys to insert should be sorted and with no duplicated keys.

Retrieving value from a store

To query a complete key-value store, use KeyValueSystem#openStore(AlluxioURI), which returns a reader to retrieve value by the key. For example:

KeyValueStoreReader reader = kvs.openStore(new AlluxioURI("alluxio://path/kvstore/"));
// Return "foo"
reader.get("100");
// Return null as no value associated with "300"
reader.get("300");
// Close the reader on the store
reader.close();

Iterating key-value pairs over a store

KeyValueStoreReader reader = kvs.openStore(new AlluxioURI("alluxio://path/kvstore/"));
KeyValueIterator iterator = reader.iterator();
while (iterator.hasNext()) {
  KeyValuePair pair = iterator.next();
  ByteBuffer key = pair.getkKey();
  ByteBuffer value = pair.getValue();
}
// Close the reader on the store
reader.close()

Examples

See more examples in the codebase.

Accessing Key-Value System in Hadoop MapReduce

MapReduce InputFormat

Alluxio provides an implementation of InputFormat for Hadoop MapReduce programs to access a key-value store. It takes a key-value URI, and emits key-value pairs stored in the store:

conf.setInputFormat(KeyValueInputFormat.class);
FileInputFormat.setInputPaths(conf, new Path("alluxio://input-store"));

MapReduce OutputFormat

Similarly, Alluxio also provides implementations of OutputFormat and OutputCommitter for Hadoop MapReduce programs to create a key-value store by taking a key-value URI, and saving key-value pairs to the key-value store:

conf.setOutputKeyClass(BytesWritable.class);
conf.setOutputValueClass(BytesWritable.class);
conf.setOutputFormat(KeyValueOutputFormat.class);
FileOutputFormat.setOutputPath(conf, new Path("alluxio://output-store"));

Examples

See an example in the codebase.

If you have configured Alluxio to use HDFS as under storage, and have enabled Key-Value system, you can run the example via

$ export HADOOP_CLASSPATH=${ALLUXIO_INSTALLATION_DIRECTORY}/assembly/target/alluxio-assemblies-${ALLUXIO_VERSION}-jar-with-dependencies.jar

$ ${HADOOP_INSTALLATION_DIRECTORY}/bin/hadoop jar \
  ${ALLUXIO_INSTALLATION_DIRECTORY}/examples/target/alluxio-examples-${ALLUXIO_VERSION}.jar \
  alluxio.examples.keyvalue.hadoop.CloneStoreMapReduce alluxio://${ALLUXIO_MASTER}:${PORT}/${INPUT_KEY_VALUE_STORE_PATH} alluxio://${ALLUXIO_MASTER}:${PORT}/${OUTPUT_KEY_VALUE_STORE_PATH} \
  -libjars=${ALLUXIO_INSTALLATION_DIRECTORY}/assembly/target/alluxio-assemblies-${ALLUXIO_VERSION}-jar-with-dependencies.jar
Need help? Ask a Question