Flume agents can store data on an HDFS filesystem. For clusters configured to use Kerberos authentication, Flume requires a Kerberos principal and keytab to authenticate to the cluster, which then interacts with HDFS, MapReduce, and other cluster services on behalf of the Flume agent.

Enabling Flume to use Kerberos authentication on a cluster assumes that cluster has been configured to integrate with Kerberos. See Configuring Hadoop Security in CDH 5 for details.

The steps below have been tested with CDH 5 and MIT Kerberos 5 only. The discussion includes an example of configuring user flume for Kerberos authentication as an HDFS client. Configuring authentication between Flume agents is not covered.

Continue reading:

• Configuring Kerberos for Flume Sinks
• Configuring Kerberos for Flume Thrift Source and Sink Using Cloudera Manager
• Configuring Kerberos for Flume Thrift Source and Sink Using the Command Line
• Flume Account Requirements
• Testing the Flume HDFS Sink Configuration
• Writing to a Secure HBase Cluster
• Using Substitution Variables for Flume Kerberos Principal and Keytab