Cloudera Enterprise 5.15.x | Other versions

Configuring Flume Security with Kafka

In CDH versions 5.15.0 and later, you can use Cloudera Manager to configure Flume to communicate with Kafka sources, sinks, and channels over TLS.

When you add a Kafka service as a dependent of the Flume service, Cloudera Manager creates jaas.conf and flume.keytab files. Cloudera Manager also scans the Flume configuration and adds Kafka security properties to it. The properties are added at the beginning of the file so you can easily locate and modify them if needed. The following Kafka security properties are added to the Flume configuration:

kafka.consumer.security.protocol
kafka.consumer.sasl.kerberos.service.name
kafka.consumer.ssl.truststore.location
kafka.consumer.ssl.truststore.password

Note that changing the Kafka broker security settings will not update the Flume configuration.

Complete the following steps to configure Flume to connect to secure Kafka:

In Cloudera Manager, open the Flume service.
Open the Configuration tab.
Use the Search field to search for Kafka. The Kafka Service property is displayed. Select the Kafka service that you want the Flume service to connect to.

The following image shows an example of the Kafka Service property with the KAFKA-1 service selected:

flume.keytab

Cloudera Manager automatically creates the flume.keytab file. However, if you need to edit the file, you can find it in the following location:

/var/run/cloudera-scm-agent/process/<latest_id>-flume-AGENT/flume.keytab

The file must not be empty on any host that runs a kerberized Flume agent.

Principal management is handled by Cloudera Manager for Flume, just as with other services. For example, principals are listed on the Administration > Security > Kerberos Credentials page in Cloudera Manager.

jaas.conf

Cloudera Manager also creates a flafka_jaas.conf file on each host that runs a Flume agent. You do not need to create or edit the file manually. The following information is provided for troubleshooting.

The configuration information in the file is used to communicate with Kafka and also provide normal Flume Kerberos support. The flafka_jaas.conf file contains two entries for the Flume principal: Client and KafkaClient. Note that the principal property is host specific. Unix user flume must have read permission for this file.

/opt/cloudera/security/flafka_jaas.conf:
    Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="flume.keytab"
    principal="flume/cornhost-1.gce.acmecorn.com@GCE.ACMECORN.COM";
    };
    
    KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    serviceName="kafka" 
    keyTab="flume.keytab"
    principal="flume/cornhost-1.gce.acmecorn.com@GCE.ACMECORN.COM";
    };

Page generated May 18, 2018.