kafka Authentication using SASL/Kerberos

Authentication using SASL/Kerberos

1. Prerequisites

   1. Kerberos
      If your organization is already using a Kerberos server (for example, by using Active Directory), there is no need to install a new server just for Kafka. Otherwise you will need to install one, your Linux vendor likely has packages for Kerberos and a short guide on how to install and configure it (Ubuntu, Redhat). Note that if you are using Oracle Java, you will need to download JCE policy files for your Java version and copy them to $JAVA_HOME/jre/lib/security.
   2. Create Kerberos Principals
      If you are using the organization's Kerberos or Active Directory server, ask your Kerberos administrator for a principal for each Kafka broker in your cluster and for every operating system user that will access Kafka with Kerberos authentication (via clients and tools).
      If you have installed your own Kerberos, you will need to create these principals yourself using the following commands:

      1 2	sudo /usr/sbin/kadmin . local -q 'addprinc -randkey kafka/{hostname}@{REALM}' sudo /usr/sbin/kadmin . local -q "ktadd -k /etc/security/keytabs/{keytabname}.keytab kafka/{hostname}@{REALM}"

   3. Make sure all hosts can be reachable using hostnames - it is a Kerberos requirement that all your hosts can be resolved with their FQDNs.

2. Configuring Kafka Brokers

   1. 1. Add a suitably modified JAAS file similar to the one below to each Kafka broker's config directory, let's call it kafka_server_jaas.conf for this example (note that each broker should have its own keytab):

         1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

         KafkaServer {      com.sun.security.auth.module.Krb5LoginModule required      useKeyTab=true      storeKey=true      keyTab="/etc/security/keytabs/kafka_server.keytab"      principal="kafka/kafka1.hostname.com@EXAMPLE.COM"; };   // Zookeeper client authentication Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka_server.keytab" principal="kafka/kafka1.hostname.com@EXAMPLE.COM"; };

   KafkaServer
   1. 1.  section in the JAAS file tells the broker which principal to use and the location of the keytab where this principal is stored. It allows the broker to login using the keytab specified in this section. See 
   notes
   1. 1.  for more details on Zookeeper SASL configuration.
      2. Pass the JAAS and optionally the krb5 file locations as JVM parameters to each Kafka broker (see here for more details):

             -Djava.security.krb5.conf=/etc/kafka/krb5.conf
                 -Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf

      3. Make sure the keytabs configured in the JAAS file are readable by the operating system user who is starting kafka broker.
      4. Configure SASL port and SASL mechanisms in server.properties as described here. For example:

             listeners=SASL_PLAINTEXT://host.name:port
                 security.inter.broker.protocol=SASL_PLAINTEXT
                 sasl.mechanism.inter.broker.protocol=GSSAPI
                 sasl.enabled.mechanisms=GSSAPI
                     

         We must also configure the service name in server.properties, which should match the principal name of the kafka brokers. In the above example, principal is "kafka/kafka1.hostname.com@EXAMPLE.com", so:

       sasl.kerberos.service.name=kafka

3. Configuring Kafka Clients

   To configure SASL authentication on the clients:
   1. Clients (producers, consumers, connect workers, etc) will authenticate to the cluster with their own principal (usually with the same name as the user running the client), so obtain or create these principals as needed. Then configure the JAAS configuration property for each client. Different clients within a JVM may run as different users by specifying different principals. The property sasl.jaas.config in producer.properties or consumer.properties describes how clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client using a keytab (recommended for long-running processes):

          sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
              useKeyTab=true \
              storeKey=true  \
              keyTab="/etc/security/keytabs/kafka_client.keytab" \
              principal="kafka-client-1@EXAMPLE.COM";

      For command-line utilities like kafka-console-consumer or kafka-console-producer, kinit can be used along with "useTicketCache=true" as in:

          sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
              useTicketCache=true;

      JAAS configuration for clients may alternatively be specified as a JVM parameter similar to brokers as described here. Clients use the login section named KafkaClient. This option allows only one user for all client connections from a JVM.
   2. Make sure the keytabs configured in the JAAS configuration are readable by the operating system user who is starting kafka client.
   3. Optionally pass the krb5 file locations as JVM parameters to each client JVM (see here for more details):

          -Djava.security.krb5.conf=/etc/kafka/krb5.conf

   4. Configure the following properties in producer.properties or consumer.properties:

          security.protocol=SASL_PLAINTEXT (or SASL_SSL)
          sasl.mechanism=GSSAPI
          sasl.kerberos.service.name=kafka