ranger kafka

Authorizing Kafka access over non-authenticated channel via Ranger

This section answers some questions one is likely to encounter when trying to authorize access to Kafka over non-authenticated channel. This Kafka feature is available in HDP releases 2.3.4 (Dal-M20) or later.

Can I authorizer access to Kafka over a non-secure channel via Ranger?

Yes. you can control access by ip-address.

Can I authorize access to Kafka over non-secure channel by user/user-groups?

No, one can’t use user/group based access to authorize Kafka access over a non-secure channel. This is because it isn't possible to assert client’s identity over the non-secure channel.

What is a recommended way to set-up policies when trying to control access to Kafka over a non-secure channel?

Ensure that all Brokers nodes haveKafka Adminaccess. This is a mandatory step. If you don’t perform this step then your cluster won’t work properly.

• Identify the nodes where brokers are running.

• Create a policy where resource is*(i.e. all topics) and grantKafka Adminaccess type to thepublicuser group. Specify ip-address of all the brokers as theip-range policy conditionon the policy item.

Ensure publishers have appropriate access.

• Identify ip address of all nodes where publishers would run along with their respective topics.

• Create policy where resources are the respective topic names and grantPublishaccess type topublicuser group. Specify ip-address of machines where those publishers would run as theip-range policy conditionon the policy item.

• Specify topic name(s) as policy resource. Note that you can specify multiple topics or even regular expressions in topic names.

Ensure consumers have appropriate access. Same process as publishers except change access type toConsumeinstead orProduce.

Why do we have to specifypublicuser group on all policies items created for authorizing Kafka access over non-secure channel?

• Kafka can’t assert the identity of client user over a non-secure channel. Thus, Kafka treats all users for such access as an anonymous user (a special user literally namedANONYMOUS).

• Ranger'spublicuser group is a means to model all users which, of course, includes this anonymous user (ANONYMOUS).

What are the specific things to watch out for when setting up authorization for accessing Kafka over non-secure channel?

• Make sure that all broker-ips haveKafka adminaccess to all topics, i.e.*.

• Make sure no publishers or consumers are running on broker nodesthat need access control. Since broker ips have open access it isn’t possible to control access on those nodes.

I have the policies as specified above, however, I am still not able to consume over an non-authenticated channel usingbin/kafka-console-consumer.shscript that is a part of the Kafka distribution! The consumer hangs and gives the error message “No brokers found in ZK.” What gives?

• Ensure that/etc/kafka/conf/kafka_client_jaas.confdoes not have specification forserviceName="zookeeper". This is typically theClientsection.

• Ensure that you are not specifying--security-protocol PLAINTEXTSASLargument to the consumer. Either specify--security-protocol PLAINTEXTor leave--security-protocolunspecified since its default value isPLAINTEXT.

I can’t edit the/etc/kafka/conf/kafka_client_jaas.conffile! What should I do to consume kafka messages over an non-authenticated channel?

• In that case just do akinitwith a valid password/ticket.

• That token will get used to authenticate you tozookeeper. After that you should be able to consume messages from kafka over non-authenticated channel. Connection to Kafka brokers correctly happens over non-authenticated channel and should get authorized as userANONYMOUS.

Why do I need to edit the/etc/kafka/conf/kafka_client_jaas.conffile?

Presence ofClientblock in/etc/kafka/conf/kafka_client_jaas.conffor servicezookeepercauses the console consumer connect to zookeeper in secure mode. To do so it needs a ticket -- which won’t exist in simple auth mode, so it fails.

Authorizing topic creation

This section describes the issues one might encounter while trying to authorize topic creation in Kafka using Ranger.

Can I authorizer topic creation via Ranger?

Yes, butonlyifthe topic is beingauto-createdby consumers or producers.

What is the recommended policy setup to authorize topic auto-creation for producers or consumers?

• Create a policy where resource is all topics, i.e.*.

• For producers, create a policy item under this policy which grants bothProduceandConfigurepermissions to the relevant user/user-groups.

• For consumers, create a policy item under this policy which grants bothConsumeandConfigurepermissions to the relevant user/user-groups.

Can I authorize topic auto-creation for producers or consumers that connect over non-authenticated channel?

• Yes, create a policy similar to that for secure producer.
• Either add user grouppublicto the policy item or specify andip-addressbase custom condition.
• Refer to FAQ about authorizing Kafka access over non-authenticated channel for additional details and rationale.

Why do I have to grant create access to all topics (via*) to allow for auto-creation to work for producers and/or consumers?

Topic creation is currently a cluster level privilege. Thus it requires access privileges over all topics in a cluster, i.e.*.

I want to allow topic auto creation for any topic that starts withfinance, e.g.finance_1,finance_2, etc. to users that are part ofFinanceuser group. But I don’t want them to be able to auto create topics that start with other strings, say,marketing_123. Can I model this sort of an authorization in Ranger Kafka plugin?

• No. Because in Kafka currently topic creation is a cluster level permissions, i.e. all topics.
• There is a pending proposal aboutHierarchical topicsin Kafka which, if and when it’s implemented, could help with that use case.

I am using the Kafka supplied console consumer to test topic auto creation by a consumer, but it is not working. Shouldn’t the new topic get auto-created the moment I startup the consumer? I have verified the recommended policy setup as indicated above! What gives?

Make sure that you specify the following two argument to the console consumer.

• --new-consumer
• --boot-strap <broker-name(s)>: Any single broker host/port would do.

Most common way of creating topic involves using thebin/kafka-tpics.shscript that is a part of the Kafka distribution. Can I authorize topic creation via that mechanism?

No.

Why can’t I authorize topic creation done via thebin/kafka-tpics.shscript!?

• This script talks directly to zookeeper. Hence, the policies of Kafka plugin don’t come into the picture.
• Script adds entries into zookeeper nodes and watchers inside the brokers monitor it and create topics.

So what are my options to authorize topic creation via thebin/kafka-tpics.shscript?

• Since this directly interacts with zookeeper this is best controlled via zookeeper acls.

Is there a Ranger plugin for Zookeeper?

Not yet.

Where can I learn more about Kafka’s support for publish/consume over non-authenticated channel?

Please refer toKAFKA-1809which implemented themultiple listeners Design.