Kafka Connector Integration with Mule 4

Introduction:

Kafka is a scalable pub/sub system, primarily used to collect & analyze large volumes of data. It enables us to pass messages from one end-point to another. Kafka messages are persisted on the disk and replicated among the cluster to prevent data loss. Kafka is constructed on top of the ZooKeeper synchronization service.

Benefits:

1. Reliability
2. Scalability
3. Durability
4. Performance

Use Case:

Send and Receive messages from Kafka messaging system by integrating with the Mulesoft platform.

Please follow the step-by-step installation procedure of Kafka Installation.

Mule Flow (Publish Message):

Step – 1: Configure the HTTP Listener by giving it a hostname, port number, and path. Along with this, specify allowed methods (Optional) in the Advanced tab of the HTTP connector.

Step – 2: Drag & Drop the Logger component to log the message that was consumed by Kafka.

Step – 3: Kafka comes as a custom connector needed to get it from Exchange. So, here Drag & Drop the Kafka Publish Message from Exchange.

From above:

Topic : Message to send.

Key : Belongs to message that is going to be sent.

Mule Flow (Consume Message):

Step-4: Drag & Drop the Kafka Message Consumer connector as the source from the palette.

From above:

Topic: Message to receive.

Group id: Unique string that identifies the consumer group that this consumer belongs to.

Step – 5: Drag & Drop the Logger component to log the consumed message.

Final Config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:kafka="http://www.mulesoft.org/schema/mule/kafka" xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/kafka http://www.mulesoft.org/schema/mule/kafka/current/mule-kafka.xsd">
	<kafka:kafka-producer-config name="Apache_Kafka_Producer_configuration" doc:name="Apache Kafka Producer configuration" doc:id="0289a213-cdf8-49ab-a13b-59af6cb25f97" >
		<kafka:basic-kafka-producer-connection bootstrapServers="localhost:9092" />
	</kafka:kafka-producer-config>
	<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="7700a54a-38e2-4385-a195-6a539d201b37" >
		<http:listener-connection host="0.0.0.0" port="8082" />
	</http:listener-config>
	<kafka:kafka-consumer-config name="Apache_Kafka_Consumer_configuration" doc:name="Apache Kafka Consumer configuration" doc:id="6e24bf87-b86e-4319-8ec2-624f68b245b5" >
		<kafka:basic-kafka-consumer-connection groupId="1" bootstrapServers="localhost:9092" />
	</kafka:kafka-consumer-config>
	<flow name="Kafka-Publish" doc:id="47de8881-580b-41d2-af4f-4f505480d9ec" >
		<http:listener doc:name="Listener" doc:id="173de33a-4b7d-4fd1-951c-a153dde6b52a" config-ref="HTTP_Listener_config" path="/kafka" allowedMethods="POST"/>
		<logger level="INFO" doc:name="Logger" doc:id="c85d216f-b005-4663-9309-28864a818da6" message="#[message.payload]"/>
		<kafka:producer doc:name="Publish Message" doc:id="e5457b47-7dcd-42bc-b83d-d6922dec0641" config-ref="Apache_Kafka_Producer_configuration" topic="muleesb" key="#[now()]">
			<kafka:message ><![CDATA[#[message.payload]]]></kafka:message>
		</kafka:producer>
	</flow>
	<flow name="Kafka-Consumer" doc:id="c0e6fd8a-0bcd-4ccb-91b6-0e97804d5607" >
		<kafka:consumer doc:name="Message Consumer" doc:id="660a5e95-a897-4cdb-a971-10df13d9d4be" topic="muleesb" config-ref="Apache_Kafka_Consumer_configuration"/>
		<logger level="INFO" doc:name="Logger" doc:id="86114247-dce0-4626-8278-81450bc928b6" message="#[payload]"/>
	</flow>
</mule>

Sample Input & Output:

H@ppy Le@rninG…@@

Conclusion

We have successfully integrated a Kafka connector to send an automatic email notification to the customers in one of the Order Management System POC.

Are you still having issues with integrating a Kafka connector with Mule 4? Need more information or assistance with MuleSoft integration? Our MuleSoft experts are here to help you, reach out today.