Implementing Apache Kafka as Live Data Streaming with NodeJS and WebSocket

Node JS and Kafka logo

The story tell that we are going to create some live data streaming that very huge. This was small tutorial to handle a huge data coming from our client and provide smoke performing after smoke processing.

We would like to create a simple huge live data streaming that was used by Spotify here! using Kafka , NodeJS and WebSocket.

What is Kafka?

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

It’s kinda hard to understand if you dont know what is Kafka , how Kafka works plus you need a deep understanding how NodeJS works. Being new at Kafka and NodeJS would crumble you.

Here’s the diagram i will show you , how this will works

Kafka, NodeJS , WebSocket processing flow

Getting started with Kafka

  1. Install Kafka on your local machine , can be found here! https://kafka.apache.org/quickstart

Use steps that mentioned there to create Kafka server, create a topic then use Kafka console producer to send message to the topic.

Creating consumer in NodeJS

Im using this https://github.com/Blizzard/node-rdkafka . The purpose is very slick, read messages from the Kafka topic. Modify your config or server.js

var kafka = require(‘kafka-node’);
var Consumer = kafka.Consumer,
client = new kafka.KafkaClient(“localhost:5000”),
consumer = new Consumer(
client, [ { topic: ‘spotifyTest’, partition: 0 } ], { autoCommit: false });consumer.on(‘message’, function (message) {
console.log(message);
});

Create WebSocket on server side

On the NodeJS server side

Our main purpose! merging these two

Its show time to return data coming from Kafka producer. The result would be like this, let’s call it Consumer here

Run with nodemon which you can find here https://www.npmjs.com/package/nodemon , to keep up the Consumer running

Now we are ready to use that live data on your interface, as i can say , you can implement it anywhere!, no boundaries. We can check the live streaming data by sending messages to Kafka console producer

No magic works here!, science doesnt involved too!

This is the most basic story about Kafka live data streaming, you can implement it anywhere! refactoring the system too. Hope this will make you understand the end to end flow

References :

Software Engineer who got D score in Introduction to Programming at college 👨‍🎓🧑‍💻