{"id":3714,"date":"2021-01-23T08:58:00","date_gmt":"2021-01-23T08:58:00","guid":{"rendered":"https:\/\/linuxways.net\/?p=3714"},"modified":"2021-01-25T18:36:16","modified_gmt":"2021-01-25T18:36:16","slug":"how-to-install-and-configure-apache-kafka-on-ubuntu-20-04","status":"publish","type":"post","link":"https:\/\/linuxways.net\/de\/ubuntu\/how-to-install-and-configure-apache-kafka-on-ubuntu-20-04\/","title":{"rendered":"How to Install and Configure Apache Kafka on Ubuntu 20.04"},"content":{"rendered":"

Introduction<\/h2>\n

Apache Kafka is a distributed events streaming platform which has the ability to handle the high-performance data pipelines. It was originally developed by Linkedin then to be public as an open-source platform and used by many IT companies in the world.<\/p>\n

This article will show you the way to install and configure Kafka on Ubuntu 20.04.<\/p>\n

Installing Apache Kafka<\/h2>\n

Prerequisite<\/span><\/h3>\n

Apache Kafka requires Java to be installed on your Ubuntu 20.04 machine. Firstly, let\u2019s update your OS by the following command:<\/p>\n

$ sudo apt update<\/pre>\n
After the OS is updated, go ahead to install Java:<\/p>\n
$ sudo apt install openjdk-11-jre-headless<\/pre>\n
Verifying that Java was successfully installed by running:<\/p>\n
$ java --version<\/pre>\n
The output:<\/p>\n
<\/p>\n
Downloading Kafka<\/span><\/h3>\n
Next, you have to download the Kafka source to your Ubuntu 20.04. It\u2019s highly recommended to download it from the official website of Apache Kafka: https:\/\/kafka.apache.org\/downloads<\/a><\/p>\n
At the time of writing this article, the latest version is 2.7.0. You can download it by the following command:<\/p>\n
$ cd $HOME<\/pre>\n
$ wget https:\/\/downloads.apache.org\/kafka\/2.7.0\/kafka-2.7.0-src.tgz<\/a><\/pre>\n
Let\u2019s create a new folder named as kafka-server in \/usr\/local directory:<\/p>\n
$ sudo mkdir \/usr\/local\/kafka-server<\/pre>\n
Then extract the downloaded source of Kafka to \/usr\/local\/kafka-server directory:<\/p>\n
$ sudo tar xf $HOME\/kafka-2.7.0-src.tgz -C \/usr\/local\/kafka-server<\/pre>\n
<\/p>\n
You\u2019ve already had the extracted Apache Kafka binary files. Listing these files by running:<\/p>\n
$ ls \/usr\/local\/kafka-server\/kafka-2.7.0-src\/bin\/<\/pre>\n
Output:<\/p>\n
<\/p>\n
Now, it\u2019s time to make Kafka and Zookeeper run as daemons in Ubuntu 20.04. To do this, you have to create Systemd unit files for both Kafka and Zookeeper.<\/p>\n
Creating Systemd Unit files for Kafka and Zookeeper<\/span><\/h3>\n
Using your favorite editor and create two files as follows:<\/p>\n
\/etc\/systemd\/system\/zookeeper.service<\/p>\n
[Unit]\n\nDescription=Apache Zookeeper Server\n\nRequires=network.target remote-fs.target\n\nAfter=network.target remote-fs.target\n\n[Service]\n\nType=simple\n\nExecStart=\/usr\/local\/kafka-server\/kafka-2.7.0-src\/bin\/zookeeper-server-start.sh \/usr\/local\/kafka-server\/kafka-2.7.0-src\/config\/zookeeper.properties\n\nExecStop=\/usr\/local\/kafka-server\/kafka-2.7.0-src\/bin\/zookeeper-server-stop.sh\n\nRestart=on-abnormal\n\n[Install]\n\nWantedBy=multi-user.target\n<\/pre>\n
\/etc\/systemd\/system\/kafka.service<\/p>\n
[Unit]\n\nDescription=Apache Kafka Server\n\nDocumentation=http:\/\/kafka.apache.org\/documentation.html\n\nRequires=zookeeper.service\n\nAfter=zookeeper.service\n\n[Service]\n\nType=simple\n\nEnvironment=\"JAVA_HOME=\/usr\/lib\/jvm\/java-11-openjdk-amd64\"\n\nExecStart=\/usr\/local\/kafka-server\/kafka-2.7.0-src\/bin\/kafka-server-start.sh \/usr\/local\/kafka-server\/kafka-2.7.0-src\/config\/server.properties\n\nExecStop=\/usr\/local\/kafka-server\/kafka-2.7.0-src\/bin\/kafka-server-stop.sh\n\nRestart=on-abnormal\n\n[Install]\n\nWantedBy=multi-user.target<\/pre>\n
In order to apply the changes, the systemd daemons need to be reloaded and you have to enable the services as well.<\/p>\n
$ sudo systemctl daemon-reload\n\n$ sudo systemctl enable --now zookeeper.service\n\n$ sudo systemctl enable --now kafka.service\n\n$ sudo systemctl status kafka zookeeper<\/pre>\n
Output:<\/p>\n
<\/p>\n
<\/p>\n
Installing Cluster Manager for Apache Kafka (CMAK)<\/span><\/h3>\n
The next step is installing the CMAK which stands for Cluster Manager for Apache Kafka. CMAK is an open-source tool for managing and monitoring Kafka services. It was originally developed by Yahoo. In order to install CMAK, run the following commands:<\/p>\n
$ cd $HOME\n\n$ git clone https:\/\/github.com\/yahoo\/CMAK.git<\/a><\/pre>\n
<\/p>\n
Configuring CMAK<\/span><\/h3>\n
Then, using your favorite editor to modify the CMAK configuration.<\/p>\n
$ vim ~\/CMAK\/conf\/application.conf<\/pre>\n
In this tutorial, we will configure the Zookeeper is localhost<\/strong>, let\u2019s change the value of cmak.zkhosts<\/strong> as localhost:2181<\/strong><\/p>\n
You can find the cmak.zkhosts<\/strong> at line 28.<\/p>\n
<\/p>\n
Now, you have to create a zip file for the purpose of deploying the application:<\/p>\n
$ cd ~\/CMAK\n\n$ .\/sbt clean dist<\/pre>\n
It will take about a minute to complete. The output will be:<\/p>\n
<\/p>\n
Starting the CMAK service<\/span><\/h3>\n
Change into ~\/CMAK\/target\/universal directory and extract the zip file:<\/p>\n
$ cd ~\/CMAK\/target\/universal\n\n$ unzip cmak-3.0.0.5.zip<\/pre>\n
After unzip the cmak-3.0.0.5.zip file, change to the directory, and execute the cmak binary:<\/p>\n
$ cd cmak-3.0.0.5\n\n$ bin\/cmak<\/pre>\n
By default, the cmak service will run on port 9000.<\/p>\n
Use the web browser and goto http:\/\/<ip-server>:9000<\/p>\n
<\/p>\n
Currently, there is no available cluster. We have to add a new one by clicking on Add Cluster <\/strong>on Cluster <\/strong>drop-down list.<\/p>\n
<\/strong><\/p>\n
Then, fill up the below form with the requested information: Cluster name, Cluster Zookeeper Hosts, Kafka Version and so on. For example:<\/p>\n
<\/p>\n
Leave other options with their default values then click Save.<\/strong><\/p>\n
<\/p>\n
Done. The cluster was successfully created.<\/p>\n
<\/strong><\/p>\n
Now, it\u2019s time to create a sample topic. Assume that we\u2019re going to create a topic named \u201cLinuxWaysTopic\u201d. Bearing in mind that the CMAK is still running and launch a new terminal then run the following command:<\/p>\n
$ cd \/usr\/local\/kafka-server\/kafka-2.7.0-src<\/pre>\n
$ bin\/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic LinuxWaysTopic<\/pre>\n
Output:<\/p>\n
<\/p>\n
Go to cluster view then click Topic > List<\/strong><\/p>\n
<\/strong><\/p>\n
Conclusion<\/h2>\n
You\u2019ve successfully installed and configured Apache Kafka on your Ubuntu 20.04 LTS machine.<\/p>\n
If you have any concerns, feel free to leave your comment and let me know. Thank you!<\/p>","protected":false},"excerpt":{"rendered":"
Introduction Apache Kafka is a distributed events streaming platform which has the ability to handle the high-performance data pipelines. It was originally developed by Linkedin then to be…<\/p>","protected":false},"author":23,"featured_media":3731,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[152,100],"class_list":["post-3714","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ubuntu","tag-apache-kafka","tag-ubuntu-20-04"],"_links":{"self":[{"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/posts\/3714","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/comments?post=3714"}],"version-history":[{"count":0,"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/posts\/3714\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/media\/3731"}],"wp:attachment":[{"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/media?parent=3714"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/categories?post=3714"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/linuxways.net\/de\/wp-json\/wp\/v2\/tags?post=3714"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}