Operations Exercise Guide PDF

Preview

Summary

Download Operations Exercise Guide PDF

Description

Confluent Operations Training for Apache Kafka Exercise Manual 5.0.0-v1.1.0

Table of Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Ê

Preparing the Lab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Ê

Investigating the Distributed Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Ê

Examine Checkpoint Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Ê

Examining Topic Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Ê

Kafka Administrative Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Ê

Modifying Partitions and Viewing Offsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Ê

Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Ê

Defining Over Consumption Trigger and Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Ê

Simulating Over Consumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Ê

Hands-On Exercise: Securing the Kafka Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Ê

Generating Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Ê

Running Kafka Connect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Ê

Appendix: Reassigning Partitions in a Topic - Alternate Method . . . . . . . . . . . . . . . . . . . . 76 Ê

1

Introduction This document provides Hands-On Exercises for the course Confluent Operations Training for Apache Kafka. You will use a setup that includes a virtual machine (VM) configured as a Docker host to demonstrate the distributed nature of Apache Kafka. The main Kafka cluster includes the following components, each running in a container: Table 1. Components of the Confluent Platform Alias

Description

zk-1

ZooKeeper 1

zk-2

ZooKeeper 2

zk-3

ZooKeeper 3

kafka-1

Kafka Broker 1

kafka-2

Kafka Broker 2

kafka-3

Kafka Broker 3

schema-registry

Schema Registry,

connect

Kafka Connect,

ksql-server

KSQL Server,

ksql-cli

KSQL CLI,

control-center

Confluent Control Center

base

a container used to run tools against the cluster

You will use Confluent Control Center to monitor the main Kafka cluster. To achieve this, we are also running the Control Center service which is backed by the same Kafka cluster.



In production, Control Center should be deployed with its own dedicated Kafka cluster, separate from the cluster with production traffic. Using a dedicated metrics cluster is more resilient because it continues to provide system health monitoring even if the production traffic cluster experiences issues.

Accessing your Lab Environment You will receive an email with a link to your lab environment from Confluent. The labs are running on a VM which is based on Ubuntu 18.04 Desktop edition. On it we have installed Docker Community Edition (CE), Google Chrome and Visual Studio Code. 1. Click the link in the email you received from Confluent:

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

1

2

2. Login (or signup) to Content Raven:

3. Access you Learning Path 4. Launch the accompanying VM 5. Login to the VM using the following credentials: ◦ Username: training ◦ Password: training

Alternatively you can also download the VM to your laptop and run it in VirtualBox. Make sure you have the newest version of VirtualBox installed. Download the VM from this link: https://s3.amazonaws.com/confluent-training-imagesus-east-1/training-ubuntu-18.04-with-images.ova

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

2

3

If you have installed Docker for Desktop on your Mac or Windows 10 Pro machine then you can run the labs there. But please note that your trainer might not be able to troubleshoot any potential problems if you are running the labs locally.

Command Line Examples Most Exercises contain commands that must be run from the command line. These commands will look like this: $ pwd /home/training

Commands you should type are shown in bold; non-bold text is an example of the output produced as a result of the command.

Continued Learning After Class Once the course ends, the VM will terminate and you will no longer have access to it, but you can continue to use this Exercise Manual for reference. We encourage you to bring up your own test environment, using Docker for Mac or Windows, or Linux and Docker. Options include: • Download the Confluent Platform: https://www.confluent.io/download/ • Run the Confluent Platform Demo including Control Center and KSQL: https://github.com/confluentinc/cp-demo • More information on clustered deployments for testing: https://docs.confluent.io/current/cpdocker-images/docs/tutorials/clustered-deployment.html

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

3

4

Preparing the Lab 1. Download all Docker images needed in this course:



This step is only needed if you’re not running the VM in the cloud but rather executing the labs in Docker for Desktop on your computer. This may take a couple of minutes!

$ CE_VERSION=5.0.0 for IMAGE in cp-zookeeper cp-enterprise-kafka cp-schema-registry \ cp-kafka-connect cp-kafka-rest cp-ksql-server cp-ksql-cli \ cp-enterprise-control-center do docker image pull confluentinc/${IMAGE}:${CE_VERSION} done 5.0.0: Pulling from confluentinc/cp-zookeeper Digest: sha256:f5cd4ac4782da1de36f263ecae0ec2d0894c96abe97fb76fd757cc1df9373a77 ...

2. Create a folder confluent-ops in your home directory for the labs and navigate to it: $ mkdir -p ~/confluent-ops && cd ~/confluent-ops



If you chose to select another folder for the labs then note that many of our samples assume that the lab folder is ~/confluent-ops. You will have to adjust all those command to fit your specific environment.

3. Download the file docker-compose.yml into this folder: $ curl -L https://bit.ly/2O7Uyx1 -o docker-compose.yml

This file will be used to run and manage the Confluent Platform with all its components. 4. Start the whole cluster with the following simple command:

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

4

5

$ docker-compose up -d Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating Creating

network "confluent-ops_confluent" with the default driver volume "confluent-ops_data-zk-1" with default driver volume "confluent-ops_data-zk-2" with default driver volume "confluent-ops_data-zk-3" with default driver volume "confluent-ops_data-kafka-1" with default driver volume "confluent-ops_data-kafka-2" with default driver volume "confluent-ops_data-kafka-3" with default driver kafka-2 ... done schema-registry ... done ksql-server ... done control-center ... done ksql-cli ... done base ... done kafka-3 ... done zk-1 ... done zk-3 ... done connect ... done kafka-1 ... done zk-2 ... done

5. Monitor the cluster with: $ docker-compose ps Name Command State Ports -------------------------------------------------------------------------------------base /bin/sh Up 8083/tcp, 9092/tcp connect /etc/confluent/docker/run Up 0.0.0.0:8083->8083/tcp, 9092/tcp control-center /etc/confluent/docker/run Up 0.0.0.0:9021->9021/tcp kafka-1 /etc/confluent/docker/run Up 9092/tcp kafka-2 /etc/confluent/docker/run Up 9092/tcp kafka-3 /etc/confluent/docker/run Up 9092/tcp ksql-cli /bin/sh Up ksql-server /etc/confluent/docker/run Up 0.0.0.0:8088->8088/tcp schema-registry /etc/confluent/docker/run Up 8081/tcp zk-1 /etc/confluent/docker/run Up 2181/tcp, 2888/tcp, 3888/tcp zk-2 /etc/confluent/docker/run Up 2181/tcp, 2888/tcp, 3888/tcp zk-3 /etc/confluent/docker/run Up 2181/tcp, 2888/tcp, 3888/tcp

All services should have State equal to Up. 6. OPTIONAL: You can also observe the stats of Docker on your VM:

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

5

6

$ docker stats CONTAINER ID 877986e719fd 42170ab1d9ec eab22caa151e e64d1f839bbf 00ce551165f2 1fabbed67d2b d597e1de0f2c 0624d0cc170e 2de9ad477a6b 30e2509018d0 582b31d8ac68 288d45be137d

NAME base zk-2 connect kafka-1 ksql-cli zk-3 zk-1 schema-registry control-center kafka-3 kafka-2 ksql-server

CPU % 0.00% 0.25% 1.20% 16.30% 0.00% 0.24% 0.22% 0.50% 70.87% 4.30% 3.95% 0.57%

MEM USAGE / LIMIT 400KiB / 7.786GiB 66.02MiB / 7.786GiB 1.912GiB / 7.786GiB 396.9MiB / 7.786GiB 364KiB / 7.786GiB 65.97MiB / 7.786GiB 59.47MiB / 7.786GiB 236.8MiB / 7.786GiB 377.4MiB / 7.786GiB 374.6MiB / 7.786GiB 381.9MiB / 7.786GiB 223.4MiB / 7.786GiB

MEM % 0.00% 0.83% 24.56% 4.98% 0.00% 0.83% 0.75% 2.97% 4.73% 4.70% 4.79% 2.80%

... ... ... ... ... ... ... ... ... ... ... ... ...

Testing the Installation 1. Use docker-compose to exec into the base container from which you can then use various command-line tools to work with the cluster: $ docker-compose exec base /bin/bash root@base:/#



The prompt root@base:/# tells you that you’re logged in as root in the container with hostname base.

2. From within the base container, use the zookeeper-shell command to verify that all Brokers have registered with ZooKeeper. You should see the three Brokers listed as [101, 102, 103] in the last line of the output. root@base:/# zookeeper-shell zk-1:2181 ls /brokers/ids Connecting to zk-1:2181 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [101, 102, 103]

3. To exit from the base container typ CTRL-d.

OPTIONAL: Analyzing the Docker Compose File 1. Open the file docker-compose.yml in your editor and: a. locate the various services that are listed in the table earlier in this section b. note that the alias (e.g. zk-1 or kafka-2) are used to resolve a particular service

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

6

7

c. note how each ZooKeeper instance (zk-1, zk-2, zk-3) i. gets a unique ID assigned via environment variable ZOOKEEPER_SERVER_ID ii. gets the information about all members of the ensemble: ZOOKEEPER_SERVERS=zk-1:2888:3888;zk-2:2888:3888;zk-3:2888:3888

d. note how each broker (kafka-1, kafka-2, kafka-3) i. gets a unique ID assigned via environment variable KAFKA_BROKER_ID ii. defines where to find the ZooKeeper instances KAFKA_ZOOKEEPER_CONNECT: zk-1:2181,zk-2:2181,zk-3:2181

iii. set the replication factor for the offsets topic to 3: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3

iv. configure the broker to send metrics to Confluent Control Center: KAFKA_METRIC_REPORTERS: "io.confluent.metrics.reporter.ConfluentMetricsReporter" CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: "kafka-1:9092,kafka-2:9092,kafka3:9092"

e. note how various services use the environment variable …_BOOTSTRAP_SERVERS to define the list of Kafka brokers that serve as bootstrap servers: ..._BOOTSTRAP_SERVERS: kafka-1:9092,kafka-2:9092,kafka-3:9092

f. note how e.g. the connect service and the ksql-server service define producer and consumer interceptors that produce data which can be monitored in Confluent Control Center: io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

Using Confluent Control Center 1. In the VM, open a new browser tab in Google Chrome. 2. Navigate to the Control Center GUI at http://localhost:9021

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

7

8

3. In the Control Center GUI, click on the top right button that shows the current date, and change Last 4 hours to Last 30 minutes.

4. In the Control Center System health landing page, observe the brokers in your cluster. Scroll down to see the table with three brokers: 1, 2, 3.

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

8

9

Running the Confluent Platform in Kubernetes For more information on how to run Confluent OSS or Confluent Enterprise in Kubernetes please refer to the following links: 1. Confluent Platform Helm Charts: https://docs.confluent.io/current/quickstart/cp-helm-charts/docs/index.html 2. Helm Charts on GitHub: https://github.com/confluentinc/cp-helm-charts

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

9

10

Investigating the Distributed Log In this exercise, you will investigate the distributed log. You will then simulate a Broker failure, and see how to recover the Broker.

Prerequisites 1. Please make sure you have prepared your environment by following → Preparing the Labs 2. Make sure your Kafka cluster is started, otherwise execute this command: $ cd ~/confluent-ops $ docker-compose up -d

Observing the Distributed Log 1. In a terminal from within your labs folder ~/confluent-ops exec into the base container from which we will run our following commands: $ docker-compose exec base /bin/bash root@base:/#

2. From base create a new Topic called replicated-topic with six Partitions and two replicas: root@base:/# kafka-topics \ --create \ --zookeeper zk-1:2181 \ --topic replicated-topic \ --partitions 6 \ --replication-factor 2 Created topic "replicated-topic".

3. View the Topic information to see that it has been created correctly. The exact output may vary depending on which brokers the topic partitions and replicas were assigned. a. Connect to Confluent Control Center. If necessary, open a browser tab to the URL http://localhost:9021 b. In Confluent Control Center, under the MANAGEMENT header, click on Topics:

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

10

11

c. Scroll through the topic list until you find replicated-topic. Click on the 3 dots next to the topic name and click on Status:

d. Notice the Topic’s partition list and the corresponding brokers where each partition replica resides. Blue replicas indicate they are in sync. A later exercise where a broker goes offline will result in some orange replicas:

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

11

12

e. You can also view the same Topic information via Kafka command line tools. In your terminal, from base, describe the Topic: root@base:/# kafka-topics \ --describe \ --zookeeper zk-1:2181 \ --topic replicated-topic

The output should be similar to this:

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

12

13

Topic:replicated-topic PartitionCount:6 Topic: replicated-topic Partition: 0 102,101 Topic: replicated-topic Partition: 1 103,102 Topic: replicated-topic Partition: 2 101,103 Topic: replicated-topic Partition: 3 102,103 Topic: replicated-topic Partition: 4 103,101 Topic: replicated-topic Partition: 5 101,102

ReplicationFactor:2 Configs: Leader: 102 Replicas: 102,101

Isr:

Leader: 103 Replicas: 103,102

Isr:

Leader: 101 Replicas: 101,103

Isr:

Leader: 102 Replicas: 102,103

Isr:

Leader: 103 Replicas: 103,101

Isr:

Leader: 101 Replicas: 101,102

Isr:

4. From base produce some data to send to the Topic replicated-topic. Leave this process running until it finishes. The following command line sends 6000 messages, 100 bytes in size, at a rate of 1000 messages/sec: root@base:/# kafka-producer-perf-test \ --topic replicated-topic \ --num-records 6000 \ --record-size 100 \ --throughput 1000 \ --producer-props bootstrap.servers=kafka-1:9092,kafka-2:9092,kafka-3:9092

The output should look similar to this: 4999 records sent, 999.8 records/sec (0.10 MB/sec), 5.2 ms avg latency, 298.0 max latency. 6000 records sent, 998.336106 records/sec (0.10 MB/sec), 4.69 ms avg latency, 298.00 ms max latency, 2 ms 50th, 25 ms 95th, 35 ms 99th, 46 ms 99.9th.

5. From base start the console Producer for the same Topic replicated-topic: root@base:/# kafka-console-producer \ --broker-list kafka-1:9092,kafka-2:9092,kafka-3:9092 \ --topic replicated-topic

6. At the > prompt, type “I heart logs” and press Enter. Add five more messages and then press Ctrl-d to exit the console Producer: > I heart logs > Hello world > All your base > Kafka rules > Don't worry > Be happy

Copyright © 2015-2018 Confluent, Inc. All rights reserved. Not to be reproduced in any form without prior written consent.

13

14

Examine Checkpoint Files For each Broker (kafka-1, kafka-2 and kafka3) examine the contents of the checkpoint files recoverypoint-offset-checkpoint and replication-offset-checkpoint at /var/lib/kafka/data. Let’s start with the first broker called kafka-1: 1. The next steps will be run on the broker. From your host system, open a new terminal and connect to the broker, here kafka-1: $ docker-compose exec kafka-1 /bin/bash root@kafka-1:/#

2. The recovery-point-offset-checkpoint file records the point up to which data has been flushed to disks. This is important as, on hard failure, the Broker needs to scan unflushed data, verify the CRC, and truncate the corrupted log. In another terminal, from the host within the folder ~/confluent-ops execute: root@kafka-1:/# cat /var/lib/kafka/data/recovery-point-offset-checkpoint | \ grep replicated-topic replicated-topic replicated-topic replicated-topic replicated-topic

5 1 4 0

0 0 0 0

In the output above, the first number is the Partition number, and the second number is the offset of the last flushing point. In the output above, the offset of the last flushing point is expected to be 0 because the open file segment has not been flushed yet 3. The replication-offset-checkpoint file is the offset of the last committed offset. The follower uses this during software upgrades if leader epoch is ...