Kafka

Kafka is a publish-subscribe message queuing system that’s designed like a distributed commit log. Metricly can help monitor the performance and throughput of your Kafka server using our Kafka collector for the Linux agent. Kafka, Kafka Consumer Lag, and Zookeeper metrics are all collected using this collector.

Prerequisites

The [Linux Agent][1] is required before proceeding with the setup of HTTPD. If you need to disable the Linux integration or view the unique API key assigned to your account, navigate to the Integrations page under the user account drop-down menu and click the integration designated as Infrastructure under the Integration column.

Configuration

  1. Download the Jolokia JVM JAR file.
  2. Move the downloaded file to the /opt/netuitive-agent/ directory.
  3. Run the following at the command line to set an environment variable: export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/netuitive-agent/jolokia-jvm-1.3.4-agent.jar"
  4. Restart Kafka, and confirm Jolokia is running by accessing http://localhost:8778/jolokia/
  5. Navigate to the collectors folder, /opt/netuitive-agent/conf/collectors.
  6. Open the KafkaJolokiaCollector.conf file.
  7. Change the enabled setting to True.
  8. Update version to be the version of Kafka running on your machine and uncomment the line.
    • Optionally, update host and port to the correct settings.
  9. Save the file, and restart the Linux Agent.

This integration’s package (computed metrics, dashboards, and policies that will give you important events and alerts) will be automatically enabled and provisioned to your account as soon as Metricly receives data from the integration. The PACKAGES button on the integration setup page will become active once data is received, so you’ll be able to disable and re-enable the package at will.

Collector Options

The path and jolokia_path settings should not be changed without consulting Metricly support first.

Option Default Description
enabled FALSE Enable collecting Kafka metrics.
path kafka The metric prefix, e.g., how you want the metrics to show up in Metricly.
jolokia_path jolokia Part of the URL path that points to where your application serves metrics. Typically jmx or jolokia.
metrics_blacklist .*Percentile$ .FifteenMinuteRate.
host localhost Hostname/IP address by which the Kafka instance can be reached by the Linux agent.
port 8778 The port that the Jolokia JAR file is listening on.
version 0.8 Specifies the Kafka major release being used on the host. Currently, Metricly supports 0.8, 0.9, and 0.10.
zookeeper localhost:2181 The hostname / IP address as well as the port by which the Zookeeper instance can be reached by the Linux agent.
bin /opt/kafka/bin/kafka-run-class.sh The location of the kafka-run-class.sh file. Certain metrics must be retrieved via command line calls, thus the location of kafka-run-class.sh is a necessity. The exception to this being if you’re running Kafka in a Docker container; see the argssetting below for more information.
args exec {kafka-container-id} {kafka-run-class.sh} The argument passed to Kafka running in a docker container. If you are running Kafka in a docker container, you’ll need to set the bin setting to the location of the docker binary. Then, you’ll need to replace {kafka-container-id} with the ID of your Kafka container and {kafka-run-class.sh} with the location of the kafka-run-class.sh file within the container.
consumer_groups Group1, Group2, Group3 For Kafka 8 or earlier: This setting specifies a list of consumer groups that you want to receive consumer lag metrics. If you do not specify this, no consumer lag metrics will be collected. For Kafka 9 or later: This setting is ignored as consumer groups can be auto-discovered.
topics Topic1, Topic2, Topic3 For Kafka 8 or earlier: This setting specifies a list of topics associated with the consumer groups. If you do not specify this, all topics are considered. For Kafka 9 or later: this setting is ignored.
byte_unit Default numeric output(s).
measure_collector_time Measure the collector’s run time in milliseconds.
metrics_whitelist Regex list to match metrics to transmit. Mutually exclusive with metrics_blacklist option.