cookbook 'confluent-platform', '= 2.1.0'
confluent-platform
(18) Versions
2.1.0
-
Follow1
Install and configure Confluent platform (Kafka)
cookbook 'confluent-platform', '= 2.1.0', :supermarket
knife supermarket install confluent-platform
knife supermarket download confluent-platform
Confluent Platform
Description
Apache Kafka, an open source technology created and maintained by the founders
of Confluent, acts as a real-time, fault tolerant, highly scalable messaging
system. It is widely adopted for use cases ranging from collecting user
activity data, logs, application metrics, stock ticker data, and device
instrumentation. Its key strength is its ability to make high volume data
available as a real-time stream for consumption in systems with very different
requirements from batch systems like Hadoop, to real-time systems that require
low-latency access, to stream processing engines that transform the data
streams as they arrive.
This infrastructure lets you build around a single central nervous system
transmitting messages to all the different systems and applications within your
company. Learn more on http://confluent.io.
This cookbook focuses on deploying Confluent Platform elements on your clusters
via Chef on systemd managed distributions. At the moment, this includes
Kafka, Schema Registry and Kafka Rest.
Requirements
Cookbooks and gems
Declared in [metadata.rb](metadata.rb) and in [Gemfile](Gemfile).
Platforms
A systemd managed distribution:
- RHEL Family 7, tested on Centos
Note: it should work quite fine on Debian 8 (with some attributes tuning) but
the official docker image does not allow systemd to work easily, so it could
not be tested.
Usage
Easy Setup
Default recipe does nothing. Each service Kafka, Schema Registry or
Kafka Rest will be installed by calling respectively recipe
[install_kafka](recipes/install_kafka.rb),
[install_registry](recipes/install_registry.rb) and
[install_rest](recipes/install_rest.rb).
By default, this cookbook installs openjdk from the official repositories
(openjdk 8 on centos 7) in services recipe, just before
launching the service. You can deactivate this behavior by setting
node['confluent-platform']['java']
to ""
, or choose your package by setting
the package name in node['confluent-platform']['java'][node[:platform]]
.
Search
The recommended way to use this cookbook is through the creation of a different
role per cluster, that is a role for Kafka, Schema Registry and
Kafka Rest. This enables the search by role feature, allowing a simple
service discovery.
In fact, there are two ways to configure the search:
1. with a static configuration through a list of hostnames (attributes hosts
like in ['confluent-platform']['kafka']['hosts']
)
2. with a real search, performed on a role (attributes role
and size
like in ['confluent-platform']['kafka']['role']
). The role should be in
the run-list of all nodes of the cluster. The size is a safety and should be
the number of nodes in the cluster.
If hosts is configured, role
and size
are ignored.
See [roles](test/integration/roles) for some examples and
Cluster Search documentation for more information.
Zookeeper Cluster
To install properly a Kafka cluster, you need a Zookeeper cluster.
This is not in the scope of this cookbook but if you need one, you should
consider using Zookeeper Platform.
The configuration of Zookeeper hosts use search and is done similarly as for
Kafka, Schema Registry and Kafka Rest hosts, ie with a static
list of hostnames or by using a search on a role.
Test
This cookbook is fully tested through the installation of the full platform
in docker hosts. This uses kitchen, docker and some monkey-patching.
If you run kitchen list
, you will see 6 suites:
- zookeeper-centos-7
- kafka-01-centos-7
- kafka-02-centos-7
- kafka-03-centos-7
- registry-01-centos-7
- rest-01-centos-7
Each corresponds to a different node in the cluster. They are connected through
a bridge network named kitchen, which is created if necessary.
For more information, see [.kitchen.yml](.kitchen.yml) and [test](test)
directory.
Local cluster
Of course, the cluster you install by running kitchen converge
is fully
working so you can use it as a local cluster to test your development (like a
new Kafka client). Moreover, compared to a single node cluster usually
installed on workstations, you can detected partition/timing/fault-tolerance
issues you could not because of the simplicity of a single-node system.
You can access it by using internal DNS of the docker network named kitchen
or by declaring each node in your hosts file. You can get each IP by
running:
docker inspect --format \
'{{.NetworkSettings.Networks.kitchen.IPAddress}}' container_name
Then to produce some messages:
kafka-console-producer.sh \
--broker-list kafka-kitchen-01.kitchen:9092 \
--topic my_topic
And to read them:
kafka-console-consumer.sh \
--zookeeper zookeeper-kafka.kitchen/kafka-kitchen \
--topic my_topic \
--from-beginning
Or you can use Rest API with http://rest-kitchen-01.kitchen:8082 and full
Schema Registry support, located at
http://registry-kitchen-01.kitchen:8081.
Attributes
Configuration is done by overriding default attributes. All configuration keys
have a default defined in [attributes/default.rb](attributes/default.rb).
Please read it to have a comprehensive view of what and how you can configure
this cookbook behavior.
Recipes
default
Does nothing.
repository
Configure confluent repository.
install_service
Install and fully configure a given service by running repository and its
4 dedicated recipes: package, user, config and service, in that order.
service_package
Install given service from confluent repository.
service_user
Create given service system user and group.
service_config
Generate service configuration. May search for dependencies (like Zookeeper
or other nodes of the same cluster) with the help of cluster-search cookbook.
service_service
Install systemd unit for the given service, then enable and start it.
Note: install java package (OpenJDK 8 on centos 7) by default, can be
disabled by setting node['confluent-platform']['java']
to "". A platform
specific configuration for the package to install is also possible.
Resources/Providers
None.
Changelog
Available in [CHANGELOG](CHANGELOG).
Contributing
Please read carefully [CONTRIBUTING.md](CONTRIBUTING.md) before making a merge
request.
License and Author
- Author:: Samuel Bernard (samuel.bernard@s4m.io)
Copyright (c) 2015-2016 Sam4Mobile Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Dependent cookbooks
cluster-search >= 0.0.0 |
yum >= 0.0.0 |
Contingent cookbooks
There are no cookbooks that are contingent upon this one.
Changelog
2.1.0
Main:
- Default confluent version to install is set to 3.0
- Scala version to install is set to 2.11
- Mandatory option ssl.client.auth is added to registry config
- Make Systemd unit path configurable
Tests:
- Start Continuous Integration with gitlab-ci
- Add security opts for docker, add package retries
- Remove sleep in recipes, wait to strengthen tests
2.0.0
Main:
- Switch to confluent 2.0
- Rename recipes to respect rubocop rules (breaking change)
Tests:
- Switch to docker_cli, use prepared docker image
- Switch kitchen driver from docker to docker_cli
- Use sbernard/centos-systemd-kitchen image instead of bare centos
- Remove privileged mode :)
- Remove some now useless monkey patching
- Remove dnsdock, use docker DNS (docker >= 1.10)
- Use "kitchen" network, create it if needed
Misc:
- Fix all rubocop offenses
- Use specific name for resources to avoid cloning
- Add more details on configuration in README
1.2.0
Main:
- Clarify and fix JVM options for services
- Use to_hash instead of dup to work on node values
- Improve readibility of default system user names
Fixes:
- Fix and clean the creation of Kafka work directories
- Fix zookeeper.connect chroot path
Test:
- Rationalize docker provision to limit images
- Fix typo in roles/rest-kitchen.json name
- Wait 15s after registry start to strengthen tests
Packaging:
- Reorganize README:
- Move changelog from README to CHANGELOG
- Move contribution guide to CONTRIBUTING.md
- Reorder README, fix Gemfile missing
- Add Apache 2 license file
- Add missing chefignore
- Fix long lines in rest and registry templates
1.1.0
- Cleaning, use only dependencies from supermarket
1.0.1
- Set java-1.8.0-openjdk-headless as default java package
1.0.0
- Initial version with Centos 7 support
Collaborator Number Metric
2.1.0 failed this metric
Failure: Cookbook has 1 collaborators. A cookbook must have at least 2 collaborators to pass this metric.
Contributing File Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file
Foodcritic Metric
2.1.0 failed this metric
FC066: Ensure chef_version is set in metadata: confluent-platform/metadata.rb:1
FC069: Ensure standardized license defined in metadata: confluent-platform/metadata.rb:1
Run with Foodcritic Version 10.3.1 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
License Metric
2.1.0 passed this metric
No Binaries Metric
2.1.0 passed this metric
Testing File Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.1.0 failed this metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must contain a CONTRIBUTING.md file
Foodcritic Metric
2.1.0 failed this metric
FC066: Ensure chef_version is set in metadata: confluent-platform/metadata.rb:1
FC069: Ensure standardized license defined in metadata: confluent-platform/metadata.rb:1
Run with Foodcritic Version 10.3.1 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
License Metric
2.1.0 passed this metric
No Binaries Metric
2.1.0 passed this metric
Testing File Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.1.0 failed this metric
FC069: Ensure standardized license defined in metadata: confluent-platform/metadata.rb:1
Run with Foodcritic Version 10.3.1 with tags metadata,correctness ~FC031 ~FC045 and failure tags any
2.1.0 passed this metric
No Binaries Metric
2.1.0 passed this metric
Testing File Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.1.0 passed this metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must contain a TESTING.md file
Version Tag Metric
2.1.0 failed this metric
Failure: To pass this metric, your cookbook metadata must include a source url, the source url must be in the form of http://github.com/user/repo, and your repo must include a tag that matches this cookbook version number
2.1.0 failed this metric