cookbook 'et_elk', '= 3.0.0'
et_elk (4) Versions 3.0.0 Follow0
Configures an Elk Cluster
cookbook 'et_elk', '= 3.0.0', :supermarket
knife supermarket install et_elk
knife supermarket download et_elk
ELK
This cookbook is the top level wrapper for the EverTrue ELK cookbook ecosystem
Requirements
-
et_elk
Ties all the peices of an ELK cluster together-
elk_forwarder
Installs and Configures Logstash forwarder -
elk_logstash
Installs and Configures a central Logstash server - Leverages the
logstash
cookbook -
elk_elasticsearch
Installs and Configures an optimized Elasticsearch for use with Logstash - Leverages the
elasticsearch
cookbook -
elk_kibana
Installs and Configures Kibana for visualizing logs - Leverages the
kibana
cookbook
-
How do I actually use this system
1. Forwarders
You need to apply the client
recipe to all your nodes. This will install and configure the Logstash Forwarder using elk_forwarder
2. Configure Forwarders
You then need to determine what logs you want to forward and you need to come up with a unique "type" to identify them. I.E. logs from rails app log files will have their type field set to rails_app
.
From your Attributes
set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log'] set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp' set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'
Or from your recipe. This one loops through a list of apps
apps.each do |app| node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"] node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = app node.set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar' end
3. Write your pattern
We need to build a pattern(s) to add to evertrue_patterns.erb or another patterns template that you have setup. Grab a bunch of sample logs and use the Grok Constructor utility to construct a pattern that will match your logs. I usually find an existing pattern online and then tweak it to properly match the logs I am parsing.
Give your pattern a name like ET_PUPPIES_APP
and add it to your patterns template.
4. Write your filter
This is pretty open ended, but here are the key requirements.
- Checks the log's type field with an
if
condition - Groks the log with the pattern you just created
- Parses a field containing some gross timestamp into the standard
@timestamp
field using thedate
filter. Note that thetimestamp
field is very different from@timestamp
.timestamp
is just a string extracted from the log which needs to be converted into a nice date/time.
Example:
filter { if [type] == "nginx_access" { grok { match => ["message", "%{ET_NGINXACCESS}"] } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } } }
Once you have written your filter, save it to a new template and call the logstash_config
resource with something like this in your cookbook's recipe
logstash_config 'nginx filter' do templates_cookbook 'yourcookbook' templates 'filter_nginx' => 'filter_nginx.erb' instance 'server' end
And now you are shipping, recieving, groking, parsing, inspecting, mutating, indexing, and storing your logs!
Recipes
default
Installs everything on the node and configures it to be a central log server that recieves logs from the clients
- Install Elasticsearch, Logstash, and kibana
- Configure logstash with inputs and outputs using
elk::server
client
Configures the node to be a client that ships logs to the central logstash server
- Install logstash-forwarder via
elk_forwarder
- Configure logstash-forwarder to forward logs to central log servers
- Central logstash servers are discovered via the
node['et_elk']['logstash_discovery']
attribute - To Disable auto-discovery simply set
node['et_elk']['logstash_discovery']
to''
- Central logstash servers are discovered via the
Usage
depends 'et_elk', '~> 1.0'
# To install and configure the forwarder include_recipe 'et_elk::client' # To install everything on the node include_recipe 'et_elk::default'
Contributing
- Fork the repository on Github
- Create a named feature branch (like
add_component_x
) - Write your change
- Write tests for your change (if applicable)
- Run the tests with
kitchen test
, ensuring they all pass - Submit a Pull Request using Github
License and Authors
Author:: EverTrue, inc. (eddie.hurtig@evertrue.com)
Dependent cookbooks
apt ~> 2.0 |
java ~> 1.0 |
runit ~> 1.5 |
elasticsearch ~> 2.1 |
storage ~> 2.2 |
logstash ~> 0.12.0 |
kibana_lwrp ~> 2.0 |
Contingent cookbooks
Change Log for et_elk
All notable changes to this project will be documented in this file.
This project adheres to Semantic Versioning.
Unreleased
Changed
[3.0.0] - 2015-12-22
Changed
- Cookbook functionality
- Move to Java 8
- Remove "client" functionality completely
- Move most "server_config" content (filters, patterns, etc) to
evertrue/logserver-cookbook
- Merge in
evertrue/elk_elasticsearch-cookbook
,evertrue/elk_logstash-cookbook
, andevertrue/elk_kibana-cookbook
- Move key/cert configuration to
evertrue/logserver-cookbook
- Drop
kibana
recipe in favor of directly includingkibana_lwrp::install
- Stop using default recipe just to install dependencies
- Give elasticsearch recipe its own attributes file
- Move kibana attributes to
server
- Clean up the recipe headers
- Remove blank/false attributes from server attributes file
- Use underscores instead of spaces for ssl-related keys
- Move attributes affecting logstash filter files into their own namespace for easy parsing
- Set lumberjack default codec to "plain" to match what we are using in prod
- Automatically include x_input_processor tag in all inputs
- Clean up a lot of useless attributes
- Remove resources that do nothing from logstash recipe
- Move inputs/outputs definition to this cookbook (from
evertrue/logserver-cookbook
) - Craft a helper method (
generate_module_config
) to simplify the rendering of the input/output templates - Upgrade
elasticsearch
cookbook to version 2 (and associated cookbook changes) - Upgrade
kibana
to v4.3.0 - Use less specific version for kibana_lwrp cookbook
- Test functionality
- Modernize TravisCI config
- Substantially clean up and improve the integration tests
- Remove Rubygems stuff completely (deprecated in favor of ChefDK)
- Remove unused default instace from Travis tests
- ServerSpec tests: Use net/http+JSON instead of cURL to parse ES status response
[2.3.3] - 2015-10-15
Changed
- Add a Java-type timestamp matcher to the java log filter
[2.3.2] - 2015-10-15
Changed
- Set flag x_input_processor on all inputs
[2.3.1] - 2015-10-15
Changed
- Add service restart notifications to logstash_config resource calls
[2.3.0] - 2015-10-15
Added
- Add log4j input for logstash
[2.2.0] - 2015-10-08
Added
- A rails_basic filter handler for handling basic rails logs without fancy multiline parsing
[2.1.1] - 2015-10-05
Added
- Attributes describing the plugins required for the logstash configs in et_elk
[2.1.0] - 2015-09-18
Changed
- Version pins for elk_elasticsearch and elk_logstash
Added
- Added sidekiq log proccessing support for events with the tag
sidekiq
[2.0.0] - 2015-08-10
Changed
- Changed the name
elk
->et_elk
- Match multiline log entries correctly
- Clean up a bunch of recipe code
- Spin off a new recipe from
default
calledserver
(and its sub-recipeserver_config
) - Add a bunch of new ServerSpec tests
- Avoid using storage cookbook w/ Vagrant
- Update kitchen-ec2 config as per deprecations
- Avoid the use of custom timestamp parsers
- Drop support for 12.04
- Change test environment name from _default to dev
1.3.5 - 2015-08-07
Added
- Java App Patterns and Filters
1.3.4 - 2015-07-01
Changed
- Specify that the multiline filter should send lines to the previous event
1.3.3 - 2015-07-01
Changed
- Added extra patterns to attempt to match mesos timestamps
- Parse and Remove the
timestamp
field for syslog events - Add Multiline support for singularity
1.3.2 - 2015-06-29
Changed
- Removed .conf from template names
1.3.1 - 2015-06-29
Changed
- The namespace for logstash input, filter, and output configuration attributes from elk_logstash to just elk
1.3.0 - 2015-06-29
Added
- The inputs, outputs, and syslog filter moved from the
elk_logstash
cookbook
1.2.5 - 2015-06-26
Added
- A common filter to perform a geoip lookup if a
client_ip
field is present - Instructions and guidlines for adding new patterns to evertrue_patterns
Changed
- The raw timestamp field from
mesos_timestamp
to justtimestamp
as it is the primary time for that log - Removed year, month, day, hour, minute, second, and timezone fields as they aren't useful when the timestamp is parsed into a searchable date
- Major cleanup of the schema. Lots of renaming of fields.
1.2.4 - 2015-06-26
Changed
- Quoted ISO8601 to make configuration valid
1.2.3 - 2015-06-26
Changed
- Used
or
instead of||
in mesos slave filter
1.2.2 - 2015-06-25
Added
- Filters to parse and remove the
timestamp
fied for haproxy and rails apps into@timestamp
1.2.1 - 2015-06-25
Changed
- Add logstash_config resource for mesos filters
1.2.0 - 2015-06-24
Added
- Test Kitchen role and attributes to converge a working All in One box
- Initial Patterns and Filters for Mesos and Singlularity
- Fauxhai to mock ohai data
- date filter to NGINX logs
1.1.1 - 2015-06-23
Changed
- Updated the ET_NGINXERROR pattern to properly match error logs
1.1.0 - 2015-06-22
Added
- NGINX log parsing configuration
1.0.7 - 2015-06-18
Added
- Fixes templates cookbook for evertrue patterns
1.0.6 - 2015-06-18
Added
- Add HAPROXY config
1.0.5 - 2015-06-17
Changed
- Loosen format of view and activerecord timing.
1.0.4 - 2015-06-17
Changed
- The multiline filter's match pattern to accomodate logs with timestamp prefixed lines
1.0.3 - 2015-06-17
Added
- SSL Certificate discovery attributes to find the CA cert
Changed
- Fixed the rails app filter
1.0.2 - 2015-06-15
Added
- A Fixup to wipe the elk_forwarder namespace if ['elk_forwarder']['config']['files'] is an array not the new format which is a hash
1.0.1 - 2015-06-15
Changed
- Updated to elk_forwarder v2.0
1.0.0 - 2015-06-15
Added
- Rails App log filter and pattern
Changed
Removed
0.0.1 - YYYY-MM-DD
Added
- Initial Release
Foodcritic Metric
3.0.0 failed this metric
FC003: Check whether you are running with chef server before using server-specific features: /tmp/cook/a305969a7badd67a5c72ced8/et_elk/recipes/elasticsearch.rb:11
FC007: Ensure recipe dependencies are reflected in cookbook metadata: /tmp/cook/a305969a7badd67a5c72ced8/et_elk/recipes/elasticsearch.rb:13
3.0.0 failed this metric
FC007: Ensure recipe dependencies are reflected in cookbook metadata: /tmp/cook/a305969a7badd67a5c72ced8/et_elk/recipes/elasticsearch.rb:13