Dive Into Graphite
And Grafana
Denis Trifonov
Denis Trifonov
QA engineer at load testing team
Previously we generated load by Yandex.Tank. Also we used it's console output and built-in reports for analysis.
Graphite — enterprise-scale monitoring tool for metrics store and graphs rendering. Yandex.Tank has a built-in Graphite support.
Carbon is receiving metrics over the network, caching them in memory for “hot queries” from the Graphite-Web application, and persisting them to disk using the Whisper or Ceres time-series library
Whisper is a fixed-size database, similar in design and purpose to RRD (round-robin-database). It provides fast, reliable storage of numeric data over time. Whisper allows for higher resolution (seconds per point) of recent data to degrade into lower resolutions for long-term retention of historical data.
Ceres is a time-series database format intended to replace Whisper as the default storage format for Graphite. In contrast with Whisper, Ceres is not a fixed-size database and is designed to better support sparse data of arbitrary fixed-size resolutions. This allows Graphite to distribute individual time-series across multiple servers or mounts.
Supports in megacarbon Carbon git branch.
A Django webapp that renders graphs on-demand using Cairo. As you send datapoints to Carbon, they become immediately available for graphing in the webapp. The webapp offers several ways to create and display graphs including a simple URL API for rendering that makes it easy to embed graphs in other webpages.
Ansible roles will be in public (on GitHub) later.
# daemon.conf
PIPELINE = write
# db.conf
DATABASE = ceres
LOCAL_DATA_DIR = /opt/graphite/storage/ceres/# listeners.conf
[plaintext-receiver]
port = 2003
type = plaintext-receiver
[pickle-receiver]
port = 2004
type = pickle-receiver# storage-rules.conf
[defaults]
match-all = true
retention = 1m:30d
aggregation-method = average# writer.conf
CACHE_QUERY_PORT = 7002
# When using Ceres you should set this to a much higher
# value like 1000, if using Whisper - set this value low
# (like at 50)
MAX_CREATES_PER_MINUTE = 1000# local_settings.py
STORAGE_FINDERS = ('graphite.finders.ceres.CeresFinder',)
# CLUSTER_SERVERS = []
CARBONLINK_HOSTS = ['127.0.0.1:7002:cache0']
You may add more Carbon cache daemons for full utilization of CPU resources and disk IO on same node, add new Graphite nodes to cluster and top-level Carbon relay daemons (or aggregation layer), setup replication and other. It's up on you.
See all on graphite.readthedocs.org/en/latest/tools.html#visualization
An open source, feature rich metrics dashboard and graph editor for Graphite, InfluxDB & OpenTSDB
Live demo play.grafana.org/#/dashboard/db/templated-graphs-nested
dashboard.rows.push({
panels: [{
targets: [{
'target': 'test.app.some.metrics'
}]
}]
});URL: http://grafana_url/#/dashboard/script/scripted.js?foo=bar
Live demo play.grafana.org/#/dashboard/db/graphite-carbon-metrics