![3. Let Prometheus scrape
a) Statically in prometheus.yml
b) Service Discovery
scrape_configs:
- job_name: 'jobmanager'
static_configs:
- targets: ['jobmanager:9999']
- job_name: 'taskmanager1'
static_configs:
- targets: ['taskmanager1:9999']
- [...]
docs](https://image.slidesharecdn.com/ff18monitoring-flink-with-prometheusbode-181007185717/75/Monitoring-Flink-with-Prometheus-8-2048.jpg)
![[Flink docs]
[Prometheus docs]
, Brian Brazil
[Prometheus docs]
Debugging & Monitoring / Metrics
Introduction / Overview
Prometheus Up & Running
prometheus/pushgateway
Remote endpoints & storage
improbable-eng/thanos](https://image.slidesharecdn.com/ff18monitoring-flink-with-prometheusbode-181007185717/75/Monitoring-Flink-with-Prometheus-11-2048.jpg)
Uploaded byMaximilian Bode
Monitoring Flink with Prometheus
The document discusses the integration of Flink with Prometheus for monitoring purposes, detailing the setup and configuration of a metrics reporting system. It outlines the use of Prometheus for scraping Flink metrics, using the PromQL query language for alerting, and configuring Prometheus to pull metrics from job and task managers. Additionally, it provides a code example for implementing custom metrics in Flink jobs.
More Related Content
Similar to Monitoring Flink with Prometheus
Monitoring Flink with Prometheus
- 1.
- 2. whoami Software Engineer with Focuson Data-Intensive Applications Site Reliability Engineering
- 3. open-source, metrics-based monitoringsystem simple yet powerful data model & query language client libraries in all popular languages high-performance and simple to run prometheus/prometheus
- 4. ☝☝ Metrics Time series of64-bit floating-point numbers Labels Key-value pairs associated with time series Scrape Act of fetching metrics via HTTP request TSDB Prometheus storage layer, PromQL Query language, used for graphing and alerting prometheus/tsdb flink_jobmanager_job_uptime{job_name="PrometheusExampleJob"}
- 5. grafana/grafana prometheus/alertmanager - alert: FlinkJobsMissing expr:sum(flink_api_jobs_running) < 2 for: 3m annotations: summary: Fewer Flink jobs than expected are running.
- 6.
- 7. PrometheusReporter 1. Copy reporterjar in lib directory 2. Configure in conf/flink-conf.yaml cp /opt/flink-metrics-prometheus-1.6.0.jar /lib metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.Prometheus metrics.reporter.prom.port: 9999 docs
- 8. 3. Let Prometheusscrape a) Statically in prometheus.yml b) Service Discovery scrape_configs: - job_name: 'jobmanager' static_configs: - targets: ['jobmanager:9999'] - job_name: 'taskmanager1' static_configs: - targets: ['taskmanager1:9999'] - [...] docs
- 9. 4. Use custommetrics in your jobs class CountingMap extends RichMapFunction<Integer, Integer> { private transient Counter eventCounter; @Override public void open(Configuration parameters) { eventCounter = getRuntimeContext().getMetricGroup().counter("events"); } @Override public Integer map(Integer value) { eventCounter.inc(); return value; } }
- 10. " " ( ) by Prayingsquirrel CC BY 2.0 Michael Seeley
- 11. [Flink docs] [Prometheus docs] ,Brian Brazil [Prometheus docs] Debugging & Monitoring / Metrics Introduction / Overview Prometheus Up & Running prometheus/pushgateway Remote endpoints & storage improbable-eng/thanos
- 12.