graphite_exporter收集spark on yarn metrics

最新推荐文章于 2025-06-01 10:44:28 发布

原创最新推荐文章于 2025-06-01 10:44:28 发布 · 3.3k 阅读

CC 4.0 BY-SA版权

文章标签：

参考：https://community.hortonworks.com/articles/222813/monitoring-spark-2-performance-via-grafana-in-amba-1.html
https://blog.csdn.net/lsshlsw/article/details/82670508
jmx 方式，端口冲突问题
https://argus-sec.com/monitoring-spark-prometheus
https://www.cnblogs.com/fengzzi/p/10033810.html

1，Ambari中设置spark 配置文件
Ambari -> Spark2 -> Configs -> Advanced spark2-metrics-properties.

*.source.jvm.class=org.apache.spark.metrics.source.JvmSource
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
*.sink.graphite.protocol=tcp
*.sink.graphite.host=10.5.147.112
*.sink.graphite.port=9109
*.sink.graphite.period=1
*.sink.graphite.unit=seconds

graphite_exporter 接收数据端口为9109
https://github.com/prometheus/prometheus/wiki/Default-port-allocations

2，配置graphite_exporter使用的映射文件spark_mapping

mappings:
- match: '*.*.executor.filesystem.*.*'
  name: spark_app_filesystem_usage
  labels:
    application: $1
    executor_id: $2
    fs_type: $3
    qty: $4

- match: '*.*.jvm.*.*'
  name: spark_app_jvm_memory_usage
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4

- match: '*.*.executor.jvmGCTime.count'
  name: spark_app_jvm_gcTime_count
  labels:
    application: $1
    executor_id: $2

- match: '*.*.jvm.pools.*.*'
  name: spark_app_jvm_memory_pools
  labels:
    application: $1
    executor_id: $2
    mem_type: $3
    qty: $4

- match: '*.*.executor.threadpool.*'
  name: spark_app_executor_tasks
  labels:
    application: $1
    executor_id: $2
    qty: $3

- match: '*.*.BlockManager.*.*'
  name: spark_app_block_manager
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4

- match: '*.*.DAGScheduler.*.*'
  name: spark_app_dag_scheduler
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4

- match: '*.*.CodeGenerator.*.*'
  name: spark_app_code_generator
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4


- match: '*.*.HiveExternalCatalog.*.*'
  name: spark_app_hive_external_catalog
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4

- match: '*.*.*.StreamingMetrics.*.*'
  name: spark_app_streaming_metrics
  labels:
    application: $1
    executor_id: $2
    app_name: $3
    type: $4
    qty: $5

原生的 Graphite 数据可以通过映射文件转化为有 label 维度的 Prometheus 数据。

例如：

- match: '*.*.HiveExternalCatalog.*.*'
  name: spark_app_hive_external_catalog
  labels:
    application: $1
    executor_id: $2
    type: $3
    qty: $4

spark metrics 原始数据格式：
http://IP:8088/proxy/application_1555401851194_0077/metrics/json

"application_1555401851194_0077.driver.HiveExternalCatalog.filesDiscovered":{"count":400},
"application_1555401851194_0077.driver.HiveExternalCatalog.hiveClientCalls":{"count":2},
"application_1555401851194_0077.driver.HiveExternalCatalog.parallelListingJobCount":{"count":0},
"application_1555401851194_0077.driver.HiveExternalCatalog.partitionsFetched":{"count":0}

转化后数据格式（注意：此处只是展示格式，使用的是application_1556625797222_0045的数据，application_1555401851194_0077的数据找不到了）：

# HELP spark_app_hive_external_catalog Graphite metric spark_app_hive_external_catalog
# TYPE spark_app_hive_external_catalog gauge
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="fileCacheHits"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="filesDiscovered"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="hiveClientCalls"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="parallelListingJobCount"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="partitionsFetched"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="fileCacheHits"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="filesDiscovered"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="hiveClientCalls"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="parallelListingJobCount"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="partitionsFetched"} 0

启动 graphite_exporter 时加载配置文件

 ./graphite_exporter --graphite.mapping-config=spark_mapping

配置 Prometheus 从 graphite_exporter 获取数据,重启Prometheus服务
/opt/appdata/disk01/app/monitor/prometheus/prometheus.yml

# spark : graphite_exporter
  - job_name: 'spark'
    static_configs:
    - targets: ['10.5.147.112:9108']

3，提交spark 应用

spark-submit --class org.apache.spark.examples.SparkPi  --name SparkPi --master yarn --deploy-mode  cluster  --executor-memory 1G --executor-cores 1  --num-executors 1 /usr/hdp/2.6.2.0-205/spark2/examples/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar 1000

访问 graphite_exporter，获取收集到的数据

curl 10.5.147.112:9108/metrics

应用application_1556625797222_0045的metrics数据如下：

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.5844e-05
go_gc_duration_seconds{quantile="0.25"} 7.1282e-05
go_gc_duration_seconds{quantile="0.5"} 9.674e-05
go_gc_duration_seconds{quantile="0.75"} 0.0001363
go_gc_duration_seconds{quantile="1"} 0.021169326
go_gc_duration_seconds_sum 9.167736982
go_gc_duration_seconds_count 18835
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 13
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.11.5"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.700416e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.1738557888e+10
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.510776e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 8.50240471e+08
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.00012865331632467894
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.379776e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.700416e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.2210048e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.079616e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 32479
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6289664e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.557544762843171e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 8.5027295e+08
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13824
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 71592
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 98304
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.956736e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 819200
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 819200
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.307084e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 14
# HELP graphite_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which graphite_exporter was built.
# TYPE graphite_exporter_build_info gauge
graphite_exporter_build_info{branch="HEAD",goversion="go1.11.5",revision="320808c4e456c899e9c223879397efc48bca7364",version="0.5.0"} 1
# HELP graphite_last_processed_timestamp_seconds Unix timestamp of the last processed graphite metric.
# TYPE graphite_last_processed_timestamp_seconds gauge
graphite_last_processed_timestamp_seconds 1.5575447648915062e+09
# HELP graphite_sample_expiry_seconds How long in seconds a metric sample is valid for.
# TYPE graphite_sample_expiry_seconds gauge
graphite_sample_expiry_seconds 300
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 473.81
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.8243584e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.55745888882e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.16604928e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 5726
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP spark_app_block_manager Graphite metric spark_app_block_manager
# TYPE spark_app_block_manager gauge
spark_app_block_manager{application="application_1556625797222_0045",executor_id="driver",qty="diskSpaceUsed_MB",type="disk"} 0
spark_app_block_manager{application="application_1556625797222_0045",executor_id="driver",qty="maxMem_MB",type="memory"} 732
spark_app_block_manager{application="application_1556625797222_0045",executor_id="driver",qty="memUsed_MB",type="memory"} 0
spark_app_block_manager{application="application_1556625797222_0045",executor_id="driver",qty="remainingMem_MB",type="memory"} 732
# HELP spark_app_code_generator Graphite metric spark_app_code_generator
# TYPE spark_app_code_generator gauge
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="count",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="count",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="count",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="count",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="max",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="max",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="max",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="max",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="mean",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="mean",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="mean",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="mean",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="min",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="min",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="min",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="min",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p50",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p50",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p50",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p50",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p75",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p75",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p75",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p75",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p95",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p95",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p95",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p95",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p98",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p98",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p98",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p98",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p99",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p99",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p99",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p99",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p999",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p999",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p999",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="p999",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="stddev",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="stddev",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="stddev",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="1",qty="stddev",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="count",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="count",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="count",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="count",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="max",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="max",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="max",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="max",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="mean",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="mean",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="mean",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="mean",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="min",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="min",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="min",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="min",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p50",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p50",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p50",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p50",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p75",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p75",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p75",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p75",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p95",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p95",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p95",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p95",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p98",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p98",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p98",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p98",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p99",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p99",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p99",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p99",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p999",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p999",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p999",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="p999",type="sourceCodeSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="stddev",type="compilationTime"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="stddev",type="generatedClassSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="stddev",type="generatedMethodSize"} 0
spark_app_code_generator{application="application_1556625797222_0045",executor_id="driver",qty="stddev",type="sourceCodeSize"} 0
# HELP spark_app_dag_scheduler Graphite metric spark_app_dag_scheduler
# TYPE spark_app_dag_scheduler gauge
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="activeJobs",type="job"} 1
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="allJobs",type="job"} 1
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="count",type="messageProcessingTime"} 1.209369e+06
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="failedStages",type="stage"} 0
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="m15_rate",type="messageProcessingTime"} 0
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="m1_rate",type="messageProcessingTime"} 0
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="m5_rate",type="messageProcessingTime"} 0
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="max",type="messageProcessingTime"} 1284.67
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="mean",type="messageProcessingTime"} 4.04
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="mean_rate",type="messageProcessingTime"} 7.66
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="min",type="messageProcessingTime"} 0
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="p50",type="messageProcessingTime"} 0.01
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="p75",type="messageProcessingTime"} 0.03
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="p95",type="messageProcessingTime"} 0.06
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="p98",type="messageProcessingTime"} 0.08
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="p99",type="messageProcessingTime"} 0.11
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="p999",type="messageProcessingTime"} 1284.67
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="runningStages",type="stage"} 1
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="stddev",type="messageProcessingTime"} 71.36
spark_app_dag_scheduler{application="application_1556625797222_0045",executor_id="driver",qty="waitingStages",type="stage"} 0
# HELP spark_app_executor_tasks Graphite metric spark_app_executor_tasks
# TYPE spark_app_executor_tasks gauge
spark_app_executor_tasks{application="application_1556625797222_0045",executor_id="1",qty="activeTasks"} 0
spark_app_executor_tasks{application="application_1556625797222_0045",executor_id="1",qty="completeTasks"} 604684
spark_app_executor_tasks{application="application_1556625797222_0045",executor_id="1",qty="currentPool_size"} 0
spark_app_executor_tasks{application="application_1556625797222_0045",executor_id="1",qty="maxPool_size"} 2.147483647e+09
# HELP spark_app_filesystem_usage Graphite metric spark_app_filesystem_usage
# TYPE spark_app_filesystem_usage gauge
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="file",qty="largeRead_ops"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="file",qty="read_bytes"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="file",qty="read_ops"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="file",qty="write_bytes"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="file",qty="write_ops"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="hdfs",qty="largeRead_ops"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="hdfs",qty="read_bytes"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="hdfs",qty="read_ops"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="hdfs",qty="write_bytes"} 0
spark_app_filesystem_usage{application="application_1556625797222_0045",executor_id="1",fs_type="hdfs",qty="write_ops"} 0
# HELP spark_app_hive_external_catalog Graphite metric spark_app_hive_external_catalog
# TYPE spark_app_hive_external_catalog gauge
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="fileCacheHits"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="filesDiscovered"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="hiveClientCalls"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="parallelListingJobCount"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="1",qty="count",type="partitionsFetched"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="fileCacheHits"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="filesDiscovered"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="hiveClientCalls"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="parallelListingJobCount"} 0
spark_app_hive_external_catalog{application="application_1556625797222_0045",executor_id="driver",qty="count",type="partitionsFetched"} 0
# HELP spark_app_jvm_memory_pools Graphite metric spark_app_jvm_memory_pools
# TYPE spark_app_jvm_memory_pools gauge
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Code-Cache",qty="committed"} 1.5335424e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Code-Cache",qty="init"} 2.555904e+06
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Code-Cache",qty="max"} 2.5165824e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Code-Cache",qty="usage"} 0.05
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Code-Cache",qty="used"} 1.3450752e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Compressed-Class-Space",qty="committed"} 4.456448e+06
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Compressed-Class-Space",qty="init"} 0
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Compressed-Class-Space",qty="max"} 1.073741824e+09
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Compressed-Class-Space",qty="usage"} 0
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Compressed-Class-Space",qty="used"} 4.260576e+06
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Metaspace",qty="committed"} 3.3030144e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Metaspace",qty="init"} 0
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Metaspace",qty="max"} -1
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Metaspace",qty="usage"} 0.98
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="Metaspace",qty="used"} 3.2358176e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Eden-Space",qty="committed"} 6.029312e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Eden-Space",qty="init"} 2.68435456e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Eden-Space",qty="max"} 3.5651584e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Eden-Space",qty="usage"} 0.05
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Eden-Space",qty="used"} 1.6893464e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Old-Gen",qty="committed"} 3.145728e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Old-Gen",qty="init"} 7.16177408e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Old-Gen",qty="max"} 7.16177408e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Old-Gen",qty="usage"} 0.1
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Old-Gen",qty="used"} 6.840348e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Survivor-Space",qty="committed"} 524288
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Survivor-Space",qty="init"} 4.456448e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Survivor-Space",qty="max"} 524288
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Survivor-Space",qty="usage"} 0.38
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Survivor-Space",qty="used"} 196608
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Code-Cache",qty="committed"} 2.5034752e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Code-Cache",qty="init"} 2.555904e+06
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Code-Cache",qty="max"} 2.5165824e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Code-Cache",qty="usage"} 0.1
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Code-Cache",qty="used"} 2.4128e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Compressed-Class-Space",qty="committed"} 7.602176e+06
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Compressed-Class-Space",qty="init"} 0
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Compressed-Class-Space",qty="max"} 1.073741824e+09
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Compressed-Class-Space",qty="usage"} 0.01
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Compressed-Class-Space",qty="used"} 7.410696e+06
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Metaspace",qty="committed"} 5.8458112e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Metaspace",qty="init"} 0
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Metaspace",qty="max"} -1
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Metaspace",qty="usage"} 0.98
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="Metaspace",qty="used"} 5.74624e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Eden-Space",qty="committed"} 3.12475648e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Eden-Space",qty="init"} 2.68435456e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Eden-Space",qty="max"} 3.12475648e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Eden-Space",qty="usage"} 0.99
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Eden-Space",qty="used"} 3.09162064e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Old-Gen",qty="committed"} 7.16177408e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Old-Gen",qty="init"} 7.16177408e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Old-Gen",qty="max"} 7.16177408e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Old-Gen",qty="usage"} 1
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Old-Gen",qty="used"} 7.15873632e+08
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Survivor-Space",qty="committed"} 2.2020096e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Survivor-Space",qty="init"} 4.456448e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Survivor-Space",qty="max"} 2.2020096e+07
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Survivor-Space",qty="usage"} 0
spark_app_jvm_memory_pools{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Survivor-Space",qty="used"} 0
# HELP spark_app_jvm_memory_usage Graphite metric spark_app_jvm_memory_usage
# TYPE spark_app_jvm_memory_usage gauge
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="PS-MarkSweep",qty="count"} 1
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="PS-MarkSweep",qty="time"} 27
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Scavenge",qty="count"} 3954
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="PS-Scavenge",qty="time"} 6375
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="heap",qty="committed"} 3.75390208e+08
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="heap",qty="init"} 1.073741824e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="heap",qty="max"} 9.54728448e+08
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="heap",qty="usage"} 0.09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="heap",qty="used"} 8.5493552e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="non-heap",qty="committed"} 5.2822016e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="non-heap",qty="init"} 2.555904e+06
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="non-heap",qty="max"} -1
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="non-heap",qty="usage"} -5.0069504e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="non-heap",qty="used"} 5.0069504e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="total",qty="committed"} 4.28212224e+08
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="total",qty="init"} 1.076297728e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="total",qty="max"} 9.54728447e+08
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="1",mem_type="total",qty="used"} 1.35563056e+08
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-MarkSweep",qty="count"} 9509
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-MarkSweep",qty="time"} 1.0854586e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Scavenge",qty="count"} 391
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="PS-Scavenge",qty="time"} 308012
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="heap",qty="committed"} 1.050673152e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="heap",qty="init"} 1.073741824e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="heap",qty="max"} 1.050673152e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="heap",qty="usage"} 0.98
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="heap",qty="used"} 1.025035696e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="non-heap",qty="committed"} 9.109504e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="non-heap",qty="init"} 2.555904e+06
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="non-heap",qty="max"} -1
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="non-heap",qty="usage"} -8.9001096e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="non-heap",qty="used"} 8.9001096e+07
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="total",qty="committed"} 1.141768192e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="total",qty="init"} 1.076297728e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="total",qty="max"} 1.050673151e+09
spark_app_jvm_memory_usage{application="application_1556625797222_0045",executor_id="driver",mem_type="total",qty="used"} 1.114036792e+09