ArgoCD GitOps Controller Prometheus

πŸ’‘ ArgoCD GitOps Controller Prometheus β€” Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for ArgoCD GitOps Controller Prometheus.


Index

Prometheus Manifest
References and Guidelines (NOTE)

Name: Prometheus
Deployed: k8s, argo-cd, bitnami
Note: Automatic creation of Node Exporter when using default value (Master + Worker)


Argo-cd Manifest, Target Definition

yaml
project: default
source:
  repoURL: 'https://charts.bitnami.com/bitnami'
  targetRevision: 8.2.2
  helm:
    valueFiles:
      - values.yaml
    parameters:
      - name: prometheus.service.nodePorts.http
        value: '30007'
      - name: prometheus.service.type
        value: NodePort
      - name: global.storageClass
        value: cmp
      - name: kubeControllerManager.service.ports.http
        value: '10257'
      - name: kubeControllerManager.service.targetPorts.http
        value: '10257'
      - name: kubeControllerManager.serviceMonitor.https
        value: 'true'
      - name: kubeControllerManager.serviceMonitor.insecureSkipVerify
        value: 'true'
      - name: kubeScheduler.service.ports.http
        value: '10259'
      - name: kubeScheduler.service.targetPorts.http
        value: '10259'
      - name: kubeScheduler.serviceMonitor.https
        value: 'true'
      - name: kubeScheduler.serviceMonitor.insecureSkipVerify
        value: 'true'
      - name: prometheus.additionalScrapeConfigs.enabled
        value: 'true'
      - name: prometheus.additionalScrapeConfigs.type
        value: internal
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[0].job_name'
        value: kube-middleware-redis-exporter
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[0].static_configs[0].targets[0]
        value: 'redis-redis-cluster-metrics.redis.svc:9121'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[1].job_name'
        value: kube-middleware-rabbitmq-exporter
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].static_configs[0].targets[0]
        value: 'rabbitmq-test.rabbitmq.svc:9419'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[2].job_name'
        value: kube-middleware-keycloak-exporter
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[2].static_configs[0].targets[0]
        value: 'portal-keycloak-metrics.keycloak.svc:8080'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[3].job_name'
        value: openstack-node-exporter
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[3].static_configs[0].targets[0]
        value: '100.0.0.245:9100'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[4].job_name'
        value: openstack-exporter
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[4].scrape_timeout'
        value: 30s
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[4].static_configs[0].targets[0]
        value: '100.0.0.245:9180'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[5].job_name'
        value: openstack-libvirt-exporter
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[5].static_configs[0].targets[0]
        value: '100.0.0.245:9177'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[6].job_name'
        value: kube-middleware-grafana-exporter
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[6].static_configs[0].targets[0]
        value: 'grafana-test.grafana-test.svc:3000'
  chart: kube-prometheus
destination:
  server: 'https://kubernetes.default.svc'
  namespace: prometheus
syncPolicy:
  syncOptions:
    - ServerSideApply=true
info:
  - name: additional jobList 경둜
    value: prometheus.additionalScrapeConfigs.internal.jobList
  - name: 'additional jobList[0]'
    value: (kube) Redis Exporter
  - name: 'additional jobList[1]'
    value: (kube) RabbitMQ Exporter
  - name: 'additional jobList[2]'
    value: (kube) Keycloak Exporter
  - name: 'additional jobList[3]'
    value: Openstack Node Exporter
  - name: 'additional jobList[4]'
    value: Openstack Exporter
  - name: 'additional jobList[5]'
    value: Openstack Libvirt Exporter
  - name: 'additional jobList[6]'
    value: Grafana Exporter


Fixing Argo CD β€œToo long must have at most 262144 bytes” error

yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  syncPolicy:
    syncOptions:
    - ServerSideApply=true


Redis Multi Cluster(Source) Option

bash
- name: 'prometheus.additionalScrapeConfigs.internal.jobList[1].job_name'
        value: redis_exporter_targets
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].static_configs[0].targets[0]
        value: 'redis://{{ CLUSTER_IP }}:{{ PORT }}'
      - name: 'prometheus.additionalScrapeConfigs.internal.jobList[1].metrics_path'
        value: /scrape
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].relabel_configs[0].source_labels[0]
        value: __address__
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].relabel_configs[0].target_label
        value: __param_target
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].relabel_configs[1].source_labels[0]
        value: __param_target
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].relabel_configs[1].target_label
        value: instance
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].relabel_configs[2].target_label
        value: __address__
      - name: >-
          prometheus.additionalScrapeConfigs.internal.jobList[1].relabel_configs[2].replacement
        value: '{{ EXPORTER_HOST }}:9121'

\ is the general exporter approach, \ is the same metric scraped with multicluster properties


(old) Values.yaml Overriding

bash
--set global.storageClass=${STORAGE_CLASS} \
--set prometheus.service.type=NodePort \
--set prometheus.service.nodePorts.http=${NODE_PORT} \
--set prometheus.additionalScrapeConfigs.enabled=true \
--set prometheus.additionalScrapeConfigs.type=internal \
--set prometheus.additionalScrapeConfigs.internal.jobList[0].job_name=redis_exporter \
--set prometheus.additionalScrapeConfigs.internal.jobList[0].static_configs[0].targets[0]=${REDIS_EXPORTER_HOST}:9121

Redis Exporter Manifest
References and Guidelines (NOTE)

Name: Redis
Deployed: k8s, argo-cd, bitnami


Argo-cd Manifest, Enable Exporter

yaml
project: default
source:
  repoURL: {{ gitrepo }}
  path: mdware-helm/charts/redis-cluster
  targetRevision: develop
  helm:
    valueFiles:
      - values.yaml
    parameters:
      - name: global.redis.password
        value: {{ password }}
      - name: global.storageClass
        value: cmp
			- name: service.nodePorts.redis
        value: 30379
      - name: service.type
        value: NodePort
      - name: metrics.enabled
        value: 'true'
destination:
  server: 'https://kubernetes.default.svc'
  namespace: redis


redis-exporter (:9121)

txt
go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
 go_goroutines Number of goroutines that currently exist.
 go_info Information about the Go environment.
 go_memstats_alloc_bytes Number of bytes allocated and still in use.
 go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
 go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
 go_memstats_frees_total Total number of frees.
 go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
 go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
 go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
 go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
 go_memstats_heap_objects Number of allocated objects.
 go_memstats_heap_released_bytes Number of heap bytes released to OS.
 go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
 go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
 go_memstats_lookups_total Total number of pointer lookups.
 go_memstats_mallocs_total Total number of mallocs.
 go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
 go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
 go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
 go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
 go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
 go_memstats_other_sys_bytes Number of bytes used for other system allocations.
 go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
 go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
 go_memstats_sys_bytes Number of bytes obtained from system.
 go_threads Number of OS threads created.
 process_cpu_seconds_total Total user and system CPU time spent in seconds.
 process_max_fds Maximum number of open file descriptors.
 process_open_fds Number of open file descriptors.
 process_resident_memory_bytes Resident memory size in bytes.
 process_start_time_seconds Start time of the process since unix epoch in seconds.
 process_virtual_memory_bytes Virtual memory size in bytes.
 process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
 redis_active_defrag_running active_defrag_running metric
 redis_allocator_active_bytes allocator_active_bytes metric
 redis_allocator_allocated_bytes allocator_allocated_bytes metric
 redis_allocator_frag_bytes allocator_frag_bytes metric
 redis_allocator_frag_ratio allocator_frag_ratio metric
 redis_allocator_resident_bytes allocator_resident_bytes metric
 redis_allocator_rss_bytes allocator_rss_bytes metric
 redis_allocator_rss_ratio allocator_rss_ratio metric
 redis_aof_base_size_bytes aof_base_size_bytes metric
 redis_aof_buffer_length aof_buffer_length metric
 redis_aof_current_rewrite_duration_sec aof_current_rewrite_duration_sec metric
 redis_aof_current_size_bytes aof_current_size_bytes metric
 redis_aof_delayed_fsync aof_delayed_fsync metric
 redis_aof_enabled aof_enabled metric
 redis_aof_last_bgrewrite_status aof_last_bgrewrite_status metric
 redis_aof_last_cow_size_bytes aof_last_cow_size_bytes metric
 redis_aof_last_rewrite_duration_sec aof_last_rewrite_duration_sec metric
 redis_aof_last_write_status aof_last_write_status metric
 redis_aof_pending_bio_fsync aof_pending_bio_fsync metric
 redis_aof_pending_rewrite aof_pending_rewrite metric
 redis_aof_rewrite_in_progress aof_rewrite_in_progress metric
 redis_aof_rewrite_scheduled aof_rewrite_scheduled metric
 redis_blocked_clients blocked_clients metric
 redis_client_recent_max_input_buffer_bytes client_recent_max_input_buffer_bytes metric
 redis_client_recent_max_output_buffer_bytes client_recent_max_output_buffer_bytes metric
 redis_clients_in_timeout_table clients_in_timeout_table metric
 redis_cluster_connections cluster_connections metric
 redis_cluster_current_epoch cluster_current_epoch metric
 redis_cluster_enabled cluster_enabled metric
 redis_cluster_known_nodes cluster_known_nodes metric
 redis_cluster_messages_received_total cluster_messages_received_total metric
 redis_cluster_messages_sent_total cluster_messages_sent_total metric
 redis_cluster_my_epoch cluster_my_epoch metric
 redis_cluster_size cluster_size metric
 redis_cluster_slots_assigned cluster_slots_assigned metric
 redis_cluster_slots_fail cluster_slots_fail metric
 redis_cluster_slots_ok cluster_slots_ok metric
 redis_cluster_slots_pfail cluster_slots_pfail metric
 redis_cluster_state cluster_state metric
 redis_cluster_stats_messages_meet_sent cluster_stats_messages_meet_sent metric
 redis_cluster_stats_messages_ping_received cluster_stats_messages_ping_received metric
 redis_cluster_stats_messages_ping_sent cluster_stats_messages_ping_sent metric
 redis_cluster_stats_messages_pong_received cluster_stats_messages_pong_received metric
 redis_cluster_stats_messages_pong_sent cluster_stats_messages_pong_sent metric
 redis_commands_duration_seconds_total Total amount of time in seconds spent per command
 redis_commands_failed_calls_total Total number of errors prior command execution per command
 redis_commands_processed_total commands_processed_total metric
 redis_commands_rejected_calls_total Total number of errors within command execution per command
 redis_commands_total Total number of calls per command
 redis_config_io_threads config_io_threads metric
 redis_config_maxclients config_maxclients metric
 redis_config_maxmemory config_maxmemory metric
 redis_connected_clients connected_clients metric
 redis_connected_slave_lag_seconds Lag of connected slave
 redis_connected_slave_offset_bytes Offset of connected slave
 redis_connected_slaves connected_slaves metric
 redis_connections_received_total connections_received_total metric
 redis_cpu_sys_children_seconds_total cpu_sys_children_seconds_total metric
 redis_cpu_sys_main_thread_seconds_total cpu_sys_main_thread_seconds_total metric
 redis_cpu_sys_seconds_total cpu_sys_seconds_total metric
 redis_cpu_user_children_seconds_total cpu_user_children_seconds_total metric
 redis_cpu_user_main_thread_seconds_total cpu_user_main_thread_seconds_total metric
 redis_cpu_user_seconds_total cpu_user_seconds_total metric
 redis_db_keys Total number of keys by DB
 redis_db_keys_expiring Total number of expiring keys by DB
 redis_defrag_hits defrag_hits metric
 redis_defrag_key_hits defrag_key_hits metric
 redis_defrag_key_misses defrag_key_misses metric
 redis_defrag_misses defrag_misses metric
 redis_dump_payload_sanitizations dump_payload_sanitizations metric
 redis_errors_total Total number of errors per error type
 redis_evicted_keys_total evicted_keys_total metric
 redis_expired_keys_total expired_keys_total metric
 redis_expired_stale_percentage expired_stale_percentage metric
 redis_expired_time_cap_reached_total expired_time_cap_reached_total metric
 redis_exporter_build_info redis exporter build_info
 redis_exporter_last_scrape_connect_time_seconds exporter_last_scrape_connect_time_seconds metric
 redis_exporter_last_scrape_duration_seconds exporter_last_scrape_duration_seconds metric
 redis_exporter_last_scrape_error The last scrape error status.
 redis_exporter_scrape_duration_seconds Durations of scrapes by the exporter
 redis_exporter_scrapes_total Current total redis scrapes.
 redis_instance_info Information about the Redis instance
 redis_io_threaded_reads_processed io_threaded_reads_processed metric
 redis_io_threaded_writes_processed io_threaded_writes_processed metric
 redis_io_threads_active io_threads_active metric
 redis_keyspace_hits_total keyspace_hits_total metric
 redis_keyspace_misses_total keyspace_misses_total metric
 redis_last_key_groups_scrape_duration_milliseconds Duration of the last key group metrics scrape in milliseconds
 redis_last_slow_execution_duration_seconds The amount of time needed for last slow execution, in seconds
 redis_latency_percentiles_usec A summary of latency percentile distribution per command
 redis_latest_fork_seconds latest_fork_seconds metric
 redis_lazyfree_pending_objects lazyfree_pending_objects metric
 redis_loading_dump_file loading_dump_file metric
 redis_master_repl_offset master_repl_offset metric
 redis_mem_clients_normal mem_clients_normal metric
 redis_mem_clients_slaves mem_clients_slaves metric
 redis_mem_fragmentation_bytes mem_fragmentation_bytes metric
 redis_mem_fragmentation_ratio mem_fragmentation_ratio metric
 redis_mem_not_counted_for_eviction_bytes mem_not_counted_for_eviction_bytes metric
 redis_memory_max_bytes memory_max_bytes metric
 redis_memory_used_bytes memory_used_bytes metric
 redis_memory_used_dataset_bytes memory_used_dataset_bytes metric
 redis_memory_used_lua_bytes memory_used_lua_bytes metric
 redis_memory_used_overhead_bytes memory_used_overhead_bytes metric
 redis_memory_used_peak_bytes memory_used_peak_bytes metric
 redis_memory_used_rss_bytes memory_used_rss_bytes metric
 redis_memory_used_scripts_bytes memory_used_scripts_bytes metric
 redis_memory_used_startup_bytes memory_used_startup_bytes metric
 redis_migrate_cached_sockets_total migrate_cached_sockets_total metric
 redis_module_fork_in_progress module_fork_in_progress metric
 redis_module_fork_last_cow_size module_fork_last_cow_size metric
 redis_net_input_bytes_total net_input_bytes_total metric
 redis_net_output_bytes_total net_output_bytes_total metric
 redis_number_of_cached_scripts number_of_cached_scripts metric
 redis_process_id process_id metric
 redis_pubsub_channels pubsub_channels metric
 redis_pubsub_patterns pubsub_patterns metric
 redis_rdb_bgsave_in_progress rdb_bgsave_in_progress metric
 redis_rdb_changes_since_last_save rdb_changes_since_last_save metric
 redis_rdb_current_bgsave_duration_sec rdb_current_bgsave_duration_sec metric
 redis_rdb_last_bgsave_duration_sec rdb_last_bgsave_duration_sec metric
 redis_rdb_last_bgsave_status rdb_last_bgsave_status metric
 redis_rdb_last_cow_size_bytes rdb_last_cow_size_bytes metric
 redis_rdb_last_save_timestamp_seconds rdb_last_save_timestamp_seconds metric
 redis_rejected_connections_total rejected_connections_total metric
 redis_repl_backlog_first_byte_offset repl_backlog_first_byte_offset metric
 redis_repl_backlog_history_bytes repl_backlog_history_bytes metric
 redis_repl_backlog_is_active repl_backlog_is_active metric
 redis_replica_partial_resync_accepted replica_partial_resync_accepted metric
 redis_replica_partial_resync_denied replica_partial_resync_denied metric
 redis_replica_resyncs_full replica_resyncs_full metric
 redis_replication_backlog_bytes replication_backlog_bytes metric
 redis_second_repl_offset second_repl_offset metric
 redis_slave_expires_tracked_keys slave_expires_tracked_keys metric
 redis_slowlog_last_id Last id of slowlog
 redis_slowlog_length Total slowlog
 redis_start_time_seconds Start time of the Redis instance since unix epoch in seconds.
 redis_target_scrape_request_errors_total Errors in requests to the exporter
 redis_total_error_replies total_error_replies metric
 redis_total_reads_processed total_reads_processed metric
 redis_total_writes_processed total_writes_processed metric
 redis_tracking_clients tracking_clients metric
 redis_tracking_total_items tracking_total_items metric
 redis_tracking_total_keys tracking_total_keys metric
 redis_tracking_total_prefixes tracking_total_prefixes metric
 redis_unexpected_error_replies unexpected_error_replies metric
 redis_up Information about the Redis instance
 redis_uptime_in_seconds uptime_in_seconds metric


Check metrics with redis command

bash
# redis client 접속
kubectl exec -it -n redis redis-redis-cluster-0 -- redis-cli
# ν•΄λ‹Ή μ»€λ§¨λ“œλ₯Ό μž…λ ₯ν•˜λ©΄ ν”„λ‘œλ©”ν…Œμš°μŠ€μ—μ„œ μ‚¬μš©ν•  수 μžˆλŠ” 쿼리의 λŒ€λž΅μ μΈ ν‚€μ›Œλ“œλ₯Ό 얻을 수 μžˆλ‹€.
redis-cli> info all

# ν΄λΌμ΄μ–ΈνŠΈ μ™ΈλΆ€μ—μ„œ μ‹€ν–‰
kubectl exec -n redis redis-redis-cluster-0 -it -- redis-cli info all


(old) Values.yaml Overriding

bash
--set metrics.enabled=true

RabbitMQ Manifest
References and Guidelines (NOTE)

Name: RabbitMQ
Deployed: k8s, argo-cd, bitnami


Argo-cd Manifest, Enable Exporter

yaml
project: default
source:
  repoURL: 'https://charts.bitnami.com/bitnami'
  targetRevision: 11.1.5
  helm:
    valueFiles:
      - values.yaml
    parameters:
      - name: global.storageClass
        value: cmp
      - name: auth.password
        value: {{ password }}
      - name: auth.username
        value: admin
      - name: ingress.enabled
        value: 'true'
      - name: ingress.hostname
        value: rabbitmq.dev.Enterprise Cloud.cld
      - name: ingress.ingressClassName
        value: kong
      - name: service.type
        value: NodePort
      - name: metrics.enabled
        value: 'true'
  chart: rabbitmq
destination:
  server: 'https://kubernetes.default.svc'
  namespace: rabbitmq


rabbitmq-exporter (:9419)

txt
erlang_mnesia_held_locks Number of held locks.
 erlang_mnesia_lock_queue Number of transactions waiting for a lock.
 erlang_mnesia_transaction_participants Number of participant transactions.
 erlang_mnesia_transaction_coordinators Number of coordinator transactions.
 erlang_mnesia_failed_transactions Number of failed (i.e. aborted) transactions.
 erlang_mnesia_committed_transactions Number of committed transactions.
 erlang_mnesia_logged_transactions Number of transactions logged.
 erlang_mnesia_restarted_transactions Total number of transaction restarts.
 erlang_mnesia_memory_usage_bytes Total number of bytes allocated by all mnesia tables
 erlang_mnesia_tablewise_memory_usage_bytes Number of bytes allocated per mnesia table
 erlang_mnesia_tablewise_size Number of rows present per table
 process_start_time_seconds Start time of the process since unix epoch in seconds.
 rabbitmq_alarms_file_descriptor_limit is 1 if file descriptor limit alarm is in effect
 rabbitmq_alarms_free_disk_space_watermark is 1 if free disk space watermark alarm is in effect
 rabbitmq_alarms_memory_used_watermark is 1 if VM memory watermark alarm is in effect
 rabbitmq_connections_opened_total Total number of connections opened
 rabbitmq_connections_closed_total Total number of connections closed or terminated
 rabbitmq_channels_opened_total Total number of channels opened
 rabbitmq_channels_closed_total Total number of channels closed
 rabbitmq_queues_declared_total Total number of queues declared
 rabbitmq_queues_created_total Total number of queues created
 rabbitmq_queues_deleted_total Total number of queues deleted
 rabbitmq_process_open_fds Open file descriptors
 rabbitmq_process_open_tcp_sockets Open TCP sockets
 rabbitmq_process_resident_memory_bytes Memory used in bytes
 rabbitmq_disk_space_available_bytes Disk space available in bytes
 rabbitmq_erlang_processes_used Erlang processes used
 rabbitmq_erlang_gc_runs_total Total number of Erlang garbage collector runs
 rabbitmq_erlang_gc_reclaimed_bytes_total Total number of bytes of memory reclaimed by Erlang garbage collector
 rabbitmq_erlang_scheduler_context_switches_total Total number of Erlang scheduler context switches
 rabbitmq_process_max_fds Open file descriptors limit
 rabbitmq_process_max_tcp_sockets Open TCP sockets limit
 rabbitmq_resident_memory_limit_bytes Memory high watermark in bytes
 rabbitmq_disk_space_available_limit_bytes Free disk space low watermark in bytes
 rabbitmq_erlang_processes_limit Erlang processes limit
 rabbitmq_erlang_scheduler_run_queue Erlang scheduler run queue
 rabbitmq_erlang_net_ticktime_seconds Inter-node heartbeat interval
 rabbitmq_erlang_uptime_seconds Node uptime
 rabbitmq_io_read_ops_total Total number of I/O read operations
 rabbitmq_io_read_bytes_total Total number of I/O bytes read
 rabbitmq_io_write_ops_total Total number of I/O write operations
 rabbitmq_io_write_bytes_total Total number of I/O bytes written
 rabbitmq_io_sync_ops_total Total number of I/O sync operations
 rabbitmq_io_seek_ops_total Total number of I/O seek operations
 rabbitmq_io_reopen_ops_total Total number of times files have been reopened
 rabbitmq_schema_db_ram_tx_total Total number of Schema DB memory transactions
 rabbitmq_schema_db_disk_tx_total Total number of Schema DB disk transactions
 rabbitmq_msg_store_read_total Total number of Message Store read operations
 rabbitmq_msg_store_write_total Total number of Message Store write operations
 rabbitmq_queue_index_read_ops_total Total number of Queue Index read operations
 rabbitmq_queue_index_write_ops_total Total number of Queue Index write operations
 rabbitmq_io_read_time_seconds_total Total I/O read time
 rabbitmq_io_write_time_seconds_total Total I/O write time
 rabbitmq_io_sync_time_seconds_total Total I/O sync time
 rabbitmq_io_seek_time_seconds_total Total I/O seek time
 rabbitmq_raft_term_total Current Raft term number
 rabbitmq_raft_log_snapshot_index Raft log snapshot index
 rabbitmq_raft_log_last_applied_index Raft log last applied index
 rabbitmq_raft_log_commit_index Raft log commit index
 rabbitmq_raft_log_last_written_index Raft log last written index
 rabbitmq_raft_entry_commit_latency_seconds Time taken for a log entry to be committed
 rabbitmq_auth_attempts_total Total number of authorization attempts
 rabbitmq_auth_attempts_succeeded_total Total number of successful authentication attempts
 rabbitmq_auth_attempts_failed_total Total number of failed authentication attempts
 rabbitmq_queue_messages_ready Messages ready to be delivered to consumers
 rabbitmq_queue_messages_unacked Messages delivered to consumers but not yet acknowledged
 rabbitmq_queue_messages Sum of ready and unacknowledged messages - total queue depth
 rabbitmq_queue_process_reductions_total Total number of queue process reductions
 rabbitmq_queue_consumers Consumers on a queue
 rabbitmq_queue_consumer_capacity Consumer capacity
 rabbitmq_queue_consumer_utilisation Same as consumer capacity
 rabbitmq_queue_process_memory_bytes Memory in bytes used by the Erlang queue process
 rabbitmq_queue_messages_ram Ready and unacknowledged messages stored in memory
 rabbitmq_queue_messages_ram_bytes Size of ready and unacknowledged messages stored in memory
 rabbitmq_queue_messages_ready_ram Ready messages stored in memory
 rabbitmq_queue_messages_unacked_ram Unacknowledged messages stored in memory
 rabbitmq_queue_messages_persistent Persistent messages
 rabbitmq_queue_messages_persistent_bytes Size in bytes of persistent messages
 rabbitmq_queue_messages_bytes Size in bytes of ready and unacknowledged messages
 rabbitmq_queue_messages_ready_bytes Size in bytes of ready messages
 rabbitmq_queue_messages_unacked_bytes Size in bytes of all unacknowledged messages
 rabbitmq_queue_messages_paged_out Messages paged out to disk
 rabbitmq_queue_messages_paged_out_bytes Size in bytes of messages paged out to disk
 rabbitmq_queue_head_message_timestamp Timestamp of the first message in the queue, if any
 rabbitmq_queue_disk_reads_total Total number of times queue read messages from disk
 rabbitmq_queue_disk_writes_total Total number of times queue wrote messages to disk
 rabbitmq_channel_consumers Consumers on a channel
 rabbitmq_channel_messages_unacked Delivered but not yet acknowledged messages
 rabbitmq_channel_messages_unconfirmed Published but not yet confirmed messages
 rabbitmq_channel_messages_uncommitted Messages received in a transaction but not yet committed
 rabbitmq_channel_acks_uncommitted Message acknowledgements in a transaction not yet committed
 rabbitmq_consumer_prefetch Limit of unacknowledged messages for each consumer
 rabbitmq_channel_prefetch Total limit of unacknowledged messages for all consumers on a channel
 rabbitmq_channel_messages_published_total Total number of messages published into an exchange on a channel
 rabbitmq_channel_messages_confirmed_total Total number of messages published into an exchange and confirmed on the channel
 rabbitmq_channel_messages_unroutable_returned_total Total number of messages published as mandatory into an exchange and returned to the publisher as unroutable
 rabbitmq_channel_messages_unroutable_dropped_total Total number of messages published as non-mandatory into an exchange and dropped as unroutable
 rabbitmq_channel_process_reductions_total Total number of channel process reductions
 rabbitmq_channel_get_ack_total Total number of messages fetched with basic.get in manual acknowledgement mode
 rabbitmq_channel_get_total Total number of messages fetched with basic.get in automatic acknowledgement mode
 rabbitmq_channel_messages_delivered_ack_total Total number of messages delivered to consumers in manual acknowledgement mode
 rabbitmq_channel_messages_delivered_total Total number of messages delivered to consumers in automatic acknowledgement mode
 rabbitmq_channel_messages_redelivered_total Total number of messages redelivered to consumers
 rabbitmq_channel_messages_acked_total Total number of messages acknowledged by consumers
 rabbitmq_channel_get_empty_total Total number of times basic.get operations fetched no message
 rabbitmq_connection_incoming_bytes_total Total number of bytes received on a connection
 rabbitmq_connection_outgoing_bytes_total Total number of bytes sent on a connection
 rabbitmq_connection_process_reductions_total Total number of connection process reductions
 rabbitmq_connection_incoming_packets_total Total number of packets received on a connection
 rabbitmq_connection_outgoing_packets_total Total number of packets sent on a connection
 rabbitmq_connection_pending_packets Number of packets waiting to be sent on a connection
 rabbitmq_connection_channels Channels on a connection
 rabbitmq_queue_messages_published_total Total number of messages published to queues
 rabbitmq_connections Connections currently open
 rabbitmq_channels Channels currently open
 rabbitmq_consumers Consumers currently connected
 rabbitmq_queues Queues available
 rabbitmq_build_info RabbitMQ & Erlang/OTP version info
 rabbitmq_identity_info RabbitMQ node & cluster identity info
 rabbitmq_global_consumers Current number of consumers
 rabbitmq_global_messages_acknowledged_total Total number of messages acknowledged by consumers
 rabbitmq_global_messages_confirmed_total Total number of messages confirmed to publishers
 rabbitmq_global_messages_dead_lettered_confirmed_total Total number of messages dead-lettered and confirmed by target queues
 rabbitmq_global_messages_dead_lettered_delivery_limit_total Total number of messages dead-lettered due to delivery-limit exceeded
 rabbitmq_global_messages_dead_lettered_expired_total Total number of messages dead-lettered due to message TTL exceeded
 rabbitmq_global_messages_dead_lettered_maxlen_total Total number of messages dead-lettered due to overflow drop-head or reject-publish-dlx
 rabbitmq_global_messages_dead_lettered_rejected_total Total number of messages dead-lettered due to basic.reject or basic.nack
 rabbitmq_global_messages_delivered_consume_auto_ack_total Total number of messages delivered to consumers using basic.consume with automatic acknowledgment
 rabbitmq_global_messages_delivered_consume_manual_ack_total Total number of messages delivered to consumers using basic.consume with manual acknowledgment
 rabbitmq_global_messages_delivered_get_auto_ack_total Total number of messages delivered to consumers using basic.get with automatic acknowledgment
 rabbitmq_global_messages_delivered_get_manual_ack_total Total number of messages delivered to consumers using basic.get with manual acknowledgment
 rabbitmq_global_messages_delivered_total Total number of messages delivered to consumers
 rabbitmq_global_messages_get_empty_total Total number of times basic.get operations fetched no message
 rabbitmq_global_messages_received_confirm_total Total number of messages received from publishers expecting confirmations
 rabbitmq_global_messages_received_total Total number of messages received from publishers
 rabbitmq_global_messages_redelivered_total Total number of messages redelivered to consumers
 rabbitmq_global_messages_routed_total Total number of messages routed to queues or streams
 rabbitmq_global_messages_unroutable_dropped_total Total number of messages published as non-mandatory into an exchange and dropped as unroutable
 rabbitmq_global_messages_unroutable_returned_total Total number of messages published as mandatory into an exchange and returned to the publisher as unroutable
 rabbitmq_global_publishers Current number of publishers
 telemetry_scrape_duration_seconds Scrape duration
 telemetry_scrape_encoded_size_bytes Scrape size, encoded
 telemetry_scrape_size_bytes Scrape size, not encoded
 erlang_vm_dist_recv_bytes Number of bytes received by the socket.
 erlang_vm_dist_recv_cnt Number of packets received by the socket.
 erlang_vm_dist_recv_max_bytes Size of the largest packet, in bytes, received by the socket.
 erlang_vm_dist_recv_avg_bytes Average size of packets, in bytes, received by the socket.
 erlang_vm_dist_recv_dvi_bytes Average packet size deviation, in bytes, received by the socket.
 erlang_vm_dist_send_bytes Number of bytes sent from the socket.
 erlang_vm_dist_send_cnt Number of packets sent from the socket.
 erlang_vm_dist_send_max_bytes Size of the largest packet, in bytes, sent from the socket.
 erlang_vm_dist_send_avg_bytes Average size of packets, in bytes, sent from the socket.
 erlang_vm_dist_send_pend_bytes Number of bytes waiting to be sent by the socket.
 erlang_vm_dist_port_input_bytes The total number of bytes read from the port.
 erlang_vm_dist_port_output_bytes The total number of bytes written to the port.
 erlang_vm_dist_port_memory_bytes The total number of bytes allocated for this port by the runtime system. The port itself can have allocated memory that is not included.
 erlang_vm_dist_port_queue_size_bytes The total number of bytes queued by the port using the ERTS driver queue implementation.
 erlang_vm_dist_proc_memory_bytes The size in bytes of the process. This includes call stack, heap, and internal structures.
 erlang_vm_dist_proc_heap_size_words The size in words of the youngest heap generation of the process. This generation includes the process stack. This information is highly implementation-dependent, and can change if the implementation changes.
 erlang_vm_dist_proc_min_heap_size_words The minimum heap size for the process.
 erlang_vm_dist_proc_min_bin_vheap_size_words The minimum binary virtual heap size for the process.
 erlang_vm_dist_proc_stack_size_words The stack size, in words, of the process.
 erlang_vm_dist_proc_total_heap_size_words The total size, in words, of all heap fragments of the process. This includes the process stack and any unreceived messages that are considered to be part of the heap.
 erlang_vm_dist_proc_message_queue_len The number of messages currently in the message queue of the process.
 erlang_vm_dist_proc_reductions The number of reductions executed by the process.
 erlang_vm_dist_proc_status The current status of the distribution process. The status is represented as a numerical value where `exiting=1', `suspended=2', `runnable=3', `garbage_collecting=4', `running=5' and `waiting=6'.
 erlang_vm_dist_node_state The current state of the distribution link. The state is represented as a numerical value where `pending=1', `up_pending=2' and `up=3'.
 erlang_vm_dist_node_queue_size_bytes The number of bytes in the output distribution queue. This queue sits between the Erlang code and the port driver.
 erlang_vm_memory_atom_bytes_total The total amount of memory currently allocated for atoms. This memory is part of the memory presented as system memory.
 erlang_vm_memory_bytes_total The total amount of memory currently allocated. This is the same as the sum of the memory size for processes and system.
 erlang_vm_memory_dets_tables Erlang VM DETS Tables count.
 erlang_vm_memory_ets_tables Erlang VM ETS Tables count.
 erlang_vm_memory_processes_bytes_total The total amount of memory currently allocated for the Erlang processes.
 erlang_vm_memory_system_bytes_total The total amount of memory currently allocated for the emulator that is not directly related to any Erlang process. Memory presented as processes is not included in this memory.
 erlang_vm_msacc_aux_seconds_total Total time in seconds spent handling auxiliary jobs.
 erlang_vm_msacc_check_io_seconds_total Total time in seconds spent checking for new I/O events.
 erlang_vm_msacc_emulator_seconds_total Total time in seconds spent executing Erlang processes.
 erlang_vm_msacc_gc_seconds_total Total time in seconds spent doing garbage collection. When extra states are enabled this is the time spent doing non-fullsweep garbage collections.
 erlang_vm_msacc_other_seconds_total Total time in seconds spent doing unaccounted things.
 erlang_vm_msacc_port_seconds_total Total time in seconds spent executing ports.
 erlang_vm_msacc_sleep_seconds_total Total time in seconds spent sleeping.
 erlang_vm_msacc_alloc_seconds_total Total time in seconds spent managing memory. Without extra states this time is spread out over all other states.
 erlang_vm_msacc_bif_seconds_total Total time in seconds spent in BIFs. Without extra states this time is part of the 'emulator' state.
 erlang_vm_msacc_busy_wait_seconds_total Total time in seconds spent busy waiting. Without extra states this time is part of the 'other' state.
 erlang_vm_msacc_ets_seconds_total Total time in seconds spent executing ETS BIFs. Without extra states this time is part of the 'emulator' state.
 erlang_vm_msacc_gc_full_seconds_total Total time in seconds spent doing fullsweep garbage collection. Without extra states this time is part of the 'gc' state.
 erlang_vm_msacc_nif_seconds_total Total time in seconds spent in NIFs. Without extra states this time is part of the 'emulator' state.
 erlang_vm_msacc_send_seconds_total Total time in seconds spent sending messages (processes only). Without extra states this time is part of the 'emulator' state.
 erlang_vm_msacc_timers_seconds_total Total time in seconds spent managing timers. Without extra states this time is part of the 'other' state.
 erlang_vm_statistics_bytes_output_total Total number of bytes output to ports.
 erlang_vm_statistics_bytes_received_total Total number of bytes received through ports.
 erlang_vm_statistics_context_switches Total number of context switches since the system started.
 erlang_vm_statistics_dirty_cpu_run_queue_length Length of the dirty CPU run-queue.
 erlang_vm_statistics_dirty_io_run_queue_length Length of the dirty IO run-queue.
 erlang_vm_statistics_garbage_collection_number_of_gcs Garbage collection: number of GCs.
 erlang_vm_statistics_garbage_collection_bytes_reclaimed Garbage collection: bytes reclaimed.
 erlang_vm_statistics_garbage_collection_words_reclaimed Garbage collection: words reclaimed.
 erlang_vm_statistics_reductions_total Total reductions.
 erlang_vm_statistics_run_queues_length Length of normal run-queues.
 erlang_vm_statistics_runtime_milliseconds The sum of the runtime for all threads in the Erlang runtime system. Can be greater than wall clock time.
 erlang_vm_statistics_wallclock_time_milliseconds Information about wall clock. Same as erlang_vm_statistics_runtime_milliseconds except that real time is measured.
 erlang_vm_dirty_cpu_schedulers The number of scheduler dirty CPU scheduler threads used by the emulator.
 erlang_vm_dirty_cpu_schedulers_online The number of dirty CPU scheduler threads online.
 erlang_vm_dirty_io_schedulers The number of scheduler dirty I/O scheduler threads used by the emulator.
 erlang_vm_ets_limit The maximum number of ETS tables allowed.
 erlang_vm_logical_processors The detected number of logical processors configured in the system.
 erlang_vm_logical_processors_available The detected number of logical processors available to the Erlang runtime system.
 erlang_vm_logical_processors_online The detected number of logical processors online on the system.
 erlang_vm_port_count The number of ports currently existing at the local node.
 erlang_vm_port_limit The maximum number of simultaneously existing ports at the local node.
 erlang_vm_process_count The number of processes currently existing at the local node.
 erlang_vm_process_limit The maximum number of simultaneously existing processes at the local node.
 erlang_vm_schedulers The number of scheduler threads used by the emulator.
 erlang_vm_schedulers_online The number of schedulers online.
 erlang_vm_smp_support 1 if the emulator has been compiled with SMP support, otherwise 0.
 erlang_vm_threads 1 if the emulator has been compiled with thread support, otherwise 0.
 erlang_vm_thread_pool_size The number of async threads in the async thread pool used for asynchronous driver calls.
 erlang_vm_time_correction 1 if time correction is enabled, otherwise 0.
 erlang_vm_wordsize_bytes The size of Erlang term words in bytes.
 erlang_vm_atom_count The number of atom currently existing at the local node.
 erlang_vm_atom_limit The maximum number of simultaneously existing atom at the local node.
 erlang_vm_allocators Allocated (carriers_size) and used (blocks_size) memory for the different allocators in the VM. See erts_alloc(3).

KeyCloak Manifest
References and Guidelines (NOTE)

Name: Keycloak
Deployed: k8s, argo-cd, bitnami


Argo-cd Manifest, Enable Exporter

yaml
project: default
source:
  repoURL: 'https://Enterprise [email protected]/Enterprise Cloudlab/cmp-helm.git'
  path: mdware-helm/charts/keycloak
  targetRevision: develop
  helm:
    valueFiles:
      - values.yaml
    parameters:
      - name: global.storageClass
        value: cmp
      - name: service.type
        value: NodePort
      - name: service.nodePorts.http
        value: '31080'
      - name: service.nodePorts.htts
        value: '31443'
      - name: ingress.enabled
        value: 'true'
      - name: ingress.ingressClassName
        value: kong
      - name: ingress.hostname
        value: keycloak.Enterprise Cloud.cld
      - name: postgresql.enabled
        value: 'true'
      - name: postgresql.auth.username
        value: keycloak
      - name: postgresql.auth.password
        value: {{ password }}
      - name: postgresql.auth.database
        value: keycloak
      - name: metrics.enabled
        value: 'true'
    releaseName: portal
    version: v3
destination:
  server: 'https://kubernetes.default.svc'
  namespace: keycloak


keycloak-exporter (:8080/metrics)

txt
base_classloader_loadedClasses_count Displays the number of classes that are currently loaded in the Java virtual machine.
 base_classloader_loadedClasses_total Displays the total number of classes that have been loaded since the Java virtual machine has started execution.
 base_classloader_unloadedClasses_total Displays the total number of classes unloaded since the Java virtual machine has started execution.
 base_cpu_availableProcessors Displays the number of processors available to the Java virtual machine. This value may change during a particular invocation of the virtual machine.
 base_cpu_processCpuLoad_percent Displays  the "recent cpu usage" for the Java Virtual Machine process. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that none of the CPUs were running threads from the JVM process during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running threads from the JVM 100% of the time during the recent period being observed. Threads from the JVM include the application threads as well as the JVM internal threads. All values betweens 0.0 and 1.0 are possible depending of the activities going on in the JVM process and the whole system. If the Java Virtual Machine recent CPU usage is not available, the method returns a negative value.
 base_cpu_systemLoadAverage Displays the system load average for the last minute. The system load average is the sum of the number of runnable entities queued to the available processors and the number of runnable entities running on the available processors averaged over a period of time. The way in which the load average is calculated is operating system specific but is typically a damped time-dependent average. If the load average is not available, a negative value is displayed. This attribute is designed to provide a hint about the system load and may be queried frequently. The load average may be unavailable on some platforms where it is expensive to implement this method.
 base_gc_time_total Displays the approximate accumulated collection elapsed time in milliseconds. This attribute displays -1 if the collection elapsed time is undefined for this collector. The Java virtual machine implementation may use a high resolution timer to measure the elapsed time. This attribute may display the same value even if the collection count has been incremented if the collection elapsed time is very short.
 base_gc_total Displays the total number of collections that have occurred. This attribute lists -1 if the collection count is undefined for this collector.
 base_jvm_uptime_seconds Displays the time from the start of the Java virtual machine in milliseconds.
 base_memory_committedHeap_bytes Displays the amount of memory in bytes that is committed for the Java virtual machine to use. This amount of memory is guaranteed for the Java virtual machine to use.
 base_memory_maxHeap_bytes Displays the maximum amount of heap memory in bytes that can be used for memory management. This attribute displays -1 if the maximum heap memory size is undefined. This amount of memory is not guaranteed to be available for memory management if it is greater than the amount of committed memory. The Java virtual machine may fail to allocate memory even if the amount of used memory does not exceed this maximum size.
 base_memory_usedHeap_bytes Displays the amount of used heap memory in bytes.
 base_thread_count Displays the current number of live threads including both daemon and non-daemon threads
 base_thread_daemon_count Displays the current number of live daemon threads.
 base_thread_max_count Displays the peak live thread count since the Java virtual machine started or peak was reset. This includes daemon and non-daemon threads.
 vendor_agroal_acquire_count_total Number of times an acquire operation succeeded.
 vendor_agroal_active_count Number of active connections. These connections are in use and not available to be acquired.
 vendor_agroal_available_count Number of idle connections in the pool, available to be acquired.
 vendor_agroal_awaiting_count Approximate number of threads blocked, waiting to acquire a connection.
 vendor_agroal_blocking_time_average_seconds Average time an application waited to acquire a connection.
 vendor_agroal_blocking_time_max_seconds Maximum time an application waited to acquire a connection.
 vendor_agroal_blocking_time_total_seconds Total time applications waited to acquire a connection.
 vendor_agroal_creation_count_total Number of created connections.
 vendor_agroal_creation_time_average_seconds Average time for a connection to be created.
 vendor_agroal_creation_time_max_seconds Maximum time for a connection to be created.
 vendor_agroal_creation_time_total_seconds Total time waiting for connections to be created.
 vendor_agroal_destroy_count_total Number of destroyed connections.
 vendor_agroal_flush_count_total Number of connections removed from the pool, not counting invalid / idle.
 vendor_agroal_invalid_count_total Number of connections removed from the pool for being idle.
 vendor_agroal_leak_detection_count_total Number of times a leak was detected. A single connection can be detected multiple times.
 vendor_agroal_max_used_count Maximum number of connections active simultaneously.
 vendor_agroal_reap_count_total Number of connections removed from the pool for being idle.
 vendor_cpu_processCpuTime_seconds Displays the CPU time used by the process on which the Java virtual machine is running in nanoseconds. The returned value is of nanoseconds precision but not necessarily nanoseconds accuracy. This method returns -1 if the the platform does not support this operation.
 vendor_cpu_systemCpuLoad_percent Displays the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values betweens 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
 vendor_memory_committedNonHeap_bytes Displays the amount of non heap memory in bytes that is committed for the Java virtual machine to use.
 vendor_memory_freePhysicalSize_bytes Displays the amount of free physical memory in bytes.
 vendor_memory_freeSwapSize_bytes Displays the amount of free swap space in bytes.
 vendor_memory_maxNonHeap_bytes Displays the maximum amount of used non-heap memory in bytes.
 vendor_memory_usedNonHeap_bytes Displays the amount of used non-heap memory in bytes.
 vendor_memoryPool_usage_bytes Current usage of the memory pool denoted by the 'name' tag
 vendor_memoryPool_usage_max_bytes Peak usage of the memory pool denoted by the 'name' tag

Grafana Manifest
References and Guidelines (NOTE)

Name: Grafana
Deployed: k8s, argo-cd, bitnami


Argo-cd Manifest, Enable Exporter

yaml
project: default
source:
  repoURL: 'https://charts.bitnami.com/bitnami'
  targetRevision: 8.2.20
  helm:
    valueFiles:
      - values.yaml
    parameters:
      - name: global.storageClass
        value: cmp
      - name: admin.user
        value: admin
      - name: admin.password
        value: {{ password }}
      - name: service.type
        value: NodePort
      - name: service.nodePorts.grafana
        value: '30009'
      - name: metrics.enabled
        value: 'true'
  chart: grafana
destination:
  server: 'https://kubernetes.default.svc'
  namespace: grafana-test


grafana-exporter (:3000/metrics)

txt
access_evaluation_duration Histogram for the runtime of evaluation function.
 access_permissions_duration Histogram for the runtime of permissions check function.
 cortex_experimental_features_in_use_total The number of experimental features in use.
 deprecated_flags_inuse_total The number of deprecated flags currently set.
 go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
 go_goroutines Number of goroutines that currently exist.
 go_info Information about the Go environment.
 go_memstats_alloc_bytes Number of bytes allocated and still in use.
 go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
 go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
 go_memstats_frees_total Total number of frees.
 go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
 go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
 go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
 go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
 go_memstats_heap_objects Number of allocated objects.
 go_memstats_heap_released_bytes Number of heap bytes released to OS.
 go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
 go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
 go_memstats_lookups_total Total number of pointer lookups.
 go_memstats_mallocs_total Total number of mallocs.
 go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
 go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
 go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
 go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
 go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
 go_memstats_other_sys_bytes Number of bytes used for other system allocations.
 go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
 go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
 go_memstats_sys_bytes Number of bytes obtained from system.
 go_sql_stats_connections_blocked_seconds The total time blocked waiting for a new connection.
 go_sql_stats_connections_closed_max_idle The total number of connections closed due to SetMaxIdleConns.
 go_sql_stats_connections_closed_max_idle_time The total number of connections closed due to SetConnMaxIdleTime.
 go_sql_stats_connections_closed_max_lifetime The total number of connections closed due to SetConnMaxLifetime.
 go_sql_stats_connections_idle The number of idle connections.
 go_sql_stats_connections_in_use The number of connections currently in use.
 go_sql_stats_connections_max_open Maximum number of open connections to the database.
 go_sql_stats_connections_open The number of established connections both in use and idle.
 go_sql_stats_connections_waited_for The total number of connections waited for.
 go_threads Number of OS threads created.
 grafana_access_evaluation_count number of evaluation calls
 grafana_alerting_active_alerts amount of active alerts
 grafana_alerting_active_configurations The number of active Alertmanager configurations.
 grafana_alerting_alerts How many alerts by state.
 grafana_alerting_discovered_configurations The number of organizations we've discovered that require an Alertmanager configuration.
 grafana_alerting_execution_time_milliseconds summary of alert execution duration
 grafana_alerting_schedule_alert_rules The number of alert rules that could be considered for evaluation at the next tick.
 grafana_alerting_schedule_alert_rules_hash A hash of the alert rules that could be considered for evaluation at the next tick.
 grafana_alerting_schedule_periodic_duration_seconds The time taken to run the scheduler.
 grafana_alerting_schedule_query_alert_rules_duration_seconds The time taken to fetch alert rules from the database.
 grafana_alerting_scheduler_behind_seconds The total number of seconds the scheduler is behind.
 grafana_alerting_ticker_interval_seconds Interval at which the ticker is meant to tick.
 grafana_alerting_ticker_last_consumed_tick_timestamp_seconds Timestamp of the last consumed tick in seconds.
 grafana_alerting_ticker_next_tick_timestamp_seconds Timestamp of the next tick in seconds before it is consumed.
 grafana_api_admin_user_created_total api admin user created counter
 grafana_api_dashboard_get_milliseconds summary for dashboard get duration
 grafana_api_dashboard_save_milliseconds summary for dashboard save duration
 grafana_api_dashboard_search_milliseconds summary for dashboard search duration
 grafana_api_dashboard_snapshot_create_total dashboard snapshots created
 grafana_api_dashboard_snapshot_external_total external dashboard snapshots created
 grafana_api_dashboard_snapshot_get_total loaded dashboards
 grafana_api_dataproxy_request_all_milliseconds summary for dataproxy request duration
 grafana_api_login_oauth_total api login oauth counter
 grafana_api_login_post_total api login post counter
 grafana_api_login_saml_total api login saml counter
 grafana_api_models_dashboard_insert_total dashboards inserted 
 grafana_api_org_create_total api org created counter
 grafana_api_response_status_total api http response status
 grafana_api_user_signup_completed_total amount of users who completed the signup flow
 grafana_api_user_signup_invite_total amount of users who have been invited
 grafana_api_user_signup_started_total amount of users who started the signup flow
 grafana_aws_cloudwatch_get_metric_data_total counter for getting metric data time series from aws
 grafana_aws_cloudwatch_get_metric_statistics_total counter for getting metric statistics from aws
 grafana_aws_cloudwatch_list_metrics_total counter for getting list of metrics from aws
 grafana_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which Grafana was built
 grafana_build_timestamp A metric exposing when the binary was built in epoch
 grafana_database_conn_idle The number of idle connections
 grafana_database_conn_in_use The number of connections currently in use
 grafana_database_conn_max_idle_closed_seconds The total number of connections closed due to SetConnMaxIdleTime
 grafana_database_conn_max_idle_closed_total The total number of connections closed due to SetMaxIdleConns
 grafana_database_conn_max_lifetime_closed_total The total number of connections closed due to SetConnMaxLifetime
 grafana_database_conn_max_open Maximum number of open connections to the database
 grafana_database_conn_open The number of established connections both in use and idle
 grafana_database_conn_wait_count_total The total number of connections waited for
 grafana_database_conn_wait_duration_seconds The total time blocked waiting for a new connection
 grafana_db_datasource_query_by_id_total counter for getting datasource by id
 grafana_emails_sent_failed Number of emails Grafana failed to send
 grafana_emails_sent_total Number of emails sent by Grafana
 grafana_encryption_cache_reads_total A counter for encryption cache reads
 grafana_encryption_ops_total A counter for encryption operations
 grafana_feature_toggles_info info metric that exposes what feature toggles are enabled or not
 grafana_frontend_boot_css_time_seconds Frontend boot initial css load
 grafana_frontend_boot_first_contentful_paint_time_seconds Frontend boot first contentful paint
 grafana_frontend_boot_first_paint_time_seconds Frontend boot first paint
 grafana_frontend_boot_js_done_time_seconds Frontend boot initial js load
 grafana_frontend_boot_load_time_seconds Frontend boot time measurement
 grafana_http_request_duration_seconds Histogram of latencies for HTTP requests.
 grafana_http_request_in_flight A gauge of requests currently being served by Grafana.
 grafana_instance_start_total counter for started instances
 grafana_ldap_users_sync_execution_time summary for LDAP users sync execution duration
 grafana_live_client_command_duration_seconds Client command duration summary.
 grafana_live_client_recover Count of recover operations.
 grafana_live_node_action_count Number of node actions called.
 grafana_live_node_build Node build info.
 grafana_live_node_messages_received_count Number of messages received.
 grafana_live_node_messages_sent_count Number of messages sent.
 grafana_live_node_num_channels Number of channels with one or more subscribers.
 grafana_live_node_num_clients Number of clients connected.
 grafana_live_node_num_nodes Number of nodes in cluster.
 grafana_live_node_num_subscriptions Number of subscriptions.
 grafana_live_node_num_users Number of unique users connected.
 grafana_live_transport_connect_count Number of connections to specific transport.
 grafana_live_transport_messages_sent Number of messages sent over specific transport.
 grafana_page_response_status_total page http response status
 grafana_plugin_build_info A metric with a constant '1' value labeled by pluginId, pluginType and version from which Grafana plugin was built
 grafana_proxy_response_status_total proxy http response status
 grafana_public_dashboard_request_count counter for public dashboards requests
 grafana_rendering_queue_size size of rendering queue
 grafana_search_dashboard_search_failures_duration_seconds 
 grafana_search_dashboard_search_successes_duration_seconds 
 grafana_stat_active_users number of active users
 grafana_stat_total_orgs total amount of orgs
 grafana_stat_total_playlists total amount of playlists
 grafana_stat_total_service_account_tokens total amount of service account tokens
 grafana_stat_total_service_accounts total amount of service accounts
 grafana_stat_total_users total amount of users
 grafana_stat_totals_active_admins total amount of active admins
 grafana_stat_totals_active_editors total amount of active editors
 grafana_stat_totals_active_viewers total amount of viewers
 grafana_stat_totals_admins total amount of admins
 grafana_stat_totals_annotations total amount of annotations in the database
 grafana_stat_totals_dashboard total amount of dashboards
 grafana_stat_totals_dashboard_versions total amount of dashboard versions in the database
 grafana_stat_totals_data_keys total amount of data keys in the database
 grafana_stat_totals_editors total amount of editors
 grafana_stat_totals_folder total amount of folders
 grafana_stat_totals_library_panels total amount of library panels in the database
 grafana_stat_totals_library_variables total amount of library variables in the database
 grafana_stat_totals_public_dashboard total amount of public dashboards
 grafana_stat_totals_viewers total amount of viewers
 grafana_stat_users_has_duplicate_user_entries instance has duplicate user entries by email or login
 grafana_stat_users_total_duplicate_user_entries total number of duplicate user entries by email or login
 grafana_stat_users_total_mixed_cased_users total number of users with upper and lower case logins or emails
 process_cpu_seconds_total Total user and system CPU time spent in seconds.
 process_max_fds Maximum number of open file descriptors.
 process_open_fds Number of open file descriptors.
 process_resident_memory_bytes Resident memory size in bytes.
 process_start_time_seconds Start time of the process since unix epoch in seconds.
 process_virtual_memory_bytes Virtual memory size in bytes.
 process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
 prometheus_template_text_expansion_failures_total The total number of template text expansion failures.
 prometheus_template_text_expansions_total The total number of template text expansions.

Enable HAProxy Exporter

OpenStack Exporter Specification


openstack-exporter (:9180/metrics) β†’ Scrape speed issue (takes more than 10 seconds)

txt
openstack_identity_domains domains
 openstack_identity_groups groups
 openstack_identity_project_info project_info
 openstack_identity_projects projects
 openstack_identity_regions regions
 openstack_identity_up up
 openstack_identity_users users
 openstack_loadbalancer_amphora_status amphora_status
 openstack_loadbalancer_loadbalancer_status loadbalancer_status
 openstack_loadbalancer_total_amphorae total_amphorae
 openstack_loadbalancer_total_loadbalancers total_loadbalancers
 openstack_loadbalancer_up up
 openstack_metric_collect_seconds Time needed to collect metric from OpenStack API
 openstack_neutron_agent_state agent_state
 openstack_neutron_floating_ip floating_ip
 openstack_neutron_floating_ips floating_ips
 openstack_neutron_floating_ips_associated_not_active floating_ips_associated_not_active
 openstack_neutron_l3_agent_of_router l3_agent_of_router
 openstack_neutron_network_ip_availabilities_total network_ip_availabilities_total
 openstack_neutron_network_ip_availabilities_used network_ip_availabilities_used
 openstack_neutron_networks networks
 openstack_neutron_port port
 openstack_neutron_ports ports
 openstack_neutron_ports_lb_not_active ports_lb_not_active
 openstack_neutron_ports_no_ips ports_no_ips
 openstack_neutron_router router
 openstack_neutron_routers routers
 openstack_neutron_routers_not_active routers_not_active
 openstack_neutron_security_groups security_groups
 openstack_neutron_subnets subnets
 openstack_neutron_up up
 openstack_nova_agent_state agent_state
 openstack_nova_availability_zones availability_zones
 openstack_nova_current_workload current_workload
 openstack_nova_flavors flavors
 openstack_nova_free_disk_bytes free_disk_bytes
 openstack_nova_local_storage_available_bytes local_storage_available_bytes
 openstack_nova_local_storage_used_bytes local_storage_used_bytes
 openstack_nova_memory_available_bytes memory_available_bytes
 openstack_nova_memory_used_bytes memory_used_bytes
 openstack_nova_running_vms running_vms
 openstack_nova_server_status server_status
 openstack_nova_total_vms total_vms
 openstack_nova_up up
 openstack_nova_vcpus_available vcpus_available
 openstack_nova_vcpus_used vcpus_used


openstack-node-exporter (:9100/metrics)

txt
go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
 go_goroutines Number of goroutines that currently exist.
 go_info Information about the Go environment.
 go_memstats_alloc_bytes Number of bytes allocated and still in use.
 go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
 go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
 go_memstats_frees_total Total number of frees.
 go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
 go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
 go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
 go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
 go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
 go_memstats_heap_objects Number of allocated objects.
 go_memstats_heap_released_bytes Number of heap bytes released to OS.
 go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
 go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
 go_memstats_lookups_total Total number of pointer lookups.
 go_memstats_mallocs_total Total number of mallocs.
 go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
 go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
 go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
 go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
 go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
 go_memstats_other_sys_bytes Number of bytes used for other system allocations.
 go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
 go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
 go_memstats_sys_bytes Number of bytes obtained from system.
 go_threads Number of OS threads created.
 node_arp_entries ARP entries by device
 node_boot_time_seconds Node boot time, in unixtime.
 node_context_switches_total Total number of context switches.
 node_cooling_device_cur_state Current throttle state of the cooling device
 node_cooling_device_max_state Maximum throttle state of the cooling device
 node_cpu_core_throttles_total Number of times this CPU core has been throttled.
 node_cpu_frequency_max_hertz Maximum cpu thread frequency in hertz.
 node_cpu_frequency_min_hertz Minimum cpu thread frequency in hertz.
 node_cpu_guest_seconds_total Seconds the CPUs spent in guests (VMs) for each mode.
 node_cpu_package_throttles_total Number of times this CPU package has been throttled.
 node_cpu_scaling_frequency_hertz Current scaled CPU thread frequency in hertz.
 node_cpu_scaling_frequency_max_hertz Maximum scaled CPU thread frequency in hertz.
 node_cpu_scaling_frequency_min_hertz Minimum scaled CPU thread frequency in hertz.
 node_cpu_seconds_total Seconds the CPUs spent in each mode.
 node_disk_discard_time_seconds_total This is the total number of seconds spent by all discards.
 node_disk_discarded_sectors_total The total number of sectors discarded successfully.
 node_disk_discards_completed_total The total number of discards completed successfully.
 node_disk_discards_merged_total The total number of discards merged.
 node_disk_info Info of /sys/block/<block_device>.
 node_disk_io_now The number of I/Os currently in progress.
 node_disk_io_time_seconds_total Total seconds spent doing I/Os.
 node_disk_io_time_weighted_seconds_total The weighted # of seconds spent doing I/Os.
 node_disk_read_bytes_total The total number of bytes read successfully.
 node_disk_read_time_seconds_total The total number of seconds spent by all reads.
 node_disk_reads_completed_total The total number of reads completed successfully.
 node_disk_reads_merged_total The total number of reads merged.
 node_disk_write_time_seconds_total This is the total number of seconds spent by all writes.
 node_disk_writes_completed_total The total number of writes completed successfully.
 node_disk_writes_merged_total The number of writes merged.
 node_disk_written_bytes_total The total number of bytes written successfully.
 node_dmi_info A metric with a constant '1' value labeled by bios_date, bios_release, bios_vendor, bios_version, board_asset_tag, board_name, board_serial, board_vendor, board_version, chassis_asset_tag, chassis_serial, chassis_vendor, chassis_version, product_family, product_name, product_serial, product_sku, product_uuid, product_version, system_vendor if provided by DMI.
 node_edac_correctable_errors_total Total correctable memory errors.
 node_edac_csrow_correctable_errors_total Total correctable memory errors for this csrow.
 node_edac_csrow_uncorrectable_errors_total Total uncorrectable memory errors for this csrow.
 node_edac_uncorrectable_errors_total Total uncorrectable memory errors.
 node_entropy_available_bits Bits of available entropy.
 node_entropy_pool_size_bits Bits of entropy pool.
 node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built.
 node_filefd_allocated File descriptor statistics: allocated.
 node_filefd_maximum File descriptor statistics: maximum.
 node_filesystem_avail_bytes Filesystem space available to non-root users in bytes.
 node_filesystem_device_error Whether an error occurred while getting statistics for the given device.
 node_filesystem_files Filesystem total file nodes.
 node_filesystem_files_free Filesystem total free file nodes.
 node_filesystem_free_bytes Filesystem free space in bytes.
 node_filesystem_readonly Filesystem read-only status.
 node_filesystem_size_bytes Filesystem size in bytes.
 node_forks_total Total number of forks.
 node_hwmon_chip_names Annotation metric for human-readable chip names
 node_hwmon_sensor_label Label for given chip and sensor
 node_hwmon_temp_celsius Hardware monitor for temperature (input)
 node_hwmon_temp_crit_alarm_celsius Hardware monitor for temperature (crit_alarm)
 node_hwmon_temp_crit_celsius Hardware monitor for temperature (crit)
 node_hwmon_temp_max_celsius Hardware monitor for temperature (max)
 node_intr_total Total number of interrupts serviced.
 node_load1 1m load average.
 node_load15 15m load average.
 node_load5 5m load average.
 node_memory_Active_anon_bytes Memory information field Active_anon_bytes.
 node_memory_Active_bytes Memory information field Active_bytes.
 node_memory_Active_file_bytes Memory information field Active_file_bytes.
 node_memory_AnonHugePages_bytes Memory information field AnonHugePages_bytes.
 node_memory_AnonPages_bytes Memory information field AnonPages_bytes.
 node_memory_Bounce_bytes Memory information field Bounce_bytes.
 node_memory_Buffers_bytes Memory information field Buffers_bytes.
 node_memory_Cached_bytes Memory information field Cached_bytes.
 node_memory_CmaFree_bytes Memory information field CmaFree_bytes.
 node_memory_CmaTotal_bytes Memory information field CmaTotal_bytes.
 node_memory_CommitLimit_bytes Memory information field CommitLimit_bytes.
 node_memory_Committed_AS_bytes Memory information field Committed_AS_bytes.
 node_memory_DirectMap1G_bytes Memory information field DirectMap1G_bytes.
 node_memory_DirectMap2M_bytes Memory information field DirectMap2M_bytes.
 node_memory_DirectMap4k_bytes Memory information field DirectMap4k_bytes.
 node_memory_Dirty_bytes Memory information field Dirty_bytes.
 node_memory_FileHugePages_bytes Memory information field FileHugePages_bytes.
 node_memory_FilePmdMapped_bytes Memory information field FilePmdMapped_bytes.
 node_memory_HardwareCorrupted_bytes Memory information field HardwareCorrupted_bytes.
 node_memory_HugePages_Free Memory information field HugePages_Free.
 node_memory_HugePages_Rsvd Memory information field HugePages_Rsvd.
 node_memory_HugePages_Surp Memory information field HugePages_Surp.
 node_memory_HugePages_Total Memory information field HugePages_Total.
 node_memory_Hugepagesize_bytes Memory information field Hugepagesize_bytes.
 node_memory_Hugetlb_bytes Memory information field Hugetlb_bytes.
 node_memory_Inactive_anon_bytes Memory information field Inactive_anon_bytes.
 node_memory_Inactive_bytes Memory information field Inactive_bytes.
 node_memory_Inactive_file_bytes Memory information field Inactive_file_bytes.
 node_memory_KReclaimable_bytes Memory information field KReclaimable_bytes.
 node_memory_KernelStack_bytes Memory information field KernelStack_bytes.
 node_memory_Mapped_bytes Memory information field Mapped_bytes.
 node_memory_MemAvailable_bytes Memory information field MemAvailable_bytes.
 node_memory_MemFree_bytes Memory information field MemFree_bytes.
 node_memory_MemTotal_bytes Memory information field MemTotal_bytes.
 node_memory_Mlocked_bytes Memory information field Mlocked_bytes.
 node_memory_NFS_Unstable_bytes Memory information field NFS_Unstable_bytes.
 node_memory_PageTables_bytes Memory information field PageTables_bytes.
 node_memory_Percpu_bytes Memory information field Percpu_bytes.
 node_memory_SReclaimable_bytes Memory information field SReclaimable_bytes.
 node_memory_SUnreclaim_bytes Memory information field SUnreclaim_bytes.
 node_memory_ShmemHugePages_bytes Memory information field ShmemHugePages_bytes.
 node_memory_ShmemPmdMapped_bytes Memory information field ShmemPmdMapped_bytes.
 node_memory_Shmem_bytes Memory information field Shmem_bytes.
 node_memory_Slab_bytes Memory information field Slab_bytes.
 node_memory_SwapCached_bytes Memory information field SwapCached_bytes.
 node_memory_SwapFree_bytes Memory information field SwapFree_bytes.
 node_memory_SwapTotal_bytes Memory information field SwapTotal_bytes.
 node_memory_Unevictable_bytes Memory information field Unevictable_bytes.
 node_memory_VmallocChunk_bytes Memory information field VmallocChunk_bytes.
 node_memory_VmallocTotal_bytes Memory information field VmallocTotal_bytes.
 node_memory_VmallocUsed_bytes Memory information field VmallocUsed_bytes.
 node_memory_WritebackTmp_bytes Memory information field WritebackTmp_bytes.
 node_memory_Writeback_bytes Memory information field Writeback_bytes.
 node_netstat_Icmp6_InErrors Statistic Icmp6InErrors.
 node_netstat_Icmp6_InMsgs Statistic Icmp6InMsgs.
 node_netstat_Icmp6_OutMsgs Statistic Icmp6OutMsgs.
 node_netstat_Icmp_InErrors Statistic IcmpInErrors.
 node_netstat_Icmp_InMsgs Statistic IcmpInMsgs.
 node_netstat_Icmp_OutMsgs Statistic IcmpOutMsgs.
 node_netstat_Ip6_InOctets Statistic Ip6InOctets.
 node_netstat_Ip6_OutOctets Statistic Ip6OutOctets.
 node_netstat_IpExt_InOctets Statistic IpExtInOctets.
 node_netstat_IpExt_OutOctets Statistic IpExtOutOctets.
 node_netstat_Ip_Forwarding Statistic IpForwarding.
 node_netstat_TcpExt_ListenDrops Statistic TcpExtListenDrops.
 node_netstat_TcpExt_ListenOverflows Statistic TcpExtListenOverflows.
 node_netstat_TcpExt_SyncookiesFailed Statistic TcpExtSyncookiesFailed.
 node_netstat_TcpExt_SyncookiesRecv Statistic TcpExtSyncookiesRecv.
 node_netstat_TcpExt_SyncookiesSent Statistic TcpExtSyncookiesSent.
 node_netstat_TcpExt_TCPSynRetrans Statistic TcpExtTCPSynRetrans.
 node_netstat_TcpExt_TCPTimeouts Statistic TcpExtTCPTimeouts.
 node_netstat_Tcp_ActiveOpens Statistic TcpActiveOpens.
 node_netstat_Tcp_CurrEstab Statistic TcpCurrEstab.
 node_netstat_Tcp_InErrs Statistic TcpInErrs.
 node_netstat_Tcp_InSegs Statistic TcpInSegs.
 node_netstat_Tcp_OutRsts Statistic TcpOutRsts.
 node_netstat_Tcp_OutSegs Statistic TcpOutSegs.
 node_netstat_Tcp_PassiveOpens Statistic TcpPassiveOpens.
 node_netstat_Tcp_RetransSegs Statistic TcpRetransSegs.
 node_netstat_Udp6_InDatagrams Statistic Udp6InDatagrams.
 node_netstat_Udp6_InErrors Statistic Udp6InErrors.
 node_netstat_Udp6_NoPorts Statistic Udp6NoPorts.
 node_netstat_Udp6_OutDatagrams Statistic Udp6OutDatagrams.
 node_netstat_Udp6_RcvbufErrors Statistic Udp6RcvbufErrors.
 node_netstat_Udp6_SndbufErrors Statistic Udp6SndbufErrors.
 node_netstat_UdpLite6_InErrors Statistic UdpLite6InErrors.
 node_netstat_UdpLite_InErrors Statistic UdpLiteInErrors.
 node_netstat_Udp_InDatagrams Statistic UdpInDatagrams.
 node_netstat_Udp_InErrors Statistic UdpInErrors.
 node_netstat_Udp_NoPorts Statistic UdpNoPorts.
 node_netstat_Udp_OutDatagrams Statistic UdpOutDatagrams.
 node_netstat_Udp_RcvbufErrors Statistic UdpRcvbufErrors.
 node_netstat_Udp_SndbufErrors Statistic UdpSndbufErrors.
 node_network_address_assign_type address_assign_type value of /sys/class/net/<iface>.
 node_network_carrier carrier value of /sys/class/net/<iface>.
 node_network_carrier_changes_total carrier_changes_total value of /sys/class/net/<iface>.
 node_network_carrier_down_changes_total carrier_down_changes_total value of /sys/class/net/<iface>.
 node_network_carrier_up_changes_total carrier_up_changes_total value of /sys/class/net/<iface>.
 node_network_device_id device_id value of /sys/class/net/<iface>.
 node_network_dormant dormant value of /sys/class/net/<iface>.
 node_network_flags flags value of /sys/class/net/<iface>.
 node_network_iface_id iface_id value of /sys/class/net/<iface>.
 node_network_iface_link iface_link value of /sys/class/net/<iface>.
 node_network_iface_link_mode iface_link_mode value of /sys/class/net/<iface>.
 node_network_info Non-numeric data from /sys/class/net/<iface>, value is always 1.
 node_network_mtu_bytes mtu_bytes value of /sys/class/net/<iface>.
 node_network_name_assign_type name_assign_type value of /sys/class/net/<iface>.
 node_network_net_dev_group net_dev_group value of /sys/class/net/<iface>.
 node_network_protocol_type protocol_type value of /sys/class/net/<iface>.
 node_network_receive_bytes_total Network device statistic receive_bytes.
 node_network_receive_compressed_total Network device statistic receive_compressed.
 node_network_receive_drop_total Network device statistic receive_drop.
 node_network_receive_errs_total Network device statistic receive_errs.
 node_network_receive_fifo_total Network device statistic receive_fifo.
 node_network_receive_frame_total Network device statistic receive_frame.
 node_network_receive_multicast_total Network device statistic receive_multicast.
 node_network_receive_packets_total Network device statistic receive_packets.
 node_network_speed_bytes speed_bytes value of /sys/class/net/<iface>.
 node_network_transmit_bytes_total Network device statistic transmit_bytes.
 node_network_transmit_carrier_total Network device statistic transmit_carrier.
 node_network_transmit_colls_total Network device statistic transmit_colls.
 node_network_transmit_compressed_total Network device statistic transmit_compressed.
 node_network_transmit_drop_total Network device statistic transmit_drop.
 node_network_transmit_errs_total Network device statistic transmit_errs.
 node_network_transmit_fifo_total Network device statistic transmit_fifo.
 node_network_transmit_packets_total Network device statistic transmit_packets.
 node_network_transmit_queue_length transmit_queue_length value of /sys/class/net/<iface>.
 node_network_up Value is 1 if operstate is 'up', 0 otherwise.
 node_nf_conntrack_entries Number of currently allocated flow entries for connection tracking.
 node_nf_conntrack_entries_limit Maximum size of connection tracking table.
 node_nfs_connections_total Total number of NFSd TCP connections.
 node_nfs_packets_total Total NFSd network packets (sent+received) by protocol type.
 node_nfs_requests_total Number of NFS procedures invoked.
 node_nfs_rpc_authentication_refreshes_total Number of RPC authentication refreshes performed.
 node_nfs_rpc_retransmissions_total Number of RPC transmissions performed.
 node_nfs_rpcs_total Total number of RPCs performed.
 node_os_info A metric with a constant '1' value labeled by build_id, id, id_like, image_id, image_version, name, pretty_name, variant, variant_id, version, version_codename, version_id.
 node_os_version Metric containing the major.minor part of the OS version.
 node_pressure_cpu_waiting_seconds_total Total time in seconds that processes have waited for CPU time
 node_pressure_io_stalled_seconds_total Total time in seconds no process could make progress due to IO congestion
 node_pressure_io_waiting_seconds_total Total time in seconds that processes have waited due to IO congestion
 node_pressure_memory_stalled_seconds_total Total time in seconds no process could make progress due to memory congestion
 node_pressure_memory_waiting_seconds_total Total time in seconds that processes have waited for memory
 node_procs_blocked Number of processes blocked waiting for I/O to complete.
 node_procs_running Number of processes in runnable state.
 node_rapl_dram_joules_total Current RAPL dram value in joules
 node_rapl_package_joules_total Current RAPL package value in joules
 node_schedstat_running_seconds_total Number of seconds CPU spent running a process.
 node_schedstat_timeslices_total Number of timeslices executed by CPU.
 node_schedstat_waiting_seconds_total Number of seconds spent by processing waiting for this CPU.
 node_scrape_collector_duration_seconds node_exporter: Duration of a collector scrape.
 node_scrape_collector_success node_exporter: Whether a collector succeeded.
 node_sockstat_FRAG6_inuse Number of FRAG6 sockets in state inuse.
 node_sockstat_FRAG6_memory Number of FRAG6 sockets in state memory.
 node_sockstat_FRAG_inuse Number of FRAG sockets in state inuse.
 node_sockstat_FRAG_memory Number of FRAG sockets in state memory.
 node_sockstat_RAW6_inuse Number of RAW6 sockets in state inuse.
 node_sockstat_RAW_inuse Number of RAW sockets in state inuse.
 node_sockstat_TCP6_inuse Number of TCP6 sockets in state inuse.
 node_sockstat_TCP_alloc Number of TCP sockets in state alloc.
 node_sockstat_TCP_inuse Number of TCP sockets in state inuse.
 node_sockstat_TCP_mem Number of TCP sockets in state mem.
 node_sockstat_TCP_mem_bytes Number of TCP sockets in state mem_bytes.
 node_sockstat_TCP_orphan Number of TCP sockets in state orphan.
 node_sockstat_TCP_tw Number of TCP sockets in state tw.
 node_sockstat_UDP6_inuse Number of UDP6 sockets in state inuse.
 node_sockstat_UDPLITE6_inuse Number of UDPLITE6 sockets in state inuse.
 node_sockstat_UDPLITE_inuse Number of UDPLITE sockets in state inuse.
 node_sockstat_UDP_inuse Number of UDP sockets in state inuse.
 node_sockstat_UDP_mem Number of UDP sockets in state mem.
 node_sockstat_UDP_mem_bytes Number of UDP sockets in state mem_bytes.
 node_sockstat_sockets_used Number of IPv4 sockets in use.
 node_softnet_dropped_total Number of dropped packets
 node_softnet_processed_total Number of processed packets
 node_softnet_times_squeezed_total Number of times processing packets ran out of quota
 node_textfile_scrape_error 1 if there was an error opening or reading a file, 0 otherwise
 node_thermal_zone_temp Zone temperature in Celsius
 node_time_clocksource_available_info Available clocksources read from '/sys/devices/system/clocksource'.
 node_time_clocksource_current_info Current clocksource read from '/sys/devices/system/clocksource'.
 node_time_seconds System time in seconds since epoch (1970).
 node_time_zone_offset_seconds System time zone offset in seconds.
 node_timex_estimated_error_seconds Estimated error in seconds.
 node_timex_frequency_adjustment_ratio Local clock frequency adjustment.
 node_timex_loop_time_constant Phase-locked loop time constant.
 node_timex_maxerror_seconds Maximum error in seconds.
 node_timex_offset_seconds Time offset in between local system and reference clock.
 node_timex_pps_calibration_total Pulse per second count of calibration intervals.
 node_timex_pps_error_total Pulse per second count of calibration errors.
 node_timex_pps_frequency_hertz Pulse per second frequency.
 node_timex_pps_jitter_seconds Pulse per second jitter.
 node_timex_pps_jitter_total Pulse per second count of jitter limit exceeded events.
 node_timex_pps_shift_seconds Pulse per second interval duration.
 node_timex_pps_stability_exceeded_total Pulse per second count of stability limit exceeded events.
 node_timex_pps_stability_hertz Pulse per second stability, average of recent frequency changes.
 node_timex_status Value of the status array bits.
 node_timex_sync_status Is clock synchronized to a reliable server (1 = yes, 0 = no).
 node_timex_tai_offset_seconds International Atomic Time (TAI) offset.
 node_timex_tick_seconds Seconds between clock ticks.
 node_udp_queues Number of allocated memory in the kernel for UDP datagrams in bytes.
 node_uname_info Labeled system information as provided by the uname system call.
 node_vmstat_oom_kill /proc/vmstat information field oom_kill.
 node_vmstat_pgfault /proc/vmstat information field pgfault.
 node_vmstat_pgmajfault /proc/vmstat information field pgmajfault.
 node_vmstat_pgpgin /proc/vmstat information field pgpgin.
 node_vmstat_pgpgout /proc/vmstat information field pgpgout.
 node_vmstat_pswpin /proc/vmstat information field pswpin.
 node_vmstat_pswpout /proc/vmstat information field pswpout.
 node_zfs_abd_linear_cnt kstat.zfs.misc.abdstats.linear_cnt
 node_zfs_abd_linear_data_size kstat.zfs.misc.abdstats.linear_data_size
 node_zfs_abd_scatter_chunk_waste kstat.zfs.misc.abdstats.scatter_chunk_waste
 node_zfs_abd_scatter_cnt kstat.zfs.misc.abdstats.scatter_cnt
 node_zfs_abd_scatter_data_size kstat.zfs.misc.abdstats.scatter_data_size
 node_zfs_abd_scatter_order_0 kstat.zfs.misc.abdstats.scatter_order_0
 node_zfs_abd_scatter_order_1 kstat.zfs.misc.abdstats.scatter_order_1
 node_zfs_abd_scatter_order_10 kstat.zfs.misc.abdstats.scatter_order_10
 node_zfs_abd_scatter_order_2 kstat.zfs.misc.abdstats.scatter_order_2
 node_zfs_abd_scatter_order_3 kstat.zfs.misc.abdstats.scatter_order_3
 node_zfs_abd_scatter_order_4 kstat.zfs.misc.abdstats.scatter_order_4
 node_zfs_abd_scatter_order_5 kstat.zfs.misc.abdstats.scatter_order_5
 node_zfs_abd_scatter_order_6 kstat.zfs.misc.abdstats.scatter_order_6
 node_zfs_abd_scatter_order_7 kstat.zfs.misc.abdstats.scatter_order_7
 node_zfs_abd_scatter_order_8 kstat.zfs.misc.abdstats.scatter_order_8
 node_zfs_abd_scatter_order_9 kstat.zfs.misc.abdstats.scatter_order_9
 node_zfs_abd_scatter_page_alloc_retry kstat.zfs.misc.abdstats.scatter_page_alloc_retry
 node_zfs_abd_scatter_page_multi_chunk kstat.zfs.misc.abdstats.scatter_page_multi_chunk
 node_zfs_abd_scatter_page_multi_zone kstat.zfs.misc.abdstats.scatter_page_multi_zone
 node_zfs_abd_scatter_sg_table_retry kstat.zfs.misc.abdstats.scatter_sg_table_retry
 node_zfs_abd_struct_size kstat.zfs.misc.abdstats.struct_size
 node_zfs_arc_access_skip kstat.zfs.misc.arcstats.access_skip
 node_zfs_arc_anon_evictable_data kstat.zfs.misc.arcstats.anon_evictable_data
 node_zfs_arc_anon_evictable_metadata kstat.zfs.misc.arcstats.anon_evictable_metadata
 node_zfs_arc_anon_size kstat.zfs.misc.arcstats.anon_size
 node_zfs_arc_arc_dnode_limit kstat.zfs.misc.arcstats.arc_dnode_limit
 node_zfs_arc_arc_loaned_bytes kstat.zfs.misc.arcstats.arc_loaned_bytes
 node_zfs_arc_arc_meta_limit kstat.zfs.misc.arcstats.arc_meta_limit
 node_zfs_arc_arc_meta_max kstat.zfs.misc.arcstats.arc_meta_max
 node_zfs_arc_arc_meta_min kstat.zfs.misc.arcstats.arc_meta_min
 node_zfs_arc_arc_meta_used kstat.zfs.misc.arcstats.arc_meta_used
 node_zfs_arc_arc_need_free kstat.zfs.misc.arcstats.arc_need_free
 node_zfs_arc_arc_no_grow kstat.zfs.misc.arcstats.arc_no_grow
 node_zfs_arc_arc_prune kstat.zfs.misc.arcstats.arc_prune
 node_zfs_arc_arc_raw_size kstat.zfs.misc.arcstats.arc_raw_size
 node_zfs_arc_arc_sys_free kstat.zfs.misc.arcstats.arc_sys_free
 node_zfs_arc_arc_tempreserve kstat.zfs.misc.arcstats.arc_tempreserve
 node_zfs_arc_async_upgrade_sync kstat.zfs.misc.arcstats.async_upgrade_sync
 node_zfs_arc_bonus_size kstat.zfs.misc.arcstats.bonus_size
 node_zfs_arc_c kstat.zfs.misc.arcstats.c
 node_zfs_arc_c_max kstat.zfs.misc.arcstats.c_max
 node_zfs_arc_c_min kstat.zfs.misc.arcstats.c_min
 node_zfs_arc_compressed_size kstat.zfs.misc.arcstats.compressed_size
 node_zfs_arc_data_size kstat.zfs.misc.arcstats.data_size
 node_zfs_arc_dbuf_size kstat.zfs.misc.arcstats.dbuf_size
 node_zfs_arc_deleted kstat.zfs.misc.arcstats.deleted
 node_zfs_arc_demand_data_hits kstat.zfs.misc.arcstats.demand_data_hits
 node_zfs_arc_demand_data_misses kstat.zfs.misc.arcstats.demand_data_misses
 node_zfs_arc_demand_hit_predictive_prefetch kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch
 node_zfs_arc_demand_hit_prescient_prefetch kstat.zfs.misc.arcstats.demand_hit_prescient_prefetch
 node_zfs_arc_demand_metadata_hits kstat.zfs.misc.arcstats.demand_metadata_hits
 node_zfs_arc_demand_metadata_misses kstat.zfs.misc.arcstats.demand_metadata_misses
 node_zfs_arc_dnode_size kstat.zfs.misc.arcstats.dnode_size
 node_zfs_arc_evict_l2_cached kstat.zfs.misc.arcstats.evict_l2_cached
 node_zfs_arc_evict_l2_eligible kstat.zfs.misc.arcstats.evict_l2_eligible
 node_zfs_arc_evict_l2_ineligible kstat.zfs.misc.arcstats.evict_l2_ineligible
 node_zfs_arc_evict_l2_skip kstat.zfs.misc.arcstats.evict_l2_skip
 node_zfs_arc_evict_not_enough kstat.zfs.misc.arcstats.evict_not_enough
 node_zfs_arc_evict_skip kstat.zfs.misc.arcstats.evict_skip
 node_zfs_arc_hash_chain_max kstat.zfs.misc.arcstats.hash_chain_max
 node_zfs_arc_hash_chains kstat.zfs.misc.arcstats.hash_chains
 node_zfs_arc_hash_collisions kstat.zfs.misc.arcstats.hash_collisions
 node_zfs_arc_hash_elements kstat.zfs.misc.arcstats.hash_elements
 node_zfs_arc_hash_elements_max kstat.zfs.misc.arcstats.hash_elements_max
 node_zfs_arc_hdr_size kstat.zfs.misc.arcstats.hdr_size
 node_zfs_arc_hits kstat.zfs.misc.arcstats.hits
 node_zfs_arc_l2_abort_lowmem kstat.zfs.misc.arcstats.l2_abort_lowmem
 node_zfs_arc_l2_asize kstat.zfs.misc.arcstats.l2_asize
 node_zfs_arc_l2_cksum_bad kstat.zfs.misc.arcstats.l2_cksum_bad
 node_zfs_arc_l2_evict_l1cached kstat.zfs.misc.arcstats.l2_evict_l1cached
 node_zfs_arc_l2_evict_lock_retry kstat.zfs.misc.arcstats.l2_evict_lock_retry
 node_zfs_arc_l2_evict_reading kstat.zfs.misc.arcstats.l2_evict_reading
 node_zfs_arc_l2_feeds kstat.zfs.misc.arcstats.l2_feeds
 node_zfs_arc_l2_free_on_write kstat.zfs.misc.arcstats.l2_free_on_write
 node_zfs_arc_l2_hdr_size kstat.zfs.misc.arcstats.l2_hdr_size
 node_zfs_arc_l2_hits kstat.zfs.misc.arcstats.l2_hits
 node_zfs_arc_l2_io_error kstat.zfs.misc.arcstats.l2_io_error
 node_zfs_arc_l2_misses kstat.zfs.misc.arcstats.l2_misses
 node_zfs_arc_l2_read_bytes kstat.zfs.misc.arcstats.l2_read_bytes
 node_zfs_arc_l2_rw_clash kstat.zfs.misc.arcstats.l2_rw_clash
 node_zfs_arc_l2_size kstat.zfs.misc.arcstats.l2_size
 node_zfs_arc_l2_write_bytes kstat.zfs.misc.arcstats.l2_write_bytes
 node_zfs_arc_l2_writes_done kstat.zfs.misc.arcstats.l2_writes_done
 node_zfs_arc_l2_writes_error kstat.zfs.misc.arcstats.l2_writes_error
 node_zfs_arc_l2_writes_lock_retry kstat.zfs.misc.arcstats.l2_writes_lock_retry
 node_zfs_arc_l2_writes_sent kstat.zfs.misc.arcstats.l2_writes_sent
 node_zfs_arc_memory_all_bytes kstat.zfs.misc.arcstats.memory_all_bytes
 node_zfs_arc_memory_direct_count kstat.zfs.misc.arcstats.memory_direct_count
 node_zfs_arc_memory_free_bytes kstat.zfs.misc.arcstats.memory_free_bytes
 node_zfs_arc_memory_indirect_count kstat.zfs.misc.arcstats.memory_indirect_count
 node_zfs_arc_memory_throttle_count kstat.zfs.misc.arcstats.memory_throttle_count
 node_zfs_arc_metadata_size kstat.zfs.misc.arcstats.metadata_size
 node_zfs_arc_mfu_evictable_data kstat.zfs.misc.arcstats.mfu_evictable_data
 node_zfs_arc_mfu_evictable_metadata kstat.zfs.misc.arcstats.mfu_evictable_metadata
 node_zfs_arc_mfu_ghost_evictable_data kstat.zfs.misc.arcstats.mfu_ghost_evictable_data
 node_zfs_arc_mfu_ghost_evictable_metadata kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata
 node_zfs_arc_mfu_ghost_hits kstat.zfs.misc.arcstats.mfu_ghost_hits
 node_zfs_arc_mfu_ghost_size kstat.zfs.misc.arcstats.mfu_ghost_size
 node_zfs_arc_mfu_hits kstat.zfs.misc.arcstats.mfu_hits
 node_zfs_arc_mfu_size kstat.zfs.misc.arcstats.mfu_size
 node_zfs_arc_misses kstat.zfs.misc.arcstats.misses
 node_zfs_arc_mru_evictable_data kstat.zfs.misc.arcstats.mru_evictable_data
 node_zfs_arc_mru_evictable_metadata kstat.zfs.misc.arcstats.mru_evictable_metadata
 node_zfs_arc_mru_ghost_evictable_data kstat.zfs.misc.arcstats.mru_ghost_evictable_data
 node_zfs_arc_mru_ghost_evictable_metadata kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata
 node_zfs_arc_mru_ghost_hits kstat.zfs.misc.arcstats.mru_ghost_hits
 node_zfs_arc_mru_ghost_size kstat.zfs.misc.arcstats.mru_ghost_size
 node_zfs_arc_mru_hits kstat.zfs.misc.arcstats.mru_hits
 node_zfs_arc_mru_size kstat.zfs.misc.arcstats.mru_size
 node_zfs_arc_mutex_miss kstat.zfs.misc.arcstats.mutex_miss
 node_zfs_arc_overhead_size kstat.zfs.misc.arcstats.overhead_size
 node_zfs_arc_p kstat.zfs.misc.arcstats.p
 node_zfs_arc_prefetch_data_hits kstat.zfs.misc.arcstats.prefetch_data_hits
 node_zfs_arc_prefetch_data_misses kstat.zfs.misc.arcstats.prefetch_data_misses
 node_zfs_arc_prefetch_metadata_hits kstat.zfs.misc.arcstats.prefetch_metadata_hits
 node_zfs_arc_prefetch_metadata_misses kstat.zfs.misc.arcstats.prefetch_metadata_misses
 node_zfs_arc_size kstat.zfs.misc.arcstats.size
 node_zfs_arc_uncompressed_size kstat.zfs.misc.arcstats.uncompressed_size
 node_zfs_dmu_tx_dmu_tx_assigned kstat.zfs.misc.dmu_tx.dmu_tx_assigned
 node_zfs_dmu_tx_dmu_tx_delay kstat.zfs.misc.dmu_tx.dmu_tx_delay
 node_zfs_dmu_tx_dmu_tx_dirty_delay kstat.zfs.misc.dmu_tx.dmu_tx_dirty_delay
 node_zfs_dmu_tx_dmu_tx_dirty_frees_delay kstat.zfs.misc.dmu_tx.dmu_tx_dirty_frees_delay
 node_zfs_dmu_tx_dmu_tx_dirty_over_max kstat.zfs.misc.dmu_tx.dmu_tx_dirty_over_max
 node_zfs_dmu_tx_dmu_tx_dirty_throttle kstat.zfs.misc.dmu_tx.dmu_tx_dirty_throttle
 node_zfs_dmu_tx_dmu_tx_error kstat.zfs.misc.dmu_tx.dmu_tx_error
 node_zfs_dmu_tx_dmu_tx_group kstat.zfs.misc.dmu_tx.dmu_tx_group
 node_zfs_dmu_tx_dmu_tx_memory_reclaim kstat.zfs.misc.dmu_tx.dmu_tx_memory_reclaim
 node_zfs_dmu_tx_dmu_tx_memory_reserve kstat.zfs.misc.dmu_tx.dmu_tx_memory_reserve
 node_zfs_dmu_tx_dmu_tx_quota kstat.zfs.misc.dmu_tx.dmu_tx_quota
 node_zfs_dmu_tx_dmu_tx_suspended kstat.zfs.misc.dmu_tx.dmu_tx_suspended
 node_zfs_dnode_dnode_alloc_next_block kstat.zfs.misc.dnodestats.dnode_alloc_next_block
 node_zfs_dnode_dnode_alloc_next_chunk kstat.zfs.misc.dnodestats.dnode_alloc_next_chunk
 node_zfs_dnode_dnode_alloc_race kstat.zfs.misc.dnodestats.dnode_alloc_race
 node_zfs_dnode_dnode_allocate kstat.zfs.misc.dnodestats.dnode_allocate
 node_zfs_dnode_dnode_buf_evict kstat.zfs.misc.dnodestats.dnode_buf_evict
 node_zfs_dnode_dnode_free_interior_lock_retry kstat.zfs.misc.dnodestats.dnode_free_interior_lock_retry
 node_zfs_dnode_dnode_hold_alloc_hits kstat.zfs.misc.dnodestats.dnode_hold_alloc_hits
 node_zfs_dnode_dnode_hold_alloc_interior kstat.zfs.misc.dnodestats.dnode_hold_alloc_interior
 node_zfs_dnode_dnode_hold_alloc_lock_misses kstat.zfs.misc.dnodestats.dnode_hold_alloc_lock_misses
 node_zfs_dnode_dnode_hold_alloc_lock_retry kstat.zfs.misc.dnodestats.dnode_hold_alloc_lock_retry
 node_zfs_dnode_dnode_hold_alloc_misses kstat.zfs.misc.dnodestats.dnode_hold_alloc_misses
 node_zfs_dnode_dnode_hold_alloc_type_none kstat.zfs.misc.dnodestats.dnode_hold_alloc_type_none
 node_zfs_dnode_dnode_hold_dbuf_hold kstat.zfs.misc.dnodestats.dnode_hold_dbuf_hold
 node_zfs_dnode_dnode_hold_dbuf_read kstat.zfs.misc.dnodestats.dnode_hold_dbuf_read
 node_zfs_dnode_dnode_hold_free_hits kstat.zfs.misc.dnodestats.dnode_hold_free_hits
 node_zfs_dnode_dnode_hold_free_lock_misses kstat.zfs.misc.dnodestats.dnode_hold_free_lock_misses
 node_zfs_dnode_dnode_hold_free_lock_retry kstat.zfs.misc.dnodestats.dnode_hold_free_lock_retry
 node_zfs_dnode_dnode_hold_free_misses kstat.zfs.misc.dnodestats.dnode_hold_free_misses
 node_zfs_dnode_dnode_hold_free_overflow kstat.zfs.misc.dnodestats.dnode_hold_free_overflow
 node_zfs_dnode_dnode_hold_free_refcount kstat.zfs.misc.dnodestats.dnode_hold_free_refcount
 node_zfs_dnode_dnode_move_active kstat.zfs.misc.dnodestats.dnode_move_active
 node_zfs_dnode_dnode_move_handle kstat.zfs.misc.dnodestats.dnode_move_handle
 node_zfs_dnode_dnode_move_invalid kstat.zfs.misc.dnodestats.dnode_move_invalid
 node_zfs_dnode_dnode_move_recheck1 kstat.zfs.misc.dnodestats.dnode_move_recheck1
 node_zfs_dnode_dnode_move_recheck2 kstat.zfs.misc.dnodestats.dnode_move_recheck2
 node_zfs_dnode_dnode_move_rwlock kstat.zfs.misc.dnodestats.dnode_move_rwlock
 node_zfs_dnode_dnode_move_special kstat.zfs.misc.dnodestats.dnode_move_special
 node_zfs_dnode_dnode_reallocate kstat.zfs.misc.dnodestats.dnode_reallocate
 node_zfs_fm_erpt_dropped kstat.zfs.misc.fm.erpt-dropped
 node_zfs_fm_erpt_set_failed kstat.zfs.misc.fm.erpt-set-failed
 node_zfs_fm_fmri_set_failed kstat.zfs.misc.fm.fmri-set-failed
 node_zfs_fm_payload_set_failed kstat.zfs.misc.fm.payload-set-failed
 node_zfs_vdev_cache_delegations kstat.zfs.misc.vdev_cache_stats.delegations
 node_zfs_vdev_cache_hits kstat.zfs.misc.vdev_cache_stats.hits
 node_zfs_vdev_cache_misses kstat.zfs.misc.vdev_cache_stats.misses
 node_zfs_vdev_mirror_non_rotating_linear kstat.zfs.misc.vdev_mirror_stats.non_rotating_linear
 node_zfs_vdev_mirror_non_rotating_seek kstat.zfs.misc.vdev_mirror_stats.non_rotating_seek
 node_zfs_vdev_mirror_preferred_found kstat.zfs.misc.vdev_mirror_stats.preferred_found
 node_zfs_vdev_mirror_preferred_not_found kstat.zfs.misc.vdev_mirror_stats.preferred_not_found
 node_zfs_vdev_mirror_rotating_linear kstat.zfs.misc.vdev_mirror_stats.rotating_linear
 node_zfs_vdev_mirror_rotating_offset kstat.zfs.misc.vdev_mirror_stats.rotating_offset
 node_zfs_vdev_mirror_rotating_seek kstat.zfs.misc.vdev_mirror_stats.rotating_seek
 node_zfs_xuio_onloan_read_buf kstat.zfs.misc.xuio_stats.onloan_read_buf
 node_zfs_xuio_onloan_write_buf kstat.zfs.misc.xuio_stats.onloan_write_buf
 node_zfs_xuio_read_buf_copied kstat.zfs.misc.xuio_stats.read_buf_copied
 node_zfs_xuio_read_buf_nocopy kstat.zfs.misc.xuio_stats.read_buf_nocopy
 node_zfs_xuio_write_buf_copied kstat.zfs.misc.xuio_stats.write_buf_copied
 node_zfs_xuio_write_buf_nocopy kstat.zfs.misc.xuio_stats.write_buf_nocopy
 node_zfs_zfetch_hits kstat.zfs.misc.zfetchstats.hits
 node_zfs_zfetch_max_streams kstat.zfs.misc.zfetchstats.max_streams
 node_zfs_zfetch_misses kstat.zfs.misc.zfetchstats.misses
 node_zfs_zil_zil_commit_count kstat.zfs.misc.zil.zil_commit_count
 node_zfs_zil_zil_commit_writer_count kstat.zfs.misc.zil.zil_commit_writer_count
 node_zfs_zil_zil_itx_copied_bytes kstat.zfs.misc.zil.zil_itx_copied_bytes
 node_zfs_zil_zil_itx_copied_count kstat.zfs.misc.zil.zil_itx_copied_count
 node_zfs_zil_zil_itx_count kstat.zfs.misc.zil.zil_itx_count
 node_zfs_zil_zil_itx_indirect_bytes kstat.zfs.misc.zil.zil_itx_indirect_bytes
 node_zfs_zil_zil_itx_indirect_count kstat.zfs.misc.zil.zil_itx_indirect_count
 node_zfs_zil_zil_itx_metaslab_normal_bytes kstat.zfs.misc.zil.zil_itx_metaslab_normal_bytes
 node_zfs_zil_zil_itx_metaslab_normal_count kstat.zfs.misc.zil.zil_itx_metaslab_normal_count
 node_zfs_zil_zil_itx_metaslab_slog_bytes kstat.zfs.misc.zil.zil_itx_metaslab_slog_bytes
 node_zfs_zil_zil_itx_metaslab_slog_count kstat.zfs.misc.zil.zil_itx_metaslab_slog_count
 node_zfs_zil_zil_itx_needcopy_bytes kstat.zfs.misc.zil.zil_itx_needcopy_bytes
 node_zfs_zil_zil_itx_needcopy_count kstat.zfs.misc.zil.zil_itx_needcopy_count
 process_cpu_seconds_total Total user and system CPU time spent in seconds.
 process_max_fds Maximum number of open file descriptors.
 process_open_fds Number of open file descriptors.
 process_resident_memory_bytes Resident memory size in bytes.
 process_start_time_seconds Start time of the process since unix epoch in seconds.
 process_virtual_memory_bytes Virtual memory size in bytes.
 process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
 promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
 promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
 promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.


openstack-libvirt-exporter (:9177/metrics)

txt
go_gc_duration_seconds A summary of the GC invocation durations.
 go_goroutines Number of goroutines that currently exist.
 go_info Information about the Go environment.
 go_memstats_alloc_bytes Number of bytes allocated and still in use.
 go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
 go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
 go_memstats_frees_total Total number of frees.
 go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
 go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
 go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
 go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
 go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
 go_memstats_heap_objects Number of allocated objects.
 go_memstats_heap_released_bytes Number of heap bytes released to OS.
 go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
 go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
 go_memstats_lookups_total Total number of pointer lookups.
 go_memstats_mallocs_total Total number of mallocs.
 go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
 go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
 go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
 go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
 go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
 go_memstats_other_sys_bytes Number of bytes used for other system allocations.
 go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
 go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
 go_memstats_sys_bytes Number of bytes obtained from system.
 go_threads Number of OS threads created.
 libvirt_up Whether scraping libvirt's metrics was successful.
 process_cpu_seconds_total Total user and system CPU time spent in seconds.
 process_max_fds Maximum number of open file descriptors.
 process_open_fds Number of open file descriptors.
 process_resident_memory_bytes Resident memory size in bytes.
 process_start_time_seconds Start time of the process since unix epoch in seconds.
 process_virtual_memory_bytes Virtual memory size in bytes.
 process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
 promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
 promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.

Activate K8s Exporter


Build and Execute

txt
git clone https://github.com/shalb-docker/kubernetes_exporter.git

cd kubernetes_exporter

docker login
docker compose -f docker-compose-build.yml build
docker compose -f docker-compose-build.yml push

docker compose up

Install docker-compose:


References
Charts
How to check MetricName with Exporter
bash
# curl {{ μ΅μŠ€ν¬ν„°SVC ClusterIP:PORT }}/metrics | grep '# TYPE' > {{ μ €μž₯ν•  txt 파일 }} && sed -i 's/# TYPE//g' {{ μ €μž₯ν•  txt 파일 }}
curl 10.233.36.195:3000/metrics | grep '# HELP' > mhkim/.txt && sed -i 's/# HELP//g' mhkim/.txt
# zsh
curl 10.233.36.195:3000/metrics | grep '# HELP' > mhkim/.txt && sed -i '' 's/# HELP//g' mhkim/.txt
PromQL example
bash
# <metric name>{<label name>=<label value>, ...}
api_http_requests_total{method="POST", handler="/messages"}
Prometheus
  • Prometheus-DefaultExporterPorts
  • Prometheus-ScrapeConfig
  • Bitnami-Prometheus
Main Exporter
Middleware Exporter
Analysis required
Alert Webhook POST


Notable Grafana Dashboard

  • node
  • openstack
  • k8s
  • middleware

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *