Long-Term K8s Monitoring: Prometheus & Thanos

💡 Long-Term K8s Monitoring: Prometheus & Thanos — Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for Long-Term K8s Monitoring: Prometheus & Thanos.

📑 Table of Contents

Declarative Management
Deployments
Action
Olds
Samples
TroubleShooting
Setups

Declarative Management

Deployments

thanos + kube-prometheus-stack

Architecture & System Flow Diagram

SVG

Pan & Zoom

thanos on mgnt

bash

{{- $clusterName := "mgnt" }}
{{- $namespace := .Values.resourceGroup.observ }}
{{- $releaseName := "thanos" }}
releases:
  - name: {{ $releaseName }}
    namespace: {{ $namespace }}
    chart: {{ .Values.helm.repoName }}/{{ $releaseName }}
    version: {{ index .Values.versions $releaseName "charts" }}
    values:
      - global:
          security:
            allowInsecureImages: true
      - image:
          registry: docker.io
          repository: dhi.io/thanos
          tag: {{ index .Values.versions $releaseName $releaseName }}
      - query:
          enabled: true
          replicaCount: 1
          stores:
            - "{{ index .Values.clusters "dev" "hostIp" }}:10901" # dev
            - "{{ index .Values.clusters "stg" "hostIp" }}:10901" # stg
            - "{{ printf "%s.%s.%s:%d" "kube-prometheus-stack-thanos-discovery" $namespace "svc.cluster.local" 10901 }}" # mgnt
          ingress:
            enabled: true
            hostname: thanos.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
            secretName: thanos.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}-tls
            ingressClassName: nginx
            annotations:
              cert-manager.io/cluster-issuer: {{ .Values.ingress.rootCAIssuer }}
            tls: true
            selfSigned: false
      - queryFrontend:
          enabled: false

kube-prometheus-stack on all

bash

{{- $clusterName := .Environment.Name }}
{{- $namespace := .Values.resourceGroup.observ }}
{{- $releaseName := "kube-prometheus-stack" }}
releases:
  - name: {{ $releaseName }}
    namespace: {{ $namespace }}
    chart: {{ .Values.helm.repoName }}/{{ $releaseName }}
    version: {{ index .Values.versions $releaseName "charts" }}
    values:
      - crds:
          enabled: true
      - alertmanager:
          enabled: true
          service:
            nodePort: 30903
            type: NodePort
          ingress:
            enabled: true
            ingressClassName: nginx
            annotations:
              cert-manager.io/cluster-issuer: {{ .Values.ingress.rootCAIssuer }}
            hosts:
              - alert.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
            tls:
              - secretName: alert.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}-tls
                hosts:
                  - alert.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
{{- if eq $clusterName "mgnt" }}
      # [Mgmt 클러스터]
      - prometheus:
          enabled: true
          thanosService:
            enabled: true
          prometheusSpec:
            externalLabels:
              cluster: {{ $clusterName }}
            thanos:
              readyTimeout: 1m
              listenLocal: false
            retention: 1d
            retentionSize: "10GB"
            serviceMonitorSelector:
              matchExpressions:
                - key: release
                  operator: In
                  values:
                    - {{ $releaseName }} #차트내 유지용
                    - {{ $namespace }} #차트외 등록용
      - grafana:
          enabled: true
          defaultDashboardsTimezone: Asia/Seoul
          adminUser: {{ .Values.globalAdmin }}
          adminPassword: {{ .Values.globalPassword }}
          ingress:
            enabled: true
            ingressClassName: nginx
            annotations:
              cert-manager.io/cluster-issuer: {{ .Values.ingress.rootCAIssuer }}
            hosts:
              - dash.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
            tls:
              - secretName: dash.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}-tls
                hosts:
                  - dash.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
          sidecar:
            dashboards:
              enabled: true
              multicluster:
                global:
                  enabled: true # dash params에 'cluster' 활성화
                etcd:
                  enabled: true # etcd-dash params에 'cluster' 활성화
            # default datasource 제거
            datasources:
              enabled: true
              defaultDatasourceEnabled: false
              isDefaultDatasource: false
          # thanos를 default datasource로 등록
          additionalDataSources:
            - name: "Thanos-Global"
              type: prometheus
              uid: thanos-global
              url: "http://thanos-query.{{ $namespace }}.svc.cluster.local:9090" # 중앙 쿼리 엔진 주소
              access: proxy
              isDefault: true
            - name: "Alertmanager-mgnt"
              type: alertmanager
              uid: alertmanager-mgnt
              url: "http://kube-prometheus-stack-alertmanager.observation-system:9093"
              access: proxy
              jsonData:
                handleGrafanaManagedAlerts: false
                implementation: prometheus
            - name: "Alertmanager-dev"
              type: alertmanager
              uid: alertmanager-dev
              url: "http://{{ index .Values.clusters "dev" "hostIp" }}:9093"
              access: proxy
              jsonData:
                handleGrafanaManagedAlerts: false
                implementation: prometheus
            - name: "Alertmanager-stg"
              type: alertmanager
              uid: alertmanager-stg
              url: "http://{{ index .Values.clusters "stg" "hostIp" }}:9093"
              access: proxy
              jsonData:
                handleGrafanaManagedAlerts: false
                implementation: prometheus
          serviceMonitor:
            enabled: true
            labels:
              release: {{ $namespace }}
      # k3s disables
      - kubeControllerManager:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
      - kubeEtcd:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
      - kubeScheduler:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
      - kubeProxy:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
{{- else }}
      # [Dev / Stg 클러스터]
      - prometheus:
          enabled: true
          thanosService:
            enabled: false
          thanosServiceExternal:
            enabled: true
            externalTrafficPolicy: Cluster
            type: NodePort
            nodePort: 30901 # grpc
            httpNodePort: 30902 # http
          prometheusSpec:
            externalLabels:
              cluster: {{ $clusterName }}
            thanos:
              readyTimeout: 1m
              listenLocal: false
            retention: 1d
            retentionSize: "10GB"
            secrets:
              - etcd-client-certs # mount
      # Daemon 기반 etcd 설정
      - kubeEtcd:
          enabled: true
          endpoints: {{ index .Values.clusters $clusterName "controlPlaneHosts" | toJson }}
          service:
            enabled: true
            port: 2379
            targetPort: 2379
          serviceMonitor:
            enabled: true
            scheme: https
            insecureSkipVerify: true
            caFile: /etc/prometheus/secrets/etcd-client-certs/etcd-ca.crt
            certFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.crt
            keyFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.key
      - grafana:
          enabled: false
{{- end }}

thanos + lgtm-distributed + alloy-operator + kube-prometheus-stack (LGTT)

lgtm-distributed on mgnt

alloy-operator on all

kube-prometheus-stack on all (for serviceMonitor)

Action

Enable external access to kube-proxy

bash

kubectl edit cm kube-proxy -n kube-system

# 127.0.0.1:10249 -> 0.0.0.0:10249
>>>vim
data:
  config.conf: |-
...
    metricsBindAddress: 0.0.0.0:10249
<<<

kubectl delete pod -l k8s-app=kube-proxy -n kube-system

daemon-based etcd metric

Test

bash

  ls -l /etc/ssl/etcd/ssl/
  export node_name=$(hostname)
  curl -v -k --cacert /etc/ssl/etcd/ssl/ca.pem \
       --cert /etc/ssl/etcd/ssl/admin-${node_name}.pem \
       --key /etc/ssl/etcd/ssl/admin-${node_name}-key.pem \
       https://127.0.0.1:2379/metrics

bash

  kubectl create secret generic etcd-client-certs \
    -n observation-system \
    --from-file=etcd-ca.crt=/etc/ssl/etcd/ssl/ca.pem \
    --from-file=etcd-client.crt=/etc/ssl/etcd/ssl/admin-${node_name}.pem \
    --from-file=etcd-client.key=/etc/ssl/etcd/ssl/admin-${node_name}-key.pem

kube-prometheus-stack chart

bash

        - prometheus:
            prometheusSpec:
  	          ...
              secrets:
                - etcd-client-certs # mount
        # Daemon 기반 etcd 설정
        - kubeEtcd:
            enabled: true
            endpoints: {{ index .Values.clusters $clusterName "controlPlaneHosts" | toJson }}
            service:
              enabled: true
              port: 2379
              targetPort: 2379
            serviceMonitor:
              enabled: true
              scheme: https
              insecureSkipVerify: true
              caFile: /etc/prometheus/secrets/etcd-client-certs/etcd-ca.crt
              certFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.crt
              keyFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.key

argocd endpoint update bug. Manual call

bash

  vi dev-etcd-ep.yaml
  ---
  apiVersion: v1
  kind: Endpoints
  metadata:
    name: kube-prometheus-stack-kube-etcd
    labels:
      app: kube-prometheus-stack-kube-etcd
      k8s-app: etcd-server
      
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/instance: kube-prometheus-stack
      app.kubernetes.io/version: "80.4.2"
      app.kubernetes.io/part-of: kube-prometheus-stack
      chart: kube-prometheus-stack-80.4.2
      release: "kube-prometheus-stack"
      heritage: "Helm"
    namespace: kube-system
  subsets:
    - addresses:
        - ip: 10.255.57.111
        - ip: 10.255.57.103
        - ip: 10.255.57.97
      ports:
        - name: http-metrics
          port: 2379
          protocol: TCP
  ---

  kubectl apply -f dev-etcd-ep.yaml

Temporarily expose Prometheus UI without ingress (checked when using Thanos)

bash

kubectl port-forward --address 0.0.0.0 -n observation-system svc/kube-prometheus-stack-prometheus 9090:9090 &
# browser 접속
http://$node_ip:9090

Separate installation required for k3s (port-forward dependency)

bash

  sudo apt-get update && sudo apt-get install -y socat

Fredric News

Leave a Reply Cancel reply

Long-Term K8s Monitoring: Prometheus & Thanos

Declarative Management

Deployments

thanos + kube-prometheus-stack

thanos + lgtm-distributed + alloy-operator + kube-prometheus-stack (LGTT)

Action

Enable external access to kube-proxy

daemon-based etcd metric

Temporarily expose Prometheus UI without ingress (checked when using Thanos)

Olds

Samples

TroubleShooting

Setups

Comments

Leave a Reply Cancel reply