Fredric News

쿠버네티스 프로메테우스·타노스 기반 장기 모니터링 설계

💡 monitoring 핵심 가이드 및 실전 아키텍처 핵심 요약 (Executive Summary)

본 가이드는 monitoring 핵심 가이드 및 실전 아키텍처의 핵심 개념, 구성 아키텍처 및 실무 엔지니어링 베스트 프랙티스를 체계적으로 정리했습니다.

📑 Table of Contents

Declarative Management

Deployments

thanos + kube-prometheus-stack

Architecture & System Flow Diagram
SVG
Architecture & System Flow
Pan & Zoom


thanos on mgnt

bash 
{{- $clusterName := "mgnt" }}
{{- $namespace := .Values.resourceGroup.observ }}
{{- $releaseName := "thanos" }}
releases:
  - name: {{ $releaseName }}
    namespace: {{ $namespace }}
    chart: {{ .Values.helm.repoName }}/{{ $releaseName }}
    version: {{ index .Values.versions $releaseName "charts" }}
    values:
      - global:
          security:
            allowInsecureImages: true
      - image:
          registry: docker.io
          repository: dhi.io/thanos
          tag: {{ index .Values.versions $releaseName $releaseName }}
      - query:
          enabled: true
          replicaCount: 1
          stores:
            - "{{ index .Values.clusters "dev" "hostIp" }}:10901" # dev
            - "{{ index .Values.clusters "stg" "hostIp" }}:10901" # stg
            - "{{ printf "%s.%s.%s:%d" "kube-prometheus-stack-thanos-discovery" $namespace "svc.cluster.local" 10901 }}" # mgnt
          ingress:
            enabled: true
            hostname: thanos.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
            secretName: thanos.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}-tls
            ingressClassName: nginx
            annotations:
              cert-manager.io/cluster-issuer: {{ .Values.ingress.rootCAIssuer }}
            tls: true
            selfSigned: false
      - queryFrontend:
          enabled: false


kube-prometheus-stack on all

bash 
{{- $clusterName := .Environment.Name }}
{{- $namespace := .Values.resourceGroup.observ }}
{{- $releaseName := "kube-prometheus-stack" }}
releases:
  - name: {{ $releaseName }}
    namespace: {{ $namespace }}
    chart: {{ .Values.helm.repoName }}/{{ $releaseName }}
    version: {{ index .Values.versions $releaseName "charts" }}
    values:
      - crds:
          enabled: true
      - alertmanager:
          enabled: true
          service:
            nodePort: 30903
            type: NodePort
          ingress:
            enabled: true
            ingressClassName: nginx
            annotations:
              cert-manager.io/cluster-issuer: {{ .Values.ingress.rootCAIssuer }}
            hosts:
              - alert.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
            tls:
              - secretName: alert.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}-tls
                hosts:
                  - alert.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
{{- if eq $clusterName "mgnt" }}
      # [Mgmt 클러스터]
      - prometheus:
          enabled: true
          thanosService:
            enabled: true
          prometheusSpec:
            externalLabels:
              cluster: {{ $clusterName }}
            thanos:
              readyTimeout: 1m
              listenLocal: false
            retention: 1d
            retentionSize: "10GB"
            serviceMonitorSelector:
              matchExpressions:
                - key: release
                  operator: In
                  values:
                    - {{ $releaseName }} #차트내 유지용
                    - {{ $namespace }} #차트외 등록용
      - grafana:
          enabled: true
          defaultDashboardsTimezone: Asia/Seoul
          adminUser: {{ .Values.globalAdmin }}
          adminPassword: {{ .Values.globalPassword }}
          ingress:
            enabled: true
            ingressClassName: nginx
            annotations:
              cert-manager.io/cluster-issuer: {{ .Values.ingress.rootCAIssuer }}
            hosts:
              - dash.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
            tls:
              - secretName: dash.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}-tls
                hosts:
                  - dash.{{ printf "%s%s" (index .Values.clusters $clusterName "ingress" "subdomain") .Values.ingress.baseDomain }}
          sidecar:
            dashboards:
              enabled: true
              multicluster:
                global:
                  enabled: true # dash params에 'cluster' 활성화
                etcd:
                  enabled: true # etcd-dash params에 'cluster' 활성화
            # default datasource 제거
            datasources:
              enabled: true
              defaultDatasourceEnabled: false
              isDefaultDatasource: false
          # thanos를 default datasource로 등록
          additionalDataSources:
            - name: "Thanos-Global"
              type: prometheus
              uid: thanos-global
              url: "http://thanos-query.{{ $namespace }}.svc.cluster.local:9090" # 중앙 쿼리 엔진 주소
              access: proxy
              isDefault: true
            - name: "Alertmanager-mgnt"
              type: alertmanager
              uid: alertmanager-mgnt
              url: "http://kube-prometheus-stack-alertmanager.observation-system:9093"
              access: proxy
              jsonData:
                handleGrafanaManagedAlerts: false
                implementation: prometheus
            - name: "Alertmanager-dev"
              type: alertmanager
              uid: alertmanager-dev
              url: "http://{{ index .Values.clusters "dev" "hostIp" }}:9093"
              access: proxy
              jsonData:
                handleGrafanaManagedAlerts: false
                implementation: prometheus
            - name: "Alertmanager-stg"
              type: alertmanager
              uid: alertmanager-stg
              url: "http://{{ index .Values.clusters "stg" "hostIp" }}:9093"
              access: proxy
              jsonData:
                handleGrafanaManagedAlerts: false
                implementation: prometheus
          serviceMonitor:
            enabled: true
            labels:
              release: {{ $namespace }}
      # k3s disables
      - kubeControllerManager:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
      - kubeEtcd:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
      - kubeScheduler:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
      - kubeProxy:
          enabled: true
          service:
            enabled: false
          serviceMonitor:
            enabled: false
{{- else }}
      # [Dev / Stg 클러스터]
      - prometheus:
          enabled: true
          thanosService:
            enabled: false
          thanosServiceExternal:
            enabled: true
            externalTrafficPolicy: Cluster
            type: NodePort
            nodePort: 30901 # grpc
            httpNodePort: 30902 # http
          prometheusSpec:
            externalLabels:
              cluster: {{ $clusterName }}
            thanos:
              readyTimeout: 1m
              listenLocal: false
            retention: 1d
            retentionSize: "10GB"
            secrets:
              - etcd-client-certs # mount
      # Daemon 기반 etcd 설정
      - kubeEtcd:
          enabled: true
          endpoints: {{ index .Values.clusters $clusterName "controlPlaneHosts" | toJson }}
          service:
            enabled: true
            port: 2379
            targetPort: 2379
          serviceMonitor:
            enabled: true
            scheme: https
            insecureSkipVerify: true
            caFile: /etc/prometheus/secrets/etcd-client-certs/etcd-ca.crt
            certFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.crt
            keyFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.key
      - grafana:
          enabled: false
{{- end }}

thanos + lgtm-distributed + alloy-operator + kube-prometheus-stack (LGTT)


lgtm-distributed on mgnt


alloy-operator on all


kube-prometheus-stack on all (for serviceMonitor)

Action

kube-proxy 외부 접근 활성화

bash 
kubectl edit cm kube-proxy -n kube-system

# 127.0.0.1:10249 -> 0.0.0.0:10249
>>>vim
data:
  config.conf: |-
...
    metricsBindAddress: 0.0.0.0:10249
<<<

kubectl delete pod -l k8s-app=kube-proxy -n kube-system

daemon 기반 etcd metric

  • 테스트
bash 
  ls -l /etc/ssl/etcd/ssl/
  export node_name=$(hostname)
  curl -v -k --cacert /etc/ssl/etcd/ssl/ca.pem \
       --cert /etc/ssl/etcd/ssl/admin-${node_name}.pem \
       --key /etc/ssl/etcd/ssl/admin-${node_name}-key.pem \
       https://127.0.0.1:2379/metrics
  • kubernetes 등록
bash 
  kubectl create secret generic etcd-client-certs \
    -n observation-system \
    --from-file=etcd-ca.crt=/etc/ssl/etcd/ssl/ca.pem \
    --from-file=etcd-client.crt=/etc/ssl/etcd/ssl/admin-${node_name}.pem \
    --from-file=etcd-client.key=/etc/ssl/etcd/ssl/admin-${node_name}-key.pem
  • kube-prometheus-stack chart
bash 
        - prometheus:
            prometheusSpec:
  	          ...
              secrets:
                - etcd-client-certs # mount
        # Daemon 기반 etcd 설정
        - kubeEtcd:
            enabled: true
            endpoints: {{ index .Values.clusters $clusterName "controlPlaneHosts" | toJson }}
            service:
              enabled: true
              port: 2379
              targetPort: 2379
            serviceMonitor:
              enabled: true
              scheme: https
              insecureSkipVerify: true
              caFile: /etc/prometheus/secrets/etcd-client-certs/etcd-ca.crt
              certFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.crt
              keyFile: /etc/prometheus/secrets/etcd-client-certs/etcd-client.key
  • argocd endpoint update 버그 있음. 수동호출
bash 
  vi dev-etcd-ep.yaml
  ---
  apiVersion: v1
  kind: Endpoints
  metadata:
    name: kube-prometheus-stack-kube-etcd
    labels:
      app: kube-prometheus-stack-kube-etcd
      k8s-app: etcd-server
      
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/instance: kube-prometheus-stack
      app.kubernetes.io/version: "80.4.2"
      app.kubernetes.io/part-of: kube-prometheus-stack
      chart: kube-prometheus-stack-80.4.2
      release: "kube-prometheus-stack"
      heritage: "Helm"
    namespace: kube-system
  subsets:
    - addresses:
        - ip: 10.255.57.111
        - ip: 10.255.57.103
        - ip: 10.255.57.97
      ports:
        - name: http-metrics
          port: 2379
          protocol: TCP
  ---

  kubectl apply -f dev-etcd-ep.yaml

ingress없이 Prometheus UI 임시 노출 (Thanos 사용시 확인)

bash 
kubectl port-forward --address 0.0.0.0 -n observation-system svc/kube-prometheus-stack-prometheus 9090:9090 &
# browser 접속
http://$node_ip:9090
  • k3s일때에 별도 설치 필요 (port-forward 종속성)
bash 
  sudo apt-get update && sudo apt-get install -y socat

Olds


Grafana With Prometheus

Samples




TroubleShooting




Setups




이 포스팅은 쿠팡 파트너스 및 알리익스프레스 어필리에이트 활동의 일환으로, 이에 따른 일정액의 수수료를 제공받습니다.

Posted:

/

Category:

Author:

fredric

Tags:

, , , , , , , , , , , , , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *