Kube-apiserver Hardening: RBAC & Admission

💡 Kube-apiserver Hardening: RBAC & Admission — Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for Kube-apiserver Hardening: RBAC & Admission.

Common

Log analysis to resolve Manifests errors

bash

# 컨테이너 재시작여부 확인
crictl ps

# Manifest 문법오류 탐색 (컨테이너가 안뜨므로 치명적임)
journalctl -u kubelet | grep manifest

# 종료된 컨테이너 에러여부 탐색
cat /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log
cat /var/log/containers/kube-apiserver-*.log

# 컨테이너 재시작: running 컨테이너를 재시작 할 경우
cd /etc/kubernetes/manifests
mv kube-apiserver.yaml ..
crictl ps
mv ../kube-apiserver.yaml .

# 컨테이너 재시작: crictl에서 컨테이너가 안뜨는 상태 (에러 후 복구)
systemctl restart kubelet

Audit-Logging

Setup for kube-apiserver

Auditing

bash

# 1. Create the directory
mkdir $target_log_dir

# 2. Edit Manifest
### 다음의 요구조건 사전 확인
#### policy_file
#### target_log_dir
#### target_log_file
#### maxsize
#### maxbackup
#### maxage

vi /etc/kubernetes/manifests/kube-apiserver.yaml
```
# enable Audit Logs
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=$policy_file
    - --audit-log-path=$target_log_dir/$target_log_file
    - --audit-log-maxsize=$maxsize #요구조건 있을 경우
    - --audit-log-maxbackup=$maxbackup #요구조건 있을 경우
    - --audit-log-maxage=$maxage #요구조건 있을 경우

# add new VolumeMounts
volumeMounts:
  - mountPath: $policy_file
    name: audit-policy
    readOnly: true
  - mountPath: $target_log_dir
    name: audit-logs
    readOnly: false

# add new Volumes
volumes:
  - name: audit-policy
    hostPath:
      path: $policy_file
      type: File
  - name: audit-logs
    hostPath:
      path: $target_log_dir
      type: DirectoryOrCreate
`

# wait
watch crictl ps

Admission-Controller

bash

vi /etc/kubernetes/manifests/kube-apiserver.yaml
```yaml
spec:
  containers:
  - command:
    - kube-apiserver
... #add 1 line
    - --enable-admission-plugins=<Plugin Name>
`

NodeRestriction

Kubernetes’ NodeRestriction Admission Controller is an important security component that limits the scope of Node and Pod objects that a Kubelet can modify through the API server, preventing privilege escalation to the entire cluster even if a specific node is hijacked.

1. Architecture structure diagram (Mermaid)

This is the overall architectural flow where Kubelet sends an API request, and inside kube-apiserver, NodeRestriction validates and blocks/allows the request.

Architecture & System Flow Diagram

SVG

Pan & Zoom

2. Main operational flow and architectural components

Identity verification

When joining a cluster, a Kubelet belongs to the system:nodes group and has a username in the format system:node: (e.g. system:node:node-A).
kube-apiserver verifies this identity via a client certificate (x509) or token.

Node Authorization

Kubelet’s API requests first go through a special-purpose Authorization module called Node Authorizer.
Node Authorizer controls read permissions so that the Kubelet can access resources such as Pods, ConfigMap, Secrets, and PVCs bound to its nodes.

NodeRestriction Admission Controller (Core)

After passing the authorization phase, the NodeRestriction plugin finely controls the final write/mutate permissions in the Admission Control phase.

Node object protection:
A Kubelet with the identity system:node:node-A can only modify its own node object (node-A).
Attempts to update or delete the state of another node (node-B) are unconditionally blocked.
Protection of Pod objects:
A Kubelet can only create, modify, or delete Pods that are bound to it.
Pods running on other nodes cannot be manipulated arbitrarily.
Restrict editing of Protected Labels:
Labels with the node-restriction.kubernetes.io/ prefix cannot be added, modified, or deleted by the Kubelet itself.
This is to prevent attacks where an attacker distorts pod scheduling (NodeSelector, Affinity) by arbitrarily attaching a specific label to a node.
Starting from version 1.16, Kubelet also prevents setting arbitrary labels (kubernetes.io/ or k8s.io/ children) on nodes other than the allowed well-known labels.

3. Security Benefits (Why NodeRestriction?)

Scenario	When NodeRestriction is not applied	When NodeRestriction is applied
Worker Node-A Violation	An attacker can obtain Node-A’s Kubelet privileges to arbitrarily change Node-B’s settings or force shutdown/reschedule sensitive Pods running on Node-B.	The attacker can only exercise influence within the scope of the compromised Node-A, and lateral movement to other nodes (Node-B) is completely blocked.
Node label modification	An attacker attaches an important label (e.g., `role=database-host`) to a node to lead sensitive data processing Pods to the node and steal them.	The Kubelet cannot change labels on its own (especially protected labels), so rogue labeling attacks that do not go through the control plane are defeated.

Refer to official document

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction

Type: Validating

This admission controller limits which Node and Pod objects the kubelet can modify. To be restricted by this admission controller, the kubelet must use credentials belonging to the system:nodes group, and the username must be in the form system:node:. These kubelets can modify only their own Node API objects, and only modify Pod API objects bound (assigned) to their Nodes. Additionally, a kubelet cannot update or remove taints from its Node API object.

The NodeRestriction admission plugin prevents the kubelet from deleting its Node API objects and enforces the following rules when modifying labels under the kubernetes-restriction.kubernetes.io/ or k8s.io/ prefix:

Forbidden (modification of Kubelet is blocked):
node-restriction.kubernetes.io/ Label with prefix: This prefix is a reserved space that allows administrators to label Node objects for the purpose of workload isolation.
Labels prefixed with node-role.kubernetes.io/ (e.g. node-role.kubernetes.io/control-plane): Restricted to prevent regular, unprivileged nodes from declaring themselves to have a specific role in the cluster.
Allowed (Kubelet can add/remove/update):
kubernetes.io/hostname
kubernetes.io/arch
kubernetes.io/os
beta.kubernetes.io/instance-type
node.kubernetes.io/instance-type
failure-domain.beta.kubernetes.io/region (Deprecated / Deprecated)
failure-domain.beta.kubernetes.io/zone (Deprecated / Deprecated)
topology.kubernetes.io/region
topology.kubernetes.io/zone
Labels with the prefix kubelet.kubernetes.io/
Labels with the prefix node.kubernetes.io/
Reserved:
The kubelet is restricted from using any other labels under the kubernetes.io or k8s.io prefix. NodeRestriction The admissions plugin disallows these labels by default to prevent unauthorized self-labeling, but may allow additional labels under this prefix as part of future new features.

When the ServiceAccountNodeAudienceRestriction Feature Gate is enabled, this admission plugin also restricts the audience from which the kubelet can request service account tokens via the TokenRequest API. A kubelet can only request tokens from targets that are already referenced by Pods on that node (via a Projected Service Account Token Volume or CSI Driver Token Request), or targets that have been explicitly granted permission through RBAC using the request-serviceaccounts-token-audience verb. For more information, see ‘Service account token audience restriction’.

Future versions may add additional restrictions to ensure that the kubelet has only the minimum set of permissions required to function properly.

Test performance tips

Edit /etc/kubernetes/manifests/kube-apiserver.yaml

bash

# add 1 flagline
  kube-apiserver --enable-admission-plugins=NodeRestriction

Application testing

bash

ssh node01 #worker-node
  export KUBECONFIG=/etc/kubernetes/kubelet.conf #kubectl 명령어 활성화

  # 노드 스스로에 대한 node-restriction.kubernetes.io/* 라벨 수정 차단됨
  k label node node01 node-restriction.kubernetes.io/two=123
  ```prompt
  Error from server (Forbidden): nodes "node01" is forbidden: is not allowed to modify labels: node-restriction.kubernetes.io/two
  `

  # 컨트롤 플레인 노드 라벨 수정 차단됨
  k label node controlplane killercoda/two=123
  ```prompt
  Error from server (Forbidden): nodes "controlplane" is forbidden: User "system:node:node01" cannot get resource "nodes" in API group "" at the cluster scope: node 'node01' cannot read 'controlplane', only its own Node object
  `

  # 노드 스스로에 대한 적용범위 내의 라벨 수정 허용됨
  k label node node01 test/two=123
  ```prompt
  node/node01 labeled
  `

  # 컨트롤플레인에서는 모두 허용됨

AlwaysPullImages

The AlwaysPullImages Admission Controller in Kubernetes is a security component that forces the container imagePullPolicy of all new Pods to be changed or verified to Always, preventing unauthorized reuse of existing images stored in the node’s local cache and always forcing the explicit download of the latest image from the registry.

1. Architecture structure diagram (Mermaid)

When a user request comes, the AlwaysPullImages plugin verifies and mutates the imagePullPolicy.

Architecture & System Flow Diagram

SVG

Pan & Zoom

2. Main operational flow and architectural components

Forced application in the Mutating phase (Mutate)

Even if the user sets imagePullPolicy to IfNotPresent or Never or omits it altogether when creating a Pod, the AlwaysPullImages admission controller detects this and forcibly mutates the field to Always.

Enforce security in multi-user and multi-tenant environments (Validate)

This plugin effectively defends against attack scenarios where unauthorized users use private images owned by other projects or users already cached on the local node by simply finding out the container image name. Forces you to always log into the registry to verify credentials (ImagePullSecrets).

3. Security Benefits (Why AlwaysPullImages?)

Scenario	When AlwaysPullImages are not applied	When AlwaysPullImages is applied
Unauthorized use of private images from other projects	If you find out the private image name already cached in the node and set it to `imagePullPolicy: IfNotPresent`, you can pull up another user’s image without your own registry token.	Forces every pod to always pull an image from the image registry when created, so without the correct `imagePullSecrets`, pod startup will fail.
Using old or tampered cache images	Even though the image tag has been updated, an outdated image in the node cache may be executed, resulting in inconsistent behavior.	Maintain execution stability by always pulling the latest version of the image from the registry.

Refer to official document

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#alwayspullimages

Type: Mutating and Validating

This admission controller is useful for enhancing security in multi-tenant clusters. This ensures that the user is double-checked against the registry for permission to pull the image before running it locally, even if the image has already been downloaded from the node.

Test performance tips

Edit /etc/kubernetes/manifests/kube-apiserver.yaml

bash

# add flag in --enable-admission-plugins
  kube-apiserver --enable-admission-plugins=NodeRestriction,AlwaysPullImages

Application testing

bash

# imagePullPolicy를 기재하지 않거나 IfNotPresent로 기재하여 Pod 배포
  kubectl run test-pod --image=nginx --restart=Never --dry-run=client -o yaml > pod.yaml

  # 생성된 yaml 확인 (기본값 또는 IfNotPresent)
  cat pod.yaml | grep imagePullPolicy

  # 클러스터에 배포
  kubectl apply -f pod.yaml

  # 실제 배포된 Pod 정보 조회 시 imagePullPolicy가 Always로 변경된 것을 확인
  kubectl get pod test-pod -o jsonpath='{.spec.containers[0].imagePullPolicy}'
  ```prompt
  Always
  `

ImagePolicyWebhook

Kubernetes’ ImagePolicyWebhook Admission Controller is a security plugin that delegates evaluation to an external policy verification webhook backend service so that the API server can block requests when container images are imported from unauthorized external registries or do not pass security vulnerability verification when creating or updating a Pod.

1. Architecture structure diagram (Mermaid)

This is a flow where the API server sends an AdmissionReview request to an external webhook server, and finally approves/rejects the creation of the pod based on the webhook server’s approval (AdmissionResponse) result.

Architecture & System Flow Diagram

SVG

Pan & Zoom

2. Main operational flow and architectural components

Definition of AdmissionConfiguration

Configuration file passed to kube-apiserver, which determines the path to the kubeconfig to be used by ImagePolicyWebhook, the local cache TTL, and whether requests should be granted (Fail-Open/defaultAllow: true) or rejected (Fail-Closed/defaultAllow: false) by default when webhook communication fails.

Communication with external webhook service (Webhook Callout)

The API server contains the image list information listed in the Pod, serializes it as a JSON object in the form of an image.k8s.io/v1alpha1 ImageReview object, and delivers an HTTPS POST request to the external backend server.

Policy validation and security certificate communication from external backend

The external backend analyzes image tags, digest hashes, etc. and returns a rejection message if it belongs to an untrusted remote registry or if a vulnerability exists. All communications must be secured via mutual TLS (mTLS).

3. Security Benefits (Why ImagePolicyWebhook?)

Scenario	When ImagePolicyWebhook is not applied	When applying ImagePolicyWebhook
Running malicious images containing vulnerabilities	Any user can distribute images containing vulnerabilities or malware from a public registry (e.g. Docker Hub) within the cluster.	The webhook is only passed if the scan result of the external security solution (e.g. Trivy, Anchore, etc.) meets the cluster policy criteria, so dangerous images are pre-blocked from entering the runtime.
Use of unauthorized private registries	Risk of confidential data leakage occurs due to arbitrary use of unconfirmed private registry images outside the company’s security network.	If it is not an allowed whitelist registry address (e.g. `.gcr.io`, `.ecr.aws`), the deployment request itself is rejected at the API stage.

Refer to official document

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook

Type: Validating

This admission controller calls the backend webhook service to evaluate the container image. Pod creation will fail if the webhook does not connect or does not meet policy requirements. For detailed operation, you must configure a configuration file (AdmissionConfiguration) and a separate kubeconfig file for communication using the --admission-control-config-file option.

Test performance tips

Create and modify /etc/kubernetes/admission/admission_config.yaml

yaml

apiVersion: apiserver.config.k8s.io/v1
  kind: AdmissionConfiguration
  plugins:
  - name: ImagePolicyWebhook
    configuration:
      imagePolicy:
        kubeConfigFile: /etc/kubernetes/admission/kubeconf.yaml
        allowTTL: 50
        denyTTL: 50
        retryBackoff: 500
        defaultAllow: false # 웹훅 통신 장애 시 Fail-Closed (안전)

Create /etc/kubernetes/admission/kubeconf.yaml (webhook information)

yaml

apiVersion: v1
  kind: Config
  clusters:
  - cluster:
      server: <https://image-policy-webhook-service.security.svc:2379/validate>
      certificate-authority: /etc/kubernetes/admission/ca.crt
    name: image-policy-webhook
  users:
  - name: api-server
    user:
      client-certificate: /etc/kubernetes/admission/api-server.crt
      client-key: /etc/kubernetes/admission/api-server.key
  contexts:
  - context:
      cluster: image-policy-webhook
      user: api-server
    name: image-policy-webhook
  current-context: image-policy-webhook

Edit /etc/kubernetes/manifests/kube-apiserver.yaml and add mount

yaml

# 1. 플러그인 활성화 및 설정파일 플래그 추가
  - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
  - --admission-control-config-file=/etc/kubernetes/admission/admission_config.yaml

  # 2. 볼륨 마운트 추가 (admission 디렉토리 내의 설정 및 인증서 사용 가능하도록 호스트 마운트)
  ...
  spec:
    containers:
    - name: kube-apiserver
      volumeMounts:
      - mountPath: /etc/kubernetes/admission
        name: admission-config
        readOnly: true
    volumes:
    - hostPath:
        path: /etc/kubernetes/admission
        type: DirectoryOrCreate
      name: admission-config

Application testing

bash

# 승인되지 않은 비인가 이미지를 사용하는 Pod 배포 시도
  kubectl run malicious-pod --image=evilregistry.com/malicious-app:latest
  ```prompt
  Error from server (Forbidden): pods "malicious-pod" is forbidden: image evilregistry.com/malicious-app:latest is denied by image policy webhook.
  `

PodSecurity

Kubernetes’ PodSecurity Admission Controller is a core security component that verifies and controls the specification of pods at the namespace level according to three predefined pod security standards (Pod Security Standards – Privileged, Baseline, Restricted), preventing pods with unnecessarily high system privileges or security settings from running. (This is the default mechanism that completely replaces PodSecurityPolicy (PSP), which was completely removed in Kubernetes version 1.25.)

1. Architecture structure diagram (Mermaid)

When requesting pod creation/modification, the policy level is evaluated according to the pod-security.kubernetes.io/* label rule of the namespace, and this flow blocks/allows differently depending on the enforce, warn, and audit modes.

Architecture & System Flow Diagram

SVG

Pan & Zoom

2. Main operational flow and architectural components

Pod Security Standards

Privileged: A completely unrestricted policy, targeting system/infrastructure pods that require host permissions (privileged: true, hostPID, hostNetwork, etc.).
Baseline: A basic constraint policy that allows common application pods to be spun up with minimal hassle while preventing popular privilege escalation. (e.g. hostPath, hostPort restrictions, etc.)
Restricted: Extremely security-restricted policy, enforcing pod hardening best practices. (Example: Prohibit root execution – runAsNonRoot: true, Linux capability whitelist restrictions, volume restrictions, etc.)

Evaluation Modes

enforce: Completely reject creation/modification of violating pods.
warn: Displays a warning message to the user screen (kubectl), but allows pod creation.
audit: Records violations in the audit log and allows pod creation.

3. Security Benefits (Why PodSecurity?)

Scenario	When PodSecurity is not applied	When applying PodSecurity
Abuse of root privileges and privileged containers	If an attacker or developer inadvertently sets the `privileged: true` setting, it may lead to takeover of the host OS when hijacking the container.	In namespaces where `enforce=baseline` or `restricted` are set, privileged pod creation is immediately blocked, preventing the risk of container escape attacks.
Share host namespace	The container directly shares the host’s process (hostPID) and network (hostNetwork), allowing sniffing and eavesdropping on the cluster network.	Allow only policy criteria with host zone sharing disabled, providing strong kernel-level isolation between container environments.

Refer to official document

https://kubernetes.io/docs/concepts/security/pod-security-admission/

Type: Validating

This admission controller blocks the creation of pods that do not meet the specified label policy level for the namespace, or generates warnings and audit logs. Since Kubernetes version 1.23 and higher is activated and operated by default, it can be used immediately through namespace labeling without any separate --enable-admission-plugins modification.

Test performance tips

Create namespace and set PSS policy label

bash

kubectl create ns secure-apps

  # 1. baseline 수준을 강제 적용 (enforce)
  kubectl label namespace secure-apps pod-security.kubernetes.io/enforce=baseline

  # 2. restricted 수준 위반 시 경고 출력 (warn)
  kubectl label namespace secure-apps pod-security.kubernetes.io/warn=restricted

Application testing

bash

# 특권 권한(Privileged)을 요구하는 Pod yaml 파일 생성
  cat <<EOF > priv-pod.yaml
  apiVersion: v1
  kind: Pod
  metadata:
    name: bad-pod
    namespace: secure-apps
  spec:
    containers:
    - name: nginx
      image: nginx
      securityContext:
        privileged: true
  EOF

  # 배포 시도 (baseline enforce 정책에 의해 생성 차단됨)
  kubectl apply -f priv-pod.yaml
  ```prompt
  Error from server (Forbidden): error when creating "priv-pod.yaml": pods "bad-pod" is forbidden: violates PodSecurity "baseline:latest": privileged (container "nginx" must not set securityContext.privileged=true)
  `

CSR Management

1. Create CSR with Openssl

bash

openssl genrsa -out /root/60099.key 2048
openssl req -new -key /root/60099.key -out /root/60099.csr -subj '/[email protected]'
# or
openssl req -new -key /root/60099.key -out 60099.csr
# Common Name에 userName 입력하기.
```prompt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:[email protected]
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
`

2-1. Sign Certificate via Openssl Manually

bash

openssl x509 -req -in 60099.csr \
    -CA /etc/kubernetes/pki/ca.crt \
    -CAkey /etc/kubernetes/pki/ca.key \
    -CAcreateserial \
    -out /root/60099.crt \
    -days 500
```prompt
Certificate request self-signature ok
subject=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = [email protected]
`

2-2. Sign Certificate via API

Create and Approve CSR

bash

cat <<EOF > csr.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: [email protected] # ADD
spec:
  groups:
  - system:authenticated
  request: $(cat 60099.csr | base64 -w 0)
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF
k apply -f csr.yaml
### or
cat <<EOF kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: [email protected] # ADD
spec:
  groups:
  - system:authenticated
  request: $(cat 60099.csr | base64 -w 0)
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF

k certificate approve [email protected]
k get csr

Save Certificate

bash

k get csr [email protected] \
    -ojsonpath="{.status.certificate}" | base64 -d > 60099.crt

3. Update Kubeconfig

bash

k config get-clusters 
```prompt
NAME
kubernetes
`
k config set-credentials [email protected] \
    --client-key=60099.key \
    --client-certificate=60099.crt 
```prompt
User "[email protected]" set.
`
k config get-users
```prompt
NAME
[email protected]
kubernetes-admin
`
k config set-context [email protected] \
    --cluster=kubernetes \
    [email protected]
```prompt
Context "[email protected]" created.
`
k config get-contexts 
```prompt
CURRENT   NAME                          CLUSTER      AUTHINFO               NAMESPACE
          [email protected]          kubernetes   [email protected]   
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin       
`

Fredric News

Leave a Reply Cancel reply

Kube-apiserver Hardening: RBAC & Admission

Common

Log analysis to resolve Manifests errors

Audit-Logging

Setup for kube-apiserver

Admission-Controller

NodeRestriction

1. Architecture structure diagram (Mermaid)

2. Main operational flow and architectural components

3. Security Benefits (Why NodeRestriction?)

Refer to official document

Test performance tips

AlwaysPullImages

1. Architecture structure diagram (Mermaid)

2. Main operational flow and architectural components

3. Security Benefits (Why AlwaysPullImages?)

Refer to official document

Test performance tips

ImagePolicyWebhook

1. Architecture structure diagram (Mermaid)

2. Main operational flow and architectural components

3. Security Benefits (Why ImagePolicyWebhook?)

Refer to official document

Test performance tips

PodSecurity

1. Architecture structure diagram (Mermaid)

2. Main operational flow and architectural components

3. Security Benefits (Why PodSecurity?)

Refer to official document

Test performance tips

CSR Management

1. Create CSR with Openssl

2-1. Sign Certificate via Openssl Manually

2-2. Sign Certificate via API

Create and Approve CSR

Save Certificate

3. Update Kubeconfig

Comments

Leave a Reply Cancel reply