Fredric News

Bitbucket Self-hosted Runner 설치 및 온프레미스 연동

💡 핵심 요약 (Executive Summary)

본 가이드는 bitbucket runner 핵심 가이드 및 실전 아키텍처의 핵심 개념, 실전 구성 명령어 및 아키텍처 베스트 프랙티스를 다룹니다. 클라우드 네이티브 및 인프라 엔지니어링 실무에 즉시 활용 가능하도록 체계적으로 정리했습니다.

📑 Table of Contents

On k8s

🔗 https://support.atlassian.com/bitbucket-cloud/docs/deploying-the-docker-based-runner-on-kubernetes/
Sample
Exec
Bitbucket Runner 생성

  1. https://bitbucket.org/\${organization}/\${repository}/admin/pipelines/runners 접근

    1. bitbucket Repository 메인페이지 접근
    2. 좌측 사이드바 하단에 Repository settings 접근
    3. Pipelines 하단의 Runners 접근

  2. Add runner 버튼 클릭

  3. Linux Docker (x86_64) 선택 후 네임 지정하고 Next

  4. Docker 실행문을 복제해둔다.

bash
docker container run -it -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock 
   -v /var/lib/docker/containers:/var/lib/docker/containers:ro 
   -e ACCOUNT_UUID=$(COPY)
   -e REPOSITORY_UUID=$(COPY)
   -e RUNNER_UUID=$(COPY)
   -e RUNTIME_PREREQUISITES_ENABLED=true 
   -e OAUTH_CLIENT_ID=$(COPY)
   -e OAUTH_CLIENT_SECRET=$(COPY)
   -e WORKING_DIRECTORY=/tmp 
   --name runner-2183e7de-fd6f-5468-b96c-3d75fc6c61b4 docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
  1. Label은 bitbucket에서 관리되는 요소이므로 정책이 따로 없으면 Finish 해도 됨
K8s 러너 생성 및 인증연동
  1. 를 참고하여 manifest를 완성한다.
  2. 이전에 복제한 Docker 실행문에서 다음을 카피하여 옮겨적기
    1. ACCOUNT_UUID
    2. REPOSITORY_UUID
    3. RUNNER_UUID
    4. OAUTH_CLIENT_ID
    5. OAUTH_CLIENT_SECRET
  3. Job과 Secret 배포 실행
  4. k apply -f ${runner}.yaml
  5. 파드 정상동작 확인
  6. k get pods -n $ns
연동완료 확인

  • 최초 생성시 runner

  • pod running 후 runner

Troubles
Container: docker-in-docker
  1. network is unreachable
yaml
Could not pull image: error pulling image configuration: Get https://docker-blobs.artifacts.atlassian.com/docker-public/docker/registry/v2/blobs/sha256/9e/9ea04d92255bdf0599c84facc7fd7be2e227bfbc07f511df9647a7ccfebdf59c/data?Expires=1745997666&Signature=4f55myfFz1Uzl5UPHc6kdhtTFQwBudvlxFnvi0Q0L-aFCoCOUVdGEulA6yC0aPHmLzPxvYhnqAY6ApBmXmdr6tvD0xXA2rGyxBfHMfxp2WYPHDEFcEa675ke5n3B-JBaGD~SQAHBsScuu~nh0ufPcO5rpJfK-4NHz0cLMUoQRnAb5i91NHeAvz5d23l6dPBvEXSLM5J~UAybGSIvO0YKNNnIOgpPpFlELiJL0xtn2QTlAydjzJncNu8CYEP0sDXJ8A1Ouw427lk~u3Y~SoFG35gbiubMIJxheFFbzVpEgyt9pAKAPIrvFNDFvjqNMb~KeN7qDXgwHKdVlujghJHO7w__&Key-Pair-Id=K3U3YXJT6R73J6: 
   dial tcp [2600:9000:2891:3400:15:4497:1cc0:93a1]:443: connect: 
   network is unreachable Refer to the Troubleshooting runners document for help resolving the issue.
  • coredns 이슈로 추측됨
    2. failed to adjust OOM score for shim
yaml
time="2025-04-30T06:46:03.754488308Z" level=error msg="Handler for POST /containers/d7d238126da63910fcc0f34b273e2ab5ed66972011e13dee5517bd0d2d55f881/start returned error: io.containerd.runc.v2: failed to adjust OOM score for shim: set shim OOM score: write /proc/416/oom_score_adj: invalid argument\n: exit status 1: unknown"
  • CRI의 docker나 containerd와 DinD image의 런타임의 버전이 다를경우 발생
    • CRI 런타임
yaml
docker version
       Client:
        Version:           26.1.3
        API version:       1.45
        Go version:        go1.22.2
        Git commit:        26.1.3-0ubuntu1~22.04.1
        Built:             Mon Oct 14 21:24:40 2024
        OS/Arch:           linux/amd64
        Context:           default

       Server:
        Engine:
         Version:          26.1.3
         API version:      1.45 (minimum version 1.24)
         Go version:       go1.22.2
         Git commit:       26.1.3-0ubuntu1~22.04.1
         Built:            Mon Oct 14 21:24:40 2024
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.7.24
         GitCommit:
        runc:
         Version:          1.1.12-0ubuntu2~22.04.1
         GitCommit:
        docker-init:
         Version:          0.19.0
         GitCommit:
 - Image 런타임
yaml
- name: docker-in-docker
                     image: docker:26.1.3-dind
Container: runner
  1. failed to create task for container: 400
yaml
Status 400: {"message":"failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: 
   exec: \"/bin/sh\": stat /bin/sh: no such file or directory: unknown"} Refer to the Troubleshooting runners document for help resolving the issue.
  • runner java app에서 생성한 컨테이너는 entrypoint=’/bin/sh’로 지정되어있는데, 생성 요청 이미지(컨테이너)에 해당 바이너리가 포함되지 않은 경우에 발생함.
Features
docker daemon.json 마운트 하기
yaml
#job/${JOB_NAME}-runner
#.spec.template.spec.containers[name=docker-in-docker].volumeMounts
                - name: docker-daemon-cm
                  mountPath: /etc/docker/daemon.json
                  subPath: daemon.json
#.spec.template.spec.volumes
            - name: docker-daemon-cm
              configMap:
                name: docker-daemon
Pipeline Log를 Container Log에 노출 활성화옵션 추가
🔗 https://support.atlassian.com/bitbucket-cloud/docs/configure-logging-in-self-hosted-runners/
yaml
#job/${JOB_NAME}-runner
#.spec.template.spec.containers[name=runner].env
                - name: BITBUCKET_PIPELINES_RUNNER_BUILD_LOG_ENABLED
                  value: 'TRUE'
Custom Logback configuration 사용
  • logback samples
default
xml
<configuration>
      <variable name="BASE_LOG_PATH" value="/tmp" />
      <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
          <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
              <evaluator>
                  <expression>com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException.class.isInstance(throwable)</expression>
              </evaluator>
              <onMatch>DENY</onMatch>
          </filter>
          <encoder>
              <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
              <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
          </encoder>
      </appender>
      <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
          <file>${BASE_LOG_PATH}/runner.log</file>
          <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
              <fileNamePattern>${BASE_LOG_PATH}/runner.%d.%i.log</fileNamePattern>
              <maxHistory>7</maxHistory>
              <maxFileSize>100MB</maxFileSize>
              <totalSizeCap>1GB</totalSizeCap>
          </rollingPolicy>
          <encoder>
              <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
              <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
          </encoder>
      </appender>
      <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
          <resetJUL>true</resetJUL>
      </contextListener>
      <root level="INFO">
          <appender-ref ref="STDOUT" />
          <appender-ref ref="FILE" />
      </root>
      <logger name="com.atlassian.pipelines.runner" level="${BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL:-INFO}" />
      <logger name="build" level="INFO" />
      <logger name="build.service.auth-proxy" level="OFF" />
      <logger name="com.netflix" level="OFF" />
      <logger name="com.atlassian.pipelines.stargate" level="OFF" />
      <logger name="com.github.dockerjava" level="OFF" />
  </configuration>
Json style
xml
<configuration>
    <variable name="BASE_LOG_PATH" value="/tmp" />
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
      <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
        <evaluator>
          <expression>com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException.class.isInstance(throwable)</expression>
        </evaluator>
        <onMatch>DENY</onMatch>
      </filter>
      <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
      <file>${BASE_LOG_PATH}/runner.log</file>
      <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
        <fileNamePattern>${BASE_LOG_PATH}/runner.%d.%i.log</fileNamePattern>
        <maxHistory>7</maxHistory>
        <maxFileSize>100MB</maxFileSize>
        <totalSizeCap>1GB</totalSizeCap>
      </rollingPolicy>
      <encoder>
        <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
        <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
      </encoder>
    </appender>
    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
      <resetJUL>true</resetJUL>
    </contextListener>
    <root level="INFO">
      <appender-ref ref="STDOUT" />
      <appender-ref ref="FILE" />
    </root>
    <logger name="com.atlassian.pipelines.runner" level="${BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL:-INFO}" />
    <logger name="build" level="INFO" />
    <logger name="build.service.auth-proxy" level="OFF" />
    <logger name="com.netflix" level="OFF" />
    <logger name="com.atlassian.pipelines.stargate" level="OFF" />
    <logger name="com.github.dockerjava" level="OFF" />
  </configuration>
  • logback configmap 생성
bash
kubectl create configmap -n bitbucket-runner logback-config --from-file=custom-logback.xml
  • 환경변수 및 볼륨 마운트 추가
yaml
#job/${JOB_NAME}-runner
  #.spec.template.spec.containers[name=runner].env
                  - name: LOGBACK_CONFIGURATION_FILE
                    value: /tmp/custom-logback.xml
  #.spec.template.spec.containers[name=runner].volumeMounts
                  - name: logback-config
                    mountPath: /tmp/custom-logback.xml
                    subPath: custom-logback.xml
                    readOnly: true
  #.spec.template.spec.volumes
              - name: logback-config
                configMap:
                  name: logback-config
Insecure Docker Registry 사용하기
Reference & Guidelines (NOTE)

보안상의 이유로 Pipelines Cloud Runners에서는 Docker 데몬을 구성할 수 없습니다. 하지만 Linux 자체 호스팅 러너에서는 Docker 데몬에 대한 사용자 지정 구성을 허용하여 안전하지 않은 레지스트리를 사용할 수 있습니다.

🔗 https://support.atlassian.com/bitbucket-cloud/kb/configuring-docker-daemon-for-insecure-registries-in-bitbucket-cloud-pipelines/
  • Dockerfile
Reference & Guidelines (NOTE)

해당 서비스에서는 entrypoint overwrite를 지원하지 않으므로, 커스텀 이미지 빌드가 필요합니다.

docker
FROM docker:26.1.3-dind
  ENTRYPOINT [ "sh", "-c", "dockerd-entrypoint.sh $DOCKER_OPTS" ]

  • pipelines.yml

  • self-hosted pipeline using custom Defined service

yaml
pipelines:
      branches:
        "feature/pipeline-test":
          - step:
              name: "on-premise-agent :: Images push & Kubernetes Deploy"
              runs-on:
                - self.hosted
              services:
                - custom-dind
              image: { client input }
              script:
                - docker info
  • Definitions for using a Docker-in-Docker (DIND) image from a private registry.
yaml
# private registry
    definitions:
      services:
        custom-dind:
          image:
            name: { insecure-dind input }
            username: "$private_docker_hub"
            password: "$private_docker_hub_pw"
          type: docker
          variables:
            DOCKER_OPTS: "--insecure-registry={ registry input }"
  • Definitions for using a Docker-in-Docker (DIND) image from a public registry.
yaml
# public registry
    definitions:
      services:
        custom-dind:
          image: { insecure-dind input }
          type: docker
          variables:
            DOCKER_OPTS: "--insecure-registry={ registry input }"

Posted:

/

Category:

Author:

fredric

Tags:

, , , , , , , , , , , , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *