こんにちは！ACS事業部の谷合です。
皆大好きGitHub Actionsにおける、GitHub社公式のSelf-hosted runnerであるActions Runner Controller(以降ARC)の紹介をシリーズでお送りしております。
前回は、アーキテクチャをご紹介しました。 techblog.ap-com.co.jp 今回は、動作解説編ということで、インストールおよび基本動作、各種機能をご紹介します。
前提条件
- ARCはKubernestesのPodをSelf-hosted runnerとして利用します。
そのため、何かしらのKubernetesクラスタを用意する必要があります。なお、Kindやminikubeでも可です。
※現在OpenShiftはサポートされていないようです。 - Helm3をインストールしておいてください
ARCインストール
インストールは以下のドキュメントを参考に進めていきます。 docs.github.com
まず、ARCをインストールします。
このとき、--version引数でバージョン指定可能が可能です。
リリース一覧はこちら
$ NAMESPACE="arc-systems"
$ helm install arc \
--namespace "${NAMESPACE}" \
--create-namespace \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller:0.5.0
Digest: sha256:c22c232462846b17b83b2ace59ccf4f8ad84f4eaf3d733d5da0afa2b4ae577ab
NAME: arc
LAST DEPLOYED: Mon Sep 4 18:49:05 2023
NAMESPACE: arc-systems
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing gha-runner-scale-set-controller.
Your release is named arc.
次に、Runner Scale Set用のPersonal Access Token(PAT)またはGitHub AppをSecretとして作成します。
今回はPATを使用します。
$ kubectl create namespace arc-runners namespace/arc-runners created $ kubectl -n arc-runners create secret generic pre-defined-secret --from-literal=github_token='<PAT>' secret/pre-defined-secret created
次に、Runner Scale Setをデプロイします。
以下、各値の解説です。
- INSTALLATION_NAME の値がWorkflowのruns-onで指定する値となる
- セキュリティのべストプラクティスとして、ARC Podとは別のnamespaceにデプロイする
- GITHUB_CONFIG_URLの値はリポジトリ、Organization、または Enterprise の URL に設定する
- GITHUB_PATには、PATやGitHub AppのSecret名を渡す
Secretを作成せずに、直接helmのgithubConfigSecret.github_tokenに渡してもいいが、セキュリティ的には良くありません。 - Actios Runner Controllerと同様に、--version引数でバージョン指定可能
リリース一覧はこちら
$ INSTALLATION_NAME="arc-runner-set"
$ NAMESPACE="arc-runners"
$ GITHUB_CONFIG_URL="<リポジトリURL>"
$ GITHUB_PAT="pre-defined-secret"
$ helm install "${INSTALLATION_NAME}" \
--namespace "${NAMESPACE}" \
--set githubConfigUrl="${GITHUB_CONFIG_URL}" \
--set githubConfigSecret="${GITHUB_PAT}" \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set:0.5.0
Digest: sha256:4ea6f440b348a8dd26942921429939d678361647d3e28e5e10b76757607e157e
NAME: arc-runner-set
LAST DEPLOYED: Mon Sep 4 19:48:33 2023
NAMESPACE: arc-runners
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing gha-runner-scale-set.
Your release is named arc-runner-set.
ここまででARCおよびRunner Scale Setのインストールは完了です。
以下コマンドで各種リソースが作成されていることが確認できるはずです。
$ helm list -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION aks-managed-azure-monitor-logs kube-system 30663 2023-09-04 10:48:29.502477051 +0000 UTC deployed azure-monitor-logs-addon-3.1.12-8cffffef20ff3a0670ed0cdef893877447a44d86 arc arc-systems 1 2023-09-04 19:47:42.823731652 +0900 JST deployed gha-runner-scale-set-controller-0.5.0 0.5.0 arc-runner-set arc-runners 1 2023-09-04 19:48:33.035946252 +0900 JST deployed gha-runner-scale-set-0.5.0 0.5.0 $ kubectl -n arc-systems get po NAME READY STATUS RESTARTS AGE arc-gha-rs-controller-7fd444476b-d8bqv 1/1 Running 0 8m55s arc-runner-set-754b578d-listener 1/1 Running 0 8m3s $ kubectl -n arc-systems get autoscalinglisteners.actions.github.com NAME GITHUB CONFIGURE URL AUTOSCALINGRUNNERSET NAMESPACE AUTOSCALINGRUNNERSET NAME arc-runner-set-754b578d-listener https://github.com/apc-jnytnai0613/arc-test arc-runners arc-runner-set $ kubectl -n arc-runners get autoscalingrunnersets.actions.github.com NAME MINIMUM RUNNERS MAXIMUM RUNNERS CURRENT RUNNERS STATE PENDING RUNNERS RUNNING RUNNERS FINISHED RUNNERS DELETING RUNNERS arc-runner-set 0 0 0 $ kubectl -n arc-runners get ephemeralrunnersets.actions.github.com NAME DESIREDREPLICAS CURRENTREPLICAS PENDING RUNNERS RUNNING RUNNERS FINISHED RUNNERS DELETING RUNNERS arc-runner-set-tcttw 0 0 0
また、GitHub上のSettings → Actions → RunnersにもRunnerが登録されていることが確認できます。
ここまでで、前回解説した以下の動作の3まで完了しました。
- AutoScalingRunnerSet CustomResourceのデプロイ
- AutoScalingRunnerSet ControllerによってEphemeralRunnerSetとAutoScalingListener CustomResourceが順番にデプロイ
- AutoScalingListener ControllerによってAutoScalingListener Podが作成され、GitHubとのロングポーリングが開始されます
- CI/CD jobが開始されると、AutoScalingListener Podに通知され、EphemeralRunnerSetの.spec.replicasを0からターゲットの数まで増やす
- EphemeralRunnerSet Controllerによって.spec.replicasの数だけEphemeralRunner CustomResourceが作成される
- EphemeralRunner Controllerによって、EphemeralRunner Podが作成され、Self-hosted runnerとして動作する
- CI/CD job終了後は自動でGitHub上のRunnerと、EphemeralRunner Podが削除される
次の動作確認にて、4～7の動作を確認します。
動作確認
今回は、以下の簡単なWorkflowで動かしてみます。
runs-onには先ほどデプロイしたRunner Scale Setの名前を指定します。
name: Test workflow on: workflow_dispatch: jobs: test1: runs-on: arc-runner-set steps: - name: Hello world run: echo "Hello" test2: runs-on: arc-runner-set steps: - name: Hello world run: echo "World"
これを動かしてみましょう。
GitHub ActionsではJobごとにRunnerが払い出されます。
ですので、上記Workflowを動かすと、以下のように2つのRunnerが動いているのが確認できます。
また、EphemeralRunnerSetリソースからEphemeralRunnerリソースが生成されていることが確認できます。
この時EphemeralRunnerリソースをWatchすることで、Jobの状況を確認することもできます。
$ kubectl -n arc-runners get ephemeralrunnersets.actions.github.com -w NAME DESIREDREPLICAS CURRENTREPLICAS PENDING RUNNERS RUNNING RUNNERS FINISHED RUNNERS DELETING RUNNERS arc-runner-set-tcttw 0 0 0 arc-runner-set-tcttw 2 0 0 0 arc-runner-set-tcttw 2 2 2 0 arc-runner-set-tcttw 2 2 1 1 arc-runner-set-tcttw 2 2 0 2 arc-runner-set-tcttw 2 1 0 1 arc-runner-set-tcttw 2 1 1 1 arc-runner-set-tcttw 2 2 1 1 arc-runner-set-tcttw 2 1 1 0 arc-runner-set-tcttw 2 2 2 0 arc-runner-set-tcttw 2 2 1 1 arc-runner-set-tcttw 2 2 0 2 arc-runner-set-tcttw 2 0 2 arc-runner-set-tcttw 1 0 1 arc-runner-set-tcttw 0 0 0 $ kubectl -n arc-runners get ephemeralrunners.actions.github.com -w NAME GITHUB CONFIG URL RUNNERID STATUS JOBREPOSITORY JOBWORKFLOWREF WORKFLOWRUNID JOBDISPLAYNAME MESSAGE AGE arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 1s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 1s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 1s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 1s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 1s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Pending 2s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Pending 1s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Running 2s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Running 3s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Running apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test2 14s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Running apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test1 13s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test2 20s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test2 20s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test2 20s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 145 0s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 145 Pending 0s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 145 Running 2s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 145 Running 6s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 145 Running 6s arc-runner-set-tcttw-runner-n8k5v https://github.com/apc-jnytnai0613/arc-test 145 Running 7s arc-runner-set-tcttw-runner-89c22 https://github.com/apc-jnytnai0613/arc-test 143 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test2 30s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test1 48s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test1 48s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test1 48s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 0s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 1s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 1s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 146 1s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 146 Pending 1s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 146 Running 3s arc-runner-set-tcttw-runner-94clm https://github.com/apc-jnytnai0613/arc-test 144 Succeeded apc-jnytnai0613/arc-test apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main 6079531861 test1 51s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 146 Running 6s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 146 Running 6s arc-runner-set-tcttw-runner-7tvzf https://github.com/apc-jnytnai0613/arc-test 146 Running 16s
上記までで、以下の4～7が実行されました。
- AutoScalingRunnerSet CustomResourceのデプロイ
- AutoScalingRunnerSet ControllerによってEphemeralRunnerSetとAutoScalingListener CustomResourceが順番にデプロイ
- AutoScalingListener ControllerによってAutoScalingListener Podが作成され、GitHubとのロングポーリングが開始されます
- CI/CD jobが開始されると、AutoScalingListener Podに通知され、EphemeralRunnerSetの.spec.replicasを0からターゲットの数まで増やす
- EphemeralRunnerSet Controllerによって.spec.replicasの数だけEphemeralRunner CustomResourceが作成される
- EphemeralRunner Controllerによって、EphemeralRunner Podが作成され、Self-hosted runnerとして動作する
- CI/CD job終了後は自動でGitHub上のRunnerと、EphemeralRunner Podが削除される
さらに、AutoScalingListener Podのログからも以下の動作が読み取れます。
2023-09-05T02:28:58Z INFO service process message. {"messageId": 42, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T02:28:58Z INFO service current runner scale set statistics. {"available jobs": 2, "acquired jobs": 0, "assigned jobs": 0, "running jobs": 0, "registered runners": 0, "busy runners": 0, "idle runners": 0}
2023-09-05T02:28:58Z INFO service process batched runner scale set job messages. {"messageId": 42, "batchSize": 2}
2023-09-05T02:28:58Z INFO service job available message received. {"RequestId": 65}
2023-09-05T02:28:58Z INFO service job available message received. {"RequestId": 66}
################
# スケール数のリクエスト
################
2023-09-05T02:28:58Z INFO auto_scaler acquiring jobs. {"request count": 2, "requestIds": "[65 66]"}
2023-09-05T02:28:59Z INFO auto_scaler acquired jobs. {"requested": 2, "acquired": 2}
2023-09-05T02:29:00Z INFO auto_scaler deleted message. {"messageId": 42}
2023-09-05T02:29:00Z INFO service waiting for message...
2023-09-05T02:29:05Z INFO service process message. {"messageId": 43, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T02:29:05Z INFO service current runner scale set statistics. {"available jobs": 0, "acquired jobs": 2, "assigned jobs": 2, "running jobs": 0, "registered runners": 0, "busy runners": 0, "idle runners": 0}
2023-09-05T02:29:05Z INFO service process batched runner scale set job messages. {"messageId": 43, "batchSize": 2}
2023-09-05T02:29:05Z INFO service job assigned message received. {"RequestId": 65}
2023-09-05T02:29:05Z INFO service job assigned message received. {"RequestId": 66}
2023-09-05T02:29:05Z INFO auto_scaler acquiring jobs. {"request count": 0, "requestIds": "[]"}
################
# Jobのアサイン
################
2023-09-05T02:29:05Z INFO service try scale runner request up/down base on assigned job count {"assigned job": 2, "decision": 2, "min": 0, "max": 2147483647, "currentRunnerCount": 0}
################
# EphemeralRunnerSetリソースのreplicasを変更
################
2023-09-05T02:29:05Z INFO KubernetesManager Created merge patch json for EphemeralRunnerSet update {"json": "{\"spec\":{\"replicas\":2}}"}
2023-09-05T02:29:05Z INFO KubernetesManager Ephemeral runner set scaled. {"namespace": "arc-runners", "name": "arc-runner-set-tcttw", "replicas": 2}
2023-09-05T02:29:07Z INFO auto_scaler deleted message. {"messageId": 43}
2023-09-05T02:29:07Z INFO service waiting for message...
2023-09-05T02:29:19Z INFO service process message. {"messageId": 44, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T02:29:19Z INFO service current runner scale set statistics. {"available jobs": 0, "acquired jobs": 2, "assigned jobs": 2, "running jobs": 2, "registered runners": 2, "busy runners": 2, "idle runners": 0}
2023-09-05T02:29:19Z INFO service process batched runner scale set job messages. {"messageId": 44, "batchSize": 2}
2023-09-05T02:29:19Z INFO service job started message received. {"RequestId": 65, "RunnerId": 143}
################
# EphemeralRunnerリソースを起動
################
2023-09-05T02:29:19Z INFO service update job info for runner {"runnerName": "arc-runner-set-tcttw-runner-89c22", "ownerName": "apc-jnytnai0613", "repoName": "arc-test", "workflowRef": "apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main", "workflowRunId": 6079531861, "jobDisplayName": "test2", "requestId": 65}
2023-09-05T02:29:19Z INFO KubernetesManager Created merge patch json for EphemeralRunner status update {"json": "{\"status\":{\"jobDisplayName\":\"test2\",\"jobRepositoryName\":\"apc-jnytnai0613/arc-test\",\"jobRequestId\":65,\"jobWorkflowRef\":\"apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main\",\"workflowRunId\":6079531861}}"}
2023-09-05T02:29:19Z INFO service job started message received. {"RequestId": 66, "RunnerId": 144}
2023-09-05T02:29:19Z INFO service update job info for runner {"runnerName": "arc-runner-set-tcttw-runner-94clm", "ownerName": "apc-jnytnai0613", "repoName": "arc-test", "workflowRef": "apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main", "workflowRunId": 6079531861, "jobDisplayName": "test1", "requestId": 66}
2023-09-05T02:29:19Z INFO KubernetesManager Created merge patch json for EphemeralRunner status update {"json": "{\"status\":{\"jobDisplayName\":\"test1\",\"jobRepositoryName\":\"apc-jnytnai0613/arc-test\",\"jobRequestId\":66,\"jobWorkflowRef\":\"apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main\",\"workflowRunId\":6079531861}}"}
2023-09-05T02:29:19Z INFO auto_scaler acquiring jobs. {"request count": 0, "requestIds": "[]"}
2023-09-05T02:29:21Z INFO auto_scaler deleted message. {"messageId": 44}
2023-09-05T02:29:21Z INFO service waiting for message...
2023-09-05T02:29:30Z INFO service process message. {"messageId": 45, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T02:29:30Z INFO service current runner scale set statistics. {"available jobs": 0, "acquired jobs": 1, "assigned jobs": 1, "running jobs": 1, "registered runners": 2, "busy runners": 1, "idle runners": 0}
各種機能紹介
以下のドキュメント内の機能を見ていきます。 docs.github.com
Runnerの最大、最小数設定
EphemeralRunnerに最大、最小数を設定することができます。
最大値の設定でJob数の多いWorkflowを動かす際のKubernetesリソースの効率化、最小数の設定でGitHub上にIdleなRunnerを常駐させておくことでJob起動のスピードアップに繋げることができます。
設定
まず、以下のhelm upgradeコマンドを使用し、Runner Scale Setの更新を行います。
この時、maxRunnersおよびminRunnersを設定します。
$ INSTALLATION_NAME="arc-runner-set"
$ NAMESPACE="arc-runners"
$ GITHUB_CONFIG_URL="https://github.com/apc-jnytnai0613/arc-test"
$ GITHUB_PAT="pre-defined-secret"
$ helm upgrade "${INSTALLATION_NAME}" \
--namespace "${NAMESPACE}" \
--set githubConfigUrl="${GITHUB_CONFIG_URL}" \
--set githubConfigSecret="${GITHUB_PAT}" \
--set maxRunners=5 \
--set minRunners=2 \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set:0.5.0
Digest: sha256:4ea6f440b348a8dd26942921429939d678361647d3e28e5e10b76757607e157e
Release "arc-runner-set" has been upgraded. Happy Helming!
NAME: arc-runner-set
LAST DEPLOYED: Tue Sep 5 16:07:24 2023
NAMESPACE: arc-runners
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Thank you for installing gha-runner-scale-set.
Your release is named arc-runner-set.
ARC更新後、以下のようにEphemeralRunnerSetの最小値が確認できます。
$ kubectl -n arc-runners get autoscalingrunnersets.actions.github.com -ojson | jq -r '.items[].spec' | grep Runners "maxRunners": 5, "minRunners": 2, $ kubectl -n arc-runners get ephemeralrunnersets.actions.github.com -w NAME DESIREDREPLICAS CURRENTREPLICAS PENDING RUNNERS RUNNING RUNNERS FINISHED RUNNERS DELETING RUNNERS arc-runner-set-tcttw 2 2 0 2 $ kubectl -n arc-runners get ephemeralrunners.actions.github.com NAME GITHUB CONFIG URL RUNNERID STATUS JOBREPOSITORY JOBWORKFLOWREF WORKFLOWRUNID JOBDISPLAYNAME MESSAGE AGE arc-runner-set-tcttw-runner-b947l https://github.com/apc-jnytnai0613/arc-test 157 Running 59m arc-runner-set-tcttw-runner-sn67d https://github.com/apc-jnytnai0613/arc-test 158 Running 58m
さらにGitHub上でもIdleなRunnerが確認できます。
動作確認
今回は以下のWorkflowを使用し、動作確認を行います。
以下のWorkflowは6つのJobから成りますので、EphemeralRunnerの最大数5つを超えた場合、どういう動作となるかを見ていきます。
name: Test workflow on: workflow_dispatch: jobs: test1: runs-on: arc-runner-set steps: - name: step1 run: echo "step1" test2: runs-on: arc-runner-set steps: - name: step2 run: echo "step2" test3: runs-on: arc-runner-set steps: - name: step3 run: echo "step3" test4: runs-on: arc-runner-set steps: - name: step4 run: echo "step4" test5: runs-on: arc-runner-set steps: - name: step5 run: echo "step5" test6: runs-on: arc-runner-set steps: - name: step6 run: echo "step6"
上記Workflowを動かすと、以下のようにEphemeralRunnerSetのreplicasが2→5に変化し、Workflow終了後は2に収束していることが確認できます。
$ kubectl -n arc-runners get ephemeralrunnersets.actions.github.com -w NAME DESIREDREPLICAS CURRENTREPLICAS PENDING RUNNERS RUNNING RUNNERS FINISHED RUNNERS DELETING RUNNERS arc-runner-set-tcttw 2 2 0 2 arc-runner-set-tcttw 5 2 0 2 arc-runner-set-tcttw 5 5 3 2 arc-runner-set-tcttw 5 5 2 3 arc-runner-set-tcttw 5 5 1 4 arc-runner-set-tcttw 5 5 0 5 arc-runner-set-tcttw 5 4 0 4 arc-runner-set-tcttw 5 5 1 4 arc-runner-set-tcttw 5 4 1 3 arc-runner-set-tcttw 5 5 2 3 arc-runner-set-tcttw 5 5 1 4 arc-runner-set-tcttw 5 5 0 5 arc-runner-set-tcttw 4 5 0 5 arc-runner-set-tcttw 4 4 0 4 arc-runner-set-tcttw 4 3 0 3 arc-runner-set-tcttw 4 3 1 3 arc-runner-set-tcttw 4 4 1 3 arc-runner-set-tcttw 2 4 1 3 arc-runner-set-tcttw 2 3 0 3 arc-runner-set-tcttw 2 2 0 2 arc-runner-set-tcttw 2 1 0 1 arc-runner-set-tcttw 2 1 1 1 arc-runner-set-tcttw 2 2 1 1 arc-runner-set-tcttw 2 1 1 0 arc-runner-set-tcttw 2 2 2 0 arc-runner-set-tcttw 2 2 1 1 arc-runner-set-tcttw 2 2 0 2
AutoScalingListener Podのログも見てみましょう。
うんうん、ログを見るとどんな動作しているか分かりやすいですね。
Workflow起動時にmaxRunnersまでScale Outし、maxRunners内のRunnerでJobを処理した後は、maxRunnersからはみ出たJobを新たにRunnerを払い出して処理しているのが見えます。また、全Job終了後はminRunnersまでScale Inしています。
2023-09-05T07:24:49Z INFO service process message. {"messageId": 48, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T07:24:49Z INFO service current runner scale set statistics. {"available jobs": 0, "acquired jobs": 6, "assigned jobs": 6, "running jobs": 2, "registered runners": 2, "busy runners": 2, "idle runners": 0}
2023-09-05T07:24:49Z INFO service process batched runner scale set job messages. {"messageId": 48, "batchSize": 8}
2023-09-05T07:24:49Z INFO service job started message received. {"RequestId": 68, "RunnerId": 148}
2023-09-05T07:24:49Z INFO service update job info for runner {"runnerName": "arc-runner-set-tcttw-runner-7d9q7", "ownerName": "apc-jnytnai0613", "repoName": "arc-test", "workflowRef": "apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main", "workflowRunId": 6081488839, "jobDisplayName": "test3", "requestId": 68}
2023-09-05T07:24:49Z INFO KubernetesManager Created merge patch json for EphemeralRunner status update {"json": "{\"status\":{\"jobDisplayName\":\"test3\",\"jobRepositoryName\":\"apc-jnytnai0613/arc-test\",\"jobRequestId\":68,\"jobWorkflowRef\":\"apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main\",\"workflowRunId\":6081488839}}"}
2023-09-05T07:24:49Z INFO service job started message received. {"RequestId": 67, "RunnerId": 147}
2023-09-05T07:24:49Z INFO service update job info for runner {"runnerName": "arc-runner-set-tcttw-runner-q9vhc", "ownerName": "apc-jnytnai0613", "repoName": "arc-test", "workflowRef": "apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main", "workflowRunId": 6081488839, "jobDisplayName": "test1", "requestId": 67}
2023-09-05T07:24:49Z INFO KubernetesManager Created merge patch json for EphemeralRunner status update {"json": "{\"status\":{\"jobDisplayName\":\"test1\",\"jobRepositoryName\":\"apc-jnytnai0613/arc-test\",\"jobRequestId\":67,\"jobWorkflowRef\":\"apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main\",\"workflowRunId\":6081488839}}"}
2023-09-05T07:24:49Z INFO service job assigned message received. {"RequestId": 69}
2023-09-05T07:24:49Z INFO service job assigned message received. {"RequestId": 70}
2023-09-05T07:24:49Z INFO service job assigned message received. {"RequestId": 68}
2023-09-05T07:24:49Z INFO service job assigned message received. {"RequestId": 67}
2023-09-05T07:24:49Z INFO service job assigned message received. {"RequestId": 71}
2023-09-05T07:24:49Z INFO service job assigned message received. {"RequestId": 72}
################
# EphemeralRunnerSetリソースのreplicasをmaxRunnersまで変更
################
2023-09-05T07:24:49Z INFO auto_scaler acquiring jobs. {"request count": 0, "requestIds": "[]"}
2023-09-05T07:24:49Z INFO service try scale runner request up/down base on assigned job count {"assigned job": 6, "decision": 5, "min": 2, "max": 5, "currentRunnerCount": 2}
2023-09-05T07:24:49Z INFO KubernetesManager Created merge patch json for EphemeralRunnerSet update {"json": "{\"spec\":{\"replicas\":5}}"}
2023-09-05T07:24:49Z INFO KubernetesManager Ephemeral runner set scaled. {"namespace": "arc-runners", "name": "arc-runner-set-tcttw", "replicas": 5}
:
################
# 5つのJob処理後、残り1つを処理
################
2023-09-05T07:25:10Z INFO service process message. {"messageId": 51, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T07:25:10Z INFO service current runner scale set statistics. {"available jobs": 0, "acquired jobs": 1, "assigned jobs": 1, "running jobs": 1, "registered runners": 1, "busy runners": 1, "idle runners": 0}
2023-09-05T07:25:10Z INFO service process batched runner scale set job messages. {"messageId": 51, "batchSize": 2}
2023-09-05T07:25:10Z INFO service job started message received. {"RequestId": 72, "RunnerId": 153}
2023-09-05T07:25:10Z INFO service update job info for runner {"runnerName": "arc-runner-set-tcttw-runner-9fz6l", "ownerName": "apc-jnytnai0613", "repoName": "arc-test", "workflowRef": "apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main", "workflowRunId": 6081488839, "jobDisplayName": "test6", "requestId": 72}
2023-09-05T07:25:10Z INFO KubernetesManager Created merge patch json for EphemeralRunner status update {"json": "{\"status\":{\"jobDisplayName\":\"test6\",\"jobRepositoryName\":\"apc-jnytnai0613/arc-test\",\"jobRequestId\":72,\"jobWorkflowRef\":\"apc-jnytnai0613/arc-test/.github/workflows/blank.yml@refs/heads/main\",\"workflowRunId\":6081488839}}"}
2023-09-05T07:25:10Z INFO service job completed message received. {"RequestId": 70, "Result": "succeeded", "RunnerId": 150, "RunnerName": "arc-runner-set-tcttw-runner-xw5ms"}
2023-09-05T07:25:10Z INFO auto_scaler acquiring jobs. {"request count": 0, "requestIds": "[]"}
################
# すべてのJob処理後、EphemeralRunnerSetリソースのreplicasをminRunnersまで変更
################
2023-09-05T07:25:10Z INFO service try scale runner request up/down base on assigned job count {"assigned job": 1, "decision": 2, "min": 2, "max": 5, "currentRunnerCount": 4}
2023-09-05T07:25:10Z INFO KubernetesManager Created merge patch json for EphemeralRunnerSet update {"json": "{\"spec\":{\"replicas\":2}}"}
2023-09-05T07:25:10Z INFO KubernetesManager Ephemeral runner set scaled. {"namespace": "arc-runners", "name": "arc-runner-set-tcttw", "replicas": 2}
2023-09-05T07:25:12Z INFO auto_scaler deleted message. {"messageId": 51}
2023-09-05T07:25:12Z INFO service waiting for message...
2023-09-05T07:25:16Z INFO service process message. {"messageId": 52, "messageType": "RunnerScaleSetJobMessages"}
2023-09-05T07:25:16Z INFO service current runner scale set statistics. {"available jobs": 0, "acquired jobs": 0, "assigned jobs": 0, "running jobs": 0, "registered runners": 2, "busy runners": 0, "idle runners": 0}
2023-09-05T07:25:16Z INFO service process batched runner scale set job messages. {"messageId": 52, "batchSize": 3}
2023-09-05T07:25:16Z INFO service job completed message received. {"RequestId": 69, "Result": "succeeded", "RunnerId": 149, "RunnerName": "arc-runner-set-tcttw-runner-nnth5"}
2023-09-05T07:25:16Z INFO service job completed message received. {"RequestId": 71, "Result": "succeeded", "RunnerId": 151, "RunnerName": "arc-runner-set-tcttw-runner-pt5b6"}
2023-09-05T07:25:16Z INFO service job completed message received. {"RequestId": 72, "Result": "succeeded", "RunnerId": 153, "RunnerName": "arc-runner-set-tcttw-runner-9fz6l"}
2023-09-05T07:25:16Z INFO auto_scaler acquiring jobs. {"request count": 0, "requestIds": "[]"}
2023-09-05T07:25:18Z INFO auto_scaler deleted message. {"messageId": 52}
2023-09-05T07:25:18Z INFO service waiting for message...
ARCでの、Workflow内Containerの使用
GitHub-hosted runnerでは、Runner内にdockerがプリインストールされているため、Workflowでdockerコンテナを何の問題もなく使えます。
しかし、Self-hosted runnerでは、そうはいきません。Self-hosted runner内にdockerをあらかじめインストールしておく必要があります。
更に厄介なのが、コンテナとして動かすSelf-hosted runnerで、Workflowでdockerコンテナを動かす場合です。
コンテナRunner内でコンテナを動かす場合は、docker.sockに接続してDocker Daemonと通信を行う必要がありますが、後述するある工夫をしないと通信ができません。
ARCの場合は以下のエラーが発生しました。
Checking docker version
/usr/bin/docker version --format '{{.Server.APIVersion}}'
'
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Error: Exit code 1 returned from process: file name '/usr/bin/docker', arguments 'version --format '{{.Server.APIVersion}}''.
この問題を解決するに当たり、コンテナのRunner内で更にコンテナを動かす場合は、以下の方法でdocker.sockと通信を行えます。
- ホスト上のdocker.sockをコンテナRunnerにマウントするDooD（Docker outside of Docker）
- コンテナRunner内にさらにDockerを起動させるDinD（Docker in Docker）
- actions/runner-container-hooks の利用 github.com
こちらについて以下の記事で非常に詳しく解説されています。 zenn.dev
上記の内、ARCでは、Dindとactions/runner-container-hooksがサポートされています。
また、ARCで使用されるコンテナイメージにはactions/runner-container-hooksが組み込まれています。
RUN curl -f -L -o runner-container-hooks.zip https://github.com/actions/runner-container-hooks/releases/download/v${RUNNER_CONTAINER_HOOKS_VERSION}/actions-runner-hooks-k8s-${RUNNER_CONTAINER_HOOKS_VERSION}.zip \
&& unzip ./runner-container-hooks.zip -d ./k8s \
&& rm runner-container-hooks.zip
ARCでは、DinDをDocker-in-Dockerモード、actions/runner-container-hooksをKubernetesモードと称して実装しておりますので、この２つを検証してみたいと思います。
Docker-in-Dockerモード
設定
Docker-in-Dockerモードではまず、values.yamlに以下の定義を記載します。
containerMode: type: "dind"
それでは、Runner Scale Setの更新を行いましょう。
この時、-f values.yaml を付与し、先ほどのDocker-in-Dockerモードの定義を渡してあげます。
$ INSTALLATION_NAME="arc-runner-set"
$ NAMESPACE="arc-runners"
$ GITHUB_CONFIG_URL="https://github.com/apc-jnytnai0613/arc-test"
$ GITHUB_PAT="pre-defined-secret"
$ helm upgrade "${INSTALLATION_NAME}" \
--namespace "${NAMESPACE}" \
--set githubConfigUrl="${GITHUB_CONFIG_URL}" \
--set githubConfigSecret="${GITHUB_PAT}" \
--set maxRunners=5 \
--set minRunners=2 \
-f values.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set:0.5.0
Digest: sha256:4ea6f440b348a8dd26942921429939d678361647d3e28e5e10b76757607e157e
Release "arc-runner-set" has been upgraded. Happy Helming!
NAME: arc-runner-set
LAST DEPLOYED: Tue Sep 5 20:17:59 2023
NAMESPACE: arc-runners
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
Thank you for installing gha-runner-scale-set.
Your release is named arc-runner-set.
更新後、EphemeralRunnerの定義を確認すると、以下のコンテナが追加されていることが確認できます。
- init コンテナ
- runner コンテナ
- dind コンテナ
また、Docker-in-Dockerモードに必要なVoumeや環境変数が追加されています。
$ k -n arc-runners get ephemeralrunners arc-runner-set-fj995-runner-ldkhn -ojson | jq -j '.spec' { "githubConfigSecret": "pre-defined-secret", "githubConfigUrl": "https://github.com/apc-jnytnai0613/arc-test", "metadata": {}, "runnerScaleSetId": 3, "spec": { "containers": [ { "command": [ "/home/runner/run.sh" ], "env": [ { "name": "DOCKER_HOST", "value": "tcp://localhost:2376" }, { "name": "DOCKER_TLS_VERIFY", "value": "1" }, { "name": "DOCKER_CERT_PATH", "value": "/certs/client" }, { "name": "RUNNER_WAIT_FOR_DOCKER_IN_SECONDS", "value": "120" } ], "image": "ghcr.io/actions/actions-runner:latest", "name": "runner", "resources": {}, "volumeMounts": [ { "mountPath": "/home/runner/_work", "name": "work" }, { "mountPath": "/certs/client", "name": "dind-cert", "readOnly": true } ] }, { "image": "docker:dind", "name": "dind", "resources": {}, "securityContext": { "privileged": true }, "volumeMounts": [ { "mountPath": "/home/runner/_work", "name": "work" }, { "mountPath": "/certs/client", "name": "dind-cert" }, { "mountPath": "/home/runner/externals", "name": "dind-externals" } ] } ], "initContainers": [ { "args": [ "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/" ], "command": [ "cp" ], "image": "ghcr.io/actions/actions-runner:latest", "name": "init-dind-externals", "resources": {}, "volumeMounts": [ { "mountPath": "/home/runner/tmpDir", "name": "dind-externals" } ] } ], "serviceAccountName": "arc-runner-set-gha-rs-no-permission", "volumes": [ { "emptyDir": {}, "name": "dind-cert" }, { "emptyDir": {}, "name": "dind-externals" }, { "emptyDir": {}, "name": "work" } ] } }
動作確認
以下のような、コンテナを使用するWorkflowを実行してみましょう。
name: Test workflow on: workflow_dispatch: jobs: test1: runs-on: arc-runner-set container: image: ubuntu steps: - name: step1 run: echo "step1"
Docker-in-Dockerモード有効化前と打って変わって、以下のように正常にPullできているのが確認できるはずです。
Checking docker version
/usr/bin/docker version --format '{{.Server.APIVersion}}'
'1.43'
Docker daemon API version: '1.43'
/usr/bin/docker version --format '{{.Client.APIVersion}}'
'1.41'
Docker client API version: '1.41'
Clean up resources from previous jobs
/usr/bin/docker ps --all --quiet --no-trunc --filter "label=b1cbc5"
/usr/bin/docker network prune --force --filter "label=b1cbc5"
Create local container network
/usr/bin/docker network create --label b1cbc5 github_network_9f3a4c0a58c645dfabc9fc8b1ed91e86
19dd9111db741a9beedf681c066afad444425ef109920f53b77809ae0730e122
Starting job container
/usr/bin/docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
445a6a12be2b: Pulling fs layer
445a6a12be2b: Verifying Checksum
445a6a12be2b: Download complete
445a6a12be2b: Pull complete
Digest: sha256:aabed3296a3d45cede1dc866a24476c4d7e093aa806263c27ddaadbdce3c1054
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest
/usr/bin/docker create --name 1cf790cecf4a4c6abd81259c8bc8f457_ubuntu_6fd18d --label b1cbc5 --workdir /__w/arc-test/arc-test --network github_network_9f3a4c0a58c645dfabc9fc8b1ed91e86 -e "HOME=/github/home" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/_work":"/__w" -v "/home/runner/externals":"/__e":ro -v "/home/runner/_work/_temp":"/__w/_temp" -v "/home/runner/_work/_actions":"/__w/_actions" -v "/home/runner/_work/_tool":"/__w/_tool" -v "/home/runner/_work/_temp/_github_home":"/github/home" -v "/home/runner/_work/_temp/_github_workflow":"/github/workflow" --entrypoint "tail" ubuntu "-f" "/dev/null"
a6cbf383a7e3dc3e81752d46aab155b92e76b5cccbc68b54d75f749842ca902a
/usr/bin/docker start a6cbf383a7e3dc3e81752d46aab155b92e76b5cccbc68b54d75f749842ca902a
a6cbf383a7e3dc3e81752d46aab155b92e76b5cccbc68b54d75f749842ca902a
/usr/bin/docker ps --all --filter id=a6cbf383a7e3dc3e81752d46aab155b92e76b5cccbc68b54d75f749842ca902a --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
a6cbf383a7e3dc3e81752d46aab155b92e76b5cccbc68b54d75f749842ca902a Up Less than a second
/usr/bin/docker inspect --format "{{range .Config.Env}}{{println .}}{{end}}" a6cbf383a7e3dc3e81752d46aab155b92e76b5cccbc68b54d75f749842ca902a
HOME=/github/home
GITHUB_ACTIONS=true
CI=true
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Waiting for all services to be ready
Kubernetesモード
設定
Kubernetesモードは若干設定が多めです。
以下の設定をvalues.yamlに書きましょう。
まず、containerModeにkubernetesを指定して、Kubernetesモードを有効化します。
次に、Kubernetesモードで利用するするPVC定義を書きます。この時、StorageClassも指定する必要があるため、事前に定義しておくか、Managed Kubernetesを利用中の場合は、そちらで事前定義されたStorageClassを指定ください。
containerMode: type: "kubernetes" kubernetesModeWorkVolumeClaim: accessModes: ["ReadWriteOnce"] storageClassName: "managed" resources: requests: storage: 1Gi
続けて、同values.yamlに以下定義を追記します。
template: spec: containers: - name: runner image: ghcr.io/actions/actions-runner:latest command: ["/home/runner/run.sh"] env: - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER value: "false"
なお、上記定義を使用してWorkflowを実行すると、Access to the path /home/runner/_work/_tool is deniedのエラーが発生することがあります。
この場合、以下ドキュメントを参考に.spec.securityContext.fsGroupにコンテナのGIDを設定するか、initContainersを追加してください。
Troubleshooting Actions Runner Controller errors - GitHub Docs
template: spec: initContainers: - name: kube-init image: ghcr.io/actions/actions-runner:latest command: ["sudo", "chown", "-R", "1001:1001", "/home/runner/_work"] volumeMounts: - name: work mountPath: /home/runner/_work containers: - name: runner image: ghcr.io/actions/actions-runner:latest command: ["/home/runner/run.sh"] env: - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER value: "false"
次に、Runner Scale Setの更新を行いましょう。
$ INSTALLATION_NAME="arc-runner-set"
$ NAMESPACE="arc-runners"
$ GITHUB_CONFIG_URL="https://github.com/apc-jnytnai0613/arc-test"
$ GITHUB_PAT="pre-defined-secret"
$ helm upgrade "${INSTALLATION_NAME}" \
--namespace "${NAMESPACE}" \
--set githubConfigUrl="${GITHUB_CONFIG_URL}" \
--set githubConfigSecret="${GITHUB_PAT}" \
--set maxRunners=5 \
--set minRunners=2 \
-f values.yaml \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
Pulled: ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set:0.5.0
Digest: sha256:4ea6f440b348a8dd26942921429939d678361647d3e28e5e10b76757607e157e
Release "arc-runner-set" has been upgraded. Happy Helming!
NAME: arc-runner-set
LAST DEPLOYED: Tue Sep 5 21:24:10 2023
NAMESPACE: arc-runners
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
Thank you for installing gha-runner-scale-set.
Your release is named arc-runner-set.
更新を行うと、PVCが作成されます。
$ k -n arc-runners get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE arc-runner-set-xnb8j-runner-9t2bc-work Bound pvc-75ebb5e5-8272-49ae-b62d-a2a2db02936a 1Gi RWO managed 24s arc-runner-set-xnb8j-runner-ppft5-work Bound pvc-2e173f9b-95e9-49ba-acc1-b0eb48f4e068 1Gi RWO managed 24s
EphemeralRunnerの定義を確認すると、runner-container-hooks用のVoumeや環境変数が追加されています。
$ k -n arc-runners get ephemeralrunners arc-runner-set-d8kph-runner-wmm6f -ojson | jq -j '.spec' { "githubConfigSecret": "pre-defined-secret", "githubConfigUrl": "https://github.com/apc-jnytnai0613/arc-test", "metadata": {}, "runnerScaleSetId": 5, "spec": { "containers": [ { "command": [ "/home/runner/run.sh" ], "env": [ { "name": "ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER", "value": "false" }, { "name": "ACTIONS_RUNNER_CONTAINER_HOOKS", "value": "/home/runner/k8s/index.js" }, { "name": "ACTIONS_RUNNER_POD_NAME", "valueFrom": { "fieldRef": { "fieldPath": "metadata.name" } } } ], "image": "ghcr.io/actions/actions-runner:latest", "name": "runner", "resources": {}, "volumeMounts": [ { "mountPath": "/home/runner/_work", "name": "work" } ] } ], "initContainers": [ { "command": [ "sudo", "chown", "-R", "1001:1001", "/home/runner/_work" ], "image": "ghcr.io/actions/actions-runner:latest", "name": "kube-init", "resources": {}, "volumeMounts": [ { "mountPath": "/home/runner/_work", "name": "work" } ] } ], "serviceAccountName": "arc-runner-set-gha-rs-kube-mode", "volumes": [ { "ephemeral": { "volumeClaimTemplate": { "metadata": {}, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "1Gi" } }, "storageClassName": "managed" } } }, "name": "work" } ] } }
動作確認
先ほどのDocker-in-Dockerモードで利用したWorkflowを実行すると、以下のようにrunner-container-hooksが実行されてことが確認できます。
Run '/home/runner/k8s/index.js'
shell: /home/runner/externals/node16/bin/node {0}
さいごに
ARCにはほかにもHA構成や、Organizationでの分離、Metricsなどもサポートされています。
シリーズの中でどれか取り上げれればと思います。
是非皆さんもARCを動かして、さまざまな機能を楽しんでみてください。
次回はいよいよARCのコード解説つまりDeepDiveです！お楽しみに！！
2023/09/19追記: 第三回コード解説 前編は以下を参照ください。 techblog.ap-com.co.jp
ACS事業部のご紹介
私達ACS事業部はAzure・AKSなどのクラウドネイティブ技術を活用した内製化やGitHub Enterpriseの導入のご支援をしております。
www.ap-com.co.jp
www.ap-com.co.jp
また、一緒に働いていただける仲間も募集中です！
今年もまだまだ組織規模拡大中なので、ご興味持っていただけましたらぜひお声がけください。
www.ap-com.co.jp
