Provide a safer way to apply sandbox with extensions

[vkryachko]

The recommended way to install agent sandbox is with

kubectl apply -f manifest.yaml -f extensions.yaml

This is not idempotent and potentially dangerous for an existing cluster. Plus won't work well with GitOps as the yamls contain a duplicate StatefulSet

Running this command on an existing cluster with agent-sandbox already installed produces the following output

namespace/agent-sandbox-system unchanged
serviceaccount/agent-sandbox-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller unchanged
service/agent-sandbox-controller unchanged
statefulset.apps/agent-sandbox-controller configured   # <--- First applied without --extensions
customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io unchanged
clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller unchanged
customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io unchanged
clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions unchanged
statefulset.apps/agent-sandbox-controller configured # <--- Second applied with --extensions
clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions unchanged

Looking at events in the cluster, we can see that the controller gets recreated multiple times:

0s          Normal   SuccessfulDelete   statefulset/agent-sandbox-controller   delete Pod agent-sandbox-controller-0 in StatefulSet agent-sandbox-controller successful
0s          Normal   Killing            pod/agent-sandbox-controller-0         Stopping container agent-sandbox-controller
0s          Normal   Scheduled          pod/agent-sandbox-controller-0         Successfully assigned agent-sandbox-system/agent-sandbox-controller-0 to gk3-sandbox-test-pool-2-e9de48dc-dh9m
0s          Normal   SuccessfulCreate   statefulset/agent-sandbox-controller   create Pod agent-sandbox-controller-0 in StatefulSet agent-sandbox-controller successful
0s          Normal   Pulled             pod/agent-sandbox-controller-0         Container image "registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0" already present on machine
0s          Normal   Created            pod/agent-sandbox-controller-0         Created container: agent-sandbox-controller
1s          Normal   Started            pod/agent-sandbox-controller-0         Started container agent-sandbox-controller
0s          Normal   SuccessfulDelete   statefulset/agent-sandbox-controller   delete Pod agent-sandbox-controller-0 in StatefulSet agent-sandbox-controller successful
0s          Normal   Killing            pod/agent-sandbox-controller-0         Stopping container agent-sandbox-controller
0s          Normal   SuccessfulCreate   statefulset/agent-sandbox-controller   create Pod agent-sandbox-controller-0 in StatefulSet agent-sandbox-controller successful
0s          Normal   Scheduled          pod/agent-sandbox-controller-0         Successfully assigned agent-sandbox-system/agent-sandbox-controller-0 to gk3-sandbox-test-pool-2-e9de48dc-dh9m
0s          Normal   Pulled             pod/agent-sandbox-controller-0         Container image "registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0" already present on machine
0s          Normal   Created            pod/agent-sandbox-controller-0         Created container: agent-sandbox-controller
0s          Normal   Started            pod/agent-sandbox-controller-0         Started container agent-sandbox-controller
60m         Normal   Pulled             pod/agent-sandbox-controller-0         Container image "registry.k8s.io/agent-sandbox/agent-sandbox-controller:v0.1.0" already present on machine
61m         Normal   Killing            pod/agent-sandbox-controller-0         Stopping container agent-sandbox-controller
60m         Normal   Started            pod/agent-sandbox-controller-0         Started container agent-sandbox-controller
60m         Normal   Created            pod/agent-sandbox-controller-0         Created container: agent-sandbox-controller
61m         Normal   Scheduled          pod/agent-sandbox-controller-0         Successfully assigned agent-sandbox-system/agent-sandbox-controller-0 to gk3-sandbox-test-pool-2-e9de48dc-dh9m

[janetkuo]

This isn't particularly dangerous, as the agent-sandbox controller (as a StatefulSet) is simply undergoing a rolling update. This happens during agent-sandbox upgrades as well.

However, I agree that the installation approach can be improved (e.g., via kustomize) to avoid triggering an extra rolling update when installing extensions.

[janetkuo]

If we use kustomize, extensions.controller.yaml can be used as a kustomization patch (instead of being deployed directly), and it will look like this (only keeping the --extensions part):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: agent-sandbox-controller
  namespace: agent-sandbox-system
spec:
  template:
    spec:
      containers:
      - name: agent-sandbox-controller
        args:
        - "--extensions"

And then the manifest can be applied with kubectl apply -k

[janetkuo]

Alternatively, we can move the controller bit to its own yaml. This requires less changes in the existing manifest structure.

So the new installation looks like this:

# core
kubectl apply -f manifest.yaml -f controller.yaml

# extensions
kubectl apply -f manifest.yaml -f extensions.yaml -f extensions.controller.yaml