Label and Annotation propagation to sandbox pods

[barney-s]

We need some clarity on the propagation of labels and annotations to pod from these sources:

1. Sandbox
2. Sandbox.spec.podtemplate
3. SandboxClaim
4. SandboxTemplate
5. Warmpool

This PR is for discussing the options wholistically. May be we will create a KEP or come up with some recommendations.

related PR:
#157
#39

[aditya-shantanu]

I'm thinking in in terms of the hierarchy/level of different constructs. In my mind, we have

0: SandboxTemplate
1: Sandbox (or Warmpool)
2: SandboxClaim

There needs to be a "merge" operation and anything defined at a higher level can override the same label/annotation at the lower layer.

(Warmpool is just a collection of Sandboxes so should be at the same level)

How are Sandbox.spec.podtemplate and Sandbox different ?

[janetkuo]

Pods should be created from a Pod Template like cookies from a cookie cutter, so pod labels/annotations should match what's specified in the Pod Template. This must happen.

Things to think about around propagating more labels/annotations to pods:

In Kubernetes convention, controllers like Deployments or StatefulSets don't propagate their own labels/annotations to their pods (propagating to history resources like ReplicaSets is fine), so this will be inconsistent with the convention.

Additionally, when metadata is propagated from multiple sources, we need a clearly defined and documented precedence order for merging them, which adds complexity. This might cause confusion or unexpected results from users.

We also want to avoid accidental propagation or unwanted metadata from parent resources to pods, which can have unintended side-effects, e.g., with monitoring or querying tools that select pods based on labels.

[peterzhongyi]

In Kubernetes convention, controllers like Deployments or StatefulSets don't propagate their own labels/annotations to their pods (propagating to history resources like ReplicaSets is fine), so this will be inconsistent with the convention.

Since the convention is to specify the pod labels in the pod template, and currently the sandbox claim is only referring to a sandbox template, I think we can add an additional field to the sandbox claim CRD called "podLabels", which will add additional labels aside from the pod labels specified in the sandbox template. WDYT?

Also, please take a look at #162 because if it gets implemented, then sandbox claim CRD can also have a "sandboxLabels" field that add additional labels to the created sandbox.

This way, the sandbox claim's labels themselves are not propagated to anywhere.

[janetkuo]

@peterzhongyi Making pod label propagation explicit is a good direction.

To make the purpose of the new field absolutely clear, I suggest we name it additionalPodLabels to make it obvious to the user that these are additional labels that will be combined with the base labels defined in the SandboxTemplate (instead of replacing).

Regarding the merge strategy, I'm leaning towards a safer approach that we should not allow overrides, by returning an error when there's a conflict. The trade-off is that users would not be able to use this field to change a label from the template, which is a reasonable trade-off for the added safety and predictability.

Before we implement the change, it would be helpful to understand the specific use cases for this feature. Could you or someone (@brandonroyal?) provide a concrete CUJ for this feature?