How-To: Set Kubernetes metadata

Background

Kubernetes Labels are key/value pairs attached to Kubernetes objects and used to identify groups of related resources by organizational structure, release, process etc. Labels are descriptive in nature and can be queried.

Kubernetes Annotations attach non-identifying information to Kubernetes objects. They are used to provide additional information to users and can also be used for operational purposes. Annotations are used to represent behavior that can be leveraged by tools and libraries and often they are not human readable or queried.

Kubernetes metadata extension

Radius enables you to retain or use your own defined tagging scheme for Kubernetes resources using Kubernetes labels and annotations. This enables users to incrementally adopt Radius for microservices built in the Kubernetes ecosystem using Kubernetes metadata concepts.

You can set labels and annotations on an environment, application, or container using the Kubernetes metadata extension. The Kubernetes objects output from your resources (Deployments, Services, etc.) will now have the defined metadata.

• Environment
• Application
• Container

resource env 'Applications.Core/environments@2023-10-01-preview' = {
  name: 'myenv'
  properties: {
    compute: {
      kind: 'kubernetes'
      namespace: 'my-ns'
    }
    extensions: [
      {
        kind: 'kubernetesMetadata'
        labels: {
          'team.key1': 'envValue1'
          'team.key2': 'envValue2'
        }
      }
    ]
  }
}

resource app 'Applications.Core/applications@2023-10-01-preview' = {
  name: 'myapp'
  properties: {
    environment: env.id
    extensions: [
      {
        kind: 'kubernetesMetadata'
        labels: {
          'team.key1': 'appValue1'
          'team.key2': 'appValue2'
          'team.contact.name': 'Operations'
          'team.contact.alias': 'ops'
        }
        annotations: {
          'prometheus.io/port': '9090'
          'prometheus.io/scrape': 'true'
        }
      }
    ]
  }
}

resource frontend 'Applications.Core/containers@2023-10-01-preview' = {
  name: 'frontend'
  properties: {
    application: app.id
    container: {
      image: 'myregistry/mycontainer:latest'
    }
    extensions: [
      {
        kind: 'kubernetesMetadata'
        labels: {
          'team.key2': 'containerValue2'
          'team.contact.name': 'Frontend'
          'team.contact.alias': 'frontend-eng'
        }
      }
    ]
  }
}

Cascading metadata

Kubernetes metadata can be applied at the environment, application, or container layers. Metadata cascades down from the environment to the application to the containers, gateway and route resources. For example, you can set labels and annotations at an environment level and all containers within the environment will gain these labels and annotation, without the need for an explicit extension on the containers. The following resources will gain the Kubernetes metadata for their output resources from labels, annotations set at Environment, Application levels:

• Applications.Core/containers
• Applications.Core/gateways

Metadata processing order

Labels and annotations are processed in the following order, combining the keys/values at each level:

1. Environment
2. Application
3. Container

Conflicts and overrides

In the case where layers have conflicting keys (i.e. Application and Container both specify a myapp.team.name label), the last level to process wins out and overrides other values (container). The metadata specified on the container will override the metadata specified on the application or environment.

Reserved keys

Certain labels/annotations have special uses to Radius internally and are not allowed to be overridden by user. Labels/Annotations with keys that have a prefix : radapp.io/ will be ignored during processing.

Extension processing order

Other extensions may set Kubernetes metadata. For example, the daprSidecar extension sets the dapr.io/enabled annotation, as well as some others. This may cause issues in the case of conflicts.

The order in which extensions are executed is as follows, from first to last:

1. Dapr sidecar extension
2. Manual scale extension
3. Kubernetes metadata extension

Conflicts

When labels/annotation have the same set of key(s) added by two or more extensions, the final value is determined by the last extension that is processed.

Example

Let’s take an example of an application with some labels and annotations. This example shows some generic ‘keys’, as well as an example of how an organization may use labels to track organization contact information inside labels for troubleshooting scenarios:

resource app 'Applications.Core/applications@2023-10-01-preview' = {
  name: 'myapp'
  properties: {
    environment: env.id
    extensions: [
      {
        kind: 'kubernetesMetadata'
        labels: {
          'team.key1': 'appValue1'
          'team.key2': 'appValue2'
          'team.contact.name': 'Operations'
          'team.contact.alias': 'ops'
        }
        annotations: {
          'prometheus.io/port': '9090'
          'prometheus.io/scrape': 'true'
        }
      }
    ]
  }
}

All resources within this application with Kubernetes outputs will gain these labels. Let’s now take a look at a frontend container, where the frontend team has overridden some of the values:

resource frontend 'Applications.Core/containers@2023-10-01-preview' = {
  name: 'frontend'
  properties: {
    application: app.id
    container: {
      image: 'myregistry/mycontainer:latest'
    }
    extensions: [
      {
        kind: 'kubernetesMetadata'
        labels: {
          'team.key2': 'containerValue2'
          'team.contact.name': 'Frontend'
          'team.contact.alias': 'frontend-eng'
        }
      }
    ]
  }
}

When this file is deployed, the metadata on the frontend deployment is:

$ kubectl describe deployment frontend
Name:             frontend
Namespace:        default-myapp
Labels:           key1=appValue1
                  key2=containerValue2
                  team.contact.name=Frontend
                  team.contact.alias=frontend-eng
                  radapp.io/application=myapp
                  radapp.io/resource=frontend
                  radapp.io/resource-type=applications.core-containers
                  ...
Annotations:      prometheus.io/port: 9090
                  prometheus.io/scrape: true
                  ...

The labels & annotations were set based on the following: