Retrieve Azure Log Analytics Workspace Keys for Kubernetes Fluent Bit

Ilias el Matani

Introduction

Over the years, I've encountered an increasingly frustrating pattern: Microsoft deprecates features without proper communication or clear alternatives. They quietly bury functionality in the UI, rename things, and expect you to figure it out.

The latest example? Workspace Keys in Log Analytics.

Microsoft is pushing hard toward Data Collection Rules (DCR) for log collection, which is architecturally sound. But they've made it unnecessarily difficult to find the traditional workspace keys—keys that are still essential for many use cases, especially in containerized environments like Kubernetes.

I discovered this frustration while setting up Fluent Bit in a Kubernetes cluster. The documentation was sparse, the Azure portal didn't make it obvious, and the CLI method required trial and error. This article walks through the exact steps to retrieve your workspace keys and configure them securely in Kubernetes with Fluent Bit.

The Classical Approach: Workspace Keys

For years, it was straightforward: you had a Workspace ID (customerId) and a Shared Key to send logs directly to your Log Analytics workspace via Fluent Bit, Logstash, or custom scripts.

In a Kubernetes cluster with Fluent Bit, your deployment would look like this:

fluent-bit:
  config:
    service: |
      [SERVICE]
          Flush         5
          Daemon        off
          Log_Level     info

    outputs: |
      [OUTPUT]
          Name azure
          Match *
          Customer_ID ${AZURE_WORKSPACE_ID}
          Shared_Key ${AZURE_SHARED_KEY}
          Log_Type KubernetesLogs

Why Workspace Keys Still Matter for Kubernetes

Lightweight and Stateless

Fluent Bit is built for containerized environments. Unlike Azure Monitor Agent, which is resource-intensive, Fluent Bit runs as a DaemonSet with minimal overhead. Workspace keys enable this simplicity without additional infrastructure.

No Agent Registration Required

Pods are ephemeral. With workspace keys, there's no registration, no state management—just deploy and logs flow.

Multi-Cluster Scalability

Running multiple Kubernetes clusters? Each one uses the same workspace keys. Simple configuration, consistent across environments.

Network Flexibility

Workspace keys work with Log Analytics Gateway for air-gapped environments. DCR's architecture assumes tighter Azure control plane integration.

Microsoft's Push Toward Data Collection Rules

Microsoft promotes Data Collection Rules (DCR) because it follows modern security best practices:

  • Granular control per agent and data type
  • Detailed audit trails for compliance requirements
  • Data transformation closer to the source
  • Multi-workspace routing with a single agent
  • Identity-based access using RBAC instead of shared secrets

The downside: DCR requires Azure Monitor Agent deployment, additional resource creation, and more complex infrastructure. For Kubernetes clusters running Fluent Bit, this introduces unnecessary overhead and administrative burden.

Why DCR Doesn't Work for Kubernetes Clusters

Fluent Bit is Purpose-Built for Containers

Azure Monitor Agent is monolithic and resource-hungry. Fluent Bit is designed for containerized environments. Adding DCR means deploying extra components—defeating the entire point of using Fluent Bit.

Ephemeral Workloads

Kubernetes pods are created and destroyed constantly. DCR requires agent registration and persistent state. Fluent Bit with workspace keys is stateless—deploy the DaemonSet and it works immediately.

Operational Complexity

Configuring DCR introduces additional Azure resources, dependency management, and troubleshooting surface area. Fluent Bit with workspace keys requires minimal setup.

Air-Gapped and Network-Restricted Environments

Some clusters operate behind firewalls or without direct Azure connectivity. Fluent Bit works with Log Analytics Gateway. DCR's architecture assumes integrated Azure control plane access.

Retrieving Workspace Keys from Azure

Microsoft doesn't advertise how to obtain workspace keys anymore, but they're still accessible via the Azure CLI. Here's the process:

Navigate to the Correct Subscription

First, ensure you're working with the right Azure subscription:

# List all available subscriptions
az account list --output table

# Set the subscription you need
az account set --subscription <subscription-id>

Retrieve the Shared Keys

Once in the correct subscription, retrieve your workspace keys:

az monitor log-analytics workspace get-shared-keys \
  --resource-group <resource-group-name> \
  --workspace-name <workspace-name>

This returns the primary and secondary shared keys. Store these securely—they grant full access to your workspace.

Retrieve the Workspace ID

You'll also need your workspace ID (customerId):

az monitor log-analytics workspace show \
  --resource-group <resource-group-name> \
  --workspace-name <workspace-name> \
  --query "customerId" -o tsv

Both values are now ready for use in your Kubernetes cluster.

Configuring Fluent Bit in Kubernetes

Create a Kubernetes Secret

First, store your workspace credentials securely as a Kubernetes secret:

kubectl create secret generic azure-log-analytics \
  --from-literal=AZURE_WORKSPACE_ID='<workspace-id>' \
  --from-literal=AZURE_SHARED_KEY='<shared-key>' \
  -n logging

Deploy Fluent Bit with Helm

Create a values.yaml file for your Fluent Bit deployment:

fluent-bit:
  enabled: true
  
  config:
    service: |
      [SERVICE]
          Flush         5
          Daemon        off
          Log_Level     info
          Parsers_File  parsers.conf

    inputs: |
      [INPUT]
          Name              tail
          Path              /var/log/containers/*.log
          Parser            docker
          Tag               kube.*
          Refresh_Interval  5

    outputs: |
      [OUTPUT]
          Name              azure
          Match             *
          Customer_ID       ${AZURE_WORKSPACE_ID}
          Shared_Key        ${AZURE_SHARED_KEY}
          Log_Type          KubernetesLogs
          Retry_Limit       5

  env:
    - name: AZURE_WORKSPACE_ID
      valueFrom:
        secretKeyRef:
          name: azure-log-analytics
          key: AZURE_WORKSPACE_ID
    - name: AZURE_SHARED_KEY
      valueFrom:
        secretKeyRef:
          name: azure-log-analytics
          key: AZURE_SHARED_KEY

Deploy the chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit \
  -f values.yaml \
  -n logging --create-namespace

Verify the deployment:

kubectl get pods -n logging
kubectl logs -n logging -l app.kubernetes.io/name=fluent-bit

Logs will now stream directly to your Log Analytics workspace.

Security Considerations

Workspace keys grant full access to your Log Analytics workspace. Treat them with the same care as database passwords.

Use Kubernetes Secrets

Never hardcode credentials in ConfigMaps or Helm values. Always use Kubernetes Secrets:

# Wrong - Credentials visible in ConfigMap
kubectl create configmap fluent-bit-config --from-literal=shared-key="xxx"

# Correct - Credentials in Kubernetes Secret
kubectl create secret generic azure-log-analytics \
  --from-literal=AZURE_SHARED_KEY="xxx"

Store Secrets in Azure Key Vault

For production environments, integrate with Azure Key Vault:

az keyvault secret show \
  --vault-name my-keyvault \
  --name azure-shared-key

Rotate Keys Regularly

Regenerate your shared keys annually:

az monitor log-analytics workspace regenerate-shared-keys \
  --resource-group <resource-group> \
  --workspace-name <workspace>

Use Both Keys for Zero-Downtime Rotation

  1. Update Fluent Bit to use the secondary key
  2. Regenerate the primary key in Azure
  3. Wait for all pods to cycle
  4. Regenerate the secondary key

Monitor Access

Review Azure Activity Logs to audit when and where your workspace keys are being used.

Conclusion

Retrieving workspace keys for your Kubernetes Fluent Bit deployment requires knowledge that Microsoft no longer documents prominently. This gap exists because they're pushing toward DCR—a valid architectural direction, but not always the right choice for containerized environments.

By following the steps in this article, you can retrieve your workspace keys via the Azure CLI and configure Fluent Bit securely in your Kubernetes cluster. The approach is simple, stateless, and scales across multiple clusters without additional Azure infrastructure.

Treat your workspace keys as production secrets: store them in Kubernetes Secrets (or Azure Key Vault), rotate them annually, and monitor their usage. With this setup in place, your cluster logs will flow directly and reliably into Azure Log Analytics.