What are the Common Issues with Secret Management in Kubernetes?
Deploying updates in Kubernetes can break services if secrets are mismanaged. A database authentication failure can occur when a password is rotated, but Kubernetes continues using an outdated version, requiring a manual pod restart.
Common secret management challenges include:
- Hardcoded secrets in manifests – Often embedded in YAML files during testing.
- Scattered secret storage – Spread across Kubernetes Secrets, CI/CD pipelines, or external providers like AWS Secrets Manager.
- Lack of automatic syncing – Kubernetes doesn’t refresh secrets when they change externally.
- Inconsistent access control – Kubernetes RBAC secures only in-cluster secrets but doesn’t control external secret providers.
For small-scale operations, these issues might seem minor. However, in large infrastructures with hundreds of microservices, secret mismanagement can lead to security breaches, downtime, and compliance violations.
How does the External Secrets Operator Fix This?
The External Secrets Operator (ESO) automates secret management in Kubernetes by syncing secrets from external providers like AWS Secrets Manager, GCP Secret Manager, and HashiCorp Vault.
Key benefits of External Secrets Operator:
- Ensures secrets are always updated – No need for manual pod restarts.
- Prevents secrets from being stored in Git – Reducing exposure risks.
- Uses external provider permissions – IAM roles or Vault policies control access.
- A developer defines an ExternalSecret in Kubernetes, specifying which secret to fetch from AWS Secrets Manager (or another provider).
- ESO continuously monitors external secret providers and fetches new values as soon as they are updated, ensuring Kubernetes always has the latest credentials.
- The secret is stored in Kubernetes as a standard Kubernetes Secret and mounted into application pods.
- Whenever a secret is updated in AWS Secrets Manager, ESO syncs it automatically - no need for a kubectl rollout restart.
This means no more hardcoded secrets, no more forgotten updates, and no more manual interventions.
For DevOps teams managing Kubernetes across multiple environments, ESO simplifies secret management while keeping security tight and operations smooth.
Simplifying Secret Management with External Secrets Operator
External Secrets Operator simplifies managing your secrets by integrating Kubernetes with external secret management systems, ensuring secrets remain secure, up-to-date, and accessible to applications without manual intervention.
ESO achieves this through Custom Resource Definitions (CRDs), which extend Kubernetes to support external secrets. The two key CRDs used are SecretStore and ExternalSecret.
The SecretStore resource defines where secrets should be fetched from. It specifies the external provider, authentication details, and region, allowing Kubernetes to securely pull secrets from AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.
Advanced Features: Secret Templating & Transformation
ESO also supports secret templating and transformation, which allow for greater flexibility in handling secrets:
- Secret templating dynamically formats secrets before storing them in Kubernetes, enabling applications to consume secrets in a predefined structure.
- Secret transformation modifies secret values before injecting them into Kubernetes workloads. This can be used to convert JSON secrets into environment variables, encode values, or mask sensitive data.
These advanced features make ESO adaptable to different use cases, allowing teams to enforce security policies while keeping secrets structured for application consumption.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secret-store
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
secretRef:
accessKeyIDSecretRef:
name: aws-credentials
key: access-key
secretAccessKeySecretRef:
name: aws-credentials
key: secret-access-key
The ExternalSecret resource defines which secret should be retrieved and how it should be stored in Kubernetes. This makes sure that external secrets map correctly to Kubernetes Secrets, making them available to applications.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
secretStoreRef:
name: aws-secret-store
kind: SecretStore
target:
name: db-secret
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: prod/db-credentials
property: username
- secretKey: password
remoteRef:
key: prod/db-credentials
property: password
ESO continuously monitors these resources and ensures that Kubernetes Secrets stay updated whenever a change happens in the external provider. If a database password is rotated in AWS Secrets Manager, ESO automatically syncs the new value in Kubernetes, eliminating the need for any restarts or redeployments.
This approach prevents secrets from being hardcoded in YAML manifests or stored in Git. It also simplifies secret access management, as permissions are controlled by external providers like AWS IAM, GCP IAM, or Azure RBAC instead of Kubernetes RBAC alone.
Setting up ESO can be complex if you're not fully familiar with Kubernetes manifests or how the underlying components interact. It requires defining multiple CRDs, configuring authentication properly, and ensuring the operator functions as expected.
Apart from ESO, you can also use Kapstan, which offers a simpler way to manage secrets in Kubernetes without requiring deep Kubernetes knowledge. It provides an intuitive UI and integrates with existing infrastructure easily. We’ll explore Kapstan in action later in this blog.
How Do You Start Using the External Secrets Operator?
Setting up External Secrets Operator (ESO) involves defining how it connects with external secret management systems. Since Kubernetes does not manage secrets efficiently on its own, ESO acts as a controller that continuously syncs secrets from providers like AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, and Azure Key Vault.
The installation process for ESO is simple and can be done using Helm or YAML manifests. To install ESO using Helm, run:
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
Once installed, ESO operates in the background, making sure that applications always receive the most up-to-date secrets.
Connecting ESO to External Secret Managers
ESO does not store secrets directly. Instead, it uses Custom Resource Definitions (CRDs) to retrieve and inject secrets into Kubernetes:
- SecretStore defines where the secrets should be fetched from, specifying the external provider, authentication details, and region.
- ExternalSecret maps external secrets to Kubernetes Secrets, ensuring they are accessible to workloads.
By using these resources, ESO standardizes secret retrieval across different cloud environments.
Integration with Cloud Secret Management Systems
ESO integrates easily with AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and HashiCorp Vault, allowing Kubernetes workloads to access secrets without exposing any secrets.
- AWS Secrets Manager serves as a central repository for API keys, database credentials, and application secrets. ESO pulls these values and syncs them with Kubernetes automatically. For additional security, AWS Secrets Manager integrates with AWS KMS, making sure that the secrets are encrypted at rest and only accessible to authorized users." To: "ESO integrates with cloud secret managers like AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault, ensuring centralized, encrypted, and access-controlled storage of secrets.
- HashiCorp Vault provides dynamic secrets with built-in access policies and expiration controls, making it ideal for handling short-lived credentials.
- GCP Secret Manager ensures secrets are securely stored and versioned, allowing applications to retrieve the latest values as needed.
- Azure Key Vault integrates well with Azure services and offers fine-grained access control for secrets.
With this flexibility, ESO enables organizations to manage secrets securely across multiple cloud providers without modifying Kubernetes configurations.
Hands-On: Managing Secrets with External Secrets Operator
Till now, we've covered how the External Secrets Operator simplifies secret management within Kubernetes. Now, let's see it in action. In this section, we'll integrate AWS Secrets Manager with Kubernetes using ESO and ensure secrets sync automatically.
Step 1: Install External Secrets Operator
First, we need to install ESO in our Kubernetes cluster. ESO runs as a controller that watches for ExternalSecret resources and pulls secrets from external providers like AWS Secrets Manager.
To install ESO, add the ESO Helm chart repository:
helm repo add external-secrets https://charts.external-secrets.io
Once executed, you should see an output like this:
"external-secrets" has been added to your repositories
This command registers the External Secrets Helm repository in Helm, allowing us to install and manage ESO with Helm.
Now, install ESO in a dedicated namespace:
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace
Confirm that ESO is running:
kubectl get all -n external-secrets
You should see an output similar to this:
This makes sure that all the ESO components (controller, webhook, and certificate manager) are running properly.
Step 2: Store AWS Credentials in Kubernetes
ESO needs AWS credentials to authenticate and pull secrets from AWS Secrets Manager. Instead of hardcoding credentials in our YAML files, we store them securely as a Kubernetes Secret.
kubectl create secret generic awssm-secret \
--from-literal=access-key=AKIAXBZV************J3UG \
--from-literal=secret-access-key=/Wbo*****96ByCE1fSdNNzpwC
- kubectl create secret generic awssm-secret: Creates a Kubernetes secret named awssm-secret.
- --from-literal=access-key=...: Stores the AWS access key.
- --from-literal=secret-access-key=...: Stores the AWS secret key.
Check if the secret was created successfully:
kubectl get secret
You should see an output similar to this:
NAME TYPE DATA AGE
awssm-secret Opaque 2 2s
This makes sure that Kubernetes has stored our AWS credentials securely.
Step 3: Define the SecretStore to Connect with AWS
To allow ESO to fetch secrets from AWS, we need to define a SecretStore. The SecretStore resource tells ESO where to pull secrets from and how to authenticate.
Create a file SecretStore.yaml and add:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: awssm-secret-store
spec:
provider:
aws:
service: SecretsManager
region: ca-central-1
auth:
secretRef:
accessKeyIDSecretRef:
name: awssm-secret
key: access-key-id
secretAccessKeySecretRef:
name: awssm-secret
key: secret-access-key
In this config:
- provider: aws – Specifies that AWS Secrets Manager is the external secret provider.
- region: ca-central-1 – Defines the AWS region where our secrets are stored.
- auth.secretRef – ESO uses a Kubernetes Secret (awssm-secret) for authentication, which contains AWS access credentials.
ESO does not directly store or manage AWS credentials. Instead, it authenticates with AWS Secrets Manager through the SecretStore resource, which references a Kubernetes Secret (awssm-secret). This secret contains the AWS access key and secret key, allowing ESO to securely fetch secrets from AWS.
Once the SecretStore is set up, ESO continuously monitors for changes in AWS Secrets Manager. Whenever a secret is updated, ESO automatically syncs it to Kubernetes, ensuring applications always have the latest credentials without requiring manual intervention.
Apply the configuration:
kubectl apply -f SecretStore.yaml
Once applied successfully, you should see an output like this:
secretstore.external-secrets.io/secretstore-sam created
This tells ESO where to fetch secrets from and how to authenticate with AWS Secrets Manager.
Step 4: Sync AWS Secrets with Kubernetes
Now, we define an ExternalSecret that pulls a secret from AWS and maps it to a Kubernetes Secret.
Create a file ExternalSecret.yaml:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: dep
spec:
refreshInterval: 3m
secretStoreRef:
name: secretstore-sam
kind: SecretStore
target:
name: kube-secret
creationPolicy: Owner
dataFrom:
- extract:
key: DB-CREDENTIALS
In this config:
- refreshInterval: 3m: ESO checks AWS Secrets Manager every 3 minutes for updates.
- secretStoreRef: References the SecretStore we created earlier (secretstore-sam).
- target.name: kube-secret: Defines the name of the Kubernetes Secret ESO will create.
- dataFrom.extract.key: DB-CREDENTIALS: Fetches all key-value pairs stored under the AWS Secrets Manager secret named DB-CREDENTIALS.
Apply the configuration:
kubectl apply -f ExternalSecret.yaml
Once applied successfully, you should see an output like this:
externalsecret.external-secrets.io/dep created
This tells ESO to pull the secret DB-CREDENTIALS from AWS Secret Manager and store it as a Kubernetes Secret named kube-secret.
Step 5: Using Secret Templating & Transformation
By default, ESO stores secrets as individual key-value pairs. However, some applications require secrets in a specific format. With Secret Templating, ESO can dynamically format secrets before storing them.
To apply secret templating, modify your ExternalSecret.yaml:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
refreshInterval: 3m
secretStoreRef:
name: aws-secret-store
kind: SecretStore
target:
name: db-secret
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: db-credentials
property: username
- secretKey: password
remoteRef:
key: db-credentials
property: password
dataFrom:
- extract:
key: db-credentials
template:
data:
connection_string: "postgres://{{ .username }}:{{ .password }}@database.internal:5432/app_db"
In this config:
- template.data - Defines a formatted secret using templating.
- connection_string - Instead of storing raw username and password, ESO transforms them into a PostgreSQL connection string:
postgres://username:password@database.internal:5432/app_db
This eliminates the need for applications to manually assemble connection strings from separate secrets.
Apply the configuration:
kubectl get secret db-secret -o yaml
Step 6: Verify Secret Syncing
Now, inspect the contents of the secret:
kubectl get secret kube-secret -o yaml
Once executed, the output should look like this:
apiVersion: v1
data:
DB_PASSWORD: aW5mcmFzaXR5QDU5MTg=
DB_USERNAME: dXNlcjE=
kind: Secret
metadata:
name: kube-secret
namespace: default
ownerReferences:
- apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
name: dep
type: Opaque
This command displays the base64-encoded secret values stored in Kubernetes.
To decode the values, use:
echo 'dXNlcjE=' | base64 -d
# Output: user1
echo 'a2Fwc3RhbkAxMjM0' | base64 -d
# Output: kapstan@1234
If Secret Templating is used, also check the connection string:
echo 'cG9zdGdyZXM6Ly91c2VyMTpzZWN1cmVwYXNzd29yZDEyM0BkYXRhYmFzZS5pbnRlcm5hbDo1NDMyL2FwcF9kYg==' | base64 -d
# Output: postgres://user1:securepassword123@database.internal:5432/app_db
Now, with this setup, Kubernetes workloads always receive the latest secrets without requiring restarts or redeployments.
Best Practices for Managing Secrets in Kubernetes
Now that we know how to manage secrets effectively with External Secrets Operator, let’s take a look at some best practices to enhance security and reliability.
- Use RBAC to Restrict Secret Access: Assign least privilege access using Kubernetes Role-Based Access Control (RBAC). Ensure only necessary workloads and users can read secrets by defining roles and role bindings at a granular level.
- Enforce Network Policies for Secret Isolation: Use Kubernetes NetworkPolicies to restrict communication between pods, preventing unauthorized access to secrets. Configure policies to allow only the External Secrets Operator to communicate with external secret stores like AWS Secrets Manager or Vault.
- Rotate Secrets Automatically: Configure external secret providers (e.g., AWS Secrets Manager, HashiCorp Vault) to auto-rotate secrets at regular intervals. Ensure ESO syncs updated secrets in Kubernetes without requiring manual intervention or pod restarts.
With these best practices in place, let's explore how Kapstan simplifies secret management even further.
Using Kapstan for Secret Management
So far, we’ve seen how the External Secrets Operator simplifies secret management by syncing Kubernetes with external secret operators. But even with ESO, managing secrets still requires DevOps teams to define CRDs, configure authentication, and maintain ESO deployments. This often leads to ticket-driven workflows where developers depend on DevOps teams to provision and update secrets.
Many teams still fall back on static configuration files, scattered environment variables, or manual external secret management. These methods introduce friction, miscommunication, and operational overhead between DevOps and development teams. Let’s say an organization is using AWS Secrets Manager to manage secrets for a new third-party API. Here, the process looks like this:
- Generate the API secret.
- Store the secret in AWS Secrets Manager.
- Modify the Kubernetes deployment or Helm chart to mount the secret.
- Reference the mounted path in the application.
At this point, the DevOps team has to step in, verify access, and modify Kubernetes configurations. This means another ticket for the DevOps team, another manual process, and more potential delays. Even with ESO, someone has to define SecretStore and ExternalSecret resources.
A manual secret management workflow like this is not scalable. It introduces “TicketOps”, where every secret change needs DevOps involvement and ESO maintenance overhead, requiring teams to make sure that ESO is correctly installed, configured, and updated.
How Kapstan Simplifies Secret Management
Kapstan eliminates this complexity, making secrets available with just three steps:
- The developer generates an API key.
- They add the secret directly to Kapstan’s UI.
- The application retrieves the secret automatically as an environment variable.
Kapstan automatically provisions the required configurations in the background, making sure that the secret is securely stored, mounted, and injected without requiring developers to modify Kubernetes manifests or request help from DevOps.
Let’s see how this works. A developer using a containerized service needs to store an API key securely. Instead of modifying Kubernetes secrets, they simply go to Kapstan’s UI, navigate to the "Environment Variables" section, and click "Add" to create a new secret.
Once the key is added, Kapstan makes sure it remains hidden(if you have clicked on the lock). The developer cannot retrieve the secret once saved, preventing accidental exposure.
Now, the developer can proceed to deployment. Kapstan makes sure that the secret is injected into the containerized application and available as an environment variable. There is no need for any manual Helm modifications, pod restarts, or external secret configuration.
How Kapstan Works Behind the Scenes
Unlike traditional secret management, where developers must manually handle secret injection, Kapstan automates the entire process. Here’s what happens under the hood:
- The secret is stored securely in AWS Secrets Manager.
- Kapstan installs and configures the necessary provider on the Kubernetes cluster.
- It mounts the secret as a volume inside the EKS cluster.
- The secret is injected and made available as an environment variable at runtime.
- It manages ESO for upgrades, ensuring the external secret sync remains operational.
That’s it - no additional YAML files, no manual Kubernetes configurations, and no need to request help from DevOps Teams. The secret is now securely injected into the application without any overhead.
Kapstan is not just a ClickOps solution. While it provides an intuitive UI for developers, all configurations remain accessible within your cloud if needed. This balance between automation and control makes Kapstan a powerful alternative to traditional secret management tools.
By removing the manual steps of secret storage, injection, and retrieval, Kapstan enables developers to focus on building applications while keeping security and compliance intact.
Conclusion
Till now, you should have a clear understanding of how External Secrets Operator automates secret management in Kubernetes. It eliminates hardcoded secrets, ensures automatic syncs, and reduces manual intervention, making secret handling more secure and efficient.
Frequently Asked Questions
Q. What is the role of External Secrets Operator?
External Secrets Operator (ESO) automates secret management in Kubernetes by fetching secrets from external providers like AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and HashiCorp Vault, ensuring applications always have the latest secret values without manual intervention.
Q. How do I get external secrets in Kubernetes?
You can get external secrets in Kubernetes using External Secrets Operator. Define a SecretStore to connect to an external secret provider, then create an ExternalSecret resource that maps the external secret to a Kubernetes Secret, making it available to workloads.
Q. How to access an external service outside of the Kubernetes cluster?
You can access an external service outside of Kubernetes by using a Service of type ExternalName, setting up an Ingress with external DNS, configuring LoadBalancer services, or directly using environment variables or ConfigMaps to store external service details.
