What Are Kubernetes Secrets?
The term Kubernetes secrets can have two meanings: managing secrets such as passwords, tokens, and SSH keys in a Kubernetes environment, and the specific Kubernetes mechanism called Secrets, which allows you to manage this sensitive information.
Most of our article will focus on the second meaning, explaining how the built-in Kubernetes Secrets mechanism works. However, Kubernetes Secrets are not meant to be a comprehensive solution for managing secret data, and you should consider using a dedicated secrets management solution compatible with Kubernetes.
The Secrets resource in Kubernetes works similar to a ConfigMap, but is intended to hold confidential data. This distinction is not enforced by Kubernetes, but rather agreed upon by convention. Here is an example of a Secret manifest in Kubernetes:
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
key1: value1
key2: value2This is part of a series of articles about secret management.
In this article:
Why Is Kubernetes Secrets Management Important?
Managing sensitive data is a critical aspect of any production system. In the context of Kubernetes, sensitive data could include items like API tokens, database passwords, or TLS certificates. If such information were exposed, it could lead to a data breach or compromise of your Kubernetes infrastructure.
The Kubernetes Secrets mechanism provides a basic way to store and manage such sensitive information. By using Secrets, you can ensure sensitive data is not exposed in logs or accidentally leaked to unauthorized users.
Additionally, Secrets provide a way to decouple sensitive data from the application code. This means you don’t have to hardcode sensitive data into applications, making it easier to manage and update it as needed.
Tips From the Expert
In my experience, here are tips that can help you better manage Kubernetes Secrets:
-
Leverage sealed secrets for better security
Use Sealed Secrets, a controller from Bitnami, to encrypt secrets before they are stored in Git. This ensures secrets can be safely stored in version control, which can then only be decrypted inside your Kubernetes cluster. -
Rotate secrets regularly
Implement automated secret rotation to reduce the risk of long-lived credentials being compromised. Integrating this with external tools like HashiCorp Vault or AWS Secrets Manager can help streamline the process. -
Use external KMS providers for enhanced security
While Kubernetes supports encryption at rest, using an external Key Management Service (KMS) provider such as AWS KMS or Google Cloud KMS adds an extra layer of security by offloading key management to a dedicated, robust service. -
Monitor secret access logs
Track and monitor access to your secrets using Kubernetes audit logs. This allows you to detect any unauthorized or abnormal access patterns and respond to potential security breaches quickly. -
Namespace isolation for sensitive secrets
Store highly sensitive secrets in a dedicated namespace with stricter Role-Based Access Control (RBAC) policies. This provides better isolation and reduces the blast radius in case of a breach.
Types of Kubernetes Secrets
Here are the main types of secrets supported by the Kubernetes Secrets resource.
Opaque Secrets
Opaque is the default type of secret. This type is used when the secret contains arbitrary key-value pairs. The keys and values can be anything you want, making opaque secrets versatile for many use cases.
For example, you might use an opaque secret to store database credentials. The secret could include keys like username and password, with corresponding values. The application could then use these credentials to connect to the database.
Service Account Token Secrets
Service account token secrets are used to store tokens that identify Kubernetes service accounts. Kubernetes automatically creates these secrets and attaches them to service accounts.
These secrets let applications running in your cluster authenticate to the Kubernetes API. This is useful for applications that need to manage Kubernetes resources.
Docker Config Secrets
Docker config secrets are used to store Docker registry login credentials. If your application’s Docker images are stored in a private registry, your pods need credentials to pull those images. Docker config secrets let you provide these credentials without including them in your pod specification or Dockerfiles.
Basic Authentication Secret
Basic authentication secrets are used to store credentials for basic HTTP authentication. These secrets contain two keys: username and password.
For example, you might use a basic authentication secret if your application needs to authenticate to a web service that uses basic HTTP authentication.
SSH Authentication Secrets
SSH authentication secrets are used to hold SSH private keys. They can be used to authenticate with services that require SSH authentication.
This type of secret is especially useful when you need to establish secure connections to external services.
TLS Secrets
TLS secrets are used to store a certificate and its corresponding private key. These secrets are useful for applications that use TLS for secure communications.
For example, if you’re running a web server in your cluster, you could use a TLS secret to provide the server’s SSL certificate and private key. The web server could then use these to establish secure connections with clients.
Bootstrap Token Secrets
Bootstrap token secrets are used for establishing trust between a client and the API server in the process of bootstrapping a new node.
These secrets are a critical part of the Kubernetes node bootstrapping process, ensuring only authorized nodes can join your cluster.
Learn more in our detailed guide to Kubernetes secrets types (coming soon)
Tutorial: Creating and Using Kubernetes Secrets
Create Secrets Using kubectl
Creating secrets using kubectl is the most straightforward method to manage your Kubernetes secrets.
To start, ensure that you have kubectl installed and properly configured to interact with your Kubernetes cluster. You can use the kubectl create secret command to make a secret. Here’s an example:
kubectl create secret generic my-secret --from-literal=key1=value1 --from-literal=key2=value2In this command, my-secret is the name of your secret, while --from-literal allows you to specify the key-value pairs that make up the secret. Another way to define secrets is to provide a file that kubectl should take the secret’s data from, using the flag --from-file=<filename>
Create Secrets in a Manifest File
Creating secrets in a manifest (YAML configuration file) provides an alternative method for managing your Kubernetes secrets. This approach is often used when you need to store secrets that have multiple key-value pairs or when you want to store your secrets in version control.
To create a secret in a configuration file, you need to create a YAML or JSON file that describes the secret. Here’s a sample YAML file for a secret:
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
key1: value1
key2: value2In this file, my-secret is the name of your secret, while data is where you specify your key-value pairs. Note that the values must be base64 encoded.
Use kubectl describe to See Existing Secrets
Once you’ve created your Kubernetes secrets, you can use the kubectl describe command to see their status and metadata, although it will not display the actual secret data.
Here’s how to use kubectl describe to view your secrets:
kubectl describe secrets my-secretThe output will look something like this:
In this command, my-secret is the name of your secret. The output will display the metadata of your secret, including the name, namespace, labels, annotations, and type.
Edit a Secret’s Metadata
There might be times when you need to edit a Kubernetes secret. This could happen when you need to update the value of a key, add a new key-value pair, or remove an existing one.
To edit a secret, you can use the kubectl edit command. However, note that this command only allows for editing the metadata of a secret. To change the data of the secret, you need to recreate the secret.
Here’s an example of how to edit a secret:
kubectl edit secrets my-secretIn this command, my-secret is the name of your secret. This will open a text editor where you can edit the metadata of your secret, like this:
Best Practices for Managing Kubernetes Secrets
Here are a few best practices that can help you manage Kubernetes secrets effectively.
Limit Access Using RBAC
Role-Based Access Control (RBAC) plays a critical role in securing your Kubernetes environment by allowing you to specify who has access to what. You can use the built-in Kubernetes RBAC functionality to restrict access to your secrets.
The first step in limiting access is to determine who needs access to the secrets. Not everyone in your organization should have access to all secrets. This principle of least privilege should guide you when configuring RBAC for secrets.
Next, define roles and role bindings that align with your access requirements. A role in Kubernetes defines a set of permissions to perform actions (like read, write, or update) on a set of resources, such as secrets. Role bindings link these permissions to specific users, groups, or service accounts.
Encrypt Secrets at Rest
When secrets are at rest, they are stored in etcd, the distributed key-value store that Kubernetes uses to maintain cluster state.
By default, data in etcd is not encrypted at rest. This means that anyone with access to the etcd data can potentially read your Kubernetes secrets. This is a significant security risk, especially if your etcd data is stored in a shared or insecure environment.
To mitigate this risk, you should enable encryption at rest for your Kubernetes secrets. This can be done by configuring a Kubernetes feature called KMS (Key Management Service) provider. A KMS provider is a plugin that provides key management for Kubernetes, allowing you to encrypt and decrypt data using a managed service.
Use Environment Variables Sparingly
Environment variables are a common way to pass configuration information to applications. However, when it comes to secrets, using environment variables should be done sparingly.
The reason for this is simple: environment variables are not secure. They can be accessed by any process running on the same machine, making them an attractive target for attackers. Furthermore, environment variables are often logged or displayed in debugging information, potentially exposing your secrets.
A better alternative to environment variables is to use Kubernetes secrets as files. By mounting a secret as a file in a pod, you can provide your application with the secret it needs without exposing it to unnecessary risks.
Automate Secret Management
Finally, try to automate as much of the process as possible. Manual secret management is error-prone and inefficient. By automating secret management, you can ensure consistent application of best practices, reduce human error, and save time.
Automation can be achieved through the use of secret management tools and automation scripts. For example, you can use a secret management tool to automatically rotate secrets, ensuring that your secrets are always fresh and reducing the chance of a secret being compromised. Secret management tools can also help you centrally manage secrets, monitor access to secrets, and create an audit trail of the usage of secrets over time.
Learn more in our detailed guide to Kubernetes Secrets best practices (coming soon)
Related content: Read our guide to GitHub secrets
Kubernetes Secret Management with Configu
Configu is a configuration management platform comprised of two main components:
Configu Orchestrator
As applications become more dynamic and distributed in microservices architectures, configurations are getting more fragmented. They are saved as raw text that is spread across multiple stores, databases, files, git repositories, and third-party tools (a typical company will have five to ten different stores).
The Configu Orchestrator, which is open-source software, is a powerful standalone tool designed to address this challenge by providing configuration orchestration along with Configuration-as-Code (CaC) approach.
Configu Cloud
Configu Cloud is the most innovative store purpose-built for configurations, including environment variables, secrets, and feature flags. It is built based on the Configu configuration-as-code (CaC) approach and can model configurations and wrap them with unique layers, providing collaboration capabilities, visibility into configuration workflows, and security and compliance standardization.
Unlike legacy tools, which treat configurations as unstructured data or key-value pairs, Configu is leading the way with a Configuration-as-Code approach. By modeling configurations, they are treated as first-class citizens in the developers’ code. This makes our solution more robust and reliable and also enables Configu to provide more capabilities, such as visualization, a testing framework, and security abilities.
