Kubernetes Reusable Elements for Designing Cloud Native Applications: Configuration Patterns
Kubernetes configuration is based on a declarative model, which means that you define what you want your deployment to look like, rather than how to achieve it. This is in contrast to imperative models, where you specify a series of steps to follow. The declarative model allows for more flexibility and easier management of complex deployments.
One of the key aspects of designing cloud native applications is how to manage the configuration of the application components. Configuration can include anything from environment variables, command-line arguments, configuration files, secrets, etc. Configuration can affect the behavior, performance, security, and scalability of the application. Therefore, it is important to follow some best practices and patterns when dealing with configuration in Kubernetes.
It’s important to separate application configuration from the source code. Storing configurations in the source code can lead to issues as it ties the life cycle of the code and configuration together. This is not ideal for a continuous delivery approach, where the application should remain unchanged as it moves through different stages of the deployment pipeline.
To achieve this separation, it’s a good practice to use external configuration data, which can be customized for each environment. This allows for flexibility in adapting configurations without needing to modify the application or recreate its container image.
EnvVar Configuration
Configuring applications using environment variables is a simple and universally applicable method, suitable for all environments and platforms.
Nontrivial applications need configuration for accessing data sources, external services, or production-level tuning. These configurations should be externalized rather than hardcoded within the application.
The 12-factor app manifesto recommends using environment variables for storing application configurations. A common pattern is to define hardcoded default values during build time, which can be overwritten at runtime.
Default values can simplify configuration but can also be problematic for evolving applications. If a default value needs to be changed, it’s often better to remove the default and throw an error if the user does not provide a configuration value. Avoid default values if you cannot be 90% sure that a reasonable default will last for a long time. Passwords or database connection parameters are good candidates for not providing default values.
Environment variables are widely used in container configurations due to their simplicity and universal support. However, they have limitations such as lack of security and difficulty in management when the number of configuration values is large.
For more complex needs, patterns like Configuration Resource, Immutable Configuration, and Configuration Template are recommended. One downside of environment variables is that they can only be set before an application starts and cannot be changed later. This promotes immutability, meaning a new application container is started with a modified configuration whenever changes are needed.
Configuration Resource
Kubernetes offers 2 resources, ConfigMap and Secret, for managing regular and confidential data respectively. These resources allow the configuration lifecycle to be separated from the application lifecycle. They store and manage key-value pairs, which can be used as environment variables or files mapped to a volume in a Pod.
However, updates to ConfigMap entries used as environment variables are not reflected in running processes. Also, when using multiple ConfigMaps with duplicate keys, the last entry in envFrom takes precedence, and any same-named environment variable set directly with env has higher priority.
Kubernetes has introduced an immutable field for ConfigMaps and Secrets to prevent updates once they’re created, enhancing cluster performance. To modify an immutable Secret, it must be deleted and recreated, and any running Pod referencing this secret would need to be restarted.
Despite their benefits, there are limitations. Secrets have a 1 MB size limit and are not suitable for storing large non-configuration application data. Also, Kubernetes clusters often impose a quota on the number of ConfigMaps per namespace or project.
Secrets hold Base64-encoded data, which is not encrypted and is considered plain text from a security perspective. The security of Secrets comes from their distribution only to nodes running Pods that require them, their storage in memory on nodes, and their potential encrypted storage in etcd.
However, there are still ways to access Secrets as a root user or by creating a Pod and mounting a Secret. Role-based access control (RBAC) can limit access to Secrets, but users who can create Pods in a namespace can still escalate their privileges within that namespace. Thus, additional encryption of sensitive information is often done at the application level.
Immutable Configuration
The Immutable Configuration pattern is a strategy to ensure that your application’s configuration data is always in a known and consistent state.
It addresses the challenge of managing numerous environment variables and the size limitation of ConfigMaps in Kubernetes.
Solutions:
- Use ConfigMaps or Secrets marked as immutable: This is the simplest and preferred option if your configuration fits into a ConfigMap and is easy to maintain. However, this becomes challenging when dealing with large or complex configuration data, such as nested
XMLorYAMLwithinYAML, or large data sets like pretrained machine learning models due to the backend size restriction of 1 MB - Use a passive data image for environment-specific configuration data: All configuration data is put into a single data image that can be distributed as a regular container image. During runtime, the application and the data image are linked together so that the application can extract the configuration from the data image. This approach allows for crafting different configuration data images for various environments, which can be versioned like any other container image
Advantages:
- Environment-specific configuration is sealed within a container, allowing it to be versioned and distributed via a container registry
- The configuration is immutable, requiring a version update and a new container image for any change
- Data images can hold large configuration data, making them useful when the data is too complex for environment variables or ConfigMaps
Drawbacks:
- The process is complex as it requires building and distributing extra container images
- It doesn’t address security concerns around sensitive configuration data
- As of 2023, only experimental CSI support is available, and there’s no image volume support for Kubernetes workloads
- Extra init container processing is required in the Kubernetes case, necessitating different Deployment objects for different environments.
An alternative approach for dealing with large configuration files that differ slightly from environment to environment is the Configuration Template pattern.
Docker Volumes
Docker volumes are a feature in Docker that allows a container to expose a volume with data from within the container. This is achieved using a VOLUME directive in a Dockerfile, which specifies a directory that can be shared. When the container starts up, the contents of this directory are copied to the shared directory.
Kubernetes Init Containers
Kubernetes, unlike Docker, does not support container volumes. Instead, it allows containers to share external volumes but not directories within the containers. To overcome this, Kubernetes’ Init Containers pattern can be used to initialize an empty shared volume during startup by copying configuration data to a shared Pod volume using a base image like busybox.
This process involves creating a configuration image with a Dockerfile. The Pod template specification in the Deployment contains a single volume and two containers. An empty directory, of type emptyDir, is created on the node hosting the Pod as the volume. During startup, an init container is called to copy its content to the specified directory, which is mounted from the volume. The application container then mounts the volume to access the copied configuration.
To switch configurations between development and production environments, you simply need to change the image of the init container either by updating the YAML definition or using kubectl. However, editing the resource descriptor for each environment is not ideal.
Configuration Template
The Configuration Template pattern is a method for managing large and complex configurations during application startup. It creates a configuration specific to the target runtime environment based on parameters used in processing the configuration template.
Large and complex configuration files can be challenging to manage when directly embedded into ConfigMaps due to special characters and size limitations. These files often contain redundant data as they only slightly differ for different execution environments.
To reduce redundancy, only differing configuration values (like database connection parameters) are stored in a ConfigMap or environment variables. During the container’s startup, these values are processed with configuration templates to create the full configuration file. Tools like Tiller or Gomplate can be used for processing templates during application initialization.
There are some techniques for live processing during runtime:
- Adding the template processor as part of the
ENTRYPOINTto aDockerfileso the template processing becomes directly part of the container image - Using an init container of a Pod in Kubernetes where the template processor runs and creates the configuration for the application containers in the Pod
The Configuration Template pattern is beneficial when operating applications in different environments with similar complex configurations. It’s particularly useful for applications that require a large amount of configuration data, with only a small fraction being environment-dependent. However, this setup is more complex and has more potential for errors.
Directly copying the entire configuration into the environment-specific ConfigMap may initially work, but it can lead to maintenance issues as the configuration is likely to diverge over time. The template approach is ideal for such situations.
Conclusion
Kubernetes Configuration patterns are a powerful way to manage the deployment and operation of your applications on a cluster. They allow you to define the desired state of your resources, automate the creation and update of those resources, and leverage the built-in features of Kubernetes such as health checks, scaling, and rolling updates.
By using Configuration patterns, you can reduce the complexity and risk of managing your applications, and focus on delivering value to your users.
However, these patterns are not exhaustive or prescriptive, and you may need to adapt them to your specific use case and requirements.
