Introduction
This blog post explains how multi-tenancy works within Kubernetes.
Kubernetes is an open-source container orchestration system that automates containerized applications’ deployment, scaling, and management. One of the key features of Kubernetes is its ability to support multi-tenancy, which refers to the ability to run multiple isolated workloads on the same cluster.
You can separate each tenant and their Kubernetes resources into their namespaces. You can then use policies to enforce tenant isolation. Policies are usually scoped by namespace and can be used to restrict API access, constrain resource usage, and restrict what containers are allowed to do.
A tenant can be defined as any of the following:
A team is responsible for developing and operating one or more workloads.
A set of related workloads (e.g., for a specific domain), whether used by one or more teams.
A single workload, such as a Deployment.
When planning to have a multi-tenant architecture, you should consider the layers of resource isolation in Kubernetes: cluster, namespace, node, pod, and container. You can have a single cluster for multi-tenancy or have a single cluster per tenant. Both strategies have their advantages and disadvantages.
Tenancy Models
Single Cluster
A single cluster with multi-tenancy is an alternative to managing many single-tenant clusters. However, the operators of multi-tenant clusters must isolate tenants from each other to minimize the damage that a compromised or malicious tenant can do to the cluster and other tenants. Also, cluster resources must be fairly allocated among tenants.
Although Kubernetes cannot guarantee perfectly secure isolation between tenants, it does offer features that may be sufficient for specific use cases. For example, you can separate each tenant and their Kubernetes resources into their namespaces. In addition, you can set a quota to limit resource usage and use network policies to enforce tenant isolation. Optionally, you can use node pools to reduce the impact of noisy neighbors.
Operating a multi-tenant cluster has several advantages over operating multiple, single-tenant clusters:
Reduced management overhead
Reduced resource fragmentation
No need to wait for cluster creation for new teams or workloads
Multiple Clusters
Multiple multi-tenancy clusters are the best solution if you need strict and secure isolation between tenants.
Operating multiple clusters has several advantages over operating a single multi-tenant cluster:
- Secure isolation between tenants, teams, or specific workloads
- Run a different Kubernetes version or have a separate maintenance window
- Billing and usage metering per tenant
Comparison
Cost / Overhead
Each cluster and its cluster add-ons need to be managed (lifecycle management, upgrades, security patching).
Each cluster will come with a control plane, and cluster add-ons (ingress controller, monitoring, log management, secrets management) which will take up cloud computing resources.
Isolation
Each cluster will have one outbound IP address for egress traffic.
A cluster can have multiple public IP addresses and load-balancers (ingress controllers).
Resource Sharing
Cloud Compute resources can be shared across tenants, but it is possible to have dedicated nodes for each tenant.
Each cluster will have a maintenance window for upgrades.
Service Complexity
Application deployments need to be prepared for multi-tenancy. Network policies should be included with deployment, usage of multiple ingress controllers, etc.
Best Practices
This chapter describes best practices and security mitigations that can be used for multi-tenant environments on Kubernetes. Some features can require additional tooling and services.
Namespace isolation
The simplest form of multi-tenancy in Kubernetes is through the use of namespaces. A namespace is a logical partition of a Kubernetes cluster that allows resources to be isolated. Each namespace has its own set of resources, such as pods, services, and secrets, and can have its own set of role-based access controls (RBAC) policies.
Also, remember that in some cases, it may be desirable to assign multiple namespaces to the same group within your Kubernetes deployment. For example, the same team of developers might need multiple namespaces for hosting different builds of their applications.
Resource Limits
Tenants should share resources fairly. A tenant using a disproportionate share of cluster resources in a cluster can impair the performance of other tenants. To resolve this problem, ensure that you set resource usage limits for every tenant.
In Kubernetes, this is managed by the ResourceQuota object.
When you apply a quota to a namespace, Kubernetes requires you to specify resource requests and limits for each container.
Network Policies
By default, Kubernetes cluster configurations allow any service on any namespace to be accessible. As a result, the pods are open to all traffic. In addition, Network policies will enable you to specify how pods can interact with various entities on the network.
Network policies can be used to implement multi-tenancy by isolating traffic between different namespaces. For example, pods in one namespace may be prevented from communicating with pods in another namespace.
RBAC
Kubernetes Role-Based Access Control is a core security feature in Kubernetes that lets you create fine-grained permissions to manage what actions users and workloads can perform on resources in your clusters.
In a multi-tenant environment, RBAC must be used to restrict tenants’ access to the appropriate namespaces and ensure that cluster-wide resources can only be accessed or modified by privileged users such as cluster administrators.
Pod Security Standard
The Kubernetes Pod Security Standards define different isolation levels for Pods. These standards let you define how you want to restrict the behavior of pods in a clear, consistent fashion.
It evaluates the pod specifications against three policy levels based on Pod Security standards:
Privileged: No restrictions, broadest permission level
Baseline: Minimal restrictions to guard against foreseeable privilege escalations
Restricted: Extreme restrictions
The policy levels can be assigned by labels to namespace resources, allowing for fine-grained policy control per namespace. For example, to enforce hardening best practices, every pod in every namespace would meet the requirements of the restricted policy.
Node Pools
You can use dedicated node pools for specific tenants so CPU and memory are not shared across tenants. This will reduce the impact of noisy tenants on a shared cluster.
Although workloads from different tenants are running on different nodes, it is important to be aware that the control plane (API service) is still a shared service.
You can constrain a Pod so that it is restricted or prefer to run on particular nodes. Read more on assigning pods to nodes.
Multiple Ingress Controllers
In order for the Ingress resource to work, the cluster must have an ingress controller running. You may deploy any number of ingress controllers using ingress class within a cluster.
You can have multiple ingress controllers so that public IP addresses and load-balancers are not shared across tenants. Besides, there’s no impact on other tenants when there’s an issue with one ingress controller.
Security
Security is always important but will be even more important when having a multi-tenant cluster.
What you can do the get a higher level of security within your cluster:
Policy engines: for configuration security best practices, such as only allowing trusted registries
Runtime scanners: to detect and report runtime security events
Service Mesh: for Mutual TLS communication between pods
SIEM: for use-case-driven security monitoring
Sources for security events:
Kubernetes Audit Trail
NGINX Ingress Access Logs
Container Runtime Security Events (by observing the behavior of your applications and containers using an IDS)
Other Application Security Events
Conclusion
All of these multi-tenancy models have their strengths and weaknesses, and the best approach will depend on your organization’s specific requirements. Some common factors to consider include the level of isolation required, the number of tenants, the resources available in the cluster, and compliance requirements.
Evaluating your specific use case and requirements is important to determine the best approach. With the suitable multi-tenancy model in place, Kubernetes can help you to effectively manage and scale your containerized applications while maintaining isolation and security.