|Developer(s)||Cloud Native Computing Foundation|
|Initial release||7 June 2014|
|Type||Cluster management software|
|License||Apache License 2.0|
Kubernetes (/ - , - , - /( ) ,, commonly stylized as K8s) is an open-source container orchestration system for automating software deployment, scaling, and management. Google originally designed Kubernetes, but the Cloud Native Computing Foundation now maintains the project.
Kubernetes works with Docker, Containerd, and CRI-O. Originally, it interfaced exclusively with the Docker runtime through a "Dockershim"; however, from 2016 up to April 2022, Kubernetes has deprecated the shim in favor of directly interfacing with the container through Containerd, or replacing Docker with a runtime that is compliant with the Container Runtime Interface (CRI). With the release of v1.24 in May 2022, "Dockershim" has been removed entirely.
Amazon, Google, IBM, Microsoft, Oracle, Red Hat, SUSE, Platform9 and VMware offer Kubernetes-based platforms or infrastructure as a service (IaaS) that deploy Kubernetes.
Kubernetes (κυβερνήτης, Greek for "helmsman," "pilot," or "governor", and the etymological root of cybernetics) was first announced by Google in mid-2014. Joe Beda, Brendan Burns, and Craig McLuckie were the initial founders of Kubernetes, but other Google engineers, including Brian Grant and Tim Hockin, joined them shortly thereafter. Google's Borg system had a significant influence on the design and development of Kubernetes. Many of the top contributors to the project previously worked on Borg. The original codename for Kubernetes within Google was Project 7, a reference to the Star Trek ex-Borg character Seven of Nine. The seven spokes on the wheel of the Kubernetes logo are a reference to that codename. The original Borg project was entirely in C++, but Kubernetes source code is in the Go language.
Kubernetes 1.0 was released on July 21, 2015, along which Google partnered with the Linux Foundation to form the Cloud Native Computing Foundation (CNCF) and offered Kubernetes as a seed technology. In February 2016, the Helm package manager for Kubernetes was released. On March 6, 2018, Kubernetes Project reached the ninth place in the list of GitHub projects by the number of commits, and second place in authors and issues, after the Linux kernel.
Until version 1.18, Kubernetes followed an N-2 support policy, meaning that the three most recent minor versions receive security updates and bug fixes. Starting with version 1.19, Kubernetes follows an N-3 support policy.
Kubernetes defines a set of building blocks ("primitives") that collectively provide mechanisms that deploy, maintain, and scale applications based on CPU, memory or custom metrics. Kubernetes is loosely coupled and extensible to meet different workloads. The internal components as well as extensions and containers that run on Kubernetes rely on the Kubernetes API. The platform exerts its control over compute and storage resources by defining resources as Objects, which can then be managed as such.
Kubernetes follows the primary/replica architecture. The components of Kubernetes can be divided into those that manage an individual node and those that are part of the control plane.
The Kubernetes master is the main controlling unit of the cluster, managing its workload and directing communication across the system. The Kubernetes control plane consists of various components, each its own process, that can run both on a single master node or on multiple masters supporting high-availability clusters. The various components of the Kubernetes control plane are as follows:
A node, also known as a worker or a minion, is a machine where containers (workloads) are deployed. Every node in the cluster must run a container runtime such as Docker, as well as the below-mentioned components, for communication with the primary for network configuration of these containers.
Kubernetes provides a partitioning of the resources it manages into non-overlapping sets called namespaces. They are intended for use in environments with many users spread across multiple teams, or projects, or even separating environments like development, test, and production.
The basic scheduling unit in Kubernetes is a pod, which consists of one or more containers that are guaranteed to be co-located on the same node. Each pod in Kubernetes is assigned a unique IP address within the cluster, allowing applications to use ports without the risk of conflict. Within the pod, all containers can reference each other. The containers can be running in different IP segment as well. However, for a container within one pod to access another container within another pod, it has to use the pod IP address. However, pod IP addresses are ephemeral; hence an application developer should never use hardcoded pod IP addresses because the specific pod that they are referencing may be assigned to another pod IP address on restart. Instead, they should use a reference to a service (see below), which holds a reference to the target pod at the specific pod IP address.
A pod can define a volume, such as a local disk directory or a network disk, and expose it to the containers in the pod. Pods can be managed manually through the Kubernetes API, or their management can be delegated to a controller. Such volumes are also the basis for the Kubernetes features of ConfigMaps (to provide access to configuration through the file system visible to the container) and Secrets (to provide access to credentials needed to access remote resources securely, by providing those credentials on the file system visible only to authorized containers).
Normally, the Kubernetes Scheduler decides where to run pods. For some use cases, though, there could be a need to run a pod on every single node in the cluster. This is useful for use cases like log collection, ingress controllers, and storage services. DaemonSets implement this kind of pod scheduling.
A ReplicaSet’s purpose is to maintain a stable set of replica pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.
The ReplicaSets can also be said to be a grouping mechanism that lets Kubernetes maintain the number of instances that have been declared for a given pod. The definition of a ReplicaSet uses a selector, whose evaluation will result in identifying all pods that are associated with it.
A Kubernetes service is a set of pods that work together, such as one tier of a multi-tier application. The set of pods that constitute a service are defined by a label selector. Kubernetes provides two modes of service discovery, using environmental variables or using Kubernetes DNS. Service discovery assigns a stable IP address and DNS name to the service, and load balances traffic in a round-robin manner to network connections of that IP address among the pods matching the selector (even as failures cause the pods to move from machine to machine). By default a service is exposed inside a cluster (e.g., back end pods might be grouped into a service, with requests from the front-end pods load-balanced among them), but a service can also be exposed outside a cluster (e.g., for clients to reach front-end pods).
File systems in the Kubernetes container provide ephemeral storage, by default. This means that a restart of the pod will wipe out any data on such containers, and therefore, this form of storage is quite limiting in anything but trivial applications. A Kubernetes Volume provides persistent storage that exists for the lifetime of the pod itself. This storage can also be used as shared disk space for containers within the pod. Volumes are mounted at specific mount points within the container, which are defined by the pod configuration, and cannot mount onto other volumes or link to other volumes. The same volume can be mounted at different points in the file system tree by different containers.
A common application challenge is deciding where to store and manage configuration information, some of which may contain sensitive data. Configuration data can be anything as fine-grained as individual properties or coarse-grained information like entire configuration files or JSON / XML documents. Kubernetes provides two closely related mechanisms to deal with this need: "configmaps" and "secrets", both of which allow for configuration changes to be made without requiring an application build. The data from configmaps and secrets will be made available to every single instance of the application to which these objects have been bound via the deployment. A secret and/or a configmap is only sent to a node if a pod on that node requires it. Kubernetes will keep it in memory on that node. Once the pod that depends on the secret or configmap is deleted, the in-memory copy of all bound secrets and configmaps are deleted as well. The data is accessible to the pod through one of two ways: a) as environment variables (which will be created by Kubernetes when the pod is started) or b) available on the container file system that is visible only from within the pod.
The data itself is stored on the master which is a highly secured machine which nobody should have login access to. The biggest difference between a secret and a configmap is that the content of the data in a secret is base64 encoded. Recent versions of Kubernetes have introduced support for encryption to be used as well. Secrets are often used to store data like certificates, passwords, pull secrets (credentials to work with image registries), and ssh keys.
Scaling stateless applications is only a matter of adding more running pods. Stateful workloads are harder, because the state needs to be preserved if a pod is restarted. If the application is scaled up or down, the state may need to be redistributed. Databases are an example of stateful workloads. When run in high-availability mode, many databases come with the notion of a primary instance and secondary instances. In this case, the notion of ordering of instances is important. Other applications like Apache Kafka distribute the data amongst their brokers; hence, one broker is not the same as another. In this case, the notion of instance uniqueness is important.
StatefulSets are controllers (see above) that enforce the properties of uniqueness and ordering amongst instances of a pod and can be used to run stateful applications.
A ReplicaSet declares the number of instances of a pod that is needed, and a Replication Controller manages the system so that the number of healthy pods that are running matches the number of pods declared in the ReplicaSet (determined by evaluating its selector).
Deployments are a higher level management mechanism for ReplicaSets. While the Replication Controller manages the scale of the ReplicaSet, Deployments will manage what happens to the ReplicaSet - whether an update has to be rolled out, or rolled back, etc. When deployments are scaled up or down, this results in the declaration of the ReplicaSet changing - and this change in declared state is managed by the Replication Controller.
Kubernetes enables clients (users or internal components) to attach keys called "labels" to any API object in the system, such as pods and nodes. Correspondingly, "label selectors" are queries against labels that resolve to matching objects. When a service is defined, one can define the label selectors that will be used by the service router/load balancer to select the pod instances that the traffic will be routed to. Thus, simply changing the labels of the pods or changing the label selectors on the service can be used to control which pods get traffic and which don't, which can be used to support various deployment patterns like blue-green deployments or A-B testing. This capability to dynamically control how services utilize implementing resources provides a loose coupling within the infrastructure.
For example, if an application's pods have labels for a system
tier (with values such as
back-end, for example) and a
release_track (with values such as
production, for example), then an operation on all of
canary nodes can use a label selector, such as:
tier=back-end AND release_track=canary
Just like labels, field selectors also let one select Kubernetes resources. Unlike labels, the selection is based on the attribute values inherent to the resource being selected, rather than user-defined categorization.
metadata.namespace are field selectors that will be present on all Kubernetes objects. Other selectors that can be used depend on the object/resource type.
Add-ons operate just like any other application running within the cluster: they are implemented via pods and services, and are only different in that they implement features of the Kubernetes cluster. The pods may be managed by Deployments, ReplicationControllers, and so on. There are many add-ons, and the list is growing. Some of the more important are:
Containers emerged as a way to make software portable. The container contains all the packages you need to run a service. The provided file system makes containers extremely portable and easy to use in development. A container can be moved from development to test or production with no or relatively few configuration changes.
Historically Kubernetes was suitable only for stateless services. However, many applications have a database, which requires persistence, which leads to the creation of persistent storage for Kubernetes. Implementing persistent storage for containers is one of the top challenges of Kubernetes administrators, DevOps and cloud engineers. Containers may be ephemeral, but more and more of their data is not, so one needs to ensure the data's survival in case of container termination or hardware failure. When deploying containers with Kubernetes or containerized applications, companies often realize that they need persistent storage. They need to provide fast and reliable storage for databases, root images and other data used by the containers.
In addition to the landscape, the Cloud Native Computing Foundation (CNCF), has published other information about Kubernetes Persistent Storage including a blog helping to define the container attached storage pattern. This pattern can be thought of as one that uses Kubernetes itself as a component of the storage system or service.
More information about the relative popularity of these and other approaches can be found on the CNCF's landscape survey as well, which showed that OpenEBS from MayaData and Rook - a storage orchestration project - were the two projects most likely to be in evaluation as of the Fall of 2019.
Container Attached Storage is a type of data storage that emerged as Kubernetes gained prominence. The Container Attached Storage approach or pattern relies on Kubernetes itself for certain capabilities while delivering primarily block, file, object and interfaces to workloads running on Kubernetes.
Common attributes of Container Attached Storage include the use of extensions to Kubernetes, such as custom resource definitions, and the use of Kubernetes itself for functions that otherwise would be separately developed and deployed for storage or data management. Examples of functionality delivered by custom resource definitions or by Kubernetes itself include retry logic, delivered by Kubernetes itself, and the creation and maintenance of an inventory of available storage media and volumes, typically delivered via a custom resource definition.
A key component of the Kubernetes control plane is the API Server, which exposes an HTTP API that can be invoked by other parts of the cluster as well as end users and external components. This API is a REST API and is declarative in nature. There are two kinds of API resources. Most of the API resources in the Kubernetes API are objects. These represent a concrete instance of a concept on the cluster, like a pod or namespace. A small number of API resource types are "virtual". These represent operations rather than objects, such as a permission check, using the "subjectaccessreviews" resource. API resources that correspond to objects will be represented in the cluster with unique identifiers for the objects. Virtual resources do not have unique identifiers.
Kubernetes can be extended using Custom Resources. These API resources represent objects that are not part of the standard Kubernetes product. These resources can appear and disappear in a running cluster through dynamic registration. Cluster administrators can update Custom Resources independently of the cluster.
Custom Controllers are another extension mechanism. These interact with Custom Resources, and allow for a true declarative API that allows for the lifecycle management of Custom Resource that is aligned with the way that Kubernetes itself is designed. The combination of Custom Resources and Custom Controllers are often referred to as an (Kubernetes) Operator. The key use case for Operators are to capture the aim of a human operator who is managing a service or set of services and to implement them using automation, and with a declarative API supporting this automation. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems. Examples of problems solved by Operators include taking and restoring backups of that application's state, and handling upgrades of the application code alongside related changes such as database schemas or extra configuration settings.
The same API design principles have been used to define an API to programmatically create, configure, and manage Kubernetes clusters. This is called the Cluster API. A key concept embodied in the API is using Infrastructure as Software, or the notion that the Kubernetes cluster infrastructure is itself a resource / object that can be managed just like any other Kubernetes resources. Similarly, machines that make up the cluster are also treated as a Kubernetes resource. The API has two pieces - the core API, and a provider implementation. The provider implementation consists of cloud-provider specific functions that let Kubernetes provide the cluster API in a fashion that is well-integrated with the cloud-provider's services and resources.
Kubernetes is commonly used as a way to host a microservice-based implementation, because it and its associated ecosystem of tools provide all the capabilities needed to address key concerns of any microservice architecture. It is available in three forms: open source, commercial, and managed. Open source distributions include the original Kubernetes, Amazon EKS-D, Red Hat OpenShift, VMware Tanzu, Mirantis Kubernetes Engine, and D2iQ Kubernetes Platform. Managed offerings include GKE, Oracle Container Engine for Kubernetes, Amazon Elastic Kubernetes Service, IBM Kubernetes Service, and Platform9 Managed Kubernetes.
|Version||Release date||End of Life date||Notes|
|Old version, no longer maintained: 1.0||10 July 2015||Original Release|
|Old version, no longer maintained: 1.1||9 November 2015||https://kubernetes.io/blog/2015/11/kubernetes-1-1-performance-upgrades-improved-tooling-and-a-growing-community|
|Old version, no longer maintained: 1.2||16 March 2016||23 October 2016||https://kubernetes.io/blog/2016/03/kubernetes-1-2-even-more-performance-upgrades-plus-easier-application-deployment-and-management|
|Old version, no longer maintained: 1.3||1 July 2016||1 November 2016||https://kubernetes.io/blog/2016/07/kubernetes-1-3-bridging-cloud-native-and-enterprise-workloads|
|Old version, no longer maintained: 1.4||26 September 2016||21 April 2017||https://kubernetes.io/blog/2016/09/kubernetes-1-4-making-it-easy-to-run-on-kuberentes-anywhere|
|Old version, no longer maintained: 1.5||12 December 2016||1 October 2017||https://kubernetes.io/blog/2016/12/kubernetes-1-5-supporting-production-workloads|
|Old version, no longer maintained: 1.6||28 March 2017||23 November 2017||https://kubernetes.io/blog/2017/03/kubernetes-1-6-multi-user-multi-workloads-at-scale|
|Old version, no longer maintained: 1.7||30 June 2017||4 April 2018||https://kubernetes.io/blog/2017/06/kubernetes-1-7-security-hardening-stateful-application-extensibility-updates|
|Old version, no longer maintained: 1.8||28 August 2017||12 July 2018||https://kubernetes.io/blog/2017/09/kubernetes-18-security-workloads-and|
|Old version, no longer maintained: 1.9||15 December 2017||29 September 2018||https://kubernetes.io/blog/2017/12/kubernetes-19-workloads-expanded-ecosystem|
|Old version, no longer maintained: 1.10||28 March 2018||13 February 2019||https://kubernetes.io/blog/2018/03/26/kubernetes-1.10-stabilizing-storage-security-networking|
|Old version, no longer maintained: 1.11||3 July 2018||1 May 2019||https://kubernetes.io/blog/2018/06/27/kubernetes-1.11-release-announcement|
|Old version, no longer maintained: 1.12||27 September 2018||8 July 2019||https://kubernetes.io/blog/2018/09/27/kubernetes-1.12-kubelet-tls-bootstrap-and-azure-virtual-machine-scale-sets-vmss-move-to-general-availability|
|Old version, no longer maintained: 1.13||3 December 2018||15 October 2019||https://kubernetes.io/blog/2018/12/03/kubernetes-1-13-release-announcement|
|Old version, no longer maintained: 1.14||25 March 2019||11 December 2019||https://kubernetes.io/blog/2019/03/25/kubernetes-1-14-release-announcement|
|Old version, no longer maintained: 1.15||20 June 2019||6 May 2020||https://kubernetes.io/blog/2019/06/19/kubernetes-1-15-release-announcement|
|Old version, no longer maintained: 1.16||22 October 2019||2 September 2020||https://kubernetes.io/blog/2019/09/18/kubernetes-1-16-release-announcement|
|Old version, no longer maintained: 1.17||9 December 2019||13 January 2021||https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-release-announcement|
|Old version, no longer maintained: 1.18||25 March 2020||18 June 2021||https://kubernetes.io/blog/2020/03/25/kubernetes-1-18-release-announcement|
|Old version, no longer maintained: 1.19||26 August 2020||28 October 2021||From Kubernetes version 1.19 on, the support window has been extended to one year of full support plus two months of maintenance mode period.|
|Old version, no longer maintained: 1.20||8 December 2020||28 February 2022||https://kubernetes.io/blog/2020/12/08/kubernetes-1-20-release-announcement/|
|Older version, yet still maintained: 1.21||8 April 2021||28 June 2022||https://kubernetes.io/blog/2021/04/08/kubernetes-1-21-release-announcement/|
|Older version, yet still maintained: 1.22||4 August 2021||28 October 2022||https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/|
|Older version, yet still maintained: 1.23||7 December 2021||28 February 2023||https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/|
|Current stable version: 1.24||3 May 2022||29 September 2023||https://kubernetes.io/blog/2022/05/03/kubernetes-1-24-release-announcement/|
The chart below visualises the period for which each release is/was supported
Compared to the 1.5 million projects on GitHub, Kubernetes is No. 9 for commits and No. 2 for authors/issues, second only to Linux.
One of the most important primary services is an API server. This is the main management point of the entire cluster as it allows a user to configure Kubernetes' workloads and organizational units. It is also responsible for making sure that the etcd store and the service details of deployed containers are in agreement. It acts as the bridge between various components to maintain cluster health and disseminate information and commands.
((cite web)): CS1 maint: url-status (link)