A workload is an application running on Kubernetes inside a set of pods. A Pod represents a set of running containers on your cluster.

• Pods have a defined lifecycle. If a pod fails, you would need to create a new Pod to recover.
• To manage pods, you can use workload resources that manage a set of pods on your behalf.

Workload Resources

Kubernetes provides several built-in workload resources:

1. Deployment and ReplicaSet: Good for managing a stateless application workload, where any Pod is interchangeable and can be replaced if needed.
2. StatefulSet: Lets you run related Pods that track state. For example, if your workload records data persistently, you can run a StatefulSet that matches each Pod with a PersistentVolume.
3. DaemonSet: Defines Pods that provide node-local facilities. These might be fundamental to the operation of your cluster, such as a networking helper tool, or be part of an add-on. Every time you add a node to your cluster that matches the specification in a DaemonSet, the control plane schedules a Pod for that DaemonSet onto the new node.
4. Job and CronJob: Define tasks that run to completion and then stop. Jobs represent one-off tasks, whereas CronJobs recur according to a schedule.

In the wider Kubernetes ecosystem, you can find third-party workload resources that provide additional behaviors. Using a custom resource definition, you can add in a third-party workload resource if you want a specific behavior that's not part of Kubernetes' core.

Supporting Concepts

• Garbage collection: Tidies up objects from your cluster after their owning resource has been removed.
• The time-to-live after finished controller: Removes Jobs once a defined time has passed since they completed.

Further Resources

Once your application is running, you might want to make it available on the internet as a Service or, for web applications only, using an Ingress.

Pods: Summary

• A Pod is the smallest deployable unit in Kubernetes. It can contain one or more containers sharing storage and network resources.
• Pods can also contain init containers that run during Pod startup and ephemeral containers for debugging, if supported by the cluster.
• Pods have shared Linux namespaces, cgroups, and potential other facets of isolation, similar to a set of containers.
• Pods can be created directly but are usually created using workload resources such as Deployment or Job.

Types of Pods

1. Single Container Pods: Most common use-case where a Pod wraps around a single container.
2. Multi-container Pods: These Pods encapsulate an application composed of multiple co-located containers that are tightly coupled and need to share resources. They form a single cohesive unit of service.

Pod Management

• Each Pod runs a single instance of a given application, with replication handled by multiple Pods.
• Pods support multiple cooperating processes (as containers) that form a cohesive unit of service.
• Workload resources can be used to manage multiple Pods.
• Controllers for workload resources create Pods from a pod template and manage those Pods.

Pod Lifecycle

• Pods are considered relatively ephemeral, disposable entities. They remain on their node until they finish execution, the Pod object is deleted, the Pod is evicted for lack of resources, or the node fails.
• Restarting a container in a Pod is different from restarting a Pod. A Pod persists until it is deleted.

Pod Networking and Storage

• Each Pod is assigned a unique IP address. All containers in a Pod share the network namespace, including the IP address and network ports.
• A Pod can specify a set of shared storage volumes. All containers in the Pod can access the shared volumes, allowing those containers to share data.

Privileged Mode for Containers

• Containers in a Pod can run in privileged mode to use operating system administrative capabilities.

Static Pods

• Managed directly by the kubelet daemon on a specific node, without the API server observing them. Useful for running a self-hosted control plane.

Container Probes

• Probes are diagnostics performed periodically by the kubelet on a container, used to check the health of the container.

Operating System Support

• As of Kubernetes v1.25, only Linux and Windows are supported. The .spec.os.name field indicates the OS on which you want the Pod to run.

Pod Templates

• Controllers for workload resources (like Deployments, Jobs, and DaemonSets) create Pods using a Pod template.
• Pod templates are specifications for creating Pods and are part of the desired state of the workload resource.
• Modifying the Pod template or switching to a new Pod template does not directly affect existing Pods. Instead, the workload resource controller creates new Pods based on the updated template.
• Each workload resource has its own rules for handling changes to the Pod template.

Pod Update and Replacement

• If the Pod template changes, the controller creates new Pods based on the updated template.
• Kubernetes allows you to update some fields of a running Pod, in place. However, limitations exist:
• Most metadata about a Pod is immutable.
• If the metadata.deletionTimestamp is set, no new entry can be added to the metadata.finalizers list.
• Pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds, or spec.tolerations.

Resource Sharing and Communication

• Pods can specify a set of shared storage volumes, accessible by all containers in the Pod.
• Each Pod is assigned a unique IP address. Every container in a Pod shares the network namespace, including the IP address and network ports.
• Containers within the Pod can communicate with each other using localhost. They must coordinate their use of shared network resources when communicating with entities outside the Pod.
• Containers in a Pod can also communicate with each other using standard inter-process communications like SystemV semaphores or POSIX shared memory.
• Containers in different Pods have distinct IP addresses and can not communicate by OS-level IPC without special configuration.
• Containers that want to interact with a container running in a different Pod can use IP networking to communicate.

Privileged Mode for Containers

• Any container in a pod can run in privileged mode to access operating system administrative capabilities.
• This is available for both Windows and Linux.

Linux Privileged Containers

• On Linux, containers can enable privileged mode using the privileged flag on the security context of the container spec.

Windows Privileged Containers

• As of Kubernetes v1.26, you can create a Windows HostProcess pod by setting the windowsOptions.hostProcess flag on the security context of the pod spec.

Static Pods

• Static Pods are managed directly by the kubelet daemon on a specific node, not observed by the API server.
• Static Pods are always bound to one Kubelet on a specific node.
• The kubelet automatically tries to create a mirror Pod on the Kubernetes API server for each static Pod.
• The Pods running on a node are visible on the API server, but cannot be controlled from there.

Container Probes

• A probe is a diagnostic performed periodically by the kubelet on a container.
• The kubelet can perform diagnostics through different actions:
• ExecAction (performed with the help of the container runtime)
• TCPSocketAction (checked directly by the kubelet)
• HTTPGetAction (checked directly by the kubelet)

Kubernetes Pod Lifecycle

Pod Lifetime

• Pods are ephemeral entities, created once, assigned a unique ID (UID), and scheduled to a node until terminated or deleted.
• If a Node fails, Pods on the node are scheduled for deletion after a timeout period.
• A Pod isn't rescheduled to a different node if it fails. Instead, a new Pod can replace it with a different UID.
• Objects tied to a Pod's lifetime, like a volume, are also destroyed when the Pod is deleted.

Pod Phases

• A Pod moves through several phases in its lifecycle:
• Pending: Accepted by the Kubernetes cluster, but some containers aren't ready yet.
• Running: Bound to a node, all containers have been created, and at least one container is still running.
• Succeeded: All containers in the Pod terminated successfully, will not be restarted.
• Failed: All containers in the Pod terminated, with at least one failure (either non-zero exit status or system termination).
• Unknown: Due to an error, the state of the Pod couldn't be obtained.

Pod and Container States

• Each container in a Pod has three possible states: Waiting, Running, and Terminated.
• A container's lifecycle can be influenced by hooks that trigger events at certain points.

Waiting

• If a container is not Running or Terminated, it's in the Waiting state. Reasons for Waiting might include pulling the container image or applying Secret data.

Running

• A Running container is executing without issues. If a postStart hook was configured, it's already executed and finished.

Terminated

• A Terminated container either ran to completion or failed. If a preStop hook was configured, it runs before the container enters the Terminated state.

Container Restart Policy

• A Pod's spec contains a restartPolicy field. Values include "Always" (default), "OnFailure", and "Never".
• restartPolicy only refers to restarts of the containers by the kubelet on the same node.
• After containers exit, the kubelet restarts them with an exponential back-off delay, capped at five minutes.

Pod Conditions

• A Pod has a PodStatus, which has an array of PodConditions.
• Kubelet manages these conditions: PodScheduled, ContainersReady, Initialized, Ready, and PodHasNetwork (alpha feature).

Pod Readiness

• Pod readiness is a feature that allows your application to provide additional signals to the PodStatus.
• It uses readinessGates in the Pod's spec to list extra conditions that the kubelet checks for Pod readiness.
• The status of these conditions is extracted from the status.condition fields of the Pod.
• If the condition isn't found, it defaults to "False".
• The condition names should conform to the Kubernetes label key format.
• Custom conditions and readiness of all containers are required for a Pod to be considered ready.
• If all containers are ready, but at least one custom condition is missing or False, the kubelet sets the Pod's condition to ContainersReady.

Pod Network Readiness

• After being scheduled on a node and admitted by the Kubelet, a Pod needs to have volumes mounted and network configuration set up.
• The PodHasNetworkCondition feature gate allows Kubelet to report the Pod's network initialization status.
• The PodHasNetwork condition can be set to False in early or later stages of the Pod's lifecycle under certain circumstances.
• This condition is set to True after successful sandbox creation and network configuration.

Pod Scheduling Readiness

• This feature is currently in alpha and additional information can be found in the Pod Scheduling Readiness documentation.

Container Probes

• Probes are diagnostics performed periodically by the kubelet on a container.
• There are four types of probes: exec, grpc, httpGet, and tcpSocket.
• Each probe has three possible outcomes: Success, Failure, Unknown.
• Kubelet can perform and react to three kinds of probes on running containers: livenessProbe, readinessProbe, and startupProbe.

Liveness Probe

• It indicates whether the container is running.
• If it fails, the kubelet kills the container and the container follows its restart policy.
• Default state is Success if a liveness probe is not provided.

Readiness Probe

• It indicates whether the container is ready to respond to requests.
• If it fails, the endpoints controller removes the Pod's IP address from all matching Services' endpoints.
• The default state before the initial delay is Failure if a readiness probe is not provided.

Startup Probe

• It indicates whether the application within the container has started.
• All other probes are disabled if a startup probe is provided, until it succeeds.
• If it fails, the kubelet kills the container and the container follows its restart policy.
• Default state is Success if a startup probe is not provided.

When to Use Probes

• Liveness probe: Useful if you want your container to be killed and restarted if a probe fails.
• Readiness probe: Useful to start sending traffic to a Pod only when a probe succeeds, or for taking the container down for maintenance.
• Startup probe: Useful for Pods that have containers that take a long time to start up.

Note

• On deletion, a Pod puts itself into an unready state regardless of the readiness probe existence.
• It stays in this state while waiting for the containers in the Pod to stop.

Kubernetes Pod Termination Process

Pods in Kubernetes represent processes running on nodes. It's crucial to allow these processes to terminate gracefully instead of abruptly stopping them with a KILL signal. The key steps involved in this process are:

1. Requesting Deletion: When you request a Pod's deletion, the cluster records and starts tracking the grace period before the Pod can be forcefully killed.
2. Graceful Shutdown: The container runtime typically sends a TERM signal to each container's main process. If the grace period expires and processes are still running, a KILL signal is sent, and the Pod is deleted from the API Server.
3. Process Interruption Handling: If the kubelet or container runtime's management service restarts during process termination, the cluster retries from the start with the full original grace period.

Deletion Flow Example:

1. Request Deletion: Delete a specific Pod using the kubectl tool with a default grace period of 30 seconds.
2. Pod Status Update: The API server updates the Pod with a "dead" status and grace period. If checked with kubectl describe, the Pod shows up as "Terminating".
3. Local Shutdown Process: The kubelet on the node starts the local Pod shutdown process once it detects the Pod as terminating.
4. PreStop Hook: If a preStop hook is defined in any of the Pod's containers, the kubelet runs it. If it's still running after the grace period, the kubelet requests a grace period extension of 2 seconds.
5. TERM Signal: The kubelet triggers the container runtime to send a TERM signal to process 1 in each container.
6. Service Interruption: The control plane evaluates whether to remove the shutting-down Pod from EndpointSlice (and Endpoints) objects. ReplicaSets and other resources no longer treat the shutting-down Pod as a valid, in-service replica.
7. Forcible Shutdown: When the grace period expires, the kubelet triggers forcible shutdown. Any remaining processes in any container in the Pod receive SIGKILL. The kubelet also cleans up a hidden pause container if the container runtime uses one.
8. Pod Transition: The kubelet transitions the Pod into a terminal phase (Failed or Succeeded) depending on the end state of its containers.
9. Forcible Removal: The kubelet triggers forcible removal of the Pod object from the API server by setting the grace period to 0 for immediate deletion.
10. Deletion: The API server deletes the Pod's API object, rendering it invisible from any client.

Forced Pod Termination

Forced deletions can be potentially disruptive. By default, all deletions are graceful within 30 seconds. However, you can override this default with the --grace-period=<seconds> option in the kubectl delete command.

Setting the grace period to 0 forcibly and immediately deletes the Pod from the API server and triggers immediate cleanup by the kubelet on the node.

Garbage Collection of Pods

For failed Pods, their API objects remain in the cluster's API until explicitly removed. The Pod garbage collector (PodGC) in the control plane cleans up terminated Pods when the number of Pods exceeds the configured threshold. PodGC also cleans up Pods that are orphan pods, unscheduled terminating pods, or terminating pods bound to a non-ready node tainted with node.kubernetes.io/out-of-service.

Kubernetes supports several types of volumes, including:

Deprecated Volumes

• awsElasticBlockStore
• azureDisk
• azureFile
• gcePersistentDisk

Migrating Volumes

• AWS EBS, Azure Disk, and Azure File have respective CSI migration paths.

Other Volume Types

• cephfs
• cinder
• OpenStack has a CSI migration path.

ConfigMap

• Injects configuration data into pods.
• You must create a ConfigMap before you can use it.
• A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates.
• Text data is exposed as files using the UTF-8 character encoding.

DownwardAPI

• Makes downward API data available to applications.
• Exposed data is in read-only files in plain text format.
• A container using the downward API as a subPath volume mount does not receive updates when field values change.

EmptyDir

• Created when a Pod is assigned to a node and exists as long as that Pod is running on that node.
• All containers in the Pod can read and write the same files in the emptyDir volume.
• The data in an emptyDir volume is safe across container crashes.
• A size limit can be specified for the default medium, which limits the capacity of the emptyDir volume.

Fibre Channel (fc)

• Allows an existing fibre channel block storage volume to mount in a Pod.
• You must configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs beforehand so that Kubernetes hosts can access them.

GCE Persistent Disk

• The contents of a PD are preserved and the volume is merely unmounted when a pod is removed.
• A gcePersistentDisk volume permits multiple consumers to simultaneously mount a persistent disk as read-only.
• Using a GCE persistent disk with a Pod controlled by a ReplicaSet will fail unless the PD is read-only or the replica count is 0 or 1.

Regional Persistent Disks

• Available in two zones within the same region.
• Must be provisioned as a PersistentVolume; referencing the volume directly from a pod is not supported.

GCE CSI Migration

• Redirects all plugin operations from the existing in-tree plugin to the pd.csi.storage.gke.io Container Storage Interface (CSI) Driver.
• The GCE PD CSI Driver must be installed on the cluster.

Please refer to the detailed documentation for more specific configuration examples and further usage details.

gitRepo (deprecated)

• The gitRepo volume type is deprecated.
• To provision a container with a git repo, use an EmptyDir with an InitContainer to clone the repo.
• A gitRepo volume is a type of volume plugin that mounts an empty directory and clones a git repository into this directory for the Pod to use.

glusterfs (removed)

• As of Kubernetes 1.27, the glusterfs volume type is no longer included.
• The GlusterFS in-tree storage driver was deprecated in Kubernetes v1.25 and removed entirely in v1.26.

hostPath

• HostPath volumes mount a file or directory from the host node's filesystem into the Pod.
• They pose security risks and should be used sparingly and with specific access controls.
• They can be used for containers needing access to Docker internals, running cAdvisor, or allowing a Pod to specify whether a given hostPath should exist prior to the Pod running.
• Can optionally specify a type for a hostPath volume. Types include DirectoryOrCreate, Directory, FileOrCreate, File, Socket, CharDevice, and BlockDevice.

iscsi

• iSCSI volume allows an existing iSCSI (SCSI over IP) volume to be mounted into your Pod.
• Its contents are preserved and the volume is merely unmounted when a Pod is removed.
• Can be mounted as read-only by multiple consumers simultaneously.

local

• A local volume represents a mounted local storage device such as a disk, partition or directory.
• They can only be used as a statically created PersistentVolume and dynamic provisioning is not supported.
• Subject to the availability of the underlying node and not suitable for all applications.
• Recommended to create a StorageClass with volumeBindingMode set to WaitForFirstConsumer when using local volumes.

nfs

• An NFS volume allows an existing NFS (Network File System) share to be mounted into a Pod.
• Its contents are preserved and the volume is merely unmounted when a Pod is removed.
• NFS can be mounted by multiple writers simultaneously.

persistentVolumeClaim

• A persistentVolumeClaim volume is used to mount a PersistentVolume into a Pod.
• Allows users to "claim" durable storage without knowing the details of the particular cloud environment.

portworxVolume (deprecated)

• A portworxVolume is an elastic block storage layer that runs hyperconverged with Kubernetes, but it's deprecated as of Kubernetes v1.25.
• Can be dynamically created through Kubernetes or pre-provisioned and referenced inside a Pod.
• Portworx CSI migration feature is in beta state as of Kubernetes v1.25, but turned off by default.
• It redirects all plugin operations from the existing in-tree plugin to the pxd.portworx.com Container Storage Interface (CSI) Driver, which must be installed on the cluster.

Projected Volume

• A projected volume maps multiple existing volume sources into the same directory.

Rados Block Device (RBD) Volume

• An RBD volume allows a Rados Block Device volume to mount into your Pod.
• Unlike emptyDir, an RBD volume's contents persist even when a pod is removed.
• This allows pre-population of data that can be shared between pods.
• You need a running Ceph installation to use RBD.
• RBD volumes can be mounted as read-only by multiple consumers simultaneously, but can only be mounted by a single consumer in read-write mode.

RBD CSI Migration

• As of Kubernetes v1.23 (alpha), all plugin operations for RBD can be redirected to the rbd.csi.ceph.com CSI driver when the CSIMigrationRBD feature gate is enabled.
• To use this, you must:
• Install the Ceph CSI driver (v3.5.0 or above) in your Kubernetes cluster.
• Create a clusterID based on the monitors hash in the CSI config map.
• If the adminId value in the StorageClass differs from admin, patch the adminSecretName with the base64 value of the adminId parameter value.

Secret Volume

• Secret volumes are used to pass sensitive information to Pods.
• Secrets can be stored in the Kubernetes API and mounted as files for use by pods.
• They are backed by tmpfs, a RAM-backed filesystem, so they are never written to non-volatile storage.
• You must create a Secret in the Kubernetes API before using it.
• A container using a Secret as a subPath volume mount won't receive Secret updates.

vSphereVolume (deprecated)

• vSphereVolume is used to mount a vSphere VMDK volume into your Pod and supports both VMFS and VSAN datastores.
• The contents of a volume persist when it is unmounted.
• The use of vSphere CSI out-of-tree driver is recommended instead of vSphereVolume.

vSphere CSI Migration

• As of Kubernetes v1.26 (stable), all operations for the in-tree vsphereVolume type are redirected to the csi.vsphere.vmware.com CSI driver.
• To migrate, you must:
• Install the vSphere CSI driver on your cluster.
• Run vSphere 7.0u2 or later.
• Some StorageClass parameters from the built-in vsphereVolume plugin are not supported by the vSphere CSI driver.

vSphere CSI Migration Complete

• As of Kubernetes v1.19 (beta), to disable the vsphereVolume plugin, you need to set InTreePluginvSphereUnregister feature flag to true.
• The csi.vsphere.vmware.com CSI driver must be installed on all worker nodes.

Using subPath

• The volumeMounts.subPath property is used to specify a sub-path inside a referenced volume instead of the root. This allows sharing a single volume for multiple uses in one pod.
• Example: A LAMP (Linux Apache MySQL PHP) stack pod configures a shared volume, mapping the PHP application's code and assets to the volume's html folder and the MySQL database to the mysql folder.

Using subPath with Expanded Environment Variables

• subPathExpr is used to construct subPath directory names from downward API environment variables. subPath and subPathExpr properties are mutually exclusive.
• Example: A Pod uses subPathExpr to create a directory pod1 within the hostPath volume /var/log/pods. The directory /var/log/pods/pod1 is mounted at /logs in the container.

Resources

• The storage media of an emptyDir volume is determined by the filesystem holding the kubelet root dir (typically /var/lib/kubelet). There's no space limit or isolation for emptyDir or hostPath volumes.

Out-of-tree Volume Plugins

• Out-of-tree volume plugins, like Container Storage Interface (CSI) and FlexVolume (deprecated), allow storage vendors to create custom storage plugins without adding their plugin source code to the Kubernetes repository.
• These plugins can be developed independently of the Kubernetes code base and deployed on Kubernetes clusters as extensions.

CSI

• CSI defines a standard interface for container orchestration systems to expose arbitrary storage systems to their container workloads.
• A CSI compatible volume driver, once deployed on a Kubernetes cluster, allows users to use the csi volume type to attach or mount the volumes exposed by the CSI driver.

CSI Ephemeral Volumes

• CSI volumes can be directly configured within the Pod specification. These volumes are ephemeral and do not persist across pod restarts.

Windows CSI Proxy

• For Windows worker nodes, privileged operations for containerized CSI node plugins are supported using csi-proxy, a community-managed, stand-alone binary that needs to be pre-installed on each Windows node.

Migrating to CSI Drivers from In-tree Plugins

• The CSIMigration feature redirects operations against existing in-tree plugins to corresponding CSI plugins. This allows a smooth transition to a CSI driver that supersedes an in-tree plugin without any configuration changes.

FlexVolume

• FlexVolume is a deprecated out-of-tree plugin interface that uses an exec-based model to interface with storage drivers. It's recommended to use an out-of-tree CSI driver for integrating external storage with Kubernetes.

Mount Propagation

• Mount propagation allows sharing volumes mounted by a container with other containers in the same pod or even other pods on the same node.
• Three propagation modes are available: None (default), HostToContainer, and Bidirectional.
• Bidirectional mount propagation is only allowed in privileged containers due to potential damage to the host operating system.

Configuration

• For mount propagation to work properly on some deployments (CoreOS, RedHat/Centos, Ubuntu), Docker's MountFlags must be configured correctly. After the configuration, Docker daemon needs to be restarted.

Kubernetes Persistent Volumes and Persistent Volume Claims

Introduction

• Persistent Volume (PV): A piece of storage in the cluster that is provisioned by an administrator or dynamically provisioned using Storage Classes. It has a lifecycle independent of any individual Pod that uses the PV. It captures the details of the implementation of the storage system.
• Persistent Volume Claim (PVC): A user's request for storage. It is similar to a Pod - as Pods consume node resources, PVCs consume PV resources. PVCs can request specific size and access modes.
• StorageClass resource: Allows cluster administrators to offer a variety of PersistentVolumes with different properties, without exposing users to the implementation details.

Lifecycle of a Volume and Claim

Provisioning: PVs can be provisioned statically or dynamically.

• Static: A cluster administrator creates a number of PVs which exist in the Kubernetes API for consumption.
• Dynamic: When no static PVs match a user's PVC, the cluster may dynamically provision a volume for the PVC based on StorageClasses.
• Binding: A control loop in the master watches for new PVCs, finds a matching PV (if possible), and binds them together. Once bound, PVC to PV binding is exclusive and one-to-one.
• Using: Pods use claims as volumes. The cluster inspects the claim to find the bound volume and mounts that volume for a Pod.
• Storage Object in Use Protection: Ensures that PVCs in active use by a Pod and PVs that are bound to PVCs are not removed from the system to prevent data loss.

Reclaiming: When a user is done with their volume, they can delete the PVC objects from the API for resource reclamation. The reclaim policy for a PV tells the cluster what to do with the volume after it has been released of its claim. Policies include:

• Retain: Allows for manual reclamation of the resource.
• Delete: Removes both the PV object from Kubernetes and the associated storage asset in the external infrastructure.
• Recycle (deprecated): Performs a basic scrub on the volume and makes it available again for a new claim.

Note

A PVC is in active use by a Pod when a Pod object exists that is using the PVC. PVC or PV removal is postponed until the PVC is no longer actively used by any Pods, or the PV is no longer bound to a PVC.

PersistentVolume Deletion Protection Finalizer

• PersistentVolumes with a Delete reclaim policy are only deleted after the backing storage is deleted.
• Two finalizers are introduced, kubernetes.io/pv-controller for in-tree plugin volumes, and external-provisioner.volume.kubernetes.io/finalizer for CSI volumes.
• When the CSIMigration{provider} feature flag is enabled for an in-tree volume plugin, kubernetes.io/pv-controller is replaced by external-provisioner.volume.kubernetes.io/finalizer.

Reserving a PersistentVolume

• You can pre-bind a PersistentVolumeClaim (PVC) to a specific PersistentVolume (PV).
• You specify a PV in a PVC to declare a binding.
• The binding happens regardless of some volume matching criteria, but the control plane still checks that storage class, access modes, and requested storage size are valid.
• To reserve a specific storage volume, specify the relevant PVC in the claimRef field of the PV so that other PVCs can't bind to it.

Expanding Persistent Volumes Claims

• PVCs can be expanded if their storage class's allowVolumeExpansion field is set to true.
• A larger volume for a PVC can be requested by editing the PVC object and specifying a larger size.
• Directly editing the size of a PersistentVolume can prevent an automatic resize of that volume.

CSI Volume Expansion

• CSI volume expansion requires a specific CSI driver to support volume expansion.
• Resizing a volume containing a file system can only be done if the file system is XFS, Ext3, or Ext4.
• File system expansion is either done when a Pod is starting up or when a Pod is running and the underlying file system supports online expansion.

Resizing an in-use PersistentVolumeClaim

• In-use PVCs automatically become available to its Pod as soon as its file system has been expanded.
• This has no effect on PVCs that are not in use by a Pod or deployment.

Recovering from Failure when Expanding Volumes

• If a new size specified is too big to be satisfied by underlying storage system, expansion of PVC will be continuously retried until action is taken.
• A manual recovery can be done by the cluster administrator to cancel the resize requests.

Types of Persistent Volumes

• PersistentVolume types are implemented as plugins.
• Current supported types: cephfs, csi, fc, hostPath, iscsi, local, nfs, rbd.
• Deprecated types: awsElasticBlockStore, azureDisk, azureFile, cinder, flexVolume, gcePersistentDisk, portworxVolume, vsphereVolume. These are still supported but will be removed in a future Kubernetes release.
• No longer supported types: photonPersistentDisk

PersistentVolume (PV) Specifications

• Spec and Status: Specification and status of the volume.
• Name: Must be a valid DNS subdomain name.
• Capacity: Storage capacity (e.g., 5Gi).
• VolumeMode: Filesystem (default) or Block.
• AccessModes: ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod (beta).
• StorageClassName: Specifies the class of the PV.
• PersistentVolumeReclaimPolicy: Retain, Recycle, or Delete.
• MountOptions: Additional options for mounting PV on a node (not supported by all PV types).

Capacity

• Storage size is the only resource that can be set or requested (for now).

Volume Mode

• Filesystem: Mounted into Pods into a directory.
• Block: Presented as a raw block device without any filesystem on it (useful for the fastest possible access).

Access Modes

• ReadWriteOnce (RWO): Volume can be mounted as read-write by a single node.
• ReadOnlyMany (ROX): Volume can be mounted as read-only by many nodes.
• ReadWriteMany (RWX): Volume can be mounted as read-write by many nodes.
• ReadWriteOncePod (RWOP, beta): Volume can be mounted as read-write by a single Pod.

Storage Class

• StorageClassName attribute specifies the class of a PV.
• PVs with no storageClassName have no class.

Reclaim Policy

• Retain: Manual reclamation.
• Recycle: Basic scrub (rm -rf /thevolume/*).
• Delete: Deletes associated storage asset.

Mount Options

• Specify additional options for mounting a PV on a node.
• Not all PV types support mount options.

Node Affinity

• Define constraints to limit what nodes a volume can be accessed from.
• Automatically populated for AWS EBS, GCE PD, and Azure Disk volume block types.

Volume Phases

• Available: Free resource not yet bound to a claim.
• Bound: Volume is bound to a claim.
• Released: Claim has been deleted, resource not yet reclaimed by the cluster.
• Failed: Volume has failed its automatic reclamation.

PersistentVolumeClaims (PVC)

Overview

A PersistentVolumeClaim (PVC) specifies how much storage a user needs and how it should be accessed. It's a request for storage by a user.

Key Components

• Spec and Status: Each PVC contains a spec and status, which outlines the specifications and status of the claim.
• Access Modes: Claims follow the same conventions as volumes when requesting storage.
• Volume Modes: Claims use the same conventions as volumes for consumption either as a filesystem or block device.
• Resources: Like Pods, claims can request specific quantities of a resource, in this case, storage.
• Selector: Claims can specify a label selector to filter the set of volumes. Only volumes whose labels match the selector can be bound to the claim.
• Class: A claim can request a particular class by specifying the name of a StorageClass using the attribute storageClassName.

Selectors

Two types of fields in selectors:

1. matchLabels: The volume must have a label with this value.
2. matchExpressions: A list of requirements made by specifying key, list of values, and operator that relates the key and values.

Class

A claim can request a particular class by specifying the name of a StorageClass using the attribute storageClassName. PVCs don't necessarily have to request a class. The handling of storageClassName depends on whether the DefaultStorageClass admission plugin is turned on or off.

Retroactive Default StorageClass Assignment

In the event that a PVC is created without specifying a storageClassName, the control plane identifies any existing PVCs without storageClassName and updates those PVCs to match the new default StorageClass when it becomes available.

Claims as Volumes

Pods access storage by using the claim as a volume. Claims must exist in the same namespace as the Pod using the claim.

Namespaces

Since PersistentVolumeClaims are namespaced objects, mounting claims with "Many" modes (ROX, RWX) is only possible within one namespace.

Raw Block Volume Support

Certain volume plugins support raw block volumes, including dynamic provisioning where applicable.

Binding Block Volumes

When a user requests a raw block volume by using the volumeMode field in the PVC spec, the binding rules change. There's a matrix for possible combinations of requesting a raw block device.

Volume Snapshot and Restore Volume from Snapshot

• Supported only by out-of-tree CSI volume plugins (In-tree volume plugins are deprecated).
• To create a PersistentVolumeClaim (PVC) from a Volume Snapshot:

yamlCopy codeapiVersion: v1 kind: PersistentVolumeClaim metadata: name: restore-pvc spec:storageClassName: csi-hostpath-sc dataSource: name: new-snapshot-test kind:VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources:requests: storage: 10Gi

Volume Cloning

• Only available for CSI volume plugins.
• To create a PVC from an existing PVC:

yamlCopy codeapiVersion: v1 kind: PersistentVolumeClaim metadata: name: cloned-pvc spec:storageClassName: my-csi-plugin dataSource: name: existing-src-pvc-name kind:PersistentVolumeClaim accessModes: - ReadWriteOnce resources: requests: storage: 10Gi

Volume Populators and Data Sources

• Kubernetes supports custom volume populators; must enable the AnyVolumeDataSource feature gate.
• The dataSourceRef field can contain a reference to any object in the same namespace, except for core objects other than PVCs.
• Use of dataSourceRef is preferred over dataSource for clusters with the feature gate enabled.

Cross Namespace Data Sources

• Kubernetes supports cross-namespace volume data sources; must enable AnyVolumeDataSource and CrossNamespaceVolumeDataSource feature gates.
• Allows you to specify a namespace in the dataSourceRef field.
• Requires ReferenceGrant from the Gateway API to use this mechanism.

Data Source References

• dataSourceRef and dataSource fields are almost identical and cannot be changed after creation.
• The dataSource field ignores invalid values while the dataSourceRef field never does.
• The dataSource field only allows PVCs and VolumeSnapshots, while the dataSourceRef field may contain different types of objects.
• When the CrossNamespaceVolumeDataSource feature is enabled, the dataSourceRef field allows objects in any namespaces and does not sync with dataSource when a namespace is specified.

Using Volume Populators

• Volume populators are controllers that create non-empty volumes determined by a Custom Resource.
• A populated volume is created by referring to a Custom Resource using the dataSourceRef field.

Using Cross-Namespace Volume Data Sources

• Create a ReferenceGrant to allow the namespace owner to accept the reference.
• Define a populated volume by specifying a cross-namespace volume data source using the dataSourceRef field.
• Requires a valid ReferenceGrant in the source namespace.

Writing Portable Configuration

• Include PVC objects in your configuration bundle.
• Do not include PersistentVolume (PV) objects in the config.
• Allow the user to provide a storage class name when instantiating the template.
• If no storage class name is provided, leave the persistentVolumeClaim.storageClassName field as nil to automatically provision a PV with the default StorageClass.
• Monitor PVCs not getting bound after some time to identify potential issues with dynamic storage support or the lack of a storage system.

What are containers?

A Container is a unit of software that packages code and its dependencies, such that it can be executed reliably across different computing environments.

In other words, a container is an executable package that includes everything an application needs to run, such as code, necessary frameworks, libraries, packages and system tools. Essentially, containers help us create reproducible environments for an application. This is particularly useful for deploying applications in a distributed system with many nodes, where dependencies need to be consistent across different machines.

Why use containers?

Scalability and Parallelization:

If the code is an isolated/decoupled manner, containers can help us scale the application to manage heavy loads and perform operations in parallel across a large number of compute instances. This means we can more easily handle spikes in traffic without having to manually provision additional servers or resources.

Portability

Containers follow standard formats for packaging an app and its dependencies, which makes it possible to move an application from one environment to another, such as from a developer's laptop to a staging environment to a production environment in a cloud provider. This enables faster development and deployment cycles, and predictable production deployment behavior.

Isolation and Security:

Containers help run applications in their own self-contained environments. This level of isolation allows for multiple apps with their own dependencies to run on the same hardware without conflicts and limits the blast radius of security vulnerabilities. Beyond that, they can be made to adhere to the principle of least privilege, subject to access control and network restrictions. This means that even if a container is compromised, it cannot affect the host system or other containers running on the same machine.

Efficiency of Resources

Containers usually define the amount of resources they require in order to run. This (combined with isolation guarantees) makes it simple to run containers of varying sizes and to optimize costs by running multiple containers on the same machine.

How do containers work?

Containers are run on a host OS and share the Host OS's kernel. "But you told us there was isolation in containers, but now you say it shares the kernel with the same OS?!?!". You're right, it does share the kernel, but it uses a few kernel tricks to achieve the isolation.

First, it uses cgroups. Cgroups allows containers to allocate and limit the system resources provisioned for it. This includes resources such as CPU, memory, and disk I/O. It ensures that a container cannot use more resources than it is allowed and allows for resource allocation to be managed at the container level.

Second, it uses namespaces. Namespaces allow the container to obtain a separate view of the system's file system resources and network interfaces. Each container has its own file system, network interfaces, and hostname, which are separate from the host system. This ensures that different containers do not interfere with each other and operate as though they are on separate machines.

I'm sure there are more tricks but knowing this should be enough for now.

How are containers run?

Now for a container to run an application, it first needs to be packaged. For that, we introduce the concept of container images.

A Container Image is a single unit of packaged code, which includes both your applications and all of its dependencies, that can be stored, distributed and run via container runtimes.

What is a container runtime?

Once you have created a container image, you need a container runtime to execute the container image.

A Container Runtime creates a container instance from a container image, and provides the necessary runtime assistance for the container, such as provisioning a read-write layer for the container to use during runtime.

The two most popular container runtimes are Docker and containerd.

Docker is a high-level container runtime that provides a simple and easy-to-use interface for building, managing, and running containers.

Containerd is a low-level container runtime that provides a more modular and extensible architecture for managing containers.

Regardless of which runtime you choose, the runtime alone is not enough to manage a large number of containers. There needs to be a component that can help manage and coordinate these containers, so they can play well with each other. This is where Kubernetes comes in, and we'll take a look at that in the next chapter.

Conclusion

Containers have become an essential component of modern-day software development. If you were previously intimidated by containers, I hope that this course helps you build an appreciation for their value. In the long term, containers can enhance your productivity and the quality of the software you build.

With their ability to provide an isolated and portable application environment, containers offer numerous benefits including efficient resource utilization, consistent deployment, and ease of management. By embracing containers, you can stay ahead of the curve and take advantage of the latest technologies to improve your development workflow.

Excited to be on this journey with you!

What is a Kubernetes Service?

A Kubernetes service is a logical abstraction layer that groups a set of pods and exposes them to the network. Services provide a stable IP address and DNS name for the pods they target, regardless of their location or health status. This enables other applications inside or outside the cluster to access the pods via the service without knowing their specific IP addresses.

Services are defined using Kubernetes manifest files in YAML or JSON format. Each service has a unique name, a set of labels that match the labels on the targeted pods, and a service type that determines how the service is exposed to the network. Services can also have optional annotations, which provide additional metadata for the service.

Types of Kubernetes Services

Kubernetes supports four types of services, each with different networking characteristics and use cases:

1. ClusterIP: The default type of service, which exposes the service on a cluster-internal IP address. This type is suitable for accessing the service within the cluster but not from outside the cluster.
2. NodePort: Exposes the service on a static port on each worker node's IP address, which makes it accessible from outside the cluster. This type is useful for testing and development purposes but is not recommended for production environments due to security and scalability concerns.
3. LoadBalancer: Creates an external load balancer that routes traffic to the service. This type is suitable for exposing the service to external clients and provides load balancing, high availability, and scalability.
4. ExternalName: Maps the service to an external DNS name without creating a selector-based service. This type is useful for accessing external services from inside the cluster.

In this post, we will dive deep into the Kubernetes Pod specification, exploring what it is, how it works, and how to configure it.

What is a Kubernetes Pod?

A Pod is the smallest deployable unit in Kubernetes. It represents a single instance of a running process in a cluster. A Pod can contain one or more containers, which share the same network namespace and can communicate with each other using localhost.

The main purpose of a Pod is to provide a way to run and manage a single container or a set of tightly coupled containers that share resources, such as CPU, memory, and storage. Pods are also designed to be ephemeral, meaning they can be created, destroyed, and replaced at any time without affecting the rest of the cluster.

Kubernetes Pod Specification

The Kubernetes Pod specification defines how a Pod should be configured and deployed in a cluster. It is defined using a YAML file or a JSON object, which includes various fields and parameters that control the Pod's behavior.