AWS EKS Kubernetes Upgrade: A Comprehensive Guide

Keeping your Kubernetes clusters up-to-date is vital for maintaining security, performance, and compatibility. An AWS EKS Kubernetes upgrade can seem intimidating, but staying on top of regular upgrades is essential to avoid vulnerabilities and ensure you're using the latest features. 

In this article, we'll cover why upgrading your EKS clusters matters, how to navigate the EKS upgrade process, and best practices to ensure a smooth and efficient upgrade experience. 

Understanding Kubernetes Version Lifecycle on Amazon EKS

Kubernetes evolves rapidly, with new minor versions released approximately every four months. Staying current ensures you benefit from the latest features, security patches, and performance improvements. Amazon EKS follows the Kubernetes version lifecycle, adhering to standard and extended support policies.

Standard and Extended Support

• Standard Support: Each Kubernetes minor version on Amazon EKS is supported for 14 months. During this period, Amazon provides critical security patches and bug fixes.
• Extended Support: After standard support ends, versions enter a 12-month extended support phase. This option comes with additional costs but allows you to stay on a specific version without immediate upgrades.

Amazon EKS supports three active Kubernetes versions at any time. Once a version exits extended support, the system automatically upgrades clusters running that version to the oldest supported version. This ensures that clusters remain secure and supported. 

A. Preparing for the Upgrade

Ensure you prepare thoroughly before upgrading. This minimizes risks and guarantees a smooth process.

1. Create an Upgrade Checklist

Begin by creating an upgrade checklist based on Amazon EKS documentation. This checklist should include all the necessary steps below:

Checklist:

• Review Kubernetes and EKS release notes.

• Back up the cluster.

• Identify and remediate deprecated API usage.

• Update kubectl and Terraform tools.

• Upgrade the control plane using Terraform.

• Upgrade node groups using Terraform.

• Validate the upgrade.

2. Review Kubernetes Change Logs and Deprecation Policies

Kubernetes minor versions often introduce new features and deprecate old ones. Reviewing the Kubernetes releases and understanding the deprecation policies is important to identify any deprecated APIs or features that might impact your workloads. 

3. Assess Add-On Compatibility

Ensure that all cluster add-ons, such as CoreDNS, kube-proxy, and the Amazon VPC CNI plugin, are compatible with the new Kubernetes version. 

Steps:

• List installed add-ons:
  aws eks list-addons --cluster-name <cluster-name>
• Retrieve the addon versions that are compatible with the Kubernetes version of your cluster:

aws eks describe-addon-versions --addon-name <addon-name> --kubernetes-version

<kubernetes-version>

Note: Reviewing the output helps identify any discrepancies or unsupported versions.

• Update add-ons using the AWS CLI:
  aws eks update-addon --cluster-name <name> --addon-name <addon-name> --addon-version <addon-version>

Add-on updates may introduce changes or deprecations in their APIs. Always review the release

notes or changelogs for the specific add-on version being updated.

4. Enable Control Plane Logging

Enabling control plane logging helps you monitor the upgrade process and troubleshoot any issues that arise. This step is important for maintaining visibility into cluster operations during the upgrade.

Steps:

• Navigate to the EKS cluster in the AWS Console.
• Select the Logging tab.
• Enable the desired log types (API, audit, authenticator, controllerManager, scheduler).

5. Backup the Cluster

Before an AWS EKS Kubernetes upgrade, back up your cluster to prevent data loss. Reliable backup solutions enable restoration if issues arise during the EKS upgrade. Since Kubernetes upgrades are irreversible, backups are essential.

Steps:

• Choose a Backup Solution: Select a backup tool compatible with AWS EKS that meets your infrastructure needs, focusing on ease of use, scalability, and support.  Tools like Velero can help you create comprehensive backups. Remember, Kubernetes upgrades are irreversible, so having a backup ensures you can restore your cluster if necessary. 
• Perform the Backup: Follow the tool’s guidelines to create a comprehensive backup, ensuring all critical namespaces and resources are included for smooth restoration.

Install Velero:

velero install --provider aws --bucket <bucket-name> --secret-file ./credentials-velero

Create a backup:

velero backup create <backup-name> --include-namespaces <namespace>

6. Define Infrastructure as Code with Terraform

Before proceeding with the upgrade, ensure that your EKS cluster is managed via Terraform. This allows for a more controlled and repeatable upgrade process.

Steps:

1. Initialize Terraform Configuration: If not already managed by Terraform, import your existing EKS cluster into Terraform.
   terraform import aws_eks_cluster.my_cluster <cluster-name>
2. ‍Define Cluster Configuration: Ensure your main.tf includes all necessary resources for the EKS cluster, control plane, and node groups.
   

resource "aws_eks_cluster" "my_cluster" {
  name      = "my-cluster"
  version   = "1.30"
  role_arn  = "arn:aws:iam::123456789012:role/EKS-ClusterRole"
  
  # ... other configurations
}

resource "aws_eks_node_group" "my_node_group" {
  cluster_name    = aws_eks_cluster.my_cluster.name
  node_group_name = "standard-workers"
  
  # ... other configurations
}

Source - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster

B. Process Overview

C. Setting Up the Tools

Properly setting up your command-line tools is fundamental to managing Kubernetes upgrades effectively.

Install and Configure kubectl

kubectl is the command-line tool for interacting with Kubernetes clusters. Ensure you have the correct version of kubectl installed, matching your cluster's Kubernetes version.

Installation Steps for macOS:

Download the kubectl binary:
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.31.2/2024-11-15/bin/darwin/amd64/kubectl

1. Verify the checksum: openssl sha256 kubectl
2. Make the binary executable:
   chmod +x ./kubectl
3. Move the binary to your PATH:
   mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$HOME/bin:$PATH
4. Add to shell profile:
   echo 'export PATH=$HOME/bin:$PATH' >> ~/.bash_profile

Install Terraform

Terraform is a powerful Infrastructure as Code (IaC) tool that enables you to define and manage your EKS clusters declaratively.

Installation Steps for macOS:

• Install Terraform:
  Install terraform for mac, windows, linux operating system.
• Unzip the Package:
  unzip terraform_1.10.0_darwin_amd64.zip
• Move the Binary to Your PATH:
  sudo mv terraform /usr/local/bin/
• Verify the Installation:
  terraform version
• Configure AWS Provider for Terraform

Ensure Terraform is configured to interact with your AWS account.

• Configure AWS Provider for Terraform

Ensure Terraform is configured to interact with your AWS account.

Create a provider.tf file:

provider "aws" { 
region = "us-west-2"
}

Source: https://registry.terraform.io/providers/hashicorp/aws/latest/docs

Best Practices for Cluster Upgrades

Adhering to best practices ensures that your upgrade process is smooth and secure.

1. Staging the Upgrade: Dev → Staging → Production

Upgrade your environments in the following order:

• Development: Start by performing an eks upgrade on your development cluster.  This environment helps you test the new Kubernetes version and identify potential issues early.
• Staging: Next, upgrade the staging environment. This stage acts as a final testing ground, allowing a 1-2 week soak time to ensure stability.
• Production: Finally, upgrade your production cluster. Aligning production with staging minimizes risks. This approach ensures that the upgrade undergoes thorough testing before impacting live workloads.

Steps:

• Upgrade the development cluster first:
  eksctl upgrade cluster --name dev-cluster --version <new-version> --approve
• Monitor and test before proceeding to the stage.

2. In-Place Upgrades vs. Blue/Green Deployment

In-place upgrades maintain existing cluster resources and configurations, allowing you to upgrade one minor version at a time. This method is straightforward but requires careful planning to avoid disruptions.

Blue/Green Deployment involves creating a parallel cluster with the new Kubernetes version. Once the new cluster is validated, you switch traffic from the old cluster to the new one. This strategy minimizes downtime but can be more complex to implement. 

D. Executing the Upgrade

With preparation complete, you can proceed to upgrade your EKS cluster.

‍

Updating the Control Plane

• Update the Kubernetes Version in Terraform Configuration:‍

resource "aws_eks_cluster" "my_cluster" {  
# ... existing configurations  
version = "1.31"  
# ... other configurations
}‍

• Apply the Changes:
  terraform plan

terraform apply

• Declarative Upgrades: By updating the version attribute in your Terraform configuration and applying the changes, Terraform handles the upgrade declaratively

Updating Add-Ons and Components

• Update Add-On Versions in Terraform Configuration:

resource "aws_eks_addon" "coredns" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  addon_name    = "coredns"
  addon_version = "v1.8.4-eksbuild.1" # Example version

  # ... other configurations
}

resource "aws_eks_addon" "vpc_cni" {
  cluster_name  = aws_eks_cluster.my_cluster.name
  addon_name    = "vpc-cni"
  addon_version = "v1.12.1-eksbuild.1" # Example version

  # ... other configurations
}

‍‍Source: https://github.com/hashicorp/terraform-provider-aws

• Apply the Changes:
  terraform plan

terraform apply

• Centralized Add-On Management: By defining add-ons in Terraform, you maintain consistency and control over their versions and configurations..

Updating Managed Node Groups

Next, update your managed node groups to match the control plane version.

• Update Node Group Configuration:
  resource "aws_eks_node_group" "my_node_group"

resource "aws_eks_node_group" "my_node_group" 
{
  // ... existing configurations
  "version": "1.31"
  // ... other configurations
}

Source: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group

• Apply the Changes:
  terraform plan

terraform apply

• Consistent Node Group Upgrades: Updating the version attribute ensures node groups align with the control plane version.

Using Tools like eksctl

Leverage tools like eksctl to automate and streamline the upgrade process. eksctl helps manage cluster upgrades efficiently, ensuring consistency and reducing the potential for human error.

E. Post-Upgrade Steps

Once the upgrade is complete, perform the following actions to ensure your cluster is functioning correctly.

1. Monitor Cluster Health

Use monitoring tools to monitor cluster health and performance. Ensure all components are running smoothly and workloads are functioning as expected.

Steps:

1. Check node status:
   kubectl get nodes
2. Monitor logs for any anomalies.

2. Verify Workload Functionality

Check that all applications and services are operating correctly. Validate that there are no disruptions or performance issues post-upgrade.

Steps:

• Test application endpoints.
• Verify deployments: kubectl get deployments --all-namespaces

3. Handle Deprecated APIs

Address any deprecated API usage identified during the upgrade. Update your manifests and workloads to use supported APIs. Refer to the Identify and Remediate Removed API Usage guide for more information.

Steps:

• Convert manifests:
  kubectl-convert -f <file> --output-version <group>/<version>
• Update deployments to use supported APIs.

4. Validate Terraform State

Ensure that Terraform's state file accurately reflects the current state of your infrastructure post-upgrade.

Steps:

• Run Terraform Plan:
  terraform plan

Ensure no unexpected changes are detected.

• Review and Commit Changes: If there are necessary updates, apply them and commit your Terraform configuration to your version control system.
  terraform apply

git add .

git commit -m "Update EKS cluster to Kubernetes version 1.31"

git push

• State Consistency: Validating the Terraform state ensures that your IaC remains in sync with the actual infrastructure, preventing configuration drift.

F. Automating and Simplifying Upgrades

Automation reduces the complexity of managing Kubernetes upgrades.

Utilize Managed Node Groups and Karpenter

Node Groups automate node provisioning, updates, and lifecycle management, simplifying the upgrade process. Karpenter adjusts node capacity dynamically based on workload demands, ensuring the cluster stays efficient and up-to-date.

Steps:

• Use eksctl to manage node groups:
  eksctl create nodegroup --cluster=<clusterName> [--name=<nodegroupName>]
• Configure Karpenter:
  yaml

apiVersion: karpenter.sh/v1alpha5
kind: NodePool

metadata:
  name: default

spec:
  ttlSecondsUntilExpired: 259200
  # ... 

Automate with Terraform

Use Terraform to automate the creation, scaling, and management of EKS clusters. Terraform provides a declarative approach to manage your infrastructure, ensuring consistency and version control.

Steps with Terraform:

• Define Infrastructure in Terraform: Ensure all EKS resources are defined in your Terraform configuration files.

resource "aws_eks_cluster" "my_cluster" {
  name      = "my-cluster"
  version   = "1.30"
  role_arn  = "arn:aws:iam::123456789012:role/EKS-ClusterRole"

  # ... other configurations
}

resource "aws_eks_node_group" "standard_workers" {
  cluster_name    = aws_eks_cluster.my_cluster.name
  node_group_name = "standard-workers"
  version         = "1.30"

  # ... other configurations
}

Source: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster

• Automate Upgrades: Modify the Kubernetes version and apply changes through Terraform to automate the upgrade process.
  terraform plan

terraform apply

• Integrate with CI/CD Pipelines: Incorporate Terraform commands into your CI/CD pipelines to automate upgrades as part of your deployment workflows.
  
  • Example CI/CD Integration (GitHub Actions):

name: Upgrade EKS Cluster

on:
  push:
    branches:
      - main

jobs:
  upgrade:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          terraform_version: 1.5.0

      - name: Initialize Terraform
        run: terraform init

      - name: Plan Terraform
        run: terraform plan

      - name: Apply Terraform
        run: terraform apply -auto-approve
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Use AWS Resilience Hub

AWS Resilience Hub supports EKS by helping you define and validate application resilience. It ensures your applications can withstand infrastructure, software, or operational issues, minimizing downtime during upgrades.

Action Steps:

• Define resilience policies.
• Validate application resilience.
• Monitor resilience scores during upgrades.

G. Handling Extended Support

Extended support provides additional time to plan and execute upgrades without incurring automatic updates.

Enabling Extended Support

By default, EKS clusters have an upgrade policy set to enable extended support. To ensure your cluster remains on its current Kubernetes version beyond the standard support period, verify that extended support is enabled. Follow the Enable EKS Extended Support guide to configure your cluster accordingly.

Steps:

• Enable extended support via AWS CLI:
  aws eks update-cluster-config --name <cluster-name> --upgrade-policy supportType=EXTENDED
• Confirm the upgrade policy:
  aws eks describe-cluster --name <cluster-name> --query 'cluster.upgradePolicy'
• Enabling Extended Support Using Terraform

Update Cluster Configuration in Terraform:

resource "aws_eks_cluster" "my_cluster" {
  name    = "my-cluster"
  version = "1.30"

  # ... other configurations

  upgrade_policy {
    support_type = "EXTENDED"
  }
}

• Apply the Changes:
  terraform plan

terraform apply

• Confirm the Upgrade Policy: Verify that the upgrade policy is correctly set.
  aws eks describe-cluster --name <cluster-name> --query 'cluster.upgradePolicy'

Source: https://registry.terraform.io/providers/hashicorp/aws/latest/doc/resources/eks_cluster

Cost Considerations

Clusters in extended support incur higher costs. Evaluate whether extended support aligns with your budget and upgrade planning. If you decide against extended support, upgrade your cluster before the end of the standard support period. This will help you avoid automatic upgrades. Cost optimization is key to managing your Kubernetes environment and can help your organization save. Considering cost optimization strategies can be useful when planning upgrades.

Planning for Automatic Upgrades

While extended support prevents automatic upgrades, it's essential to proactively plan and execute upgrades to maintain cluster security and performance. Relying solely on automatic processes can lead to unexpected issues and potential downtime.

Steps:

• Schedule regular upgrade reviews.
• Allocate resources for timely upgrades.
• Use monitoring tools to track cluster health.

Ready to Upgrade Your Kubernetes Clusters?

Upgrading AWS EKS Kubernetes Clusters is vital for security, performance, and compatibility. To minimize risks use automation tools and AWS resources. 

At Kapstan, we specialize in Kubernetes cluster management and upgrades. Our expert team ensures your AWS EKS clusters are secure, efficient, and up-to-date. Book a demo today to streamline your Kubernetes upgrade process and optimize your cloud infrastructure.