EKS Image Hardening: What Is It?
EKS The practice of safeguarding AMIs based on specific benchmarks is known as image hardening. Standards from CIS levels 1 and 2 are used as a baseline. CIS (Center for Internet Security) level 1 and 2 standards refer to a set of security benchmarks and best practices developed by the Center for Internet Security. These standards provide guidelines and recommendations for securing various components of an IT infrastructure, including operating systems, applications, and cloud services.
Why is it required?
EKS Image Hardening is required for several key reasons. Firstly, it enables organizations to meet compliance requirements by securing AMIs based on CIS level 1 and 2 standards. This ensures adherence to regulatory standards and demonstrates a commitment to maintaining a robust security posture. Secondly, EKS Image Hardening enhances the overall security of an Amazon EKS environment by reducing the attack surface. Implementing recommended security measures mitigates the risk of vulnerabilities, data breaches, and unauthorized access.
Lastly, it safeguards against emerging threats by continuously updating and hardening AMIs based on the latest security benchmarks. This proactive approach ensures organizations stay ahead of evolving cyber threats, keeping their cloud infrastructure resilient and protected
Prerequisites
Before starting EKS Image Hardening, ensure you have Packer and Make installed on your local computer. For EKS image files, contact us here. Download the cis-audit.sh and hardened-ami-service-account.zip files from the provided link. Unzip the EKS image file to obtain the necessary EKS-hardened AMI folder. These prerequisites are essential for a smooth EKS Image Hardening process, ensuring the secure configuration of your AMIs.
Phase 1 – Creating an AMI: Enhancing Security for Your Amazon EKS Environment
To begin the process of strengthening the security of your Amazon EKS environment through image hardening, follow these steps:
- Configure AWS Access: Open the terminal on your local machine and ensure that AWS is properly configured with the access token or credentials of your AWS account. This will enable you to interact with the AWS services seamlessly.
Command Execution: Run the following command in your terminal to initiate the creation of an AMI:
$ make build-al2-1.22
This command triggers the AMI creation process, forming a foundation for further hardening. It prepares a partially hardened AMI, which will serve as the starting point for subsequent security enhancements.
- Review the AMI: Proceed to the AWS console and log in to your account. Take a moment to review the newly created AMI. This step allows you to assess the initial level of hardening achieved by the previous command. It serves as an opportunity to verify that the necessary progress has been made and sets the stage for the subsequent phases of the image-hardening process.
By carefully following these steps, you lay the groundwork for reinforcing the security of your Amazon EKS environment, ensuring that your AMIs are on the path to becoming fortified against potential threats.
Phase 2: Building an Instance from the AMI
To proceed with the creation of an instance from the chosen AMI, follow the below steps:
- Launch an Instance: Take the decisive step of launching an instance from the AMI that you have carefully selected for building purposes. This pivotal action sets the foundation for configuring and customizing the instance according to your specific requirements.
- Provide Necessary Information: As you initiate the instance creation journey, you will be prompted to furnish essential information that will shape the characteristics of the instance. This includes crucial details such as the instance type, network configuration, security groups, and other proper settings. By providing accurate and thoughtful information, you ensure that the instance is tailored to suit your unique needs.
- Connecting to the Initialized Instance: Once the instance creation process is complete, you will have the opportunity to connect to the instance. This pivotal moment allows you to establish a secure connection and gain access to the instance’s resources, where you can further customize and configure it as desired. Seamlessly connect to the initialized instance, and prepare to unveil its full potential in securing your Amazon EKS environment
Phase 3: Expanding /var/lib/docker: Unlocking Storage Potential
To expand the capacity of /var/lib/docker and accommodate your evolving needs, follow these steps:
- Select the EC2 Instance: Identify and choose the specific EC2 instance that was created from the AMI. This instance serves as the target for expanding /var/lib/docker.
- Verify /var/lib/docker Size: Run the following command within the instance’s terminal to confirm the current size of /var/lib/docker:
| $ df -h |
Expected Output:
| Filesystem Size Used Avail Use% Mounted on devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 1.9G 364K 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/nvme0n1p1 8.0G 1.7G 6.4G 21% / tmpfs 1.9G 4.0K 1.9G 1% /tmp /dev/nvme1n1p4 589M 19M 528M 4% /home /dev/nvme1n1p1 1.2G 442M 655M 41% /var tmpfs 1.9G 0 1.9G 0% /var/tmp /dev/nvme1n1p2 1.2G 33M 1.1G 4% /var/log /dev/nvme1n1p5 1.2G 260K 1.1G 1 % /var/lib/docker /dev/nvme1n1p3 1.2G 480K 1.1G 1% /var/log/audit tmpfs 388M 0 388M 0% /run/user/1000 |
This command provides information about storage utilization, helping you assess the need for expansion.
3. Note Available Memory: Take note of the available memory within /var/lib/docker, as this will be crucial for comparing the size after expansion.
4. Increase Partition Size: Execute the following command to increase the partition size:
| $ sudo growpart /dev/nvme1n1 5 |
Expected output:
| CHANGED: partition=5 start=8808448 old: size=2516992 end=11325440 new: size=33134559 end=41943007 |
| $ sudo resize2fs /dev/nvme1n1p5 |
| resize2fs 1.42.9 (28-Dec-2013) Filesystem at /dev/nvme1n1p5 is mounted on /var/lib/docker; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 2 The filesystem on /dev/nvme1n1p5 is now 4141819 blocks long. |
5. This command expands the partition size to enable more storage allocation for /var/lib/docker.
6. Verify Increased Size: Run the command below once again to confirm the increased size of /var/lib/docker:
| $ df -h |
Expected output:
| Filesystem Size Used Avail Use% Mounted on devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 1.9G 364K 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/nvme0n1p1 8.0G 1.7G 6.4G 21% / tmpfs 1.9G 4.0K 1.9G 1% /tmp /dev/nvme1n1p4 589M 19M 528M 4% /home /dev/nvme1n1p1 1.2G 442M 655M 41% /var tmpfs 1.9G 0 1.9G 0% /var/tmp /dev/nvme1n1p2 1.2G 33M 1.1G 4% /var/log /dev/nvme1n1p5 16G 260K 15G 1% /var/lib/docker /dev/nvme1n1p3 1.2G 516K 1.1G 1% /var/log/audit tmpfs 388M 0 388M 0% /run/user/1000 |
This step ensures that the expansion was successful and that /var/lib/docker now has the desired additional capacity.
7. Confirm the Expansion: Finally, double-check the increased size of /var/lib/docker to validate that the storage expansion process was completed successfully, meeting your requirements.
By following these simple steps, you effectively expand the storage capacity of /var/lib/docker within your Amazon EKS environment, empowering it to accommodate your growing needs with ease.
Phase 4: Copying cis-audit.sh File to Your EC2 Instance: Bridging Local and Cloud Environments
To seamlessly transfer the cis-audit.sh file from your local machine to the EC2 instance, need to follow simple steps:
- Connect to the EC2 Instance: Access the EC2 instance from the terminal on your local machine. Establishing this connection serves as the bridge between your local and cloud environments, enabling smooth file transfer.
- Open a Second Terminal Window: In a separate terminal window, navigate to the directory where you have stored the cis-audit.sh file. This prepares you for executing the necessary commands to initiate the file transfer.
- Transfer the File: Run the following scp command in the second terminal window, replacing the placeholders with the appropriate details:
$ scp -i <key pair name> cis-audit.sh ec2-user@<ip of the instance>:~/cis-audit.sh
Expected output$ scp -i <key pair name> cis-audit.sh ec2-user@<ip of the instance>:~/cis-audit.sh
This command securely copies the cis-audit.sh file to the specified EC2 instance, ensuring that it is placed in the home directory of the ec2-user.-This is a CIS Hardened Level – 1 OS.
-The AMI of this image is not officially supported by AWS
-Hardening is done on best effort basis and does not guarantee compliance.
-Please check with your Compliance Teams for specific Hardening Compliance Standard requirements.
cis-audit.sh 100% 121KB 667.8KB/s 00:00 - Verify the Copy: To confirm the successful transfer, execute the ls command in the terminal window connected to the EC2 instance:
Expected output:$ ls
This command allows you to see the contents of the current directory and verify if the cis-audit.sh file is present.awscli–bundle cis-audit.sh
By using these easy steps, you seamlessly bridge the gap between your local machine and the EC2 instance, copying the cis-audit.sh file and making it readily available within the instance. This sets the stage for further actions and ensures a smooth continuation of the Amazon EKS Image Hardening process.
Phase 5: Remediation: Solidifying Your AMI with Essential Fixes
To effectively remediate any vulnerabilities discovered during the Amazon EKS Image Hardening process, carefully follow these steps:
- Change to Root User: Once the cis-audit.sh file is successfully copied into the EC2 instance, switch to the root user by executing the following command:
This step grants you the necessary privileges to perform system-wide changes and address the identified security issues.$ sudo su
- Level 1 Check: Launch the level 1 check by running the cis-audit bash script:
Expected output:$ bash cis-audit.sh –level 1
This script initiates the evaluation of the AMI against the CIS Amazon Linux 2 Benchmark, highlighting any failing test cases.[00:00:16] (✓) 176 of 176 tests completed
.
.
.
Passed 151 of 175 tests in 16 seconds (13 Skipped, 0 Errors)
For the failed test cases, refer the cis-benchmark-doc pdf to remediate. - Identify Failing Test Cases: Analyze the results and identify the specific test cases that have failed. Take note of these areas where security improvements are required.
- Consult CIS Benchmark: Open the CIS Amazon Linux 2 Benchmark PDF and search for each failing test case. Follow the remediation steps provided within the benchmark document for each identified issue.
- Level 2 Checks: Execute the bash script command to commence the level 2 checks. Similar to the previous step, refer to the CIS Amazon Linux 2 Benchmark PDF and apply the recommended remediation steps for any failing test cases encountered.
- Reboot and Verify: Reboot the EC2 instance to ensure that the remediated test cases no longer persist. This step confirms the effectiveness of the applied fixes and validates the enhanced security measures.
Create a New AMI: Once a significant number of checks have passed and the necessary remediation steps have been implemented, create a new AMI from the instance. This new AMI will serve as the hardened image AMI that meets the required security standards.
By precisely following these steps, you actively address vulnerabilities within your AMI, applying essential fixes and strengthening its security posture. This ensures that your Amazon EKS environment is safeguarded against potential threats, providing a resilient foundation for your cloud-native applications.
Automation for Remediation: Streamlining the Hardening Process
To expedite the remediation process and eliminate the need for manual intervention, an efficient automation solution has been developed. This section outlines the steps for leveraging the bash script automation to swiftly address failing checks:
- Save the Automation Script: Ensure the eks-hardening-automationfile.sh is saved in the same directory as the cis-audit.sh file. This consolidation facilitates seamless execution and simplifies the automation workflow.
- Copy the Script to the EC2 Instance: Utilize the following command in a separate terminal window to securely copy the bash script to the EC2 instance:
This command ensures the script is transferred and accessible within the instance.$ scp -i <key pair name> eks-hardening-automationfile.sh ec2-user@<ip of the instance>:~/eks-hardening-automationfile.sh
- Switch to Root User: Log into the EC2 instance and elevate the user privileges to root by executing the following command:
This step grants the necessary permissions to execute system-wide changes.$ sudo su
- Execute the Automation Script: Run the bash script using the following command:
This automated script effectively remediates the identified failing checks, streamlining the remediation process.$ bash eks-hardening-automationfile.sh
- Verify Score Increase: Once the script has completed its execution, validate the improvement in passing check scores by executing the following commands:
For Level 1 Checks:
Expected output:$ bash cis-audit.sh -level 1
For Level 2 Checks:[00:00:16] (✓) 176 of 176 tests completed
.
.
.
Passed 164 of 175 tests in 16 seconds (13 Skipped, 0 Errors)
Expected Output:$ bash cis-audit.sh -level 2
These commands provide a comprehensive overview of the passing check scores, reflecting the effectiveness of the automation.[00:00:04] (✓) 33 of 33 tests completed
.
.
.
Passed 32 of 33 tests in 4 seconds (1 Skipped, 0 Errors) - Reboot and Confirm: Reboot the EC2 instance to ensure that the remediated test cases remain resolved. This step confirms the stability and integrity of the applied fixes.
- Create a new AMI: As a final step, generate a new AMI from the instance. This AMI captures the hardened image, incorporating the successful remediation efforts.
By following these streamlined steps, you leverage the power of automation to expedite the remediation process, ensuring a robust and secure environment for your Amazon EKS instances.
Packer Automation: Simplifying AMI Creation
To streamline the AMI creation process and ensure adequate hardening, the powerful tool called Packer is employed. Packer enables the creation of AMIs from a source template, allowing for the inclusion of multiple scripts to strengthen the image. Follow these steps to automate the process using Packer:
- Obtain the Necessary Files: Download the hardened-ami-service-account.zip and cis-audit.sh file. Note that the eks-hardening-automationfile.sh has already been added, and a new zip file has been created to consolidate the required resources.
- Unzip the EKS Image: Extract the contents of the EKS image file to acquire the essential EKS-hardened AMI folder. This folder serves as a crucial component for the subsequent steps.
- Configure AWS on Local Machine: Open the terminal on your local machine and configure AWS by providing the appropriate access token/credentials for your AWS account. This step ensures seamless communication with the AWS services.
- Execute the Command: Run the following command to create the AMI:
This command triggers the AMI creation process. The resulting AMI will be the desired hardened image, incorporating the hardening scripts passed within the packer source template.$ make build-al2-1.22
By following these streamlined steps, you leverage the capabilities of Packer to automate the creation of AMIs. This ensures a standardized and efficient approach to hardening your Amazon EKS images, enabling you to establish a secure foundation for your cloud-native applications.
Next Step for Automation: Streamlining /var/lib/docker Expansion
Having acknowledged the manual nature of phase 3, which involves expanding the /var/lib/docker library, the logical progression is to automate this process entirely. By doing so, we can enhance the overall efficiency and reliability of the expansion.
The next crucial step is to automate phase 3, eliminating the need for manual intervention when expanding the /var/lib/docker library.
Creating an EKS AMI with version 1.23 involves a few straightforward steps:
- Navigate to the eks-custom-amis folder and open the Makefile.
- In the Amazon Linux 2 section, insert the build command for EKS Version 1.23, following the provided snapshot.
3. Save the file and execute the command below in your terminal:
$ make build-al2-1.23 |
By following these steps, you can effortlessly create an EKS AMI with the desired version 1.23.
Passing Checks Analysis:
Here’s an overview of the analysis conducted on the passing checks for CIS benchmark level 1 and level 2 standards:
- Initially, there were 175 checks for CIS benchmark level 1 and 33 checks for level 2.
- At the beginning, the number of passing checks stood at 151 for level 1 and 30 for level 2.
- This remarkable improvement signifies that the AMI taken is now more secure than its initial state, offering enhanced protection.
By continuously striving to increase the number of passing checks, the overall security of the AMI has been significantly enhanced.
In summary, for the Kubernetes cluster to be more secure, Amazon EKS images must be hardened. Leveraging CIS benchmark standards alongside automation tools such as bash scripts and packers streamlines the image hardening process, saving valuable time and effort. By adopting a proactive approach through regular monitoring and updates, organizations can effectively shield their applications and data from potential threats, fostering a secure environment for their EKS deployments.
About Author
Bhupali is a seasoned technology leader with a passion for innovation and a deep understanding of the cloud computing industry. With extensive experience in cloud architecture and a proven track record of delivering successful AWS implementations, Bhupali is a trusted advisor to Comprinno’s clients. She is a thought leader in the industry and loves to channel her passion for technology through her insightful blogs.
Copyright 2022