My Journey to CKA

A long long time ago (in a galaxy far away)…

At the beginning of 2020 I’ve got interested in Kubernetes (K8s from now on) and I’ve decided to buy The Kubernetes Book from Nigel Poulton (yearly updates for free), thus starting digging into this very interesting piece of technology.

Unfortunately, as we all know, the beginning of 2020 brought us a very big problem which had a lot of impact on my daily job as Network Engineer (moving from few hundreds to thousands of users working at home had a great impact on the infrastructure and required a lot of work) and so I’ve stopped studying K8S at the beginning of my journey.

Important: if you don’t want to read the whole story, you can directly jump to my list of personal hints at the end of the article ;)

Time to start again… with a clear objective

In summer 2022 I’ve decided that it was time to try again to learn a bit about K8s, because it’s something becoming more and more important (not only K8s but also OpenShift etc) in our IT world.

The best way to study something, for me, is having a clear objective that forces you to constantly work on a specific topic. It is something very difficult nowadays since I already work 8+ hours a day on a personal computer and mental resources after work are almost exhausted. By the way, I wanted to give it a try and so I’ve decided to aim at the Certified Kubernetes Administrator certification by The Linux Foundation.

Since I had a lot of doubts during my journey, I’d like to share my experience toward the CKA certification, hoping it can help someone else to achieve the same result on first attempt.

Let’s go hunting on Udemy

After a bit of hunting for discounts, I’ve decided to buy for less than 10€ the Certified Kubernetes Administrator (CKA) with Practice Tests from Mumshad Mannambeth and KodeKloud on Udemy.

The course is very well done, with practice labs on some real Kubernetes clusters on KodeKloud (they give you free access to those labs on their learning platform) and I went through the course from the beginning to the end doing all the labs while progressing, which is the best way to fix in your mind what you are learning.

My first K8s installation with Multipass+MicroK8s

KodeKloud labs are fine to experiment but if you really want to lear K8s and become confident with how it works, how to do troubleshooting and how to configure objects, I strongly recommend you to build your own K8s cluster on your laptop or some home server.

When I’ve started looking at K8s in 2020, before my big pause, I’ve created my single-node kubernetes environment on a Multipass Ubuntu VM on my laptop, using microk8s. When I started studying again K8s this summer, I decided to go with Kubernetes and not one of the light & quick versions, by installing it via kubeadm since this is what is used in the exam. So, I’ve built my first cluster with a Master/Controlplane node and a Worker node, again using Multipass as virtualization tool on my MacBook Pro. This allowed me to make some experience, but I wanted to move toward an high-availability cluster and so I restarted from scratch after some weeks of experience.

Moving to Vagrant and increasing the number of nodes

I got rid of my first K8s installation and I replaced Multipass with Vagrant tool to automatically build my Ubuntu VMs relying on the VMWare Fusion virtualization environment I was already using, in order to be able to use snapshots, something I was missing on Multipass and which is very important when you want to experiment and break things :)

Important: the implementation of an highly available cluster, with redundant load balancers etc is definitely not required to pass CKA, you can learn and experiment on KodeKloud labs and/or on a single-node test cluster, so don’t panic if you don’t have enough resources on your PC to build such a cluster or you simply don’t have time to do that.

This time I decided to double the Controlplane nodes, to be a bit more redundant (later on I’ve realized that 2 CPs with stacked ETCD where not better than a single node because with a single CP dead, the other one does not have the consensus to operate.. so you must go with 1CP, 3CPS, 5CPs in a stacked ETCD configuration). This required me to add a load balancer to balance requests to the apiserver (everything is explained on the official Creating Highly Available Clusters with kubeadm guide on kubernetes.io and on some other resources linked on that page) and forced me to customize kubeadm installation in order to specify the loadbalancer virtual server as ApiServer to be advertised to the nodes, along with some other parameters. This made me learn how to customize kubeadm init and join procedures via YAML files, since those options can not be specified on the command line. This is not required for CKA, but again more experience can’t never be a waste of time. Since I’m a Network Engineer, a single-node load balancer was not enough for me so I’ve build a cluster of LBs with Keepalived to expose via VRRP the IP address of the virtual server and with HAProxy to implement the LB itself. You can find instructions about how to implement a Keepalived+HAProxy installation in very few time on Options for Software Load Balancing, a document referenced within the high availability guide I’ve linked above. I wanted to have the K8s nodes communicating between themselves on an internal network not exposed to my Macbook Pro and so I had to add a second interface to the VMs. This again forced me to learn some details about kubeadm customization because I had to specify which IP address use as node address, since the default interface was not the one to use.

Vagrant K8s Cluster — Logical View

When things became serious

Now it should be clear that I’m never satisfied with simple things and I wanted to upgrade my testing environment. There were also some reasons why the lab on my MacBook Pro needed to be changed:

• Few disk space: I was running out of space and my VMs were already using about 50Gbs of disk space. A possible solution could have been moving everything on an external SSD, but I didn’t have one and it wouldn’t have solved the following issues
• Few ram: I wasn’t really running out of memory with 6 VMs with 2GBs of ram allocated, but there was not very much available ram for the OS and for other apps like Visual Studio Code
• High CPU/Power usage: my quad-core 2013 i7 cpu was constantly running at high load, with the VMs and the other apps running in the background, fans were always running at full speed, cpu temperature was always around 90°C and power consumption was between 60 and 90W constantly.

As I’ve already explained, CKA doesn’t require such a multi-node lab to be done, and I could have simply built a 1 or 2 nodes’ K8s cluster… But you know how it works… this wasn’t an option for me :)

So, I needed to build a dedicated home lab, on another piece of hardware other than my MacBook Pro. This desire was already building up in my mind since Apple released Apple Silicon platform, an amazing piece of technology that has a drawback for me: the impossibility to virtualize x86 hardware. So, building up a dedicated virtualization environment could solve both the K8s cluster problem and future requirements for x86 VMs.

Let’s go hunting on Ebay

When you think about a small and low-power machine that can be used to run VMs, one of the first thing you should think of is Intel NUC platform… or at least this is what happened to me several times in the past 2–3 years. I’ve never gone the NUC way because of its price, now increased so much due to the problems of the last two years. My problem is that I didn’t want to spend money on a small i3 intel NUC, with few ram and disk space, so the price tag was always around 1.000 € or more. I’ve started hunting for something used or refurbished on Ebay and other similar platforms and after a rare planetary alignment, I’ve found what I wanted, for about 600€:

• Intel NUC Performance NUC10i7FNH
• 32GB (2x16GB) DDR4 2666Mhz of RAM by Crucial
• 2TB NVMe Sabrent Rocket 4 PCI Gen4 SSD for OS and VMs
• 2TB Crucial MX500 SATA SSD for automatic backups via Proxmox Backup Server

This marvellous piece of hardware was all condensed in less than 12 x 12 x 5 cm, with a power consumption of about 15W when running non-CPU intensive VMs.

So, it was time to learn again something new: Proxmox VE

I went through the user manual, the forums, some guides and in few time my new virtualization solution was up and running, ready to host a K8s cluster! This caused a small break in my CKA learning process but it was worth the price, definitely.

A new K8s cluster on Proxmox VE

After having a running Proxmox VE I’ve quickly learnt how to build an Ubuntu 22.10 cloudimage and how to deploy it with Terraform+Proxmox provider and I was ready to start again with a new K8s cluster, built with kubeadm. This time I choose to go with 3 ControlPlane + 2 Worker nodes, but instead of placing 2 LoadBalancer VMs in front of the CP nodes, thus having to manage & update two more machines, I decided to integrate LBs within the CP VMs:

Proxmox K8s Cluster — Logical View

Since my PVE (Proxmox VE) box was not meant to run only the K8s cluster but also other home services (view my posts here on Medium about my Pihole+Unbound recursive DNS resolver with AD-blocking features), I didn’t want to expose on my “production” home LAN all the K8s VMs and so I’ve placed them on an isolated bridge (vmbr1) along with a dual-homed gateway VM (GW) connected to the K8s bridge and to the main PVE bridge vmbr0 which faces on the home LAN. The GW machine is, as the name suggests, a gateway toward my LAN and Internet for K8s VMs and an SSH jump server from which I can reach the K8s machines:

A quick explanation of the configuration above:

• gw is the gateway VM with IP 10.0.0.100 on my home LAN, which can be directly reached from my MacBook Pro
• c01 is controlplane01, which has an IP 192.168.100.11 on the K8s network. This IP can not be directly reached from my MBP but the configuration of c01 tells ssh to first jump (i.e. establish an SSH session) to gw and then reach the 192.168.100.11 IP address (which must be reachable from the GW VM, not from the MBP, which is exactly what I wanted).
• LocalForward on c01 allows me to reach the stats page of HAProxy running con controlplane01 by pointing http://localhost:8081/stats
• DynamicForward on gw allows me to reach every service exposed on 192.168.100.0/24 network by simply configuring proxy settings on a browser like Firefox pointing at localhost:3128 as Socks proxy.

This is the “physical” view of the new K8s cluster setup:

Proxmox K8s Cluster — “Physical” View

Everything was up and running, ready to deploy the K8s cluster and so I went through the setup of an highly available K8s cluster with Kubeadm once again. I used Tigera Calico as CNI and in order to have an on-premise implementation of LoadBalancer services I’ve installed MetalLB in Layer-3 mode, with BGP peerings between the K8s controlplane/worker nodes and GW, which was running Bird daemon. This allowed me to announce services via BGP to the GW VM that could be used to test reachability of services exposed as LoadBalancers within the K8s cluster (such as the Nginx ingress).

More details about my Intel NUC/Proxmox VE lab here.

Time to study hard and make practice, a lot of practice

Now that the cluster was up and running, there were no more excuses and it was time to study hard and make practice, after practice after practice. Did I say that you should practice a lot?

I think that you can pass the exam with only studying on Udemy and going through the tasks on KodeKloud but you will surely be much more prepared if you do a lot of practice, imagine new scenarios and try to implement them on your own cluster.

My personal suggestions about how to pass CKA on first attempt

Finally, here we are with some suggestions about how to pass the CKA on first attempt, focused on my own experience:

• Follow the Udemy course from the beginning to the end, studying a bit every day and completing all the labs on the KodeKloud platform, asking questions everytime something is not clear.
• Perform some K8s installations with kubeadm.
• Go through the topics of the course and the CKA exam (which obviously are almost the same) once again, experimenting on your own cluster (I’ve also documented my tests in order to fix concepts into my mind).
• Become very confident with all the section of kubernetes.io, in order to be very fast in finding YAML skeletons of the different kinds of resources, like PersistentVolumes, NetworkPolicies and the other stuff that can not be created with kubectl imperative commands.
• Try to master kubectl imperative commands that allow you to create a YAML skeleton for resources like Pods, Deployments, Roles etc via the — dry-run=client -o yaml flags.
• Learn how to quickly find info about specific sections of YAML files via the kubectl explain command, like kubectl explain pod.spec.containers.volumeMounts to know the parameters you can specify for volume mounts within a container.
• When you are quite confident to be ready to face the exam, try the free exam environment on killer.sh (free if you buy the CKA exam): complete as much tasks as possible in the first two hours to see how you will perform during a real exam and then complete the remaining tasks, see what is wrong, try to fix remaining issues and then go through the solutions to understand your mistakes.
• Keep an eye on The Linux Foundation Exams page and wait for discounts: I’ve seen some 35% discount codes from KodeKloud this summer and I’ve finally managed to buy the exam with a 50% discount on Cyber Monday. I’ve bought also for a 10$ extra the Kubernetes Fundamentals (LFS258) course which is usually priced 299$: it has been useful but it is definitely not worth the full price, compared to the CKA Udemy course, even at full price (less than 90€).
• Carefully read the exam info/requirements and verify that your computer satisfies the requirements for the PSI browser that must be installed and used to take the proctored CKA exam.
• Setup terminal/shell at the beginning of the exam in order to be as quick as possible and work in the best way: configure bash completion for kubectl (source <(kubectl completion bash), which should be placed in .bashrc if you will use multiple terminal sessions), configure a shortcut like $do to create YAML skeletons (export do=“ — dry-run=client -o yaml” in .bashrc) which can be appended at the end of the kubectl imperative commands, check the preferences of the terminal app to be sure that “unsafe past warning” is disabled.
• During the exam, jump over low-score tasks (1–3%) if you can’t solve them quickly and try to solve the high-score ones. At the end, go over the tasks once again.
• Always test with curl or wget from the shell or within temporary pods if what you implement is correctly working (check services’ reachability, network policy implementation etc) and look at the details of the objects with kubectl describe. For example you can launch a test pod with an alpine image, exec into the pod, install curl and then check if it is blocked/allowed to reach a service before and after implementing a network policy required by a specific task.
• Always copy the names of objects or target files that are specified in the exam task to avoid typos.

Conclusion

On Dec. 3 I tried the killer.sh exam and completed 19 tasks (with some errors) within 2 hours, and then went on with the remaining tasks, including bonus/extra tasks for other two hours. The score was good and given the fact that killer.sh exam should be more difficult than the real one, I wanted to try CKA as soon as I could. So, I immediately scheduled it for next day in the evening. On Dec. 4 at 7pm I started the exam and completed the tasks in about 100 minutes, with 20 minutes remaining to go through the tasks again and fix some errors.

The time ended and I was happy with my performance, but then I had to wait for 23 hours and a half in order to receive the exam results… quite a long wait for me… and BOOM: CKA passed with 95%!

Verify my achievement from The Linux Foundation

I worked hard to crack it in the first attempt and I’m very happy I’ve been able to complete this personal challenge :)

I hope that this article can be useful for someone else aiming at passing CKA, feel free to ask questions, I’ll be happy to help if the answer would not violate exam’s NDA ;)