Sec 1: CKA :

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/docs/04-Logging-and-Monitoring/02-Monitor-Cluster-Components.md

Kubernetes Release Page for Downloads

https://kubernetes.io/releases/

CKA Exam :
https://www.youtube.com/watch?v=9UqkWcdy140

Exam Point of view : Imperative commands

https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/learn/lecture/15018998#overview

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands

https://kubernetes.io/docs/reference/kubectl/conventions/

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Sec: 2 : Core Concepts :

11. Cluster Architecture

We are taking Two ships as an analogy for the explaination.

12. Docker-vs-ContainerD

CLI tools

1. ctr
2. crictl
3. nerdctl

What are the CLI tools and which one should you be using.

Back then kubernetes only worked with docker and didn't support any other container runtime.
now over a period of time Kubernetes as a container orchestration tool earned popularity and the developers wanted to work with other containers such as Rockt .

Therefore Kubernetes introduced an interface called the container run time interface or CRI
CRI allowed any vendor to work as a container run time for as long as they adhere to OCI standards.

OCI Stands for Open Container Initiative. It consists of

1. Imagespec
2. RuntimeSpec

-- ImageSpec means the specification on how an image should be build.
-- RuntimeSpec meant how a container run time should be developed.

However Docker wasn't built to support the CRI standards. Docker was built way before CRI introduced. Docker was still the dominant container tool used by most . Kubernetes had to continue to support Docker as well.

Therefore Kubernetes introduced what is know as Dockershim

which was a hacky but a temporary way to continue to support docker outside of CRI (Container Runtime Interface),while most of the containers continued to work with CRI docker continued to workout it.

You see Docker isn't just a container run time alone. Docker consists of multiple tools such as

Also the container run time called Run-c and the deamon that managed Run-c that was called as containerD . Container D is CRI compatible and can work directly with Kubernetes .

Therefore ContainerD can be used as its own and can work independently without docker.

With Kubernetes 1.24 -- they decided to remove dockershim completely . But support for docker was removed.

All the images that were build before the docker was removed they continued to work because docker followed the imagespec that OCI followed they contained to work with container-D . But Docker itself was removed as a supported runtime from Kubernetes

Now lets look into container-D more specifically .

ContainerD though is a part of docker it is a seperate Project own its own and is a member of CNCF .

How to run containerD . Once you install containerD it comes with a command line tool called CTL - ctr and this tool is solely made for debugging and not very user-friendly. It only supports limited feature.

Apart from the limited feature provided by containerD , if you want to interact with it in any other way you need to make API call. The most user-friendly way to operate.

ctr command to pull images.

not meant for Prod usecases.

A better alternative recommended is the nerdctl tool

nerdctl tool works very similar to docker CLI . Instead of docker you simply need to replace it with "nerdctl"

There is another utility crictl

This tool is developed and maintained by the kubernetes community . And this tool works across all the container run time . Mainly used for debugging purposes not as effective as Docker for building containers. It kind of works along with Kublet

13. ETCD For Beginners

ETCD : It is a distributed, reliable key-value store that is simple, secure and fast.

Keyvalue store information as a document so each individual gets a document. It is easy to work with ETCD,

download the binary
extract it
And run

when you start ETCD there is service that is started and listens to the port 2379. you can then attach any client to ETCD to store and retrieve information.

the default client that comes with ETCD is etcd control > etcdctl client . it is command line client for etcd . used to store and retrieve key value pairs.

How do you know what version etcdctl is configured to work with.

./etcdctl --version

To change the etcdctl to work with API version 3

ETCDCTL_API=3 ./etcdctl version

You may do this for writing before you run each command or you can set it in your environment_variable

export ETCDCTL_API=3
./etcdctl version

14. ETCD in Kubernetes

Etcd's role in kubernetes

The etcd data store store information about the cluster such as the Nodes , Pods, Configs, secrets, accounts, roles, bindings, others

Every information you see when you run the kubectl get command is from the etcd server

Every change you make to the cluster such as adding additional nodes, define Pods or replicasets are updated in the etcd server . Only once it is updated in the etcd server is the change considered to be complete. Depending upon how your cluster is configured etcd is deployed differently.

Through out this section we discuss about two types of deployments .

1. one deployed from scratch
2. And other using the Kubeadm tool

If you set the cluster from scratch you begin by deploying etcd by downloading the etcd binaries . Installing and deploying as service in your Master Node. There are many options passed on to the service a number of them relate to certificates

The only think to note for now is the

--advertise-client -urls https://$INTERNAL_IP):2379

This is the address to which etcd listens to that happens to be on the IP of the server. And on port 2379 which is its default port.

This is the URL that needs to configured on KubeAPI server when it tries to reach the etcd server .

If you setup your setup your cluster using kubeadm , then kubeadm deploys the etcd server for you as Pod in the kube-system namespace.

You can explore the etcd database by using etcdctl utility within this Pod.

To list all keys stored by kubernetes

kubectl exec etcd-master -n kube-system etcdctl get / -- prefix -keys-only

Kuberenetes stores data in a specific directory structure . A root directory is a registry and under that there are various kubernetes constructs such as .

In a HA environment you will have multiple masters in your cluster . Then you will multiple ETCD instances spread across the master nodes. In that make sure the etcd instances know about each other by setting the right parameter in etcd service configuration.

The initial cluster option is where you have to specify different instances of of etcd service .
We speak about high availability in much more detail later in the course.

15 . ETCD - Commands (Optional)

(Optional) Additional information about ETCDCTL Utility

ETCDCTL is the CLI tool used to interact with ETCD.

ETCDCTL can interact with ETCD Server using 2 API versions - Version 2 and Version 3. By default its set to use Version 2. Each version has different sets of commands.

For example ETCDCTL version 2 supports the following commands:

etcdctl backup
etcdctl cluster-health
etcdctl mk
etcdctl mkdir
etcdctl set

Whereas the commands are different in version 3

etcdctl snapshot save 
etcdctl endpoint health
etcdctl get
etcdctl put

To set the right version of API set the environment variable ETCDCTL_API command

export ETCDCTL_API=3

When API version is not set, it is assumed to be set to version 2. And version 3 commands listed above don't work. When API version is set to version 3, version 2 commands listed above don't work.

Apart from that, you must also specify path to certificate files so that ETCDCTL can authenticate to the ETCD API Server. The certificate files are available in the etcd-master at the following path. We discuss more about certificates in the security section of this course. So don't worry if this looks complex:

--cacert /etc/kubernetes/pki/etcd/ca.crt     
--cert /etc/kubernetes/pki/etcd/server.crt     
--key /etc/kubernetes/pki/etcd/server.key

So for the commands I showed in the previous video to work you must specify the ETCDCTL API version and path to certificate files. Below is the final form:

kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl get / --prefix --keys-only --limit=10 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt  --key /etc/kubernetes/pki/etcd/server.key"

16. Kube-API Server

Kube-Api server is the primary management component in kubernetes

When you run a kubectl command : The kubectl utility is infact reaching to the kubeapi server .
the kubeapi server first authenticates the requests and validates it then retrieves the data from the ETCD server and responds back with the requested information.

you really do not need to use the kubectl command instead you can also invoke the APIs directly by sending a Post request like this.

Lets look at an example of creating a Pod. when you do that as before the request is authenticated and then validated in this case the API server creates a Pod object without assigning it to a node. updates the information in the etcd server and updates the user that the Pod is been created.

The scheduler continuously monitors API server realizes a new pod is created without the node . The scheduler identifies a node to assign the new pod on and communicates back to the kubeapi server. The api server then updates the information in the etcd cluster . The API server then passes that information to kubelet in the appropriate worker node. The kubelet then creates the Pod on the node and instructs the container run time engine to deploy the application image. once done the kubelet then updates the status back to the API server. and the API server then updates the information back in the etcd cluster.

Kube-Api server is at the center of all the tasks that needs to be performed to make a change in the cluster.

If you Bootstrapped your cluster using the kubeadm tool . Then you don't need to know this. But if you are setting up the hard way then the kube-api server is available as a binary in the kubernetes release page. Download it and configure it to run as a service on your kubernetes Master node.

The kubeapi server is run with loads of parameters as you can see here below.

Through out this session we are going to take a peak at how to configure these individual components of the kubernetes architecture.

The option etcd servers specify the location of the etcd servers . This is how kubeapi-servers connects to the etcd servers .

So how do you view the kubeapi server options in an existing cluster. It depends on how you setup your cluster. If you set it up with a kubeadm tool , the kubeadm deploys the "kubeapi-server" as a Pod in the kube-system namespace on the Master Node.

You can see the options in the Pod definitions file -- used kudeadm

cat /etc/kubernetes/manifests/kube-apiserver.yaml

In a non-kubeadm setup you can inspect the options in the kubeapi service located @

cat /etc/systemd/system/kube-apiserver.service

You can also see the running process and the effective options by listing the process on master node. by searching for kube-apiserver .

ps -aux | grep kube-apiserver

17 : Kube-Controller Manager

kube-controller manager manages various controllers in kubernetes. A controller is like an office or a department in the master ship that have its own set of responsibilities. An office for the ships will be responsible for the monitoring and taking appropriate actions for the ships whenever a new ship arrives , when a ship leaves or destroyed.

Another office could that it manages the containers on the ship. That take care of containers that are damaged or fall of ships . So these offices are continuously on the look out for the status of the ships and takes necessary actions to remediate the situation.

In Kubernetes terms Controller is a process that continuously monitoring the status of different components within the system and to bring the whole system into a desired functioning state.

Node Controller : Node controller is responsible for monitoring the status of the nodes and taking necessary actions to keep the application running. it does this with the help of the kube-apiserver . The node controller checks the status every 5 seconds that way the node controller can monitor the health of the nodes. If it stops receiving heart beat from a node, it marks the node as unreachable -- but it waits for 40 secs before marking it unreachable. After marking a node as unreachable it gives it 5 minutes to come back up if it does not . It removes the pods assigned to it and provisions it on the healthy ones if the Pods are part of replicaset .

Replication Controller :

It is responsible for monitor the status of the replicasets and the desired number of Pods are available at all times in the set . If a Pod dies it creates another one

There are many more such controllers in kubernetes .

Whatever components that we seen in kubernetes . or whatever intelligence that we have build into this constructs is implemented through these various controllers.

How do you see these controllers and where are they located in your cluster. They are packaged into a single process called "Kube-Controller-Manager" . When you install the Kube-Controller-Manager different controllers get installed as well.

So how do you install and view the Kubernetes controller manager . Download the Kube-Controller-Manager from the kubernetes release page , extract it and run it as a service.

kube-controller-manager.service

you can see a list of options provided.

This is where you will . This is where you will give additional options to customize your controller.

There is additional options called controllers that you can use to specify which controllers to enable. Bu default all of them are enabled but you can choose to a select few.

In case if any of your controllers do not seem to exist or work . This would be a good starting point to look at.

How do you view kube-controller-manager server options.

Again it depends upon how you setup your cluster . If you set it up with kubeadm tool . Kubeadm deploys a kube-controller-manager as a pod in the kube-system namespace on the master node

You can see the options within the pod definitions file

cat /etc/kubernetes/manifests/kube-controller-manager.yaml

In a non-kube admin set up

you can you can inspect the options by viewing the

cat /etc/systemd/system/kube-controller-manager.service

you can also see the running process and the effective options by listing the process on the master node. And searching for kube-controller-manager

ps -aux | grep kube-controller-manager

Kube-scheduler is only responsible of decising which Pod goes to which Node . it does not actually place the pod on the nodes . That is the job of the kublet , the kubelet is the captain on the ship(worker-node) who creates the Pod on the node. The scheduler only decides which Pod go where .

In detail.

Why do you a scheduler , when there are many containers and many ships you want to make sure the right container end up on the right ship. For example there could be different sizes ships and containers .
You want to make sure the ship has sufficient capacity to host those containers .

Different ships might be going to different destinations. you want to make sure your containers are placed on the right ship so they end up in the right destination.

In kubernetes the scheduler decides which node the pods are placed on depending on certain criteria . You may have Pods with different resource requirements . You can have nodes in the cluster dedicated to certain applications . So how does the scheduler assigns these pods ?

The scheduler look at each Pod and tries to find the best node for it. for example lets take one of these Pods. Thick blue one . It has a set of CPU and memory requirements

The scheduler goes into two Phases to identify the best node for the Pod.

1st Phase : The scheduler tries to filters out the nodes that do not fit the profile for this pod. For example the nodes that do not have sufficient cpu and memory requirement requested by the Pod. So the first two nodes are filtered out , diagram below.

So now we are left with two pods on which it can be placed. Now how does the scheduler pick one from the two.
2nd Phase:
The scheduler ranks the node as the best fit for the pod, it used a priority function to assign a scod to the node on the scale of 0-10. For example the scheduler calculates number of resources that would be free after placing the nodes on the node. In the above case the one on the right will have 6 CPUs free. The Pod was placed on it four more than the other one and it gets a better rank and so it wins. And of course this can be customized and you can write your own scheduler as well.

How do you install the kube-schdudler ?

Download the kube-scheduler binary from the kubernetes page extract it and run it as a service.

kube-schduler.service

when you run it as a service you specify the scheduler configuration file.

ps -aux | grep kube-scheduler

19: Kublet

Earlier we discussed that the kublet is like the captain on the ship they lead all the activities on the ship . They are responsible for doing all the paperwork that are necessary to be the part of the cluster. They are the sole point of contact from the master ship . they load and unload containers on the ship as instructed by the scheduler on the master. They also send back reports on the status of the ship and the containers on them .

The kubelet in the kubernetes worker node registers the node with the kubernetes cluster. When it receives instruction to load a container or a pod to a node, it requests the container run time engine which might be docker to pull the required image and run an instance. The kublet then continues to monitor the state of the pod and containers in it and reports to the kubeapi server on a timely basis.

So how you install the kubelet.

If you use the kubeadm tool to deploy the cluster . It does not automatically deploy the kubelet. That is the difference between other components . You must always manually deploy the kublets in your worker nodes.

Download the Installer and extract and run it .

ps -aux | grep kubelet

20. Kube Proxy

Within a kubernetes cluster every pod can reach every other pod . This is accomplished by deploying a Pod networking solution to the cluster . A Pod Network is an internal virtual network that spans across all the nodes in a cluster to which all the Pods connect to. Through this network they are able to communicate with each other . There are many solutions for deploying such a network . In this case I have a web-application deployed on the first node and a database application deployed on the second.

The web-application can reach the database simply by using the IP on the Pod . But there is no guarantee that the Ip of the database pod will always remain the same. If you gone through the lecture of services as discussed in the beginners course you must know a better way for the web-application is using a service so we create a service to expose the database application across the cluster. The web application can now access the database by using the name of the service:db

The service also gets an IP address assigned to it whenever a pod reach the service using the IP or the service name it forwards the request to the backend pod in this case the database.

But what is a service and how does it get an IP ? Does the service join the same pod network . The service cannot join the Pod Network . Because a service is not an actual thing. It is not a container like pods. It does not have any interfaces or an actively listening process .

Service : It is a virtual component that only lives in Kubernetes Memory.

But we also said that a service must accessible from across the cluster from any nodes. So how is that achieved -- That's where kube-proxy comes in.

Kube-Proxy is a process that runs on each node in the kubernetes cluster -- Its jobs is to look for services and every time a new service is created it creates an appropriate rule on each node to forward traffic to those services to forward traffic to the backend pods one way it does is by using IP Table rules.
it creates an IP tables rule in each node in the cluster to forward traffic heading to the IP of the service to the pod that has the database server in it.

This is the high level overview for now.

Installing Kube-Proxy

Download the kube-proxy from the kubernetes release page

Download it from the kubernetes release page extract it and run it as a service.

kube-proxy.service

The kubeadm tool deploys kube-proxy as a pods on each node , in fact it is deployed as a deamon set . So a single pod is deployed in each node in a cluster .2

If you do not know about deamon set you we will be discussing about it in a detail lecture .

21. Recap - Pods

till 21 - 40 : Is covered in Absoluted Beginners.

41 : Namespace

Name : SMITH from two house are taken as and analogy .

So far in this course we created objects such as Pods, Deployments and Services in our cluster what ever we have been doing. We have been doing in a namespace.

We were inside a house all this while. This house is called the default namespace. And it is created automatically by kubernetes when the cluster is first setup .

Kubernetes creates a set of Pods and services for its internal purpose such as that is required by the networking solution the DNS service etc

44 : Imperative and declarative :

In the imperative approach we have two ways.

Declarative Approach :

Is to create a set of files that defines the expected state of the applications and service on the kubernetes cluster and with a single kubectl apply command you should be able to read the configuration files and decide by itself what needs to be done to bring the infrastructure to the expected state.

In declarative approach you will use kubectl apply command , for creating , updating or deleting an object

The apply command will look at the existing configuration and figure-out what needs to be done and what changes needs to be made to the system.

In cases you may want to completely delete and recreate objects in those cases .

kubectl replace --force -f nginx.yml

The best way is to update the local copy of the nginx.yml file and updating the changes and then use the replace command to update it .

kubectl replace --force -f nginx.yaml

or delete command to delete the object.

What if you run the create command if the object already exists. Then it would fail saying that a Pod already exists. When you update an object you must always make sure that the object exist before running a replace command.

In an IMperative approach you must always check if the things are in place, which requires manual intervention.

Declarative :

You use the same object configuration files that we were working for

Instead of the create or replace command , we use kubectl apply command to manage objects.

kubectl apply -f nginx.yaml

This is intelligent enough to create an object if the object does not already exists.

If there are multiple object configuration files as you would usually then you may specify a path of the directory instead of a direct file. That way all the objects are created at once.

example:

kubectl apply -f /path/to/config-files

When changes are to be made we simply update the object configuration files. And run the kubectl appy command again. And this time it knows that the objects exists. So it only updates the objects with the new changes.

kubectl apply -f nginx.yaml

In the next lecture we will learn how the kubectl command works in the backend.

from the Exam point of view you can use the imperative approach to save time as much as possible.