1. Logs
- Components level logs
-
kubelet,kube-apiserver,kube-scheduler,kube-controller,etcd,kube-proxy…
-
- Pod level logs
- generate by container
1.1 Log files
The base path for logs is /var/log/
| Component | Log Path | Description |
|---|---|---|
| kubelet | /var/log/syslog |
Kubelet logs if not using journald. |
| kube-apiserver | /var/log/kube-apiserver.log |
API server logs (depends on setup). |
| kube-scheduler | /var/log/kube-scheduler.log |
Scheduler logs (depends on setup). |
| kube-controller-manager | /var/log/kube-controller-manager.log |
Controller manager logs. |
| Pod logs |
/var/log/pods/ /var/log/containers/
|
Logs for all containers running on the node. Docker: /var/lib/docker/containers/<container-id>/<container-id>-json.log and containerd: /var/log/pods/<namespace>/<pod-name>/<container-name>.log
|
| etcd | /var/log/etcd.log |
etcd logs, if configured |
1.2 CMD
kubelet logs:
grep kubelet /var/log/syslog
journalctl -u [kubectl|kube-controller-manager|kube-scheduler|kube-apiserver|etcd|docker|containerd]
#
kubectl describe pod <pod-name> -n kube-system
kubectl logs <pod-name> -c <container-name>
kubectl logs <pod-name> -n kube-system
crictl ps
crictl logs
docker ps
docker logs # (in case when Docker is used)
2. Config
2.1 Config Files
The basic path for kubernetes config file store in /etc/kubernetes/ folder, for kubelet normaly
stores at /var/lib/kubelet/config.yaml
Default Paths:
-
/etc/kubernetes/: Central directory for most Kubernetes configuration files. -
/var/lib/kubelet/: Directory for kubelet-related data and configuration. -
/var/lib/etcd/: Directory for etcd data. -
/etc/systemd/system/: Directory for systemd service configuration files.
| Component | Config Path | Description | Params |
|---|---|---|---|
| kubelet | /var/lib/kubelet/config.yaml |
Main kubelet configuration file. | |
/etc/kubernetes/kubelet.conf |
Kubeconfig for API server communication. |
--config: Points to the kubelet’s configuration file (e.g., /var/lib/kubelet/config.yaml). --pod-manifest-path: Specifies the directory for static pod manifests (default: /etc/kubernetes/manifests). --kubeconfig: Path to the kubelet’s kubeconfig file for API server communication (default: /etc/kubernetes/kubelet.conf). |
|
| kube-apiserver | /etc/kubernetes/manifests/kube-apiserver.yaml |
Static pod manifest for API server. |
--etcd-servers: Specifies the etcd endpoints. --secure-port: Port for secure communication (default: 6443). --service-cluster-ip-range: Range of IPs for cluster services. --tls-cert-file and --tls-private-key-file: Certificates for secure |
| kube-scheduler | /etc/kubernetes/manifests/kube-scheduler.yaml |
Static pod manifest for scheduler. |
--kubeconfig: Path to the kubeconfig file for the scheduler. --leader-elect: Ensures only one active scheduler in HA setups. |
| kube-controller-manager | /etc/kubernetes/manifests/kube-controller-manager.yaml |
Static pod manifest. |
--kubeconfig: Path to the kubeconfig file for the controller manager. --service-account-private-key-file: Key for signing service account tokens. --cluster-signing-cert-file and --cluster-signing-key-file: Certificates for signing. |
| kubeconfig files | /etc/kubernetes/admin.conf |
Admin kubeconfig for kubectl. | |
/etc/kubernetes/controller-manager.conf |
Kubeconfig for controller manager. | ||
/etc/kubernetes/scheduler.conf |
Kubeconfig for scheduler. | ||
| CNI plugins | /etc/cni/net.d/ |
Directory for CNI plugin configurations. | |
| Static pods | /etc/kubernetes/manifests/ |
Directory for static pod definitions. | |
| etcd | /etc/kubernetes/manifests/etcd.yaml |
etcd configuration. |
--data-dir: Directory for storing etcd data. --advertise-client-urls: URLs advertised to clients. --listen-client-urls: URLs where etcd listens for client requests. |
| kube-proxy | /var/lib/kube-proxy/config.conf |
Often referenced in the kube-proxy pod or daemonset |
--cluster-cidr: Range of IPs for the cluster network. --kubeconfig: Path to the kubeconfig file for API server communication. |
2.1.1 /etc/kubernetes/manifests/ file
As kube-controller always monitor the manifests folder files, so we can directly edit in manifests file. But it’s better to backup first.
## always make a backup
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.bk
## make the change
vim /etc/kubernetes/manifests/kube-apiserver.yaml
## wait till container restarts
watch crictl ps
## check for apiserver pod
k -n kube-system get pod
2.2 CMD
# View the active kubeconfig settings.
kubectl config view
# check the corresponding pods
kubectl describe pod <pod-name> -n kube-system
# use systemctl cat
systemctl cat [kubectl|kube-controller-manager|kube-scheduler|kube-apiserver|etcd|docker|containerd]
# can even use find
find / | grep kubeadm
By understanding these logs, configurations and their purposes, we can better manage and troubleshoot our Kubernetes cluster.