Kubernetes troubleshooting

Kubernetes cluster

Kubernetes API

Kubernetes stores the cluster state in the ETCD database among other metrics. This information can be obtained through an API. In order to access that data, you must run the commands explained in the following section in a host where the k8s certificates are configured to allow the access to the cluster API.

This section provides some troubleshooting ideas and solutions to common problems you might encounter with the Kubernetes cluster. The most relevant tool to learn here is the kubectl command line tool, which lets you control Kubernetes clusters. For configuration, kubectl looks for a file named config in the $HOME/.kube directory. You can specify other kubeconfig files by setting the KUBECONFIG environment variable or by setting the –kubeconfig flag. For more information about kubectl syntax, full command operations and details about each command, including all the supported flags and subcommands, see the kubectl reference documentation.

The first step should be checking the cluster API connectivity with the following command:

kubectl config view

In case that the API access is not set up in the VM where the command is executed, the output will be have no data:

quobis@ops-node1-delorean:~$ kubectl config view
apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

The complete output must contain all the data related with selected cluster:

quobis@OPS-master-delorean:~$  kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://IP:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: dev-nightly
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

Make sure the selected context are the cluster we what to manage. The row with an asterisk in the beginning defines the current context and the last column defines the default namespace selected (“my-context” and “dev-nightly” in the example below):

console:~/Devops/k8s-installer/ansible$ kubectl config get-contexts

        CURRENT NAME             CLUSTER                    AUTHINFO                                                NAMESPACE
            akspreGDQuobis           akspreGDQuobis                 clusterUser_preGDQuobis_akspreGDQuobis                  quobis
            aksproGDQuobis           aksproGDQuobis                 clusterUser_proGDQuobis_aksproGDQuobis                  wac-quobis
            aksstaging               aksstaging                     clusterUser_testGDQuobis_aksstaging                     stagingquobis
            cluster.quobis.com       cluster.quobis.com             cluster.quobis.com                                      quobis-qa
            clusterquobis.quobis.com clusterquobis.quobis.com   clusterquobis.quobis.com                                    kube-system
*           my-context               kubernetes                     kubernetes-admin                                        dev-nightly
            preprod                  preprod                        clusterUser_test_preprod                                quobis
            proGD                    proGD                          clusterUser_pro_gemelo_digital_proGD                    wac-quobis

In order to change to another context, use the following command:

kubectl config use-context <desired-context>

In order to change the default namespace, use the following command:

kubectl config set-context --current --namespace=<desired-namespace>

Kubernetes cluster state

The main element in a Kubernetes cluster are the nodes (VM) where containers are running, so checking the nodes’ state should be the first step once the k8s API is configured. The cluster node must have a “Ready” state, any other case indicates one of the problems.

quobis@OPS-master-delorean:~$ kubectl get nodes -o wide
NAME                        STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION   CONTAINER-RUNTIME
k8s-master                  Ready    master   140d   v1.15.5   10.1.72.104   <none>            Debian GNU/Linux 9 (stretch)   4.9.0-16-amd64   docker://18.9.9
kubernetes-janus-edu        NotReady <none>   140d   v1.15.5   10.1.72.107   <none>            Debian GNU/Linux 9 (stretch)   4.9.0-8-amd64    docker://18.9.9
ops-k3s                     Ready    <none>   107d   v1.15.5   10.1.72.71    <none>            Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.9.9
ops-kubernetes-janus2-edu   Ready    <none>   140d   v1.15.5   10.1.72.108   <none>            Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.9.9
ops-node1-delorean          Ready    <none>   140d   v1.15.5   10.1.72.105   <none>            Debian GNU/Linux 9 (stretch)   4.9.0-12-amd64   docker://18.9.9
ops-node2-delorean          Ready    <none>   140d   v1.15.5   10.1.72.106   <none>            Debian GNU/Linux 9 (stretch)   4.9.0-9-amd64    docker://18.9.9

Kubernetes node troubleshooting

Kubectl describe node <unhealth_node_name>

• Check the state of VM: the Virtual Machine must accomplish the following requirements:

  • It must be on.

  • It must have network connectivity to the rest of the cluster machines.

• Check kubernetes dependencies running on the VM:

  • Kubelet service must be running: sudo systemctl status kubelet

  • Kubelet logs: sudo journalctl -u kubelet (error logs should not appear)

  • The iptables command must contain the following entries (sudo iptables -L), several entries can insert additional delays in the system.

• Check the native k8s pod in kube-system namespace:

kubectl -n kube-system get pods

• Make sure kube-flannel is deployed and running.

In order to check the state of the pods of a given namespace pod, we need to use the following command:

kubectl get pods -n namespace_name

In order to get additional information of a specific pod, the instruction to issue is the following one:

$ kubectl -n <namespace> describe pod <pod_name>

In order to reboot a pod, we just need to delete it and Kubernetes will recreate it again. The command to issue is:

$ kubectl -n <namespace> delete pod <pod_name>

Kubernetes namespaces and pod list

The following namespaces are always available in our platform after a clean installation:

• “kube-system” namespace: used internally by Kubernetes.

• “monitoring” namespace: host the monitoring services (Grafana, Loki, etc)

• “quobis” namespace: hosts the application services (authentication server, signalling server, databases, etc..)

This is the list of the pods that run in the kube-system namespace, that can be listed with the kubectl get pods -n kube-system:

quobis@vm-bar-quo-mas11-1:~$ kubectl get pods -n monitoring
NAME                          READY STATUS  RESTARTS AGE
controller-6c4449fc67-pfn48   1/1   Running 1        55d
coredns-5c98db65d4-9r99t      1/1   Running 1        55d
coredns-5c98db65d4-l8v4v      1/1   Running 1        55d
etcd-vm-bar-quo-mas11-1       1/1   Running 9        55d
kube-apiserver-vm             1/1   Running 10       55d
kube-controller-manager-vm    1/1   Running 18       55d
kube-flannel-ds-2z2qm         1/1   Running 0        55d
kube-proxy-9mflq              1/1   Running 0        55d
kube-scheduler-vm-bar         1/1   Running 14       55d
nginx-ingress-controller      1/1   Running 0        22d
node-exporter-6xm5h           1/1   Running 1        55d
speaker-cxf8r                 1/1   Running 1        55d

Note

• There might be more that one pod with the same name but different suffix (for example, coredns-5c98db65d4-9r99t and 5c98db65d4-l8v4v). These are just replicas of the same image running at the same time into this namespace.

• You might not find all these services in your deployment, as some of them are optional and/or might be deactivated. For example, the “teal” service is not deployed if the deployment does not require blue/green support.

This is the list of the pods that run in the monitoring namespace, that can be listed with the kubectl get pods -n monitoring:

quobis@vm-bar-quo-mas11-1:~$ kubectl get pods -n monitoring
NAME                          READY  STATUS    RESTARTS AGE
grafana-54d5f96758-m4gtj      1/1    Running   0        27d
loki-5d8c457556-8qdhb         1/1    Running   0        51d
prometheus-b9d4f8f54-dpxtv    1/1    Running   0        21d
promtail-c626r                1/1    Running   0        20d

Pods included in the monitoring namespace

Pod name	Functionality	Deployment considerations
grafana	Grafana instance for logging and troubleshooting.	No HA required.
loki	Loki instance for log collection and management.	No HA required.
prometheus	Prometheus instance as a storage for Grafana and Loki	No HA required.
promtail	Agent which ships the contents of local logs to a the Grafana Loki instance	No HA required.

This is the list of the pods that run in the quobis namespace, that can be listed with the kubectl get pods -n quobis:

quobis@vm-bar-quo-mas11-1:~$ kubectl get pods -n quobis
NAME                                        READY   STATUS  RESTARTS AGE
audiomixer-sfu1-0                           1/1     Running 0        22d
audiomixer-sfu2-0                           1/1     Running 0        22d
database-mongo-0                            2/2     Running 0        6d4h
database-mongo-1                            2/2     Running 0        6d4h
database-mongo-2                            2/2     Running 0        6d4h
erebus-787567f497-lnrcr                     1/1     Running 0        6d15h
kapi-759d58f644-m5db2                       2/2     Running 0        22d
keycloak-77f6796b89-9ngw7                   1/1     Running 0        6d16h
kitter-5b5647c888-dbdfw                     1/1     Running 0        13d
kube-state-metrics-bc95796c9-hb42c          2/2     Running 0        43d
message-broker-5675fb65b9-755s6             1/1     Running 0        6d16h
mongodb-exporter-548d4dcddd-7bt67           1/1     Running 0        43d
mongodb-kubernetes-operator-6c6cbdf9bf-8jwg 1/1     Running 0        6d16h
nfs-client-provisioner-5697c964d6-pwgl5     1/1     Running 0        6d16h
nginx-gw-0                                  1/1     Running 0        13d
podhandler-1640265360-4n5vw                 0/1     Running 0        13d
postgresql-5c9f585dc8-bw5t9                 1/1     Running 0        6d16h
qss-audiomixersio-sfu1-86cf778b84-28ct2     1/1     Running 0        6d15h
qss-audiomixersio-sfu2-57454f59d-brswp      1/1     Running 0        6d15h
qss-auth-http-679d4475d7-w65ch              1/1     Running 0        6d15h
qss-calls-574cbbbcdd-j94wx                  1/1     Running 0        6d15h
qss-calltransfer-basic-5c66bf9764-lg2vk     1/1     Running 0        6d15h
qss-conference-state-7b8d4f448-dnbhk        1/1     Running 0        6d15h
qss-invites-rooms-7d7749896f-rhbjw          1/1     Running 0        6d15h
qss-io-websockets-6bcd86694c-pzd8p          1/1     Running 0        6d15h
qss-io-websockets-6bcd86694c-rv8fq          1/1     Running 0        6d15h
qss-log-conference-7fdd67d9f5-gkdrt         1/1     Running 0        6d15h
qss-meeting-basic-6d6789b86-h7642           1/1     Running 0        6d15h
qss-peer-jt-6468689c87-7t5hg                1/1     Running 0        6d15h
qss-quick-conference-844d87c4f5-4v8j8       1/1     Running 0        6d15h
qss-registry-authenticated-59dc88c8ff-b6w4d 1/1     Running 0        3h14m
qss-resolver-wac-6f4787c4f4-9fcv5           1/1     Running 0        6d15h
qss-rooms-basic-6798d49d48-lmf9v            1/1     Running 0        6d15h
qss-trunk-85dcc5cb9f-jd87x                  1/1     Running 0        3h13m
qss-watchdog-invites-6dc97486f6-qstg2       1/1     Running 0        6d15h
qss-watchdog-registry-675f8b55c9-58bm7      1/1     Running 0        6d15h
sfu-dispatcher-66f58bf667-bll57             1/1     Running 0        8d
sfu-wrapper-sfu1-598d59fd79-mplxb           1/1     Running 4        8d
sfu1-0                                      1/1     Running 0        22d
sfu2-0                                      1/1     Running 0        22d
sip-proxy-1-0                               1/1     Running 0        22d
sip-proxy-2-0                               1/1     Running 0        22d
sip-proxy-database-6bf67bc969-q4v2j         1/1     Running 1        55d
sippo-exporter-678ff77cff-4dztt             1/1     Running 0        14d
sippo-maintainer-1640228400-tldsm           0/1     Completed        0d10h
sippo-server-755f6d8f9b-ckgzx               1/1     Running 7        22d
sippo-server-755f6d8f9b-pzmdh               1/1     Running 0        6d15h
sippo-server-755f6d8f9b-q7gmr               1/1     Running 7        22d
sippo-storage-86c7bcb6d4-bhbnf              1/1     Running 0        6d16h
webphone-angular-8677ccbcc8-gkmh9           1/1     Running 0        12d
xmpp-server-58ccd4bb86-b6s4t                1/1     Running 0        41d

The list below gives a brief explanation of each service and its impact in case or an issue with this pod.

Pods included in quobis namespace

Pod name	Functionality	Deployment considerations
audiomixer	Audio MCU, handles the mixing of the audio streams and media interconnection with the PSTN/NGN.	Deployed in HA. If one instance gets down, all the active calls using it will fail. Following calls will use the active instance.
database-mongo	Internal database to storage user data.	Deployed in HA as a cluster.
erebus	Responsible for listening to WebSocket connections from clients and feeding them into our internal message broker	No HA, needs to be re-created in case of failure as all the clients traffic goes through this element
kapi	Provides a REST API used for the Kubernetes cluster maintenance.	HA not required. No critical impact in the service in case of failure.
keycloack	Identity and access management solution	No HA. Needs to be re-created in case of failure, otherwise users without a session token won’t be able to login.
kitter	Provides an API to manage user accounts (CRUD).	No HA. Needs to be re-created in case of failure, otherwise no news users can be added to the system.
message-broker	Message broker using AMQP protocol based on RabbitMQ	No HA. Needs to be re-created in case of failure, otherwise there won’t be message interchange between several services.
mongodb-exporter	Exports data from the Mongo database to the monitoring system	No HA. No impact in the service in case of failure.
mongodb-kubernetes-operator	Manages the lifecycle events for a MongoDB cluster by using the K8S API	No HA. No impact in the service in case of failure.
nfs-client-provisioner	Atomatic provisioner that uses existing NFS server to support dynamic provisioning of Kubernetes Persistent Volumes.	No HA. No impact in the service in case of failure.
nginx	Web server, reverse proxying, caching and load balancing for HTTP, TCP, and UDP servers.	HA provided by the cluster.
postgresql	Postgresql database.	No HA. Used by the chat messaging server, voice and audio calls are not affected by a potential failure.
qss	Quobis Signaling Server	All QSS microservices can be replicated. More info here.
recording	Responsible for post-call processing to create audio and video recordings.	No HA, needs to be re-created in case of failure as calls won’t be recorded.
sfu-dispatcher	Balances the traffic between different video SFUs	No HA, needs to be re-created in case of failure. The call information is kept in the DB to be persistent across restarts.
sfu-wrapper	Connector between the SFU dispatcher and the SFU itself	Supports HA. If one instance gets down, all the active calls using it will fail. Following calls will use the active instance.
sfu	Video SFU.	Supports HA that is managed by the SFU wrapper and dispatcher services.
sip-proxy	SIP application server that acts as a gateway to an external telephony network	Supports HA.
sip-proxy-database	MySQL database backend for the SIP proxy. Supports HA if required (not deployed by default)	Supports HA if required (not deployed by default)
sippo-exporter	Exports aggregated data to the monitoring system.	No HA needed.
sippo-maintainer	Runs daily operations for system maintenance.	No HA needed.
sippo-server	Main process that orchestrates the sytem.	HA supported, several replicas can be deployed and traffic is balanced from the message broker.
sippo-storage	Storage backend	No HA needed.
teal	Responsible for parsing login responses to support blue/green canary deployments (not needed for standard deployments)	No HA. Needs to be re-created in case of failure, otherwise users without a session cooking won’t be able to login.
turn-server	Responsible for the implementation of TURN media relay	HA supported, several replicas can be deployed.
xmpp-server	Messaging server for chat services.	No HA. Needs to be re-created in case of failure, otherwise the chat services won’t work.