Calico Node CrashLoopBackOff

1. Desc

MicroK8s 설치 후, calico-node에서 IPv4를 찾지 못해, pod 생성 실패
calico-node 생성 yaml 파일에서 IP auto detection 전략이 안맞음

2. Solution

https://github.com/ubuntu/microk8s/issues/1554#issuecomment-691426908
/var/snap/microk8s/current/args/cni-network/cni.yaml파일의 IP_AUTODETECTION_METHOD를 "first-found"에서 "interface=wlo.*"로 변경
- microk8s kubectl apply -f /var/snap/microk8s/current/args/cni-network/cni.yaml
Calico의 IP auto detection이 잘 적용되지 않음. DaemonSet 구동후 pod별로 CIDR 수동 설정이 필요
- `kubectl patch pod {podName} -p '{"containers": {"env":{"

3. Trials

hostname에 대문자가 있으면 Kubernetes에서 문제가 발생한다는 이슈 확인. hostname을 소문자로 변경. hostnamectl set-hostname seungbae-pn41. 해결안됨.
ip a 명령어로 조회시, wlo1에서 네트워크 제공. lo가 있어서 이를 인식하지 않음. 해결됨.

3. Reference

Details

seungbae@seungbae-PN41:~$ kubectl get pods -n kube-system
NAME                                         READY   STATUS              RESTARTS   AGE
calico-kube-controllers-f7868dd95-d9f4f      0/1     ContainerCreating   0          40m
metrics-server-8bbfb4bdb-7j2tt               0/1     ContainerCreating   0          26m
dashboard-metrics-scraper-78d7698477-5t4jg   0/1     ContainerCreating   0          25m
kubernetes-dashboard-85fd7f45cb-qd6hb        0/1     ContainerCreating   0          25m
calico-node-hcctb                            0/1     CrashLoopBackOff    16         40m
seungbae@seungbae-PN41:~$ kubectl logs calico-node-hcctb -n kube-system
2021-09-23 03:57:50.911 [INFO][8] startup.go 290: Early log level set to info
2021-09-23 03:57:50.911 [INFO][8] startup.go 306: Using NODENAME environment for node name
2021-09-23 03:57:50.911 [INFO][8] startup.go 318: Determined node name: seungbae-pn41
2021-09-23 03:57:50.913 [INFO][8] startup.go 350: Checking datastore connection
2021-09-23 03:57:50.927 [INFO][8] startup.go 374: Datastore connection verified
2021-09-23 03:57:50.927 [INFO][8] startup.go 102: Datastore is ready
2021-09-23 03:57:50.931 [INFO][8] customresource.go 101: Error getting resource Key=GlobalFelixConfig(name=CalicoVersion) Name="calicoversion" Resource="GlobalFelixConfigs" error=the server could not find the requested resource (get GlobalFelixConfigs.crd.projectcalico.org calicoversion)
2021-09-23 03:57:50.937 [INFO][8] startup.go 416: Initialize BGP data
2021-09-23 03:57:50.937 [WARNING][8] startup.go 648: Unable to auto-detect an IPv4 address: no valid IPv4 addresses found on the host interfaces
2021-09-23 03:57:50.937 [WARNING][8] startup.go 438: Couldn't autodetect an IPv4 address. If auto-detecting, choose a different autodetection method. Otherwise provide an explicit address.
2021-09-23 03:57:50.937 [INFO][8] startup.go 244: Clearing out-of-date IPv4 address from this node IP=""
2021-09-23 03:57:50.946 [WARNING][8] startup.go 1187: Terminating
Calico node failed to start

seungbae@seungbae-PN41:~$ kubectl describe pods kubernetes-dashboard-85fd7f45cb-qd6hb -n kube-system
Name:           kubernetes-dashboard-85fd7f45cb-qd6hb
Namespace:      kube-system
Priority:       0
Node:           seungbae-pn41/192.168.0.15
Start Time:     Thu, 23 Sep 2021 12:32:24 +0900
Labels:         k8s-app=kubernetes-dashboard
                pod-template-hash=85fd7f45cb
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/kubernetes-dashboard-85fd7f45cb
Containers:
  kubernetes-dashboard:
    Container ID:
    Image:         kubernetesui/dashboard:v2.0.0
    Image ID:
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --auto-generate-certificates
      --namespace=kube-system
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /certs from kubernetes-dashboard-certs (rw)
      /tmp from tmp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9twxv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kubernetes-dashboard-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubernetes-dashboard-certs
    Optional:    false
  tmp-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-9twxv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              beta.kubernetes.io/os=linux
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                     From     Message
  ----     ------                  ----                    ----     -------
  Warning  MissingClusterDNS       35m (x1736 over 6h55m)  kubelet  pod: "kubernetes-dashboard-85fd7f45cb-qd6hb_kube-system(a1d0d0cd-ab17-4e58-81ff-90baed758413)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  Warning  FailedCreatePodSandBox  33m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0d96af449a8a95ca1ff4339876470f46936ca9241354fd219473fe715d74470c": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  33m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b00568d04bd177d47b2604fd90f4497b2baa103367d48184d0dcd66b8656366f": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  33m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c5a20eac4b10bd0a16032d682195e52840b5b7b166f23a2f01dbcd04a78b3925": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  32m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "812d69d44917b9a92672842acfcaceda4f3cb41c5dd81001a1f78abed86697fa": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  32m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ddb1fb0d370af512b26e7431f49fe0331d3b049d2337cd842ad5888ceeae4fce": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  32m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3d15c5de1a83b96d040918a58eab7ccd91681b5618de43e7a45b4a39b065bc1b": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  32m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c73de045cfa9c2b3cf71fefa0caab64568801c590780cae21cb5e4edadec5c56": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  32m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4c01695896578e047787abbdfd4caa7fbfe75f69ba0eb762bcb43c169eff7bd7": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  31m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e882d2dbdb9a296584359fec96415e6ad3dfea179063e001b9a43943d4c81b27": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
  Warning  FailedCreatePodSandBox  3m29s (x129 over 31m)   kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b502ebc34f8a587f3128d989af11be4ab21e11b1f6001b95aacb527bccbc7695": error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found