microk8s.reset does not remove resources #3243
Comments
|
Hi @tmpm697 Thanks for reaching out.
If this is not the case, could you clarify the issue a bit more? Thanks. |
|
yes, i did enable that when the mess becomes. eventually I recently tried with v1.23 and same issue but with below output when Then My question is simple: why |
|
Calico is responsible for the networking of pods. It is expected that reset will give you back a clean and working cluster, therefore calico should be running after the reset command finishes. |
|
so what is default service will be enabled by default when you have a fresh microk8s? because from what i can recall before the issue, only when you |
|
Only the calico CNI and its services are available after a fresh install (or after a |
Are those pods, replicaset, deployment, daemonset are default when you have a fresh microk8s? like I mentioned above: |
|
Yes, this is the calico CNI. |
|
I thought a fresh microk8s would have none resources in it? How to remove all resources? |
|
You can remove the CNI with: |
|
I saw some yaml files under |
No. Only the calico CNI is available in a fresh install (or after a microk8s.reset). |
|
Hi, I have a similar issue. I had a cluster with some deployments, among others, vault, prometheus, and cert-manager. I wanted to tear down the cluster by using admin@node1:~$ sudo microk8s reset
Disabling all addons
Disabling addon : core/cert-manager
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/kube-ovn
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/observability
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Restarting cluster
Stopped.
Setting up the CNI
admin@node1:~$ kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/calico-node-bnjn4 1/1 Running 0 61s
kube-system pod/calico-kube-controllers-d8b9b6478-6pdwg 1/1 Running 0 60s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 10d
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 10d
kube-system service/prometheus-stack-kube-prom-kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 9d
default service/cert-manager ClusterIP 10.152.183.62 <none> 9402/TCP 7d9h
default service/cert-manager-webhook ClusterIP 10.152.183.179 <none> 443/TCP 7d9h
kube-system service/prometheus-stack-kube-prom-kube-scheduler ClusterIP None <none> 10259/TCP 2d20h
kube-system service/prometheus-stack-kube-prom-kube-etcd ClusterIP None <none> 2381/TCP 2d20h
kube-system service/prometheus-stack-kube-prom-kube-controller-manager ClusterIP None <none> 10257/TCP 2d20h
kube-system service/prometheus-stack-kube-prom-kube-proxy ClusterIP None <none> 10249/TCP 2d20h
... # more services
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 2m3s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 2m3s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/calico-kube-controllers-d8b9b6478 1 1 1 60s
|
|
Hi @m4rc3l-h3, Looks like we in fact do not clean up any of the services in the cluster https://github.com/ubuntu/microk8s/blob/master/scripts/wrappers/reset.py#L121 Looks like a simple fix. |
|
Update: Hi @neoaggelos Thanks for your feedback, much appreciated. admin@node1:~$ sudo microk8s reset
[sudo] password for admin:
Disabling all addons
Disabling addon : core/cert-manager
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/kube-ovn
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/observability
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Cleaning resources in namespace tigera-operator
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Removing namespace/tigera-operator
Restarting cluster
Stopped.
Setting up the CNI
admin@node1:~$ kubectl get configmaps
NAME DATA AGE
kube-root-ca.crt 1 10d
cert-manager-webhook 0 8d
configmap-ca-certificates-vault 1 47h
release-name-vault-config 1 47h
configmap-vault-agent-prom 1 47h
admin@node1:~$ kubectl get secrets
NAME TYPE DATA AGE
local-ca-root-cert Opaque 2 10d
sh.helm.release.v1.reloader.v1 helm.sh/release.v1 1 10d
cert-manager-webhook-ca Opaque 3 8d
sh.helm.release.v1.cert-manager.v1 helm.sh/release.v1 1 8d
prometheus-stack-kube-prom-admission Opaque 6 8d
prometheus-stack-server-tls kubernetes.io/tls 3 3d15h
vault-server-tls kubernetes.io/tls 3 47h
vault-client-tls kubernetes.io/tls 3 47h
cert-manager-vault-approle Opaque 1 47h
example-com-tls kubernetes.io/tls 3 6d3h
|
|
I did some digging and I think I found the issue in line 235 of the def remove_extra_resources(ns_name):
# Remove all resource types except the standard k8s apiservices themselves
cmd = [KUBECTL, "api-resources", "-o", "name", "--verbs=delete", "--namespaced=true"]
res = run_silently(cmd)
if not res:
return
extra_resources = res.split()
for rs in extra_resources:
if rs.startswith("apiservices"):
continue
cmd = [KUBECTL, "delete", "--all", "-n", ns_name, "--timeout=60s"]
subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)I would change the respective line to cmd = [KUBECTL, "delete", rs, "--all", "-n", ns_name, "--timeout=60s"]As I am new to this, would you please advise on what to do next? Would that be a good starting point to contribute to the project? Would be glad to do so. |
|
Hi @m4rc3l-h3, sorry for the late reply. Indeed, nice catch, this should be a good starting point. Feel free to create a PR with a fix. See the CONTRIBUTING for details, and feel free to ask here if something is not clear. Thanks a lot! |
|
Hi @neoaggelos, no problem and thanks for the reply and the offering to ask if something is unclear. I will keep you updated here. Thank you for your support! |
|
Hi @neoaggelos Thank you again for the information and the opportunity to contribute, at least to some degree e to this great project. PR #3774 was created, please let me know if anything further (e.g., information, unit tests) would help or if I can be of further assistance in other cases. |
Summary
What Should Happen Instead?
All k8s resources should be remove, no exception!
System info
Reproduction Steps
So as you can see above steps, resources weren't removed after
microk8s.resetcommand.I tried to do a clean refresh and re-installed snapd like below:
Surprisingly is that even I've not enabled
dnsyet, after above command and then I've just issued the commandmicrok8s.kubectl get all -A, all calico pods and all of the resources still there (even I removed /var/lib/snapd and reboot the pc)So why resources weren't removed and how to have a clean reset microk8s?
Full
microk8s inspectlogs hereThe text was updated successfully, but these errors were encountered: