Backup & Restore data in Kubernetes

Backing up your kubernetes cluster data by backuping up etcd is of crucial importance. etcd is the backend storage solution for the deployed Kubernetes cluster. All the K8s objects, applications & configuration are stored in etcd.

1. Backing up ‘etcd’ data is done using etcd command line tool: etcdctl

ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save <filename>

2. Restoring etcd can be done from a backup using the etcdctl snapshot restore command:

ETCDCTL_API=3 etcdctl snapshot restore <filename>

Lets have a handson demo session on the above operations. Steps are as follows:

1) Execute etcdctl command to check Cluster name. Lookup for the cluster.name in the etcd cluster:

ETCDCTL_API=3 etcdctl get cluster.name \
>   --endpoints=https://10.0.1.101:2379 \
>   --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
>   --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
>   --key=/home/cloud_user/etcd-certs/etcd-server.key
cluster.name
beebox

The returned value for the Cluster.name is beebox

arguments:
–endpoint -> instructing etcdctl how to reach out to etcd server
–cacert -> public certificate for certificate authority
–cert -> the client certificate
–key -> certificate key

2) Snapshot save will save the backup with the name supplied as argument:

ETCDCTL_API=3 etcdctl snapshot save /home/cloud_user/etcd_backup.db \
>   --endpoints=https://10.0.1.101:2379 \
>   --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
>   --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
>   --key=/home/cloud_user/etcd-certs/etcd-server.key

3) Now, stop etcd, Delete etcd data dir. Reset etcd by removing all the existing etcd data:

4) Snapshot restore on the etcd data will restore the data back as a temporary etcd cluster. This command will spinup a temp etcd cluster, saving the data from the backup file to a new data dir in the same location where the previous data dir was:

sudo ETCDCTL_API=3 etcdctl snapshot restore /home/cloud_user/etcd_backup.db \
>   --initial-cluster etcd-restore=https://10.0.1.101:2380 \
>   --initial-advertise-peer-urls https://10.0.1.101:2380 \
>   --name etcd-restore \
>   --data-dir /var/lib/etcd

arguements:
Cluster with etcd restore
Required settings for temp cluster
Temp name as ‘etcd-restore’
Data directory to be used

5) Since I executed the operation as Root, I will have to make etcd as the user.
Set ownership on the new directory & start etcd:

sudo chown -R etcd:etcd /var/lib/etcd
sudo systemctl start etcd

6) Verify the restored data by looking up the value for the key cluster.name again:

ETCDCTL_API=3 etcdctl get cluster.name \
>   --endpoints=https://10.0.1.101:2379 \
>   --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
>   --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
>   --key=/home/cloud_user/etcd-certs/etcd-server.key
cluster.name
beebox

At this point, I can say with confidence that our etcd database is properly restored.
With this demonstration, we come to the end of topic: “High availability in Kubernetes”.