Velero

https://github.com/vmware-tanzu/velero

Overview

Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. You can run Velero with a cloud provider or on-premises. Velero lets you:

• Take backups of your cluster and restore in case of loss.
• Migrate cluster resources to other clusters.
• Replicate your production cluster to development and testing clusters.

Velero consists of:

• A server that runs on your cluster
• A command-line client that runs locally

In the recent 1.4 release of Velero, we announced a new feature of supporting CSI snapshotting using the Kubernetes CSI Snapshot Beta APIs. With this capability of CSI volume snapshotting, Velero can now support any volume provider that has a CSI driver with snapshotting capability, without requiring a Velero-specific volume snapshotter plugin to be available.

This post has the necessary instructions for you to start using this feature.

Getting Started

Using the CSI volume snapshotting features in Velero involves the following steps.

1. Set up a CSI environment with a driver supporting the Kubernetes CSI snapshot beta APIs.
2. Install Velero with CSI snapshotting feature enabled.
3. Deploy csi-app: a stateful application that uses CSI backed volumes that we will backup and restore.
4. Use Velero to backup and restore the csi-app.

Set up a CSI environment.

As the Kubernetes CSI Snapshot Beta API is available starting from Kubernetes 1.17, you need to run Kubernetes 1.17 or later.

This post uses an AKS cluster running Kubernetes 1.17, with Azure disk CSI driver as an example.

Following instructions to install the Azure disk CSI driver from here run the below command

curl -skSL https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/install-driver.sh | bash -s master snapshot -- 

This script will deploy the following CSI components, CRDs, and necessary RBAC:

• deployment.apps/csi-azuredisk-controller
• daemonset.apps/csi-azuredisk-node
• deployment.apps/csi-snapshot-controller

Install Velero with CSI support enabled

The CSI volume snapshot capability is currently, as of Velero 1.4, a beta feature behind the EnableCSI feature flag and is not enabled by default.

Following instructions from our docs website, install Velero with the velero-plugin-for-csi and using the Azure Blob Store as our BackupStorageLocation. Please refer to our velero-plugin-for-microsoft-azure documentation for instructions on setting up the BackupStorageLocation. Please note that the BackupStorageLocation should be set up before installing Velero.

Install Velero by running the below command

velero install \ 
--provider azure \ 
--plugins velero/velero-plugin-for-microsoft-azure:v1.1.0,velero/velero-plugin-for-csi:v0.1.1 \ 
--bucket $BLOB_CONTAINER \ 
--secret-file <PATH_TO_CREDS_FILE>/aks-creds \ 
--backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \ 
--snapshot-location-config apiTimeout=5m,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID \ 
--image velero/velero:v1.4.0 \ 
--features=EnableCSI 

Deploy a stateful application with CSI backed volumes

Before installing the stateful application with CSI backed volumes, install the storage class and the volume snapshot class for the Azure disk CSI driver by applying the below yaml to our cluster.

apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata: 
  name: disk.csi.azure.com 
provisioner: disk.csi.azure.com 
parameters: 
  skuname: StandardSSD_LRS 
reclaimPolicy: Delete 
volumeBindingMode: Immediate 
allowVolumeExpansion: true
---
apiVersion: snapshot.storage.k8s.io/v1beta1 
kind: VolumeSnapshotClass 
metadata: 
  name: csi-azuredisk-vsc 
driver: disk.csi.azure.com 
deletionPolicy: Retain 
parameters:
  tags: 'foo=aaa,bar=bbb' 

NOTE: The above yaml was sourced from StorageClass and VolumeSnapshotClass.

Deploy the stateful application that is using CSI backed PVCs, in the csi-app namespace by applying the below yaml.

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: csi-app
---
kind: Pod
apiVersion: v1
metadata:
  namespace: csi-app
  name: csi-nginx
spec:
  nodeSelector:
    kubernetes.io/os: linux
  containers:
    - image: nginx
      name: nginx
      command: [ "sleep", "1000000" ]
      volumeMounts:
        - name: azuredisk01
          mountPath: "/mnt/azuredisk"
  volumes:
    - name: azuredisk01
      persistentVolumeClaim:
        claimName: pvc-azuredisk
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: csi-app
  name: pvc-azuredisk
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: disk.csi.azure.com
---

For demonstration purposes, instead of relying on the application writing data to the mounted CSI volume, exec into the pod running the stateful application to write data into /mnt/azuredisk, where the CSI volume is mounted. This is to let us get a consistent checksum value of the data and verify that the data on restore is exactly same as that in the backup.

$ kubectl -n csi-app exec -ti csi-nginx bash 
root@csi-nginx:/# while true; do echo -n "FOOBARBAZ " >> /mnt/azuredisk/foobar; done 
^C 
root@csi-nginx:/# cksum /mnt/azuredisk/foobar 
2279846381 1726530 /mnt/azuredisk/foobar 

Back Up

Back up the csi-app namespace by running the below command

$ velero backup create csi-b2 --include-namespaces csi-app --wait 
Backup request "csi-b2" submitted successfully. 
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background. 
.................. 
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe csi-b2` and `velero backup logs csi-b2`.

Restore

Before restoring from the backup simulate a disaster by running

kubectl delete ns csi-app

Once the namespace has been deleted, restore the csi-app from the backup csi-b2.

$ velero create restore --from-backup csi-b2 --wait 
Restore request "csi-b2-20200518085136" submitted successfully. 
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background. 
.... 
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe csi-b2-20200518085136` and `velero restore logs csi-b2-20200518085136`. 

Now that the restore has completed and our csi-nginx pod is Running, confirm that contents of /mnt/azuredisk/foobar have been correctly restored.

$ kubectl -n csi-app exec -ti csi-nginx bash 
root@csi-nginx:/# cksum /mnt/azuredisk/foobar 
2279846381 1726530 /mnt/azuredisk/foobar 
root@csi-nginx:/# 

The stateful application that we deployed has been successfully restored with its data intact. And that’s all it takes to backup and restore a stateful application that uses CSI backed volumes!

Community Engagement

Please try out the CSI support in Velero 1.4. Feature requests, suggestions, bug reports, PRs are all welcome.

Get in touch with us on Kubernetes Slack #velero, Twitter, or Our weekly community calls

Resources

More details about CSI volume snapshotting and its support in Velero may be found in the following links:

• Kubernetes 1.17 Feature: Kubernetes Volume Snapshot Moves to Beta for more information on the CSI beta snapshot APIs.
• Prerequisites to use this feature is available on our website.
• Kubernetes CSI docs: To understand components in a CSI environment
• Velero plugin for CSI snapshots for implementation details of the CSI plugin.