Verrazzano Enterprise Container Platform – Back Up and Restore

Docs: Argo CD

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides an Argo CD deployment that automates the deployment of the desired application states in the specified target environments. There are scenarios where you may want to back up your Argo CD data and restore it.

Verrazzano uses Velero to facilitate backing up and restoring Argo CD data.

Velero operator prerequisites

Before proceeding with an Argo CD backup or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore Argo CD, you must first enable Velero.

The following configuration shows you how to enable velero with a prod installation profile.
```
  $ kubectl apply -f -<<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: prod
      components:
        velero:
          enabled: true
EOF
```
After Verrazzano installation is completed, verify that the Velero pods are running in the verrazzano-backup namespace.
```
$ kubectl get pod -n verrazzano-backup
```
```
# Sample of pods running after enabling the velero component
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h
```
Velero requires a secret to communicate with the S3 compatible object store, so we create a backup-secret.txt file, which has the object store credentials.
```
[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>
```
In the namespace verrazzano-backup, create a Kubernetes secret, for example verrazzano-backup-creds.
```
$ kubectl create secret generic -n verrazzano-backup <secret-name> --from-file=<key>=<full_path_to_creds_file>
```
The following is an example:
```
$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt
```
NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.
Create a BackupStorageLocation resource, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example, which uses Oracle Cloud Object Storage. For more information, see Backup Storage Location in the Velero documentation.
```
 $ kubectl apply -f -<<EOF
   apiVersion: velero.io/v1
   kind: BackupStorageLocation
   metadata:
     name: verrazzano-backup-location
     namespace: verrazzano-backup
   spec:
     provider: aws
     objectStorage:
       bucket: example-verrazzano
       prefix: backup-demo
     credential:
       name: verrazzano-backup-creds
       key: cloud
     config:
       region: us-phoenix-1
       s3ForcePathStyle: "true"
       s3Url: https://<myobjstorenamespace>.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```

Argo CD backup using Velero

If you created applications with resources running in different namespaces, other than argocd, then based on the following criteria you can back up and restore Argo CD:

If applications running in different namespaces use persistent volumes, then you can back up the namespace where the applications are running with the PV.
If applications running in different namespaces do not use persistent storage, then:
- Take a backup of all the namespaces where the application is running by specifying the namespaces as a list.
- Take a backup of only the argocd namespace, create all the namespaces of different applications, and then restore from the backup.

The following example shows a sample Velero Backup API resource that you can create to initiate an Argo CD backup.

  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: verrazzano-argocd-backup
    namespace: verrazzano-backup
  spec:
    includedNamespaces:
    - argocd
    defaultVolumesToRestic: false
    storageLocation:  verrazzano-backup-location
EOF

The preceding example backs up the Argo CD components:

The defaultVolumesToRestic is set to false so that Velero ignores the associated PVCs.
If the deployed applications refer to a database or persistent volumes, then you need to manually create a backup.

Argo CD backup status

# To display the status of the backup, run the following command
$ kubectl get backup.velero.io -n verrazzano-backup verrazzano-argocd-backup -o yaml

# Sample output showing just the status portion of the backup
status:
  completionTimestamp: ...
  expiration: ...
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 232
    totalItems: 232
  startTimestamp: ...
  version: 1

Argo CD scheduled backups

Velero supports a Schedule API that is a repeatable request that is sent to the Velero server to perform a backup for a given cron notation. After the Schedule object is created, the Velero server will start the backup process. Then, it will wait for the next valid point in the given cron expression and run the backup process on a repeating basis.

Argo CD restore using Velero

To initiate an Argo CD restore operation, first delete the existing Argo CD running on the system and all related data. Make sure that the data you are deleting is not needed. The restore operation will only restore data from the specified backup, and any additional data since that backup will be destroyed by the deletion.

If you restore Argo CD to a different cluster, create a Velero BackupStorageLocation resource in the new cluster that points to the same backup storage location configured in the original cluster. This ensures that the Velero resources created by the original cluster’s backup are automatically synced to the new cluster. Once the sync completes, you will be able to access the backup from the original cluster on the new cluster. It is recommended to configure the BackupStorageLocation on the new cluster as read-only by setting accessMode to ReadOnly in the BackupStorageLocation spec. This ensures that the backup in the object store is not modified from the new cluster. For more information, see Backup Storage Location in the Velero documentation.

Delete the Argo CD components and the namespace.
```
# These are sample commands to demonstrate the Argo CD restore process
$ kubectl delete sts -l app.kubernetes.io/part-of=argocd -n argocd
$ kubectl delete deploy -l app.kubernetes.io/part-of=argocd -n argocd
$ kubectl delete pvc -n argocd <pvc name>
$ kubectl delete ns argocd
```
To perform an Argo CD restore operation, you can invoke the following example Velero Restore API object.

NOTE: For Argo CD, includedNamespaces should list all the namespaces across which the applications are deployed.
```
 $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: verrazzano-argocd-restore
    namespace: verrazzano-backup
  spec:
    backupName: verrazzano-argocd-backup
    includedNamespaces:
    - argocd
EOF
```
The preceding example will restore an Argo CD cluster from an existing backup.
Wait for all the Argo CD pods to be in the RUNNING state.
```
  $ kubectl wait -n argocd --for=condition=ready pod -l app.kubernetes.io/instance=argocd
```
Sample output:
```
    pod/argocd-application-controller-0 condition met
    pod/argocd-applicationset-controller-8489bfbb8-4f686 condition met
    pod/argocd-notifications-controller-c4f5c9684-8qzl8 condition met
    pod/argocd-redis-548968fdd9-4jcrf condition met
    pod/argocd-repo-server-5889c8cc68-5n8j6 condition met
    pod/argocd-server-67b6994987-j9z99 condition met
```

Argo CD restore status

# To display the status of the restore, run the following command
$ kubectl get restore -n verrazzano-backup verrazzano-argocd-restore -o yaml

# Sample output showing just the status portion of the restore
status:
  completionTimestamp: ...
  phase: Completed
  progress:
    itemsRestored: 75
    totalItems: 75
  startTimestamp: ...

Docs: Keycloak

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano stores user login information in Keycloak. In turn, Keycloak uses MySQL as a back end to store all persistent data. This document shows you how to back up persistent data stored in MySQL from the original cluster and restore it in a new cluster. If you are restoring data to the same cluster, then the terms original cluster and new cluster refer to same cluster.

MySQL Operator prerequisites

MySQL is deployed using the MySQL Operator for Kubernetes. Apart from managing the life cycle of MySQL instances, MySQL Operator provides the capability to back up and restore data using an Amazon S3 compatible object storage.

Before proceeding with a MySQL back up or restore operation, keep the following details handy:

Object storage bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object storage solution.
Object storage prefix name. This will be a child folder under the bucket, which the backup component creates.
Object storage region name.
Object storage signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object storage; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure (OCI), you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

The following example creates a secret mysql-backup-secret in the namespace keycloak. The instructions in this document back up data from MySQL to an Oracle Cloud Object Storage bucket and restore it from there.

MySQL Operator requires a secret to communicate with the S3 compatible object storage, so we create a backup-secret.txt file, which has the object storage credentials.
```
[default]
aws_access_key_id=<object storage access key>
aws_secret_access_key=<object storage secret key>
```
MySQL Operator requires the region name where the bucket is created, so we create a backup-region.txt file, which contains the region information. The following is an example of a backup-region.txt file indicating that the object storage is created in region us-phoenix-1:
```
[default]
region=us-phoenix-1
```
In the namespace keycloak, create a Kubernetes secret, for example mysql-backup-secret.
```
$ kubectl create secret generic --namespace <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file> --from-file=<key>=<full_path_to_config_file>
```
The following is an example to create a Kubernetes secret consisting of credentials to connect to OCI Object Storage.
```
$ kubectl create secret generic --namespace keycloak mysql-backup-secret --from-file=credentials=backup-secret.txt --from-file=config=backup-region.txt
```

NOTE

The secret must be created in the namespace keycloak.
To restore Keycloak on a new cluster, create the secret in the namespace, keycloak, in the new cluster.
To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.

MySQL Operator backup

To initiate a MySQL backup on the original cluster, create the following example custom resource YAML file that uses an OCI Object Storage as a back end. The operator uses the secret referenced in spec.backupProfile.dumpInstance.storage.s3.config to authenticate with the OCI Object Storage.
```
$ kubectl apply -f - <<EOF
    apiVersion: mysql.oracle.com/v2
    kind: MySQLBackup
    metadata:
      name: <backup name>
      namespace: keycloak
    spec:
      clusterName: mysql
      backupProfile:
        name: <backupProfileName>
        dumpInstance:
          storage:
            s3:
              bucketName: <The Object Storage bucket. See the MySQL Operator prerequisites section.>
              config: <Kubernetes secret name. See the MySQL Operator prerequisites section.>
              endpoint: < OCI S3 Object Storage endpoint.>
              prefix: <The prefix name. This folder will be automatically created.>
              profile: default
EOF
```
NOTE
- The config value is mysql-backup-secret, which is the name of the secret that you created previously in the keycloak namespace.
- The clustername has to be mysql.
- The namespace has to be keycloak.
- The profile value is the profile for the security credentials. In this case, it is default.
The following is an example of a MySQLBackup resource to initiate a MySQL backup:
```
$ kubectl apply -f - <<EOF
    apiVersion: mysql.oracle.com/v2
    kind: MySQLBackup
    metadata:
      name: mysql-backup
      namespace: keycloak
    spec:
      clusterName: mysql
      backupProfile:
        name: mysqlOneTime
        dumpInstance:
          storage:
            s3:
              bucketName: mysql-bucket
              config: mysql-backup-secret
              endpoint: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
              prefix: mysql-test
              profile: default
EOF
```
Confirm that the backup operation is complete. Run the following command on the original cluster and ensure that the STATUS is Completed.
```
$ kubectl get MySQLBackup --namespace keycloak
```
```
# Sample output
NAME           CLUSTER   STATUS      OUTPUT                         AGE
mysql-backup   mysql     Completed   mysql-backup-20221025-180836   119s
```
A successful backup of MySQL creates a backup folder in the object storage. Make note of the backup folder prefix name that the MySQL backup created on the original cluster.
```
$ kubectl get mysqlbackup --namespace keycloak <mysql-backup-name> -o jsonpath={.status.output}
```
The following is an example:
```
$ kubectl get mysqlbackup --namespace keycloak mysql-backup -o jsonpath={.status.output}
mysql-backup-20221025-180836
```
Back up MySQL Helm chart and values.

Back up the values in the MySQL Helm chart, in the original cluster to a file, mysql-values.yaml .
```
$ helm get values --namespace keycloak mysql > mysql-values.yaml
```
MySQL Helm charts are present inside the Verrazzano platform operator. Retrieve the charts from the original cluster to a local directory.

The following example retrieves the MySQL charts to a directory mysql-charts under the current directory. In order to avoid data corruption, ensure that the directory, mysql-charts, doesn’t already exist under the current directory.
```
$ kubectl cp --namespace verrazzano-install \
    $(kubectl get pod --namespace verrazzano-install -l app=verrazzano-platform-operator \
    -o custom-columns=:metadata.name --no-headers):platform-operator/thirdparty/charts/mysql \
    -c verrazzano-platform-operator mysql-charts/
```

Scheduled backups

You can also implement schedules for running MYSQL backups. For more information, see the Handling MySQL Backups section, “A PersistentVolumeClaim Scheduled Backup Example.”

MySQL Operator restore

Before you begin, read the MySQL Operator prerequisites. In addition, you must have at least one healthy backup before starting a restore operation.

To initiate a MySQL restore operation from an existing backup, you need to recreate the MySQL cluster. Use the following steps for a successful MySQL restore operation:

Delete the MySQL pods and PersistentVolumeClaim from the system on the new cluster.
```
$ helm delete mysql --namespace keycloak
$ kubectl delete pvc --namespace keycloak -l tier=mysql
```
Start a MySQL restore operation by installing the Helm chart by using the chart from the original cluster.
```
 $ helm install mysql <path to directory mysql-charts, where original charts are extracted> \
         --namespace keycloak \
         --set initDB.dump.name=<dump-name> \
         --set initDB.dumpOptions.loadUsers=true \
         --set initDB.dump.s3.profile=default \
         --set initDB.dump.s3.prefix=<prefixName/backup folder name> \
         --set initDB.dump.s3.bucketName=<OCI bucket name> \
         --set initDB.dump.s3.config=<Kubernetes secret name, see MySQL Operator prerequisites section.> \
         --set initDB.dump.s3.endpoint=<OCI S3 endpoint> \
         --values <mysql values file>
```
The following is an example:
```
 $ helm install mysql mysql-charts \
         --namespace keycloak \
         --set initDB.dump.name=alpha \
         --set initDB.dump.s3.profile=default \
         --set initDB.dump.s3.prefix=mysql-test/mysql-backup-20221025-180836 \
         --set initDB.dump.s3.bucketName=mysql-bucket \
         --set initDB.dump.s3.config=mysql-backup-secret \
         --set initDB.dump.s3.endpoint=https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com \
         --values mysql-values.yaml
```
After performing the restore command, wait for the MySQL cluster to be online. Ensure that the STATUS is ONLINE and the count under ONLINE matches the INSTANCES.
```
$ kubectl get innodbclusters --namespace keycloak mysql
```
```
# Sample output
NAME    STATUS   ONLINE   INSTANCES   ROUTERS   AGE
mysql   ONLINE   3        3           3         2m23s
```
Wait for all the MySQL pods to be in the RUNNING state.
```
$ kubectl wait --namespace keycloak --for=condition=ready pod -l tier=mysql --timeout=600s
```
```
# Sample output
pod/mysql-0 condition met
pod/mysql-1 condition met
pod/mysql-2 condition met
pod/mysql-router-746d9d75c7-6pc5p condition met
pod/mysql-router-746d9d75c7-bhrkw condition met
pod/mysql-router-746d9d75c7-t8bhb condition met
```
At this point, the MySQL cluster has been restored successfully from the backup, along with the PersistentVolumeClaim that was deleted previously.
If you are restoring Keycloak on a new cluster, then update the Keycloak secret.

On the original cluster, if you are restoring Keycloak on a new cluster, then run the following command for the keycloak-http secret in keycloak namespace:
```
 $ kubectl get secret --namespace keycloak keycloak-http -o jsonpath={.data.password}; echo
```
On the new cluster, replace the existing password value with the value displayed from the previous command.
```
 kubectl patch secret keycloak-http --namespace keycloak -p '{"data": {"password": "<password displayed in the step above>"}}'
```
Restart the Keycloak pods.

The removal and recreation of the MySQL cluster may bring down the Keycloak pods because MySQL goes offline during the restore operation. Run the following commands to restart the Keycloak pods:
```
 KEYCLOAK_REPLICAS=$(kubectl get sts --namespace keycloak keycloak -o custom-columns=:status.replicas --no-headers)
 kubectl scale sts --namespace keycloak keycloak --replicas=0
 kubectl scale sts --namespace keycloak keycloak --replicas=${KEYCLOAK_REPLICAS}
 kubectl wait --namespace keycloak --for=condition=ready pod -l app.kubernetes.io/instance=keycloak --timeout=600s
```

Update Verrazzano secrets in the new cluster

The following steps are applicable only if you are restoring Keycloak on a new cluster.

After you complete the MySQL restore operation, the password for the following secrets in the verrazzano-system namespace must be updated in the new cluster:

verrazzano
verrazzano-es-internal
verrazzano-prom-internal

On the original cluster, run the following command for the verrazzano secret:
```
 $ kubectl get secret --namespace verrazzano-system verrazzano -o jsonpath={.data.password}; echo
```
On the new cluster, replace the existing password value with the value displayed in step 1.
```
 kubectl patch secret verrazzano --namespace verrazzano-system -p '{"data": {"password": "<password displayed in step 1>"}}'
```
Repeat steps 1 and 2 for the verrazzano-es-internal and verrazzano-prom-internal secrets.
Restart the fluentd pods in the new cluster to use the original cluster password to connect to OpenSearch.
```
 $ kubectl delete pod -l app=fluentd --namespace verrazzano-system
```

Docs: OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides a ready-to-use, OpenSearch deployment that gives you access to all the log messages from various microservices running on the platform. There are scenarios where you may want to back up your OpenSearch data and restore it.

Verrazzano uses Velero to facilitate backing up and restoring OpenSearch data.

Velero operator prerequisites

Before proceeding with an OpenSearch backup or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore OpenSearch, you must first enable Velero.

The following configuration shows you how to enable velero with a prod installation profile.
```
  $ kubectl apply -f -<<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: prod
      components:    
        velero:
          enabled: true
EOF
```

NOTE: The OpenSearch back up and restore operation is supported only on prod installation profiles with a multinode OpenSearch configuration.

After Velero is enabled, verify that the Velero pods are running in the verrazzano-backup namespace.
```
# Sample of pods running after enabling the velero component
$ kubectl get pod -n verrazzano-backup
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h
```
Velero requires a secret to communicate with the S3 compatible object store, so we create a backup-secret.txt file, which has the object store credentials.
```
[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>
```
In the namespace verrazzano-backup, create a Kubernetes secret, for example verrazzano-backup-creds.
```
$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file>
```
The following is an example:
```
$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt
```
NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.
Create a BackupStorageLocation resource, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example. For more information, see Backup Storage Location in the Velero documentation.
```
 $ kubectl apply -f -<<EOF
 apiVersion: velero.io/v1
 kind: BackupStorageLocation
 metadata:
   name: verrazzano-backup-location
   namespace: verrazzano-backup
 spec:
   provider: aws
   objectStorage:
     bucket: example-verrazzano
     prefix: backup-demo
   credential:
     name: verrazzano-backup-creds
     key: cloud
   config:
     region: us-phoenix-1
     s3ForcePathStyle: "true"
     s3Url: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```

OpenSearch backup Using Velero

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero while invoking a backup. Due to the nature of transient data handled by OpenSearch, the hook invokes the OpenSearch snapshot APIs to back up data streams appropriately, thereby ensuring that there is no loss of data and avoids data corruption as well.

The following example shows a sample Velero Backup API resource that you can create to initiate an OpenSearch backup.

  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: verrazzano-opensearch-backup
    namespace: verrazzano-backup
  spec:
    includedNamespaces:
      - verrazzano-system
    labelSelector:
      matchLabels:
        verrazzano-component: opensearch
    defaultVolumesToRestic: false
    storageLocation:  verrazzano-backup-location
    hooks:
      resources:
        - name: opensearch-backup-test
          includedNamespaces:
            - verrazzano-system
          labelSelector:
            matchLabels:
              statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
          post:                           
            - exec:
                container: es-master
                command:
                  - /usr/share/opensearch/bin/verrazzano-backup-hook
                  - -operation
                  - backup
                  - -velero-backup-name
                  - verrazzano-opensearch-backup
                onError: Fail
                timeout: 10m
EOF

The preceding example backs up the OpenSearch components:

In this case, you are not backing up the PersistentVolumes directly, rather running a hook that invokes the OpenSearch APIs to take a snapshot of the data.
The defaultVolumesToRestic is set to false so that Velero ignores the associated PVCs.
In this case, the hook can be pre or post.
The command used in the hook requires an operation flag and the Velero backup name as an input.
The container on which the hook needs to be run defaults to the first container in the pod. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

After the backup is processed, you can see the hook logs using the velero backup logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch backup logs

# To display the logs from the backup, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-backup-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

OpenSearch scheduled backups

OpenSearch restore using Velero

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero to perform a restore operation. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to restore data streams appropriately, thereby ensuring there is no loss of data and avoids data corruption as well.

If you restore OpenSearch to a different cluster, create a Velero BackupStorageLocation resource in the new cluster that points to the same backup storage location configured in the original cluster. This ensures that the Velero resources created by the original cluster’s backup are automatically synced to the new cluster. Once the sync completes, you will be able to access the backup from the original cluster on the new cluster. It is recommended to configure the BackupStorageLocation on the new cluster as read-only by setting accessMode to ReadOnly in the BackupStorageLocation spec. This ensures that the backup in the object store is not modified from the new cluster. For more information, see Backup Storage Location in the Velero documentation.

To initiate an OpenSearch restore operation, first delete the existing OpenSearch cluster running on the system and all related data.

Scale down the Verrazzano Monitoring Operator. This is required because the operator manages the life cycle of the OpenSearch cluster, so scaling it down to zero ensures that it does not interfere with the restore operation. The restore operation also ensures that this operator is scaled back up to return the system to its previous state.
```
 $ kubectl scale deploy -n verrazzano-system verrazzano-monitoring-operator --replicas=0
```
Delete the OpenSearch components.
```
# These are sample commands to demonstrate the OpenSearch restore process
$ kubectl delete sts -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete deploy -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete pvc -n verrazzano-system -l verrazzano-component=opensearch
```
To perform an OpenSearch restore operation, you can invoke the following example Velero Restore API object.
```
  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: verrazzano-opensearch-restore
    namespace: verrazzano-backup
  spec:
    backupName: verrazzano-opensearch-backup
    includedNamespaces:
      - verrazzano-system
    labelSelector:
      matchLabels:
        verrazzano-component: opensearch
    restorePVs: false
    hooks:
      resources:
      - name: opensearch-test
        includedNamespaces:
        - verrazzano-system       
        labelSelector:
          matchLabels:            
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        postHooks:
        - exec:
            container: es-master
            command:
            - /usr/share/opensearch/bin/verrazzano-backup-hook
            - -operation
            - restore
            - -velero-backup-name
            - verrazzano-opensearch-backup
            waitTimeout: 30m
            execTimeout: 30m
            onError: Fail
EOF
```
The preceding example will restore an OpenSearch cluster from an existing backup.
- In this case, you are not restoring PersistentVolumes directly, rather running a hook that invokes the OpenSearch APIs to restore them from an existing snapshot of the data.
- The restorePVs is set to false so that Velero ignores restoring PVCs.
- The command used in the hook requires an -operation flag and the Velero backup name as an input.
- The postHook will invoke the OpenSearch APIs that restore the snapshot data.
- The container on which the hook needs to be run defaults to the first container in the pod. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.
NOTE: The hook needs to be a postHook because it must be applied after the Kubernetes objects are restored.
Wait for all the OpenSearch pods to be in the RUNNING state.
```
  $ kubectl wait -n verrazzano-system --for=condition=ready pod -l verrazzano-component=opensearch --timeout=600s
    pod/vmi-system-es-data-0-6f49bdf6f5-fc6mz condition met
    pod/vmi-system-es-data-1-8f8785994-4pr7n condition met
    pod/vmi-system-es-data-2-d5f569d98-q8p2v condition met
    pod/vmi-system-es-ingest-6ddd86b9b6-fpl6j condition met
    pod/vmi-system-es-ingest-6ddd86b9b6-jtmrh condition met
    pod/vmi-system-es-master-0 condition met
    pod/vmi-system-es-master-1 condition met
    pod/vmi-system-es-master-2 condition met
```

After the restore operation is processed, you can see the hook logs using the velero restore logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch restore logs

# To display the logs from the restore, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-restore-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

OpenSearch restore in an existing cluster using OpenSearch API

OpenSearch restore using Velero typically is used for disaster recovery scenarios where you need to restore the entire cluster. But, if you want to restore the OpenSearch data within an already existing cluster, then you can use the OpenSearch API.

Assuming that you have previously created a backup using Velero, within the same cluster that would have completed the repository registration, and want to solely restore the OpenSearch data from that specific backup, run the following commands.

To get the registered repositories:

# To see all snapshot repositories
$ GET _snapshot/_all

To get all the snapshots in the registered repository:

# To see all snapshots in a repository
$ GET _snapshot/<backup_repository_name>/_all

To restore the specific snapshot:

# To restore the existing snapshot, run the following command
$ POST _snapshot/<backup_repository_name>/<snapshot-name>/_restore

<backup_repository_name>: The name of the backup repository where your OpenSearch snapshot is stored. Replace <backup_repository_name> with the name of your backup repository.
<snapshot-name>: The name of the OpenSearch snapshot you want to restore. Replace <snapshot-name> with the name of your snapshot.

For more information, see OpenSearch Restore in the OpenSearch documentation.

Docs: Rancher

Mon, 01 Jan 0001 00:00:00 +0000

Rancher maintains many configurations, like user credentials and cluster credentials, as ConfigMaps and namespace values. The rancher-backup Operator provides a seamless way to back up and restore Rancher installations, configuration, and data.

rancher-backup Operator prerequisites

Before proceeding with a Rancher back up or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore Rancher, you must first enable rancherBackup.

The following configuration shows you how to enable rancherBackup.
```
$ kubectl apply -f -<<EOF
 apiVersion: install.verrazzano.io/v1beta1
  kind: Verrazzano
 metadata:
    name: example-verrazzano
  spec:
   profile: dev
    components:    
     rancherBackup:
        enabled: true
EOF
```
For rancher-backup, the pods will be created in the cattle-resources-system namespace.
```
  # Sample of pods running after enabling the rancherBackup component

  $ kubectl get pod -n cattle-resources-system
  NAME                              READY   STATUS    RESTARTS   AGE
  rancher-backup-5c4b985697-xw7md   1/1     Running   0          2d4h
```
Rancher requires a secret to communicate with the S3 compatible object store. So, in the namespace verrazzano-backup, create a Kubernetes secret rancher-backup-creds.
```
 $ kubectl create secret generic -n <backup-namespace> <secret-name> --from-literal=accessKey=<accesskey> --from-literal=secretKey=<secretKey>
```
The following is an example:
```
 $ kubectl create secret generic -n verrazzano-backup rancher-backup-creds --from-literal=accessKey="s5VLpXwa0xNZQds4UTVV" --from-literal=secretKey="nFFpvyxpQvb0dIQovsl0"
```

Rancher backup

The rancher-backup Operator creates the backup file, in *.tar.gz format, on the S3 compatible object store.

To initiate a Rancher backup, create the following example custom resource YAML file that uses an Amazon S3 compatible object store as a back end. The operator uses the credentialSecretNamespace value to determine where to look for the Amazon S3 backup secret.
```
 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Backup
   metadata:
     name: <rancher-backup-name>
   spec:
     storageLocation:
       s3:
         credentialSecretName: <rancher backup credential name>
         credentialSecretNamespace: <namespace where credential object was created>
         bucketName: <object store bucket. This must be exist as noted in pre-requisites section>
         folder: <folder name. This folder will be auto created>
         region: <region name where bucket exists>
         endpoint: <object store endpoint configuration>
     resourceSetName: rancher-resource-set
EOF
```

NOTE: In Step 3. of the example in the prerequisites section, you created the secret in the verrazzano-backup namespace.

The following is an example:

 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Backup
   metadata:
     name: rancher-backup-test
   spec:
      storageLocation:
       s3:
         credentialSecretName: rancher-backup-creds
         credentialSecretNamespace: verrazzano-backup
         bucketName: myvz-bucket
         folder: rancher-backup
         region: us-phoenix-1
         endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
     resourceSetName: rancher-resource-set
EOF

The *.tar.gz file is stored in a location configured in the storageLocation field. When the backup is complete, then the rancher-backup Operator creates a file on the S3 compatible object store.

You can retrieve the backed up file name, as shown:
```
 $ kubectl get backups.resources.cattle.io rancher-backup-test
   NAME                 LOCATION   TYPE       LATEST-BACKUP                                                                     RESOURCESET            AGE   STATUS
   rancher-backup-test             One-time   rancher-615034-957d182d-44cb-4b81-bbe0-466900049124-2022-11-14T16-42-28Z.tar.gz   rancher-resource-set   54s   Completed
```

Rancher scheduled backups

To implement scheduled Rancher backups, see Backup Configuration in the Rancher documentation.

Rancher restore

During the restore operation, Rancher ensures that it recreates all the CRDs related to Rancher and configurations. Restoring Rancher is done by creating a custom resource that indicates to rancherBackup to start the restore process.

To initiate a Rancher restore operation, create the following example custom resource YAML file. When a Restore custom resource is created, the operator accesses the backup *.tar.gz file specified and restores the application data from that file.
```
 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Restore
   metadata:
     name: s3-restore
   spec:
     backupFilename: rancher-615034-957d182d-44cb-4b81-bbe0-466900049124-2022-11-14T16-42-28Z.tar.gz
     storageLocation:
       s3:
         credentialSecretName: rancher-backup-creds
         credentialSecretNamespace: verrazzano-backup
         bucketName: myvz-bucket
         folder: rancher-backup
         region: us-phoenix-1
         endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```
The rancher-backup Operator scales down the Rancher deployment during the restore operation and scales it back up after the restoration completes.

Resources are restored in this order:
1. Custom Resource Definitions (CRDs)
2. Cluster-scoped resources
3. Namespace resources
NOTE: The backupFilename is retrieved from the Rancher backup created previously.
Wait for all the Rancher pods to be in the RUNNING state.
```
$ kubectl wait -n cattle-system --for=condition=ready pod -l app=rancher --timeout=600s
  pod/rancher-69976cffc6-bbx4p condition met
  pod/rancher-69976cffc6-fr75t condition met
  pod/rancher-69976cffc6-pcdf2 condition met
```