Restore a backup

You can restore an ArcGIS Enterprise on Kubernetes organization from a backup using ArcGIS Enterprise Manager. All configuration data, settings, services, and infrastructure objects will be restored. Any services that reference data in registered data stores will be re-created.

If your organization has configured services using volumes through the Admin API and there have been changes in the volumes since the backup was created, you must patch any related persistent volumes (PVs) in the Released state to make them Available before running the restore. If not completed, services may fail to refresh and will need to be restarted once the PVC binds to the existing PV. To patch the associated PVs, run the following command:

kubectl patch pv <pv name> -p '{"spec": {"claimRef": {"uid": null}}}'

The way in which a backup is restored depends on whether there is an existing organization that is compatible with the backup you want to restore.

Restore a backup using the in-place process

The in-place restore process can be used to restore an organization from a backup without undeploying and redeploying ArcGIS Enterprise on Kubernetes. You can use the in-place restore process in the event of data corruption or data loss, as long as the backup is compatible with your existing organization. The original organization will maintain a record of all backups, and you can restore from one of them using ArcGIS Enterprise Manager.

To restore a backup using the in-place process, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator of your organization.
  2. Click the Backups button.
  3. From the backups page, determine which backup you want to restore to and click the options button.
  4. Click Restore.
  5. Provide the encryption pass phrase that was used to create the backup.
  6. Click Restore.

When the restore operation begins, a job is created to manage the operation, and the restore page will appear and show the progress of the restore operation. The organization will be restored to the point in time when the backup was created and will be inaccessible until the restore is complete.

In the in-place restore process, pod placement policies for the object, relational, and spatiotemporal data stores are retained from the current configuration and ignored from the backup. This avoids restarting the data store StatefulSets as well as potential conflicts with the current cluster configuration during the restore operation. For system, utility, and user-published services, if the deployment does not refresh successfully, the deployment ID will be recorded in the results, and the operation will report that it completed successfully with warnings. An administrator should correct the placement policy for each following the restore operation.

Restore a backup using the out-of-place process

The out-of-place restore process can be used to restore a backup when the original organization is not accessible, or when the backup was taken at a different version or with a different relational store type than your current organization.

Before using the out-of-place restore process, undeploy your existing organization and redeploy a new organization. Ensure that the original persistent volume (PV) contains labels that can be used as part of a label selector when registering a new backup store. You must also ensure the following settings are identical across your source and target environments:

  • Storage type for the relational store (either persistent-volume or cloud service)
  • ArcGIS Enterprise on Kubernetes version
  • Fully Qualified Domain Name (FQDN) and Context Path (that is, https://dnsalias.domain.com/context)
  • Kubernetes Namespace (that is, arcgis)
  • Kubernetes Cluster Domain (that is, cluster.local)
  • Kubernetes Service DNS Suffix (that is, svc.cluster.local)
  • FSGroup and Supplemental Group ID (if deployed using a custom value)

Note:

These settings were specified during deployment.

Prepare an existing PV for static binding

After you undeploy ArcGIS Enterprise on Kubernetes, the original PV's status will be set as Released. The existing PV should not be patched until the new organization is created. Once the new organization is available, you will need to remove the PV's claimRef to allow for binding using label selectors. To add the appropriate labels and ensure that the PV is available to bind with the new organization's PVC, complete the following steps:

  1. Use kubectl to identify the PV that was used for the previous backup store.
    kubectl get pv
    
  2. Use kubectl to get the PV specifications.
    kubectl get pv <PV> -o yaml
    

    These values can be used as a reference when registering the backup store.

    Note:

    If you have forgotten the previous backup store name, you can retrieve it by deconstructing the PV's spec.claimRef.name. For example, if the PV's spec.claimRef.name is data-volume-arcgis-backup-store-backups-111-backups-main-0, the previous backup store name is backups-111.

  3. Use kubectl to assign a label to the PV.

    This is used to bind the PV to the new backup store pod.

    kubectl label pv <pv name> <key>=<value>
    

    For example, use "arcgis/purpose":"backups" as the label selector:

    kubectl label pv <pv name> arcgis/purpose=backups
    
  4. Patch the PV to remove the uid value from the claimRef field. This is to ensure it can be bound to the new backup store pod.
    kubectl patch pv <pv name> -p '{"spec":{"claimRef":{"uid":null}}}'
    

Register the backup store when another backup store does not already exist

If you have not registered a staging location and a backup store, complete the following steps to register a new backup store and bind it to the existing PV:

  1. Sign in to ArcGIS Enterprise Manager as an administrator of your organization.
  2. Click the Backups button.
  3. From the backups page, click Register a backup store.
  4. If using static provisioning, see the Admin API reference on defining the staging location settings. If using dynamic provisioning, provide the following information for the staging location:
    • Size (GiB)—Defines the size of the PV for the staging location. The minimum size is 16 GiB and should be large enough to contain each store's backup.
    • Storage class name—Defines the storage class name.
  5. Select the type of backup store being registered.
    • Amazon S3
    • Azure Blob
    • Google Cloud Storage
    • System managed storage
    1. If using Amazon S3, provide the following information:
      • Backup store name—The name of the backup store. The name can only contain lowercase letters, numbers, and hyphens, and must not begin or end with a hyphen.
      • Bucket name—The name of the bucket created in Amazon S3 to store backups.
      • Region—The region where the bucket was created.
      • Authentication type—Choose either Access key or IAM role.
      • If using access key as the authentication type, provide the following information:
        • Access key—Paste or type the access key for the IAM user.
        • Secret key—Paste or type the secret key for the IAM user.
    2. If using Azure Blob, provide the following information:
      • Backup store name—The name of the backup store. The name can only contain lowercase letters, numbers, and hyphens, and must not begin or end with a hyphen
      • Container name—The name of the container created in Azure to store backups.
      • Storage account—The name of the parent storage account the container exists under.
      • Authentication type—Choose either Storage account key or Managed identity.
      • If using storage account key as the authentication type, provide the following information:
        • Account key—Paste or type the primary or secondary account key for the associated storage account.
    3. If using Google Cloud Storage, provide the following information:
      • Backup store name—The name of the backup store. The name can only contain lowercase letters, numbers, and hyphens, and must not begin or end with a hyphen.
      • Bucket name—The name of the bucket created in Google Cloud Storage to store backups.
      • Access key—Paste or type the access key for the service account.
      • Secret key—Paste or type the secret key for the service account.
    4. If using system managed storage, provide the following information:
      • Volume type—To bind to the existing PV, Static should be selected. Administrators should confirm the PV has the required labels for binding and that it's available to bind.
      • Backup store name—Defines the name of the backup store. It must match the name of the previously registered backup store. The name can only contain lowercase letters, numbers, and hyphens and must not begin or end with a hyphen.
      • Size (GiB)—Defines the size of the PV for the backup store. The minimum size is 32 GiB and the value should match the size of the existing PV when using static binding. If the value is higher than the size of the existing PV, the PVC will not bind with the PV.
      • Storage class name—The storage class must match the storage class of the existing PV.
      • Label selector—Required for static provisioning, and the label or labels must match those of the existing PV.
  6. Click Register.
  7. Note:
    When binding to a precreated PV that has no storage class defined, the storage class name should be left blank. If a default storage class is configured within the cluster, the DefaultStorageClass admission controller will add the default storage class and prevent the PVC from binding. In this case, administrators should either add a storage class spec to the PV or remove the default storage class configuration.

Once a backup store is registered, all existing backups in the backup store will be listed on the backups page, but only backups from the same release will be available to restore.

Register the backup store when another backup store already exists

If you have registered a staging location and a backup store, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator of your organization.
  2. Click the Backups button.
  3. From the backups page, click Backup stores.
  4. Click Register store.
  5. Select the type of backup store being registered.
    • Amazon S3
    • Azure Blob
    • Google Cloud Storage
    • System managed storage
    1. If using Amazon S3, provide the following information:
      • Backup store name—The name of the backup store. The name can only contain lowercase letters, numbers, and hyphens, and must not begin or end with a hyphen.
      • Bucket name—The name of the bucket created in Amazon S3 to store backups.
      • Region—The region where the bucket was created.
      • Authentication type—Choose either Access key or IAM role.
      • If using access key as the authentication type, provide the following information:
        • Access key—Paste or type the access key for the IAM user.
        • Secret key—Paste or type the secret key for the IAM user.
    2. If using Azure Blob, provide the following information:
      • Backup store name—The name of the backup store. The name can only contain lowercase letters, numbers, and hyphens, and must not begin or end with a hyphen
      • Container name—The name of the container created in Azure to store backups.
      • Storage account—The name of the parent storage account the container exists under.
      • Authentication type—Choose either Storage account key or Managed identity.
      • If using storage account key as the authentication type, provide the following information:
        • Account key—Paste or type the primary or secondary account key for the associated storage account.
    3. If using Google Cloud Storage, provide the following information:
      • Backup store name—The name of the backup store. The name can only contain lowercase letters, numbers, and hyphens, and must not begin or end with a hyphen.
      • Bucket name—The name of the bucket created in Google Cloud Storage to store backups.
      • Access key—Paste or type the access key for the service account.
      • Secret key—Paste or type the secret key for the service account.
    4. If using system managed storage, provide the following information:
      • Volume type—To bind to the existing PV, Static should be selected. Administrators should confirm the PV has the required labels for binding and that it's available to bind.
      • Backup store name—Defines the name of the backup store. It must match the name of the previously registered backup store. The name can only contain lowercase letters, numbers, and hyphens and must not begin or end with a hyphen.
      • Size (GiB)—Defines the size of the PV for the backup store. The minimum size is 32 GiB and the value should match the size of the existing PV when using static binding. If the value is higher than the size of the existing PV, the PVC will not bind with the PV.
      • Storage class name—The storage class must match the storage class of the existing PV.
      • Label selector—Required for static provisioning, and the label or labels must match those of the existing PV.
  6. Click Register.
  7. Note:
    When binding to a precreated PV that has no storage class defined, the storage class name should be left blank. If a default storage class is configured within the cluster, the DefaultStorageClass admission controller will add the default storage class and prevent the PVC from binding. In this case, administrators should either add a storage class spec to the PV or remove the default storage class configuration.

Once a backup store is registered, all existing backups in the backup store will be listed on the backups page, but only backups from the same release will be available to restore.

Restore the backup

To restore a backup, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator of your organization.
  2. Click the Backups button.
  3. From the backups page, determine which backup you want to restore to and click the options button.
  4. Click Restore.
  5. Provide the encryption pass phrase that was used to create the backup.
  6. Click Restore.
Note:

If you use ArcGIS Web Adaptor for client access, you must unregister it from the Admin API and reregister it to pick up the configuration of the restored organization.

Check the status of a restore operation

Since the original organization was not accessible prior to the restore, the existing user session will be invalidated and the administrator must sign back in to ArcGIS Enterprise Manager to continue viewing the status of the restore. When prompted, sign back in to Enterprise Manager and the restore status page will appear.

In the out-of-place restore process, pod placement policies for the object, relational, and spatiotemporal data stores as well as the PublishingTools service are retained from the current configuration and ignored from the backup. This avoids restarting the data store StatefulSets as well as potential conflicts with the current cluster infrastructure configuration during the restore operation. For other system, utility, and user-published services, if the deployment does not refresh successfully, the deployment ID will be recorded in the results, and the operation will report that it completed successfully with warnings. An administrator should correct the placement policy for each following the restore operation.

Review restore result

For both the in-place and out-of-place restore processes, the following table summarizes the possible result states of a completed restore operation.

Result stateMeaning
success

No action necessary, all restore stages including configuration changes and deployment refreshes completed successfully and the organization is ready for use.

success_with_warnings

Objects failed to mirror, which may affect the organization functionality. An administrator should review the logs and messages to determine the areas of impact and assess the integrity of critical workflows. If this condition persists across multiple restore attempts, contact Esri Technical Support.

failure

One or more critical stages failed and any backups created after this point should be considered at-risk. The organization should be undeployed, deployed and configured, and restored from a backup created before this point in time.