Back to Top

Manage data stores

In this topic

  1. Manage folder data stores
  2. Manage required storage
  3. Manage raster stores

ArcGIS Enterprise Manager provides several options for managing your organization's data stores, including viewing folder data stores registered with your organization and adding and removing persistent volume (PV)-based folder data stores. You can also update cloud object and relational store credentials, increase the volume size of stores, and add storage to support raster analytics in your organization.

Manage folder data stores

From the User storage tab of the Data stores page in ArcGIS Enterprise Manager, you can view the folder data stores registered with your organization. If your organization has file- or folder-based GIS data in PVs, you can add the PVs as folder data stores and unregister them if needed. Adding PV-based folder data stores is only supported for administrators from ArcGIS Enterprise Manager. Other user managed data stores can be added from the portal.

Persistent volumes are a native Kubernetes solution for exposing data. For more information, refer to the Kubernetes documentation. ArcGIS Enterprise on Kubernetes supports using NFS shares, Windows (SMB) shares, or local persistent volumes as PV-based folder data stores.

Benefits of adding PVs as folder data stores

There are a variety of reasons why organizations may want to expose folder-based GIS data as PVs and add the PVs as folder data stores:

  • The flexibility of choosing Network File System (NFS) versions and mount options
  • The ability to register a Windows share directly with ArcGIS Enterprise on Kubernetes using SMB protocol
  • To meet requirements for cases in which data must be more local to service pods to have better performance—for example, local volumes are the recommended option for routing and geocoding services
  • To add a folder data store in organizations enforcing the restricted pod security standard, as registration of folder data stores is limited to PV-based volumes.

Common scenarios for organizations adding a PV-based folder data store include the following:

  • Windows share as a persistent volume—An organization has folder data for map image services in a Windows share that is used by publishers to connect to data from ArcGIS Pro. Instead of configuring an NFS share separately with duplicate data or configuring dual protocol on their file server for ArcGIS Pro and ArcGIS Enterprise to have access to the data, the administrator registers the Windows share directly to ArcGIS Enterprise on Kubernetes.
  • NFS share as a persistent volume—An organization has folder data for map and image services in an NFS share. The administrator registers this share with ArcGIS Enterprise on Kubernetes, using their desired NFS version and specifying additional mount options. Organization members can now publish map and image services that reference the data in the share.
  • Local persistent volume—An organization wants to share a geocode service as a locator to ArcGIS Enterprise on Kubernetes. To ensure the best performance, the locator data, including ArcGIS StreetMap Premium data, must be in close proximity to the compute-geocode service pods. To do this, the administrator creates a local PV, places locator data there, then adds the local PV as a PV-based folder data store. For more information see Configure the organization to geocode addresses.

Meet prerequisites for adding a PV-based folder data store

You must create the associated Persistent Volume before adding a PV-based folder data store. Each type of PV-based folder data store also has additional prerequisites that must be met.

  1. Before adding a PV-based folder data store for an NFS share, complete the following steps:
    1. Ensure the NFS server is accessible to the cluster.
      Note:

      For an AKS cluster, the cluster needs to be on the same or peered Azure virtual network (VNet) as the NFS Server. The cluster must be created on an existing VNet, which can be the same VNet as your NFS Server VM.

    2. Create the PV. See the sample NFS share .yaml file for more information.
  2. Before adding a PV-based folder data store for a Windows (SMB) share, complete the following steps:
    1. Install the SMB CSI Driver for Kubernetes in your cluster.
    2. Create the PV. See the sample Windows share .yaml file for more information.
  3. Before adding a PV-based folder data store for routing or geocoding, compete the following steps:
    1. Copy the data (for example, a locator for geocoding services) to the nodes in your cluster.
      Note:

      It is recommended that you use pod placement settings for services that will use the local PV so that they are only scheduled to nodes in a dedicated node pool. Then you can automate copying data to nodes in that pool using launch templates or other automation strategies.

    2. Create the PV. See the sample local PV .yaml file for more information.

Add a PV-based folder data store

To add a PV-based folder data store, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. Click the User storage tab.
  4. Click Add store and do the following:
    1. Enter a name for the folder data store.
    2. In the Publisher folder path field, provide individual shared locations where source data can be accessed and published by clients, such as ArcGIS Pro.

      This is usually a Windows path. If you register multiple folder data stores, ensure that each of them is a unique file share. No publisher folder paths should be a subdirectory of any other registered publisher folder path.

    3. If using a Publisher folder path with a drive letter, provide the Publisher folder hostname.
  5. If you are adding an NFS share, complete the following steps:
    1. Choose Linux for Share type.
    2. For NFS share path, provide an NFS path on the host machine where shared data can be stored and accessed across the organization.

      The NFS share path is available in the persistent volume definition file.

    3. For NFS hostname, provide the machine name where shared data is stored and accessed through an NFS path.
  6. If you are adding a Windows share, complete the following steps:
    1. Choose Windows for Share type.
    2. For Volume path, provide the source path of the Windows share.

      The volume path is available in the persistent volume definition file.

  7. If you are adding a local PV-based folder data store for routing or geocoding services, complete the following steps:
    1. Check the box next to This is a local persistent volume for routing or geocoding services.
    2. Choose Linux for Share type.
    3. Provide the local path on the nodes for the local PV in the Volume path field.

      The volume path is available in the persistent volume definition file.

  8. Provide the following information about the volume specification of the PV:
    • Access mode
    • Storage class name (optional)
    • Size (GiB)
    • Persistent volume name
    • Persistent volume claim name (optional)
  9. Click Add store.

View details of a PV-based folder data store

For a PV-based folder data store that has been previously added, you can view its details by clicking the More options button next to the data store. Details include the following information:

  • Publisher folder path
  • Volume path (local persistent volumes and Windows shares only)
  • NFS share path and NFS hostname (Linux only)
  • Persistent Volume name
  • Persistent Volume Claim name (if applicable)

Unregister a PV-based data store

If you unregister a PV-based folder data store, the services that reference it will be unable to connect to the data.

To unregister a data store, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. Click the User managed tab.
  4. Click the More options button next to the PV-based data store you want to unregister.
  5. Click Unregister and click Unregister again to confirm.
  6. Restart GIS services that referenced the PV-based data store.

    You can restart all GIS services in bulk using the Admin API.

Manage required storage

You may need to update the credentials of cloud object stores and cloud relational stores or increase the volume size of object stores, relational stores, and spatiotemporal index stores.

Update credentials

To update credentials for cloud object stores and cloud relational stores, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. Click the More options button for the store and click Update credentials.
  4. If you are updating a cloud object store, provide the following information:
    1. If using Amazon S3, choose Access key or IAM role as the authentication type and provide the following information:
      • If you selected Access key, paste or type the access key and secret key for the IAM user.
    2. If using Azure Blob, choose Storage account key or Managed identity as the authentication type and provide the following information:
      • If you selected Storage account key, enter the storage account name and paste or type the storage account key.
      • If you selected Managed identity, enter the storage account name and select System-assigned or User-assigned as the identity type.
        • If you selected user-assigned identity, provide the client ID.
    3. If using Google Cloud Storage, paste or type the access key and the secret key for the service account.
  5. If you are updating a cloud relational store, provide the database username and password.
  6. Click Update.

    The credentials have been updated.

Increase volume size

You can increase the volume size of object stores, relational stores, and spatiotemporal and index stores.

Note:

This option applies when storage was provisioned as persistent volumes. It is not supported for storage that was configured with cloud services.

To update the volume size, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. Click the More options button for the data store and click Increase volume size.
  4. Increase the volume size and click Update.

    If a failure occurs when updating the volume size, the Increase volume size window will contain information about that attempt. The Requested size column lists the previously requested volume size. To perform this workflow again, the new value must be greater than the previously requested size.

Migrate to a cloud relational store

You can use ArcGIS Enterprise Manager to migrate your relational store from persistent volume backed storage to a cloud database service.

Before migrating, review the considerations for using a cloud relational store. It is also recommended that you create a full backup of your ArcGIS Enterprise organization and disable the option to automatically delete backups.

During the migration process, the organization will automatically be put in read-only mode. Users will be able to sign in to the organization during the upgrade process, except for brief periods while critical system components are restarted. To minimize operational impacts, it is recommended that you perform the migration during a scheduled maintenance period.

To migrate the relational store, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. Click Migrate to cloud service for your relational store
  4. Confirm that the last backup created is recent enough to meet your requirements for restoring your organization using that backup and click Continue.
  5. If you need to create a new backup, click Go to backups to exit the migration process.
  6. Specify the configuration of the staging location.

    The staging location is a persistent volume where data from your existing relational store is temporarily copied before being migrated to the new cloud relational store. Based on the provisioning type, provide the following information for the staging location:

    • Storage class name—In the case of dynamic provisioning, specify the storage class name you've created. This field is optional for static provisioning.
    • Size (GiB)—The size of the PV for the staging location. The minimum required size for the staging location is calculated based on the amount of data you have in your relational store or 16 GiB, whichever is higher.
    • Label selector—Specify the label or labels that match those of the existing PV. This is mandatory for static provisioning and optional for dynamic provisioning.

  7. Click Continue, and click Cloud configuration.
  8. On the Cloud configuration page, select the database service from the drop-down menu and provide the following information:
    • Database endpoint (read-write)—The database endpoint that allows both read and write operations. This typically points to the primary instance of the database where all data modifications occur.
    • Database endpoint (read-only)—The database endpoint that is used only for read operations and is optional. This typically points to one or more read replicas of the database, which are synchronized with the primary instance. Read-only endpoints help distribute the load, improving performance and availability.
    • Database port—The network port on which the database server is listening for connections. The default port is 5432.
    • Username—The username of the database administrator account used to authenticate with the database. Ensure it has the necessary permissions to perform required operations.
    • Password—The password of the database administrator.
  9. Click Migrate and wait for the migration process to complete.
  10. Click Finish when the migration completes successfully.
  11. Create a full backup.

    Older backups before the migration will not be compatible with your new configuration using the cloud relational store. A new backup is required if you need to restore using the in-place restore process.

If you need to restore from a backup taken before the migration, use the out-of-place restore process.

Manage raster stores

There are two types of rasters stores: cloud and relational. A cloud store uses Amazon S3, Microsoft Azure Blob Storage, or Google Cloud Storage to store raster output. A relational store uses a database to store mosaic datasets. You can add, update the properties of, or unregister raster stores using Enterprise Manager.

Note:

You will not be able to add a new raster store if you already have a raster store of that same type.

Add a raster store

Image hosting and raster analytics capabilities both require one relational store and one cloud store. To add the required storage, complete the following steps for each type of raster store.

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. In the Raster stores section, click Add store.
    Note:

    You will not be able to add a new raster store if you already have a raster store of that same type.

  4. Enter the name of the new raster store.
  5. Select the storage service from the drop-down menu:
    • Amazon S3
    • Azure Blob
    • Google Cloud Storage
    • Relational store
    1. If using Amazon Simple Storage Service (S3), provide the following information:
      • Bucket name—The name of the bucket created in Amazon S3.
      • Region—The region where the bucket exists.
        Note:

        To connect to a private S3 endpoint, choose Custom as the region. Type or paste the URL to access the private cloud in the Private URL field.

      • Folder (optional)—The name of an existing or nonexisting root directory in Amazon S3 that can be used for the raster store.
      • Use existing credentials—Select this option to use existing cloud storage credentials for this raster store. This option will only appear if your organization was configured to use cloud storage when it was created.
      • Authentication type—If not using existing credentials, choose Access key or IAM role.
      • If using access key as the authentication type, provide the following information:
        • Access key—Paste or type the access key for the IAM user.
        • Secret key—Paste or type the secret key for the IAM user.
    2. If using Azure Blob, provide the following information:
      • Container name—The name of the container created in Azure.
      • Storage account—The name of the parent storage account the container exists under.
      • Storage domain—The Azure storage domain where the Blob storage container exists.
        Note:

        To connect to a custom Azure Blob endpoint, choose Other as the storage domain and provide the URL or DNS endpoint to access the storage location.

      • Folder (optional)—The name of an existing or nonexisting root directory in Azure Blob that can be used for the raster store.
      • Use existing credentials—Select this option to use existing cloud storage credentials for this raster store. This option will only appear if your organization was configured to use cloud storage when it was created.
      • Authentication type—If not using existing credentials, choose Shared key, Microsoft Entra ID, or Shared Access Signature.
      • If you selected Microsoft Entra ID, and choose user-assigned managed identity as the identity type, provide the client ID. If you choose service principal, also include the tenant ID and client secret.
        Note:

        User-assigned identities must have at least read-only access to the storage account container. If more than one user-assigned managed identity is assigned to an instance or storage container, you must specify a client ID.

      • If you selected Shared key, paste or type the primary or secondary account key for the associated storage account.
      • If you selected Shared Access Signature, enter the token.
    3. If using Google Cloud Storage, provide the following information:
      • Bucket name—The name of the bucket created in Google Cloud Storage.
      • Folder (optional)—The name of an existing or nonexisting root directory in Google Cloud Storage that can be used for the raster store.
      • Use existing credentials—Select this option to use existing cloud storage credentials for this raster store. This option will only appear if your organization was configured to use cloud storage when it was created.
      • Access key—If not using existing credentials, paste or type the access key for the service account.
      • Secret key—If not using existing credentials, paste or type the secret key for the service account.
    4. If using a relational store to store the mosaic datasets for hosted imagery layers, choose a database connection file.
  6. Click Add store.

    The raster store is added to your organization and appears in the Raster stores section.

Update raster store properties

It may become necessary to update the credentials or authentication method you specified when adding the raster store. To update raster store properties, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar.

    The data stores page appears.

  3. In the Raster stores section, click the More options button next to the raster store you want to update.
  4. If you are updating a cloud raster store, update the properties available for the storage service:
    • Amazon S3—Choose an authentication type, and provide the information required for the authentication type.
    • Azure Blob—Select an authentication type, and provide the information required for the authentication type.
    • If using Google Cloud Storage—Provide the access key and secret key.

    For descriptions of the properties, review the step for your storage service in the Add a raster store section above.

  5. If you are updating a hosted raster store, choose the updated database connection file.
    Caution:

    Choosing the incorrect connection file or changing the database username could break the existing published services associated with this raster store.

  6. Click Save.
Note:

Updating the properties of a cloud raster store other than credentials or authentication mode is not supported using Enterprise Manager.

Validate a raster store

To determine whether a cloud raster store is healthy or unhealthy, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar of ArcGIS Enterprise Manager.
  3. Click the Validate button next to the raster store you want to validate.

    Validation will succeed if the raster store is accessible and can be used. Validation will fail if the raster store is inaccessible, cannot be written to, or has some other issue, such as expired credentials. In the case of system failure or other unforeseen event, validation may also fail because the status of the raster store could not be determined.

Validation for relational raster stores is not currently available in ArcGIS Enterprise Manager. Use a database client like ArcGIS Pro to validate the username and password for a relational raster store. Use network management tools to ensure network traffic is open between the cluster and the database.

Unregister a raster store

To unregister a raster store, complete the following steps:

  1. Sign in to ArcGIS Enterprise Manager as an administrator.
  2. Click the Data stores button in the sidebar of ArcGIS Enterprise Manager.

    The data stores page appears.

  3. Click the More options button next to the raster store you want to unregister.
  4. Click Unregister and click Unregister again to confirm.

Feedback on this topic?

In this topic
  1. Manage folder data stores
  2. Manage required storage
  3. Manage raster stores