Oracle Cloud Infrastructure Documentation

Storage Mounts

Similar to ML Jobs, Pipelines enable customers to mount File Storage and Object Storage as local folders for pipeline steps.

This simplifies the process of accessing data stored in File or Object Storage, removing the need for installing extra libraries or writing specialized code. Using standard OSs or language libraries, users can access shared file systems and Object Storage buckets in the same way they access local files and folders.

You can also override storage mounts in pipeline runs, enabling one-off changes for a specific run. This is essential for SaaS customers who use pipelines within ML Applications and need to override storage mounts for a particular SaaS customer (tenant).

Configuring Storage Mounts When Creating a Pipeline

Ensure you have the appropriate Policies applied.

Storage mounts can be configured at two levels:

  • Pipeline level: The storage mount configuration is applied to all steps, unless overridden.
  • Step level: The storage mount configuration is applied only to the specific step.
Note

Storage mounts apply only to the following step types:
  • Script step type. Steps defined by a script.
  • BYOC step type. Steps defined by a container image.

The configuration is managed in the Storage Mounts configuration panel.

You can configure no more than five storage mounts for a pipeline or a step.

  1. On the Projects list page, select the project that contains the pipelines that you want to work with. If you need help finding the list page or the project, see Listing Projects.
  2. On the project details page, select Pipelines.
  3. Select Create pipeline.
  4. On the Create pipeline page, under Storage mounts (at either the pipeline or step level), enter the following information.
    Note

    Creating Pipelines with Custom Networking Using APIs must be used to use a file storage mount.
    • File storage mounts (Optional): Select Add file storage mount and enter the following information.
      • Compartment: Select the compartment that contains the target that you want to mount.
      • Mount target: The mount target that you want to use.
      • Export path: The export path that you want to use.
      • Destination path and directory: Enter the path to use for mounting the storage.

        The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

        You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

    • Object storage mounts (Optional): Select Add object storage mount and enter the following information.
      • Compartment: Select the compartment that contains the bucket that you want to mount.
      • Bucket: Select the bucket that you want to use.
      • Object name prefix (Optional): Object name prefix. The prefix must start with an alphanumeric character. The allowed characters are alphanumerics, slash ( / ), hyphen ( - ) and underscore ( _ ).
      • Destination path and directory: Enter the path to use for mounting the storage.

        The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

        You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

      Note

      If using custom networking:
      1. Create the service gateway in the VCN.
      2. For the route table configurations in the private subnet, add the service gateway.
      3. Change the egress rules of security list of the required subnet to let traffic to all services in the network.
  5. Update other fields as needed.
    For field descriptions, see Creating a Pipeline.
  6. Select Create.

Storage Mount Overrides

You can specify storage mount overrides for Pipeline Runs the same way as for Pipelines. For details about storage mounts, see Storage Mounts.

When creating a pipeline run, you can define the override at the pipeline level or for a pipeline step.