Provisioning PVCs on the File Storage Service

When using the File Storage service to provision PVCs, you specify in-transit encryption independently of at-rest encryption.

The File Storage service always encrypts data at rest, using Oracle-managed encryption keys by default. However, you have the option to specify at-rest encryption using your own master encryption keys that you manage yourself in the Vault service. How to specify at-rest encryption depends on whether you are provisioning a PVC on:

• a new file system dynamically created by the CSI volume plugin (see Encrypting Data At Rest on a New File System)
• an existing file system that you've manually created (see Encrypting Data At Rest on an Existing File System)

If you want to manage your own master encryption keys to encrypt data at rest, you have to:

• Create a suitable master encryption key in Vault (or obtain the OCID of such a key). See Managing Keys.
• Create a policy granting access to the master encryption key. See Create Policy to Access User-Managed Encryption Keys for Encrypting Boot Volumes, Block Volumes, and/or File Systems.

The File Storage service can optionally encrypt data in transit. If you specify in-transit encryption, data in transit is encrypted using a TLS (Transport Layer Security) certificate that is always Oracle-managed, regardless of whether data at rest is encrypted using Oracle-managed keys or using user-managed keys. In-transit encryption secures data being transferred between instances and mounted file systems using TLS v. 1.2 encryption. For more information about in-transit encryption and the File Storage service, see Using In-transit TLS Encryption. How to specify in-transit encryption depends on whether you are provisioning a PVC on:

• a new file system dynamically created by the CSI volume plugin (see Encrypting Data In Transit on a New File System)
• an existing file system that you've manually created (see Encrypting Data In Transit on an Existing File System)

Note that when using the File Storage service to provision PVCs, in-transit encryption is only supported when the compute instances hosting worker nodes are running Oracle Linux 7 and Oracle Linux 8.

To dynamically provision a PVC on a new file system dynamically created by the CSI volume plugin in the File Storage service:

1. (Optional) Create a mount target in the File Storage service. See Creating a Mount Target.

   The CSI volume plugin requires an active mount target (that is, a mount target with an assigned IP address) to create a new file system. If you do not create a mount target in advance, specify the subnet in which the CSI volume plugin is to create a new mount target when you define the storage class.

2. Define a new storage class that uses the fss.csi.oraclecloud.com provisioner:
   1. Create a manifest file (for example, in a file named fss-dyn-st-class.yaml), specify a name for the new storage class, and specify values for required and optional parameters:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: <storage-class-name>
      provisioner: fss.csi.oraclecloud.com
      parameters:
        availabilityDomain: <ad-name>
        mountTargetOcid: <mt-ocid> | mountTargetSubnetOcid: <mt-subnet-ocid>
        compartmentOcid: <compartment-ocid>
        kmsKeyOcid: <key-ocid>
        exportPath: <path>
        exportOptions: "[{<options-in-json-format>}]"
        encryptInTransit: "true"|"false"
        oci.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}'
        oci.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}'

      where:

      • name: <storage-class-name>: Required. A name of your choice for the storage class.
      • availabilityDomain: <ad-name>: Required. The name of the availability domain in which to create the new file system (and in which to create the new mount target, if an existing mount target OCID is not specified). For example, US-ASHBURN-AD-1. To find out the availability domain name to use, run the oci iam availability-domain list CLI command (or use the ListAvailabilityDomains operation). For more information, see Your Tenancy's Availability Domain Names.
      • mountTargetOcid: <mt-ocid> | mountTargetSubnetOcid: <mt-subnet-ocid>: Required. Either the OCID of an existing active mount target (mountTargetOcid: <mt-ocid>), or the OCID of a subnet in which to create a new mount target (mountTargetSubnetOcid: <mt-subnet-ocid>). Specify either mountTargetOcid or mountTargetSubnetOcid. If you specify both mountTargetOcid and mountTargetSubnetOcid in the storage class definition, the existing mount target specified by mountTargetOcid is used, and mountTargetSubnetOcid is ignored. For example:
        • mountTargetSubnetOcid: ocid1.subnet.oc1.iad.aaaaaaaa2xpk______zva
        • mountTargetOcid: ocid1.mounttarget.oc1.iad.aaaaaaaa4np______fuy
      • compartmentOcid: <compartment-ocid>: Optional. The OCID of the compartment to which the new file system (and the new mount target, if an existing mount target OCID is not specified) is to belong. If not specified, defaults to the same compartment as the cluster. For example, compartmentOcid: ocid1.compartment.oc1..aaaaaaaay______t6q
      • kmsKeyOcid: <key-ocid>: Optional. The OCID of a master encryption key that you manage, with which to encrypt data at rest. If not specified, data is encrypted at rest using encryption keys managed by Oracle. See Encrypting Data At Rest on a New File System. For example, kmsKeyOcid: ocid1.key.oc1.iad.anntl______usjh
      • exportPath: <path>: Optional. The path in an export that uniquely identifies the file system within the mount target. The export path must start with a slash (/) followed by a sequence of zero or more slash-separated elements. For example, /FileSystem1. For more information, see Paths in File Systems.

        If you include exportPath: <path> in a storage class definition, do not specify a path (either as a path or as the sub-path of an existing path) that already exists. If you specify a path that already exists, an error is returned because multiple file systems with the same mount target cannot have the same export path. Therefore, if you do include exportPath: <path> in the storage class definition, only use this storage class definition to create one file system.

        If you don't include exportPath: <path> in the storage class definition, the path defaults to the display name of the file system created by the CSI volume plugin (starting with /csi-fss-).

      • exportOptions: "[{<options-in-json-format>}]" Optional. A set of parameters (in valid JSON format) within the export that specifies the level of access granted to NFS clients when they connect to a mount target. An NFS export options entry within an export defines access for a single IP address or CIDR block range. For more information, see Working with NFS Exports and Export Options. If not specified, the following default is used:

        exportOptions: "[{\"source\":\"0.0.0.0/0\",\"requirePrivilegedSourcePort\":false,\"access\":\"READ_WRITE\",\"identitySquash\":\"NONE\"}]"

      • encryptInTransit: "true"|"false": Optional. Indicates whether to encrypt data in transit. If you specify "true", be sure to complete the setup steps described in Encrypting Data In Transit on a New File System. If not specified, defaults to "false". For example, encryptInTransit: "true".
      • oci.oraclecloud.com/initial-defined-tags-override: '{"<tag-namespace>": {"<tag-key>": "<tag-value>"}}' Optional. Specifies a defined tag for the new file system. For example, oci.oraclecloud.com/initial-defined-tags-override: '{"Operations": {"CostCenter": "42"}}'

        Note that to apply defined tags from a tag namespace belonging to one compartment to a filesystem resource belonging to a different compartment, you must include a policy statement to allow the cluster to use the tag namespace. See Additional IAM Policy when a Cluster and a Tag Namespace are in Different Compartments.

      • oci.oraclecloud.com/initial-freeform-tags-override: '{"<tag-key>": "<tag-value>"}' Optional. Specifies a free-form tag for the new file system. For example, oci.oraclecloud.com/initial-freeform-tags-override: '{"Department": "Finance"}'

      For example:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: fss-dyn-storage
      provisioner: fss.csi.oraclecloud.com
      parameters:
        availabilityDomain: US-ASHBURN-AD-1
        mountTargetSubnetOcid: ocid1.subnet.oc1.iad.aaaaaaaa2xpk______zva
        compartmentOcid: ocid1.compartment.oc1..aaaaaaaay______t6q
        kmsKeyOcid: ocid1.key.oc1.iad.anntl______usjh
        exportPath: /FileSystem1
        exportOptions: "[{\"source\":\"0.0.0.0/0\",\"requirePrivilegedSourcePort\":false,\"access\":\"READ_WRITE\",\"identitySquash\":\"NONE\"}]"
        encryptInTransit: "true"

   2. Create the storage class from the manifest file by entering:

      kubectl create -f <filename>

      For example:

      kubectl create -f fss-dyn-st-class.yaml

3. Create security rules in either a network security group (recommended) or a security list for both the mount target referenced in (or created by) the storage class definition, and for the cluster's worker nodes.

   The security rules to create depend on the relative network locations of the mount target and the worker nodes, according to the following scenarios:

   • Scenario B: Mount target and instance in the same subnet
   • Scenario A: Mount target and instance in different subnets (recommended)

   These scenarios, the security rules to create, and where to create them, are fully described in the File Storage service documentation (see Configuring VCN Security Rules for File Storage).

4. Create a PVC to be provisioned by the new file system in the File Storage service, as follows:
   1. Create a manifest file to define the PVC:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: <pvc-name>
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: "<storage-class-name>"
        resources:
          requests:
            storage: 50Gi

      where:

      • name: <pvc-name>: Required. For example, fss-dynamic-claim
      • storageClassName: "<storage-class-name>": Required. The name of the storage class you defined earlier. For example, fss-dyn-storage.
      • accessModes: - ReadWriteMany: Required. Note that accessModes: must specify ReadWriteMany.
      • storage: 50Gi: Required. Note that Kubernetes requires that you specify a value for storage: in the PVC manifest. However, the File Storage service does not support the specification of a file system size and creates a new file system with a default size, regardless of the value you specify for storage:.

      For example, the following manifest file (named fss-dyn-claim.yaml) defines a PVC named fss-dynamic-claim that is provisioned by the file system defined in the fss-dyn-storage storage class:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: fss-dynamic-claim
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: "fss-dyn-storage"
        resources:
          requests:
            storage: 50Gi

   2. Create the PVC from the manifest file by entering:

      kubectl create -f <filename>

      For example:

      kubectl create -f fss-dyn-claim.yaml

   A new PVC is created. The CSI volume plugin creates a new persistent volume (PV) and a new file system in the File Storage service (along with a new mount target if you didn't specify an existing mount target).

   The new PVC is bound to the new PV. Data is encrypted at rest, using encryption keys managed either by Oracle or by you.

5. Verify that the PVC has been bound to the new persistent volume by entering:

   kubectl get pvc

   The output from the above command confirms that the PVC has been bound successfully:

   			
   NAME                STATUS    VOLUME                 CAPACITY   ACCESSMODES   STORAGECLASS      AGE
   fss-dynamic-claim   Bound     csi-fss-<unique_ID>    50Gi       RWX           fss-dyn-storage   4m

6. Use the new PVC when creating other objects, such as pods. For example, you could create a new pod from the following pod definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: fss-dynamic-app
   spec:
     containers:
       - name: nginx
         image: nginx:latest
         ports:
           - name: http
             containerPort: 80
         volumeMounts:
           - name: persistent-storage
             mountPath: /usr/share/nginx/html
     volumes:
     - name: persistent-storage
       persistentVolumeClaim:
         claimName: fss-dynamic-claim

7. Having created a new pod as described in the example in the previous step, you can verify that the pod is using the new PVC by entering:

   kubectl describe pod nginx

To create a PVC on an existing file system in the File Storage service (using Oracle-managed encryption keys to encrypt data at rest):

1. Create a file system with a mount target in the File Storage service, selecting the Encrypt using Oracle-managed keys encryption option. See Creating a File System.
2. Create security rules in either a network security group (recommended) or a security list for both the mount target that exports the file system, and for the cluster's worker nodes.
   The security rules to create depend on the relative network locations of the mount target and the worker nodes, according to the following scenarios:

   • Scenario B: Mount target and instance in the same subnet
   • Scenario A: Mount target and instance in different subnets (recommended)

   These scenarios, the security rules to create, and where to create them, are fully described in the File Storage service documentation (see Configuring VCN Security Rules for File Storage).

3. Create a PV backed by the file system in the File Storage service as follows:

   1. Create a manifest file to define a PV and in the csi: section, set:

      • driver to fss.csi.oraclecloud.com
      • volumeHandle to <FileSystemOCID>:<MountTargetIP>:<path>
        where:

        • <FileSystemOCID> is the OCID of the file system defined in the File Storage service.
        • <MountTargetIP> is the IP address assigned to the mount target.
        • <path> is the mount path to the file system relative to the mount target IP address, starting with a slash.

        For example: ocid1.filesystem.oc1.iad.aaaa______j2xw:10.0.0.6:/FileSystem1

      For example, the following manifest file (named fss-pv.yaml) defines a PV called fss-pv backed by a file system in the File Storage service:

      apiVersion: v1
      kind: PersistentVolume
      metadata:
        name: fss-pv
      spec:
        capacity:
          storage: 50Gi
        volumeMode: Filesystem
        accessModes:
          - ReadWriteMany
        persistentVolumeReclaimPolicy: Retain
        csi:
          driver: fss.csi.oraclecloud.com
          volumeHandle: ocid1.filesystem.oc1.iad.aaaa______j2xw:10.0.0.6:/FileSystem1

   2. Create the PV from the manifest file by entering:

      kubectl create -f <filename>

      For example:

      kubectl create -f fss-pv.yaml

4. Create a PVC that is provisioned by the PV you have created, as follows:
   1. Create a manifest file to define the PVC and set:
      • storageClassName to ""
      • volumeName to the name of the PV you created (for example, fss-pv)

      For example, the following manifest file (named fss-pvc.yaml) defines a PVC named fss-pvc that is provisioned by a PV named fss-pv:

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: fss-pvc
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: ""
        resources:
          requests:
            storage: 50Gi
        volumeName: fss-pv

      Note that the requests: storage: element must be present in the PVC's manifest file, and its value must match the value specified for the capacity: storage: element in the PV's manifest file. Apart from that, the value of the requests: storage: element is ignored.

   2. Create the PVC from the manifest file by entering:

      kubectl create -f <filename>

      For example:

      kubectl create -f fss-pvc.yaml

The PVC is bound to the PV backed by the File Storage service file system. Data is encrypted at rest, using encryption keys managed by Oracle.