Enabling Logs in Oracle Cloud Infrastructure Logging

Data Integration lets you use Oracle Cloud Infrastructure Logging to enable logging and retrieve service logs in the Console or from the CLI.

Public logs are available for the following types:

  • Execution logs for different task runs including integration, data loader, REST, SQL, and pipeline task runs
  • Create, edit, and delete data flow logs

Only administrative users can enable logging. Non-administrative users can only view log contents, and only after administrative users have enabled and set up logging, and given you access to logging groups and content.

Currently, Data Integration service logs are enabled at the workspace resource level using the DIS Workspace Logs log category. For example, when logging is enabled and active, integration task runs associated with data flows in a specified Data Integration workspace have the logging feature turned on.

Resource logs are grouped in logical containers called log groups in Oracle Cloud Infrastructure Logging. Administrative users must create at least one log group.

Before you start using Oracle Cloud Infrastructure Logging:

  • Get familiar with basic concepts and terminology used in the OCI Logging service. See Logging Overview in the OCI Logging documentation.

  • Create a group to manage access to log groups and log content. See Managing Groups in the OCI IAM with Identity Domains documentation.

  • Add the policy to enable public logging for the Data Integration service.

    Note

    To restrict the enable permission to specific users, replace any-user with a specific user.

    ALLOW any-user TO {DIS_WORKSPACE_OBJECT_CREATE} in tenancy WHERE ALL {request.service.name = 'dataintegration', request.principal.type = 'service'}
  • Add the policies to allow you to use OCI Logging with the Data Integration service.

    allow group <group-name> to manage log-groups in compartment <compartment-name>
    allow group <group-name> to manage log-content in compartment <compartment-name>
  • Ensure that you have the policies that are required to use OCI Logging. See Required Permissions for Working with Logs and Log Groups in the OCI Logging documentation.

  • In the OCI Logging service, create one or more log groups to organize and manage logs in logical containers. See Creating a Log Group in the OCI Logging documentation.

Enabling Logs in the Console

Follow these steps in the Console to use Oracle Cloud Infrastructure Logging to enable Data Integration service logging. Only administrative users can enable logging.

Currently, Data Integration service logs are enabled at the workspace level. To enable logging, you need to know the workspace and the name of the compartment where the workspace is located.

  1. Open the navigation menu and click Observability & Management. Under Logging, click Logs.
  2. Click Enable service log. The Enable Resource Log panel is displayed.
  3. Under Select Resource, do the following:
    1. Select the Resource Compartment that has the Data Integration workspace for which you want to collect service logs.
    2. For Service, select Data Integration Service to enable resource logging in the service.
    3. For Resource, select the name of the Data Integration workspace on which you want to enable public logs for collection.
  4. Under Configure Log, do the following:
    1. Select the Log Category to specify the type of log to create for this resource.

      Currently, there's only one category, DIS Workspace Logs, which contains all logs for task runs in a specified Data Integration workspace. For example, integration task runs associated with data flows.

    2. In Log Name, enter a name for this log. Use the following log name guidelines.
      • The first character must start with a letter.
      • Use from 1 to 256 characters.
      • A log name is case-sensitive. Valid characters are letters (upper or lowercase), numbers, hyphens, underscores, and periods.
      • Avoid entering confidential information.
    Note

    You can only have one log for any combination of service, resource, and log category.
  5. If needed, click Show Advanced Options to specify where you want to create the log.
    1. Select the Compartment.
    2. Select the Log Group.

      The OCI Logging service uses log groups to organize and manage logs in logical containers.

      If there are no log groups in the selection menu, click Create New Group to create one. See Creating a Log Group in the OCI Logging documentation.

  6. (Optional) Under Log Retention, you can specify how long you want to retain the log, and also apply tags for tracking.
  7. Click Enable Log.
The Log details page is displayed. The Status shows as "Creating" and the message "Creating log" is shown under Explore Log while the log is being created. When the log is created, the Status changes to "Active." To learn more about the Log details page, see Getting a Log's Details in the OCI Logging documentation.

Enabling Logs in the CLI

Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging to enable Data Integration service logging for a workspace.

  1. Create a public log group.
    Request
    oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dis-logs"
    Response
    {
      "data": {
        "compartment-id": "<compartment-ocid>",
        "id": "<log-workrequest-ocid>",
        "operation-type": "CREATE_LOG_GROUP",
        "percent-complete": 100.0,
        "resources": [
          {
            "action-type": "CREATED",
            "entity-type": "loggroup",
            "entity-uri": "/logGroups/<log-group-ocid>",
            "identifier": "<log-group-ocid>"
          }
        ],
        "status": "SUCCEEDED",
        "time-accepted": "2022-23-08T18:53:29.065000+00:00",
        "time-finished": "2022-23-08T18:53:29.065000+00:00",
        "time-started": "2022-23-08T18:53:29.065000+00:00"
      }
    }
  2. Enable Data Integration service logs for the workspace.
    Request
    export LOG_GROUP_ID=<log-group-ocid>
    oci logging log create --display-name "MyWorkspace Logs" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json
    config.json file
    {
        "compartment-id" : "<compartment-ocid>",
        "source": {
            "resource" : "<workspace-ocid>",
            "service" : "dataintegration",
            "source-type" : "OCISERVICE",
            "category" : "disworkspacelogs"
        }
    }
    Response
    {
      "opc-work-request-id": "<log-workrequest-ocid>"
    }
  3. Get the log ID of the created public log.

    The log ID is the OCID that's uniquely generated for the specified resource and log category in the specified compartment.

    Request
    export LOG_WR_ID=<log-workrequest-ocid>
    oci logging work-request get --work-request-id $LOG_WR_ID
    Response
    {
      "data": {
        "compartment-id": "<compartment-ocid>",
        "id": "<log-workrequest-ocid>",
        "operation-type": "CREATE_LOG",
        "percent-complete": 100.0,
        "resources": [
          {
            "action-type": "CREATED",
            "entity-type": "log",
            "entity-uri": "/logs/<log-ocid>",
            "identifier": "<log-ocid>"
          }
        ],
        "status": "SUCCEEDED",
        "time-accepted": "2022-23-08T18:58:29.065000+00:00",
        "time-finished": "2022-23-08T18:58:29.065000+00:00",
        "time-started": "2022-23-08T18:58:29.065000+00:00"
      }
    }

Example of a Data Integration Public Log

The following is an example of a Data Integration service public log in JSON format.

The Data Integration fields in the output are:

Field Description

logLevel

The level of the log. For example: INFO,WARN/ERROR.

message

The actual log message.

objectId

The taskRunId or dataflowId, depending on the type of operation logged.

The Id is the Universal Unique Identifier (UUID) of the object for which the log is generated. For example, for a task run, it's the TaskRunKey UUID. For a data flow save log, it's the dataflow UUID.

opcRequestId

The opcRequestId of the request that came in.

workspaceId

The workspace OCID.

{
  "datetime": 1660885646612,
  "logContent": {
    "data": {
      "logLevel": "INFO",
      "message": "ACTION: INTEGRATION_TASK-TASK-RUN- Submit on TASK b7fba0c4-bf0d-4c29-b49f-b21673cc2791 - SUCCESS",
      "objectId": "b7fba0c4-bf0d-4c29-b49f-b21673cc2791",
      "opcRequestId": "csidb3d7aef846ab99c53689c81107f0/f11c0c8e208a49e8a1f79176bdf611a6/D4927163D0EAC4B699CD3076A17C5FD0",
      "workspaceId": "ocid1.disworkspace.oc1.phx.<long-string-letters-12345>"
    },
    "id": "2d0da51e-e02d-4ba2-b766-6255ed1b1269",
    "oracle": {
      "compartmentid": "ocid1.tenancy.oc1..<long-string-letters-12345>",
      "ingestedtime": "2022-08-19T05:07:55.326Z",
      "loggroupid": "ocid1.loggroup.oc1.phx.<long-string-letters-12345>",
      "logid": "ocid1.log.oc1.phx.<long-string-letters-12345>",
      "tenantid": "ocid1.tenancy.oc1..<long-string-letters-12345>"
    },
    "source": "ocid1.disworkspace.oc1.phx.<long-string-letters-12345>",
    "specversion": "1.0",
    "time": "2022-08-19T05:07:26.612Z",
    "type": "public.log.dis.execute"
  }
}