Oracle Cloud Infrastructure Documentation

Scenario: Sending Log Data to an Autonomous AI Database

Send log data from Logging to an Autonomous AI Database using a function (Functions service).

This scenario involves creating a function and then referencing that function in a connector (Connector Hub)  to process and move log data from Logging to an Autonomous AI Database.

Required IAM Policy

If you're a member of the Administrators group, you already have the required access to execute this scenario. Otherwise, you need access to Functions .

The workflow for creating the connector includes a default policy when needed to provide permission for writing to the target service. If you're new to policies, see IAM Policies Overview.

Setting Up This Scenario

This scenario involves creating a connector (Connector Hub)  to send log data from Logging to an Autonomous AI Database (JSON).

Before you can create the connector, you must set up an Autonomous AI JSON Database that you want to receive the log data and set up the function to copy that log data.

Your function must be deployed. In this example, the connector moves log data from Logging to an Autonomous AI Database using the function you created using the function code sample.

For help with troubleshooting, see Troubleshooting Connectors.

Autonomous AI JSON Database setup details:

  • Provision Autonomous AI Database
  • Duplicate the browser tab.
  • In the duplicated tab, copy the ORDS base URL: From the details page for the Autonomous AI Database, select Service Console, select Development, and then select Copy under RESTful Services and SODA.'
  • Create a "logs" collection to store the log data that will be moved by the function and connector:

    1. Go back to the details page for the Autonomous AI Database (the original browser tab).

    2. Select Database Actions.

    3. Log in with the admin user and the password you set when you created the database.

      The Database Actions | Launchpad window is displayed.

    4. Select SQL.

      The Database Actions | SQL window is displayed.

    5. Enter the following command:

      soda create logs

    6. Select Run Statement (Run Statement).

    To query documents in the collection after the connector copies log data, enter the following command:

    soda get logs -f {}

Function setup details:

Once the database and function are set up, you're ready to create the connector. Creating the connector is easy in the Console. Alternatively, you can use the Oracle Cloud Infrastructure CLI or API, which lets you execute the individual operations yourself.

Function code sample

The following code sample is for a function to send log data from the Logging service to an Autonomous AI Database. For instructions on creating and deploying functions, see Creating and Deploying Functions.

Note

The following code sample isn't meant for production workloads. Update it for your production environment.
import io
import json
import logging
import requests

from fdk import response

# soda_insert uses the Autonomous AI Database REST API to insert JSON documents
def soda_insert(ordsbaseurl, dbschema, dbuser, dbpwd, collection, logentries):
    auth=(dbuser, dbpwd)
    sodaurl = ordsbaseurl + dbschema + '/soda/latest/'
    bulkinserturl = sodaurl + 'custom-actions/insert/' + collection + "/"
    headers = {'Content-Type': 'application/json'}
    resp = requests.post(bulkinserturl, auth=auth, headers=headers, data=json.dumps(logentries))
    return resp.json()

def handler(ctx, data: io.BytesIO=None):
    logger = logging.getLogger()
    logger.info("function start")

    # Retrieving the Function configuration values
    try:
        cfg = dict(ctx.Config())
        ordsbaseurl = cfg["ordsbaseurl"]
        dbschema = cfg["dbschema"]
        dbuser = cfg["dbuser"]
        dbpwd = cfg["dbpwd"]
        collection = cfg["collection"]
    except:
        logger.error('Missing configuration keys: ordsbaseurl, dbschema, dbuser, dbpwd and collection')
        raise
    
    # Retrieving the log entries from Connector Hub as part of the Function payload
    try:
        logentries = json.loads(data.getvalue())
        if not isinstance(logentries, list):
            raise ValueError
    except:
        logger.error('Invalid payload')
        raise
    
    # The log entries are in a list of dictionaries. We can iterate over the the list of entries and process them.
    # For example, we are going to put the Id of the log entries in the function execution log
    logger.info("Processing the following LogIds:")
    for logentry in logentries:
        logger.info(logentry["oracle"]["logid"])

    # Now, we are inserting the log entries in the JSON Database
    resp = soda_insert(ordsbaseurl, dbschema, dbuser, dbpwd, collection, logentries)
    logger.info(resp)
    if "items" in resp:
        logger.info("Logs are successfully inserted")
        logger.info(json.dumps(resp))
    else:
        raise Exception("Error while inserting logs into the database: " + json.dumps(resp))

    # The function is done. Return empty response.
    logger.info("function end")
    return response.Response(
        ctx, 
        response_data="",
        headers={"Content-Type": "application/json"}
    )
  • Note

    For complete descriptions of fields on the Create connector page, see Creating a Connector.

    On the Connectors list page, select Create connector. If you need help finding the list page, see Listing Connectors.

    1. Basic Connector Information

    Enter identifying information.

    • Connector name: Enter a user-friendly name for the new connector and an optional description. Avoid entering confidential information. Example: Send Logs to My Autonomous AI Database
    • Select a compartment: Select the compartment to store the new connector in.

    Select Next.

    2. Configure Connector Source

    • Select source: Select Logging.
    • Configure source: Select the compartment, log group, and log.
    • Create policy: Select to accept the default policy provided for the entered source configuration.

    Select Next.

    3. Configure Connector Task

    Select Next (skip this section).

    4. Configure Connector Target

    Enter the metric namespace and metric name that you want to use for the filtered log data.

    • Target: Select Functions.
    • Configure target: Select the function application and function that you created using the function code sample.
    • Create policy: Select to accept the default policy provided for the entered target configuration.

    Select Next.

    Connector Preview

    Review the connector configuration and then select Create.

    The creation process begins, and its progress is displayed. On completion, the connector's details page opens.

  • For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.

    • Create a connector: Open a command prompt and run the oci sch service-connector create command: 

      oci sch service-connector create --display-name
"<display_name>" --compartment-id <compartment_OCID> --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]

    For help with troubleshooting, see Troubleshooting Connectors.

  • For information about using the API and signing requests, see REST API documentation and Security Credentials. For information about SDKs, see SDKs and the CLI.

    Use the following operations:

    • CreateServiceConnector: Create a connector that moves log data to your function (which then moves the data to an Autonomous AI Database).

      Example CreateServiceConnector request
      POST /20200909/serviceConnectors
Host: service-connector-hub.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "compartmentId": "<compartment_OCID>",
  "description": "My connector description",
  "displayName": "My Connector",
  "source": {
    "kind": "logging",
    "logSources": [
      {
        "compartmentId": "<compartment_OCID>",
        "logGroupId": "<log_group_OCID>",
        "logId": "<log_OCID>"
      }
    ]
  },
  "target": {
    "compartmentId": "<compartment_OCID>",
    "kind": "functions",
    "functionId": "<function_OCID>"
  },
  "tasks": []
  }
}

    For help with troubleshooting, see Troubleshooting Connectors.