Ingesting Data into an OpenSearch Cluster with Fluent Bit

Prerequisites 🔗

Complete the following tasks before proceeding with the steps described in this topic:

• Create an OpenSearch cluster and make note of the connection information. See Creating an OpenSearch Cluster and Getting an OpenSearch Cluster's Details.

• Create and connect to a compute instance from your local machine. The instance must be in the same VCN as the OpenSearch cluster. See Create a VM Instance (Optional) and Connecting to an Instance.

Install and Configure Fluent Bit 🔗

1. Download Fluent Bit and install on the VM instance. To install the latest version on Linux-based instances, see Linux Packages. For other operating systems, see Supported Platforms.

2. Run the applicable command for the VM instance's operating system to install the Fluent Bit agent. For example, run the following command to install the Fluent Bit agent on a Linux instance:

   sudo rpm -i <fluent-bit-rpm file>

3. Update the OUTPUT section of the Fluent Bit configuration file, /fluent-bit/fluent-bit.conf to direct the output plugin to the OpenSearch cluster. For more information about this section, see Configuration File - Output.

   The following is sample code for this section of the configuration file. Replace the applicable text with the details for your cluster. Specify the OpenSearch cluster's OCID for host, see Getting an OpenSearch Cluster's Details. For http_User and http_Passwd, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.

   name - es
   host - <cluster_ID>
   port - 9200
   tls - on
   tls.verify - on
   index - <index_name>
   http_User - <cluster_user>
   http_Passwd - <cluster_password>
   Suppress_Type_Name - On
   Replace_Dots - On
   Retry_Limit - 6
   Buffer_Size - 400MB

4. Update the INPUT section of the Fluent Bit configuration file to specify the source for the log data and how it should be ingested. For more information about this section, see Configuration File - Input.

   For example:

   Name  tail
   Tag   file_log
   Path  /etc/fluent-bit/project/file.log
   Buffer_Max_Size  600m
   Multiline  On
   Parser_Firstline  multiline
   Buffer_Chunk_Size  2m
   Skip_Long_Lines  Off
   Skip_Empty_Lines  Off
   Mem_Buf_Limit  700m
   DB  /etc/fluent-bit/db/file.db
   DB.Sync  normal
   Refresh_Interval  30
   Rotate_Wait  20
   Exit_On_Eof  Off
   Ignore_Older  5m
   storage.type  filesystem

Test Data Ingestion with Fluent Bit 🔗

This section walks through the steps to test log data ingestion with Fluent Bit based on the configuration you specified in Install and Configure Fluent Bit. Perform these steps while you are connected to the VM instance. The sample commands included are for Linux-based operating systems, for other operating systems, use the applicable commands for that operating system.

1. Run the following commands to install Java and Python:

   sudo yum install python3

   sudo yum install java

2. Run the following command to create the directory for the log data:

   mkdir /etc/fluent-bit/project

3. Run the following command to start Fluent Bit:

   sudo fluent-bit -c /etc/fluent-bit/fluent-bit.confb

4. Create the python script file generate_log.py in /etc/fluent-bit/project and copy the following code into the file:

   Import logging
    
   # Configure logging to write to a file
    
   logging.basicConfig(filename='file.log', level=logging.DEBUG)
    
   # Generate some log messages
    
   logging.debug('This is a debug message.')
    
   logging.info('This is an informational message.')
    
   logging.warning('This is a warning message.')
    
   logging.error('This is an error message.')
    
   logging.critical('This is a critical message.')
    
   # Example of logging additional details using formatting
    
   name = 'John'
    
   age = 30
    
   logging.info('User %s, age %d, logged in.', name, age)
    
   # Example of logging an exception stack trace
    
   try:
    
       result = 10 / 0
    
   except Exception as e:
    
    logging.exception('An error occurred: %s', str(e))

5. Use the following command to run the script created in the previous step:

   python3 /etc/fluent-bit/project/generate_log.py

   Running this script creates a file file.log and adds the specified logging messages to the file.

After you run the python script, you can verify that the log data was ingested into the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check for the index you specified in the OUTPUT section of the Fluent Bit configuration file. See Task 6: Connect to OpenSearch Dashboards and Quickstart guide for OpenSearch Dashboards.