Common Load Balancer Errors

Learn about common load balancer errors associated with the load balancers.

Common load balancer errors include, series 500 and series 400 errors, health check errors, client errors, and SSL errors. The subsequent topics in this section describe these common errors and detail troubleshooting procedures for each, where applicable.

Server Errors (500-599)

504

Error messages:

  • lbStatusCode: "504"

  • backendStatusCode: ""

Oracle Cloud Infrastructure log category: Access log

Symptoms:

The client fails with a 504 error.

Possible causes:

The load balancer is not able to establish connections with any of the backends, even though the health check is marking the backends as available.

Possible solutions:

Configure the health check correctly.

Troubleshooting documentation: Editing a Load Balancer's Health Check Policies

502, 502

Error messages:
  • lbStatusCode: "502"

  • backendStatusCode: "502"

Oracle Cloud Infrastructure log category: Access log and error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The backend health check succeeds.

  • The backend returns a 502 error.

Possible causes:
  • An application on the backend is returning a 502 error.

  • The backend is configured incorrectly.

  • The backend is likely another reverse proxy or load balancer.

Possible solutions:

Examine the backend application logs to determine why a 502 error is returned.

Troubleshooting documentation: HTTP 502 Bad Gateway Errors and Testing TCP and HTTP Backend Servers.

502

Error messages:
  • lbStatusCode: "502"

  • backendStatusCode: ""

  • No healthy backends available in associated backend set

Oracle Cloud Infrastructure log category: Access log and error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The backend health check fails.

  • No traffic observed to a specific backend or all backends.

Possible causes:
  • A backend application is not responding to the health check with the expected response.

  • If no error occurs from the backend, then a TCP health check is configured.

  • A single backend or all backends are configured in drain mode.

Possible solutions:
  • Determine why TCP health check is failing.

  • Convert to HTTP health check.

  • Change the drain mode to false (undrain) for a given backend or all backends.

Troubleshooting documentation: HTTP 502 Bad Gateway Errors and Testing TCP and HTTP Backend Servers.

Session Persistence Issue

Error message:
Persistence selected backend ip_address which failed and no_fallback is selected

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • Session persistence is failing.

Possible causes
  • Backend set is configured with session persistence and the expected backend is not available because the connection failed or timed out.

  • Fallback option is disabled.

Possible solutions:
  • Determine why backend application is not reachable.

  • Enable fallback option in case the selected server is unavailable.

Troubleshooting documentation: Fallback

For all other 5nn errors, the most likely causes are issues with the backend server.

Client Errors (400-499)

400

Error messages:
  • lbStatusCode: "400"

  • backendStatusCode: ""

  • 400 bad request header or cookie too large

Oracle Cloud Infrastructure log category: Access log

Symptoms:
  • The load balancer returns a status code 400.

  • The backend does not return a status code.

Possible causes:

The client is sending a request that exceeds the configured buffer size.

Possible solutions:

Increase the HTTP request header size on the load balancer. By default, the size limit is 8 KB but raising it to 64 KB resolves the issue.

Troubleshooting documentation: HTTP Header Rules

404, 404

Error messages:
  • lbStatusCode: "404"

  • backendStatusCode: "404"

Oracle Cloud Infrastructure log category: Access log

Symptoms:
  • The load balancer returns a 404 status code.

  • The backend returns a 404 status code.

Possible causes:

The expected page does not exist on the backend.

Possible solutions:
  • Create the missing page.

  • Configure the client to call the correct page.

403, 403

Error messages:
  • lbStatusCode: "403"

  • backendStatusCode: "403"

Oracle Cloud Infrastructure log category: Access log

Symptoms:
  • The load balancer returns a 403 status code.

  • The backend returns a 403 status code.

Possible causes:
  • Expected page does not have sufficient permission on the backend.

  • Expected authentication token is missing or not being forwarded.

Possible solutions:

  • Create missing permissions on backend.

  • Adjust client configuration to ensure that tokens are sent properly.

  • Ensure that all tokens being sent are arriving at the backend.

  • If the header is missing:

    • Adjust header size on the load balancer or the client.

    • Allow headers with special characters.

Troubleshooting documentation: HTTP Header Rules

Health Check Errors

No Healthy Backends

Error message:
No healthy backends available in associated backendSet

Oracle Cloud Infrastructure log category: Error log

Symptoms:

The client fails with a 502 Bad Gateway error.

Possible causes:
  • No backends in the backend set.

  • No backends responding to health check.

Possible solutions:
  • Determine why backends aren't responding to health check.

  • Check and adjust any health check settings, including status code, regular expressions, interval timeout, port, and protocol.

Troubleshooting documentation: Editing a Load Balancer's Health Check Policies

Status Code Issues

Backend health status failure reason: Status code mismatch

Oracle Cloud Infrastructure category: Backend Health Status

Error message:
Status code mismatch

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The backend fails the health check.

  • The client fails with a 502 Bad Gateway error.

  • invalid statusCode appears in the error logs.

Possible causes:
  • The backend is responding with an incorrect response code.

  • The backend health check fails because of response code mismatch.

  • The health check failures are because of an unexpected status code in the regular expression body.

Possible solutions:
  • Determine why the backend is sending the incorrect response code.

  • Adjust the path or status code of the health check to match the backend.

Troubleshooting documentation: Editing a Load Balancer's Health Check Policies

Response Match Failed

Backend Health Status Failure Reason: Regular expression mismatch

Oracle Cloud Infrastructure category: Backend Health Status

Error message:
"response match result: failed"

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The backend fails the health check.

  • The client fails with a 502 Bad Gateway error.

  • "response match result: failed" appears in the error logs.

Possible causes:

The backend health check fails because of regular expression mismatch, incorrect value returned, or incorrect value provided to the health check.

Possible solutions:
  • Determine why the backend is sending the incorrect body.

  • Adjust the path or regular expression pattern of the health check to match the backend.

Troubleshooting documentation: Editing a Load Balancer's Health Check Policies

Unreachable Host

Backend Health Status Failure Reason: Connection failed

Oracle Cloud Infrastructure category: Backend Health Status

Error messages:
"errno":"EHOSTUNREACH","syscall":"connect"
"ECONNREFUSED","errno":"ECONNREFUSED"

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The backend fails the health check.

  • The client fails with a 502 Bad Gateway error.

  • "EHOSTUNREACH" appears in error logs.

Possible causes:
  • The backend health check fails because of an unreachable host.

  • The backend health check fails because of a connection reset.

  • An application or firewall is actively refusing the connection.

Possible solutions:
  • Check the local instance firewall to confirm that traffic is being allowed.

  • Check the local instance to confirm that the application is running.

  • Check the network security group and security lists to confirm that traffic is allowed.

Troubleshooting documentation: Access and Security

Health Status Issues

Error messages:
"healthStatus":"Unhealthy to Healthy"
"healthStatus":"Healthy to Unhealthy"

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client behaves as expected but fails periodically.

  • The backend switches between passing and failing the health check.

  • "Unhealthy to Healthy" or "Healthy to Unhealthy" appears in error logs.

Possible causes:
  • An unhealthy backend becomes healthy.

  • If the health status of the backend changes often, it can indicate a chronic problem.

Possible solutions:
  • Ensure that the instance is not changing health status abnormally.

  • Check application logs on the backend server for any application-specific issues.

Connection Issues

Backend Health Status Failure Reason: Timed out

Oracle Cloud Infrastructure category: Backend Health Status

Error messages:
"msg":"connect timed out","elapsed":3000}

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The backend is periodically or chronically failing health checks.

  • "connect timed out" appears in the error logs.

Possible causes:
  • The backend server is not responding to health checks in the expected time period.

  • Slow upstream dependency including, database, application service or API, or slow storage services, such as Oracle Cloud Infrastructure File Storage service, Elastic Block Store, or Object Storage.

Possible solutions:
  • Perform a local test to the backend to eliminate the load balancer as a cause.

  • Check the performance of all upstream dependencies.

  • Check application logs on the backend server for any dependencies reporting any sort of timeout.

Troubleshooting documentation: Testing TCP and HTTP Backend Servers.

SSL Errors

SSL Virtual Listener Issues

Error message:
Not all SSL virtual listeners on port 443 have the same set of SSL protocols defined

Symptoms:

You cannot create backends for an existing load balancer nor can you add new servers to the backend created previously within the same load balancer.

Possible causes:

Mismatch of transport layer security (TLS) versions.

Possible solutions:

Match TLS versions on the listeners.

Troubleshooting documentation: SSL Certificates for Load Balancers

SSL Handshake Issues

Error message:
(SSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) while SSL handshake error

Oracle Cloud Infrastructure log category: Client log

Symptoms:

The client experiences SSL handshake failures in Load Balancer metrics (see Load Balancer Metrics).

Possible causes:

The backend is not configured to accept SSL.

Possible solutions:
  • Confirm that the backend certificate matches the certificate authority that is provided.

  • Ensure that all certificates in the chain are provided in the correct order in the Certificate field.

  • Ensure that you provide the correct certificate depth.

Troubleshooting documentation: SSL Certificates for Load Balancers

Backend SSL Handshake Issues

Error messages:
Peer backend_ip_address closed connection in SSL handshake

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The client experiences SSL handshake failures in Oracle Cloud Infrastructure metrics (see Load Balancer Metrics).

Possible causes:
  • The backend is not configured to accept SSL.

  • The backend certificate is invalid.

Possible solutions:
  • Confirm that the backend certificate matches the certificate authority that is provided.

  • Ensure that all certificates in the chain are provided in the correct order in the Certificate field.

  • Ensure that you provide the correct certificate depth.

Troubleshooting documentation: SSL Certificates for Load Balancers

SSL Certificate Issues

Error:

Client backend_ip_address has SSL certificate verify error.

Oracle Cloud Infrastructure log category: Error log

Symptoms:

The client experiences SSL handshake failures in Oracle Cloud Infrastructure metrics (see Load Balancer Metrics).

Possible causes:
  • The client certificate is invalid.

  • The client certificate is not trusted.

  • Invalid peer certification verify depth.

Possible solutions:
  • Ensure that the client certificate is valid.

  • Remove Peer Cert Verify feature on the listener.

Troubleshooting documentation: Key Pair Mismatch and Private Key Consistency.

Client SSL Certificate Issues

Error message:
Client backend_ip_address sent no required SSL certificate

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client experiences a 400 Response error.

  • no required SSL certificate appears in error logs.

Possible causes:

The client is not sending a client certificate.

Possible solutions:
  • Update the client to send the correct client certificate.

  • Remove Peer Cert Verify feature on the listener.

  • Adjust the certificate verification depth.

Troubleshooting documentation: Configuring Peer Certificate Verification.

SSL Error Causes Backend Health Check Failure

Error message:
"code":"EPROTO","errno":"EPROTO"

Oracle Cloud Infrastructure log category: Error log

Symptoms:

The backend health check fails because of the SSL error.

Possible causes:

The backend is configured to accept SSL but the health check protocol selected does not match that of the backend.

Possible solutions:

Confirm that you are using non-TLS health check on a backend that has TLS enabled.

Troubleshooting documentation: Editing a Load Balancer's Health Check Policies

SSL Host Name Verification Fails

Error message:
SSL host name verification failed for host_name

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • Error message contains SSL host name verification failed.

Possible causes:

Host name provided does not match what is expected.

Possible solutions:
  • Configure client to use the expected host name.

  • Configure certificate to match the host name sent by the client.

Troubleshooting documentation: SSL Certificates for Load Balancers

Client-Side Errors

Client Access Denied

Error:

Access for client_ip_address denied by HTTP ACL rule.

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The backend doesn't pass health check.

  • forbidden by HTTP ACL rule appears in the error log.

Possible causes:

Access control rule set is enabled bud doesn't include the source IP address.

Possible solutions:

Check and apply respective rule set to include the source IP address.

Troubleshooting documentation: Access Control Rules

Client Timeout Issue

Error:

Client client_name timed out

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The client experiences SSL handshake failures in Oracle Cloud Infrastructure metrics (see Load Balancer Metrics).

Possible causes:

The client terminated the connection sooner than the configured timeout for the load balancer.

Possible solutions:
  • Configure client timeout to match expected application configuration.

  • Determine why the backend server didn't respond in the configured amount of time.

Troubleshooting documentation: Testing TCP and HTTP Backend Servers.

Client Connection Closed Abruptly

Error:

Connection to address was abruptly closed by

Oracle Cloud Infrastructure log category: Error log

Symptoms:

The client fails with a 502 Bad Gateway error.

Possible causes:

The listener has an Max listener connection rule and an IP tried to make more connections to the listener than allowed by the rule.

Possible solutions

  • Increase the allowed number of connections an IP can make to the listener.

  • Reduce the number of connections the IP is making to the listener.

Troubleshooting documentation: Max Listener Connection Rules.

Backend Server Errors

Backend Server Connection Issue

Error:

Backend ip_address abruptly closes connection.

Oracle Cloud Infrastructure log category: Error log

Symptoms:
  • The client fails with a 502 Bad Gateway error.

  • The client reports IO error in load balancer metrics.

  • The backend server set uses HTTPS and the cipher suites or TLS versions aren't compatible.

Possible causes:

  • The backend server connection timeout is configured incorrectly, with a lower timeout value than the load balancer.

  • The backend server or its containing backend set has maxConnections set and the number of connections to the backend server has reached the specified limit.

Possible solutions:

  • Determine why the backend server application is timing out.

  • If the backend server timeout value needs to be adjusted, then adjust it to be greater than the load balancer timeout value.

  • Add more backend servers to handle the load.

  • Increase the maxConnections setting.

No Healthy Backend Servers

Error:

No healthy backends available in associated backendSet

Oracle Cloud Infrastructure log category: Error log

Symptoms:

The client fails with a 502 Bad Gateway error.

Possible causes:

  • No backend servers in the backend set.

  • No backend servers responding to health check.

  • All health backend servers in the backend set have reached their maxConnections limit.

Possible solutions:

  • Determine why backends aren't responding to health check.

  • Check and adjust any health check settings, including status code, regular expressions, interval timeout, port, and protocol.

  • Check if backend servers have a maxConnections limit set. If so, add more backend servers to handle the load or increase the maxConnections limit

Troubleshooting documentation: Testing TCP and HTTP Backend Servers.