Oracle Cloud Infrastructure Documentation

Adding a Model to a Private Endpoint

Learn how to associate a model endpoint for a pretrained, custom, or imported model with a private endpoint in OCI Generative AI.

You can associate one or more model endpoints with a private endpoint. A private endpoint provides the private network path, while each model endpoint provides access to a model on a hosting dedicated AI cluster. Each model endpoint counts against the remaining endpoint capacity of its hosting cluster.

  • Create Endpoint

    1. On the Private Endpoints list page, select the private endpoint that you want to work with. If you need help finding the list page for private endpoints, see Listing Private Endpoints.
    2. Select Endpoints and then select Create endpoint.

    Endpoint Information

    1. Select a compartment to create the endpoint in. The default compartment is the same as the list page, but you can select any compartment that you have permission to work in.
      Tip

      We recommend that you create the endpoint in the same compartment as the model.
    2. (Optional) Enter a name for the endpoint. Start the name with a letter or underscore, followed by letters, numbers, hyphens, or underscores. The length can be 1 to 255 characters. If you don't enter a name, the system generates a name that you can change later.
      The generated name has the format generativeaiendpoint<timestamp>. Example: generativeaiendpoint20250531235319
    3. (Optional) Enter a description for the model.

    Hosting configuration

    1. Select the compartment that hosts the model that you want to add an endpoint to.
    2. Select the model that you want to add an endpoint to. This model can be a custom model, imported model, or a ready-to-use pretrained foundational model available in the region that you're working in.
    3. If the model that you selected has several versions, select a model version.
      For the ready-to-use pretrained foundational models, this field populates when you select the model.
    4. Select a hosting dedicated AI cluster by performing one of the following actions:
      • Select a Dedicated AI cluster from the list. If you created a cluster a few minutes ago, wait for that cluster to become active. Ensure that the cluster was created for the selected pretrained or imported model, or for the base model used to create the selected custom model.
      • Select Create new dedicated AI cluster and perform the following steps:
        1. (Optional) Enter a name and description.
        2. For Base model, select the pretrained model that you're hosting, the base model used to create the selected custom model, or the selected imported model.
        3. If you selected an imported model, select a recommended Unit size. See Hardware Unit Shapes for Imported Models.
        4. Use at least one model replica for the hosting cluster. The cluster's replicas serve its active model endpoints. Add replicas to increase inference throughput for the endpoints; creating additional endpoints alone doesn't increase throughput.
        5. Review the billing and commitment information for the hosting dedicated AI cluster and, if prompted, select the checkbox to agree to the commitment. Hosting clusters for imported models don't require the 744-unit-hour minimum commitment that applies to OCI Generative AI pretrained and custom models.
        6. (Optional) Select Add tag and assign tags to this dedicated AI cluster. See Resource Tags.
        7. Select Create and wait for the cluster to become active.
        8. From the Dedicated AI cluster list, select the dedicated AI cluster that you created.

    Networking resources

    The private endpoint is preselected.

    Guardrails (for pretrained and custom models)

    Guardrails aren't available for imported models.

    1. Select whether to enable the following guardrail.
      • Content moderation
        • Off: Don't apply content moderation and output explicit content.
        • Block: Help identify and apply content moderation.
        • Inform: Don't apply content moderation, but aim to inform the user if the model detects content that needs moderation.
    2. (Optional) Select Add tag and assign tags to this endpoint. See Resource Tags.
    3. Select Create.
      You're directed to the endpoint details page where you can track the state of the endpoint. The new model endpoint counts against the hosting cluster's remaining endpoint capacity.

  • Use the endpoint create command and required parameters to create an endpoint:

    oci generative-ai endpoint create 
--model-id <model-OCID>
--compartment-id <compartment-OCID> 
--dedicated-ai-cluster-id <hosting-dedicated-AI-cluster-OCID> 
[OPTIONS]

    For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

    Note

    For pretrained models, instead of an OCID, you can use the model name exactly as listed in the Console's playground. You can also find this OCI model name, in the model's detail page in Offered Pretrained Foundational Models in Generative AI.

  • Run the CreateEndpoint operation to create an endpoint.

    Note

    For pretrained models, instead of an OCID, you can use the model name exactly as listed in the Console's playground. You can also find this OCI model name, in the model's detail page in Offered Pretrained Foundational Models in Generative AI.