Bring Your Own Models
If you have models you want to use instead of the cached models provided by Data Science, you can bring them into AI Quick Actions from Object Storage or from Hugging Face by registering the model.
Hugging Face is an open source model repository. You can bring in models from here to use in AI Quick Actions. Hugging Face offers certain gated models that require the acceptance of user agreement. To bring a gated model from Hugging Face into AI Quick Actions, sign in to Hugging Face using the Hugging Face CLI and your Hugging Face token from a terminal inside the Notebook. This is to verify your access to the model. See the Hugging Face guides to see how to sign in with the Hugging Face CLI. If you don't have a Hugging Face token, see this Hugging Face article on security tokens to generate one.
- Register a verified model.
- Register an unverified model.
The difference between a service cached model and a verified model is that, for a verified model, you must register the model in AI Quick Actions before using it.
Service Managed Inference Containers
Three inference containers are available to use with Bring Your Own Model.
For cached and verified models, Data Science has tested which inference container works best with each model and so the inference container can't be chosen. For unverified models, you must decide which inference container is most suitable for each model. Three service managed inference containers are available, one for models compatible with inference engine vLLM 0.5.3 post1 another container for models compatible with TGI 2.0.1, and the third for models compatible with inference framework llama.cpp for models in the GGUF format.
Download Model Artifacts
Before you can bring a model from Object Storage and register it in AI quick Actions, you need to download the model artifacts, such as scoring logic, ML model, and dependent files, from the model source and upload them to a versioned OCI Object Storage bucket. If you don't have a versioned Object Storage bucket, follow these steps to create one. Then you can upload the model artifacts to Object Storage.
Register Verified Models
Data Science has models you can select to use that have been tested.
Follow the steps in Prerequisites. Steps 3 and 4 show different ways of registering a model. Model registration is a necessary process for a model to be brought into AI Quick Actions. For a complete list of parameters and values for AI Quick Actions CLI commands, see AI Quick Actions CLI
This task can't be performed using the API.
Register Unverified Models
Follow these steps to use models that haven't been tested by Data Science.
Follow the steps in Prerequisites. Model registration is a necessary process for a model to be brought into AI Quick Actions. For a complete list of parameters and values for AI Quick Actions CLI commands, see AI Quick Actions CLI
This task can't be performed using the API.