diff --git a/content/docs/02-foundations/02-providers-and-models.mdx b/content/docs/02-foundations/02-providers-and-models.mdx index 51ab862e940b..29788199a978 100644 --- a/content/docs/02-foundations/02-providers-and-models.mdx +++ b/content/docs/02-foundations/02-providers-and-models.mdx @@ -53,6 +53,7 @@ You can also use the [OpenAI Compatible provider](/providers/openai-compatible-p - [LM Studio](/providers/openai-compatible-providers/lmstudio) - [Baseten](/providers/openai-compatible-providers/baseten) +- [Heroku](/providers/openai-compatible-providers/heroku) Our [language model specification](https://github.com/vercel/ai/tree/main/packages/provider/src/language-model/v2) is published as an open-source package, which you can use to create [custom providers](/providers/community-providers/custom-providers). diff --git a/content/providers/02-openai-compatible-providers/45-heroku.mdx b/content/providers/02-openai-compatible-providers/45-heroku.mdx new file mode 100644 index 000000000000..2ad7f60641ed --- /dev/null +++ b/content/providers/02-openai-compatible-providers/45-heroku.mdx @@ -0,0 +1,119 @@ +--- +title: Heroku +description: Use a Heroku OpenAI compatible API with the AI SDK. +--- + +# Heroku Provider + +[Heroku](https://heroku.com/) is a cloud platform that allows you to deploy and run applications, including AI models with OpenAI API compatibility. +You can deploy models that are OpenAI API compatible and use them with the AI SDK. + +## Setup + +The Heroku provider is available via the `@ai-sdk/openai-compatible` module as it is compatible with the OpenAI API. +You can install it with + + + + + + + + + + + + + +### Heroku Setup + +1. Create a test app in Heroku: + +```bash +heroku create +``` + +2. Inference using claude-3-5-haiku: + +```bash +heroku ai:models:create -a $APP_NAME claude-3-5-haiku +``` + +3. Export Variables: + +```bash +export INFERENCE_KEY=$(heroku config:get INFERENCE_KEY -a $APP_NAME) +export INFERENCE_MODEL_ID=$(heroku config:get INFERENCE_MODEL_ID -a $APP_NAME) +export INFERENCE_URL=$(heroku config:get INFERENCE_URL -a $APP_NAME) +``` + +## Provider Instance + +To use Heroku, you can create a custom provider instance with the `createOpenAICompatible` function from `@ai-sdk/openai-compatible`: + +```ts +import { createOpenAICompatible } from '@ai-sdk/openai-compatible'; + +const heroku = createOpenAICompatible({ + name: 'heroku', + baseURL: process.env.INFERENCE_URL + '/v1', + apiKey: process.env.INFERENCE_KEY, +}); +``` + +Be sure to have your `INFERENCE_KEY`, `INFERENCE_MODEL_ID`, and `INFERENCE_URL` set in your environment variables. + +## Language Models + +You can create Heroku models using a provider instance. +The first argument is the served model name, e.g. `claude-3-5-haiku`. + +```ts +const model = heroku('claude-3-5-haiku'); +``` + +### Example + +You can use Heroku language models to generate text with the `generateText` function: + +```ts +import { createOpenAICompatible } from '@ai-sdk/openai-compatible'; +import { generateText } from 'ai'; + +const heroku = createOpenAICompatible({ + name: 'heroku', + baseURL: process.env.INFERENCE_URL + '/v1', + apiKey: process.env.INFERENCE_KEY, +}); + +const { text } = await generateText({ + model: heroku('claude-3-5-haiku'), + prompt: 'Tell me about yourself in one sentence', +}); + +console.log(text); +``` + +Heroku language models are also able to generate text in a streaming fashion with the `streamText` function: + +```ts +import { createOpenAICompatible } from '@ai-sdk/openai-compatible'; +import { streamText } from 'ai'; + +const heroku = createOpenAICompatible({ + name: 'heroku', + baseURL: process.env.INFERENCE_URL + '/v1', + apiKey: process.env.INFERENCE_KEY, +}); + +const result = streamText({ + model: heroku('claude-3-5-haiku'), + prompt: 'Tell me about yourself in one sentence', +}); + +for await (const message of result.textStream) { + console.log(message); +} +``` + +Heroku language models can also be used in the `generateObject`, and `streamObject` functions. diff --git a/content/providers/02-openai-compatible-providers/index.mdx b/content/providers/02-openai-compatible-providers/index.mdx index 5e4d879dee67..678e87e531d1 100644 --- a/content/providers/02-openai-compatible-providers/index.mdx +++ b/content/providers/02-openai-compatible-providers/index.mdx @@ -14,6 +14,7 @@ We provide detailed documentation for the following OpenAI compatible providers: - [LM Studio](/providers/openai-compatible-providers/lmstudio) - [NIM](/providers/openai-compatible-providers/nim) - [Baseten](/providers/openai-compatible-providers/baseten) +- [Heroku](/providers/openai-compatible-providers/heroku) The general setup and provider instance creation is the same for all of these providers.