apps/www/_blog/2024-08-21-mozilla-llamafile-in-supabase-edge-functions.mdx
A few months back, we introduced support for running AI Inference directly from Supabase Edge Functions.
Today we are adding Mozilla Llamafile, in addition to Ollama, to be used as the Inference Server with your functions.
Mozilla Llamafile lets you distribute and run LLMs with a single file that runs locally on most computers, with no installation! In addition to a local web UI chat server, Llamafile also provides an OpenAI API compatible server, that is now integrated with Supabase Edge Functions.
<div className="video-container"> <iframe className="w-full" src="https://www.youtube-nocookie.com/embed/_6L-dnBn2wg" title="The Supabase Book by David Lorenz" allow="accelerometer; autoplay; clipboard-write; encrypted-media; fullscreen; gyroscope; picture-in-picture; web-share" allowfullscreen /> </div> <Admonition type="info">Want to jump straight into the code? You can find the examples on GitHub!
</Admonition>Follow the Llamafile Quickstart Guide to get up and running with the Llamafile of your choice.
Once your Llamafile is up and running, create and initialize a new Supabase project locally:
npx supabase bootstrap scratch
If using VS Code, when promptedt Generate VS Code settings for Deno? [y/N] select y and follow the steps. Then open the project in your favoiurte code editor.
Supabase Edge Functions now comes with an OpenAI API compatible mode, allowing you to call a Llamafile server easily via @supabase/functions-js.
Set a function secret called AI_INFERENCE_API_HOST to point to the Llamafile server. If you don't have one already, create a new .env file in the functions/ directory of your Supabase project.
AI_INFERENCE_API_HOST=http://host.docker.internal:8080
Next, create a new function called llamafile:
npx supabase functions new llamafile
Then, update the supabase/functions/llamafile/index.ts file to look like this:
import 'jsr:@supabase/functions-js/edge-runtime.d.ts'
const session = new Supabase.ai.Session('LLaMA_CPP')
Deno.serve(async (req: Request) => {
const params = new URL(req.url).searchParams
const prompt = params.get('prompt') ?? ''
// Get the output as a stream
const output = await session.run(
{
messages: [
{
role: 'system',
content:
'You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.',
},
{
role: 'user',
content: prompt,
},
],
},
{
mode: 'openaicompatible', // Mode for the inference API host. (default: 'ollama')
stream: false,
}
)
console.log('done')
return Response.json(output)
})
Since Llamafile provides an OpenAI API compatible server, you can alternatively use the OpenAI Deno SDK to call Llamafile from your Supabase Edge Functions.
For this, you will need to set the following two environment variables in your Supabase project. If you don't have one already, create a new .env file in the functions/ directory of your Supabase project.
OPENAI_BASE_URL=http://host.docker.internal:8080/v1
OPENAI_API_KEY=sk-XXXXXXXX # need to set a random value for openai sdk to work
Now, replace the code in your llamafile function with the following:
import OpenAI from 'https://deno.land/x/[email protected]/mod.ts'
Deno.serve(async (req) => {
const client = new OpenAI()
const { prompt } = await req.json()
const stream = true
const chatCompletion = await client.chat.completions.create({
model: 'LLaMA_CPP',
stream,
messages: [
{
role: 'system',
content:
'You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.',
},
{
role: 'user',
content: prompt,
},
],
})
if (stream) {
const headers = new Headers({
'Content-Type': 'text/event-stream',
Connection: 'keep-alive',
})
// Create a stream
const stream = new ReadableStream({
async start(controller) {
const encoder = new TextEncoder()
try {
for await (const part of chatCompletion) {
controller.enqueue(encoder.encode(part.choices[0]?.delta?.content || ''))
}
} catch (err) {
console.error('Stream error:', err)
} finally {
controller.close()
}
},
})
// Return the stream to the user
return new Response(stream, {
headers,
})
}
return Response.json(chatCompletion)
})
Note that the model parameter doesn't have any effect here! The model depends on which Llamafile is currently running!
</Admonition>To serve your functions locally, you need to install the Supabase CLI as well as Docker Desktop or Orbstack.
You can now serve your functions locally by running:
supabase start
supabase functions serve --env-file supabase/functions/.env
Execute the function
curl --get "http://localhost:54321/functions/v1/llamafile" \
--data-urlencode "prompt=write a short rap song about Supabase, the Postgres Developer platform, as sung by Nicki Minaj" \
-H "Authorization: $ANON_KEY"
There is a great guide on how to containerize a Lllamafile by the Docker team.
You can then use a service like Fly.io to deploy your dockerized Llamafile.
Set the secret on your hosted Supabase project to point to your deployed Llamafile server:
supabase secrets set --env-file supabase/functions/.env
Deploy your Supabase Edge Functions:
supabase functions deploy
Execute the function:
curl --get "https://project-ref.supabase.co/functions/v1/llamafile" \
--data-urlencode "prompt=write a short rap song about Supabase, the Postgres Developer platform, as sung by Nicki Minaj" \
-H "Authorization: $ANON_KEY"
Access to open-source LLMs is currently invite-only while we manage demand for the GPU instances. Please get in touch if you need early access.
We plan to extend support for more models. Let us know which models you want next. We're looking to support fine-tuned models too!