How I Built a Lightweight Bridge for LLM Frontend Tool Calling in Django
I’ve been experimenting with giving LLMs the ability to control a web app’s interface as it runs. Not just streaming text into a chat window, but actually invoking JavaScript functions directly in the browser.
During a recent project, I needed to give an LLM the ability to filter and refresh a list, if it felt the need to. And after implementing that, it got me thinking… wouldn’t it be great if I could just define whatever javascript functions I wanted in the front-end and make them available to the LLM at the backend.
I have put together a small local development app and also this finely written blog post to demonstrate this exact idea.
The Design
- A set of frontend tools are defined and passed to the LLM alongside any backend tools.
- When the model calls a front end tool, the function name and any arguments are broadcast to the browser using SSE.
- The frontend receives the event, looks up the right listener by tool name, and calls it with the arguments.
The Demo
To demonstrate this in action, a simple notes-and-map app was created with the classic chat bot interface. The baseline functionality was adding, deleting, and refreshing notes, but to really showcase the power of this idea I added a map on a separate tab. Telling the AI to “switch to the map and put a pin on [a place]”, then seeing the map open and a pin drop on that location feels very interactive. But it’s so simple! And really if you already have SSE or websockets set up you can implement this very easily.
See the video of the demo app in action.
Note: the parts of the video where I’m typing are sped up. Everything else is real time. (I type slowly)
Defining Frontend Tools
Frontend tools are defined using the standard tool schema, the same as any other tool. They are even merged with the normal backend tooling but are executed differently if they are in the list of FRONTEND_TOOL_NAMES.
FRONTEND_TOOLS = [
...
{
"type": "function",
"function": {
"name": "open_console",
"description": "Open the LLM console panel so the user can see log output. Use close_console to hide it.",
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "drop_pin",
"description": "Drop a marker on the map at a given location with an optional label.",
"parameters": {
"type": "object",
"properties": {
"lat": {"type": "number", "description": "Latitude"},
"lng": {"type": "number", "description": "Longitude"},
"label": {"type": "string", "description": "Popup label shown on the marker (optional)"},
},
"required": ["lat", "lng"],
},
},
},
...
]
FRONTEND_TOOL_NAMES = {t["function"]["name"] for t in FRONTEND_TOOLS}
Dispatching to the Frontend
When a tool call comes in, the backend checks whether it’s a frontend tool. If it is, instead of executing a function on the server it publishes the tool call to the frontend using SSE. If it’s a backend tool it executes normally.
def _execute(name: str, arguments: dict) -> str:
if name in FRONTEND_TOOL_NAMES:
publish({"type": "tool_call", "tool": name, "args": arguments})
return json.dumps({"status": "dispatched"})
return execute_tool(name, arguments)
Ideally we would have an acknowledge mechanism here but for a proof-of-concept the fire and forget approach is fine.
The Relay Object
On the frontend, a small Relay class handles the SSE connection and routes incoming tool calls to registered handler functions. Handlers are registered using the .on() method, keeping the interface clean and readable. Note: this is not the full class definition, just showing the relevant parts
export class Relay {
constructor() {
// initialise a map to store our handlers
this._handlers = new Map()
}
on(name, fn) {
this._handlers.set(name, fn)
return this
}
connect() {
this._startSSE()
return this
}
_startSSE() {
...
// when a tool call event is received forward it to _dispatch
es.addEventListener('tool_call', (event) => {
const data = JSON.parse(event.data)
this._dispatch(data)
})
...
}
// dispatch the tool call with arguments to the relevant handler
_dispatch({ tool, args }) {
this._handlers.get(tool)?.(args)
}
}
Front-end listeners
Here is the Relay instance from the demo app. Each one of those listeners, open_console, close_console, etc. has been defined at the back end. So when the LLM calls that tool and it is transmitted by SSE to the front end, this is where it ends up.
window._relay = new Relay()
.registerHtmxTriggers()
.on("open_console", () => {
if (document.getElementById("console-panel").classList.contains("collapsed")) toggleConsole();
})
.on("close_console", () => {
if (!document.getElementById("console-panel").classList.contains("collapsed")) toggleConsole();
})
.on("log", ({ message }) => appendToConsole(message))
.on("refresh_note_list", () => loadNotes())
.on("set_dark_mode", ({ dark }) => setDark(dark))
.on("go_to_coordinates", ({ lat, lng, zoom = 14 }) =>
map?.flyTo([lat, lng], zoom),
)
.on("drop_pin", (args) => dropPin(args))
.connect();
HTMX Integration
You might have also noticed registerHtmxTriggers() in there as well. This relay object works really well with HTMX. Using a generic function the LLM can send HX triggers to the front end, giving the model the ability to fire HTMX requests directly. In the example below, swap-to-map and swap-to-notes each trigger a hx-get request for their respective elements. By giving them a description the LLM has context to know when the user says “open the map”, that it probably wants to call the htmx_trigger tool with the value “swap-to-map”.
FRONTEND_TOOLS = [
{
"type": "function",
"function": {
"name": "htmx_trigger",
"description": (
'Fire an HTMX action by name. Available triggers: '
'"swap-to-map" — Switch the left panel to the map view; '
'"swap-to-notes" — Switch the left panel back to the note list'
),
"parameters": {
"type": "object",
"properties": {
"trigger": {"type": "string", "description": "One of: swap-to-map, swap-to-notes"},
},
"required": ["trigger"],
},
},
},
...
]
The Relay class handles this with a single generic method:
export class Relay {
...
registerHtmxTriggers() {
return this.on('htmx_trigger', ({ trigger }) => {
document.querySelector(`[data-trigger="${trigger}"]`)?.dispatchEvent(new Event(trigger))
})
}
...
}
The SSE Tab Limit Problem
One issue I ran into is the HTTP connection limit browsers enforce per domain. An SSE connection holds open a long-lived HTTP connection and browsers support up to six of them. Open a seventh tab and it hangs.
Web Lock & BroadcastChannel APIs
The solution uses two browser APIs: the Web Locks API and BroadcastChannel.
The idea is straightforward. Each tab races to acquire a named lock. Whichever tab wins becomes the leader and opens the SSE connection. When a tool call arrives, the leader broadcasts it to all other tabs via BroadcastChannel. If the leader tab closes, the lock is released and another tab picks it up, reopening the connection automatically. This means only one SSE connection is ever open, regardless of how many tabs are open.
_startSSE() {
// navigator.locks ensures only one tab holds the SSE connection
navigator.locks.request('relay_lock', () => {
const es = new EventSource('/api/events/')
let release
const held = new Promise((resolve) => { release = resolve })
es.addEventListener('tool_call', (event) => {
const data = JSON.parse(event.data)
// Call this _dispatch since BroadcastChannel doesn't deliver to the sender
this._dispatch(data)
// Notify other tabs
this._channel.postMessage(data)
})
es.onerror = () => {
es.close()
// Release the lock before retrying. Without this, the tab holds the
// lock forever while also queuing a new request for it, deadlocking
// all other tabs that try to open.
release()
setTimeout(() => this._startSSE(), 3000)
}
// Close the SSE connection on unload so the server frees the thread immediately.
// The browser releases the lock naturally when the tab is gone.
window.addEventListener('beforeunload', () => es.close())
return held
})
// Other tabs receive via BroadcastChannel
this._channel.onmessage = ({ data }) => this._dispatch(data)
}
Gunicorn Configuration
This might not be relevant to all projects but caught me out. SSE requires long-lived connections, which means threads are held open for the duration. The default Gunicorn sync workers time out and close these connections. The fix is to use threaded workers with a timeout of zero.
web:
build: .
command: sh -c "python manage.py migrate --no-input && gunicorn demo.wsgi:application --bind 0.0.0.0:8000 --worker-class gthread --workers 1 --threads 50 --timeout 0"
...
--worker-class gthread switches to threaded workers, --threads 50 allows up to 50 concurrent connections, and --timeout 0 disables the worker timeout so long-lived SSE connections aren’t killed.
What’s Missing
I used SSE because that is what I used when I originally met this idea, and I also wanted to shoehorn my SSE multiple tabs solution into this post. But I think this could be used more effectively with websockets. It’s a bit more setup but due to the full duplex communication it would allow a proper acknowledge mechanism and even make it so the model could query information about the frontend.
Existing Solutions
There are of course, as with every possible idea someone might have about AI, many existing solutions.
AG-UI
A protocol by CopilotKit that standardises how an agent backend and frontend communicate in real time. Any agent framework and any frontend can interoperate without custom glue code.
A2UI
A Google spec that lets agents respond with UI components instead of text. The agent sends a JSON description of a form, chart, button etc. and the client renders it natively.
Browser Use
A Python library that gives an LLM access to any website. It reads the DOM, clicks things, fills forms. You give it a task in plain English and it figures out the steps.
Conclusion
The existing tools for agentic AI controlling the UI are incredibly impressive and just playing around by my comparatively very simple implementation I can only imagine the type of user interfaces that can be achieved with these tools. But that doesn’t mean this simple idea doesn’t have a place. Sometimes you just want to extend the functionality of what you already have without wrapping your head around a whole new framework. If SSE or WebSockets are already in place, this pattern slots in with minimal code and no new dependencies. The concept is simple enough to understand in an afternoon and an easy way to enhance the capability of your app.
Repo
Check out the repo below for the demo app and give it a go yourself!