Sometimes it seems appropriate to let a text generator make important and irreversible decisions about your life. Other times you just want to let it control a web UI.
I’ve been experimenting with giving LLMs the ability to control a web app’s interface as it runs. Not just streaming text into a chat window, but actually invoking JavaScript functions directly in the browser.
It started with something small: giving an LLM a tool to refresh a list. If it decided a refresh was warranted, it would call the tool, the backend would broadcast a message over SSE, and the client would receive it and fire an HTMX request. That got me thinking. What if I could define a whole set of tools that lived in the frontend and simply tell the LLM they existed?
What Are Tools to an LLM?
Tools are how LLMs escape the confines of a text box. From emailing sensitive data to all your contacts to deleting your entire filesystem, tools give models real capability.
The canonical example is a get_time_now() function. LLMs are notoriously bad at knowing the current date; they’re trained on historical data and have no concept of now. Without a tool, ask one what day it is and it will confidently tell you something wrong. Give it a function that returns today’s date and suddenly it stops guessing. The model sees the tool, calls it, and grounds its response in reality. Simple, but it illustrates the pattern perfectly.
The Problem (and the Idea)
What if you defined a set of frontend tools, functions that live in the browser, and passed them to an LLM the same way you’d pass any other tool? The model wouldn’t know or care that calling drop_pin(lat, lng) moves a map marker instead of writing to a database. It would just call it.
That’s the experiment.
The Demo
The demo app is a simple notes-and-map interface, not something you’d actually ship but a decent sandbox for the concept. The baseline functionality was adding, deleting, and refreshing notes. The map came later, and I think it’s the more compelling showcase: watching the LLM switch tabs and drop pins gives it that copilot quality that makes human-in-the-loop apps feel genuinely useful rather than gimmicky.
Note: the parts of the video where I’m typing are sped up. Everything else is real time. (I type slowly ;p)
How Normal Tool Use Works
In a standard LLM integration, tools are defined as JSON schemas and passed to the model alongside the user’s message. When the model decides a tool is relevant, it doesn’t call it directly. It returns a structured response describing which tool it wants to use and with what arguments. Your backend intercepts that, executes the function, and feeds the result back into the conversation.
Frontend tools follow exactly the same protocol. Same schema format, same turn-based handshake, and they can be mixed freely with regular backend tools in a single request. The only thing that changes is where the execution happens.
Defining Frontend Tools
Frontend tools are defined using the standard tool schema format, the same as any other tool. Their names are then used in the tool execution handler to route calls to the frontend dispatcher. This way the LLM can use both backend and frontend tooling simultaneously.
FRONTEND_TOOLS = [
...
{
"type": "function",
"function": {
"name": "open_console",
"description": "Open the LLM console panel so the user can see log output. Use close_console to hide it.",
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "drop_pin",
"description": "Drop a marker on the map at a given location with an optional label.",
"parameters": {
"type": "object",
"properties": {
"lat": {"type": "number", "description": "Latitude"},
"lng": {"type": "number", "description": "Longitude"},
"label": {"type": "string", "description": "Popup label shown on the marker (optional)"},
},
"required": ["lat", "lng"],
},
},
},
...
]
FRONTEND_TOOL_NAMES = {t["function"]["name"] for t in FRONTEND_TOOLS}
Dispatching to the Frontend
When a tool call comes in, the backend checks whether it’s a frontend tool. If it is, instead of executing a function on the server it publishes the tool call to the frontend over SSE. If it’s a backend tool it executes normally.
def _execute(name: str, arguments: dict) -> str:
if name in FRONTEND_TOOL_NAMES:
publish({"type": "tool_call", "tool": name, "args": arguments})
return json.dumps({"status": "dispatched"})
return execute_tool(name, arguments)
This is fire and forget for now. The backend doesn’t wait for acknowledgement from the frontend. For a production implementation you’d want some form of confirmation, but for this demo it works reliably enough.
The Relay Object
On the frontend, a small Relay class handles the SSE connection and routes incoming tool calls to registered handler functions. Handlers are registered using the .on() method, keeping the interface clean and readable.
export class Relay {
constructor() {
this._handlers = new Map()
}
on(name, fn) {
this._handlers.set(name, fn)
return this
}
connect() {
this._startSSE()
return this
}
_startSSE() {
...
es.addEventListener('tool_call', (event) => {
const data = JSON.parse(event.data)
this._dispatch(data)
})
...
}
_dispatch({ tool, args }) {
this._handlers.get(tool)?.(args)
}
}
Here is the Relay instance from the demo app, showing all the registered handlers:
window._relay = new Relay()
.registerHtmxTriggers()
.on("open_console", () => {
if (document.getElementById("console-panel").classList.contains("collapsed")) toggleConsole();
})
.on("close_console", () => {
if (!document.getElementById("console-panel").classList.contains("collapsed")) toggleConsole();
})
.on("log", ({ message }) => appendToConsole(message))
.on("refresh_note_list", () => loadNotes())
.on("set_dark_mode", ({ dark }) => setDark(dark))
.on("go_to_coordinates", ({ lat, lng, zoom = 14 }) =>
map?.flyTo([lat, lng], zoom),
)
.on("drop_pin", (args) => dropPin(args))
.connect();
HTMX Integration
This implementation works well with HTMX too. By defining a set of named triggers and passing them to the LLM with descriptions, the model gains the ability to fire HTMX requests directly. In the example below, swap-to-map and swap-to-notes each trigger an hx-get request for their respective elements.
Frontend tools for HTMX triggers are defined on the backend:
FRONTEND_TOOLS = [
{
"type": "function",
"function": {
"name": "htmx_trigger",
"description": (
'Fire an HTMX action by name. Available triggers: '
'"swap-to-map" — Switch the left panel to the map view; '
'"swap-to-notes" — Switch the left panel back to the note list'
),
"parameters": {
"type": "object",
"properties": {
"trigger": {"type": "string", "description": "One of: swap-to-map, swap-to-notes"},
},
"required": ["trigger"],
},
},
},
...
]
The Relay class handles this with a single generic method:
export class Relay {
...
registerHtmxTriggers() {
return this.on('htmx_trigger', ({ trigger }) => {
document.querySelector(`[data-trigger="${trigger}"]`)?.dispatchEvent(new Event(trigger))
})
}
...
}
The SSE Tab Limit Problem
One issue I ran into is the HTTP connection limit browsers enforce per domain. An SSE connection holds open a long-lived HTTP connection and browsers support up to six of them. Open a seventh tab and it hangs.
The solution uses two browser APIs: the Web Locks API and BroadcastChannel.
The idea is straightforward. Each tab races to acquire a named lock. Whichever tab wins becomes the leader and opens the SSE connection. When a tool call arrives, the leader dispatches it locally and broadcasts it to all other tabs via BroadcastChannel. If the leader tab closes, the lock is released and another tab picks it up, reopening the connection automatically.
Only one SSE connection is ever open per domain, regardless of how many tabs are open.
_startSSE() {
// navigator.locks ensures only one tab holds the SSE connection
navigator.locks.request('relay_lock', () => {
const es = new EventSource('/api/events/')
let release
const held = new Promise((resolve) => { release = resolve })
es.addEventListener('tool_call', (event) => {
const data = JSON.parse(event.data)
// Call this _dispatch since BroadcastChannel doesn't deliver to the sender
this._dispatch(data)
// Notify other tabs
this._channel.postMessage(data)
})
es.onerror = () => {
es.close()
// Release the lock before retrying. Without this, the tab holds the
// lock forever while also queuing a new request for it, deadlocking
// all other tabs that try to open.
release()
setTimeout(() => this._startSSE(), 3000)
}
// Close the SSE connection on unload so the server frees the thread immediately.
// The browser releases the lock naturally when the tab is gone.
window.addEventListener('beforeunload', () => es.close())
return held
})
// Other tabs receive via BroadcastChannel
this._channel.onmessage = ({ data }) => this._dispatch(data)
}
Gunicorn Configuration
SSE requires long-lived connections, which means threads are held open for the duration. The default Gunicorn sync workers time out and close these connections. The fix is to use threaded workers with a timeout of zero.
web:
build: .
command: sh -c "python manage.py migrate --no-input && gunicorn demo.wsgi:application --bind 0.0.0.0:8000 --worker-class gthread --workers 1 --threads 50 --timeout 0"
...
--worker-class gthread switches to threaded workers, --threads 50 allows up to 50 concurrent connections, and --timeout 0 disables the worker timeout so long-lived SSE connections aren’t killed.
Existing Solutions
AG-UI is a full protocol for agent-user interaction, backed by CopilotKit and supported by LangGraph and CrewAI. It handles streaming, shared state, multi-agent coordination, cancellation, and more. It’s the right choice for complex agentic applications.
This is not that. This is a minimal pattern for Django developers who want to add LLM frontend tool calling to an existing app without adopting a new protocol or framework. If you have a Django app with a chat interface and you want the LLM to be able to manipulate your UI, this gets you there with a small amount of code and no new dependencies.
What’s Missing
Two obvious things worth adding for anything beyond a demo: WebSockets instead of SSE for lower latency and genuine bidirectional communication, and acknowledgement from tool calls so the backend knows the frontend actually received and acted on them.
Conclusion
There are better solutions to this, and there will be more. But I think the concept is genuinely interesting. Existing protocols are more robust and holistic, though anyone who has worked on legacy systems knows you can’t always bolt a new technology onto an existing stack. This pattern earns its place when you have one or two tools you’d like to give the LLM access to and the system already has SSE or WebSockets in place. In that case the whole thing comes together with very little code.
Repo
Requires Docker and Docker Compose. Create a .env file using the example, add your OpenAI API key, and give it a try.