centersl/open-webui-docs
1
0
Fork 0
mirror of https://github.com/open-webui/docs.git synced 2026-03-28 13:38:43 +07:00
Code Issues Packages Projects Releases Wiki Activity
Files
dev
open-webui-docs/docs/features/extensibility/plugin/development/rich-ui.mdx
Jakob Gruber d7d8ea862b api
2026-03-26 14:54:10 +01:00

496 lines
20 KiB
Plaintext
Raw Permalink Blame History

---
sidebar_position: 4
title: "Rich UI Embedding"
---
# Rich UI Element Embedding
Tools and Actions both support rich UI element embedding, allowing them to return HTML content and interactive iframes that display directly within chat conversations. This feature enables sophisticated visual interfaces, interactive widgets, charts, dashboards, and other rich web content — regardless of whether the function was triggered by the model (Tool) or by the user (Action).
When a function returns an `HTMLResponse` with the appropriate headers, the content will be embedded as an interactive iframe in the chat interface rather than displayed as plain text.
## Tool Usage
To embed HTML content, your tool should return an `HTMLResponse` with the `Content-Disposition: inline` header:
```python
from fastapi.responses import HTMLResponse
def create_visualization_tool(self, data: str) -> HTMLResponse:
"""
Creates an interactive data visualization that embeds in the chat.
:param data: The data to visualize
"""
html_content = """
<!DOCTYPE html>
<html>
<head>
<title>Data Visualization</title>
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
</head>
<body>
<div id="chart" style="width:100%;height:400px;"></div>
<script>
// Your interactive chart code here
Plotly.newPlot('chart', [{
y: [1, 2, 3, 4],
type: 'scatter'
}]);
</script>
</body>
</html>
"""
headers = {"Content-Disposition": "inline"}
return HTMLResponse(content=html_content, headers=headers)
```
### Custom Result Context
By default, when a tool returns an `HTMLResponse`, the LLM receives a generic message: `"<tool_name>: Embedded UI result is active and visible to the user."` This gives the model no information about *what* was actually generated.
To provide the LLM with actionable context about the embed, return a **tuple** of `(HTMLResponse, context)` where the second element is a `str`, `dict`, or `list`:
```python
from fastapi.responses import HTMLResponse
def create_chart(self, data: str) -> tuple:
"""
Creates an interactive chart and returns context to the LLM.
:param data: The data to chart
"""
html_content = "<html>...</html>"
headers = {"Content-Disposition": "inline"}
# The LLM receives this context instead of the generic message
result_context = {
"status": "success",
"chart_type": "scatter",
"data_points": 42,
"description": "Scatter plot showing correlation between X and Y"
}
return HTMLResponse(content=html_content, headers=headers), result_context
```
The context can be:
- A **string** — sent as-is to the LLM (e.g., `"Generated a bar chart with 5 categories"`)
- A **dict** — serialized as JSON for structured context
- A **list** — serialized as JSON for multiple items
If the second element is missing or not one of these types, the generic fallback message is used.
:::tip When to use this
This is particularly useful when your tool generates dynamic content and the LLM needs to reference what was generated in follow-up conversation — for example, telling the LLM which parameters were used, what data is being displayed, or what actions the user can take next.
:::
## Action Usage
Actions work exactly the same way. The rich UI embed is delivered to the chat via the event emitter:
**Option A — HTMLResponse:**
```python
from fastapi.responses import HTMLResponse
async def action(self, body, __event_emitter__=None):
html = "<html><body><h1>Dashboard</h1></body></html>"
return HTMLResponse(content=html, headers={"Content-Disposition": "inline"})
```
**Option B — Tuple with headers:**
```python
async def action(self, body, __event_emitter__=None):
html = "<h1>Interactive Chart</h1><script>...</script>"
return (html, {"Content-Disposition": "inline", "Content-Type": "text/html"})
```
## Iframe Height and Auto-Sizing
Rich UI embeds are rendered inside a sandboxed iframe. The iframe needs to know how tall its content is in order to display without scrollbars. There are two mechanisms for this:
### postMessage Height Reporting (Recommended)
When `allowSameOrigin` is **off** (the default), the parent page cannot read the iframe's content height directly. Your HTML must report its own height by posting a message to the parent window:
```html
<script>
function reportHeight() {
const h = document.documentElement.scrollHeight;
parent.postMessage({ type: 'iframe:height', height: h }, '*');
}
window.addEventListener('load', reportHeight);
// Also re-report when content changes size
new ResizeObserver(reportHeight).observe(document.body);
</script>
```
Add this script to the end of your `<body>` in every Rich UI embed. Without it, the iframe will stay at a small default height and your content will be cut off with a scrollbar.
### Same-Origin Auto-Resize
When `allowSameOrigin` is **on** (via the user setting `iframeSandboxAllowSameOrigin`), the parent page can directly measure the iframe's content height and resize it automatically — no script needed in your HTML. However, this comes with security trade-offs (see below).
## Sandbox and Security
Embedded iframes run inside a [sandbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/iframe#sandbox). The following sandbox flags are always enabled by default:
- `allow-scripts` — JavaScript execution
- `allow-popups` — Popups (e.g. window.open)
- `allow-downloads` — File downloads
Two additional flags can be toggled by the user in **Settings → Interface**:
| Setting | Default | Description |
|---|---|---|
| Allow Iframe Same-Origin Access | ❌ Off | Allows the iframe to access parent page context |
| Allow Iframe Form Submissions | ❌ Off | Allows form submissions within embedded content |
### allowSameOrigin
This is the most important flag to be aware of. It is **off by default** for security reasons.
**When off (default):**
- The iframe is fully isolated from the parent page
- It **cannot** read cookies, localStorage, or DOM of the parent
- The parent **cannot** read the iframe's content height (so you must use the postMessage pattern above)
- This is the safest option and recommended for most use cases
**When on:**
- The iframe can interact with the parent page's context
- Auto-resizing works without any script in your HTML
- Chart.js and Alpine.js dependencies are automatically injected if detected
- ⚠️ **Use with caution** — only enable this when you trust the embedded content
Users can toggle this setting in **Settings → Interface → Iframe Same-Origin Access**.
:::caution Practical Impact of Sandbox Settings
When `allowSameOrigin` is **off** (default), the Rich UI iframe is heavily sandboxed. This means:
- **Downloads from within the embed** are difficult or impossible — especially on iOS, where sandboxed iframes cannot trigger file downloads at all
- **JavaScript in the embed cannot interact with Open WebUI itself** — the iframe has no access to the parent page's DOM, cookies, localStorage, or any Open WebUI APIs
- **Cross-frame communication** is limited to `postMessage` only — however, [prompt submission](#prompt-submission) works cross-origin with a user confirmation dialog
If your Rich UI embed needs to trigger downloads, interact with Open WebUI's frontend, or execute JavaScript that impacts the parent page, **enabling same-origin iframe access is required**. Enable it in **Settings → Interface → Iframe Same-Origin Access**.
As an alternative for ephemeral interactions that need full page access, consider using the [`execute` event](/features/extensibility/plugin/development/events#execute-requires-__event_call__) instead, which runs unsandboxed in the main page context.
:::
## Rendering Position
- **Tool embeds** inside a tool call result render **inline** at the tool call indicator (the "View Result from..." line)
- **Action embeds** and message-level embeds render **above** the message text content
## Advanced Communication
The iframe and parent window can communicate beyond just height reporting. The following patterns are available:
### Payload Requests
The iframe can request a data payload from the parent. This is useful for passing dynamic data into the embed after it loads:
```html
<script>
// Request payload from parent
window.addEventListener('message', (e) => {
if (e.data?.type === 'payload') {
const data = e.data.payload;
// Use the payload data to populate your UI
console.log('Received payload:', data);
}
});
// Trigger the request
parent.postMessage({ type: 'payload', requestId: 'my-request' }, '*');
</script>
```
The parent responds with `{ type: 'payload', requestId: ..., payload: ... }` containing the configured payload data.
### Tool Args Injection (Tools Only)
When a **Tool** returns a Rich UI embed, the tool call arguments (the parameters the model passed to the tool) are automatically injected into the iframe's `window.args`. This allows your embedded HTML to access the tool's input:
```html
<script>
window.addEventListener('load', () => {
// window.args contains the JSON arguments the model passed to this tool
const args = window.args;
if (args) {
document.getElementById('output').textContent = JSON.stringify(args, null, 2);
}
});
</script>
```
:::note
This only works for Tool embeds rendered via the tool call display. Action embeds do not have `window.args` since they are triggered by the user, not the model.
:::
### Auto-Injected Libraries
When `allowSameOrigin` is enabled, the iframe component auto-detects usage of certain libraries in your HTML and injects them automatically — no CDN `<script>` tags needed:
- **Alpine.js** — Detected when any `x-data`, `x-init`, `x-show`, `x-bind`, `x-on`, `x-text`, `x-html`, `x-model`, `x-for`, `x-if`, `x-effect`, `x-transition`, `x-cloak`, `x-ref`, `x-teleport`, or `x-id` directives are found
- **Chart.js** — Detected when `new Chart(` or `Chart.` appears in the HTML
This means you can write Alpine or Chart.js code directly in your HTML and it will just work when same-origin is enabled, without importing scripts.
### Ping/Pong Connectivity
The iframe can test connectivity with the parent window using a simple ping/pong pattern:
```html
<script>
window.addEventListener('message', (e) => {
if (e.data?.type === 'pong:ack') {
console.log('Parent is listening!');
}
});
// Send a pong to test connectivity
parent.postMessage({ type: 'pong' }, '*');
</script>
```
### Prompt Submission
Rich UI embeds can interact with the chat input using three message types:
| Message Type | Behavior |
|---|---|
| `input:prompt` | Fills the chat input box with text (does not submit) |
| `input:prompt:submit` | Fills and submits the prompt to the chat |
| `action:submit` | Submits the current text already in the chat input |
```html
<script>
// Fill the chat input without submitting
parent.postMessage({ type: 'input:prompt', text: 'Analyze this data' }, '*');
// Fill and submit a prompt
parent.postMessage({ type: 'input:prompt:submit', text: 'Show me a summary' }, '*');
// Submit whatever is currently in the chat input
parent.postMessage({ type: 'action:submit', text: '' }, '*');
</script>
```
**Same-origin vs cross-origin behavior:**
- When `allowSameOrigin` is **on**, `input:prompt:submit` submits the prompt **immediately** — no user interaction required.
- When `allowSameOrigin` is **off** (default), `input:prompt:submit` from cross-origin iframes shows a **confirmation dialog** to the user before submitting. This prevents abuse while still enabling interactive embeds without requiring same-origin access.
- `input:prompt` and `action:submit` work the same regardless of origin — they only fill or submit text that the user can already see.
:::tip
This means your Rich UI embed can include interactive buttons (e.g., "Explain this chart", "Regenerate with different parameters") that submit prompts to the chat, **without requiring the user to enable same-origin access**. The user simply sees a confirmation dialog and clicks "Confirm" to proceed.
:::
## Rich UI Embeds vs Execute Event
Rich UI embeds and the [`execute` event](/features/extensibility/plugin/development/events#execute-requires-__event_call__) are complementary ways to create interactive experiences. Choose based on your needs:
| | Rich UI Embed | `execute` Event |
|---|---|---|
| **Runs in** | Sandboxed iframe | Main page context (no sandbox) |
| **Persistence** | Persistent — saved in chat history | Ephemeral — gone on reload/navigate |
| **Page access** | Isolated from parent by default | Full (DOM, cookies, localStorage) |
| **Forms** | Requires `allowForms` setting enabled | Always works (no sandbox) |
| **Best for** | Persistent visual content, dashboards, charts | Transient interactions, side effects, downloads, DOM manipulation |
Use Rich UI embeds for persistent visual content you want to stay in the conversation. Use `execute` for transient interactions like custom dialogs, triggering downloads, or reading page state.
## Use Cases
Rich UI embedding is perfect for:
- **Interactive dashboards** — Real-time data visualization and controls
- **Charts and graphs** — Interactive plotting with libraries like Plotly, D3.js, or Chart.js
- **Form interfaces** — Complex input forms with validation and dynamic behavior
- **Media players** — Video, audio, or interactive media content
- **Download triggers** — Especially useful for iOS PWA where native download links are blocked
- **Custom widgets** — Specialized UI components for specific tool functionality
- **External integrations** — Embedding content from external services or APIs
- **Human-triggered visualizations** — Actions that display results when a user clicks a button, e.g. generating a report or triggering a download
## Full Sample Action
<details>
<summary>Complete working Sample Action with Rich UI embed</summary>
This Action returns a styled card with stats, including the recommended height-reporting script:
```python
"""
title: Rich UI Demo Action
author: open-webui
version: 0.1.0
description: Demonstrates Rich UI embedding from an Action function.
"""
from pydantic import BaseModel, Field
class Action:
class Valves(BaseModel):
pass
def __init__(self):
self.valves = self.Valves()
async def action(self, body: dict, __user__=None, __event_emitter__=None) -> None:
from fastapi.responses import HTMLResponse
html = """
<!DOCTYPE html>
<html>
<head>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
padding: 24px;
color: #fff;
}
.card {
background: rgba(255,255,255,0.15);
backdrop-filter: blur(10px);
border-radius: 16px;
padding: 24px;
border: 1px solid rgba(255,255,255,0.2);
}
h1 { font-size: 1.4em; margin-bottom: 8px; }
p { opacity: 0.9; line-height: 1.5; margin-bottom: 12px; }
.badge {
display: inline-block;
background: rgba(255,255,255,0.25);
padding: 4px 12px;
border-radius: 20px;
font-size: 0.85em;
font-weight: 600;
}
.stats {
display: flex;
gap: 16px;
margin-top: 16px;
}
.stat {
flex: 1;
text-align: center;
background: rgba(255,255,255,0.1);
border-radius: 12px;
padding: 12px;
}
.stat-value { font-size: 1.8em; font-weight: 700; }
.stat-label { font-size: 0.8em; opacity: 0.8; margin-top: 4px; }
</style>
</head>
<body>
<div class="card">
<h1>Rich UI Embed Demo</h1>
<p>This embed renders <strong>above</strong> the message text.</p>
<span class="badge">Action Embed</span>
<div class="stats">
<div class="stat">
<div class="stat-value">42</div>
<div class="stat-label">Answers</div>
</div>
<div class="stat">
<div class="stat-value">99%</div>
<div class="stat-label">Accuracy</div>
</div>
<div class="stat">
<div class="stat-value">0ms</div>
<div class="stat-label">Latency</div>
</div>
</div>
</div>
<script>
// Report height to parent so the iframe auto-sizes
function reportHeight() {
const h = document.documentElement.scrollHeight;
parent.postMessage({ type: 'iframe:height', height: h }, '*');
}
window.addEventListener('load', reportHeight);
new ResizeObserver(reportHeight).observe(document.body);
</script>
</body>
</html>
"""
return HTMLResponse(content=html, headers={"Content-Disposition": "inline"})
```
</details>
## External Tool Example
For external tools served via HTTP endpoints:
```python
@app.post("/tools/dashboard")
async def create_dashboard():
html = """
<div style="padding: 20px;">
<h2>System Dashboard</h2>
<canvas id="myChart" width="400" height="200"></canvas>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<script>
const ctx = document.getElementById('myChart').getContext('2d');
new Chart(ctx, {
type: 'line',
data: { /* your chart data */ }
});
</script>
</div>
"""
return HTMLResponse(
content=html,
headers={"Content-Disposition": "inline"}
)
```
The embedded content automatically inherits responsive design and integrates seamlessly with the chat interface, providing a native-feeling experience for users interacting with your tools.
## CORS and Direct Tools
Direct external tools are tools that run directly from the browser. In this case, the tool is called by JavaScript in the user's browser.
Because we depend on the Content-Disposition header, when using CORS on a remote tool server, the Open WebUI cannot read that header due to Access-Control-Expose-Headers, which prevents certain headers from being read from the fetch result.
To prevent this, you must set Access-Control-Expose-Headers to Content-Disposition. Check the example below of a tool using Node.js:
```javascript
const app = express();
const cors = require('cors');
app.use(cors())
app.get('/tools/dashboard', (req,res) => {
let html = `
<div style="padding: 20px;">
<h2>System Dashboard</h2>
<canvas id="myChart" width="400" height="200"></canvas>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<script>
const ctx = document.getElementById('myChart').getContext('2d');
new Chart(ctx, {
type: 'line',
data: { /* your chart data */ }
});
</script>
</div>
`
res.set({
'Content-Disposition': 'inline'
,'Access-Control-Expose-Headers':'Content-Disposition'
})
res.send(html)
})
```
More info about the header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Access-Control-Expose-Headers
Reference in New Issue View Git Blame Copy Permalink