📘 feat: Update Agents, OCR, and MCP Docs with new Features (#336)
* 📚 docs: Update OCR documentation for clarity and structure * 📚 docs: Add Model Context Protocol (MCP) documentation and update features * 📚 docs: Add Model Context Protocol (MCP) section to features documentation * 📚 docs: Enhance MCP documentation with OAuth authentication details and configuration examples * 📚 docs: Update MCP documentation with improved configuration examples and OAuth authentication details * 📚 docs: Update dotenv documentation to reflect new index header constants * 📚 docs: Revise MCP integration documentation for Agents with clearer instructions and new images - Updated the steps for adding MCP tools to agents for better clarity and organization. - Introduced new images to illustrate the MCP tools interface. - Removed outdated image and improved the overall structure of the documentation. * 📚 docs: Enhance MCP documentation with custom user variables support --------- Co-authored-by: Aleš Kůtek <kutekales@gmail.com>
@@ -10,4 +10,9 @@
|
|||||||
- Added additional user field placeholders to [MCP Servers Object Structure](/docs/configuration/librechat_yaml/object_structure/mcp_servers):
|
- Added additional user field placeholders to [MCP Servers Object Structure](/docs/configuration/librechat_yaml/object_structure/mcp_servers):
|
||||||
- `{{LIBRECHAT_USER_ID}}`: Will be replaced with the current user's ID, enabling multi-user support.
|
- `{{LIBRECHAT_USER_ID}}`: Will be replaced with the current user's ID, enabling multi-user support.
|
||||||
- `{{LIBRECHAT_USER_*}}`: Dynamic user field placeholders. Replace `*` with the UPPERCASE version of any allowed field.
|
- `{{LIBRECHAT_USER_*}}`: Dynamic user field placeholders. Replace `*` with the UPPERCASE version of any allowed field.
|
||||||
- See: **[`headers`](/docs/configuration/librechat_yaml/object_structure/mcp_servers#headers)** and **[`url`](/docs/configuration/librechat_yaml/object_structure/mcp_servers#url)** fields, which both support these placeholders.
|
- See: **[`headers`](/docs/configuration/librechat_yaml/object_structure/mcp_servers#headers)** and **[`url`](/docs/configuration/librechat_yaml/object_structure/mcp_servers#url)** fields, which both support these placeholders.
|
||||||
|
- Added `customUserVars` to MCP Servers for per-user credentials:
|
||||||
|
- Allows users to securely provide their own API keys and credentials for MCP servers
|
||||||
|
- Configurable through UI via settings icon in tool selection or MCP Settings panel
|
||||||
|
- Ensures authentication details remain private in multi-user environments
|
||||||
|
- See: [MCP Servers Object Structure - customUserVars](/docs/configuration/librechat_yaml/object_structure/mcp_servers#customuservars)
|
||||||
@@ -98,9 +98,9 @@ Sets the [Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Heade
|
|||||||
|
|
||||||
<OptionTable
|
<OptionTable
|
||||||
options={[
|
options={[
|
||||||
['INDEX_HTML_CACHE_CONTROL', 'string', 'Cache-Control header for index.html','INDEX_HTML_CACHE_CONTROL=no-cache, no-store, must-revalidate'],
|
['INDEX_CACHE_CONTROL', 'string', 'Cache-Control header for index.html','INDEX_CACHE_CONTROL=no-cache, no-store, must-revalidate'],
|
||||||
['INDEX_HTML_PRAGMA', 'string', 'Pragma header for index.html','INDEX_HTML_PRAGMA=no-cache'],
|
['INDEX_PRAGMA', 'string', 'Pragma header for index.html','INDEX_PRAGMA=no-cache'],
|
||||||
['INDEX_HTML_EXPIRES', 'string', 'Expires header for index.html','INDEX_HTML_EXPIRES=0'],
|
['INDEX_EXPIRES', 'string', 'Expires header for index.html','INDEX_EXPIRES=0'],
|
||||||
]}
|
]}
|
||||||
/>
|
/>
|
||||||
|
|
||||||
|
|||||||
@@ -401,6 +401,7 @@ see: [Model Specs Object Structure](/docs/configuration/librechat_yaml/object_st
|
|||||||
- `timeout`: Timeout in milliseconds for MCP server requests. Determines how long to wait for a response for tool requests.
|
- `timeout`: Timeout in milliseconds for MCP server requests. Determines how long to wait for a response for tool requests.
|
||||||
- `initTimeout`: Timeout in milliseconds for MCP server initialization. Determines how long to wait for the server to initialize.
|
- `initTimeout`: Timeout in milliseconds for MCP server initialization. Determines how long to wait for the server to initialize.
|
||||||
- `serverInstructions`: Controls whether server instructions are included in agent context. Can be `true` (use server-provided), `false` (disabled), or a custom string (overrides server-provided).
|
- `serverInstructions`: Controls whether server instructions are included in agent context. Can be `true` (use server-provided), `false` (disabled), or a custom string (overrides server-provided).
|
||||||
|
- `customUserVars`: (Optional) Defines custom variables (e.g., API keys, URLs) that individual users can set for an MCP server. These per-user values, provided through the UI, can then be referenced in the server's `headers` or `env` configurations using `{{VARIABLE_NAME}}` syntax. This allows for per-user authentication or customization for MCP tools.
|
||||||
- see: [MCP Servers Object Structure](/docs/configuration/librechat_yaml/object_structure/mcp_servers)
|
- see: [MCP Servers Object Structure](/docs/configuration/librechat_yaml/object_structure/mcp_servers)
|
||||||
|
|
||||||
**Example:**
|
**Example:**
|
||||||
@@ -442,6 +443,16 @@ mcpServers:
|
|||||||
headers:
|
headers:
|
||||||
Authorization: "Bearer ${API_TOKEN}"
|
Authorization: "Bearer ${API_TOKEN}"
|
||||||
timeout: 30000
|
timeout: 30000
|
||||||
|
per-user-crendentials-example:
|
||||||
|
type: sse
|
||||||
|
url: "https//some.mcp/sse"
|
||||||
|
headers:
|
||||||
|
X-Custom-Auth-Token: "{{USER_API_KEY}}" # Placeholder for the user-provided API key, defined in `customUserVars` below.
|
||||||
|
customUserVars:
|
||||||
|
USER_API_KEY:
|
||||||
|
title: "Service API Key"
|
||||||
|
description: "Your personal API key for this service. You can get it <a href='https://example.com/api-keys' target='_blank'>here</a>."
|
||||||
|
serverInstructions: true
|
||||||
```
|
```
|
||||||
|
|
||||||
see: [MCP Servers Object Structure](/docs/configuration/librechat_yaml/object_structure/mcp_servers)
|
see: [MCP Servers Object Structure](/docs/configuration/librechat_yaml/object_structure/mcp_servers)
|
||||||
|
|||||||
@@ -39,6 +39,15 @@ mcpServers:
|
|||||||
streamable-http-server:
|
streamable-http-server:
|
||||||
type: streamable-http
|
type: streamable-http
|
||||||
url: https://example.com/api/
|
url: https://example.com/api/
|
||||||
|
per-user-credentials-example:
|
||||||
|
type: streamable-http
|
||||||
|
url: "https://example.com/api/"
|
||||||
|
headers:
|
||||||
|
X-Auth-Token: "{{MY_SERVICE_API_KEY}}"
|
||||||
|
customUserVars:
|
||||||
|
MY_SERVICE_API_KEY:
|
||||||
|
title: "My Service API Key"
|
||||||
|
description: "Enter your personal API key for the service. You can generate one at <a href='https://myservice.example.com/developer/keys' target='_blank'>Service Developer Portal</a>."
|
||||||
```
|
```
|
||||||
|
|
||||||
## `<serverName>`
|
## `<serverName>`
|
||||||
@@ -66,6 +75,7 @@ mcpServers:
|
|||||||
['initTimeout', 'Number', '(Optional) Timeout in milliseconds for MCP server initialization. Determines how long to wait for the server to initialize.', 'initTimeout: 10000'],
|
['initTimeout', 'Number', '(Optional) Timeout in milliseconds for MCP server initialization. Determines how long to wait for the server to initialize.', 'initTimeout: 10000'],
|
||||||
['env', 'Object', '(Optional, `stdio` type only) Environment variables to use when spawning the process.', 'env:\n NODE_ENV: "production"'],
|
['env', 'Object', '(Optional, `stdio` type only) Environment variables to use when spawning the process.', 'env:\n NODE_ENV: "production"'],
|
||||||
['stderr', 'String or Stream or Number', '(Optional, `stdio` type only) How to handle `stderr` of the child process. Defaults to `"inherit"`.', 'stderr: "inherit"'],
|
['stderr', 'String or Stream or Number', '(Optional, `stdio` type only) How to handle `stderr` of the child process. Defaults to `"inherit"`.', 'stderr: "inherit"'],
|
||||||
|
['customUserVars', 'Object', '(Optional) Defines custom variables that users can set for this MCP server, allowing for per-user credentials or configurations (e.g., API keys). These variables can then be referenced in `headers` or `env` fields.', 'customUserVars:\n API_KEY:\n title: "API Key"\n description: "Your personal API key."'],
|
||||||
]}
|
]}
|
||||||
/>
|
/>
|
||||||
|
|
||||||
@@ -97,11 +107,12 @@ mcpServers:
|
|||||||
#### `headers`
|
#### `headers`
|
||||||
|
|
||||||
- **Type:** Object (Optional, for `sse` and `streamable-http` types)
|
- **Type:** Object (Optional, for `sse` and `streamable-http` types)
|
||||||
- **Description:** Custom headers to send with the request.
|
- **Description:** Custom headers to send with the request. Supports various placeholder types for dynamic value substitution.
|
||||||
- **Special Values:**
|
- **Placeholder Support:**
|
||||||
- `{{LIBRECHAT_USER_ID}}`: Will be replaced with the current user's ID, enabling multi-user support.
|
- `{{LIBRECHAT_USER_ID}}`: Will be replaced with the current user's ID, enabling multi-user support.
|
||||||
- `{{LIBRECHAT_USER_*}}`: Dynamic user field placeholders. Replace `*` with the UPPERCASE version of any allowed field.
|
- `{{LIBRECHAT_USER_*}}`: Dynamic user field placeholders. Replace `*` with the UPPERCASE version of any allowed field.
|
||||||
- `${ENV_VAR}`: Will be replaced with the value of the environment variable `ENV_VAR`.
|
- `{{CUSTOM_VARIABLE_NAME}}`: Replaced with the value provided by the user for a variable defined in `customUserVars` (e.g., `{{MY_API_KEY}}`).
|
||||||
|
- `${ENV_VAR}`: Will be replaced with the value of the environment variable `{{ENV_VAR}}`.
|
||||||
|
|
||||||
**Available User Field Placeholders:**
|
**Available User Field Placeholders:**
|
||||||
|
|
||||||
@@ -178,7 +189,12 @@ mcpServers:
|
|||||||
#### `env`
|
#### `env`
|
||||||
|
|
||||||
- **Type:** Object (Optional, `stdio` type only)
|
- **Type:** Object (Optional, `stdio` type only)
|
||||||
- **Description:** Environment variables to use when spawning the process. Supports the same dynamic user field placeholders as headers (`{{LIBRECHAT_USER_*}}`) and environment variable substitution (`${ENV_VAR}`).
|
- **Description:** Environment variables to use when spawning the process.
|
||||||
|
- **Placeholder Support:**
|
||||||
|
- `{{LIBRECHAT_USER_ID}}`: Replaced with the current user's ID.
|
||||||
|
- `{{LIBRECHAT_USER_*}}`: Dynamic user field placeholders (e.g., `{{LIBRECHAT_USER_EMAIL}}`).
|
||||||
|
- `{{CUSTOM_VARIABLE_NAME}}`: Replaced with the value provided by the user for a variable defined in `customUserVars` (e.g., `{{MY_API_KEY}}`).
|
||||||
|
- `${ENV_VAR}`: Replaced with the value of the server-side environment variable `{{ENV_VAR}}`.
|
||||||
|
|
||||||
#### `timeout`
|
#### `timeout`
|
||||||
|
|
||||||
@@ -198,6 +214,41 @@ mcpServers:
|
|||||||
- **Description:** How to handle `stderr` of the child process. This matches the semantics of Node's [`child_process.spawn`](https://nodejs.org/api/child_process.html#child_processspawncommand-args-options).
|
- **Description:** How to handle `stderr` of the child process. This matches the semantics of Node's [`child_process.spawn`](https://nodejs.org/api/child_process.html#child_processspawncommand-args-options).
|
||||||
- **Default Value:** `"inherit"` (messages to `stderr` will be printed to the parent process's `stderr`).
|
- **Default Value:** `"inherit"` (messages to `stderr` will be printed to the parent process's `stderr`).
|
||||||
|
|
||||||
|
#### `customUserVars`
|
||||||
|
|
||||||
|
- **Type:** Object (Optional)
|
||||||
|
- **Description:** Defines custom variables that users can set for this MCP server. This allows administrators to specify variables (e.g., API keys, URLs) that each user must configure individually. These user-provided values can then be used in `headers` configuration.
|
||||||
|
- **Structure:**
|
||||||
|
- The `customUserVars` object contains keys, where each key represents a variable name (e.g., `MY_API_KEY`). This name will be used in placeholders like `{{MY_API_KEY}}`.
|
||||||
|
- Each variable name is an object with the following subkeys:
|
||||||
|
- `title`: String (Required) - A user-friendly title for the variable, displayed in the configuration UI.
|
||||||
|
- `description`: String (Optional) - A description or instructions for the variable, also displayed in the UI to guide the user. HTML can be used in this field (e.g., to create a link: `<a href="https://example.com" target="_blank">More info</a>`).
|
||||||
|
- **Usage in `headers` and `env`:**
|
||||||
|
- Once defined under `customUserVars`, these variables can be referenced in the `headers` (for `sse` and `streamable-http` types) or `env` (for `stdio` type) sections using the `{{VARIABLE_NAME}}` syntax.
|
||||||
|
- Users provide these values through the UI. These settings can be accessed in two ways:
|
||||||
|
- **From Assistant Chat Input**: When selecting MCP tools for an assistant, a settings icon will appear next to configurable MCP servers in the tool selection dropdown. Clicking this icon opens a dialog to manage credentials for that server.
|
||||||
|
<img src="/images/mcp/mcp_per_user_vars_assistant.png" alt="MCP Per-User Variables Configuration - Assistant Access" width={600} />
|
||||||
|
<img src="/images/mcp/mcp_per_user_vars_assistant_dialog.png" alt="MCP Per-User Variables Configuration - Assistant Access Dialog" width={800} />
|
||||||
|
- **From Settings Panel**: A dedicated "MCP Settings" section in the right panel lists all MCP servers with definable custom variables. Users can click on a server to open the configuration dialog to set or update their credentials for that specific MCP server.
|
||||||
|
<img src="/images/mcp/mcp_per_user_vars_side_panel.png" alt="MCP Per-User Variables Configuration - Settings Panel Access" width={300} />
|
||||||
|
- These user-provided values are stored securely, associated with the individual user and the specific MCP server, and substituted at runtime.
|
||||||
|
- **Example:**
|
||||||
|
```yaml
|
||||||
|
customUserVars:
|
||||||
|
MY_SERVICE_API_KEY:
|
||||||
|
title: "My Service API Key"
|
||||||
|
description: "Your personal API access key for My Service. Find it at <a href='https://myservice.example.com/settings/api' target='_blank'>My Service API Settings</a>."
|
||||||
|
SOME_OTHER_VAR:
|
||||||
|
title: "Some Other Variable"
|
||||||
|
description: "The specific value for some other configuration (e.g., a specific path or identifier)."
|
||||||
|
```
|
||||||
|
Usage in `headers`:
|
||||||
|
```yaml
|
||||||
|
headers:
|
||||||
|
X-Auth-Token: "{{MY_SERVICE_API_KEY}}"
|
||||||
|
X-Some-Other-Config: "{{SOME_OTHER_VAR}}"
|
||||||
|
```
|
||||||
|
|
||||||
### Notes
|
### Notes
|
||||||
|
|
||||||
- **Type Inference:**
|
- **Type Inference:**
|
||||||
@@ -273,6 +324,20 @@ user-aware-server:
|
|||||||
Authorization: "Bearer ${API_TOKEN}"
|
Authorization: "Bearer ${API_TOKEN}"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### MCP Server with Per-User Credentials via `customUserVars`
|
||||||
|
|
||||||
|
```yaml filename="MCP Server with customUserVars"
|
||||||
|
my-mcp-server:
|
||||||
|
type: streamable-http
|
||||||
|
url: "https://api.example-service.com/api/" # Example URL
|
||||||
|
headers:
|
||||||
|
X-Auth-Token: "{{API_KEY}}" # Uses the API_KEY defined below
|
||||||
|
customUserVars:
|
||||||
|
API_KEY: # This key will be used as {{API_KEY}} in headers/url
|
||||||
|
title: "API Key" # This is the label shown above the input field
|
||||||
|
description: "Get your API key <a href='https://example.com/api-keys' target='_blank'>here</a>." # This description appears below the input
|
||||||
|
```
|
||||||
|
|
||||||
### MCP Server with Custom Icon
|
### MCP Server with Custom Icon
|
||||||
|
|
||||||
```yaml filename="MCP Server with Icon"
|
```yaml filename="MCP Server with Icon"
|
||||||
|
|||||||
@@ -112,6 +112,6 @@ ocr:
|
|||||||
|
|
||||||
**Available Strategies:**
|
**Available Strategies:**
|
||||||
|
|
||||||
- `mistral_ocr`: Uses Mistral's OCRcapabilities via the standard [Mistral API](/docs/features/ocr#mistral-ocr) .
|
- `mistral_ocr`: Uses Mistral's OCRcapabilities via the standard [Mistral API](/docs/features/ocr#1-mistral-ocr-default) .
|
||||||
- `azure_mistral_ocr`: Uses Mistral OCR models deployed on [Azure AI Foundry](/docs/features/ocr#2-azure-deployed-mistral-ocr).
|
- `azure_mistral_ocr`: Uses Mistral OCR models deployed on [Azure AI Foundry](/docs/features/ocr#2-azure-mistral-ocr).
|
||||||
- `custom_ocr`: Uses a custom OCR service specified by the `baseURL` [(not yet implemented)](/docs/features/ocr#future-enhancements).
|
- `custom_ocr`: Uses a custom OCR service specified by the `baseURL` [(not yet implemented)](/docs/features/ocr#future-enhancements).
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
export default {
|
export default {
|
||||||
index: 'Overview',
|
index: 'Overview',
|
||||||
|
mcp: 'MCP',
|
||||||
agents: 'Agents',
|
agents: 'Agents',
|
||||||
code_interpreter: 'Code Interpreter API',
|
code_interpreter: 'Code Interpreter API',
|
||||||
artifacts: 'Artifacts - Generative UI',
|
artifacts: 'Artifacts - Generative UI',
|
||||||
|
|||||||
@@ -71,6 +71,41 @@ The File Context (OCR) capability allows your agent to extract and process text
|
|||||||
- [More info about OCR](/docs/features/ocr)
|
- [More info about OCR](/docs/features/ocr)
|
||||||
- **Currently uses Mistral OCR API which may incur costs**
|
- **Currently uses Mistral OCR API which may incur costs**
|
||||||
|
|
||||||
|
### Model Context Protocol (MCP)
|
||||||
|
|
||||||
|
MCP is an open protocol that standardizes how applications provide context to Large Language Models (LLMs), acting like a universal adapter for AI tools and data sources.
|
||||||
|
|
||||||
|
For more information, see documentation on [MCP](/docs/features/mcp).
|
||||||
|
|
||||||
|
#### Agents with MCP Tools
|
||||||
|
|
||||||
|
|
||||||
|
1. Configure MCP servers in your `librechat.yaml` file
|
||||||
|
2. Restart LibreChat to initialize the connections
|
||||||
|
3. Create or edit an agent
|
||||||
|
4. Click the "Add Tools" button in the Agent Builder panel to open the Tools Dialog shown below
|
||||||
|
5. Select the MCP server(s) you want to add - each server appears as a single entry
|
||||||
|
6. Save your changes to the agent
|
||||||
|
|
||||||
|
In this example, we've added the **[Spotify MCP server](https://github.com/LibreChat-AI/spotify-mcp)** to an agent.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
#### Managing MCP Tools
|
||||||
|
|
||||||
|
Once an MCP server is added to an agent, you can fine-tune which specific tools are available:
|
||||||
|
|
||||||
|
- After adding an MCP server, expand it to see all available tools
|
||||||
|
- Check/uncheck individual tools as needed
|
||||||
|
- For example, the Spotify MCP server provides ~20 tools (search, playback control, playlist management, etc.)
|
||||||
|
- This granular control lets you limit agents to only the tools they need
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Learn More:
|
||||||
|
- [Configuring MCP Servers](/docs/features/mcp)
|
||||||
|
- [Model Context Protocol Introduction](https://modelcontextprotocol.io/introduction)
|
||||||
|
|
||||||
### Artifacts
|
### Artifacts
|
||||||
|
|
||||||
The Artifacts capability enables your agent to generate and display interactive content:
|
The Artifacts capability enables your agent to generate and display interactive content:
|
||||||
@@ -174,94 +209,6 @@ A single, non-tool response is 1 step. A singular round of tool usage is usually
|
|||||||
|
|
||||||
1. API Request -> 2. Tool Usage (1 or many tools) -> 3. Follow-up API Request
|
1. API Request -> 2. Tool Usage (1 or many tools) -> 3. Follow-up API Request
|
||||||
|
|
||||||
### Model Context Protocol (MCP)
|
|
||||||
|
|
||||||
MCP is an open protocol that standardizes how applications provide context to Large Language Models (LLMs), acting like a universal adapter for AI tools and data sources.
|
|
||||||
|
|
||||||
Imagine MCP as the "USB-C of AI" - just as USB-C provides a universal connection standard for electronic devices, MCP offers a standardized way to connect AI models to diverse tools, data sources, and services.
|
|
||||||
|
|
||||||
By configuring `mcpServers` in the `librechat.yaml` file, you can:
|
|
||||||
- Add custom tools from various sources
|
|
||||||
- Integrate specialized data access tools
|
|
||||||
- Extend AI capabilities beyond default offerings
|
|
||||||
|
|
||||||
Here are some tools added by configuring the ["Filesystem" MCP server](https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem):
|
|
||||||

|
|
||||||
|
|
||||||
Key Features:
|
|
||||||
- Upon app restart, tools are available in the "Add Tools" button within the Agent Builder panel
|
|
||||||
- Seamlessly connect custom tools and data sources
|
|
||||||
- Leverage a growing ecosystem of MCP-compatible servers and integrations
|
|
||||||
|
|
||||||
Learn More:
|
|
||||||
- [Configuring MCP Servers](/docs/configuration/librechat_yaml/object_structure/mcp_servers)
|
|
||||||
- [Model Context Protocol Introduction](https://modelcontextprotocol.io/introduction)
|
|
||||||
|
|
||||||
#### MCP Server Types and Multi-User Support
|
|
||||||
|
|
||||||
MCP servers can be configured to use different transport mechanisms:
|
|
||||||
|
|
||||||
**STDIO Servers**
|
|
||||||
- Work well for local, single-user environments
|
|
||||||
- Not scalable for remote or cloud deployments
|
|
||||||
- Limited multi-user support
|
|
||||||
|
|
||||||
**Server-Sent Events (SSE) Servers**
|
|
||||||
- More scalable transport mechanism for multi-user environments
|
|
||||||
- Enhanced multi-user support with user-specific connections
|
|
||||||
|
|
||||||
**Streamable HTTP Servers**
|
|
||||||
- Uses HTTP POST for sending messages and supports streaming responses
|
|
||||||
- Operates as an independent process that can handle multiple client connections
|
|
||||||
- Supports both basic requests and streaming via Server-Sent Events (SSE)
|
|
||||||
- More performant alternative to the legacy HTTP+SSE transport
|
|
||||||
- Supports proper multi-user server configurations
|
|
||||||
|
|
||||||
**Multi-User Features**
|
|
||||||
- User-level and app-level connections with proper establishment and cleanup
|
|
||||||
- MCP Servers can distinguish connections by session ID
|
|
||||||
- Dynamic user field placeholders in headers, URLs, and environment variables:
|
|
||||||
- `{{LIBRECHAT_USER_ID}}` - User's unique identifier
|
|
||||||
- Basic fields: `{{LIBRECHAT_USER_EMAIL}}`, `{{LIBRECHAT_USER_USERNAME}}`, `{{LIBRECHAT_USER_NAME}}`, `{{LIBRECHAT_USER_ROLE}}`, `{{LIBRECHAT_USER_PROVIDER}}`
|
|
||||||
- Provider IDs: `{{LIBRECHAT_USER_GOOGLEID}}`, `{{LIBRECHAT_USER_OPENIDID}}`, `{{LIBRECHAT_USER_SAMLID}}`, `{{LIBRECHAT_USER_LDAPID}}`, etc.
|
|
||||||
- Boolean fields: `{{LIBRECHAT_USER_EMAILVERIFIED}}`, `{{LIBRECHAT_USER_TWOFACTORENABLED}}`, `{{LIBRECHAT_USER_TERMSACCEPTED}}`
|
|
||||||
- See [MCP Servers documentation](/docs/configuration/librechat_yaml/object_structure/mcp_servers#headers) for complete field list
|
|
||||||
- Environment variable support: `${SOME_API_KEY}`
|
|
||||||
- User idle timeout management for efficient resource usage
|
|
||||||
|
|
||||||
**Example Configuration:**
|
|
||||||
```yaml
|
|
||||||
mcpServers:
|
|
||||||
googlesheets:
|
|
||||||
type: sse
|
|
||||||
url: https://mcp.composio.dev/googlesheets/some-endpoint
|
|
||||||
headers:
|
|
||||||
X-User-ID: "{{LIBRECHAT_USER_ID}}"
|
|
||||||
X-User-Email: "{{LIBRECHAT_USER_EMAIL}}"
|
|
||||||
X-API-Key: "${SOME_API_KEY}"
|
|
||||||
|
|
||||||
streamable-http-example:
|
|
||||||
type: streamable-http
|
|
||||||
url: https://example.com/mcp/users/{{LIBRECHAT_USER_USERNAME}}
|
|
||||||
headers:
|
|
||||||
X-User-ID: "{{LIBRECHAT_USER_ID}}"
|
|
||||||
X-User-Role: "{{LIBRECHAT_USER_ROLE}}"
|
|
||||||
Authorization: "Bearer ${API_TOKEN}"
|
|
||||||
|
|
||||||
user-context-server:
|
|
||||||
type: stdio
|
|
||||||
command: node
|
|
||||||
args: ["server.js"]
|
|
||||||
env:
|
|
||||||
USER_EMAIL: "{{LIBRECHAT_USER_EMAIL}}"
|
|
||||||
USER_PROVIDER: "{{LIBRECHAT_USER_PROVIDER}}"
|
|
||||||
API_KEY: "${MCP_API_KEY}"
|
|
||||||
```
|
|
||||||
|
|
||||||
LibreChat is at the forefront of implementing flexible, scalable MCP server integrations to support diverse usage scenarios.
|
|
||||||
|
|
||||||
For detailed configuration options, see [Configuring MCP Servers](/docs/configuration/librechat_yaml/object_structure/mcp_servers).
|
|
||||||
|
|
||||||
## File Management
|
## File Management
|
||||||
|
|
||||||
Agents support four distinct file upload categories:
|
Agents support four distinct file upload categories:
|
||||||
|
|||||||
@@ -45,6 +45,14 @@ import Image from 'next/image'
|
|||||||
- **Configuration Required**: Must be explicitly configured in `librechat.yaml` to work
|
- **Configuration Required**: Must be explicitly configured in `librechat.yaml` to work
|
||||||
- **[Learn More →](/docs/features/memory)**
|
- **[Learn More →](/docs/features/memory)**
|
||||||
|
|
||||||
|
### 🔌 [Model Context Protocol (MCP)](/docs/features/mcp)
|
||||||
|
|
||||||
|
- **Universal AI Tool Integration**: Connect AI models to any tool or service through a standardized protocol
|
||||||
|
- **Production-Ready Transports**: Support for scalable HTTP-based servers ideal for multi-user deployments
|
||||||
|
- **Dynamic User Context**: Pass user-specific information to MCP servers for personalized experiences
|
||||||
|
- **Flexible Usage**: Use MCP tools directly in chat or integrate them with custom agents
|
||||||
|
- **[Learn More →](/docs/features/mcp)**
|
||||||
|
|
||||||
### 🪄 **[Artifacts](/docs/features/artifacts)**
|
### 🪄 **[Artifacts](/docs/features/artifacts)**
|
||||||
|
|
||||||
- **Generative UI:** React, HTML, Mermaid diagrams
|
- **Generative UI:** React, HTML, Mermaid diagrams
|
||||||
|
|||||||
297
pages/docs/features/mcp.mdx
Normal file
@@ -0,0 +1,297 @@
|
|||||||
|
---
|
||||||
|
title: Model Context Protocol (MCP) - Extend LibreChat with Universal AI Tools
|
||||||
|
description: Learn how LibreChat leverages Model Context Protocol (MCP) to provide seamless integration with external tools, data sources, and specialized services.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Model Context Protocol (MCP)
|
||||||
|
|
||||||
|
Model Context Protocol (MCP) is an open protocol that standardizes how applications provide context to Large Language Models (LLMs). Think of MCP as the **"USB-C of AI"** - just as USB-C provides a universal connection standard for electronic devices, MCP offers a standardized way to connect AI models to diverse tools, data sources, and services.
|
||||||
|
|
||||||
|
LibreChat leverages MCP to dramatically expand what your AI agents can do, allowing you to integrate everything from file system access, web browsers, specialized APIs, to custom business tools.
|
||||||
|
|
||||||
|
## Why MCP Matters
|
||||||
|
|
||||||
|
LLMs are limited to their built-in capabilities. With MCP, LibreChat breaks down these walls by:
|
||||||
|
|
||||||
|
- **Connecting to any tool or service** that provides an MCP server
|
||||||
|
- **Standardizing integrations** so you don't need to edit LibreChat's code for each tool
|
||||||
|
- **Supporting multi-user environments** with proper authentication and isolation
|
||||||
|
- **Providing a growing ecosystem** of dynamic, ready-to-use integrations
|
||||||
|
|
||||||
|
## How MCP Works in LibreChat
|
||||||
|
|
||||||
|
LibreChat provides two ways to use MCP servers.
|
||||||
|
|
||||||
|
1. First, configure MCP servers in your `librechat.yaml` file ([see example below](#basic-configuration)).
|
||||||
|
2. Restart LibreChat to initialize the connections.
|
||||||
|
|
||||||
|
### In Chat Area
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
LibreChat displays configured MCP servers directly in the chat area when using traditional endpoints (OpenAI, Anthropic, Google, Bedrock, etc.):
|
||||||
|
|
||||||
|
- Select any non-agent endpoint first, and a tool-compatible model
|
||||||
|
- MCP servers appear in a dropdown in the chat interface below your text input
|
||||||
|
- When selected, all tools from that server become available to your current model
|
||||||
|
- Quick access to MCP tools without creating an agent, allowing multiple servers to be used at once
|
||||||
|
|
||||||
|
To disable MCP servers from appearing in the chat dropdown (keeping them agent-only), set [`chatMenu: false`](/docs/configuration/librechat_yaml/object_structure/mcp_servers#chatmenu) in your configuration:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
internal-tools:
|
||||||
|
command: npx
|
||||||
|
args: ["-y", "internal-mcp-server"]
|
||||||
|
chatMenu: false # Only available in agent builder
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Agents
|
||||||
|
|
||||||
|
MCP servers integrate seamlessly with LibreChat Agents:
|
||||||
|
|
||||||
|
1. Create or edit an agent
|
||||||
|
2. Click "Add Tools" to open the Tools Dialog from the Agent Builder panel
|
||||||
|
3. Select MCP servers once added, each appears as a single entry
|
||||||
|
4. Fine-tune by enabling/disabling individual tools after adding
|
||||||
|
5. Save your agent
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
This higher-level organization keeps the interface manageable - even servers with 20+ tools (like Spotify) appear as single entries that can be expanded for granular control.
|
||||||
|
|
||||||
|
### Basic Configuration
|
||||||
|
|
||||||
|
Add MCP servers to your `librechat.yaml` file:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
# File system access
|
||||||
|
filesystem:
|
||||||
|
command: npx
|
||||||
|
args:
|
||||||
|
- -y
|
||||||
|
- "@modelcontextprotocol/server-filesystem"
|
||||||
|
- /path/to/your/documents
|
||||||
|
|
||||||
|
# Web browser automation
|
||||||
|
puppeteer:
|
||||||
|
command: npx
|
||||||
|
args:
|
||||||
|
- -y
|
||||||
|
- "@modelcontextprotocol/server-puppeteer"
|
||||||
|
|
||||||
|
# Production-ready cloud service
|
||||||
|
business-api:
|
||||||
|
type: streamable-http
|
||||||
|
url: https://api.yourbusiness.com/mcp
|
||||||
|
headers:
|
||||||
|
X-User-ID: "{{LIBRECHAT_USER_ID}}"
|
||||||
|
Authorization: "Bearer ${API_TOKEN}"
|
||||||
|
timeout: 30000
|
||||||
|
serverInstructions: true
|
||||||
|
```
|
||||||
|
|
||||||
|
For detailed configuration options and examples, see:
|
||||||
|
- [MCP Servers Configuration](/docs/configuration/librechat_yaml/object_structure/mcp_servers)
|
||||||
|
- [Agent Configuration](/docs/configuration/librechat_yaml/object_structure/agents)
|
||||||
|
- [Advanced Agent Features](/docs/features/agents#model-context-protocol-mcp)
|
||||||
|
|
||||||
|
## LibreChat-Specific Features
|
||||||
|
|
||||||
|
LibreChat's MCP implementation is designed for highly configurable, real-world, multi-user environments.
|
||||||
|
|
||||||
|
### User-Specific Connections
|
||||||
|
- Each user gets their own isolated connection to MCP servers
|
||||||
|
- User authentication and permissions are respected
|
||||||
|
- Personal data and context remain private
|
||||||
|
|
||||||
|
### Dynamic User Context
|
||||||
|
MCP servers can access user information through placeholders in **URLs and headers** (for SSE and Streamable HTTP transports):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
user-api:
|
||||||
|
type: streamable-http
|
||||||
|
url: https://api.example.com/users/{{LIBRECHAT_USER_USERNAME}}/mcp
|
||||||
|
headers:
|
||||||
|
X-User-ID: "{{LIBRECHAT_USER_ID}}"
|
||||||
|
X-User-Email: "{{LIBRECHAT_USER_EMAIL}}"
|
||||||
|
X-User-Role: "{{LIBRECHAT_USER_ROLE}}"
|
||||||
|
Authorization: "Bearer ${API_TOKEN}"
|
||||||
|
```
|
||||||
|
|
||||||
|
Available placeholders include:
|
||||||
|
- `{{LIBRECHAT_USER_ID}}` - Unique user identifier
|
||||||
|
- `{{LIBRECHAT_USER_EMAIL}}` - User's email address
|
||||||
|
- `{{LIBRECHAT_USER_ROLE}}` - User role (admin, user, etc.)
|
||||||
|
- `{{LIBRECHAT_USER_USERNAME}}` - Username
|
||||||
|
- And many more (see [MCP Servers Configuration](/docs/configuration/librechat_yaml/object_structure/mcp_servers#headers) for complete list)
|
||||||
|
|
||||||
|
### Server Instructions
|
||||||
|
|
||||||
|
`serverInstructions` is a LibreChat feature that dynamically adds configured instructions when any tool from that MCP server is selected:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
filesystem:
|
||||||
|
command: npx
|
||||||
|
args: ["-y", "@modelcontextprotocol/server-filesystem", "/docs"]
|
||||||
|
serverInstructions: |
|
||||||
|
When accessing files:
|
||||||
|
- Always check file permissions first
|
||||||
|
- Use absolute paths for reliability
|
||||||
|
- Handle errors gracefully
|
||||||
|
```
|
||||||
|
|
||||||
|
Options:
|
||||||
|
- `true`: Use server-provided instructions
|
||||||
|
- `false`: Disable instructions
|
||||||
|
- `string`: Custom instructions (shown above)
|
||||||
|
|
||||||
|
### Timeout Configuration
|
||||||
|
|
||||||
|
For long-running MCP operations, configure appropriate timeouts for both initialization and tool operations.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
data-processor:
|
||||||
|
type: streamable-http
|
||||||
|
url: https://api.example.com/mcp
|
||||||
|
initTimeout: 15000 # 15 seconds for server initialization
|
||||||
|
timeout: 60000 # 60 seconds for tool operations
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note**: If operations are still being cut short, check your proxy configuration (e.g., nginx, traefik, etc.) which may be severing connections prematurely due to default timeouts.
|
||||||
|
|
||||||
|
### User Provided Credentials
|
||||||
|
|
||||||
|
You can allow users to provide their own credentials for MCP servers through `customUserVars`. This enables secure, user-specific authentication without storing credentials in configuration files.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
my-api-server:
|
||||||
|
type: streamable-http
|
||||||
|
url: "https://api.example.com/mcp"
|
||||||
|
headers:
|
||||||
|
X-Auth-Token: "{{MY_API_KEY}}" # Uses the user-provided value
|
||||||
|
customUserVars:
|
||||||
|
MY_API_KEY:
|
||||||
|
title: "API Key"
|
||||||
|
description: "Enter your personal API key from <a href='https://example.com/keys' target='_blank'>your account settings</a>"
|
||||||
|
```
|
||||||
|
|
||||||
|
Users can configure these credentials:
|
||||||
|
- **From Chat Area**: Click the settings icon next to configurable MCP servers in the tool selection dropdown
|
||||||
|
- **From MCP Settings Panel**: Access "MCP Settings" in the right panel to manage credentials for all configured servers
|
||||||
|
|
||||||
|
### OAuth Authentication
|
||||||
|
|
||||||
|
LibreChat supports OAuth authentication for MCP servers, following Anthropic's recommendation for secure MCP connections. OAuth provides a standardized, secure way to authenticate without storing long-lived credentials.
|
||||||
|
|
||||||
|
#### Supported OAuth Flows
|
||||||
|
|
||||||
|
LibreChat MCP servers support OAuth 2.0 with:
|
||||||
|
- **Authorization Code Flow with PKCE**: Recommended for maximum security
|
||||||
|
- **Client Discovery**: Automatic client registration when supported by the OAuth provider
|
||||||
|
- **Refresh Tokens**: Automatic token renewal when available
|
||||||
|
|
||||||
|
#### Configuration Examples
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcpServers:
|
||||||
|
# Public remote MCP server for PayPal, uses OAuth Client Discovery
|
||||||
|
# ❌ Refresh Tokens: you may need to re-authenticate periodically
|
||||||
|
# More info: https://developer.paypal.com/tools/mcp-server/
|
||||||
|
paypal:
|
||||||
|
type: "sse"
|
||||||
|
initTimeout: 150000 # higher timeout to allow for initial authentication
|
||||||
|
url: "https://mcp.paypal.com/sse"
|
||||||
|
|
||||||
|
# Example self-hosted remote MCP server for Spotify, uses OAuth Client Discovery
|
||||||
|
# ✅ Refresh Tokens: refreshes token for authentication automatically
|
||||||
|
# Hosted on Cloudflare Workers, more info: https://github.com/LibreChat-AI/spotify-mcp
|
||||||
|
spotify:
|
||||||
|
type: "streamable-http"
|
||||||
|
initTimeout: 150000
|
||||||
|
url: "https://mcp-spotify-oauth-example.account.workers.dev/mcp"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### OAuth Authentication Flow
|
||||||
|
|
||||||
|
When you first configure an OAuth-enabled MCP server:
|
||||||
|
|
||||||
|
1. **Initial Connection**: LibreChat attempts to connect to the MCP server
|
||||||
|
2. **Authentication Required**: If no valid token exists, you'll see an authentication URL in the console
|
||||||
|
3. **Browser Authentication**: Copy the URL and authenticate in your browser
|
||||||
|
4. **Token Storage**: LibreChat securely stores the tokens for future use
|
||||||
|
5. **Automatic Refresh**: If refresh tokens are supported, LibreChat handles renewal automatically
|
||||||
|
|
||||||
|
Example authentication prompt:
|
||||||
|
```bash
|
||||||
|
═══════════════════════════════════════════════════════════════════════
|
||||||
|
Please visit the following URL to authenticate:
|
||||||
|
|
||||||
|
https://mcp-server.example.com/authorize?response_type=code&client_id=...
|
||||||
|
|
||||||
|
═══════════════════════════════════════════════════════════════════════
|
||||||
|
```
|
||||||
|
|
||||||
|
Soon, you will be able to add MCP Servers and configure OAuth authentication directly from the LibreChat interface.
|
||||||
|
|
||||||
|
#### Token Management
|
||||||
|
|
||||||
|
LibreChat handles OAuth tokens intelligently:
|
||||||
|
|
||||||
|
- **Secure Storage**: Tokens are encrypted and stored securely
|
||||||
|
- **Automatic Refresh**: When refresh tokens are available, LibreChat automatically renews expired access tokens
|
||||||
|
- **Session Management**: Each user maintains their own OAuth sessions for multi-user environments
|
||||||
|
|
||||||
|
Each user will be prompted to authenticate with their own OAuth login when they first use an OAuth-enabled MCP server. This ensures that connection and authentication details are unique to each user, maintaining security and privacy in multi-user environments.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
> Note: The tokens shown during app startup are for app-level initialization only and are not used for individual user connections.
|
||||||
|
|
||||||
|
Example of automatic token refresh:
|
||||||
|
```bash
|
||||||
|
[MCP][spotify] Access token missing
|
||||||
|
[MCP][spotify] Attempting to refresh token
|
||||||
|
[MCP][spotify] Successfully refreshed and stored OAuth tokens
|
||||||
|
[MCP][spotify] ✓ Initialized
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Best Practices
|
||||||
|
|
||||||
|
1. **Use OAuth when available**: Prefer OAuth over API keys for better security
|
||||||
|
2. **Configure appropriate timeouts**: Set higher `initTimeout` for OAuth flows (e.g., 150000ms)
|
||||||
|
3. **Monitor token expiration**: Check logs for authentication issues
|
||||||
|
4. **Plan for re-authentication**: Some providers don't support refresh tokens
|
||||||
|
|
||||||
|
**Note**: UI-based OAuth configuration is coming soon, which will streamline the authentication process directly from the LibreChat interface.
|
||||||
|
|
||||||
|
## Server Transports
|
||||||
|
|
||||||
|
MCP servers can be configured to use different transport mechanisms:
|
||||||
|
|
||||||
|
**STDIO Servers**
|
||||||
|
- Work wells for local, single-user environments
|
||||||
|
- Not scalable for remote or cloud deployments
|
||||||
|
|
||||||
|
**Server-Sent Events (SSE) Servers**
|
||||||
|
- Remote transport mechanism but not recommended for production environment
|
||||||
|
|
||||||
|
**Streamable HTTP Servers**
|
||||||
|
- Uses HTTP POST for sending messages and supports streaming responses
|
||||||
|
- Operates as an independent process that can handle multiple client connections
|
||||||
|
- Supports both basic requests and streaming via Server-Sent Events (SSE)
|
||||||
|
- More performant alternative to the legacy HTTP+SSE transport
|
||||||
|
- Supports proper multi-user server configurations
|
||||||
|
|
||||||
|
**For production environments**, only MCP servers with ["Streamable HTTP" transports](https://modelcontextprotocol.io/specification/draft/basic/transports#streamable-http) are recommended. Unlike SSE which maintains long-running connections, Streamable HTTP offers stateless options that are better suited for scalable, multi-user deployments.
|
||||||
|
|
||||||
|
LibreChat is at the forefront of implementing flexible, scalable MCP server integrations to support diverse usage scenarios and help you build the AI workflows of tomorrow.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Ready to extend your AI capabilities?** Start by configuring your first MCP server and discover how LibreChat can connect to virtually any tool or service your organization needs.
|
||||||
@@ -21,91 +21,77 @@ OCR functionality in LibreChat allows agents to:
|
|||||||
|
|
||||||
Currently, OCR is **only available as an agent capability**. This means you must use an agent via the Agents endpoint to leverage OCR functionality.
|
Currently, OCR is **only available as an agent capability**. This means you must use an agent via the Agents endpoint to leverage OCR functionality.
|
||||||
|
|
||||||
## Configuration
|
## OCR Strategies
|
||||||
|
|
||||||
OCR can be enabled in the LibreChat configuration file (`librechat.yaml`). The OCR configuration supports three strategies:
|
LibreChat supports multiple OCR strategies to meet different deployment needs and requirements. Choose the strategy that best fits your infrastructure and compliance requirements.
|
||||||
|
|
||||||
1. **Mistral OCR** (Default)
|
### 1. Mistral OCR (Default)
|
||||||
2. **Azure Mistral OCR** (For Azure-deployed models)
|
|
||||||
3. **Custom OCR** (Planned for future releases)
|
|
||||||
|
|
||||||
### Basic Configuration Example
|
The default strategy uses Mistral's cloud API service directly. This is the simplest setup and requires only an API key from Mistral.
|
||||||
|
|
||||||
If using the Mistral OCR API, you only need the following environment variables to get started:
|
|
||||||
|
|
||||||
|
**Environment Variables:**
|
||||||
```.env
|
```.env
|
||||||
# `.env`
|
# `.env`
|
||||||
OCR_API_KEY=your-mistral-api-key
|
OCR_API_KEY=your-mistral-api-key
|
||||||
# OCR_BASEURL=https://api.mistral.ai/v1 # this is the default value
|
# OCR_BASEURL=https://api.mistral.ai/v1 # this is the default value
|
||||||
```
|
```
|
||||||
|
|
||||||
For additional, detailed configuration options, see the [OCR Config Object Structure](/docs/configuration/librechat_yaml/object_structure/ocr).
|
**Configuration:**
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# `librechat.yaml`
|
# `librechat.yaml`
|
||||||
ocr:
|
ocr:
|
||||||
mistralModel: "mistral-ocr-latest" # Optional: Specify Mistral model, defaults to "mistral-ocr-latest"
|
mistralModel: "mistral-ocr-latest" # Optional: Specify Mistral model, defaults to "mistral-ocr-latest"
|
||||||
apiKey: "your-mistral-api-key" # Optional: Defaults to OCR_API_KEY env variable
|
apiKey: "your-mistral-api-key" # Optional: Defaults to OCR_API_KEY env variable
|
||||||
baseURL: "https://api.mistral.ai/v1" # Optional: Defaults to OCR_BASEURL env variable, or Mistral's API if no variable set
|
baseURL: "https://api.mistral.ai/v1" # Optional: Defaults to OCR_BASEURL env variable, or Mistral's API if no variable set
|
||||||
strategy: "mistral_ocr" # Optional: Defaults to "mistral_ocr"
|
strategy: "mistral_ocr" # Optional: Defaults to "mistral_ocr"
|
||||||
```
|
```
|
||||||
|
|
||||||
### Azure Mistral OCR Configuration
|
**Key Features:**
|
||||||
|
|
||||||
For organizations using Azure AI Foundry, you can deploy Mistral OCR models to your Azure infrastructure:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# `librechat.yaml`
|
|
||||||
ocr:
|
|
||||||
mistralModel: "deployed-mistral-ocr-2503" # Should match your Azure deployment name
|
|
||||||
apiKey: "${AZURE_MISTRAL_OCR_API_KEY}" # Reference to your Azure API key in .env
|
|
||||||
baseURL: "https://your-deployed-endpoint.models.ai.azure.com/v1" # Your Azure endpoint
|
|
||||||
strategy: "azure_mistral_ocr" # Use Azure strategy
|
|
||||||
```
|
|
||||||
|
|
||||||
## Mistral OCR
|
|
||||||
|
|
||||||
LibreChat supports Mistral's OCR capabilities through two deployment options:
|
|
||||||
|
|
||||||
### 1. Standard Mistral OCR API
|
|
||||||
|
|
||||||
The default option uses Mistral's cloud API service directly. This is the simplest setup and requires only an API key from Mistral.
|
|
||||||
|
|
||||||
### 2. Azure-Deployed Mistral OCR
|
|
||||||
|
|
||||||
For organizations with specific data residency or compliance requirements, Mistral OCR models can be deployed on Azure AI Foundry. Currently, the **Mistral OCR 2503** model is available for Azure deployment.
|
|
||||||
|
|
||||||
Benefits of Azure deployment:
|
|
||||||
- **Data Residency**: Keep data processing within your Azure region
|
|
||||||
- **Compliance**: Meet organizational security and compliance requirements
|
|
||||||
- **Integration**: Leverage existing Azure infrastructure and security policies
|
|
||||||
- **Control**: Manage your own deployment and scaling
|
|
||||||
|
|
||||||
You can explore the latest Mistral OCR model available on Azure AI Foundry here (requires Azure subscription):
|
|
||||||
|
|
||||||
https://ai.azure.com/explore/models/mistral-ocr-2503
|
|
||||||
|
|
||||||
### Key Features of Mistral OCR
|
|
||||||
|
|
||||||
- **Document Structure Preservation**: Maintains formatting like headers, paragraphs, lists, and tables
|
- **Document Structure Preservation**: Maintains formatting like headers, paragraphs, lists, and tables
|
||||||
- **Multilingual Support**: Processes text in multiple languages and scripts
|
- **Multilingual Support**: Processes text in multiple languages and scripts
|
||||||
- **Complex Layout Handling**: Handles multi-column text and mixed content
|
- **Complex Layout Handling**: Handles multi-column text and mixed content
|
||||||
- **Mathematical Expression Recognition**: Accurately processes equations and formulas
|
- **Mathematical Expression Recognition**: Accurately processes equations and formulas
|
||||||
- **High-Speed Processing**: Processes up to 2000 pages per minute
|
- **High-Speed Processing**: Processes up to 2000 pages per minute
|
||||||
|
|
||||||
### Important Considerations
|
**Considerations:**
|
||||||
|
|
||||||
- **Cost**: Using Mistral OCR may incur costs as it's a paid API service (though free trials may be available)
|
- **Cost**: Using Mistral OCR may incur costs as it's a paid API service (though free trials may be available)
|
||||||
- **Data Privacy**: Data processed through Mistral OCR is subject to Mistral's cloud environment and their terms of service
|
- **Data Privacy**: Data processed through Mistral OCR is subject to Mistral's cloud environment and their terms of service
|
||||||
- **Document Limitations**:
|
- **Document Limitations**:
|
||||||
- Maximum file size: 50 MB
|
- Maximum file size: 50 MB
|
||||||
- Maximum document length: 1,000 pages
|
- Maximum document length: 1,000 pages
|
||||||
|
|
||||||
### Future Plans
|
### 2. Azure Mistral OCR
|
||||||
|
|
||||||
- Mistral plans to make their OCR API available through their cloud partners, such as GCP and AWS, and enterprise self-hosting for organizations with stringent data privacy requirements ([source](https://mistral.ai/fr/news/mistral-ocr)).
|
For organizations using Azure AI Foundry, you can deploy Mistral OCR models to your Azure infrastructure. Currently, the **Mistral OCR 2503** model is available for Azure deployment.
|
||||||
- LibreChat will continue to support Mistral OCR and explore additional OCR providers, including open-source solutions, for enhanced functionality.
|
|
||||||
- LibreChat currently does not include the parsed image content from the OCR process in its responses, even though services like [Mistral's OCR API may provide](https://docs.mistral.ai/api/#tag/ocr) these in the result. This feature may be supported in future updates.
|
**Configuration:**
|
||||||
|
```yaml
|
||||||
|
# `librechat.yaml`
|
||||||
|
ocr:
|
||||||
|
mistralModel: "deployed-mistral-ocr-2503" # Should match your Azure deployment name
|
||||||
|
apiKey: "${AZURE_MISTRAL_OCR_API_KEY}" # Reference to your Azure API key in .env
|
||||||
|
baseURL: "https://your-deployed-endpoint.models.ai.azure.com/v1" # Your Azure endpoint
|
||||||
|
strategy: "azure_mistral_ocr" # Use Azure strategy
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benefits:**
|
||||||
|
- **Data Residency**: Keep data processing within your Azure region
|
||||||
|
- **Compliance**: Meet organizational security and compliance requirements
|
||||||
|
- **Integration**: Leverage existing Azure infrastructure and security policies
|
||||||
|
- **Control**: Manage your own deployment and scaling
|
||||||
|
|
||||||
|
**Azure Model Information:**
|
||||||
|
You can explore the latest Mistral OCR model available on Azure AI Foundry here (requires Azure subscription):
|
||||||
|
|
||||||
|
https://ai.azure.com/explore/models/mistral-ocr-2503
|
||||||
|
|
||||||
|
### 3. Custom OCR (Planned)
|
||||||
|
|
||||||
|
Support for custom OCR providers and user-defined strategies is planned for future releases.
|
||||||
|
|
||||||
|
## Detailed Configuration
|
||||||
|
|
||||||
|
For additional, detailed configuration options, see the [OCR Config Object Structure](/docs/configuration/librechat_yaml/object_structure/ocr).
|
||||||
|
|
||||||
## Using File Context (OCR) in LibreChat
|
## Using File Context (OCR) in LibreChat
|
||||||
|
|
||||||
@@ -160,6 +146,8 @@ LibreChat plans to expand OCR capabilities in future releases:
|
|||||||
- Integration with open-source OCR solutions
|
- Integration with open-source OCR solutions
|
||||||
- Enhanced document processing options
|
- Enhanced document processing options
|
||||||
- More granular control over OCR settings
|
- More granular control over OCR settings
|
||||||
|
- Mistral plans to make their OCR API available through their cloud partners, such as GCP and AWS, and enterprise self-hosting for organizations with stringent data privacy requirements ([source](https://mistral.ai/fr/news/mistral-ocr))
|
||||||
|
- LibreChat currently does not include the parsed image content from the OCR process in its responses, even though services like [Mistral's OCR API may provide](https://docs.mistral.ai/api/#tag/ocr) these in the result. This feature may be supported in future updates.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
BIN
public/images/agents/mcp_agent_tools.png
Normal file
|
After Width: | Height: | Size: 23 KiB |
BIN
public/images/agents/mcp_chat.png
Normal file
|
After Width: | Height: | Size: 21 KiB |
BIN
public/images/agents/mcp_oauth_flow.png
Normal file
|
After Width: | Height: | Size: 22 KiB |
BIN
public/images/agents/mcp_oauth_success.png
Normal file
|
After Width: | Height: | Size: 20 KiB |
|
Before Width: | Height: | Size: 113 KiB |
BIN
public/images/agents/mcp_tools_v2.png
Normal file
|
After Width: | Height: | Size: 143 KiB |
BIN
public/images/mcp/mcp_per_user_vars_assistant.png
Normal file
|
After Width: | Height: | Size: 71 KiB |
BIN
public/images/mcp/mcp_per_user_vars_assistant_dialog.png
Normal file
|
After Width: | Height: | Size: 245 KiB |
BIN
public/images/mcp/mcp_per_user_vars_side_panel.png
Normal file
|
After Width: | Height: | Size: 90 KiB |