khanh.pndc/nextcloud-docs
1
0
Fork 0
mirror of https://github.com/nextcloud/documentation.git synced 2026-01-02 17:59:36 +07:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #13934 from nextcloud/chore/update-ai

Browse Source 
chore(admin): Update AI docs
This commit is contained in:
Marcel Klehr
2025-12-15 09:23:12 +01:00
committed by GitHub
parent 021b2e3590 63739be4f6
commit 0334145fc9
6 changed files with 91 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
admin_manual/ai/app_assistant.rst
View File

@@ -57,7 +57,7 @@ Speech-To-Text
In order to make use of Speech-to-Text, you will need an app that provides a Speech-To-Text backend:
* :ref:`stt_whisper2<ai-app-stt_whisper2>` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Text processing
~~~~~~~~~~~~~~~
@@ -65,7 +65,7 @@ Text processing
In order to make use of text processing features in the assistant, you will need an app that provides a Text processing backend:
* :ref:`llm2<ai-app-llm2>` - Runs open source AI language models locally on your own server hardware (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* * `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_watsonx* - Integrates with the IBM watsonx.ai API to provide AI functionality from IBM Cloud servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
These apps currently implement the following Assistant Tasks:
@@ -85,18 +85,40 @@ Text-To-Image
In order to make use of Text-To-Image features, you will need an app that provides an image generation backend:
* :ref:`tex2image_stablediffusion2<ai-app-text2image_stablediffusion2>` (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* * `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)
Context Chat
~~~~~~~~~~~~
In order to make use of our special Context Chat feature, offering in-context insights about your own data stored in Nextcloud, you will need the following apps:
In order to make use of our special Context Chat feature, offering insights about your own documents and data stored in Nextcloud, you will need the following apps:
* :ref:`context_chat + context_chat_backend<ai-app-context_chat>` - (Customer support available upon request)
You will also need a text processing provider as specified above (ie. llm2, integration_openai or integration_watsonx).
Chat
~~~~
In order to make use of our "Chat with AI" feature you will need any one of the following apps:
* :ref:`llm2<ai-app-llm2>` - Runs open source AI language models locally on your own server hardware (Customer support available upon request)
* * `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Voice Chat
~~~~~~~~~~
In order to make use of our "Audio chat" feature which allows you to interact with the assistant Chat via your Voice and Ears as if in a real conversation, you will need any of the following set of apps:
* Either
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* Or
* :ref:`llm2<ai-app-llm2>` - Runs open source AI language models locally on your own server hardware (Customer support available upon request)
* :ref:`stt_whisper2<ai-app-stt_whisper2>` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request)
* :ref:`text2speech_kokoro<ai-app-text2speech_kokoro>` - Runs open source AI Text-To-Speech models on your own server hardware (Customer support available upon request)
Context Agent
~~~~~~~~~~~~~
@@ -111,10 +133,10 @@ You will also need a text processing provider as specified above (ie. *llm2* or
Text-To-Speech
~~~~~~~~~~~~~~
In order to make use of Text-To-Speech, you will need an app that provides a Text-To-Speech backend:
In order to make use of Text-To-Speech, you will need an app that provides a Text-To-Speech backend, which is one of the following:
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *text2speech_kokoro* - Runs a local model
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* :ref:`text2speech_kokoro<ai-app-text2speech_kokoro>` - Runs open source AI Text-To-Speech models on your own server hardware (Customer support available upon request)
Configuration
-------------

2
admin_manual/ai/app_context_agent.rst
View File

@@ -4,7 +4,7 @@ App: Context Agent (context_agent)
.. _ai-app-context_agent:
The *context_agent* app is the app that provides AI agent functionality in Nextcloud and acts as a backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
The *context_agent* app is the app that provides AI agent functionality in Nextcloud's "Chat with AI" feature and acts as a backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
When the Context Agent app is installed the AI Chat in Nextcloud Assistant will be able to interact with your Nextcloud apps via virtual integrations that are called "tools". They allow the Assistant to perform actions in your Nextcloud upon sending instructions in a chat message.
Tool groups are only available if their requirements are met. They can be en- and disabled in the AI admin settings.

2
admin_manual/ai/app_stt_whisper2.rst
View File

@@ -10,7 +10,7 @@ This app supports input and output in languages other than English if the underl
This app uses `faster-whisper <https://github.com/SYSTRAN/faster-whisper>`_ under the hood. Output quality will differ depending on which model you use, we recommend the following models:
* OpenAI Whisper large-v2 or v3 (multilingual)
* OpenAI Whisper large v3 turbo (multilingual)
* OpenAI Whisper medium.en (English only)
Whisper large v3 supports about ~100 languages and shows outstanding performance in ~10 of them. For more details see the `OpenAI Whisper paper <https://cdn.openai.com/papers/whisper.pdf>`_

BIN
admin_manual/ai/images/ai_overview.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 90 KiB

After

Width:  |  Height:  |  Size: 225 KiB

2
admin_manual/ai/images/ai_overview.svg
View File

File diff suppressed because one or more lines are too long
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 280 KiB

After

Width:  |  Height:  |  Size: 84 KiB

71
admin_manual/ai/overview.rst
View File

@@ -12,7 +12,7 @@ Overview of AI features
Nextcloud uses modularity to separate raw AI functionality from the Graphical User interfaces and apps that make use of said functionality. Each instance can thus make use of various backends that provide the functionality for the same frontends and the same functionality can be implemented by multiple apps using on-premises processing or third-party AI service providers.
.. figure:: images/ai_overview.png
:scale: 80%
:scale: 60%
.. csv-table::
:header: "Feature","App","Rating","Open source","Freely available model","Freely available training data","Privacy: Keeps data on premises"
@@ -53,7 +53,7 @@ Nextcloud uses modularity to separate raw AI functionality from the Graphical Us
"","`OpenAI and LocalAI integration (via Groqcloud) <https://apps.nextcloud.com/apps/integration_openai>`_","Orange","No","Yes","No","No"
"","`OpenAI and LocalAI integration (via MistralAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Orange","No","Yes","No","No"
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No"
"Image generation","`Local Stable Diffusion <https://apps.nextcloud.com/apps/text2image_stablediffusion>`_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes"
"Image generation","`Local Stable Diffusion 2 (ExApp) <https://apps.nextcloud.com/apps/text2image_stablediffusion2>`_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes"
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","No"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Green","Yes","Yes","Yes","Yes"
@@ -65,10 +65,11 @@ Nextcloud uses modularity to separate raw AI functionality from the Graphical Us
"Context Chat","`Nextcloud Assistant Context Chat <https://apps.nextcloud.com/apps/context_chat>`_","Yellow","Yes","Yes","No","Yes"
"","`Nextcloud Assistant Context Chat (Backend) <https://apps.nextcloud.com/apps/context_chat_backend>`_","Yellow","Yes","Yes","No","Yes"
"Context Chat Search","`Nextcloud Assistant Context Chat <https://apps.nextcloud.com/apps/context_chat>`_","Yellow","Yes","Yes","No","Yes"
"Context Agent","`Nextcloud Context Agent <https://apps.nextcloud.com/apps/context_agent>`_","Green","Yes","Yes","Yes","Yes"
"Context Agent","`Nextcloud Context Agent (ExApp) <https://apps.nextcloud.com/apps/context_agent>`_","Green","Yes","Yes","Yes","Yes"
"Text To Speech","`Open AI Text To Speech <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`Local Text To Speech <https://apps.nextcloud.com/apps/text2speech_kokoro>`_","Yellow","Yes","Yes","No","Yes"
"","`Local Text To Speech (ExApp) <https://apps.nextcloud.com/apps/text2speech_kokoro>`_","Yellow","Yes","Yes","No","Yes"
"Document generation","`Nextcloud Office <https://apps.nextcloud.com/apps/richdocuments>`_","Green","Yes","Yes","Yes","Yes"
"Live Transcription","`Local Live Transcription <https://apps.nextcloud.com/apps/live_transcription>`_","Yellow","Yes","Yes","No","Yes"
Ethical AI Rating
@@ -94,7 +95,7 @@ Features used by other apps
---------------------------
Some of our AI features are realized as generic APIs that any app can use and any app can provide an implementation for by registering a provider. So far, these are
Machine translation, Speech-To-Text, Image generation, Text processing and Context Chat.
Machine translation, Speech-To-Text, Text-To-Speech, Image generation, Text processing and Context Chat.
Text processing
^^^^^^^^^^^^^^^
@@ -108,9 +109,10 @@ Frontend apps
~~~~~~~~~~~~~
* *Text* for offering an inline graphical UI for the various tasks
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI for the various tasks and a smart picker
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI for the various tasks, a smart picker and "Chat with AI" functionality
* `Mail <https://apps.nextcloud.com/apps/mail>`_ for summarizing mail threads (see :ref:`the Nextcloud Mail docs<mail_thread_summary>` for how to enable this)
* `Summary Bot <https://apps.nextcloud.com/apps/summary_bot>`_ for summarizing chat histories in `Talk <https://apps.nextcloud.com/apps/spreed>`_
* `Talk <https://apps.nextcloud.com/apps/spreed>`_ for summarizing chat history (see `Nextcloud Talk docs <https://nextcloud-talk.readthedocs.io/en/latest/settings/#app-configuration>`_ for how to enable this)
Backend apps
@@ -140,6 +142,7 @@ Backend apps
~~~~~~~~~~~~
* :ref:`translate2 (ExApp)<ai-app-translate2>` - Runs open source AI translation models locally on your own server hardware (Customer support available upon request)
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_deepl* - Integrates with the deepl API to provide translation functionality from Deepl.com servers (Only community supported)
Speech-To-Text
@@ -152,13 +155,13 @@ As you can see in the table above we have multiple apps offering Speech-To-Text
Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ offering a graphical translation UI and a smart picker
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ offering a graphical translation UI, a smart picker and Audio Chat
* `Talk <https://apps.nextcloud.com/apps/spreed>`_ for transcribing calls (see `Nextcloud Talk docs <https://nextcloud-talk.readthedocs.io/en/latest/settings/#app-configuration>`_ for how to enable this)
Backend apps
~~~~~~~~~~~~
* :ref:`stt_whisper2<ai-app-stt_whisper2>` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request)
* :ref:`stt_whisper2<ai-app-stt_whisper2>` - Runs open weights AI Speech-To-Text models on your own server hardware (Customer support available upon request)
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
@@ -177,7 +180,7 @@ Frontend apps
Backend apps
~~~~~~~~~~~~
* text2image_stablediffusion (Customer support available upon request)
* `Local Stable Diffusion 2 (ExApp) <https://apps.nextcloud.com/apps/text2image_stablediffusion2>`_ (Customer support available upon request)
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)
@@ -198,7 +201,7 @@ Backend apps
~~~~~~~~~~~~
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *text2speech_kokoro* (Runs a local model)
* `Local Text To Speech (ExApp) <https://apps.nextcloud.com/apps/text2speech_kokoro>`_ (Customer support available upon request)
Context Chat
@@ -209,6 +212,7 @@ Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI for the context chat tasks
* `Nextcloud Context Agent <https://apps.nextcloud.com/apps/context_agent>`_ for offering Context chat as a tool that the agent can execute in the "Chat with AI" feature
Backend apps
~~~~~~~~~~~~
@@ -222,8 +226,8 @@ Apps can integrate their content with Context Chat to make it available for quer
* *files*
* `Analytics <https://apps.nextcloud.com/apps/analytics>`_
* `Mail <https://apps.nextcloud.com/apps/mail>`_ (coming soon)
* `Bookmarks <https://apps.nextcloud.com/apps/bookmarks>`_
Context Chat Search
^^^^^^^^^^^^^^^^^^^
@@ -244,6 +248,28 @@ Provider apps
See *Context Chat* section above.
Context Agent
^^^^^^^^^^^^^
Our Context Agent feature was introduced in Nextcloud Hub 9 (v30). It allows asking the assistant to execute tasks related to Nextcloud. You will need to install both the context_agent app as well as a text processing provider.
Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI for the "Chat with AI" feature
Backend apps
~~~~~~~~~~~~
* `Nextcloud Context Agent <https://apps.nextcloud.com/apps/context_agent>`_ for agentic AI capabilities in the "Chat with AI" feature (Customer support available upon request)
Provider apps
~~~~~~~~~~~~~
* :ref:`llm2<ai-app-llm2>` - Runs open source AI LLM models on your own server hardware (Customer support available upon request)
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Document generation
^^^^^^^^^^^^^^^^^^^
Since Hub 11 you can let Nextcloud automatically generate Office documents with content.
@@ -259,6 +285,27 @@ Backend apps
* `Nextcloud Office <https://apps.nextcloud.com/apps/richdocuments>`_
Provider apps
~~~~~~~~~~~~~
* :ref:`llm2<ai-app-llm2>` - Runs open source AI LLM models on your own server hardware (Customer support available upon request)
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* `IBM watsonx.ai integration (via IBM watsonx.ai as a Service) <https://apps.nextcloud.com/apps/integration_watsonx>`_ - Integrates with the IBM watsonx.ai API to provide AI functionality from IBM Cloud servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Live transcription
^^^^^^^^^^^^^^^^^^
Since Hub 25 Autumn you can let Nextcloud automatically generate produce subtitles for video and audio calls in Nextcloud Talk.
Frontend apps
~~~~~~~~~~~~~
* `Talk <https://apps.nextcloud.com/apps/spreed>`_ for displaying the subtitles in calls
Backend apps
~~~~~~~~~~~~
* :ref:`live_transcription<ai-live-transcription>` - Runs open weights AI Speech-To-Text models on your own server hardware (Customer support available upon request)
.. _ai-overview_improve-ai-task-pickup-speed: