diff --git a/admin_manual/ai/app_assistant.rst b/admin_manual/ai/app_assistant.rst index 35abe1068..9e18f9faa 100644 --- a/admin_manual/ai/app_assistant.rst +++ b/admin_manual/ai/app_assistant.rst @@ -57,7 +57,7 @@ Speech-To-Text In order to make use of Speech-to-Text, you will need an app that provides a Speech-To-Text backend: * :ref:`stt_whisper2` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request) -* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) +* `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) Text processing ~~~~~~~~~~~~~~~ @@ -65,7 +65,7 @@ Text processing In order to make use of text processing features in the assistant, you will need an app that provides a Text processing backend: * :ref:`llm2` - Runs open source AI language models locally on your own server hardware (Customer support available upon request) -* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) +* * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) * *integration_watsonx* - Integrates with the IBM watsonx.ai API to provide AI functionality from IBM Cloud servers (Customer support available upon request; see :ref:`AI as a Service`) These apps currently implement the following Assistant Tasks: @@ -85,18 +85,40 @@ Text-To-Image In order to make use of Text-To-Image features, you will need an app that provides an image generation backend: * :ref:`tex2image_stablediffusion2` (Customer support available upon request) -* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) +* * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) * *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service`) Context Chat ~~~~~~~~~~~~ -In order to make use of our special Context Chat feature, offering in-context insights about your own data stored in Nextcloud, you will need the following apps: +In order to make use of our special Context Chat feature, offering insights about your own documents and data stored in Nextcloud, you will need the following apps: * :ref:`context_chat + context_chat_backend` - (Customer support available upon request) You will also need a text processing provider as specified above (ie. llm2, integration_openai or integration_watsonx). +Chat +~~~~ + +In order to make use of our "Chat with AI" feature you will need the following apps: + +* :ref:`llm2` - Runs open source AI language models locally on your own server hardware (Customer support available upon request) +* * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) + + +Voice Chat +~~~~~~~~~~ + +In order to make use of our "Audio chat" feature which allows you to interact with the assistant Chat via your Voice and Ears as if in a real conversation, you will need the following apps: + +* Either + * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) + +* Or + * :ref:`llm2` - Runs open source AI language models locally on your own server hardware (Customer support available upon request) + * :ref:`stt_whisper2` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request) + * :ref:`text2speech_kokoro` - Runs open source AI Text-To-Speech models on your own server hardware (Customer support available upon request) + Context Agent ~~~~~~~~~~~~~ @@ -111,10 +133,10 @@ You will also need a text processing provider as specified above (ie. *llm2* or Text-To-Speech ~~~~~~~~~~~~~~ -In order to make use of Text-To-Speech, you will need an app that provides a Text-To-Speech backend: +In order to make use of Text-To-Speech, you will need an app that provides a Text-To-Speech backend, which is one of the following: -* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) -* *text2speech_kokoro* - Runs a local model +* `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) +* :ref:`text2speech_kokoro` - Runs open source AI Text-To-Speech models on your own server hardware (Customer support available upon request) Configuration ------------- diff --git a/admin_manual/ai/app_context_agent.rst b/admin_manual/ai/app_context_agent.rst index 46a569146..9da158762 100644 --- a/admin_manual/ai/app_context_agent.rst +++ b/admin_manual/ai/app_context_agent.rst @@ -4,7 +4,7 @@ App: Context Agent (context_agent) .. _ai-app-context_agent: -The *context_agent* app is the app that provides AI agent functionality in Nextcloud and acts as a backend for the :ref:`Nextcloud Assistant app`. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. +The *context_agent* app is the app that provides AI agent functionality in Nextcloud's "Chat with AI" feature and acts as a backend for the :ref:`Nextcloud Assistant app`. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. When the Context Agent app is installed the AI Chat in Nextcloud Assistant will be able to interact with your Nextcloud apps via virtual integrations that are called "tools". They allow the Assistant to perform actions in your Nextcloud upon sending instructions in a chat message. Tool groups are only available if their requirements are met. They can be en- and disabled in the AI admin settings. diff --git a/admin_manual/ai/app_stt_whisper2.rst b/admin_manual/ai/app_stt_whisper2.rst index 92c28c0d9..9dbae154e 100644 --- a/admin_manual/ai/app_stt_whisper2.rst +++ b/admin_manual/ai/app_stt_whisper2.rst @@ -10,7 +10,7 @@ This app supports input and output in languages other than English if the underl This app uses `faster-whisper `_ under the hood. Output quality will differ depending on which model you use, we recommend the following models: - * OpenAI Whisper large-v2 or v3 (multilingual) + * OpenAI Whisper large v3 turbo (multilingual) * OpenAI Whisper medium.en (English only) Whisper large v3 supports about ~100 languages and shows outstanding performance in ~10 of them. For more details see the `OpenAI Whisper paper `_ diff --git a/admin_manual/ai/images/ai_overview.png b/admin_manual/ai/images/ai_overview.png index 509cc40d8..3ca9e1838 100644 Binary files a/admin_manual/ai/images/ai_overview.png and b/admin_manual/ai/images/ai_overview.png differ diff --git a/admin_manual/ai/images/ai_overview.svg b/admin_manual/ai/images/ai_overview.svg index e98f4bf9f..ea0ace7a1 100644 --- a/admin_manual/ai/images/ai_overview.svg +++ b/admin_manual/ai/images/ai_overview.svg @@ -1,4 +1,4 @@ -
OpenAI integration
llm2
translate2
stt_whisper2
Text processing
(e.g. Summary)
Translation
Transcription
Image generation
OpenAI integration
OpenAI integration
OpenAI integration
text2image_
stablediffusion
Context Chat
AI backend Apps
AI frontend Apps
AI task types
Deepl integration
Context Agent
Nextcloud Assistant
Nextcloud Mail
Nextcloud Office
Nextcloud Talk
Replicate
integration
 \ No newline at end of file +
OpenAI integration
llm2
translate2
stt_whisper2
Text processing
(e.g. Summary, Chat)
Translation
Transcription
Image generation
OpenAI integration
OpenAI integration
OpenAI integration
text2image_
stablediffusion2
Context Chat
AI backend Apps
AI frontend Apps
AI task types
Deepl integration
Context Agent
Nextcloud Mail
Nextcloud Office
Replicate
integration
Text-To-Speech
OpenAI integration
text2speech_
kokoro
Nextcloud Assistant
Live Transcription
Nextcloud Talk
Data provider Apps
Bookmarks
Mail
Analytics
Files
 \ No newline at end of file diff --git a/admin_manual/ai/overview.rst b/admin_manual/ai/overview.rst index e75beb508..6efbf0820 100644 --- a/admin_manual/ai/overview.rst +++ b/admin_manual/ai/overview.rst @@ -12,7 +12,7 @@ Overview of AI features Nextcloud uses modularity to separate raw AI functionality from the Graphical User interfaces and apps that make use of said functionality. Each instance can thus make use of various backends that provide the functionality for the same frontends and the same functionality can be implemented by multiple apps using on-premises processing or third-party AI service providers. .. figure:: images/ai_overview.png - :scale: 80% + :scale: 60% .. csv-table:: :header: "Feature","App","Rating","Open source","Freely available model","Freely available training data","Privacy: Keeps data on premises" @@ -53,7 +53,7 @@ Nextcloud uses modularity to separate raw AI functionality from the Graphical Us "","`OpenAI and LocalAI integration (via Groqcloud) `_","Orange","No","Yes","No","No" "","`OpenAI and LocalAI integration (via MistralAI) `_","Orange","No","Yes","No","No" "","`Replicate integration `_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No" - "Image generation","`Local Stable Diffusion `_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes" + "Image generation","`Local Stable Diffusion 2 (ExApp) `_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes" "","`Replicate integration `_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","No" "","`OpenAI and LocalAI integration (via OpenAI API) `_","Red","No","No","No","No" "","`OpenAI and LocalAI integration (via LocalAI) `_","Green","Yes","Yes","Yes","Yes" @@ -67,8 +67,9 @@ Nextcloud uses modularity to separate raw AI functionality from the Graphical Us "Context Chat Search","`Nextcloud Assistant Context Chat `_","Yellow","Yes","Yes","No","Yes" "Context Agent","`Nextcloud Context Agent `_","Green","Yes","Yes","Yes","Yes" "Text To Speech","`Open AI Text To Speech `_","Red","No","No","No","No" - "","`Local Text To Speech `_","Yellow","Yes","Yes","No","Yes" + "","`Local Text To Speech (ExApp) `_","Yellow","Yes","Yes","No","Yes" "Document generation","`Nextcloud Office `_","Green","Yes","Yes","Yes","Yes" + "Live Transcription","`Local Live Transcription `_","Green","Yes","Yes","Yes","Yes" Ethical AI Rating @@ -94,7 +95,7 @@ Features used by other apps --------------------------- Some of our AI features are realized as generic APIs that any app can use and any app can provide an implementation for by registering a provider. So far, these are -Machine translation, Speech-To-Text, Image generation, Text processing and Context Chat. +Machine translation, Speech-To-Text, Text-To-Speech, Image generation, Text processing and Context Chat. Text processing ^^^^^^^^^^^^^^^ @@ -108,9 +109,10 @@ Frontend apps ~~~~~~~~~~~~~ * *Text* for offering an inline graphical UI for the various tasks -* `Assistant `_ for offering a graphical UI for the various tasks and a smart picker +* `Assistant `_ for offering a graphical UI for the various tasks, a smart picker and "Chat with AI" functionality * `Mail `_ for summarizing mail threads (see :ref:`the Nextcloud Mail docs` for how to enable this) * `Summary Bot `_ for summarizing chat histories in `Talk `_ +* `Talk `_ for summarizing chat history (see `Nextcloud Talk docs `_ for how to enable this) Backend apps @@ -140,6 +142,7 @@ Backend apps ~~~~~~~~~~~~ * :ref:`translate2 (ExApp)` - Runs open source AI translation models locally on your own server hardware (Customer support available upon request) +* `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) * *integration_deepl* - Integrates with the deepl API to provide translation functionality from Deepl.com servers (Only community supported) Speech-To-Text @@ -152,7 +155,7 @@ As you can see in the table above we have multiple apps offering Speech-To-Text Frontend apps ~~~~~~~~~~~~~ -* `Assistant `_ offering a graphical translation UI and a smart picker +* `Assistant `_ offering a graphical translation UI, a smart picker and Audio Chat * `Talk `_ for transcribing calls (see `Nextcloud Talk docs `_ for how to enable this) Backend apps @@ -177,7 +180,7 @@ Frontend apps Backend apps ~~~~~~~~~~~~ -* text2image_stablediffusion (Customer support available upon request) +* `Local Stable Diffusion 2 (ExApp) `_ (Customer support available upon request) * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) * *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service`) @@ -198,7 +201,7 @@ Backend apps ~~~~~~~~~~~~ * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) -* *text2speech_kokoro* (Runs a local model) +* `Local Text To Speech (ExApp) `_ (Customer support available upon request) Context Chat @@ -209,6 +212,7 @@ Frontend apps ~~~~~~~~~~~~~ * `Assistant `_ for offering a graphical UI for the context chat tasks +* `Nextcloud Context Agent `_ for offering Context chat as a tool that the agent can execute in the "Chat with AI" feature Backend apps ~~~~~~~~~~~~ @@ -222,8 +226,8 @@ Apps can integrate their content with Context Chat to make it available for quer * *files* * `Analytics `_ - - +* `Mail `_ (coming soon) +* `Bookmarks `_ Context Chat Search ^^^^^^^^^^^^^^^^^^^ @@ -244,6 +248,28 @@ Provider apps See *Context Chat* section above. + +Context Agent +^^^^^^^^^^^^^ +Our Context Agent feature was introduced in Nextcloud Hub 9 (v30). It allows asking the assistant to execute tasks related to Nextcloud. You will need to install both the context_agent app as well as a text processing provider. + +Frontend apps +~~~~~~~~~~~~~ + +* `Assistant `_ for offering a graphical UI for the "Chat with AI" feature + +Backend apps +~~~~~~~~~~~~ + +* `Nextcloud Context Agent `_ (Customer support available upon request) + +Provider apps +~~~~~~~~~~~~~ + +* :ref:`llm2` - Runs open source AI LLM models on your own server hardware (Customer support available upon request) +* `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) + + Document generation ^^^^^^^^^^^^^^^^^^^ Since Hub 11 you can let Nextcloud automatically generate Office documents with content. @@ -259,6 +285,27 @@ Backend apps * `Nextcloud Office `_ +Provider apps +~~~~~~~~~~~~~ + +* :ref:`llm2` - Runs open source AI LLM models on your own server hardware (Customer support available upon request) +* `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) +* `IBM watsonx.ai integration (via IBM watsonx.ai as a Service) `_ - Integrates with the IBM watsonx.ai API to provide AI functionality from IBM Cloud servers (Customer support available upon request; see :ref:`AI as a Service`) + + +Live transcription +^^^^^^^^^^^^^^^^^^ +Since Hub 25 Autumn you can let Nextcloud automatically generate produce subtitles for video and audio calls in Nextcloud Talk. + +Frontend apps +~~~~~~~~~~~~~ + +* `Talk `_ for displaying the subtitles in calls + +Backend apps +~~~~~~~~~~~~ + +* :ref:`live_transcription` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request) .. _ai-overview_improve-ai-task-pickup-speed: