khanh.pndc/nextcloud-docs
1
0
Fork 0
mirror of https://github.com/nextcloud/documentation.git synced 2026-01-03 18:26:42 +07:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #11665 from nextcloud/admin/ai-setup

Browse Source 
enh(admin_manual): Extend AI section to contain AI setup
This commit is contained in:
Marcel Klehr
2024-03-28 15:10:56 +01:00
committed by GitHub
parent 1f8a32bb1f 91b8e2f3e5
commit c5917f135d
9 changed files with 639 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
admin_manual/ai/ai_as_a_service.rst Normal file
View File

@@ -0,0 +1,17 @@
===============
AI as a Service
===============
.. _ai-ai_as_a_service:
At Nextcloud we focus on creating on-premise AI apps that run fully self-hosted on your own servers in order to preserve your privacy and data sovereignty. However, you can also offload these resource-heavy tasks to an "AI as a Service" provider offering API access in exchange for payment. Examples of such providers are `OpenAI <https://platform.openai.com/>`_, with its ChatGPT APIs providing language model access among other APIs as well as `replicate <https://replicate.com/>`_.
Installation
------------
In order to use these providers you will need to install the respective app from the app store:
* ``integration_openai``
* ``integration_replicate``
You can then add your API token and rate limits in the administration settings and set the providers live in the "Artificial intelligence" section of the admins settings.

36
admin_manual/ai/app_api_and_external_apps.rst Normal file
View File

@@ -0,0 +1,36 @@
========================
AppAPI and External Apps
========================
.. _ai-app_api:
Previously, Nextcloud only supported applications written in the PHP programming language. In order to support a wider range of use cases,
the External App ecosystem was introduced, which allows installing apps that are docker containers.
Most of our AI apps nowadays are ExApps (External Apps) and thus require some preparation of your Nextcloud before you can install them.
Installation
------------
1. You will need to install the `AppAPI <https://apps.nextcloud.com/apps/app_api>`_ Nextcloud app from the app store.
2. You will need to set up a Deploy Daemon in the AppAPI admin settings. A Deploy Daemon is a way for Nextcloud to install and communicate with and control External Apps.
1. You will need to `setup a docker container called docker-socket-proxy that proxies access to docker for your Nextcloud instance <https://github.com/cloud-py-api/docker-socket-proxy#readme>`_
2. Now you can connect your Nextcloud to the docker-socket-proxy by entering its details in the Deploy Daemon creation form in the AppAPI settings.
3. Make sure to enable GPU support if you want the installed ExApps to be able to use the GPU
3. You can now install ExApps from the Nextcloud Appstore by clicking "Install" on the respective app in the AppAPI apps page.
FAQ
---
* I have two graphic cards XXX with 6/8/Y GB of ram each, how can I run something what does not fit in one graphic card?
* Distributing models across multiple GPUs is currently not supported. You will need a GPU that fits all of the model you are trying to use.
* I have YYY graphic card that does not supports CUDA - can I use it and how?
* No, our AI apps require GPUs with CUDA support to function
* What is the minimum VRAM size requirement for a GPU if I want to install multiple apps?
* When running multiple ExApps on the same GPU, it is currently required that the GPU can hold the largest model of the apps you install
* Is it possible to add more graphics cards for my instance to enable parallel requests or to speed up one request?
* Parallel processing of AI workloads of the same app with multiple GPUs is currently not supported
* Can I use in parallel CPU and GPU for AI processing?
* No, you can only process AI workloads for one app either on CPU or GPU. Between apps you can decide which app you want to run on CPU or GPU.

78
admin_manual/ai/app_assistant.rst Normal file
View File

@@ -0,0 +1,78 @@
===================
Nextcloud Assistant
===================
.. _ai-app-assistant:
Nextcloud assistant is the primary graphical user interface for interacting with artificial intelligence features in Nextcloud.
It offers the graphical user interface for text processing tasks like summarizing text, generating headlines, and asking arbitrary questions, for Speech-To-Text transcription of media files, for Text-To-Image picture generation and it integrates with the context_chat app to offer in-context answers about your own data stored in Nextcloud. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
Installation
------------
You can install the *assistant* app via the "Apps" page in Nextcloud, or by executing
.. code-block::
php occ app:enable assistant
App store
---------
You can also find the app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/assistant>`_
Repository
----------
You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/assistant>`_
Nextcloud customers should file bugs directly with our Customer Support.
Related apps
------------
Artificial intelligence at Nextcloud is built in a modular way, allowing you to choose from a variety of solutions for your needs. In order to make use of the various features of the Assistant you will need additional apps that act as backends to provide the actual implementation of the AI functionality. In the Nextcloud administration settings under "Artificial intelligence" you can select which AI backend app to use for which tasks. Note that some of the backend apps are only community maintained, while others are available for Customer support upon request.
**Note**: At Nextcloud we focus on creating on-premise AI apps that run fully self-hosted on your own servers in order to preserve your privacy and data sovereignty. However, you can also offload these resource-heavy tasks to an :ref:`"AI as a Service" provider<ai-ai_as_a_service>`.
Machine translation
~~~~~~~~~~~~~~~~~~~
In order to make use of machine translation features in the assistant, you will need an app that provides a translation backend:
* :ref:`translate<ai-app-translate>` - Runs open source AI translation models locally on your own server hardware (Customer support available upon request)
* *integration_deepl* - Integrates with the deepl API to provide translation functionality from Deepl.com servers (Only community supported)
* *integration_libretranslate* - Integrates with the open source LibreTranslate API to provide translation functionality hosted commercially or on your own hardware (Only community supported)
Speech-To-Text
~~~~~~~~~~~~~~
In order to make use of Speech-to-Text, you will need an app that provides a Speech-To-Text backend:
* :ref:`stt_whisper2<ai-app-stt_whisper2>` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Text processing
~~~~~~~~~~~~~~~
In order to make use of text processing features in the assistant, you will need an app that provides a Text processing backend:
* :ref:`llm2<ai-app-llm2>` - Runs open source AI language models locally on your own server hardware (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Text-To-Image
~~~~~~~~~~~~~
In order to make use of Text-To-Image features, you will need an app that provides an image generation backend:
* text2image_stablediffusion2 (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)
Context Chat
~~~~~~~~~~~~
In order to make use of our special Context Chat feature, offering in-context insights about your own data stored in Nextcloud, you will need the following apps:
* :ref:`context_chat + context_chat_backend<ai-app-context_chat>` - (Customer support available upon request)

73
admin_manual/ai/app_context_chat.rst Normal file
View File

@@ -0,0 +1,73 @@
=================
App: Context Chat
=================
.. _ai-app-context_chat:
Context Chat is an :ref:`assistant<ai-app-assistant>` feature that is implemented via an ensemble of two apps:
* the *context_chat* app, written purely in PHP
* the *context_chat_backend* ExternalApp written in Python
Together they provide the ContextChat text processing tasks accessible via the :ref:`Nextcloud Assistant app<ai-app-assistant>`.
The *context_chat* and *context_chat_backend* apps run only open source models and do so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
Requirements
------------
* Minimal Nextcloud version: 28
* The *context_chat_backend* app is built as an External App and thus depends on AppAPI v2.3.0
* Nextcloud AIO is supported
* Using GPU processing is currently required; we will soon make available a version that works on CPU as well
* We currently only support NVIDIA GPUs
* GPU Sizing
* You will need a GPU with at least 6GB VRAM
Space usage
~~~~~~~~~~~
This app employs a bundled Vector DB called `Chroma<https://github.com/chroma-core/chroma>`. All the users' textual data is duplicated, chunked and stored on disk in this vector DB along with semantic embedding vectors for the content.
Assuming no shared files between users you can calculate with roughly the amount of textual data in user files (e.g. for PDFs, only the text counts, no images are kept). Any shared files will be duplicated per user, however, so, assuming all files are shared with all users you need to calculate with "the amount of textual data in user files * number of users". The reality will lie between these two estimates, of course.
Installation
------------
0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
2. Install the *context_chat_backend* ExApp via the "External Apps" admin page in Nextcloud
3. Install the *context_chat* app via the "Apps" page in Nextcloud, or by executing
.. code-block::
occ app:enable context_chat
**Note**: Both apps need to be installed and both major version and minor version of the two apps must match for the functionality to work (ie. "v1.3.4" and "v1.3.1"; but not "v1.3.4" and "v2.1.6"; and not "v1.3.4" and "v1.4.5"). Keep this in mind when updating.
Scaling
-------
It is currently not possible to scale ExApps like Context Chat, we are working on this. Based on our calculations an instance has a rough capacity of 1000 user requests per hour. However, this number is based on theory and we do appreciate real-world feedback on this.
App store
---------
You can also find the *context_chat* app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/context_chat>`_
Repository
----------
You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/context_chat>`_ and `<https://github.com/nextcloud/context_chat_backend>`_
Nextcloud customers should file bugs directly with our Customer Support.
Known Limitations
-----------------
* We currently only support the English language
* Language models are likely to generate false information and should thus only be used in situations that are not critical. It's recommended to only use AI at the beginning of a creation process and not at the end, so that outputs of AI serve as a draft for example and not as final product. Always check the output of language models before using it.
* Make sure to test this app for whether it meets your use-case's quality requirements
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)

76
admin_manual/ai/app_llm2.rst Normal file
View File

@@ -0,0 +1,76 @@
======================================
App: Local large language model (llm2)
======================================
.. _ai-app-llm2:
The *llm2* app is one of the apps that provide text processing functionality using Large language models in Nextcloud and act as a text processing backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`, the *mail* app and :ref:`other apps making use of the core Translation API<tp-consumer-apps>`. The *llm2* app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
This app uses `ctransformers <https://github.com/marella/ctransformers>`_ under the hood and is thus compatible with any model in *gguf* format. Output quality will differ depending on which model you use, we recommend the following models:
* `Llama2 7b Chat <https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF>`_ (Slightly older; good quality; good acclaim)
* `NeuralBeagle14 7B <https://huggingface.co/mlabonne/NeuralBeagle14-7B-GGUF>`_ (Newer; good quality; less well known)
Requirements
------------
* This app is built as an External App and thus depends on AppAPI v2.3.0
* Nextcloud AIO is supported
* Using GPU processing is supported, but not required; be prepared for slow performance unless you are using GPU
* We currently only support NVIDIA GPUs
* GPU Sizing
* You will need a GPU with enough VRAM to hold the model you choose
* for 7B parameter models, 5bit-quantized variants and lower should fit on a 8GB VRAM, but of course have lower quality
* for 7B parameter models, 6bit-quantized variants and up will need 12GB VRAM
* If you want better reasoning capabilities, you will need to look for models with more parameters, like 14B and higher, which of course also need more VRAM
* CPU Sizing
* If you don't have a GPU, this app will utilize your CPU cores
* The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores
* The app will hog all cores by default, so it is usually better to run it on a separate machine
Installation
------------
0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
2. Install the "Local large language model" ExApp via the "External Apps" page in the Nextcloud web admin user interface
Supplying alternate models
~~~~~~~~~~~~~~~~~~~~~~~~~~
This app allows supplying alternate LLM models as *gguf* files in the ``/nc_app_llm2_data`` directory of the docker container.
1. Download a **gguf** model e.g. from huggingface
2. Copy the **gguf** file to ``/nc_app_llm2_data`` inside the docker container
3. Restart the llm2 ExApp
4. Select the new model in the Nextcloud AI admin settings
Scaling
-------
It is currently not possible to scale this app, we are working on this. Based on our calculations an instance has a rough capacity of 1000 user requests per hour. However, this number is based on theory and we do appreciate real-world feedback on this.
App store
---------
You can also find the app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/llm2>`_
Repository
----------
You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/llm2>`_
Nextcloud customers should file bugs directly with our Support system.
Known Limitations
-----------------
* We currently only support the English language
* Language models are likely to generate false information and should thus only be used in situations that are not critical. It's recommended to only use AI at the beginning of a creation process and not at the end, so that outputs of AI serve as a draft for example and not as final product. Always check the output of language models before using it.
* Make sure to test the language model you are using it for whether it meets the use-case's quality requirements
* Language models notoriously have a high energy consumption, if you want to reduce load on your server you can choose smaller models or quantized models in excahnge for lower accuracy
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)

80
admin_manual/ai/app_stt_whisper2.rst Normal file
View File

@@ -0,0 +1,80 @@
================================================
App: Local Whisper Speech-To-Text (stt_whisper2)
================================================
.. _ai-app-stt_whisper2:
The *stt_whisper2* app is one of the apps that provide Speech-To-Text functionality in Nextcloud and act as a media transcription backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`, the *talk* app and :ref:`other apps making use of the core Translation API<stt-consumer-apps>`. The *stt_whisper2* app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
This app uses `faster-whisper <https://github.com/SYSTRAN/faster-whisper>`_ under the hood. Output quality will differ depending on which model you use, we recommend the following models:
* OpenAI Whisper large-v2 or v3
* OpenAI Whisper medium.en
Requirements
------------
* Minimal Nextcloud version: 28
* This app is built as an External App and thus depends on AppAPI v2.3.0
* Nextcloud AIO is supported
* Using GPU processing is supported, but not required; be prepared for slow performance unless you are using GPU
* We currently only support NVIDIA GPUs
* GPU Sizing
* You will need a GPU with enough VRAM to hold the model you choose
* the small model should fit on 2GB VRAM
* the large-v2 (the best and largest) will need 6GB VRAM
* The distil-whisper variants have half the parameters of the original models while supposedly staying within 1% of the original error rate (your mileage may vary)
* CPU Sizing
* If you don't have a GPU, this app will utilize your CPU cores
* The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores
* The app will hog all cores by default, so it is usually better to run it on a separate machine
Installation
------------
0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
2. Install the *stt_whisper2* "Local Speech-To-Text" ExApp via the "External Apps" page in the Nextcloud web admin user interface
Supplying alternate models
~~~~~~~~~~~~~~~~~~~~~~~~~~
This app allows supplying alternate LLM models as *gguf* files in the ``/nc_app_llm2_data`` directory of the docker container. You can use any `*faster-whisper* model by Systran on hugging face <https://huggingface.co/Systran>`_ by simply
1. git cloning the respective repository
2. Copying the folder with the git repository to ``/nc_app_llm2_data`` inside the docker container.
3. Restarting the Whisper ExApp
4. Selecting the respective model in the Nextcloud AI admin settings
Scaling
-------
It is currently not possible to scale this app, we are working on this. Based on our calculations an instance has a rough capacity of 4h of transcription throughput per minute (measured with 8 CPU threads on an Intel(R) Xeon(R) Gold 6226R). It is unclear how close to real-world usage this number is, so we do appreciate real-world feedback on this.
App store
---------
You can also find this app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/stt_whisper2>`_
Repository
----------
You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/stt_whisper2>`_
Nextcloud customers should file bugs directly with our customer support.
Known Limitations
-----------------
* We currently do not support live transcription
* We currently only support languages supported by the underlying Whisper models
* The whisper models perform unevenly across languages, and may show lower accuracy on low-resource and/or low-discoverability languages or languages where there was less training data available. The models also exhibit disparate performance on different accents and dialects of particular languages, which may include higher word error rate across speakers of different genders, races, ages, or other demographic criteria.
* Language models are likely to generate false information and should thus only be used in situations that are not critical. It's recommended to only use AI at the beginning of a creation process and not at the end, so that outputs of AI serve as a draft for example and not as final product. Always check the output of language models before using it.
* Make sure to test the language model you are using it for whether it meets the use-case's quality requirements
* Language models notoriously have a high energy consumption, if you want to reduce load on your server you can choose smaller models or quantized models in excahnge for lower accuracy
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)

103
admin_manual/ai/app_translate.rst Normal file
View File

@@ -0,0 +1,103 @@
==========================================
App: Local Machine translation (translate)
==========================================
.. _ai-app-translate:
The *translate* app is one of the apps that provide machine translation functionality in Nextcloud and act as a translation backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`, the *text* app and :ref:`other apps making use of the core Translation API<mt-consumer-apps>`. The *translate* app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
The app currently supports the following languages:
* English (en)
* German (de)
* French (fr)
* Spanish (es)
* Chinese (zh)
As the models are entirely open source, the quality of translations may not be comparable to commercially available machine translation services.
Requirements
------------
* Minimal Nextcloud version: 26
* x86 CPU
* GNU lib C (musl is not supported)
* This app does not support using GPU for processing and may thus not be performing ideally for long texts
(*Note*: Nextcloud AIO is currently not supported due to it using musl)
Space usage
~~~~~~~~~~~
* ~1GB per language pair
* ~10GB in total
Installation
------------
0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
1. Install the *translate* app via the "Apps" page in Nextcloud, or by executing
occ app:enable translate
Setup
~~~~~
After installing this app you will need to run:
.. code-block::
occ translate:download-models
You may also download only specific languages by using the following command:
.. code-block::
occ translate:download-models <languages>
For example
.. code-block::
occ translate:download-models de en
will download both en->de and de->en.
.. code-block::
occ translate:download-models de en es
will download en->de, de->en, en->es, es->en, es->de, de->es
App store
---------
You can also find the app in our app store, where you can write a review: `<https://apps.nextcloud.com/apps/translate>`_
Repository
----------
You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: `<https://github.com/nextcloud/translate>`_
Nextcloud customers should file bugs directly with our Customer Support.
Ethical AI Rating
-----------------
Rating: 🟢
~~~~~~~~~~
Positive:
* the software for training and inference of this model is open source
* the trained model is freely available, and thus can be run on-premises
* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage.
Learn more about the Nextcloud Ethical AI Rating `in our blog<https://nextcloud.com/blog/nextcloud-ethical-ai-rating/>`.
Known Limitations
-----------------
* Language models are likely to generate false information and should thus only be used in situations that are not critical. It's recommended to only use AI at the beginning of a creation process and not at the end, so that outputs of AI serve as a draft for example and not as final product. Always check the output of language models before using it.
* Make sure to test the translation model you are using it for whether it meets the use-case's quality requirements
* Language models notoriously have a high energy consumption
* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying models. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI)

120
admin_manual/ai/index.rst
View File

@@ -2,114 +2,14 @@
Artificial Intelligence
=======================
We strive to bring Artificial Intelligence features to Nextcloud. This section highlights these features, how they work and where to find them.
All of these features are completely optional and need to be installed via separate Nextcloud Apps.
.. toctree::
:maxdepth: 2
Overview of AI features
-----------------------
.. csv-table::
:header: "Feature","App","Rating","Open source","Freely available model","Freely available training data","Privacy: Keeps data on premises"
"Smart inbox","`Mail <https://apps.nextcloud.com/apps/mail>`_","Green","Yes","Yes","Yes","Yes"
"Image object recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Image face recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Video action recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Audio music genre recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Suspicious login detection","`Suspicious Login <https://apps.nextcloud.com/apps/suspicious_login>`_","Green","Yes","Yes","Yes","Yes"
"Related resources","`Related Resources <https://apps.nextcloud.com/apps/related_resources>`_","Green","Yes","Yes","Yes","Yes"
"Recommended files","recommended_files","Green","Yes","Yes","Yes","Yes"
"Machine translation","`Translate <https://apps.nextcloud.com/apps/translate>`_","Green","Yes","Yes - Opus models by University Helsinki","Yes","Yes"
"","`LibreTranslate integration <https://apps.nextcloud.com/apps/integration_libretranslate>`_","Green","Yes","Yes - OpenNMT models","Yes","Yes"
"","`DeepL integration <https://apps.nextcloud.com/apps/integration_deepl>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Green","Yes","Yes","Yes","Yes"
"Speech-To-Text","`Whisper Speech-To-Text <https://apps.nextcloud.com/apps/stt_whisper>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","Yes"
"","`OpenAI and LocalAI integration <https://apps.nextcloud.com/apps/integration_openai>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No"
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No"
"Image generation","`Local Stable Diffusion <https://apps.nextcloud.com/apps/text2image_stablediffusion>`_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","Yes"
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","No"
"Text generation","`Local large language model (via GPT4all Falcon) <https://apps.nextcloud.com/apps/llm>`_","Green","Yes","Yes","Yes","Yes"
"","`Local large language model (via Llama 2) <https://apps.nextcloud.com/apps/llm>`_","Yellow","Yes","Yes","No","Yes"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Green","Yes","Yes","Yes","Yes"
"Context Chat","`Nextcloud Assistant Context Chat <https://apps.nextcloud.com/apps/context_chat>`_","Yellow","Yes","Yes","No","Yes"
"","`Nextcloud Assistant Context Chat (Backend) <https://apps.nextcloud.com/apps/context_chat_backend>`_","Yellow","Yes","Yes","No","Yes"
Ethical AI Rating
-----------------
Until Hub 3, we succeeded in offering features without relying on proprietary blobs or third party services. Yet, while there is a large community developing ethical, safe and privacy-respecting technologies, there are many other relevant technologies users might want to use. We want to provide users with these cutting-edge technologies – but also be transparent. For some use cases, ChatGPT might be a reasonable solution, while for more private, professional or sensitive data, it is paramount to have a local, on-prem, open solution. To differentiate these, we developed an Ethical AI Rating.
The rating has four levels:
* Red
* Orange
* Yellow
* Green
It is based on points from these factors:
* Is the software (both for inferencing and training) under a free and open source license?
* Is the trained model freely available for self-hosting?
* Is the training data available and free to use?
If all of these points are met, we give a Green label. If none are met, it is Red. If 1 condition is met, it is Orange and if 2 conditions are met, Yellow.
Features used by other apps
---------------------------
Some of our AI features are realized as generic APIs that any app can use and any app can provide an implementation for by registering a provider. So far, these are
Machine translation, Speech-To-Text and Text processing.
Machine translation
^^^^^^^^^^^^^^^^^^^
As you can see in the table above we have multiple apps offering machine translation capabilities. Each app brings its own set of supported languages.
In downstream apps like the Text app, users can use the translation functionality regardless of which app implements it behind the scenes.
Implementing apps
~~~~~~~~~~~~~~~~~
* *Text* for offering the translation menu
* `Analytics <https://apps.nextcloud.com/apps/analytics>`_ for translating graph labels
Speech-To-Text
^^^^^^^^^^^^^^
As you can see in the table above we have multiple apps offering Speech-To-Text capabilities. In downstream apps like the Talk app, users can use the transcription functionality regardless of which app implements it behind the scenes.
Implementing apps
~~~~~~~~~~~~~~~~~
* `Speech-to-Text Helper <https://apps.nextcloud.com/apps/stt_helper>`_ for providing a Speech-To-Text smart picker
* `Talk <https://apps.nextcloud.com/apps/spreed>`_ for transcribing calls (see `Nextcloud Talk docs <https://nextcloud-talk.readthedocs.io/en/latest/settings/#app-configuration>`_ for how to enable this)
Text processing
^^^^^^^^^^^^^^^
As you can see in the table above we have multiple apps offering Text processing capabilities. In downstream apps like the Nextcloud Assistant app, users can use the text processing functionality regardless of which app implements it behind the scenes.
Implementing apps
~~~~~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for various tasks
* `Mail <https://apps.nextcloud.com/apps/mail>`_ for summarizing mail threads (see :ref:`the Nextcloud Mail docs<mail_thread_summary>` for how to enable this)
* `SummarAI <https://apps.nextcloud.com/apps/summarai>`_ for summarizing chat histories in `Talk <https://apps.nextcloud.com/apps/spreed>`_
Image generation
^^^^^^^^^^^^^^^^
As you can see in the table above we have multiple apps offering Image generation capabilities. In downstream apps like the Text-to-Image helper app, users can use the image generation functionality regardless of which app implements it behind the scenes.
Implementing apps
~~~~~~~~~~~~~~~~~
* `Text-to-Image Helper <https://apps.nextcloud.com/apps/stt_helper>`_ for providing a Text-to-Image smart picker
Context Chat (Tech preview)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Our Context Chat feature was introduced in Nextcloud Hub 7 (v28). It allows asking questions to the assistant related to your documents in Nextcloud. You will need to install both the context_chat app as well as the context_chat_backend External App. Be prepared that things might break or be a little rough around the edges. We look forward to your feedback!
Implementing apps
~~~~~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for the context chat task
overview
app_assistant
app_translate
app_llm2
app_stt_whisper2
app_context_chat
app_api_and_external_apps
ai_as_a_service

166
admin_manual/ai/overview.rst Normal file
View File

@@ -0,0 +1,166 @@
========
Overview
========
We strive to bring Artificial Intelligence features to Nextcloud. This section highlights these features, how they work and where to find them.
All of these features are completely optional. If you want to have them on your server, you need install them via separate Nextcloud Apps.
Overview of AI features
-----------------------
Nextcloud uses modularity to separate raw AI functionality from the Graphical User interfaces and apps that make use of said functionality. Each instance can thus make use of various backends that provide the functionality for the same frontends and the same functionality can be implemented by multiple apps using on-premises processing or third-party AI service providers.
.. csv-table::
:header: "Feature","App","Rating","Open source","Freely available model","Freely available training data","Privacy: Keeps data on premises"
"Smart inbox","`Mail <https://apps.nextcloud.com/apps/mail>`_","Green","Yes","Yes","Yes","Yes"
"Image object recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Image face recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Video action recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Audio music genre recognition","`Recognize <https://apps.nextcloud.com/apps/recognize>`_","Green","Yes","Yes","Yes","Yes"
"Suspicious login detection","`Suspicious Login <https://apps.nextcloud.com/apps/suspicious_login>`_","Green","Yes","Yes","Yes","Yes"
"Related resources","`Related Resources <https://apps.nextcloud.com/apps/related_resources>`_","Green","Yes","Yes","Yes","Yes"
"Recommended files","recommended_files","Green","Yes","Yes","Yes","Yes"
"Machine translation","`Translate <https://apps.nextcloud.com/apps/translate>`_","Green","Yes","Yes - Opus models by University Helsinki","Yes","Yes"
"","`LibreTranslate integration <https://apps.nextcloud.com/apps/integration_libretranslate>`_","Green","Yes","Yes - OpenNMT models","Yes","Yes"
"","`DeepL integration <https://apps.nextcloud.com/apps/integration_deepl>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Green","Yes","Yes","Yes","Yes"
"","`Local Whisper Speech-To-Text 2 (ExApp) <https://apps.nextcloud.com/apps/stt_whisper2>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","Yes"
"","`OpenAI and LocalAI integration <https://apps.nextcloud.com/apps/integration_openai>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No"
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No"
"Image generation","`Local Stable Diffusion <https://apps.nextcloud.com/apps/text2image_stablediffusion>`_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","Yes"
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","No"
"","`Local large language model 2 (ExApp) <https://apps.nextcloud.com/apps/llm2>`_","Yellow","Yes","Yes","No","Yes"
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Green","Yes","Yes","Yes","Yes"
"Context Chat","`Nextcloud Assistant Context Chat <https://apps.nextcloud.com/apps/context_chat>`_","Yellow","Yes","Yes","No","Yes"
"","`Nextcloud Assistant Context Chat (Backend) <https://apps.nextcloud.com/apps/context_chat_backend>`_","Yellow","Yes","Yes","No","Yes"
Ethical AI Rating
-----------------
Until Hub 3, we succeeded in offering features without relying on proprietary blobs or third party services. Yet, while there is a large community developing ethical, safe and privacy-respecting technologies, there are many other relevant technologies users might want to use. We want to provide users with these cutting-edge technologies – but also be transparent. For some use cases, ChatGPT might be a reasonable solution, while for more private, professional or sensitive data, it is paramount to have a local, on-prem, open solution. To differentiate these, we developed an Ethical AI Rating.
The rating has four levels:
* Red
* Orange
* Yellow
* Green
It is based on points from these factors:
* Is the software (both for inferencing and training) under a free and open source license?
* Is the trained model freely available for self-hosting?
* Is the training data available and free to use?
If all of these points are met, we give a Green label. If none are met, it is Red. If 1 condition is met, it is Orange and if 2 conditions are met, Yellow.
Features used by other apps
---------------------------
Some of our AI features are realized as generic APIs that any app can use and any app can provide an implementation for by registering a provider. So far, these are
Machine translation, Speech-To-Text, Image generation, Text processing and Context Chat.
Machine translation
^^^^^^^^^^^^^^^^^^^
.. _mt-consumer-apps:
As you can see in the table above we have multiple apps offering machine translation capabilities. Each app brings its own set of supported languages.
In downstream apps like the Text app, users can use the translation functionality regardless of which app implements it behind the scenes.
Frontend apps
~~~~~~~~~~~~~
* *Text* for offering the translation menu
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ offering a graphical translation UI
* `Analytics <https://apps.nextcloud.com/apps/analytics>`_ for translating graph labels
Backend apps
~~~~~~~~~~~~
* :ref:`translate<ai-app-translate>` - Runs open source AI translation models locally on your own server hardware (Customer support available upon request)
* *integration_deepl* - Integrates with the deepl API to provide translation functionality from Deepl.com servers (Only community supported)
* *integration_libretranslate* - Integrates with the open source LibreTranslate API to provide translation functionality hosted commercially or on your own hardware (Only community supported)
Speech-To-Text
^^^^^^^^^^^^^^
.. _stt-consumer-apps:
As you can see in the table above we have multiple apps offering Speech-To-Text capabilities. In downstream apps like the Talk app, users can use the transcription functionality regardless of which app implements it behind the scenes.
Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ offering a graphical translation UI and a smart picker
* `Speech-to-Text Helper <https://apps.nextcloud.com/apps/stt_helper>`_ for providing a Speech-To-Text smart picker (deprecated; was merged into assistant)
* `Talk <https://apps.nextcloud.com/apps/spreed>`_ for transcribing calls (see `Nextcloud Talk docs <https://nextcloud-talk.readthedocs.io/en/latest/settings/#app-configuration>`_ for how to enable this)
Backend apps
~~~~~~~~~~~~
* :ref:`stt_whisper2<ai-app-stt_whisper2>` - Runs open source AI Speech-To-Text models on your own server hardware (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Text processing
^^^^^^^^^^^^^^^
.. _tp-consumer-apps:
As you can see in the table above we have multiple apps offering Text processing capabilities. In downstream apps like the Nextcloud Assistant app, users can use the text processing functionality regardless of which app implements it behind the scenes.
Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI for the various tasks and a smart picker
* `GPTFreePrompt <https://apps.nextcloud.com/apps/gptfreeprompt>`_ for providing an llm smart picker (deprecated; was merged into assistant)
* `Mail <https://apps.nextcloud.com/apps/mail>`_ for summarizing mail threads (see :ref:`the Nextcloud Mail docs<mail_thread_summary>` for how to enable this)
* `SummarAI <https://apps.nextcloud.com/apps/summarai>`_ for summarizing chat histories in `Talk <https://apps.nextcloud.com/apps/spreed>`_
Backend apps
~~~~~~~~~~~~
* :ref:`llm2<ai-app-llm2>` - Runs open source AI language models locally on your own server hardware (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
Image generation
^^^^^^^^^^^^^^^^
.. _t2i-consumer-apps:
As you can see in the table above we have multiple apps offering Image generation capabilities. In downstream apps like the Text-to-Image helper app, users can use the image generation functionality regardless of which app implements it behind the scenes.
Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI and a smart picker
* `Text-to-Image Helper <https://apps.nextcloud.com/apps/stt_helper>`_ for providing a Text-to-Image smart picker (deprecated; was merged into assistant)
Backend apps
~~~~~~~~~~~~
* text2image_stablediffusion2 (Customer support available upon request)
* *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)
Context Chat (Tech preview)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Our Context Chat feature was introduced in Nextcloud Hub 7 (v28). It allows asking questions to the assistant related to your documents in Nextcloud. You will need to install both the context_chat app as well as the context_chat_backend External App. Be prepared that things might break or be a little rough around the edges. We look forward to your feedback!
Frontend apps
~~~~~~~~~~~~~
* `Assistant <https://apps.nextcloud.com/apps/assistant>`_ for offering a graphical UI for the context chat tasks
Backend apps
~~~~~~~~~~~~
* :ref:`context_chat + context_chat_backend<ai-app-context_chat>` - (Customer support available upon request)