diff --git a/admin_manual/configuration_server/config_sample_php_parameters.rst b/admin_manual/configuration_server/config_sample_php_parameters.rst index 3edbcd699..da5b61cb4 100644 --- a/admin_manual/configuration_server/config_sample_php_parameters.rst +++ b/admin_manual/configuration_server/config_sample_php_parameters.rst @@ -609,7 +609,7 @@ more outdated. Tokens are still checked every 5 minutes for validity max value: 300 -Defaults to ``300`` +Defaults to ``60`` auth.bruteforce.protection.enabled ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/developer_manual/client_apis/OCS/index.rst b/developer_manual/client_apis/OCS/index.rst index db65a3fe4..f74cb84f5 100644 --- a/developer_manual/client_apis/OCS/index.rst +++ b/developer_manual/client_apis/OCS/index.rst @@ -20,4 +20,5 @@ The old documentation is still kept as it provides some additional documentation ocs-translation-api ocs-textprocessing-api ocs-text2image-api + ocs-taskprocessing-api ocs-out-of-office-api diff --git a/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst b/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst new file mode 100644 index 000000000..8298f80b6 --- /dev/null +++ b/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst @@ -0,0 +1,256 @@ +.. _ocs-taskprocessing-api: + +====================== +OCS TaskProcessing API +====================== + +.. versionadded:: 30.0.0 + +The OCS Text processing API allows you to run text processing tasks, like prompting large language models implemented by apps using :ref:`the backend Text Processing API`. + +The base URL for all calls to this API is: ``/ocs/v2.php/taskprocessing/`` + +All calls to OCS endpoints require the ``OCS-APIRequest`` header to be set to ``true``. + + +Get available task types +------------------------ + +.. versionadded:: 30.0.0 + +* Method: ``GET`` +* Endpoint: ``/tasktypes`` +* Response: + - Status code: + + ``200 OK`` + - Data: + ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +| field | type | Description | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``types`` | array | A map of supported task types. The keys are the task type IDs See below for the values. | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ + +Task type fields: + ++-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +| field | type | Description | ++-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``name`` | string | The name of the task type in the user's language | ++-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``description`` | string | A description of the task type in the user's language | ++-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``inputShape`` | array | The input shape of this task type | ++-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``outputShape`` | array | The output shape of this task type | ++-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ + +Input and output shape fields are maps from slot key to slot metadata: + ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +| field | type | Description | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``name`` | string | The name of the I/O slot | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``description`` | string | A description of the I/O slot | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``type`` | int | The I/O slot type (See backend API for the available types) | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ +|``mandatory`` | bool | Whether this slot is mandatory or not | ++----------------------+--------+---------------------------------------------------------------------------------------------------------------+ + +Schedule a task +--------------- + +.. versionadded:: 30.0.0 + +.. note:: The endpoint is rate limited as it can be quite resource intensive. Users can make 20 requests in 2 minutes, guests only 5 + +* Method: ``POST`` +* Endpoint: ``/schedule`` +* Data: + ++-----------------+-------------+--------------------------------------------------------------------------------+ +| field | type | Description | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``input`` | array | The input text for the task | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``type`` | string | Id of this task's type. | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``appId`` | string | The id of the requesting app | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``customId`` | string | An custom app-defined identifier for the task (optional) | ++-----------------+-------------+--------------------------------------------------------------------------------+ + +* Response: + - Status code: + + ``200 OK`` + + ``400 Bad Request`` - When the task type is invalid + + ``412 Precondition Failed`` - When the task type is not available currently + + ``401 Unauthenticated`` - When the input references a file that the user doesn't have access to + + ``429 Too Many Requests`` - When the rate limiting was exceeded + + - Data: + + ``input`` - Only provided in case of ``200 OK``, the task input, array + + ``type`` - Only provided in case of ``200 OK``, the task type, string + + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int + + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API + + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string + + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string + + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string + + ``output`` - Only provided in case of ``200 OK``, null + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed + +Fetch a task by ID +------------------ + +.. versionadded:: 30.0.0 + +.. note:: The endpoint is rate limited as it can be quite resource intensive. Users can make 20 requests in 2 minutes, guests only 5 + +* Method: ``POST`` +* Endpoint: ``/task/{id}`` + +* Response: + - Status code: + + ``200 OK`` + + ``404 Not Found`` - When the task could not be found + + - Data: + + ``input`` - Only provided in case of ``200 OK``, the task input, array + + ``type`` - Only provided in case of ``200 OK``, the task type, string + + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int + + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API + + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string + + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string + + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string + + ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed + + +Cancel a task +------------- + +.. versionadded:: 30.0.0 + +* Method: ``POST`` +* Endpoint: ``/task/{id}/cancel`` + +* Response: + - Status code: + + ``200 OK`` + + ``404 Not Found`` - When the task could not be found + + - Data: + - Data: + + ``input`` - Only provided in case of ``200 OK``, the task input, array + + ``type`` - Only provided in case of ``200 OK``, the task type, string + + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int + + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API + + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string + + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string + + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string + + ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed + + +Delete a task +------------- + +.. versionadded:: 30.0.0 + +* Method: ``DELETE`` +* Endpoint: ``/task/{id}`` + +* Response: + - Status code: + + ``200 OK`` + + ``404 Not Found`` - When the task could not be found + + - Data: + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed + + +Get task file contents +---------------------- + +.. versionadded:: 30.0.0 + +* Method: ``GET`` +* Endpoint: ``/task/{id}/file/{fileId}`` + +* Response: + - Status code: + + ``200 OK`` + + ``404 Not Found`` - When the task could not be found + + - Data: + + If ``200 OK`` this endpoint returns the raw data of the file + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed + +Set task progress +----------------- + +.. versionadded:: 30.0.0 + +* Method: ``GET`` +* Endpoint: ``/task/{id}/progress`` +* Data: + ++-----------------+-------------+--------------------------------------------------------------------------------+ +| field | type | Description | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``progress`` | float | A number between 0-1 indicating the task progress | ++-----------------+-------------+--------------------------------------------------------------------------------+ + + +* Response: + - Status code: + + ``200 OK`` + + ``404 Not Found`` - When the task could not be found + + - Data: + - Data: + + ``input`` - Only provided in case of ``200 OK``, the task input, array + + ``type`` - Only provided in case of ``200 OK``, the task type, string + + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int + + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API + + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string + + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string + + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string + + ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed + +Set task result +--------------- + +.. versionadded:: 30.0.0 + +* Method: ``POST`` +* Endpoint: ``/task/{id}/result`` +* Data: + ++-----------------+-------------+--------------------------------------------------------------------------------+ +| field | type | Description | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``output`` | array | The task output if the task was successful (optional) | ++-----------------+-------------+--------------------------------------------------------------------------------+ +|``errorMessage`` | string | The error message if the task failed (optional) | ++-----------------+-------------+--------------------------------------------------------------------------------+ + +* Response: + - Status code: + + ``200 OK`` + + ``404 Not Found`` - When the task could not be found + + - Data: + - Data: + + ``input`` - Only provided in case of ``200 OK``, the task input, array + + ``type`` - Only provided in case of ``200 OK``, the task type, string + + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int + + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API + + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string + + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string + + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string + + ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null + + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed diff --git a/developer_manual/digging_deeper/index.rst b/developer_manual/digging_deeper/index.rst index 51b969f68..374de7c19 100644 --- a/developer_manual/digging_deeper/index.rst +++ b/developer_manual/digging_deeper/index.rst @@ -29,6 +29,7 @@ Digging deeper talk translation text_processing + task_processing text2image two-factor-provider users diff --git a/developer_manual/digging_deeper/task_processing.rst b/developer_manual/digging_deeper/task_processing.rst new file mode 100644 index 000000000..af6b2ee60 --- /dev/null +++ b/developer_manual/digging_deeper/task_processing.rst @@ -0,0 +1,346 @@ +.. _task_processing: + +=============== +Task Processing +=============== + +.. versionadded:: 30.0.0 + +Nextcloud offers a **Task Processing** API which replaces the previously introduced :ref:`Text Processing`, :ref:`TextToImage` and :ref:`Speech-To-Text` APIs. The overall idea is that there is a central OCP API that apps can use to schedule all kinds of tasks (mainly inteded for AI tasks). To be technology agnostic any other app can provide this task functionality by registering Task Processing providers for specific Task types. + +Consuming the Task Processing API +--------------------------------- + +To consume the Task Processing API, you will need to :ref:`inject` ``\OCP\TaskProcessing\IManager``. This manager offers the following methods: + + * ``hasProviders()`` This method returns a boolean which indicates if any providers have been registered. If this is false you cannot use the TextProcessing feature. + * ``getAvailableTaskTypes()`` This method returns an array of task types indexed by their ID with their names and additional metadata. + * ``scheduleTask(Task $task)`` This method provides the actual scheduling functionality. The task is defined using the Task class. This method runs the task asynchronously in a background job. + * ``getTask(int $id)`` This method fetches a task specified by its id. + * ``deleteTask(Task $task)`` This method deletes a task + * ``cancelTask(int $id)`` This method cancels a task specified by its id. + +If you would like to use the text processing functionality in a client, there are also OCS endpoints available for this: :ref:`OCS Text Processing API` + +Tasks types +^^^^^^^^^^^ +The following built-in task types are available: + + * ``'core:text2text'``: This task allows passing an arbitrary prompt to the language model. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToText`` + * Input shape: + * ``input``: ``Text`` + * Output shape: + * ``output``: ``Text`` + * ``'core:text2text:headline'``: This task will generate a headline for the passed input text. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextHeadline`` + * Input shape: + * ``input``: ``Text`` + * Output shape: + * ``output``: ``Text`` + * ``'core:text2text:topics'``: This task will generate a comma-separated list of topics for the passed input text. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextTopics`` + * Input shape: + * ``input``: ``Text`` + * Output shape: + * ``output``: ``Text`` + * ``'core:text2text:summary'``: This task will summarize the passed input text. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextSummary`` + * Input shape: + * ``input``: ``Text`` + * Output shape: + * ``output``: ``Text`` + * ``'core:audio2text'``: This task type is for transcribing audio to text. It is implemented by ``\OCP\TaskProcessing\TaskTypes\AudioToText`` + * Input shape: + * ``input``: ``Audio`` + * Output shape: + * ``output``: ``Text`` + * ``'core:text2image'``: This task type is for generating images from text prompts. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToImage`` + * Input shape: + * ``input``: ``Text`` + * ``numberOfImages``: ``Number`` + * Output shape: + * ``output``: ``ListOfImages`` + +Input and output shapes +~~~~~~~~~~~~~~~~~~~~~~~ + +Each task type defines how its input and output should look. This is called the input and output shape. + +For example the TextToImage type defines its input shape as follows: + +.. code-block:: php + + /** + * @return ShapeDescriptor[] + * @since 30.0.0 + */ + public function getInputShape(): array { + return [ + 'input' => new ShapeDescriptor( + $this->l->t('Prompt'), + $this->l->t('Describe the image you want to generate'), + EShapeType::Text + ), + 'numberOfImages' => new ShapeDescriptor( + $this->l->t('Number of images'), + $this->l->t('How many images to generate'), + EShapeType::Number + ), + ]; + } + +The task input and output are always represented by an associative array. In this case, the task input for TextToImage must have an array key named ``'input'`` which must contain a text and an array key named ``'numberOfImages'`` which must contain a number. + +If you want to simply use a task type, you can look up it's input and output shapes above or, if it is not built-in, in the documentation or implementation of the app introducing the task type. If you would like to use task types dynamically without knowing their shapes in advance, you can get their shape information from the ``IManager#getAvailableTaskTypes()`` menthod. The ShapeDescriptor class allows accessing the type data as well as human readable name and description using the ``getName()``, ``getDescription()`` and ``getShapeType()`` methods. + +Shape types +~~~~~~~~~~~ + +Input and output shape keys can have one of a pre-defined set of types, which are enumerated in the ``\OCP\TaskProcessing\EShapeType`` Enum: + +.. code-block:: php + + enum EShapeType: int { + case Number = 0; + case Text = 1; + case Image = 2; + case Audio = 3; + case Video = 4; + case File = 5; + case ListOfNumbers = 10; + case ListOfTexts = 11; + case ListOfImages = 12; + case ListOfAudio = 13; + case ListOfVideo = 14; + case ListOfFiles = 15; + } + +When consuming the task processing API, ``Image``, ``Audio``, ``Video`` and ``File`` slots are filled with Nextcloud file IDs, so instead of supplying the image data directly as a string to the task you create a file for it and pass the id. Similarly, if the task outputs an image, you will receive a file ID in that slot. + +Tasks +^^^^^ +To create a task we use the ``\OCP\TaskProcessing\Task`` class. Its constructor takes the following arguments: ``new \OCP\TaskProcessing\Task(string $taskTypeId, array $input, string $appId, ?string $userId, string $customId = '')``. For example: + +.. code-block:: php + + if (isset($textprocessingManager->getAvailableTaskTypes()[TextToTextSummary::ID]) { + $summaryTask = new Task(TextToTextSummary::ID, $emailText, "my_app", $userId, (string) $emailId); + } else { + // cannot use summarization + } + +The task class objects have the following methods available: + + * ``getTaskTypeId()`` This returns the task type. + * ``getStatus()`` This method returns one of the below statuses. + * ``getId()`` This method will return ``null`` before the task has been passed to ``scheduleTask`` otherwise it will return the unique ID of the task. + * ``getInput()`` This returns the input array. + * ``getOutput()`` This method will return ``null`` unless the task was successfully run, in that case it will return the output array + * ``getAppId()`` This returns the originating application ID of the task. + * ``getCustomId()`` This returns the original scheduler-defined identifier for the task + * ``getUserId()`` This returns the originating user ID of the task. + +You could now schedule the task as follows: + +.. code-block:: php + + try { + $taskprocessingManager->scheduleTask($summaryTask); + } catch (OCP\TaskProcessing\Exception\Exception|OCP\TaskProcessing\Exception\PreConditionNotMetException|OCP\TaskProcessing\Exception\UnauthorizedException|OCP\TaskProcessing\Exception\ValidationException $e) { + // scheduling task failed + } + +Task statuses +^^^^^^^^^^^^^ + +All tasks always have one of the below statuses: + +.. code-block:: php + + Task::STATUS_CANCELLED = 5; + Task::STATUS_FAILED = 4; + Task::STATUS_SUCCESSFUL = 3; + Task::STATUS_RUNNING = 2; + Task::STATUS_SCHEDULED = 1; + Task::STATUS_UNKNOWN = 0; + + +Listening to the task processing events +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Since ``scheduleTask`` does not block, you will need to listen to the following events in your app to obtain the output or be notified of any failure. + + * ``OCP\TaskProcessing\Events\TaskSuccessfulEvent`` This event class offers the ``getTask()`` method which returns the up-to-date task object, with the task output. + * ``OCP\TaskProcessing\Events\TaskFailedEvent`` In addition to the ``getTask()`` method, this event class provides the ``getErrorMessage()`` method which returns the error message as a string (only in English and for debugging purposes, so don't show this to the user) + + +For example, in your ``lib/AppInfo/Application.php`` file: + +.. code-block:: php + + $context->registerEventListener(OCP\TaskProcessing\Events\TaskSuccessfulEvent::class, MyPromptResultListener::class); + $context->registerEventListener(OCP\TaskProcessing\Events\TaskFailedEvent::class, MyPromptResultListener::class); + +The corresponding ``MyPromptResultListener`` class can look like: + +.. code-block:: php + + getTask()->getAppId() !== Application::APP_ID) { + return; + } + + if ($event instanceof TaskSuccessfulEvent) { + $output = $event->getTask()->getOutput() + // store $output somewhere + } + + if ($event instanceof TaskFailedEvent) { + $error = $event->getErrorMessage() + $userId = $event->getTask()->getUserId() + // Notify relevant user about failure + } + } + } + + +Implementing a TaskProcessing provider +-------------------------------------- + +A **Task processing provider** will usually be a class that implements the interface ``OCP\TaskProcessing\ISynchrounousProvider``. + +.. code-block:: php + + l->t('My awesome summary provider'); + } + + public function getTaskTypeId(): string { + return TextToTextSummary::ID; + } + + public function process(?string $userId, array $input, callable $reportProgress): array { + // Return the output here + } + + public function getExpectedRuntime() { + // usually takes 1min on average + return 60; + } + } + +The method ``getName`` returns a string to identify the registered provider in the user interface. + +The method ``process`` implements the text processing step. In case execution fails for some reason, you should throw a ``\OCP\TaskProcessing\Exception\ProcessingException`` with an explanatory error message. Important to note here is that ``Image``, ``Audio``, ``Video`` and ``File`` slots in the input array will be filled with ``\OCP\Files\File`` objects for your convenience. When outputting one of these you should simply return a string, the API will turn the data into a proper file for convenience. The ``$reportProgress`` parameter is a callback that you may use at will to report the task progress as a single float value between 0 and 1. Its return value will indicate if the task is still running (``true``) or if it was cancelled (``false``) and processing should be terminated. + +This class would typically be saved into a file in ``lib/TextProcessing`` of your app but you are free to put it elsewhere as long as it's loadable by Nextcloud's :ref:`dependency injection container`. + +Providing more task types +^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to implement providers that handle additional task types, you can create your own Task type classes implementing the ``OCP\TaskProcessing\ITaskType`` interface: + +.. code-block:: php + + new ShapeDescriptor('Audio', 'The audio', EShapeType::Audio), + ]; + } + + public function getOutputShape(): array { + return [ + 'spectrogram' => new ShapeDescriptor('Spectrogram', 'The audio spectrogram', EShapeType::Image), + ]; + } + } + +Provider and task type registration +----------------------------------- + +Providers and task types are registered via the :ref:`bootstrap mechanism` of the ``Application`` class. + +.. code-block:: php + :emphasize-lines: 17,18 + + registerTaskProcessingProvider(Provider::class); + $context->registerTaskProcessingTaskType(AudioToImage::class); + } + + public function boot(IBootContext $context): void {} + + }