From 7d2ba0d653bef576414aabd2e65cc983f51e9af5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marius=20Blu=CC=88m?= Date: Wed, 4 Jan 2017 15:40:00 +0100 Subject: [PATCH] Enterprise-Edition Fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Oracle DB is only available with Nextcloud Enterprise Support currently, not with an non-existing Enterprise Edition * S3 is rewritten now (…and does not require an non-existing Enterprise Edition) Signed-off-by: Marius Blüm --- .../deployment_recommendations.rst | 304 +++++++++--------- 1 file changed, 152 insertions(+), 152 deletions(-) diff --git a/admin_manual/installation/deployment_recommendations.rst b/admin_manual/installation/deployment_recommendations.rst index 90078e1bc..568fbba91 100644 --- a/admin_manual/installation/deployment_recommendations.rst +++ b/admin_manual/installation/deployment_recommendations.rst @@ -2,23 +2,23 @@ Nextcloud Deployment Recommendations ==================================== -What is the best way to install and maintain Nextcloud? The answer to that is -*"it depends"* because every Nextcloud customer has their own -particular needs and IT infrastructure. Nextcloud and the LAMP stack are -highly-configurable, so we will present three typical scenarios and make +What is the best way to install and maintain Nextcloud? The answer to that is +*"it depends"* because every Nextcloud customer has their own +particular needs and IT infrastructure. Nextcloud and the LAMP stack are +highly-configurable, so we will present three typical scenarios and make best-practice recommendations for both software and hardware. General Recommendations ----------------------- -.. note:: Whatever the size of your organization, always keep one thing in mind: +.. note:: Whatever the size of your organization, always keep one thing in mind: the amount of data stored in Nextcloud will only grow. Plan ahead. -Consider setting up a scale-out deployment, or using Federated Cloud Sharing to +Consider setting up a scale-out deployment, or using Federated Cloud Sharing to keep individual Nextcloud instances to a manageable size. -.. comment: Federating instances seems the best way to grow organically in - an enterprise. A lookup server to tie all the instances together under a +.. comment: Federating instances seems the best way to grow organically in + an enterprise. A lookup server to tie all the instances together under a single domain is being worked on. * Operating system: Linux. @@ -32,8 +32,8 @@ keep individual Nextcloud instances to a manageable size. high load and limited RAM environments. For the best results we recommend working with the Nextcloud GmbH enterprise suppor team for large deployments. -.. comment: mod_php is easier to set up, php-fpm with apache event MPM seems to - scale better under load and limited RAM restrictions: +.. comment: mod_php is easier to set up, php-fpm with apache event MPM seems to + scale better under load and limited RAM restrictions: http://blog.bitnami.com/2014/06/performance-enhacements-for-apache-and.html Small Workgroups or Departments @@ -46,14 +46,14 @@ Small Workgroups or Departments 100 GB to 10TB. * High availability level - Zero-downtime backups via Btrfs snapshots, component failure leads to - interruption of service. Alternate backup scheme on other filesystems: + Zero-downtime backups via Btrfs snapshots, component failure leads to + interruption of service. Alternate backup scheme on other filesystems: nightly backups with service interruption. - + Recommended System Requirements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -One machine running the application server, Web server, database server and +One machine running the application server, Web server, database server and local storage. Authentication via an existing LDAP or Active Directory server. @@ -65,55 +65,55 @@ Authentication via an existing LDAP or Active Directory server. One server with at least 2 CPU cores, 16GB RAM, local storage as needed. * Operating system - Enterprise-grade Linux distribution with full support from OS vendor. We + Enterprise-grade Linux distribution with full support from OS vendor. We recommend Red Hat Enterprise Linux or SUSE Linux Enterprise Server 12. * SSL Configuration - The SSL termination is done in Apache. A standard SSL certificate is + The SSL termination is done in Apache. A standard SSL certificate is needed, installed according to the Apache documentation. * Load Balancer - None. + None. * Database - MySQL, MariaDB or PostgreSQL. We currently recommend MySQL / MariaDB, as our - customers have had good experiences when moving to a Galera cluster to + MySQL, MariaDB or PostgreSQL. We currently recommend MySQL / MariaDB, as our + customers have had good experiences when moving to a Galera cluster to scale the DB. * Backup - Install Nextcloud, Nextcloud data directory and database on Btrfs filesystem. - Make regular snapshots at desired intervals for zero downtime backups. + Install Nextcloud, Nextcloud data directory and database on Btrfs filesystem. + Make regular snapshots at desired intervals for zero downtime backups. Mount DB partitions with the "nodatacow" option to prevent fragmentation. - + Alternatively, make nightly backups with service interruption: - + * Shut down Apache. * Create database dump. * Push data directory to backup. * Push database dump to backup. * Start Apache. - - Then optionally rsync to a backup storage or tape backup. (See the - `Maintenance`_ section of the Administration manual for tips on backups + + Then optionally rsync to a backup storage or tape backup. (See the + `Maintenance`_ section of the Administration manual for tips on backups and restores.) * Authentication User authentication via one or several LDAP or Active Directory servers. (See - `User Authentication with LDAP`_ for information on configuring Nextcloud to + `User Authentication with LDAP`_ for information on configuring Nextcloud to use LDAP and AD.) * Session Management - Local session management on the application server. PHP sessions are stored - in a tmpfs mounted at the operating system-specific session storage - location. You can find out where that is by running ``grep -R - 'session.save_path' /etc/php5`` and then add it to the ``/etc/fstab`` file, - for example: - ``echo "tmpfs /var/lib/php5/pool-www tmpfs defaults,noatime,mode=1777 0 0" + Local session management on the application server. PHP sessions are stored + in a tmpfs mounted at the operating system-specific session storage + location. You can find out where that is by running ``grep -R + 'session.save_path' /etc/php5`` and then add it to the ``/etc/fstab`` file, + for example: + ``echo "tmpfs /var/lib/php5/pool-www tmpfs defaults,noatime,mode=1777 0 0" >> /etc/fstab``. * Memory Caching - A memcache speeds up server performance, and Nextcloud supports four - memcaches; refer to `Configuring Memory Caching`_ for information on + A memcache speeds up server performance, and Nextcloud supports four + memcaches; refer to `Configuring Memory Caching`_ for information on selecting and configuring a memcache. * Storage @@ -124,12 +124,12 @@ Mid-sized Enterprises * Number of users 150 to 1,000 users. - + * Storage size Up to 200TB. - + * High availability level - Every component is fully redundant and can fail without service interruption. + Every component is fully redundant and can fail without service interruption. Backups without service interruption Recommended System Requirements @@ -153,42 +153,42 @@ Authentication via an existing LDAP or Active Directory server. * NFS storage server as needed. * Operating system - Enterprise grade Linux distribution with full support from OS vendor. Red + Enterprise grade Linux distribution with full support from OS vendor. Red Hat Enterprise Linux or SUSE Linux Enterprise Server 12 are recommended. * SSL Configuration - The SSL termination is done in the HAProxy load balancer. A standard SSL + The SSL termination is done in the HAProxy load balancer. A standard SSL certificate is needed, installed according to the `HAProxy documentation`_. * Load Balancer - HAProxy running on a dedicated server in front of the application servers. - Sticky session needs to be used because of local session management on the - application servers. + HAProxy running on a dedicated server in front of the application servers. + Sticky session needs to be used because of local session management on the + application servers. -.. comment: (please add configuration details here) -.. comment: why sticky sessions? the nice thing about haproxy is that it can - send requests to the application server with the least load. redis or - memcached seem more appropriate. this is mid size already. the software +.. comment: (please add configuration details here) +.. comment: why sticky sessions? the nice thing about haproxy is that it can + send requests to the application server with the least load. redis or + memcached seem more appropriate. this is mid size already. the software stack should be the same as for L`_ - Frank: Yes. But this only works if haproxy can read the http stream which - means that we have to terminate SSL in the haproxy instead of the Web server. + Frank: Yes. But this only works if haproxy can read the http stream which + means that we have to terminate SSL in the haproxy instead of the Web server. Totally possible. Whatever you prefer :-) - Jörn: AFAIK you need to do SSL offloading to do sticky sessions, because the - load balancer has to look into the http stream or rely on the client IP to - determine the Web server for the session. Not doing SSL offloading instead - requires you to use a shared session (via memcached or redis) because the - requests are distributed via round robin or least load. It allows you to - scale out the ssl load by adding more applicaton servers. So ... I think it + Jörn: AFAIK you need to do SSL offloading to do sticky sessions, because the + load balancer has to look into the http stream or rely on the client IP to + determine the Web server for the session. Not doing SSL offloading instead + requires you to use a shared session (via memcached or redis) because the + requests are distributed via round robin or least load. It allows you to + scale out the ssl load by adding more applicaton servers. So ... I think it is exactly the other way round. * Database MySQL/MariaDB Galera cluster with master-master replication. * Backup - Minimum daily backup without downtime. All MySQL/MariaDB statements should + Minimum daily backup without downtime. All MySQL/MariaDB statements should be replicated to a backup MySQL/MariaDB slave instance. - - * Create a snapshot on the NFS storage server. + + * Create a snapshot on the NFS storage server. * At the same time stop the MySQL replication. * Create a MySQL dump of the backup slave. * Push the NFS snapshot to the backup. @@ -197,30 +197,30 @@ Authentication via an existing LDAP or Active Directory server. * Restart MySQL replication. * Authentication - User authentication via one or several LDAP or Active Directory servers. - (See `User Authentication with LDAP`_ for information on configuring + User authentication via one or several LDAP or Active Directory servers. + (See `User Authentication with LDAP`_ for information on configuring Nextcloud to use LDAP and AD.) - -* LDAP - Read-only slaves should be deployed on every application server for + +* LDAP + Read-only slaves should be deployed on every application server for optimal scalability * Session Management - Session management on the application server. PHP sessions are stored - in a tmpfs mounted at the operating system-specific session storage - location. You can find out where that is by running ``grep -R - 'session.save_path' /etc/php5`` and then add it to the ``/etc/fstab`` file, - for example: - ``echo "tmpfs /var/lib/php5/pool-www tmpfs defaults,noatime,mode=1777 0 0" + Session management on the application server. PHP sessions are stored + in a tmpfs mounted at the operating system-specific session storage + location. You can find out where that is by running ``grep -R + 'session.save_path' /etc/php5`` and then add it to the ``/etc/fstab`` file, + for example: + ``echo "tmpfs /var/lib/php5/pool-www tmpfs defaults,noatime,mode=1777 0 0" >> /etc/fstab``. * Memory Caching - A memcache speeds up server performance, and Nextcloud supports four - memcaches; refer to `Configuring Memory Caching`_ for information on + A memcache speeds up server performance, and Nextcloud supports four + memcaches; refer to `Configuring Memory Caching`_ for information on selecting and configuring a memcache. - + * Storage - Use an off-the-shelf NFS solution, such as IBM Elastic Storage or RedHat + Use an off-the-shelf NFS solution, such as IBM Elastic Storage or RedHat Ceph. Large Enterprises and Service Providers @@ -228,14 +228,14 @@ Large Enterprises and Service Providers * Number of users 5,000 to >100,000 users. - + * Storage size Up to 1 petabyte. - + * High availabily level Every component is fully redundant and can fail without service interruption. - Backups without service interruption - + Backups without service interruption + Recommended System Requirements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -251,7 +251,7 @@ Authentication via an existing LDAP or Active Directory server, or SAML. .. figure:: images/deprecs-3.png :scale: 60% - :alt: Network diagram for large enterprise. + :alt: Network diagram for large enterprise. * Components * 4 to 20 application servers with 4 sockets and 64GB RAM. @@ -263,54 +263,54 @@ Authentication via an existing LDAP or Active Directory server, or SAML. RHEL 7 with latest service packs. * SSL Configuration - The SSL termination is done in the load balancer. A standard SSL certificate - is needed, installed according to the load balancer documentation. + The SSL termination is done in the load balancer. A standard SSL certificate + is needed, installed according to the load balancer documentation. * Load Balancer - A redundant hardware load-balancer with heartbeat, for example `F5 Big-IP`_. + A redundant hardware load-balancer with heartbeat, for example `F5 Big-IP`_. This runs two load balancers in front of the application servers. * Database MySQL/MariaDB Galera Cluster with 4x master -- master replication. * Backup - Minimum daily backup without downtime. All MySQL/MariaDB statements should + Minimum daily backup without downtime. All MySQL/MariaDB statements should be replicated to a backup MySQL/MariaDB slave instance. - - * Create a snapshot on the NFS storage server. + + * Create a snapshot on the NFS storage server. * At the same time stop the MySQL replication. * Create a MySQL dump of the backup slave. * Push the NFS snapshot to the backup. * Push the MySQL dump to the backup. * Delete the NFS snapshot. * Restart MySQL replication. - + * Authentication - User authentication via one or several LDAP or Active Directory + User authentication via one or several LDAP or Active Directory servers, or SAML/Shibboleth. (See `User Authentication with LDAP`_.) * LDAP - Read-only slaves should be deployed on every application server for + Read-only slaves should be deployed on every application server for optimal scalability. * Session Management Redis should be used for the session management storage. * Caching - Redis for distributed in-memory caching (see `Configuring Memory + Redis for distributed in-memory caching (see `Configuring Memory Caching`_). - + * Storage - An off-the-shelf NFS solution should be used. Examples are IBM Elastic - Storage or RedHAT Ceph. Optionally, an S3 compatible object store can also + An off-the-shelf NFS solution should be used. Examples are IBM Elastic + Storage or RedHAT Ceph. Optionally, an S3 compatible object store can also be used. - + Hardware Considerations ----------------------- * Solid-state drives (SSDs) for I/O. * Separate hard disks for storage and database, SSDs for databases. -* Multiple network interfaces to distribute server synchronisation and backend +* Multiple network interfaces to distribute server synchronisation and backend traffic across multiple subnets. Single Machine / Scale-Up Deployment @@ -320,17 +320,17 @@ The single-machine deployment is widely used in the community. Pros: -* Easy setup: no session storage daemon, use tmpfs and memory caching to +* Easy setup: no session storage daemon, use tmpfs and memory caching to enhance performance, local storage. * No network latency to consider. -* To scale buy a bigger CPU, more memory, larger hard drive, or additional hard +* To scale buy a bigger CPU, more memory, larger hard drive, or additional hard drives. Cons: * Fewer high availability options. -* The amount of data in Nextcloud tends to continually grow. Eventually a - single machine will not scale; I/O performance decreases and becomes a +* The amount of data in Nextcloud tends to continually grow. Eventually a + single machine will not scale; I/O performance decreases and becomes a bottleneck with multiple up- and downloads, even with solid-state drives. Scale-Out Deployment @@ -338,13 +338,13 @@ Scale-Out Deployment Provider setup: -* DNS round robin to HAProxy servers (2-n, SSL offloading, cache static +* DNS round robin to HAProxy servers (2-n, SSL offloading, cache static resources) * Least load to Apache servers (2-n) * Memcached/Redis for shared session storage (2-n) -* Database cluster with single Master, multiple slaves and proxy to split +* Database cluster with single Master, multiple slaves and proxy to split requests accordingly (2-n) -* GPFS or Ceph via phprados (2-n, 3 to be safe, Ceph 10+ nodes to see speed +* GPFS or Ceph via phprados (2-n, 3 to be safe, Ceph 10+ nodes to see speed benefits under load) Pros: @@ -357,7 +357,7 @@ Cons: * More complicated to setup. * Network becomes the bottleneck (10GB Ethernet recommended). -* Currently DB filecache table will grow rapidly, making migrations painful in +* Currently DB filecache table will grow rapidly, making migrations painful in case the table is altered. What About Nginx / PHP-FPM? @@ -369,9 +369,9 @@ But on uploads stores the whole file on disk before handing it over to PHP-FPM. A Single Master DB is Single Point of Failure, Does Not Scale ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When master fails another slave can become master. However, the increased -complexity carries some risks: Multi-master has the risk of split brain, and -deadlocks. Nextcloud tries to solve the problem of deadlocks with high-level +When master fails another slave can become master. However, the increased +complexity carries some risks: Multi-master has the risk of split brain, and +deadlocks. Nextcloud tries to solve the problem of deadlocks with high-level file locking. Software Considerations @@ -380,81 +380,81 @@ Software Considerations Operating System ^^^^^^^^^^^^^^^^ -We are dependent on distributions that offer an easy way to install the various -components in up-to-date versions. Nextcloud has a partnership with RedHat -and SUSE for customers who need commercial support. Canonical, the parent -company of Ubuntu Linux, also offers enterprise service and support. Debian -and Ubuntu are free of cost, and include newer software packages. CentOS is the -community-supported free-of-cost Red Hat Enterprise Linux clone. openSUSE is -community-supported, and includes many of the same system administration tools +We are dependent on distributions that offer an easy way to install the various +components in up-to-date versions. Nextcloud has a partnership with RedHat +and SUSE for customers who need commercial support. Canonical, the parent +company of Ubuntu Linux, also offers enterprise service and support. Debian +and Ubuntu are free of cost, and include newer software packages. CentOS is the +community-supported free-of-cost Red Hat Enterprise Linux clone. openSUSE is +community-supported, and includes many of the same system administration tools as SUSE Linux Enterprise Server. Web server ^^^^^^^^^^ -Taking Apache and Nginx as the contenders, Apache with mod_php is currently the -best option, as Nginx does not support all features necessary for enterprise -deployments. Mod_php is recommended instead of PHP_FPM, because in scale-out +Taking Apache and Nginx as the contenders, Apache with mod_php is currently the +best option, as Nginx does not support all features necessary for enterprise +deployments. Mod_php is recommended instead of PHP_FPM, because in scale-out deployments separate PHP pools are simply not necessary. Relational Database ^^^^^^^^^^^^^^^^^^^ -More often than not the customer already has an opinion on what database to -use. In general, the recommendation is to use what their database administrator -is most familiar with. Taking into account what we are seeing at customer -deployments, we recommend MySQL/MariaDB in a master-slave deployment with a -MySQL proxy in front of them to send updates to master, and selects to the +More often than not the customer already has an opinion on what database to +use. In general, the recommendation is to use what their database administrator +is most familiar with. Taking into account what we are seeing at customer +deployments, we recommend MySQL/MariaDB in a master-slave deployment with a +MySQL proxy in front of them to send updates to master, and selects to the slave(s). -.. comment: MySQL locks tables for schema updates and might even have to copy - the whole table. That is pretty much a non-starter for migrations unless you - are using a scale out deployment where you can apply the schema changes to - each slave individually. Even then each migration might take several hours. +.. comment: MySQL locks tables for schema updates and might even have to copy + the whole table. That is pretty much a non-starter for migrations unless you + are using a scale out deployment where you can apply the schema changes to + each slave individually. Even then each migration might take several hours. Make sure you have enough disk space. You have been warned. -.. comment: Currently, Nextcloud uses the utf8 character set with utf8_bin - collation on MySQL installations. As a result 4 byte UTF characters like - emojis cannot be used. This can be fixed by [moving to +.. comment: Currently, Nextcloud uses the utf8 character set with utf8_bin + collation on MySQL installations. As a result 4 byte UTF characters like + emojis cannot be used. This can be fixed by [moving to utf8mb4/utf8mb4_bin](https://github.com/owncloud/core/issues/7030). -The second best option is PostgreSQL (alter table does not lock table, which -makes migration less painful) although we have yet to find a customer who uses a +The second best option is PostgreSQL (alter table does not lock table, which +makes migration less painful) although we have yet to find a customer who uses a master-slave setup. -.. comment: PostgreSQL may produce excessive amounts of dead tuples due to +.. comment: PostgreSQL may produce excessive amounts of dead tuples due to Nextcloud transactions preventing the execution of the autovacum process. What about the other DBMS? -* Sqlite is adequate for simple testing, and for low-load single-user +* Sqlite is adequate for simple testing, and for low-load single-user deployments. It is not adequate for production systems. * Microsoft SQL Server is not a supported option. -* Oracle DB is the de facto standard at large enterprises and is fully - supported with Nextcloud Enterprise Edition only. +* Oracle DB is the de facto standard at large enterprises and is currently only + supported with an Nextcloud Enterprise Support Subscription only. File Storage ------------ -While many customers are starting with NFS, sooner or later that requires scale-out storage. Currently the options are GPFS or GlusterFS, or an object store protocol like S3 (supported in Enterprise Edition only) or Swift. S3 also allows access to Ceph Storage. +While many customers are starting with NFS, sooner or later that requires scale-out storage. Currently the options are GPFS or GlusterFS, or an object store protocol like S3 or Swift. S3 also allows access to Ceph Storage. -.. comment: A proof of concept implementation based on - [phprados](https://github.com/ceph/phprados) that talks directly to a - [ceph](http://ceph.com/) cluster without having to use temp files is [in +.. comment: A proof of concept implementation based on + [phprados](https://github.com/ceph/phprados) that talks directly to a + [ceph](http://ceph.com/) cluster without having to use temp files is [in development](https://github.com/owncloud/objectstore/pull/26). -.. comment: NFS can be used but needs to be micro-managed to distribute users - on multiple storages. If you want to go that route configure ldap to provide - a custom home folder location. That allows you to move each users data +.. comment: NFS can be used but needs to be micro-managed to distribute users + on multiple storages. If you want to go that route configure ldap to provide + a custom home folder location. That allows you to move each users data folder to different nfs mounts. Session Storage --------------- -* Redis: provides persistence, nice graphical inspection tools available, +* Redis: provides persistence, nice graphical inspection tools available, supports Nextcloud high-level file locking. - -* If Shibboleth is a requirement you must use Memcached, and it can also be + +* If Shibboleth is a requirement you must use Memcached, and it can also be used to scale-out shibd session storage (see `Memcache StorageService`_). .. comment: High Availability / Failover deployment @@ -464,33 +464,33 @@ References ---------- `Database High Availability`_ - + `Performance enhancements for Apache and PHP`_ `How to Set Up a Redis Server as a Session Handler for PHP on Ubuntu 14.04`_ .. TODO ON RELEASE: Update version number below on release -.. _Maintenance: +.. _Maintenance: https://docs.nextcloud.org/server/11/admin_manual/maintenance/index.html .. _User Authentication with LDAP: https://docs.nextcloud.org/server/11/admin_manual/configuration_user/user_auth_ldap.html -.. _Configuring Memory Caching: +.. _Configuring Memory Caching: https://docs.nextcloud.org/server/11/admin_manual/configuration_server/caching_configuration.html -.. _Nextcloud Server or Enterprise Edition: +.. _Nextcloud Server or Enterprise Edition: https://nextcloud.com/enterprise/ .. _F5 Big-IP: https://f5.com/products/big-ip/ -.. _Memcache StorageService: +.. _Memcache StorageService: https://wiki.shibboleth.net/confluence/display/SHIB2/ NativeSPStorageService#NativeSPStorageService-MemcacheStorageService - -.. _Database High Availability: + +.. _Database High Availability: http://www.severalnines.com/blog/become-mysql-dba-blog-series-database-high- availability -.. _Performance enhancements for Apache and PHP: - http://blog.bitnami.com/2014/06/performance-enhacements-for-apache-and.html -.. _How to Set Up a Redis Server as a Session Handler for PHP on Ubuntu 14.04: +.. _Performance enhancements for Apache and PHP: + http://blog.bitnami.com/2014/06/performance-enhacements-for-apache-and.html +.. _How to Set Up a Redis Server as a Session Handler for PHP on Ubuntu 14.04: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-redis-server -as -a-session-handler-for-php-on-ubuntu-14-04 .. _HAProxy documentation: