diff --git a/ee/ucp/images/ingress-deploy.png b/ee/ucp/images/ingress-deploy.png new file mode 100644 index 0000000000..cadbf33928 Binary files /dev/null and b/ee/ucp/images/ingress-deploy.png differ diff --git a/ee/ucp/kubernetes/layer-7-routing.md b/ee/ucp/kubernetes/layer-7-routing.md index d12a15a961..a57ef051d2 100644 --- a/ee/ucp/kubernetes/layer-7-routing.md +++ b/ee/ucp/kubernetes/layer-7-routing.md @@ -7,6 +7,8 @@ redirect_from: - /ee/ucp/kubernetes/deploy-ingress-controller/ --- +## Overview + When you deploy a Kubernetes application, you may want to make it accessible to users using hostnames instead of IP addresses. @@ -19,87 +21,180 @@ Use an ingress controller when you want to: * Give your Kubernetes app an externally-reachable URL. * Load-balance traffic to your app. -Kubernetes provides an NGINX ingress controller that you can use in Docker EE -without modifications. -Learn about [ingress in Kubernetes](https://v1-8.docs.kubernetes.io/docs/concepts/services-networking/ingress/). +Ingress is an API object that manages external access to the services in a cluster, typically HTTP. In UCP 3.1, Ingress must be deployed separately. There are a variety of ingress controllers, but for UCP, the supported one is the [NGINX Ingress Controller](https://github.com/kubernetes/ingress-nginx). The following instructions provide highly-available production deployment of this controller. -## Create a dedicated namespace +## Prerequisites -1. Navigate to the **Namespaces** page, and click **Create**. -2. In the **Object YAML** editor, append the following text. - ```yaml - metadata: - name: ingress-nginx - ``` +- UCP deployed and properly configured +- Two or three dedicated infrae nodes deployed as UCP worker nodes +- An external load-balancer fronting these nodes with an associated VIP that resolves the application DNS (for example, `*.app.docker.mycompany.com`) - The finished YAML should look like this. +### Step 1: Labeling infrastructure nodes - ```yaml - apiVersion: v1 - kind: Namespace - metadata: - name: ingress-nginx - ``` -3. Click **Create**. -4. In the **ingress-nginx** namespace, click the **More options** icon, - and in the context menu, select **Set Context**. +To deploy the NGINX controller on the infra nodes, the nodes must be labeled. Identify 2 or 3 nodes as the infra nodes to host the ingress controller. The following example uses three nodes: `dockeree-worker-linux-1`, `dockeree-worker-linux-2`, and `dockeree-worker-linux-3`: - ![](../images/deploy-ingress-controller-1.png){: .with-border} +``` + 🐳 → kubectl get nodes +NAME STATUS ROLES AGE VERSION +dockeree-worker-linux-1 Ready 5d v1.8.11-docker-8d637ae +dockeree-worker-linux-2 Ready 5d v1.8.11-docker-8d637ae +dockeree-worker-linux-3 Ready 5d v1.8.11-docker-8d637ae -## Create a grant +``` -The default service account that's associated with the `ingress-nginx` -namespace needs access to Kubernetes resources, so create a grant with -`Restricted Control` permissions. +For this example, the label `infra.role=ingress` is used to label the nodes: -1. From UCP, navigate to the **Grants** page, and click **Create Grant**. -2. Within the **Subject** pane, select **Service Account**. For the - **Namespace** select **ingress-nginx**, and select **default** for - the **Service Account**. Click **Next**. -3. Within the **Role** pane, select **Restricted Control**, and then click - **Next**. -4. Within the **Resource Set** pane, select the **Type** **Namespace**, and - select the **Apply grant to all existing and new namespaces** toggle. -5. Click **Create**. +``` +🐳 → kubectl label node dockeree-worker-linux-1 infra.role=ingress +node "dockeree-worker-linux-1" labeled +🐳 → kubectl label node dockeree-worker-linux-2 infra.role=ingress +node "dockeree-worker-linux-2" labeled +🐳 → kubectl label node dockeree-worker-linux-3 infra.role=ingress +node "dockeree-worker-linux-3" labeled -## Deploy NGINX ingress controller +``` -The cluster is ready for the ingress controller deployment, which has three -main components: +### Step 2: Create a Dedicated Namespace -- a simple HTTP server, named `default-http-backend`, -- an ingress controller, named `nginx-ingress-controller`, and -- a service that exposes the app, named `ingress-nginx`. +A dedicated namespace, for example, `infra`, is needed for all infrastructure deployment activities. You also need a service account to enable the ingress controller to work with the Kubernetes API. After creating a namespace and a service account, you must create an RBAC policy to only allow infrastructure deployments on the dedicated nodes that were labelled in the previous example. -Navigate to the **Create Kubernetes Object** page, and in the **Object YAML** -editor, paste the following YAML. +To create a namespace and a service account, simply use the following [YAML file](config/ns-and-sa.yaml) and apply it via the CLI or the UI: -```yaml -apiVersion: apps/v1beta2 +``` +🐳 → cat ns-and-sa.yaml +apiVersion: v1 +kind: Namespace +metadata: + name: infra +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: nginx-ingress-service-account + namespace: infra +``` + +Then apply the namespace and service account via `kubectl`: + +``` +🐳 → kubectl apply -f ns-and-sa.yaml +namespace "infra" created +serviceaccount "nginx-ingress-service-account" created + +``` + +### Step 3: Create an RBAC Policy + +Apply an RBAC role-binding for NGINX controller access to the API Server, as shown in the following example: + + +``` +🐳 → cat ingress-rbac.yaml +## Source: https://github.com/nginxinc/kubernetes-ingress/blob/master/deployments/rbac/rbac.yaml +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1beta1 +metadata: + name: nginx-ingress-cluster-role +rules: +- apiGroups: + - "" + resources: + - services + - endpoints + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - secrets + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - configmaps + verbs: + - get + - list + - watch + - update + - create +- apiGroups: + - "" + resources: + - pods + verbs: + - list +- apiGroups: + - "" + resources: + - events + verbs: + - create + - patch +- apiGroups: + - extensions + resources: + - ingresses + verbs: + - list + - watch + - get +- apiGroups: + - "extensions" + resources: + - ingresses/status + verbs: + - update +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1beta1 +metadata: + name: nginx-ingress-cluster-rb +subjects: +- kind: ServiceAccount + name: nginx-ingress-service-account + namespace: infra +roleRef: + kind: ClusterRole + name: nginx-ingress-cluster-role + apiGroup: rbac.authorization.k8s.io +``` + +Then apply it with `kubectl`: + +``` +🐳 → kubectl apply -f ingress-rbac.yaml +``` + +### Step 4: Deploy NGINX Controller + +Note that following example uses hostPorts for controller ports. This exposes the host port ( selected in a high range using `hostPort: 38080`) directly into the nodes. Port 38080 is used for HTTP and port 38443 is used for HTTPS. Make sure that your loadbalancer forwards to the applicable ports on the nodes. You can change them as needed. + + +``` +🐳 → cat nginx-ingress-deployment.yaml +apiVersion: extensions/v1beta1 kind: Deployment metadata: name: default-http-backend labels: app: default-http-backend - namespace: ingress-nginx + namespace: infra spec: - replicas: 1 - selector: - matchLabels: - app: default-http-backend + replicas: 3 template: metadata: labels: app: default-http-backend - annotations: - seccomp.security.alpha.kubernetes.io/pod: docker/default spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend - # Any image is permissible as long as: - # 1. It serves a 404 page at / - # 2. It serves 200 on a /healthz endpoint image: gcr.io/google_containers/defaultbackend:1.4 livenessProbe: httpGet: @@ -117,12 +212,14 @@ spec: requests: cpu: 10m memory: 20Mi + nodeSelector: + infra.role: ingress --- apiVersion: v1 kind: Service metadata: name: default-http-backend - namespace: ingress-nginx + namespace: infra labels: app: default-http-backend spec: @@ -136,7 +233,7 @@ kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration - namespace: ingress-nginx + namespace: infra labels: app: ingress-nginx --- @@ -144,21 +241,21 @@ kind: ConfigMap apiVersion: v1 metadata: name: tcp-services - namespace: ingress-nginx + namespace: infra --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services - namespace: ingress-nginx + namespace: infra --- -apiVersion: apps/v1beta2 +apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-ingress-controller - namespace: ingress-nginx + namespace: infra spec: - replicas: 1 + replicas: 3 selector: matchLabels: app: ingress-nginx @@ -169,7 +266,6 @@ spec: annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' - seccomp.security.alpha.kubernetes.io/pod: docker/default spec: initContainers: - command: @@ -181,9 +277,10 @@ spec: name: sysctl securityContext: privileged: true + serviceAccountName: nginx-ingress-service-account containers: - name: nginx-ingress-controller - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0 + image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend @@ -191,7 +288,6 @@ spec: - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --annotations-prefix=nginx.ingress.kubernetes.io - - --enable-ssl-passthrough env: - name: POD_NAME valueFrom: @@ -204,8 +300,12 @@ spec: ports: - name: http containerPort: 80 + hostPort: 38443 + protocol: TCP - name: https containerPort: 443 + hostPort: 38443 + protocol: TCP livenessProbe: failureThreshold: 3 httpGet: @@ -225,27 +325,121 @@ spec: periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 ---- -apiVersion: v1 -kind: Service -metadata: - name: ingress-nginx - namespace: ingress-nginx -spec: - type: NodePort - ports: - - name: http - port: 80 - targetPort: 80 - protocol: TCP - - name: https - port: 443 - targetPort: 443 - protocol: TCP - selector: - app: ingress-nginx + nodeSelector: + infra.role: ingress ``` +Deploy the controller using `kubectl` and verify pods are deployed successfully: + +``` +🐳 → kubectl apply -f nginx-ingress-deployment.yaml +deployment.extensions "default-http-backend" created +service "default-http-backend" created +configmap "nginx-configuration" created +configmap "tcp-services" created +configmap "udp-services" created +deployment.extensions "nginx-ingress-controller" created + + 🐳 → kubectl get pod -n infra -o wide +NAME READY STATUS RESTARTS AGE IP NODE +default-http-backend-7ff9774865-hsj46 1/1 Running 0 1m 192.168.145.6 dockeree-worker-linux-1 +default-http-backend-7ff9774865-kcqhj 1/1 Running 0 1m 192.168.116.145 dockeree-worker-linux-3 +default-http-backend-7ff9774865-xq566 1/1 Running 0 1m 192.168.247.210 dockeree-worker-linux-2 +nginx-ingress-controller-6b987cbbc6-4qqz8 1/1 Running 0 1m 192.168.145.7 dockeree-worker-linux-1 +nginx-ingress-controller-6b987cbbc6-h6rmg 1/1 Running 0 1m 192.168.116.146 dockeree-worker-linux-3 +nginx-ingress-controller-6b987cbbc6-hkw86 1/1 Running 0 1m 192.168.247.211 dockeree-worker-linux-2 +``` + +### Step 5: Deploy an application and its ingress rule + +After the controller is successfully deployed, you can create ingress objects to expose applications externally. The following example application deployment uses deployment yaml consisting of `Service` and `Deployment` objects that deploy the application with the Docker image `ehazlett/docker-demo` and create a service associated with it. + + +``` +🐳 → cat dockerdemo.yaml + +kind: Service +apiVersion: v1 +metadata: + namespace: default + name: docker-demo-svc +spec: + selector: + app: dockerdemo + ports: + - protocol: TCP + port: 8080 + targetPort: 8080 +--- +apiVersion: apps/v1beta2 +kind: Deployment +metadata: + namespace: default + name: dockerdemo-deploy + labels: + app: dockerdemo +spec: + selector: + matchLabels: + app: dockerdemo + strategy: + type: Recreate + template: + metadata: + labels: + app: dockerdemo + spec: + containers: + - image: ehazlett/docker-demo + name: docker-demo-container + env: + - name: app + value: dockerdemo + ports: + - containerPort: 8080 +``` + + +To expose this application with a valid URL, for example, `dockerdemo.app.docker.example.com`, you must create an `Ingress` rule. The ingress rule is used by the NGINX controller to generate the proper NGINX configuration for proper load-balancing. For example: + +``` +🐳 → cat dockerdemo.ingress.yaml + +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + name: dockerdemo-ingress + namespace: default + annotations: + kubernetes.io/ingress.class: "nginx" +spec: + rules: + - host: dockerdemo.app.docker.example.com + http: + paths: + - path: / + backend: + serviceName: docker-demo-svc + servicePort: 8080 +``` + +``` + 🐳 → kubectl apply -f dockerdemo.ingress.yaml +``` + +You can apply ingress rules separately or combine them with the application deployment yaml file. + +``` + + 🐳 → kubectl get ingress +NAME HOSTS ADDRESS PORTS AGE +dockerdemo-ingress dockerdemo.app.docker.example.com 80 7d +``` + +Assuming you have already registered a DNS record for your application pointing to the external load-balancer fronting the `infra` nodes, you should be able to access your application using the URL.You can also scale the docker demo deployment to test how ingress correctly routes traffic to all the backend pods! + +![dockerdemo.png](../images/ingress-deploy.png) + ## Check your deployment The `default-http-backend` provides a simple service that serves a 404 page