diff --git a/.github/vale/config/vocabularies/Docker/accept.txt b/.github/vale/config/vocabularies/Docker/accept.txt index 1129461f1f..f77e3a17bd 100644 --- a/.github/vale/config/vocabularies/Docker/accept.txt +++ b/.github/vale/config/vocabularies/Docker/accept.txt @@ -27,6 +27,7 @@ DHCP DNS DSOS DVP +Datadog Ddosify Dev Environments? Django diff --git a/content/scout/images/datadog_dashboard_1.png b/content/scout/images/datadog_dashboard_1.png new file mode 100644 index 0000000000..9263b5088b Binary files /dev/null and b/content/scout/images/datadog_dashboard_1.png differ diff --git a/content/scout/images/datadog_dashboard_2.png b/content/scout/images/datadog_dashboard_2.png new file mode 100644 index 0000000000..0898550312 Binary files /dev/null and b/content/scout/images/datadog_dashboard_2.png differ diff --git a/content/scout/images/datadog_metrics_summary.png b/content/scout/images/datadog_metrics_summary.png new file mode 100644 index 0000000000..fdf902ca6e Binary files /dev/null and b/content/scout/images/datadog_metrics_summary.png differ diff --git a/content/scout/images/scout-metrics-grafana-policy.png b/content/scout/images/scout-metrics-grafana-policy.png new file mode 100644 index 0000000000..99042b35fe Binary files /dev/null and b/content/scout/images/scout-metrics-grafana-policy.png differ diff --git a/content/scout/images/scout-metrics-grafana-vulns.png b/content/scout/images/scout-metrics-grafana-vulns.png new file mode 100644 index 0000000000..a48a279bf2 Binary files /dev/null and b/content/scout/images/scout-metrics-grafana-vulns.png differ diff --git a/content/scout/images/scout-metrics-grafana.png b/content/scout/images/scout-metrics-grafana.png deleted file mode 100644 index 0e49733e0d..0000000000 Binary files a/content/scout/images/scout-metrics-grafana.png and /dev/null differ diff --git a/content/scout/metrics-exporter.md b/content/scout/metrics-exporter.md index 758016ceb9..bb16c7ea7e 100644 --- a/content/scout/metrics-exporter.md +++ b/content/scout/metrics-exporter.md @@ -7,9 +7,9 @@ keywords: scout, exporter, prometheus, grafana, metrics, dashboard, api, compose --- Docker Scout exposes a metrics HTTP endpoint that lets you scrape vulnerability -and policy data from Docker Scout, using Prometheus. With this you can create -your own, self-hosted Docker Scout dashboards for visualizing supply chain -metrics. +and policy data from Docker Scout, using Prometheus or Datadog. With this you +can create your own, self-hosted Docker Scout dashboards for visualizing supply +chain metrics. ## Metrics @@ -30,7 +30,7 @@ The metrics endpoint exposes the following metrics: > > Streams is mostly an internal concept in Docker Scout, > with the exception of the data exposed through this metrics endpoint. -{ .tip } +{ .tip #stream } ## Creating an access token @@ -43,7 +43,9 @@ To create a PAT, follow the steps in [Create an access token](/security/for-deve Once you have created the PAT, store it in a secure location. You will need to provide this token to the exporter when scraping metrics. -## Configure Prometheus +## Prometheus + +This section describes how to scrape the metrics endpoint using Prometheus. ### Add a job for your organization @@ -64,16 +66,6 @@ scrape_configs: The address in the `targets` field is set to the domain name of the Docker Scout API, `api.scout.docker.com`. Make sure that there's no firewall rule in place preventing the server from communicating with this endpoint. -### Scrape interval - -By default, Prometheus scrapes the metrics every 15 seconds. -You can change the scrape interval by setting the `scrape_interval` field in the Prometheus configuration file at the global or job level. -A scraping interval of 60 minutes or higher is recommended. - -Because of the own nature of vulnerability data, the metrics exposed through this API are unlikely to change at a high frequency. -For this reason, the metrics endpoint has a 60-minute cache by default. -If you set the scrape interval to less than 60 minutes, you will see the same data in the metrics for multiple scrapes during that time window. - ### Add bearer token authentication To scrape metrics from the Docker Scout Exporter endpoint using Prometheus, you need to configure Prometheus to use the PAT as a bearer token. @@ -100,7 +92,7 @@ If you are running Prometheus in a Docker container or Kubernetes pod, mount the Finally, restart Prometheus to apply the changes. -## Sample project +### Prometheus sample project If you don't have a Prometheus server set up, you can run a [sample project](https://github.com/dockersamples/scout-metrics-exporter) using Docker Compose. The sample includes a Prometheus server that scrapes metrics for a Docker organization enrolled in Docker Scout, @@ -111,16 +103,17 @@ alongside Grafana with a pre-configured dashboard to visualize the vulnerability ```console $ git clone git@github.com:dockersamples/scout-metrics-exporter.git + $ cd scout-metrics-exporter/prometheus ``` 2. [Create a Docker access token](/security/for-developers/access-tokens/#create-an-access-token) - and store it in a plain text file at `prometheus/token` under the template directory. + and store it in a plain text file at `/prometheus/prometheus/token` under the template directory. ```plaintext {title=token} $ echo $DOCKER_PAT > ./prometheus/token ``` -3. In the Prometheus configuration file at `prometheus/prometheus.yml`, +3. In the Prometheus configuration file at `/prometheus/prometheus/prometheus.yml`, replace `ORG` in the `metrics_path` property on line 6 with the namespace of your Docker organization. ```yaml {title="prometheus/prometheus.yml",hl_lines="6",linenos=1} @@ -170,11 +163,187 @@ Prometheus UI at . To view the Grafana dashboards, go to , and sign in using the credentials defined in the Docker Compose file (username: `admin`, password: `grafana`). -![Policy dashboard in Grafana](./images/scout-metrics-grafana.png "Policy dashboard in Grafana") +![Vulnerability dashboard in Grafana](./images/scout-metrics-grafana-vulns.png "Vulnerability dashboard in Grafana") + +![Policy dashboard in Grafana](images/scout-metrics-grafana-policy.png "Policy dashboard in Grafana") The dashboards are pre-configured to visualize the vulnerability and policy metrics scraped by Prometheus. -### Revoke an access token +## Datadog + +This section describes how to scrape the metrics endpoint using Datadog. +Datadog pulls data for monitoring by running a customizable +[agent](https://docs.datadoghq.com/agent/?tab=Linux) that scrapes available +endpoints for any exposed metrics. The OpenMetrics and Prometheus checks are +included in the agent, so you don’t need to install anything else on your +containers or hosts. + +This guide assumes you have a Datadog account and a Datadog API Key. Refer to +the [Datadog documentation](https://docs.datadoghq.com/agent) to get started. + +### Configure the Datadog agent + +To start collecting the metrics, you will need to edit the agent’s +configuration file for the OpenMetrics check. If you're running the agent as a +container, such file must be mounted at +`/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml`. + +The following example shows a Datadog configuration that: + +- Specifies the OpenMetrics endpoint targeting the `dockerscoutpolicy` Docker organization +- A `namespace` that all collected metrics will be prefixed with +- The [`metrics`](#metrics) you want the agent to scrape (`scout_*`) +- An `auth_token` section for the Datadog agent to authenticate to the Metrics + endpoint, using a Docker PAT as a Bearer token. + +```yaml +instances: + - openmetrics_endpoint: "https://api.scout.docker.com/v1/exporter/org/dockerscoutpolicy" + namespace: "scout-metrics-exporter" + metrics: + - scout_* + auth_token: + reader: + type: file + path: /var/run/secrets/scout-metrics-exporter/token + writer: + type: header + name: Authorization + value: Bearer +``` + +> **Important** +> +> Do not replace the `` placeholder in the previous configuration +> example. It must stay as it is. Only make sure the Docker PAT is correctly +> mounted into the Datadog agent in the specified filesystem path. Save the +> file as `conf.yaml` and restart the agent. +{ .important } + +When creating a Datadog agent configuration of your own, make sure to edit the +`openmetrics_endpoint` property to target your organization, by replacing +`dockerscoutpolicy` with the namespace of your Docker organization. + +### Datadog sample project + +If you don't have a Datadog server set up, you can run a [sample project](https://github.com/dockersamples/scout-metrics-exporter) +using Docker Compose. The sample includes a Datadog agent, running as a +container, that scrapes metrics for a Docker organization enrolled in Docker +Scout. This sample project assumes that you have a Datadog account, an API key, +and a Datadog site. + +1. Clone the starter template for bootstrapping a Datadog Compose service for + scraping the Docker Scout metrics endpoint: + + ```console + $ git clone git@github.com:dockersamples/scout-metrics-exporter.git + $ cd scout-metrics-exporter/datadog + ``` + +2. [Create a Docker access token](/security/for-developers/access-tokens/#create-an-access-token) + and store it in a plain text file at `/datadog/token` under the template directory. + + ```plaintext {title=token} + $ echo $DOCKER_PAT > ./token + ``` + +3. In the `/datadog/compose.yaml` file, update the `DD_API_KEY` and `DD_SITE` environment variables + with the values for your Datadog deployment. + + ```yaml {hl_lines="5-6"} + datadog-agent: + container_name: datadog-agent + image: gcr.io/datadoghq/agent:7 + environment: + - DD_API_KEY=${DD_API_KEY} # e.g. 1b6b3a42... + - DD_SITE=${DD_SITE} # e.g. datadoghq.com + - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - ./conf.yaml:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml:ro + - ./token:/var/run/secrets/scout-metrics-exporter/token:ro + ``` + + The `volumes` section mounts the Docker socket from the host to the + container. This is required to obtain an accurate hostname when running as a + container ([more details here](https://docs.datadoghq.com/agent/troubleshooting/hostname_containers/)). + + It also mounts the agent's config file and the Docker access token. + +4. Edit the `/datadog/config.yaml` file by replacing the placeholder `` in + the `openmetrics_endpoint` property with the namespace of the Docker + organization that you want to collect metrics for. + + ```yaml {hl_lines=2} + instances: + - openmetrics_endpoint: "https://api.scout.docker.com/v1/exporter/org/<>" + namespace: "scout-metrics-exporter" + # ... + ``` + +5. Start the Compose services. + + ```console + docker compose up -d + ``` + +If configured properly, you should see the OpenMetrics check under Running +Checks when you run the agent’s status command whose output should look similar +to: + +```text +openmetrics (4.2.0) +------------------- + Instance ID: openmetrics:scout-prometheus-exporter:6393910f4d92f7c2 [OK] + Configuration Source: file:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml + Total Runs: 1 + Metric Samples: Last Run: 236, Total: 236 + Events: Last Run: 0, Total: 0 + Service Checks: Last Run: 1, Total: 1 + Average Execution Time : 2.537s + Last Execution Date : 2024-05-08 10:41:07 UTC (1715164867000) + Last Successful Execution Date : 2024-05-08 10:41:07 UTC (1715164867000) +``` + +For a comprehensive list of options, take a look at this [example config file](https://github.com/DataDog/integrations-core/blob/master/openmetrics/datadog_checks/openmetrics/data/conf.yaml.example) for the generic OpenMetrics check. + +### Visualizing your data + +Once the agent is configured to grab Prometheus metrics, you can use them to build comprehensive Datadog graphs, dashboards, and alerts. + +Go into your [Metric summary page](https://app.datadoghq.com/metric/summary?filter=scout_prometheus_exporter) +to see the metrics collected from this example. This configuration will collect +all exposed metrics starting with `scout_` under the namespace +`scout_metrics_exporter`. + +![datadog_metrics_summary](images/datadog_metrics_summary.png) + +The following screenshots show examples of a Datadog dashboard containing +graphs about vulnerability and policy compliance for a specific [stream](#stream). + +![datadog_dashboard_1](images/datadog_dashboard_1.png) +![datadog_dashboard_2](images/datadog_dashboard_2.png) + +> The reason why the lines in the graphs look flat is due to the own nature of +> vulnerabilities (they don't change too often) and the short time interval +> selected in the date picker. + +## Scrape interval + +By default, Prometheus and Datadog scrape metrics at a 15 second interval. +Because of the own nature of vulnerability data, the metrics exposed through this API are unlikely to change at a high frequency. +For this reason, the metrics endpoint has a 60-minute cache by default, +which means a scraping interval of 60 minutes or higher is recommended. +If you set the scrape interval to less than 60 minutes, you will see the same data in the metrics for multiple scrapes during that time window. + +To change the scrape interval: + +- Prometheus: set the `scrape_interval` field in the Prometheus configuration + file at the global or job level. +- Datadog: set the `min_collection_interval` property in the Datadog agent + configuration file, see [Datadog documentation](https://docs.datadoghq.com/developers/custom_checks/write_agent_check/#updating-the-collection-interval). + +## Revoke an access token If you suspect that your PAT has been compromised or is no longer needed, you can revoke it at any time. To revoke a PAT, follow the steps in the [Create and manage access tokens](/security/for-developers/access-tokens/#modify-existing-tokens).