Applications Configuration¶

The Telemetruum Leaf is able to monitor not only the ICOS system, but also to get metrics and logs from user applications (that can be used by the ICOS System to evaluate application policies). To make it possible, a specific configuration is needed during the application deployment.

Metrics¶

User applications can get their metrics collected by the Telemetruum Leaf running in the same node of the application in two ways:

1. expose a Prometheus-compatible endpoint (e.g. /metrics API) and configure properly Kubernetes and Docker to let this endpoint discovered by the Telemetruum Leaf
2. push the metrics directly to the Telemetruum Leaf using the Open Telemetry format

Next subsections explain how to configure an application with method (1), while method (2) is still under development and not yet documented.

Kuberentes Prometheus Metrics Endpoints Discovery¶

In Kubernetes, the Telemetruum Leaf is able to discover endpoints that exposes metrics in two ways: using annotations or using CRDs.

Annotations¶

To configure scraping of metrics from an endpoint exposed by a service or a pod, the following annotations are supported when added to a service or a pod resource:

annotations:

  # enable/disable the scraping of metrics from this service/pod
  # Default: false
  telemetry.icos.eu/scrape: true

  # the interval of time at which the endponint will be scraped
  # Default: 60s
  telemetry.icos.eu/interval: 120s

  # the timeout for the scraping call
  # Default: 2s
  telemetry.icos.eu/timeout: 10s

  # the protocol to use (e.g, http, https)
  # Default: http
  telemetry.icos.eu/scheme: http

  # the path at which scrape
  # Default: /
  telemetry.icos.eu/path: /metrics

  # the port at which scrape
  # Default: 80
  telemetry.icos.eu/path: 9100

CRDs¶

The Telemetry agent watches Service Monitor and PodMonitor CRDs and configure itself to scrape metrics from the endpoints pointed by those resources.

More detailed instrcutions on how to use them can be found here.

Docker Prometheus Metrics Endpoints Discovery¶

In Docker, the Telemetruum Leaf is able to discover endpoints that exposes metrics using container labels. The Telemetruum Leaf queires the Docker Engine looking for containers that have a telemetry.icos.eu/scrape=true label and will try to call the /metrics endpoint exposed by that container (customizable using other labels).

labels:
  # enable/disable the scraping of metrics from this service/pod
  # Default: false
  telemetry.icos.eu/scrape: "true"

  # the interval of time at which the endponint will be scraped
  # Default: 60s
  telemetry.icos.eu/interval: 120s

  # the timeout for the scraping call
  # Default: 2s
  telemetry.icos.eu/timeout: 10s

  # the protocol to use (e.g, http, https)
  # Default: http
  telemetry.icos.eu/scheme: http

  # the path at which scrape
  # Default: /
  telemetry.icos.eu/path: /metrics

  # the port at which scrape
  # Default: 80
  telemetry.icos.eu/path: 9100

Logs¶

Kuberentes Pod Logs Discovery¶

By default, logs from pods are not collected. To enable them, just add an annotation on the pod.

annotations:

  # enable/disable the collection of logs from this pod
  # Default: false
  logs.icos.eu/scrape: true

TODO: multi-container pods configuration

Custom configuration values¶

Custom values can be set by other ICOS components to keep track of the application configuration. For instance, the ICOS Job Manager sets the following values:

app.icos.eu/component: hello-world
app.icos.eu/instance: 15e24fce-b24d-4d83-8b22-9ecbefd739b7
jobmanager.icos.eu/manifest: 53e0668e-7276-4251-91b4-99365b61eeb1

Any value that matches the format <prefix>.icos.eu/<name>=<value> is considered valid and it is exposed by the icos_workload_info metric as icos_<prefix>_<name>=<value> label.

For instance a Kuberentes pod that define the following annotations (or a Docker container that defined the following labels):

metadata:
  annotations:
    app.icos.eu/component: hello-world
    logs.icos.eu/scrape: true
    custom.icos.eu/conf: myvalue

will produce the following labels in the corresponding tlum_workload_info metric:

tlum_workload_info{
    ...
    icos_app_component="hello-world",
    icos_logs_scrape="true",
    icos_custom_conf="myvalue"
    ...
}