v3.15.1
Starting with v3.12.0, the default scrape jobs are defined under scrapeConfigs, keyed by job name. To change a default job, set its fields under its key; to turn it off, set its enabled field to false. To add a scrape job, add a key under scrapeConfigs.
Jobs set under serverFiles."prometheus.yml".scrape_configs are added to the scrapeConfigs jobs, not used instead of them. A serverFiles job with the same job_name as a scrapeConfigs job replaces that job. Other job names must be unique across scrapeConfigs, serverFiles and extraScrapeConfigs, or Prometheus rejects the configuration.
The package build 3.15.1-383aa49f… (the CurrentVersion column of kubectl get packages) doesn’t apply serverFiles."prometheus.yml".scrape_configs jobs, and setting extraScrapeConfigs makes its configuration invalid. With that build, use scrapeConfigs only.
Configuring Prometheus in EKS Anywhere package spec
Example
apiVersion: packages.eks.amazonaws.com/v1alpha1
kind: Package
metadata:
name: generated-prometheus
namespace: eksa-packages-<cluster-name>
spec:
packageName: prometheus
targetNamespace: observability
config: |
server:
replicaCount: 2
statefulSet:
enabled: true
Configurable parameters and default values under spec.config
| Parameter | Description | Default |
|---|---|---|
| General | ||
| rbac.create | Specifies if clusterRole / role and clusterRoleBinding / roleBinding will be created for prometheus-server and node-exporter | true |
| sourceRegistry | Specifies image source registry for prometheus-server and node-exporter | "783794618700.dkr.ecr.us-west-2.amazonaws.com" |
| Node-Exporter | ||
| nodeExporter.enabled | Indicates if node-exporter is enabled | true |
| nodeExporter.hostNetwork | Indicates if node-exporter shares the host network namespace | true |
| nodeExporter.hostPID | Indicates if node-exporter shares the host process ID namespace | true |
| nodeExporter.image.pullPolicy | Specifies node-exporter image pull policy: IfNotPresent, Always, Never |
"IfNotPresent" |
| nodeExporter.image.repository | Specifies node-exporter image repository | "prometheus/node-exporter" |
| nodeExporter.resources | Specifies resource requests and limits of the node-exporter container. Refer to the Kubernetes API documentation ResourceRequirements field for more details | {} |
| nodeExporter.service | Specifies how to expose node-exporter as a network service | See footnote 1 |
| nodeExporter.tolerations | Specifies node tolerations for node-exporter scheduling to nodes with taints. Refer to the Kubernetes API documentation toleration field for more details. | See footnote 2 |
| serviceAccounts.nodeExporter.annotations | Specifies node-exporter service account annotations | {} |
| serviceAccounts.nodeExporter.create | Indicates if node-exporter service account will be created | true |
| serviceAccounts.nodeExporter.name | Specifies node-exporter service account name | "" |
| Prometheus-Server | ||
| server.global.evaluation_interval | Specifies how frequently the prometheus-server rules are evaluated | "1m" |
| server.global.scrape_interval | Specifies how frequently prometheus-server will scrape targets | "1m" |
| server.global.scrape_timeout | Specifies how long until a prometheus-server scrape request times out | "10s" |
| server.image.pullPolicy | Specifies prometheus-server image pull policy: IfNotPresent, Always, Never |
Not set; Kubernetes uses IfNotPresent |
| server.image.repository | Specifies prometheus-server image repository | "prometheus/prometheus" |
| server.name | Specifies prometheus-server container name | "server" |
| server.persistentVolume.accessModes | Specifies prometheus-server data Persistent Volume access modes | "ReadWriteOnce" |
| server.persistentVolume.enabled | Indicates if prometheus-server will create/use a Persistent Volume Claim | true |
| server.persistentVolume.existingClaim | Specifies prometheus-server data Persistent Volume existing claim name. It requires server.persistentVolume.enabled: true. If defined, PVC must be created manually before volume will be bound |
"" |
| server.persistentVolume.size | Specifies prometheus-server data Persistent Volume size | "8Gi" |
| server.remoteRead | Specifies prometheus-server remote read configs. Refer to Prometheus docs remote_read for more details | [] |
| server.remoteWrite | Specifies prometheus-server remote write configs. Refer to Prometheus docs remote_write for more details | [] |
| server.replicaCount | Specifies the replicaCount for prometheus-server deployment / statefulSet. Note: server.statefulSet.enabled should be set to true if server.replicaCount is greater than 1 |
1 |
| server.resources | Specifies resource requests and limits of the prometheus-server container. Refer to the Kubernetes API documentation ResourceRequirements field for more details | {} |
| server.retention | Specifies prometheus-server data retention period | "15d" |
| server.service | Specifies how to expose prometheus-server as a network service | See footnote 3 |
| server.statefulSet.enabled | Indicates if prometheus-server is deployed as a statefulSet. If set to false, prometheus-server will be deployed as a deployment |
false |
| scrapeConfigs | Specifies prometheus-server scrape jobs, keyed by job name. Each entry takes the fields of a Prometheus scrape_config
, plus enabled (defaults to true), job_name (defaults to the key), and pre_relabel_configs / post_relabel_configs, templated YAML strings whose rules are added before and after relabel_configs |
See footnote 4 |
| serverFiles.“prometheus.yml”.scrape_configs | Specifies additional prometheus-server scrape jobs, as a list of Prometheus scrape_config
entries. They are added to the scrapeConfigs jobs; a job with the same job_name as a scrapeConfigs job replaces that job. Not applied by package build 3.15.1-383aa49f… |
Not set |
| extraScrapeConfigs | Specifies additional prometheus-server scrape jobs, as a templated YAML string with a list of Prometheus scrape_config
entries. Their job names must not match any other job. Don’t set it with package build 3.15.1-383aa49f… |
"" |
| serviceAccounts.server.annotations | Specifies prometheus-server service account annotations | {} |
| serviceAccounts.server.create | Indicates if prometheus-server service account will be created | true |
| serviceAccounts.server.name | Specifies prometheus-server service account name | "" |
-
Node-exporter service is exposed as a
clusterIPwithport: 9100(controlled bynodeExporter.service.servicePortbelow) andtargetPort: 9100(controlled bynodeExporter.service.hostPortbelow) by default. Note the annotationprometheus.io/scrape: "true"is mandatory in order for node-exporter to be discovered by prometheus-server as a scrape target. See below specification for details.↩︎apiVersion: packages.eks.amazonaws.com/v1alpha1 kind: Package ... spec: config: | nodeExporter: service: annotations: prometheus.io/scrape: "true" hostPort: 9100 servicePort: 9100 type: ClusterIP -
Node-exporter pods have the following toleration by default, which allows daemonSet to be scheduled on control plane node.
↩︎apiVersion: packages.eks.amazonaws.com/v1alpha1 kind: Package ... spec: config: | nodeExporter: tolerations: # For K8 version prior to 1.24 - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" # For K8 version 1.24+ - key: "node-role.kubernetes.io/control-plane" operator: "Exists" effect: "NoSchedule" -
Prometheus-server service is exposed as a
clusterIPwithport: 9090(controlled byserver.service.servicePortbelow) andtargetPort: 9090(not overridable) by default. See below specification for details.↩︎apiVersion: packages.eks.amazonaws.com/v1alpha1 kind: Package ... spec: config: | server: service: enabled: true servicePort: 9090 type: ClusterIP -
Prometheus-server by default has the following scrape configs.
↩︎apiVersion: packages.eks.amazonaws.com/v1alpha1 kind: Package ... spec: config: | scrapeConfigs: prometheus: enabled: true job_name: "" static_configs: - targets: - localhost:9090 kubernetes-api-servers: enabled: true job_name: "" kubernetes_sd_configs: - role: endpointslice scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: - __meta_kubernetes_namespace - __meta_kubernetes_service_name - __meta_kubernetes_endpointslice_port_name action: keep regex: default;kubernetes;https pre_relabel_configs: "" post_relabel_configs: "" kubernetes-nodes: enabled: true job_name: "" scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) pre_relabel_configs: "" post_relabel_configs: "" kubernetes-nodes-cadvisor: enabled: true job_name: "" scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token metrics_path: /metrics/cadvisor kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - source_labels: [__metrics_path__] target_label: metrics_path pre_relabel_configs: "" post_relabel_configs: "" kubernetes-service-endpoints: enabled: true job_name: "" honor_labels: true kubernetes_sd_configs: - role: endpointslice relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow] action: drop regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: - __address__ - __meta_kubernetes_service_annotation_prometheus_io_port action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+) replacement: __param_$1 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: service - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: node pre_relabel_configs: "" post_relabel_configs: "" kubernetes-service-endpoints-slow: enabled: true job_name: "" honor_labels: true scrape_interval: 5m scrape_timeout: 30s kubernetes_sd_configs: - role: endpointslice relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: - __address__ - __meta_kubernetes_service_annotation_prometheus_io_port action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+) replacement: __param_$1 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: service - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: node prometheus-pushgateway: enabled: true job_name: "" honor_labels: true kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] action: keep regex: pushgateway pre_relabel_configs: "" post_relabel_configs: "" kubernetes-services: enabled: true job_name: "" honor_labels: true metrics_path: /probe params: module: [http_2xx] kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] action: keep regex: true - source_labels: [__address__] target_label: __param_target - target_label: __address__ replacement: blackbox - source_labels: [__param_target] target_label: instance - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] target_label: namespace - source_labels: [__meta_kubernetes_service_name] target_label: service pre_relabel_configs: "" post_relabel_configs: "" kubernetes-pods: enabled: true job_name: "" honor_labels: true kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow] action: drop regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme] action: replace regex: (https?) target_label: __scheme__ - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_port - __meta_kubernetes_pod_ip action: replace regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4}) replacement: '[$2]:$1' target_label: __address__ - source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_port - __meta_kubernetes_pod_ip action: replace regex: (\d+);((([0-9]+?)(\.|$)){4}) replacement: $2:$1 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod - source_labels: [__meta_kubernetes_pod_phase] regex: Pending|Succeeded|Failed|Completed action: drop - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: node pre_relabel_configs: "" post_relabel_configs: "" kubernetes-pods-slow: enabled: true job_name: "" honor_labels: true scrape_interval: 5m scrape_timeout: 30s kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_slow] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme] action: replace regex: (https?) target_label: __scheme__ - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_port - __meta_kubernetes_pod_ip action: replace regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4}) replacement: '[$2]:$1' target_label: __address__ - source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_port - __meta_kubernetes_pod_ip action: replace regex: (\d+);((([0-9]+?)(\.|$)){4}) replacement: $2:$1 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+) replacement: __param_$1 - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: pod - source_labels: [__meta_kubernetes_pod_phase] regex: Pending|Succeeded|Failed|Completed action: drop - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: node pre_relabel_configs: "" post_relabel_configs: ""