Verrazzano Enterprise Container Platform – Configure Monitoring Components

Docs: Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus is a system for monitoring cloud native applications and is used by Verrazzano to monitor applications. Prometheus is used in Verrazzano to collect system performance metrics and metrics for applications deployed or managed by Verrazzano. Prometheus analyzes the metrics and provides visualization using Grafana.

Customize Prometheus configuration

Verrazzano installs Prometheus components, including Prometheus Operator and Prometheus, using the kube-prometheus-stack Helm chart. You can customize the installation configuration using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the number of Prometheus replicas.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            prometheus:
              prometheusSpec:
                replicas: 3

For more information about setting component overrides, see Installation Overrides.

For information about all the overrides supported by the kube-prometheus-stack chart in Verrazzano, see values.yaml.

For instructions to customize persistent storage settings for Prometheus, see Customize Persistent Storage.

Configure Alertmanager

To configure Alertmanager to send alerts as SMTP notifications, complete the following steps:

Create a secret named smtp-secret in the verrazzano-monitoring namespace that contains the SMTP server credentials. For example:
```
$ kubectl -n verrazzano-monitoring create secret generic smtp-secret \
 --from-literal=username="<smtp server username>" \
 --from-literal=password="<smtp server password>"
```
Configure the Prometheus Operator component of the Verrazzano custom resource. For example:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            alertmanager:
              enabled: true
              alertmanagerSpec:
                podMetadata:
                  annotations:
                    sidecar.istio.io/inject: "false"
                secrets:
                - smtp-secret
              config:
                global:
                  smtp_auth_password_file: /etc/alertmanager/secrets/smtp-secret/password
                  smtp_auth_username: "<smtp server username>"
                  smtp_from: "<e-mail address used when sending out emails>"
                  smtp_smarthost: "<host or host:port for the smtp server>"
              receivers:
              - email_configs:
                - send_resolved: true
                  to: "<e-mail address of the receiver>"
                name: email-notifications
              route:
                group_by:
                - alertname
                receiver: email-notifications
                routes:
                - matchers:
                  - alertname =~ "InfoInhibitor|Watchdog"
                  receiver: email-notifications
```

For more information about Alertmanager configurations, see the Alertmanager Documentation.

After you have enabled Alertmanager in Step 2, you can deploy alert rules to get proactive alerts. To create a TestAlertRule, run the following command.

$ kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus-operator
  name: prometheus-operator-test
  namespace: verrazzano-monitoring
spec:
  groups:
    - name: test
      rules:
        - alert: TestAlertRule
          annotations:
            description: Test alert rule
            runbook_url: test-runbook-url
            summary: Test alert rule
          expr: |-
            prometheus_config_last_reload_successful{job="prometheus-operator-kube-p-prometheus",namespace="verrazzano-monitoring"} == 0
          for: 10m
          labels:
            severity: critical
EOF

For more information, see Deploying Prometheus Rules.

Configure data retention settings

Verrazzano configures Prometheus with a default data retention setting of 10 days. The rate of metrics data collected depends on many factors, including the number of monitors, the monitor scrape intervals, and the number of metrics returned by each monitor.

When using persistent storage for Prometheus, it is possible to consume all storage. If Prometheus uses all available persistent storage, then queries return no data and new metrics cannot be saved. You can customize the persistent storage settings, and change the data retention days and configure a maximum retention size. When configuring retention size, a good rule of thumb is to set the value to no more than 85 percent of the persistent storage size.

The following example configures Prometheus to store at most three days or 40 GB of metrics data.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            prometheus:
              prometheusSpec:
                retention: 3d
                retentionSize: 40GB

Docs: Thanos

Mon, 01 Jan 0001 00:00:00 +0000

Thanos is a group of components that seamlessly integrate with Prometheus to monitor your applications. You can enable and configure Thanos components with Verrazzano and use long-term storage to store metrics. By using the Thanos console, you can query for metrics across all Prometheus instances and long-term storage. Thanos also makes it easier to scale Prometheus horizontally and obtain a global view of data from multiple Prometheus servers.

Advantages of using Thanos:

Long-term metrics retention
High availability
Easy backup for metrics
Efficient data access

For more information on Thanos, see the Thanos website.

Thanos components

Verrazzano currently supports the following Thanos components:

Components	Description
Sidecar	Container that resides in the Prometheus pod. It connects to Prometheus, reads its data for queries, and uploads it to long-term storage.
Store Gateway	Serves metrics from long-term storage.
Query	Implements Prometheus API to aggregate data from the underlying components and provides a user interface for querying across all Prometheus stores.
Query Frontend	Implements Prometheus API and proxies it to Query while caching the response and optionally splits queries.

Verrazzano installs these Thanos components using a Helm chart. You can customize the installation configuration using Helm overrides specified in the Verrazzano custom resource.

Enable Thanos

To enable Thanos in Verrazzano, add the following:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  components:
    prometheusOperator:
      enabled: true
      overrides:
      - values:
          prometheus:
            thanos:
              integration: sidecar
    thanos:
      enabled: true

Enable long-term storage using OCI Object Storage

Optionally, you can configure Thanos to use OCI Object Storage for long-term storage of metrics.

To enable this behavior complete the following steps:

Step 1: Create a YAML configuration file

Create a local file named storage.yaml that identifies your OCI Object Storage bucket name, the region and compartment where it is located within your OCI tenancy, and a valid set of credentials for Thanos to use when accessing it.

type: OCI
config:
  provider: "raw"
  bucket: "thanos"
  compartment_ocid: "ocid1.compartment.oc1....."
  region: "us-ashburn-1"
  tenancy_ocid: "ocid1.tenancy.oc1....."
  user_ocid: "ocid1.user.oc1....."
  fingerprint: "12:d3:4c:..."
  privatekey: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----

Step 2: Create a secret

Create the secret for object storage configuration using the storage.yaml file you created in Step 1.

The Thanos Store Gateway requires the key in the secret to be objstore.yml.

$ kubectl create namespace verrazzano-monitoring
$ kubectl create secret generic objstore-config -n verrazzano-monitoring --from-file=objstore.yml=storage.yaml

Step 3: Enable storage and Thanos Store Gateway

The following example enables storage, creates the required secret, and enables Thanos Store Gateway in the Verrazzano custom resource. It also configures the Thanos Sidecar to write to object storage and the Store Gateway to read from object storage.

Note: objstore-config is the secret that you created in Step 2.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  components:
    prometheusOperator:
      enabled: true
      overrides:
      - values:
          prometheus:
            thanos:
              integration: sidecar
            prometheusSpec:
              thanos:
                objectStorageConfig:
                  name: objstore-config
                  key: objstore.yml
    thanos:
      enabled: true
      overrides:
      - values:
          existingObjstoreSecret: objstore-config
          storegateway:
            enabled: true

Access the Thanos Query console

You can access the Thanos Query console using the instructions at Get the consoles URLs.

Docs: Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Grafana provides visualization for your Prometheus metric data.

Single pod per cluster.
Named vmi-system-grafana-* in the verrazzano-system namespace.
Provides dashboards for metrics visualization.

To access Grafana:

Get the host name from the Grafana ingress.
```
$ kubectl get ingress vmi-system-grafana -n verrazzano-system

# Sample output
NAME                 CLASS    HOSTS                                              ADDRESS          PORTS     AGE
vmi-system-grafana   <none>   grafana.vmi.system.default.123.456.789.10.nip.io   123.456.789.10   80, 443   26h
```
Get the password for the user verrazzano.
```
$ kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo
```
Access Grafana in a browser using the host name.
Log in using the verrazzano user and the password.

Grafana dashboards

Verrazzano installs a number of preconfigured dashboards in Grafana. These dashboards are organized logically into folders. To view the folders, locate the Dashboards icon and select Manage.

Within each folder, you will find one or more preconfigured dashboards. The dashboards provide a visual representation of metrics for monitoring your applications, the Kubernetes clusters where your applications are running, and Verrazzano itself. You can add to these dashboards, and you can copy and modify the preconfigured dashboards.

Here are the folders and dashboard descriptions:

Coherence - A set of dashboards for monitoring Coherence applications. They provide metrics from Coherence clusters, Caches, Elastic Data, Federation Details, HTTP Servers, Kubernetes, Machines, Cluster Members, Persistence Summary, Proxy Servers, and Coherence Services.
Helidon - For Helidon workloads, provides the JVM details for Helidon applications, such as Status, Heap Usage, JVM Heap Sizes, Thread Count, HTTP Requests, and such.
Istio - A set of dashboards for monitoring the Verrazzano Istio service mesh and the workloads running in the mesh.
JVM - For monitoring applications deployed in Verrazzano.
NGINX - For monitoring the Verrazzano system ingress controller and the ingresses created using the ingress controller. Metrics include Controller Request Volume, Controller Connections, Controller Success Rate, Config Reloads, Network Pressure, and such.
OpenSearch - Provides metrics from OpenSearch clusters, such as Cluster Health, Shard Details, Index Details, JVM Metrics, and such.
Prometheus Operator - Provides metrics for the cluster compute resources, such as CPU Utilization, Memory Limits, and such.
WebLogic - For WebLogic workloads, provides WebLogic Server runtime metrics to monitor and diagnose the runtime deployment of WebLogic Server.

In addition, there are the following preconfigured Grafana Verrazzano dashboards. These dashboards are located in Verrazzano Application, Verrazzano Monitoring, and Verrazzano System folders:

Verrazzano Application contains the Application Status dashboard. This dashboard provides information on the overall health and performance of OAM applications that are deployed in Verrazzano and the details of the running pods. You can filter this information based on cluster, application, and component.
- When you deploy a new application in Verrazzano, the application is automatically added to the dashboard and becomes available in the list of Applications. You can select the specific application that you want to monitor and select which application metrics to view. Metrics are available in three rows:
  - Summary provides CPU and memory usage and pod status.
  - Storage provides disk and PVC (Persistent Volume Claims) usage.
  - Requests provides HTTP requests and request metrics.
Verrazzano Monitoring contains dashboards for monitoring the Kubernetes clusters where Verrazzano is running:
- alertmanager-mixin displays alerts created by Alertmanager, including the successful and invalid notifications sent by Alertmanager.
- etcd-mixin provides cluster metrics and shows additional information, such as active streams, RPC rate, database size, memory usage, client traffic, peer traffic, and such.
- kubernetes-mixin provides CPU, memory and network metrics based on cluster, pod, workload type, and nodes; network bandwidth based on cluster, pod, workload type, and nodes; persistent volume usage; and metrics for proxy and scheduler.
- node-exporter-mixin provides node level status for CPU, memory, disk, and network usage.

The following graphic shows some Verrazzano Monitoring dashboards:

Verrazzano System has a set of dashboards that provide information on the health of all Verrazzano system components and the resource usage of Kubernetes resources across clusters.
- Resource Usage Detailed gives a detailed view of the resource usage for each of the Verrazzano system components, such as the application operator, cluster operator, monitoring operator, and platform operator. You can filter information based on cluster, component, and so on.
- Resource Usage Summary gives a summary of the resource usage that can be viewed for all components or for a specific component. You can filter information based on cluster, component, and so on. Metrics are available in three rows:
  - Summary provides CPU and memory usage and pod status.
  - Storage provides disk and PVC (Persistent Volume Claims) usage.
  - Requests provides HTTP requests and request metrics.
- System Health gives the health for all Verrazzano system components that can be viewed for local clusters or any other registered clusters.

Dynamic dashboard discovery and portability

Verrazzano enables dynamic discovery of Grafana dashboards. Using this feature, you can package the dashboard configuration and settings as a Kubernetes ConfigMap, and then deploy it individually or along with your application.

To configure a dashboard as a ConfigMap:

Create a dashboard as a JSON file. Or, you can export an existing Grafana dashboard as a JSON file. See Manage Dashboards.
Create a ConfigMap on the same cluster running Grafana that includes:
- The label, grafana_dashboard:"1", so that Grafana discovers the dashboard.
- The JSON code that defines and configures the dashboard (the dashboard JSON from step 1).
- Optionally, a k8s-sidecar-target-directory annotation, which specifies the Grafana folder name in which you want the dashboard to appear. The value of the annotation is the Grafana folder name. If not specified, the dashboard will appear in the default Grafana folder.

Here is an example dashboard ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    k8s-sidecar-target-directory: My App Dashboards
  labels:
    grafana_dashboard: "1"
  name: app-dashboard
  namespace: app
data:
  app_dashboard.json: |-
    {
      "title": "My App Dashboard",
      "uid": "Q4Bkkx",
      "version": 2,
      "panels": [
        {
          ...
        }
      ...
    }

After the ConfigMap is created, the dashboard gets loaded in Grafana.

Alternatively, you can create a ConfigMap from a file using kubectl create configmap, or generate ConfigMap from files. See Create a ConfigMap.

You can use dashboard ConfigMaps to deploy custom dashboards; these dashboards can be modified but cannot be saved in Grafana. If you want to modify the dashboard, then either update the ConfigMap or save the dashboard to a new location from the Grafana console.