Verrazzano Enterprise Container Platform – Monitoring

Docs: Understand Monitoring Components in Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus, Thanos, and Grafana components. Metrics sources expose system and application metrics. The Prometheus and Thanos components retrieve and store the metrics and Grafana provides dashboards to visualize them.

The following is a representation of metrics collection in a single cluster environment.

Metrics sources

The following sections describe metrics sources that Verrazzano provides for OAM and standard Kubernetes applications.

OAM

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.
```
$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list
```
Get the user name and password used to access the metrics source from the corresponding secret.
```
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo
```
Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Standard Kubernetes workloads

Verrazzano supports enabling metric sources for Kubernetes workloads deployed without OAM Components. To enable metrics for Kubernetes workloads, you must create a Service Monitor or Pod Monitor, as applicable.

Metrics Traits use Service Monitors which require Services for metrics collection.
For details on Service Monitor and Pod Monitor, refer to the Prometheus Operator documentation.

When creating the Service Monitor or Pod Monitor for your workload, include the label release, with the value prometheus-operator on the monitor resource.

Verify metrics collection

To verify that the metrics are being collected for your workload, follow these steps.

Access the Prometheus console.
From the console, use the navigation bar to access Status/Targets.
On this page, you will see a target name with this formatting: <monitor-type>/<workload-namespace>_<workload-name>_<workload-type>, where monitor-type may be serviceMonitor or podMonitor.
Copy this job name from the target labels for use in future queries.
Verify that the State of this target is UP.
Next, use the navigation bar to access the Graph.
Here, use the job name you copied to construct this expression: {job="<job_name>"}
Use the graph to run this expression and verify that you see application metrics appear.

If you are unable to verify metrics collection, you might need to manually create a Service for the workload.

For more information on Prometheus solutions, see Troubleshooting Prometheus.

Legacy workloads

Standard Kubernetes workloads that were metrics sources in earlier versions of Verrazzano (1.3.x or earlier), will continue to be supported when upgrading to later versions of Verrazzano.

For workloads that used the legacy default metrics template, Verrazzano will create a Service Monitor in the workload’s namespace to ensure that metrics continue to be scraped. You can make any ongoing changes to the metrics source configuration by editing the Service Monitor.

For workloads that used a legacy custom metrics template, Verrazzano will configure the Prometheus Operator to ensure that metrics continue to be scraped.

Metrics server

Verrazzano installs the Prometheus Operator in the verrazzano-monitoring namespace.
A single Prometheus pod is created by Prometheus Operator in the same namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Docs: Configure Monitoring Components

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Multicluster Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano federates metrics from managed clusters to the admin cluster. This ensures that metrics in managed clusters can be queried from the admin cluster.

If you enable Thanos on the managed clusters, Verrazzano installs the required set of Thanos components on the managed clusters. Then, on the admin cluster, Verrazzano automatically configures the managed cluster’s Thanos endpoint in Thanos Query. This allows you to query metrics across all clusters from Thanos Query on the admin cluster.

For Thanos to query metrics from managed clusters, you must enable Thanos on the admin cluster and managed clusters. If Thanos is disabled on either the admin or managed clusters, then Prometheus federation is used to scrape metrics from managed clusters.

When enabling long-term metric storage in object storage, it is recommended that you use a separate bucket for each managed cluster. This provides metric data isolation and improves overall query performance. For more information, see Thanos.

The following is a representation of metrics collection in a multicluster environment.

Docs: Troubleshoot Prometheus Issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes cluster monitors are in a DOWN state

When viewing targets in the Prometheus console, some Kubernetes cluster monitors may be down (kube-etcd, kube-proxy, and such). This is likely caused by the configuration of the Kubernetes cluster itself. Depending on the type of cluster, certain metrics may be disabled by default. Enabling metrics is cluster dependent; for details, refer to the documentation for your cluster type.

For example, to enable kube-proxy metrics on Kind clusters, edit the kube-proxy ConfigMap.

$ kubectl edit cm/kube-proxy -n kube-system

Replace the metricsBindAddress value with the following and save the ConfigMap.

metricsBindAddress: 0.0.0.0:10249

Then, restart the kube-proxy pods.

$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system

For more information, see this GitHub issue.

Metrics Trait Service Monitor not discovered

Metrics Traits use Service Monitors which require a Service to collect metrics. If your OAM workload is created with a Metrics Trait and no Ingress Trait, a Service might not be generated for your workload and will need to be created manually.

This troubleshooting example uses the hello-helidon application.

Verify a Service Monitor exists for your application workload.

$ kubectl get servicemonitors -n hello-helidon

Verify a Service exists for your application workload.

$ kubectl get services -n hello-helidon

If no Service exists, create one manually. This example uses the default Prometheus port.

apiVersion: v1
kind: Service
metadata:
  name: hello-helidon-service
  namespace: hello-helidon
spec:
  selector:
    app: hello-helidon
  ports:
    - name: tcp-hello-helidon
      port: 8080
      protocol: TCP
      targetPort: 8080

After you’ve completed these steps, you can verify metrics collection has succeeded.

Metrics queries no longer return metrics

If Prometheus storage reaches capacity, then metrics queries will no longer return results. Check the Prometheus logs.

$ kubectl logs -l app.kubernetes.io/instance=prometheus-operator-kube-p-prometheus -n verrazzano-monitoring

If there are messages indicating that the disk is full, then it will be necessary to either expand the storage or free disk space. If the default storage class supports volume expansion, then you can attempt to expand the volume.

Check if the default storage class allows volume expansion.

$ kubectl get storageclass

If the default storage class allows expansion, then modify the persistent volume claim and the Prometheus resource storage request to use the larger size.

For example, to increase the storage to 100 Gi:

$ kubectl patch pvc prometheus-prometheus-operator-kube-p-prometheus-db-prometheus-prometheus-operator-kube-p-prometheus-0 -n verrazzano-monitoring \
   --type=merge -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

$ kubectl patch prometheus prometheus-operator-kube-p-prometheus -n verrazzano-monitoring \
   --type=merge -p '{"spec":{"storage":{"volumeClaimTemplate":{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}}}}'

Alternatively, delete existing metrics data in the Prometheus pods to free space.

$ kubectl exec statefulset.apps/prometheus-prometheus-operator-kube-p-prometheus -n verrazzano-monitoring -- rm -fr /prometheus/wal

$ kubectl rollout restart statefulset.apps/prometheus-prometheus-operator-kube-p-prometheus -n verrazzano-monitoring

For information on how to configure Prometheus data retention settings to avoid filling up persistent storage in the Prometheus pods, see Configure data retention settings.