Verrazzano Enterprise Container Platform – Logging

Docs: Jaeger Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger is a distributed tracing system used for monitoring and troubleshooting microservices. Distributed tracing lets you trace errors across your microservice architecture. You can track application requests as they flow from frontend devices to backend services and databases. You use distributed tracing to troubleshoot requests that exhibit high latency or errors. For more information on Jaeger distributed tracing, see the Jaeger website.

Install Jaeger Operator

To install the Jaeger Operator, enable the jaegerOperator component in your Verrazzano custom resource. Here is an example YAML file that enables the Jaeger Operator. Verrazzano installs the Jaeger Operator in the verrazzano-monitoring namespace. If OpenSearch and Keycloak components are enabled in the Verrazzano custom resource, then a default Jaeger instance is also created by the Jaeger Operator in the verrazzano-monitoring namespace.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true

The Jaeger Operator will create Service custom resources for query and collection. After applying the Verrazzano custom resource, listing Jaeger resources will show output similar to the following.

$ kubectl get services,deployments -l app.kubernetes.io/instance=jaeger-operator-jaeger -n verrazzano-monitoring

#sample output
NAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                  AGE
service/jaeger-operator-jaeger-collector            ClusterIP   10.96.120.223   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   79m
service/jaeger-operator-jaeger-collector-headless   ClusterIP   None            <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   79m
service/jaeger-operator-jaeger-query                ClusterIP   10.96.209.196   <none>        16686/TCP,16685/TCP                      79m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-operator-jaeger-collector   1/1     1            1           79m
deployment.apps/jaeger-operator-jaeger-query       1/1     1            1           79m

Customize Jaeger

Verrazzano installs the Jaeger Operator and Jaeger using the jaeger-operator Helm chart. Using Helm overrides specified in the Verrazzano custom resource, you can customize the installation configuration. For more information about setting component overrides, see Customizing the Chart Before Installing.

The following sections describe some common customizations you might want to employ:

Customize a Jaeger instance to use an external OpenSearch for storage

You can use the default Jaeger instance with an external OpenSearch cluster. The following example shows you how to configure Jaeger Operator Helm overrides in the Verrazzano custom resource to use an external OpenSearch cluster with a TLS CA certificate mounted from a volume and the user/password stored in a secret. For more details, see the Jaeger documentation.

Prior to configuring the external OpenSearch for Jaeger in the Verrazzano custom resource, create a secret containing the OpenSearch credentials and certificates in the verrazzano-install namespace. Jaeger will use these credentials to connect to OpenSearch.
```
$ kubectl create secret generic jaeger-secret \
 --from-literal=OS_PASSWORD=<OPENSEARCH PASSWORD> \
 --from-literal=OS_USERNAME=<OPENSEARCH USERNAME> \
 --from-file=ca-bundle=<path to the file containing CA certs> \
 -n verrazzano-install
```
Use the Verrazzano custom resource to update the Jaeger resource:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger-external-opensearch
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              create: true
              spec:
                strategy: production
                storage:
                  type: opensearch
                  options:
                    es:
                      # Enter your OpenSearch cluster endpoint here.
                      server-urls: <External OpenSearch URL>
                      index-prefix: jaeger
                      tls:
                        ca: /verrazzano/certificates/ca-bundle
                  secretName: jaeger-secret
                volumeMounts:
                  - name: certificates
                    mountPath: /verrazzano/certificates/
                    readOnly: true
                volumes:
                  - name: certificates
                    secret:
                      secretName: jaeger-secret
```

Enable Jaeger Service Performance Monitoring

NOTE: The Service Performance Monitoring (SPM) feature is currently considered experimental.

To enable Service Performance Monitoring in the default Jaeger instance created by Verrazzano, use the following Verrazzano custom resource. Verrazzano sets jaeger.spec.query.options.prometheus.server-url to the Prometheus server URL managed by Verrazzano, if it exists. To configure an external Prometheus server for your use case, override jaeger.spec.query.options.prometheus.server-url, jaeger.spec.query.options.prometheus.tls.enabled, and jaeger.spec.query.options.prometheus.tls.ca, appropriately, in the Verrazzano custom resource. For more details, see the Jaeger documentation.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              spec:
                query:
                  metricsStorage:
                    type: prometheus

Disable default Jaeger instance creation

If you have a custom Jaeger instance and want to disable the default Jaeger instance created by Verrazzano, then use the following Verrazzano custom resource:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              create: false

Jaeger Operator Helm chart values that you cannot override

The following Jaeger Operator Helm values are not supported to be overridden in the Verrazzano custom resource:

nameOverride
fullnameOverride
serviceAccount.name
ingress.enabled
jaeger.spec.storage.dependencies.enabled

If you try to override these Helm values in the Verrazzano custom resource, the request will be rejected and an error message returned.

NOTE: Verrazzano does not support Jaeger Spark dependencies and hence, the Helm chart value jaeger.spec.storage.dependencies.enabled, which is set to false for the Jaeger instance managed by Verrazzano, cannot be overridden.

Configure the Istio mesh to use Jaeger tracing

You can view Istio mesh traffic by enabling Istio’s distributed tracing integration. Traces from the Istio mesh provide observability on application traffic that passes through Istio’s ingress and egress gateways.

Istio tracing is disabled by default. To turn on traces, customize your Istio component like the following example:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true

After enabling tracing, Istio will automatically configure itself with the the Jaeger instance managed by Verrazzano in your cluster, and Istio-injected pods will begin exporting traces to Jaeger.

To export traces to a different Jaeger instance than the one managed by Verrazzano, set meshConfig.defaultConfig.tracing.zipkin.address to the intended Jaeger Collector URL. Any new Istio-injected pods will begin exporting traces to the newly configured Jaeger instance. Existing pods require a restart to pull the new Istio configuration and start sending traces to the newly configured Jaeger instance.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true
                defaultConfig:
                  tracing:
                    zipkin:
                      address: <address:port of your Jaeger collector service>

Istio’s default sampling rate is 1%, meaning 1 in 100 requests will be traced in Jaeger. If you want a different sampling rate, configure your desired rate using the meshConfig.defaultConfig.tracing.sampling Istio installation argument.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true
                defaultConfig:
                  tracing:
                    sampling: 25.0

Manage Jaeger indices in OpenSearch

To clean old Jaeger data from OpenSearch, Verrazzano uses the index management provided by Jaeger. By default, a cron job with the following default values is created to clean old traces. To configure it to your use case, override the following Jaeger spec values in the Verrazzano custom resource with your desired values.

storage:
  type: opensearch
  esIndexCleaner:
    enabled: true                                 // turn the cron job deployment on and off
    numberOfDays: 7                               // number of days to wait before deleting a record
    schedule: "55 23 * * *"                       // cron expression for it to run

Configure applications to export traces to Jaeger

The Jaeger agent sidecar is injected to application pods by the "sidecar.jaegertracing.io/inject": "true" annotation. You may apply this annotation to namespaces or pod controllers, such as Deployments. The subsequent snippet shows how to annotate an OAM Component for Jaeger agent injection.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-component
spec:
  workload:
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
    metadata:
      name: example-workload
      annotations:
        # The component's Deployment will carry the Jaeger annotation.
        "sidecar.jaegertracing.io/inject": "true"

If you have multiple Jaeger instances in your cluster, specify the name of the Jaeger instance to which you intend to send the traces, as a value for the annotation sidecar.jaegertracing.io/inject. For more details, see the Jaeger documentation.

NOTE: Using the Jaeger agent is not supported in Helidon 3.x. To use Jaeger tracing, the Helidon application should connect directly to the Jaeger collector. See the following example YAML file, where "TRACING_HOST" is set to "jaeger-operator-jaeger-collector.verrazzano-monitoring" and "TRACING_PORT" to "9411". For Jaeger tracing in a multicluster Verrazzano environment, set the "TRACING_HOST" to "jaeger-verrazzano-managed-cluster-collector.verrazzano-monitoring.svc.cluster.local".

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
        version: v1
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:1.0.0-1-20220513221156-7da0d32"
              env:
                - name: "TRACING_SERVICE"
                  value: "hello-helidon"
                - name: "TRACING_PORT"
                  value: "9411"
                - name: "TRACING_HOST"
                  value: "jaeger-operator-jaeger-collector.verrazzano-monitoring"
              ports:
                - containerPort: 8080
                  name: http

View traces on the Jaeger console

After the installation has completed, you can use the Verrazzano Jaeger console to view the traces. For information on how to get the Verrazzano Jaeger console URL and credentials, see Access Verrazzano.

Use Jaeger tracing in a multicluster Verrazzano environment

If the Jaeger Operator component is enabled in the managed cluster, after successful registration with the admin cluster, a Jaeger collector service runs in the managed cluster, which exports the traces to the OpenSearch storage configured in the admin cluster.

NOTE: Traces are exported to the admin cluster only when the Jaeger instance in the admin cluster is configured with the OpenSearch storage.

Listing Jaeger resources in the managed cluster shows output similar to the following.

$ kubectl get jaegers -n verrazzano-monitoring

#sample output
NAME                                STATUS    VERSION   STRATEGY     STORAGE         AGE
jaeger-verrazzano-managed-cluster   Running   1.34.1    production   opensearch      11m

Configure the Istio mesh in a managed cluster to export Jaeger traces to the admin cluster

To export the Istio mesh traces in the managed cluster to the admin cluster, set meshConfig.defaultConfig.tracing.zipkin.address to the Jaeger Collector URL created in the managed cluster that exports the traces to the OpenSearch storage configured in the admin cluster.

Configure the Istio mesh on the managed cluster at the time of the Verrazzano installation, as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: managed-cluster
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            meshConfig:
              enableTracing: true
              defaultConfig:
                tracing:
                  zipkin:
                    address: jaeger-verrazzano-managed-cluster-collector.verrazzano-monitoring.svc.cluster.local.:9411

View the managed cluster traces

You can see the managed cluster traces from the Jaeger console in the admin cluster only. To find the Jaeger console URL for your admin cluster, follow the instructions for Accessing Verrazzano.

The spans include the Process tag verrazzano_cluster, which has the name of the managed cluster. To see the traces for the managed cluster only, search based on the tag verrazzano_cluster=<managed cluster name>.

Sample output of Jager console screens

Additional information

Read the blog, Jaeger, Fluentd, and OpenSearch with Verrazzano, for a detailed look at how these Verrazzano components work together to provide a complete observability stack.

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano logging stack consists of Fluentd, OpenSearch, and OpenSearch Dashboards components.

Fluentd: a log aggregator that collects, processes, and formats logs from Kubernetes clusters.
OpenSearch: a scalable search and analytics engine for storing Kubernetes logs.
OpenSearch Dashboards: a visualization layer that provides a user interface to query and visualize collected logs.

As shown in the following diagram, logs written to stdout by a container running on Kubernetes are picked up by the kubelet service running on that node and written to /var/log/containers.

Fluentd sidecar

For components with multiple log streams or that cannot log to stdout, Verrazzano deploys a Fluentd sidecar which parses and translates the log stream. The resulting log is sent to stdout of the sidecar container and then written to /var/log/containers by the kubelet service.

For example, in a WebLogic deployment, AdminServer.log is consumed, translated, and written to stdout by the Fluentd sidecar. You can view these logs using kubectl on the container named fluentd-stdout-sidecar.

$ kubectl logs tododomain-adminserver \
   -n todo-list \
   -c fluentd-stdout-sidecar

The Verrazzano Fluentd Docker image comes with these plug-ins:

The Verrazzano Fluentd Docker image also has two local default plug-ins, kubernetes_parser and kubernetes_multiline_parser. These plug-ins help to parse Kubernetes management log files.

Here are example use cases for these plug-ins:

# ---- fluentd.conf ----
# kubernetes parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>

# kubernetes multi-line parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>
# ----   EOF      ----

Fluentd DaemonSet

Verrazzano deploys a Fluentd DaemonSet which runs one Fluentd replica per node in the verrazzano-system namespace. Each instance pulls logs from the node’s /var/log/containers directory and writes them to the target OpenSearch data stream. Verrazzano system applications receive special handling, and write their logs to the verrazzano-system data stream. Verrazzano application logs are exported to a data stream based on the application’s namespace, following this format: verrazzano-application-<application namespace>.

For example, vmi-system-opensearchDashboards logs written to /var/log/containers will be pulled by Fluentd and written to OpenSearch. The logs are exported to the verrazzano-system data stream, because vmi-system-opensearchDashboards is a Verrazzano system application. For a non-system application, if it is in the myapp namespace, then its logs will be exported to the verrazzano-application-myapp data stream.

OpenSearch

Verrazzano creates an OpenSearch cluster as the store and search engine for the logs processed by Fluentd. Records written by Fluentd can be queried using the OpenSearch REST API.

For example, you can use curl to get all of the OpenSearch data streams. First, you must get the password for the verrazzano user and the host for the Verrazzano Monitoring Instance (VMI) OpenSearch.

$ PASS=$(kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo)
$ HOST=$(kubectl get ingress \
    -n verrazzano-system vmi-system-os-ingest \
    -o jsonpath={.spec.rules[0].host})

$ curl -ik \
   --user verrazzano:$PASS https://$HOST/_data_stream

To see all of the records for a specific data stream, do the following:

$ DATA_STREAM=verrazzano-application-todo-list

$ curl -ik \
    --user verrazzano:$PASS https://$HOST/$DATA_STREAM/_search?q=message:*

Verrazzano provides support for Installation Profiles. The production profile (prod), which is the default, provides a 3-node OpenSearch and persistent storage for the VMI. The development profile (dev) provides a single node OpenSearch and no persistent storage for the VMI. The managed-cluster profile does not install OpenSearch or OpenSearch Dashboards in the local cluster; all logs are forwarded to the admin cluster’s OpenSearch instance.

If you want the logs sent to an external OpenSearch, instead of the default VMI OpenSearch, specify opensearchURL and opensearchSecret in the Fluentd Component configuration in your Verrazzano custom resource.

The following is an example of a Verrazzano custom resource to send the logs to the OpenSearch endpoint https://external-os.default.172.18.0.231.nip.io.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: default
spec:
  components:
    fluentd:
      opensearchURL: https://external-os.default.172.18.0.231.nip.io
      opensearchSecret: external-os-secret

For information on OpenSearch, see the Customize OpenSearch.

OpenSearch Dashboards

OpenSearch Dashboards is a visualization dashboard for the content indexed on an OpenSearch cluster. Verrazzano creates a OpenSearch Dashboards deployment to provide a user interface for querying and visualizing the log data collected in OpenSearch.

To access the OpenSearch Dashboards, read Access Verrazzano.

To see the records of an OpenSearch index or data stream through OpenSearch Dashboards, create an index pattern to filter for records under the desired data stream or index.

For example, to see the log records of a WebLogic application deployed to the todo-list namespace, create an index pattern of verrazzano-application-todo-*.

Docs: Oracle Cloud Infrastructure Logging Service

Mon, 01 Jan 0001 00:00:00 +0000

The Oracle Cloud Infrastructure Logging service is a highly scalable and fully managed single view for all the logs in your tenancy. You can configure Verrazzano to send logs to Oracle Cloud Infrastructure Logging instead of OpenSearch. For general information, see Oracle Cloud Infrastructure Logging Overview.

Set up custom logs

Verrazzano can send its logs to Oracle Cloud Infrastructure custom logs. You will need to provide two Oracle Cloud Infrastructure Log identifiers in your Verrazzano installation resource: one for Verrazzano system logs and one for application logs. Follow the steps in Creating Custom Logs to create two custom logs. Do not create an agent configuration when creating a custom log, otherwise the log records will be duplicated.

Configure credentials

The Fluentd plug-in included with Verrazzano will use Oracle Cloud Infrastructure instance principal authentication by default. Optionally, you can configure Verrazzano with a user API signing key. API signing key authentication is required to send logs to Oracle Cloud Infrastructure Logging if the cluster is running outside of Oracle Cloud Infrastructure.

Instance principal authentication

Create a dynamic group that includes the compute instances in your cluster’s node pools and assign the appropriate policy, so that the dynamic group is allowed to send log entries to the custom logs you created earlier. Pay close attention to the required permissions.

If the dynamic group and policy are configured incorrectly, then Fluentd will fail to send logs to Oracle Cloud Infrastructure Logging.

User API signing key

If you do not already have an API signing key, then see Required Keys and OCIDs in the Oracle Cloud Infrastructure documentation. You need to create an Oracle Cloud Infrastructure configuration file with the credential details and then use that configuration file to create a secret.

The following requirements must be met for Fluentd Oracle Cloud Infrastructure Logging to work:

The profile name in the Oracle Cloud Infrastructure configuration file must be DEFAULT.
The key_file path in the Oracle Cloud Infrastructure configuration file must be /root/.oci/key. The actual key file does not need to be in that location, because you will be providing the actual key file location in a secret.
The user associated with the API key must have the appropriate Oracle Cloud Infrastructure Identity and Access Management (IAM) policy in place to allow the Fluentd plug-in to send logs to Oracle Cloud Infrastructure. See Details for Logging in the Oracle Cloud Infrastructure documentation for the IAM policies used by the Oracle Cloud Infrastructure Logging service.

After the Verrazzano platform operator has been installed, create an opaque secret in the verrazzano-install namespace from the Oracle Cloud Infrastructure configuration and private key files. The key for the configuration file must be config and the key for the private key file data must be key.

Here is an example kubectl command that will create the secret.

$ kubectl create secret generic oci-fluentd -n verrazzano-install \
      --from-file=config=/home/myuser/oci_config --from-file=key=/home/myuser/keys/oci_api.pem

The secret should look something like this.

apiVersion: v1
data:
  config: W0RFRkFVTFRdCnVzZXI9b2NpZDEudXN...
  key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS...
kind: Secret
metadata:
  name: oci-fluentd
  namespace: verrazzano-install
type: Opaque

For convenience, there is a helper script available here that you can point at an existing Oracle Cloud Infrastructure configuration file and it will create the secret for you. The script allows you to override the default configuration file location, profile name, and the name of the secret.

Install Verrazzano

Oracle Cloud Infrastructure Logging is enabled in your cluster when installing Verrazzano. The Verrazzano installation custom resource has fields for specifying two custom logs: one for system logs and one for application logs. Here is an example Verrazzano installation YAML file for each type of credential.

Instance principal credentials

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
    opensearch:
      enabled: false
    opensearchDashboards:
      enabled: false

User API credentials

When using user API credentials, you need to configure the name of the secret in the Verrazzano custom resource, under the Oracle Cloud Infrastructure section of the Fluentd component settings. Your YAML file should look something like this.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-oci-logging
spec:
  profile: dev
  components:
    fluentd:
      enabled: true
      oci:
        systemLogId: ocid1.log.oc1.iad.system.example
        defaultAppLogId: ocid1.log.oc1.iad.app.example
        apiSecret: oci-fluentd
    opensearch:
      enabled: false
    opensearchDashboards:
      enabled: false

The apiSecret value must match the secret you created earlier when configuring the user API credentials.

Override the default log objects

You can override the Oracle Cloud Infrastructure Log object on an individual namespace. To specify a log identifier on a namespace, add an annotation named verrazzano.io/oci-log-id to the namespace. The value of the annotation is the Oracle Cloud Infrastructure Log object identifier.

Here is an example namespace.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    verrazzano.io/oci-log-id: ocid1.log.oc1.iad.ns.app.example
  creationTimestamp: "2022-01-14T15:09:19Z"
  labels:
    istio-injection: enabled
    verrazzano-managed: "true"
  name: example
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Note that if you add and subsequently remove the annotation, then the logs will revert to the default Oracle Cloud Infrastructure Log object specified in the Verrazzano custom resource.

Search logs

To search Verrazzano logs, you can use the Oracle Cloud Infrastructure Console, Oracle Cloud Infrastructure CLI, or Oracle Cloud Infrastructure SDK.

For example, use the Oracle Cloud Infrastructure CLI to search the system logs for records emitted by the verrazzano-application-operator container:

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.container_name\" = 'verrazzano-application-operator' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

Search for all application log records in the springboot namespace:

$ oci logging-search search-logs --search-query=\
     "search \"ocid1.compartment.oc1..example/ocid1.loggroup.oc1.iad.example/ocid1.log.oc1.iad.example\" | \
     where \"data\".\"kubernetes.namespace_name\" = 'springboot' | sort by datetime desc" \
     --time-start 2021-12-07 --time-end 2021-12-17

For more information on searching logs, see the Logging Query Language Specification.

Troubleshooting

If you are not able to view Verrazzano logs in Oracle Cloud Infrastructure Logging, then check the Fluentd container logs in the cluster to see if there are errors.

$ kubectl logs -n verrazzano-system -l app=fluentd --tail=-1

If you see not authorized error messages, then there is likely a problem with the Oracle Cloud Infrastructure Dynamic Group or IAM policy that is preventing the Fluentd plug-in from communicating with the Oracle Cloud Infrastructure API.

To ensure the appropriate permissions are in place, review the Oracle Cloud Infrastructure Logging required permissions documentation.