Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Developing and deploying an application to Verrazzano consists of:

Packaging the application as a Docker image.
Publishing the application’s Docker image to a container registry.
Applying the application’s Verrazzano components to the cluster.
Applying the application’s Verrazzano applications to the cluster.

This guide does not provide the full details for the first two steps. An existing example application Docker image has been packaged and published for use.

Verrazzano supports application definition using Open Application Model (OAM). Verrrazzano applications are composed of components and application configurations. This document demonstrates creating OAM resources that define an application as well as the steps required to deploy those resources.

What you need

About 10 minutes.
Access to an existing Kubernetes cluster with Verrazzano installed.
Access to the application’s image in GitHub Container Registry.

Confirm access using this command to pull the example’s Docker image:
```
$ docker pull ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.12-1-20210218160249-d8db8f3
```

Application development

This guide uses an example application which was written with Java and Helidon. For the implementation details, see the Helidon MP tutorial. See the application source code in the Verrazzano examples repository.

The example application is a JAX-RS service and implements the following REST endpoints:

/greet - Returns a default greeting message that is stored in memory. This endpoint accepts the GET HTTP request method.
/greet/{name} - Returns a greeting message including the name provided in the path parameter. This endpoint accepts the GET HTTP request method.
/greet/greeting - Changes the greeting message to be used in future calls to the other endpoints. This endpoint accepts the PUT HTTP request method and a JSON payload.

The following code shows a portion of the application’s implementation. The Verrazzano examples repository contains the complete implementation. An important detail here is that the application contains a single resource exposed on path /greet.

package io.helidon.examples.quickstart.mp;
...
@Path("/greet")
@RequestScoped
public class GreetResource {

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getDefaultMessage() {
        ...
    }

    @Path("/{name}")
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getMessage(@PathParam("name") String name) {
        ...
    }

    @Path("/greeting")
    @PUT
    @Consumes(MediaType.APPLICATION_JSON)
    ...
    public Response updateGreeting(JsonObject jsonObject) {
        ...
    }

}

A Dockerfile is used to package the completed application JAR file into a Docker image. The following code shows a portion of the Dockerfile. The Verrazzano examples repository contains the complete Dockerfile. Note that the Docker container exposes a single port 8080.

FROM ghcr.io/oracle/oraclelinux:7-slim
...
CMD java -cp /app/helidon-quickstart-mp.jar:/app/* io.helidon.examples.quickstart.mp.Main
EXPOSE 8080

Application deployment

When you deploy applications with Verrazzano, the platform sets up connections, network policies, and ingresses in the service mesh, and wires up a monitoring stack to capture the metrics, logs, and traces. Verrazzano employs OAM Components to define the functional units of a system that are then assembled and configured by defining associated application configurations.

Verrazzano components

A Verrazzano OAM Component is a Kubernetes Custom Resource describing an application’s general composition and environment requirements. The following code shows the component for the example application used in this guide. This resource describes a component which is implemented by a single Docker image containing a Helidon application exposing a single endpoint.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
  namespace: hello-helidon
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:0.1.10-3-20201016220428-56fb4d4"
              ports:
                - containerPort: 8080
                  name: http

A brief description of each field of the component:

apiVersion - Version of the component custom resource definition
kind - Standard name of the component custom resource definition
metadata.name - The name used to create the component’s custom resource
metadata.namespace - The namespace used to create this component’s custom resource
spec.workload.kind - VerrazzanoHelidonWorkload defines a stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.metadata.name - The name used to create the stateless workload of Kubernetes
spec.workload.spec.deploymentTemplate.podSpec.containers - The implementation containers
spec.workload.spec.deploymentTemplate.podSpec.containers.ports - Ports exposed by the container

Verrazzano application configurations

A Verrazzano application configuration is a Kubernetes Custom Resource which provides environment specific customizations. The following code shows the application configuration for the example used in this guide. This resource specifies the deployment of the application to the hello-helidon namespace. Additional runtime features are specified using traits, or runtime overlays that augment the workload. For example, the ingress trait specifies the ingress host and path, while the metrics trait optionally provides the Prometheus scraper used to obtain the application related metrics. If no metrics trait is specified, the Verrazzano-supplied Prometheus component is used by default.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
metadata:
  name: hello-helidon-appconf
  namespace: hello-helidon
  annotations:
    version: v1.0.0
    description: "Hello Helidon application"
spec:
  components:
    - componentName: hello-helidon-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: MetricsTrait
            spec:
              scraper: <optionally specify custom scraper>
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            metadata:
              name: hello-helidon-ingress
            spec:
              rules:
                - paths:
                    - path: "/greet"
                      pathType: Prefix

A brief description of each field in the application configuration:

apiVersion - Version of the ApplicationConfiguration custom resource definition
kind - Standard name of the application configuration custom resource definition
metadata.name - The name used to create this application configuration resource
metadata.namespace - The namespace used for this application configuration custom resource
spec.components - Reference to the application’s components leveraged to specify runtime configuration
spec.components[].traits - The traits specified for the application’s components

To explore traits, we can examine the fields of an ingress trait:

apiVersion - Version of the OAM trait custom resource definition
kind - IngressTrait is the name of the OAM application ingress trait custom resource definition
spec.rules.paths - The context paths for accessing the application

Deploy the application

The following steps are required to deploy the example application. Steps similar to the apply steps would be used to deploy any application to Verrazzano.

Create a namespace for the example application and add labels identifying the namespace as managed by Verrazzano and enabled for Istio.
```
$ kubectl create namespace hello-helidon
$ kubectl label namespace hello-helidon verrazzano-managed=true istio-injection=enabled
```
Apply the application’s component.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/hello-helidon/hello-helidon-comp.yaml -n hello-helidon
```
This step causes the validation and creation of the Component resource. No other resources or objects are created as a result. Application configurations applied in the future may reference this Component resource.
Apply the application configuration.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/hello-helidon/hello-helidon-app.yaml -n hello-helidon
```
This step causes the validation and creation of the application configuration resource. This operation triggers the activation of a number of Verrazzano operators. These operators create other Kubernetes objects (for example, Deployments, ReplicaSets, Pods, Services, Ingresses) that collectively provide and support the application.
Configure the application’s DNS resolution.

After deploying the application, configure DNS to resolve the application’s ingress DNS name to the application’s load balancer IP address. The generated host name is obtained by querying Kubernetes for the gateway:
```
$ kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-gw \
    -n hello-helidon \
    -o jsonpath='{.spec.servers[0].hosts[0]}'
```
The load balancer IP is obtained by querying Kubernetes for the Istio ingress gateway status:
```
$ kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```
DNS configuration steps are outside the scope of this guide. For DNS infrastructure that can be configured and used, see the Oracle Cloud Infrastructure DNS documentation. In some small non-production scenarios, DNS configuration using /etc/hosts or an equivalent may be sufficient.

Verify the deployment

Applying the application configuration initiates the creation of several Kubernetes objects. Actual creation and initialization of these objects occurs asynchronously. The following steps provide commands for determining when these objects are ready for use.

NOTE: Many other Kubernetes objects unrelated to the example application may also exist. Those have been omitted from the lists.

Verify the Helidon application pod is running.
```
$ kubectl get pods -n hello-helidon -l app=hello-helidon

# Sample output
NAME                                        READY   STATUS    RESTARTS   AGE
hello-helidon-deployment-8664954995-wcb9d   2/2     Running   0          5m5s
```
Verify that the Verrazzano application operator pod is running.
```
$ kubectl get pod -n verrazzano-system -l app=verrazzano-application-operator

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
verrazzano-application-operator-79849b89ff-lr9w6   1/1     Running   0          13m
```
The namespace verrazzano-system is used by Verrazzano for non-application objects managed by Verrazzano. A single verrazzano-application-operator manages the life cycle of all OAM based applications within the cluster.
Verify the Verrazzano monitoring infrastructure is running.
```
$ kubectl get pods -n verrazzano-system | grep '^NAME\|vmi-system'

# Sample output
NAME                                               READY   STATUS    RESTARTS   AGE
vmi-system-os-master-0                             2/2     Running   0          47m
vmi-system-grafana-799d79648d-wsdp4                2/2     Running   0          47m
vmi-system-kiali-574c6dd94d-f49jv                  2/2     Running   0          51m
vmi-system-opensearchDashboards-77f8d998f4-zzvqr   2/2     Running   0          47m
```
```
$ kubectl get pods -n verrazzano-monitoring

# Sample output
NAME                                                   READY   STATUS    RESTARTS   AGE
prometheus-node-exporter-fstc7                         1/1     Running   0          14h
prometheus-operator-kube-p-operator-857fb66b74-szv4h   1/1     Running   0          14h
prometheus-prometheus-operator-kube-p-prometheus-0     3/3     Running   0          14h
```
These pods in the verrazzano-system and verrazzano-monitoring namespaces constitute a monitoring stack created by Verrazzano for the deployed applications.

The monitoring infrastructure comprises several components:
- vmi-system-os - OpenSearch for log collection
- vmi-system-grafana - Grafana for metric visualization
- vms-system-kiali - Kiali for management console of istio service mesh
- vmi-system-opensearchDashboards - OpenSearch Dashboards for log visualization
- prometheus-prometheus-operator-kube-p-prometheus - Prometheus for metric collection
Diagnose failures.

View the event logs of any pod not entering the Running state within a reasonable length of time, such as five minutes.
```
$ kubectl describe pod -n hello-helidon -l app=hello-helidon
```
Use the specific namespace and name for the pod being investigated.

Explore the application

Follow these steps to explore the application’s functionality. If DNS was not configured, then use the alternative commands.

Save the host name and IP address of the load balancer exposing the application’s REST service endpoints for later.
```
 $ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-gw \
       -n hello-helidon \
       -o jsonpath='{.spec.servers[0].hosts[0]}')
 $ ADDRESS=$(kubectl get service \
       -n istio-system istio-ingressgateway \
       -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
```
NOTE:
- The value of ADDRESS is used only if DNS has not been configured.
- The following alternative commands may not work in conjunction with firewalls that validate HTTP Host headers.
Get the default message.
```
  $ curl -sk \
      -X GET \
      "https://${HOST}/greet"

  # Expected response
  {"message":"Hello World!"}
```
If DNS has not been configured, then use this command.

  $ curl -sk \
      -X GET \
      "https://${HOST}/greet" \
      --resolve ${HOST}:443:${ADDRESS}

Get a message for Robert.
```
  $ curl -sk \
      -X GET \
      "https://${HOST}/greet/Robert"

  # Expected response
  {"message":"Hello Robert!"}
  ```
  If DNS has not been configured, then use this command.
  ```
  $ curl -sk \
      -X GET
      "https://${HOST}/greet/Robert" \
      --resolve ${HOST}:443:${ADDRESS}
```
Update the default greeting.
```
  $ curl -sk \
      -X PUT \
      "https://${HOST}/greet/greeting" \
      -H 'Content-Type: application/json' \
      -d '{"greeting" : "Greetings"}'
```
If DNS has not been configured, then use this command.
```
  $ curl -sk \
      -X PUT \
      "https://${HOST}/greet/greeting" \
      -H 'Content-Type: application/json' \
      -d '{"greeting" : "Greetings"}' \
      --resolve ${HOST}:443:${ADDRESS}
```
Get the new message for Robert.
```
  $ curl -sk \
      -X GET \
      "https://${HOST}/greet/Robert"

  # Expected response
  {"message":"Greetings Robert!"}
```
If DNS has not been configured, then use this command.
```
  $ curl -sk \
      -X GET \
      "https://${HOST}/greet/Robert" \
      --resolve ${HOST}:443:${ADDRESS}
```

Access the application’s logs

Deployed applications have log collection enabled. These logs are collected using OpenSearch and can be accessed using OpenSearch Dashboards. OpenSearch and OpenSearch Dashboards are examples of infrastructure Verrazzano creates in support of an application as a result of applying an application configuration. For more information on creating an index pattern and visualizing the log data collected in OpenSearch, see OpenSearch Dashboards.

Determine the URL to access OpenSearch Dashboards:

$ OSD_HOST=$(kubectl get ingress \
     -n verrazzano-system vmi-system-opensearchDashboards \
     -o jsonpath='{.spec.rules[0].host}')
$ OSD_URL="https://${OSD_HOST}"
$ echo "${OSD_URL}"
$ open "${OSD_URL}"

The user name to access OpenSearch Dashboards defaults to verrazzano during the Verrazzano installation.

Determine the password to access OpenSearch Dashboards:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Access the application’s metrics

Deployed applications have metric collection enabled. Grafana can be used to access these metrics collected by Prometheus. Prometheus and Grafana are additional components Verrazzano creates as a result of applying an application configuration. For more information on visualizing Prometheus metrics data, see Grafana.

Determine the URL to access Grafana:

$ GRAFANA_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-grafana \
      -o jsonpath='{.spec.rules[0].host}')
$ GRAFANA_URL="https://${GRAFANA_HOST}"
$ echo "${GRAFANA_URL}"
$ open "${GRAFANA_URL}"

The user name to access Grafana is set to the default value verrazzano during the Verrazzano installation.

Determine the password to access Grafana:

$ echo $(kubectl get secret \
      -n verrazzano-system verrazzano \
      -o jsonpath={.data.password} | base64 \
      --decode)

Alternatively, metrics can be accessed directly using Prometheus. Determine the URL for this access:

$ PROMETHEUS_HOST=$(kubectl get ingress \
      -n verrazzano-system vmi-system-prometheus \
      -o jsonpath='{.spec.rules[0].host}')
$ PROMETHEUS_URL="https://${PROMETHEUS_HOST}"
$ echo "${PROMETHEUS_URL}"
$ open "${PROMETHEUS_URL}"

The user name and password for both Prometheus and Grafana are the same.

Suppress Kiali console warnings

For some applications, the Kiali console may show warnings for VirtualService and Gateway objects that replicate hostname/port configurations across multiple IngressTraits. These warnings do not impact functionality and can be suppressed with the following component override:

kiali:
  overrides:
    - values:
        kiali_feature_flags:
          validations:
            ignore: ["KIA1106", "KIA0301"]

Remove the application

Run the following commands to delete the application configuration, and optionally the component and namespace.

Delete the application configuration.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/hello-helidon/hello-helidon-app.yaml
```
The deletion of the application configuration will result in the destruction of all application-specific Kubernetes objects.
(Optional) Delete the application’s component.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/hello-helidon/hello-helidon-comp.yaml
```
NOTE: This step is not required if other application configurations for this component will be applied in the future.
(Optional) Delete the namespace.
```
$ kubectl delete namespace hello-helidon
```

Docs: Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

During application deployment, the oam-kubernetes-runtime and verrazzano-application-operator cooperate through the generation and update of Kubernetes resources. The oam-kubernetes-runtime processes the ApplicationConfiguration and Component resources provided by the user and generates workload and Trait resources. The verrazzano-application-operator processes Verrazzano specific workload and Trait resources. These are then used to generate additional child and related resources.

Troubleshooting application deployments should follow three general steps:

Review the status of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the logs of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the resources generated by the oam-kubernetes-runtime and the verrazzano-application-operator.

Review oam-kubernetes-runtime operator status

For application deployment to succeed, the oam-kubernetes-runtime pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

If the pod status is not Running, then see the instructions for reviewing the oam-kubernetes-runtime pod logs.

Review verrazzano-application-operator operator status

For application deployment to succeed, the verrazzano-application-operator pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If the pod status is not Running, then see the instructions for reviewing the verrazzano-application-operator logs.

Review oam-kubernetes-runtime operator logs

Review the oam-kubernetes-runtime pod logs for any indication that pod startup or the generation of workloads or traits has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

Review verrazzano-application-operator logs

Review the verrazzano-application-operator logs for any indication that pod startup or resource generation has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

Review generated workload resources

The processing of a Component reference within an ApplicationConfiguration results in the generation of workloads. For example, a referenced Component might result in the generation of a VerrazzanoHelidonWorkload workload resource. In turn, the VerrazzanoHelidonWorkload workload resource will be processed and result in the generation of related Deployment and Service resources.

If the expected workload resource, for example VerrazzanoHelidonWorkload, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Deployment or Service, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to a VerrazzanoHelidonWorkload deployment:

$ kubectl get -n hello-helidon verrazzanohelidonworkload hello-helidon-workload
$ kubectl get -n hello-helidon deployment hello-helidon-deployment
$ kubectl get -n hello-helidon service hello-helidon-deployment

Review generated Trait resources

The processing of traits embedded with an ApplicationConfiguration results in the generation of Trait resources. For example, an IngressTrait embedded within an ApplicationConfiguration will result in the generation of an IngressTrait resource. In turn, the IngressTrait resource will be processed and result in the generation of related Certificate, Gateway, and VirtualService resources.

If the expected Trait resource, for example IngressTrait, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Certificate, Gateway, and VirtualService, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to an IngressTrait:

$ kubectl get -n hello-helidon ingresstrait hello-helidon-ingress
$ kubectl get -n istio-system Certificate hello-helidon-hello-helidon-appconf-cert
$ kubectl get -n hello-helidon gateway hello-helidon-hello-helidon-gw
$ kubectl get -n hello-helidon virtualservice hello-helidon-ingress-rule-0-vs

Check for RBAC privilege issues

The use of generic Kubernetes resources as workloads and traits can result in deployment failures if privileges are insufficient. In this case, the oam-kubernetes-runtime logs will contain errors containing the term forbidden.

The following command shows how to query for this type of failure message:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime | grep forbidden

Check resource owners

Kubernetes maintains the child to parent relationship within metadata fields.

The following example returns the parent of the IngressTrait, named hello-helidon-ingress, in the hello-helidon namespace:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .metadata.ownerReferences[*]}{.name}{"\n"}{end}'

The results of this command can help identify the lineage of a given resource.

Some resources also record the related resources affected during their processing. For example, when processed, an IngressTrait will create related Gateway, VirtualService, and Certificate resources.

The following command is an example of how to obtain the related resources of an IngressTraits:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .status.resources[*]}{.kind}: {.name}{"\n"}{end}'

The results of this command can help identify which other resources, the given resource affected.

Docs: Application Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides the following support.

Keycloak

Applications can use the Verrazzano Keycloak server as an Identity Provider. Keycloak supports SAML 2.0 and OpenID Connect (OIDC) authentication and authorization flows. Verrazzano does not provide any explicit integrations for applications.

NOTE

If using Keycloak for application authentication and authorization, create a new realm to contain application users and clients. Do not use the verrazzano-system realm, or the default (Keycloak system) realm. The Keycloak root user account (keycloakadmin) has privileges to create realms.

Network security

Verrazzano uses Istio to authenticate and authorize incoming network connections for applications. Verrazzano also provides support for configuring Kubernetes NetworkPolicy on Verrazzano projects. NetworkPolicy rules control where network connections can be made.

NOTE

Enforcement of NetworkPolicy requires that a Kubernetes Container Network Interface (CNI) provider, such as Calico, be configured for the cluster.

For more information on how Verrazzano secures network traffic, see Network Security.

Pod security

By default, all containers within a pod run as root (UID 0) within the container. Most applications do not require this level of access and doing so is considered a security risk.

It is recommended that applications attempt to meet the requirements of the Kubernetes restricted Pod Security Standard. This essentially means running the container within a pod as a non-root user with minimal capabilities, and without the ability to escalate privileges. Each container image also should define a non-root user identity that the container process will run, as by default, for added security.

In the Kubernetes Pod specification, there is a Pod SecurityContext for defining security at the pod level and a Container SecurityContext used to define security for containers. Some fields are common between the two security contexts, and others are unique. For details, see the API specifications for each. Where there is overlap, settings defined at the container level override settings defined at the pod level.

The following sections describe implementing these standards in more detail.

Specify a non-root user in the container image

Unless otherwise specified, all containers run as the root user. It is recommended that each container image build explicitly creates an unprivileged, non-root user and group, and then uses that with the USER instruction in the Dockerfile for the container.

To achieve this, modify the container’s image build and use the USER <UID> instruction.

For example:

# Run as user 1000
USER 1000

This will make the process within the container run as UID 1000. Even if there is no entry in /etc/passwd matching the UID declared, the container will run as the specified UID with minimal privileges.

For example, this is illustrated by a running image using the kubectl run command with the defaults:

% kubectl run -it --rm myol --image=ghcr.io/oracle/oraclelinux:7-slim --restart=Never -- bash
If you don't see a command prompt, try pressing enter.
bash-4.2# whoami
root
bash-4.2# id
uid=0(root) gid=0(root) groups=0(root)
bash-4.2#

To run the same image as a non-root user, you can override the default user and group for the container process, as shown:

% kubectl run -it --rm myol --image=ghcr.io/oracle/oraclelinux:7-slim --restart=Never --overrides='{ "spec": { "securityContext": { "runAsUser": 1000, "runAsGroup": 1000, "runAsNonRoot": true } } }' -- bash
If you don't see a command prompt, try pressing enter.
bash-4.2$
bash-4.2$ whoami
whoami: cannot find name for user ID 1000
bash-4.2$ id
uid=1000 gid=1000 groups=1000

In the second example, the container is running as UID 1000 with a GID of 1000. Running whoami from within the container returns an error because USER 1000 is not defined in /etc/passwd, but running the id command from the shell shows that the container process is indeed running as the desired UID (1000).

Specify security settings for the Pod

By default, containers within Kubernetes pods run as the image default user, which in turn defaults to the root user (UID 0).

You can use the pod and container securityContext fields to force containers within a pod to run as non-root and prevent the container from acquiring escalated privileges. These will override any USER setting within the image.

apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
    ...
    spec:
      # Define a security context for all containers in the pod
      securityContext:
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: some-container
        ...
        # Define a security context for the container; settings defined here have precedence over the pod securityContext
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
      ...

As mentioned previously, where there is overlap between the pod and container security settings, the settings defined at the container level override settings defined at the pod level.

Helidon pod security

The following YAML shows how to explicitly specify the pod security context for a Helidon application. With these settings, the Helidon application will meet the requirements of the Kubernetes restricted Pod Security Standard.

Note that the runAsUser 2000 UID does not exist in the container, as described previously.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
        version: v1
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          securityContext:
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: hello-helidon-container
...
              securityContext:
                runAsNonRoot: true
                runAsGroup: 2000
                runAsUser: 2000
                privileged: false
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL

Pod security for ContainerizedWorkload applications

The only means for controlling pod security for the ContainerizedWorkload type is to specify a non-root user, using the USER instruction in the container image build, as described in this section, Specify a non-root user in the container image.

Pod security for applications using standard Kubernetes resources

You can deploy applications using standard Kubernetes resources, as described in the Standard Kubernetes Resources example.

You configure security for these resources as you typically would for any Kubernetes Deployment resource.

For example:

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-deployment
spec:
  workload:
    kind: Deployment
    apiVersion: apps/v1
    name: oam-kube-dep
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: oam-kube-app
      template:
        metadata:
          labels:
            app: oam-kube-app
        spec:
          securityContext:
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: oam-kube-cnt
              image: hashicorp/http-echo
              args:
                - "-text=hello"
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false

Docs: Argo CD

Mon, 01 Jan 0001 00:00:00 +0000

Argo CD is a Kubernetes deployment tool that uses Git repositories as the source of truth. It monitors running applications and compares the deployed state against the desired one in Git. Argo CD lets you visualize the differences and provides methods to automatically or manually update the live state with the desired target state. For more information, see the Argo CD documentation.

In a multicluster Verrazzano environment, Argo CD integration depends on Rancher being enabled on the admin cluster. Argo CD connects to managed clusters using the Rancher proxy, to create the required resources for Argo CD cluster registration.

Before you begin

Install Verrazzano and set up a multicluster environment by following these instructions.
- Because Argo CD is not enabled by default, you must first enable argoCD on the admin cluster.
- When you register managed clusters, they are automatically registered in Argo CD.
Access the Argo CD console using the instructions at Access Verrazzano.
After you set up your application in the Argo CD console, those registered clusters will be available for you to select, deploy, and manage applications.

Configure repositories

In the Argo CD console, configure repositories that will contain the Kubernetes resources for deploying an application.

The following is a sample procedure to configure a private Git repository through HTTPS.

Log in to the Argo CD console.
In the left navigation, click Settings.
Click Repositories.
Click Connect Repo.
Select VIA HTTPS as the connection method.
For Project, specify default.
NOTE: Unless they are grouped together, all the projects are defined in the default level.
For Repository URL, provide the required URL.
If it is a private repository and a user name and password is required to connect to the repository, enter the required credentials.
NOTE: The other fields are optional and based on how the Git repository is configured.
Click Connect and verify that the connection status displayed is Successful.

Deploy applications

To deploy applications in a custom namespace, create Argo CD applications that specify the Git repository path, which Argo CD requires to synchronize and deploy the applications in the specified namespace.

NOTE: You can either pre-create a namespace and label it or auto-create a namespace when deploying an application.

This example provides information about how to deploy the Hello-helidon application. The Hello-helidon application and component YAML files are available at Hello World Helidon.

Log in to the Argo CD console.
Click New App.
Specify a name for the application.
For Project Name, select default.
Select the required Sync Policy option:
- Automatic - By default, every three minutes, Argo CD checks the specified Git repository and synchronizes the updates in Kubernetes to the Git repository.
- Manual - For manually synchronizing the updates to the Git repository, use the Manual option.
If you want to auto-create a namespace, in the Sync Options section, select Auto-Create Namespace.
By auto-creating the namespace, the application will be deployed outside of the service mesh. The best practice is to pre-create the namespace and label it.
Under the Source section, enter the following:
- Repository URL: https://github.com/verrazzano/verrazzano/
- Revision: master
- Path: Path in the repository where the Kubernetes resource definitions are listed. For example: examples/helidon-config
Under the Destination section, do the following:
- Cluster URL: Select the cluster to which you want to deploy the applications.
- Namespace: Specify the namespace in which you want to deploy the applications. The instructions in this sample use hello-helidon as the namespace.
Click Create.
If you selected Manual as the Sync Policy option, then click Sync. In the Synchronize Resources section, select all the resources, and then click Synchronize.
This creates the Argo CD application and a pictorial representation of the deployed applications is displayed.

Verify the deployed application

The Hello World Helidon microservices application implements a REST API endpoint, /config, which returns a message {"message":"Hello World!"} when invoked.

NOTE: The following instructions assume that you are using a Kubernetes environment such as OKE. Other environments or deployments may require alternative mechanisms for retrieving addresses, ports, and such.

Follow these steps to test the endpoints.

Get the generated host name for the application.

$ HOST=$(kubectl get gateways.networking.istio.io hello-helidon-helidon-config-appconf-gw \
     -n hello-helidon \
     -o jsonpath='{.spec.servers[0].hosts[0]}')
$ echo $HOST

# Sample output
helidon-config-appconf.hello-helidon.11.22.33.44.nip.io

Get the EXTERNAL_IP address of the istio-ingressgateway service.

$ ADDRESS=$(kubectl get service \
     -n istio-system istio-ingressgateway \
     -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $ADDRESS

# Sample output
11.22.33.44

Access the application.
- Using the command line
```
$ curl -sk \
   -X GET \
   https://${HOST}/config \
   --resolve ${HOST}:443:${ADDRESS}

# Expected response output
{"message":"Hello World!"}
```
  If you are using nip.io, then you do not need to include --resolve.
- Local testing with a browser
  
  Temporarily, modify the /etc/hosts file (on Mac or Linux) or c:\Windows\System32\Drivers\etc\hosts file (on Windows 10), to add an entry mapping the host name to the ingress gateway’s EXTERNAL-IP address. For example:
```
11.22.33.44 hello-helidon.example.com
```
  Then you can access the application in a browser at https://<host>/config.
  - If you are using nip.io, then you can access the application in a browser using the HOST variable (for example, https://${HOST}/config).
  - If you are going through a proxy, then you may need to add *.nip.io to the NO_PROXY list.
- Using your own DNS name
  
  Point your own DNS name to the ingress gateway’s EXTERNAL-IP address.
  - In this case, you would need to edit the hello-helidon-app.yaml file to use the appropriate value under the hosts section (such as yourhost.your.domain), before deploying the hello-helidon application.
  - Then, you can use a browser to access the application at https://<yourhost.your.domain>/config.
A variety of endpoints associated with the deployed application are available to further explore the logs, metrics, and such. You can access them according to the directions here.

Undeploy applications

Log in to the Argo CD console.
Select the application that you want to undeploy and then click Delete.
Enter the name of the application and then click OK.
This deletes all the resources created by the specific application except for the namespaces.

Docs: AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

Path to Field	Description
`spec.components.authProxy.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.authProxy.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes. spec: components: authProxy: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - verrazzano-authproxy topologyKey: kubernetes.io/hostname

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authProxy:
      overrides:
      - values:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

Docs: Bob's Books

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install Verrazzano by following the installation instructions.
To download the example image, you must first accept the license agreement.
- In a browser, navigate to https://container-registry.oracle.com/ and sign in.
- Search for example-bobbys-coherence, example-bobbys-front-end, example-bobs-books-order-manager, example-roberts-coherence, and weblogic.
- For each one:
  - Select the image name in the results.
  - From the drop-down menu, select your language and click Continue.
  - Then read and accept the license agreement.
NOTE: The Bob’s Books example application deployment files are contained in the Verrazzano project located at <VERRAZZANO_HOME>/examples/bobs-books, where <VERRAZZANO_HOME> is the root of the Verrazzano project.

Overview

Bob’s Books consists of three main parts:

A back-end “order processing” application, which is a Java EE application with REST services and a very simple JSP UI, which stores data in a MySQL database. This application runs on WebLogic Server.
A front-end web store “Robert’s Books”, which is a general book seller. This is implemented as a Helidon microservice, which gets book data from Coherence, uses a Coherence cache store to persist data for the order manager, and has a React web UI.
A front-end web store “Bobby’s Books”, which is a specialty children’s book store. This is implemented as a Helidon microservice, which gets book data from a (different) Coherence cache store, interfaces directly with the order manager, and has a JSF web UI running on WebLogic Server.

For more information and the source code of this application, see the Verrazzano Examples.

Deploy the application

NOTE

To run this application in the default namespace:

$ kubectl label namespace default verrazzano-managed=true istio-injection=enabled

If you chose the default namespace, you can skip Step 1 and ignore the -n option in the rest of the commands.

Create a namespace for the example and add a label identifying the namespace as managed by Verrazzano.
```
 $ kubectl create namespace bobs-books
 $ kubectl label namespace bobs-books verrazzano-managed=true istio-injection=enabled
```
Create a docker-registry secret to enable pulling the example image from the registry.
```
$ kubectl create secret docker-registry bobs-books-repo-credentials \
        --docker-server=container-registry.oracle.com \
        --docker-username=YOUR_REGISTRY_USERNAME \
        --docker-password=YOUR_REGISTRY_PASSWORD \
        --docker-email=YOUR_REGISTRY_EMAIL \
        -n bobs-books
```
Replace YOUR_REGISTRY_USERNAME, YOUR_REGISTRY_PASSWORD, and YOUR_REGISTRY_EMAIL with the values you use to access the registry.
Create secrets for the WebLogic domains:
```
  # Replace the values of the WLS_USERNAME and WLS_PASSWORD environment variables as appropriate.
  $ export WLS_USERNAME=<username>
  $ export WLS_PASSWORD=<password>
  $ kubectl create secret generic bobbys-front-end-weblogic-credentials \
      --from-literal=password=$WLS_PASSWORD \
      --from-literal=username=$WLS_USERNAME \
      -n bobs-books

  $ kubectl create secret generic bobs-bookstore-weblogic-credentials \
      --from-literal=password=$WLS_PASSWORD \
      --from-literal=username=$WLS_USERNAME \
      -n bobs-books

  $ kubectl create secret generic mysql-credentials \
      --from-literal=username=$WLS_USERNAME \
      --from-literal=password=$WLS_PASSWORD \
      --from-literal=url=jdbc:mysql://mysql.bobs-books.svc.cluster.local:3306/books \
      -n bobs-books
```
Note that the example application is preconfigured to use specific secret names. For the source code of this application, see the Verrazzano Examples. If you want to use secret names that are different from what is specified in the source code, you will need to update the corresponding YAML file and rebuild the Docker images for the example application.
To deploy the application, apply the example resources.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-comp.yaml -n bobs-books
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-app.yaml -n bobs-books
```
Wait for all of the pods in the Bob’s Books example application to be ready. You can monitor their progress by listing the pods and inspecting the output, or you can use the kubectl wait command.

You may need to repeat the kubectl wait command several times before it is successful. The WebLogic Server and Coherence pods may take a while to be created and Ready.
```
$ kubectl get pods -n bobs-books

# -or- #

$ kubectl wait \
    --for=condition=Ready pods \
    --all -n bobs-books \
    --timeout=600s
```
Get the EXTERNAL_IP address of the istio-ingressgateway service.
```
$ ADDRESS=$(kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $ADDRESS

# Sample output
11.22.33.44
```
Get the generated host name for the application.
```
$ HOST=$(kubectl get gateways.networking.istio.io bobs-books-bobs-books-gw \
    -n bobs-books \
    -o jsonpath='{.spec.servers[0].hosts[0]}')
$ echo $HOST

# Sample output
bobs-books.bobs-books.11.22.33.44.nip.io
```
Access the application. To access the application in a browser, you will need to do one of the following:
- Option 1: If you are using nip.io, then you can access the application using the generated host name. For example:
  - Robert’s Books UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/.
  - Bobby’s Books UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/bobbys-front-end/.
  - Bob’s order manager UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/bobs-bookstore-order-manager/orders.
- Option 2: Temporarily, modify the /etc/hosts file (on Mac or Linux) or c:\Windows\System32\Drivers\etc\hosts file (on Windows 10), to add an entry mapping the host used by the application to the external IP address assigned to your gateway. For example:
```
11.22.33.44 bobs-books.example.com
```
  Then, you can use a browser to access the application, as shown:
  - Robert’s Books UI at https://bobs-books.example.com/.
  - Bobby’s Books UI at https://bobs-books.example.com/bobbys-front-end/.
  - Bob’s order manager UI at https://bobs-books.example.com/bobs-bookstore-order-manager/orders.
- Option 3: Alternatively, point your own DNS name to the load balancer’s external IP address. In this case, you would need to have edited the bobs-books-app.yaml file to use the appropriate values under the hosts section for the application (such as your-roberts-books-host.your.domain), before deploying the application. Then, you can use a browser to access the application, as shown:
  - Robert’s Books UI at https://<your-roberts-books-host.your.domain>/.
  - Bobby’s Books UI at https://<your-bobbys-books-host.your.domain>/bobbys-front-end/.
  - Bob’s order manager UI at https://<your-bobs-orders-host.your.domain>/.

Access the applications using the WebLogic Server Administration Console

Use the WebLogic Server Administration Console to access the applications as follows.

NOTE

It is recommended that the WebLogic Server Administration Console not be exposed publicly.

Access bobs-bookstore

Set up port forwarding.
```
$ kubectl port-forward pods/bobs-bookstore-adminserver 7001:7001 -n bobs-books
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

Access bobbys-front-end

Set up port forwarding.
```
$ kubectl port-forward pods/bobbys-front-end-adminserver 7001:7001 -n bobs-books
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

Verify the deployed application

Verify that the application configuration, domains, Coherence resources, and ingress trait all exist.
```
$ kubectl get ApplicationConfiguration -n bobs-books
$ kubectl get Domain -n bobs-books
$ kubectl get Coherence -n bobs-books
$ kubectl get IngressTrait -n bobs-books
```
Verify that the service pods are successfully created and transition to the READY state. Note that this may take a few minutes and that you may see some of the services terminate and restart.
```
$ kubectl get pods -n bobs-books

# Sample output
NAME                                                READY   STATUS    RESTARTS   AGE
bobbys-helidon-stock-application-868b5965c8-dk2xb   3/3     Running   0          19h
bobbys-coherence-0                                  2/2     Running   0          19h
bobbys-front-end-adminserver                        3/3     Running   0          19h
bobbys-front-end-managed-server1                    3/3     Running   0          19h
bobs-bookstore-adminserver                          3/3     Running   0          19h
bobs-bookstore-managed-server1                      3/3     Running   0          19h
mysql-669665fb54-9m8wq                              2/2     Running   0          19h
robert-helidon-96997fcd5-kzjkf                      3/3     Running   0          19h
robert-helidon-96997fcd5-nlswm                      3/3     Running   0          19h
roberts-coherence-0                                 2/2     Running   0          17h
roberts-coherence-1                                 2/2     Running   0          17h
```

Undeploy the application

To undeploy the application, delete the Bob’s Books OAM resources.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-app.yaml -n bobs-books
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-comp.yaml -n bobs-books
```
Delete the namespace bobs-books after the application pods are terminated. The secrets created for the WebLogic domain also will be deleted.
```
$ kubectl delete namespace bobs-books
```

Docs: Considerations

Mon, 01 Jan 0001 00:00:00 +0000

Before uninstalling Verrazzano, you should delete your Verrazzano applications because they may not function properly after the uninstall operation is done.

When you uninstall Verrazzano:

All of the Verrazzano components are uninstalled.
The CRDs installed by Verrazzano are not deleted.
Any applications that were deployed will still exist, but they may not be functional.

Docs: Generic Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the generic install

Verrazzano requires that your Kubernetes cluster provides an implementation of network load balancers (Services of type LoadBalancer) for a production environment. If your generic Kubernetes implementation provides this feature, then you can use a default configuration of the Verrazzano custom resource with no customizations and follow the Installation Guide.

NOTE

Remember to not overlap network Classless Inter-Domain Routing (CIDR) blocks when designing and implementing your Kubernetes cluster; proper routing relies on that.

You can install a load balancer, such as MetalLB. This setup requires knowledge of networking both inside and outside your Kubernetes cluster. This would include specifics of your Container Network Interface (CNI) implementation, IP address allocation schemes, and routing that goes beyond the scope of this documentation. For a Kind implementation, see Install and configure MetalLB.

It is possible to use a Kubernetes Service of type NodePort to test aspects of Verrazzano. This requires a good working knowledge of networking and has limited use cases.

Customizations

Verrazzano is highly customizable. If your Kubernetes implementation requires a custom configuration, see Customize Verrazzano.

Next steps

To continue, see the Installation Guide.

Docs: Install Multicluster

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Before you begin, read this document, Verrazzano in a multicluster environment.
To set up a multicluster Verrazzano environment, you will need two or more Kubernetes clusters. One of these clusters will be admin cluster; the others will be managed clusters. For instructions on preparing Kubernetes platforms for installing Verrazzano, see Platform Setup.

NOTE

If Rancher is not enabled, then refer to Verrazzano multicluster installation without Rancher because additional steps are required to register a managed cluster.

The following instructions assume an admin cluster and a single managed cluster. For each additional managed cluster, simply repeat the managed cluster instructions.

Install Verrazzano

To install Verrazzano on each Kubernetes cluster, complete the following steps:

On one cluster, install Verrazzano using the dev or prod profile; this will be the admin cluster.
On the other cluster, install Verrazzano using the managed-cluster profile; this will be a managed cluster. The managed-cluster profile contains only the components that are required for a managed cluster.
NOTE: You also can use the dev or prod profile.

For detailed instructions on how to install and customize Verrazzano on a Kubernetes cluster using a specific profile, see the Installation Guide and Installation Profiles.

Register managed clusters using the console

To register a cluster, complete the following steps:

Enable syncClusters.
For information about syncClusters, see Enable syncClusters.
```
kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: admin
spec:
  profile: prod
  components:
    clusterOperator:
      overrides:
      - values:
          syncClusters:
            enabled: true
EOF
```
On the admin cluster, open the Rancher console.
You can find the Rancher console URL for your cluster by following the instructions here.
Select Cluster Management, and then click Import Existing Cluster.
Provide a name for your managed cluster, for example, managed1.
Optional. In your Verrazzano configuration, if you specified a cluster label selector, then under Labels & Annotations provide the label and value for the cluster. The label and value information must match the cluster selection matchExpression in your Verrazzano configuration for selective cluster registration to occur.
Click Create.
On the next screen, follow the on-screen instructions to complete the registration by running the provided command against the managed cluster.

After the cluster reaches the Active state in the console, synchronization with Verrazzano will happen automatically and a VerrazzanoManagedCluster resource will be created in the verrazzano-mc namespace.

Run the following command to view the details and status of Verrazzano’s multicluster initialization operations:

$ kubectl get vmc -n verrazzano-mc <Rancher_cluster_name> -o yaml

For more information, see Registering Existing Clusters in the Rancher documentation.

NOTE: You can also register managed clusters using kubectl, see Register Managed Clusters using kubectl.

Next steps

Verify your multicluster Verrazzano environment set up by following the instructions at Verify Multicluster Installation.
Deploy multicluster example applications. See Examples of using Verrazzano in a multicluster environment.

NOTE

To deregister a managed cluster, see Deregister a Managed Cluster.

Docs: Install with CLI

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster using the CLI.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster.

Prepare for the installation

Before installing Verrazzano, see instructions on preparing Kubernetes platforms and installing the Verrazzano CLI (optional). Make sure that you have a valid kubeconfig file pointing to the Kubernetes cluster that you want to use for installing Verrazzano.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For instructions on how to install a CNI plug-in, see the documentation for your Kubernetes cluster.

Perform the installation

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations. When applying the Verrazzano custom resource, the Verrazzano CLI deploys and installs the Verrazzano platform operator; you need only to install Verrazzano as described in the following section.

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information, see Installation Profiles.

This document shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE: Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For an overview of how to configure Verrazzano, see Modify Verrazzano Installations. For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see Customizing DNS for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands.

Install Verrazzano with its dev profile.
```
    $ vz install -f - <<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: dev
      defaultVolumeSource:
        persistentVolumeClaim:
          claimName: verrazzano-storage
      volumeClaimSpecTemplates:
        - metadata:
            name: verrazzano-storage
          spec:
            resources:
              requests:
                storage: 2Gi
    EOF
```
This command installs the Verrazzano platform operator and applies the Verrazzano custom resource.
Wait for the installation to complete. Installation logs will be streamed to the command window until the installation has completed or until the default timeout (30m) has been reached.

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

Verify the installation

To verify the Verrazzano installation, you can use the vz status command to determine the status of your installation. After a successful installation, Verrazzano should be in the Ready state.

$ vz status

# Sample output for a dev profile install
Verrazzano Status
  Name: example-verrazzano
  Namespace: default
  Profile: prod
  Version: v1.5.1
  State: Ready
  Available Components: 23/23
  Access Endpoints:
    consoleUrl: https://verrazzano.default.10.0.0.1.nip.io
    grafanaUrl: https://grafana.vmi.system.default.10.0.0.1.nip.io
    keyCloakUrl: https://keycloak.default.10.0.0.1.nip.io
    kialiUrl: https://kiali.vmi.system.default.10.0.0.1.nip.io
    openSearchDashboardsUrl: https://osd.vmi.system.default.10.0.0.1.nip.io
    openSearchUrl: https://opensearch.vmi.system.default.10.0.0.1.nip.io
    prometheusUrl: https://prometheus.vmi.system.default.10.0.0.1.nip.io
    rancherUrl: https://rancher.default.10.0.0.1.nip.io

For installation troubleshooting help, see the Analysis Advice.

After the installation has completed, you can use the Verrazzano consoles. For information on how to get the consoles URLs and credentials, see Access Verrazzano.

Next steps

(Optional) Run the example applications located here.

Docs: Install with kubectl

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster using kubectl.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster.

Prepare for the installation

You can install Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

Install the Verrazzano platform operator

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations.

To install the Verrazzano platform operator:

Deploy the Verrazzano platform operator.
```
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/master/verrazzano-platform-operator.yaml
```
Wait for the deployment to complete.
```
$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
```
```
# Expected response
deployment "verrazzano-platform-operator" successfully rolled out
```
Confirm that the operator pod is correctly defined and running.
```
 $ kubectl -n verrazzano-install get pods
```
```
 # Sample output
 NAME                                            READY   STATUS    RESTARTS   AGE
 verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s
```

Perform the installation

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information, see Installation Profiles.

This document shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE: Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see Customzing DNS for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands.

$ kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: ${VZ_PROFILE:-dev}
EOF

$ kubectl wait \
    --timeout=20m \
    --for=condition=InstallComplete verrazzano/example-verrazzano

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

If an error occurs, check the log output of the installation. You can view the logs with the following command.

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

Verify the installation

To verify the Verrazzano installation, you can use kubectl to view the status of the Verrazzano resource. After a successful installation, Verrazzano status should be InstallComplete.

$ kubectl get vz

# Example response
NAME                 AVAILABLE   STATUS            VERSION
example-verrazzano   23/23       InstallComplete   v1.5.2

For installation troubleshooting help, see Analysis Advice.

After the installation has completed, you can use the Verrazzano consoles. For information on how to get the consoles URLs and credentials, see Access Verrazzano.

Next steps

(Optional) Run the example applications located here.

Docs: Jaeger Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Jaeger is a distributed tracing system used for monitoring and troubleshooting microservices. Distributed tracing lets you trace errors across your microservice architecture. You can track application requests as they flow from frontend devices to backend services and databases. You use distributed tracing to troubleshoot requests that exhibit high latency or errors. For more information on Jaeger distributed tracing, see the Jaeger website.

Install Jaeger Operator

To install the Jaeger Operator, enable the jaegerOperator component in your Verrazzano custom resource. Here is an example YAML file that enables the Jaeger Operator. Verrazzano installs the Jaeger Operator in the verrazzano-monitoring namespace. If OpenSearch and Keycloak components are enabled in the Verrazzano custom resource, then a default Jaeger instance is also created by the Jaeger Operator in the verrazzano-monitoring namespace.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true

The Jaeger Operator will create Service custom resources for query and collection. After applying the Verrazzano custom resource, listing Jaeger resources will show output similar to the following.

$ kubectl get services,deployments -l app.kubernetes.io/instance=jaeger-operator-jaeger -n verrazzano-monitoring

#sample output
NAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                  AGE
service/jaeger-operator-jaeger-collector            ClusterIP   10.96.120.223   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   79m
service/jaeger-operator-jaeger-collector-headless   ClusterIP   None            <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP   79m
service/jaeger-operator-jaeger-query                ClusterIP   10.96.209.196   <none>        16686/TCP,16685/TCP                      79m

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-operator-jaeger-collector   1/1     1            1           79m
deployment.apps/jaeger-operator-jaeger-query       1/1     1            1           79m

Customize Jaeger

Verrazzano installs the Jaeger Operator and Jaeger using the jaeger-operator Helm chart. Using Helm overrides specified in the Verrazzano custom resource, you can customize the installation configuration. For more information about setting component overrides, see Customizing the Chart Before Installing.

The following sections describe some common customizations you might want to employ:

Customize a Jaeger instance to use an external OpenSearch for storage

You can use the default Jaeger instance with an external OpenSearch cluster. The following example shows you how to configure Jaeger Operator Helm overrides in the Verrazzano custom resource to use an external OpenSearch cluster with a TLS CA certificate mounted from a volume and the user/password stored in a secret. For more details, see the Jaeger documentation.

Prior to configuring the external OpenSearch for Jaeger in the Verrazzano custom resource, create a secret containing the OpenSearch credentials and certificates in the verrazzano-install namespace. Jaeger will use these credentials to connect to OpenSearch.
```
$ kubectl create secret generic jaeger-secret \
 --from-literal=OS_PASSWORD=<OPENSEARCH PASSWORD> \
 --from-literal=OS_USERNAME=<OPENSEARCH USERNAME> \
 --from-file=ca-bundle=<path to the file containing CA certs> \
 -n verrazzano-install
```
Use the Verrazzano custom resource to update the Jaeger resource:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger-external-opensearch
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              create: true
              spec:
                strategy: production
                storage:
                  type: opensearch
                  options:
                    es:
                      # Enter your OpenSearch cluster endpoint here.
                      server-urls: <External OpenSearch URL>
                      index-prefix: jaeger
                      tls:
                        ca: /verrazzano/certificates/ca-bundle
                  secretName: jaeger-secret
                volumeMounts:
                  - name: certificates
                    mountPath: /verrazzano/certificates/
                    readOnly: true
                volumes:
                  - name: certificates
                    secret:
                      secretName: jaeger-secret
```

Enable Jaeger Service Performance Monitoring

NOTE: The Service Performance Monitoring (SPM) feature is currently considered experimental.

To enable Service Performance Monitoring in the default Jaeger instance created by Verrazzano, use the following Verrazzano custom resource. Verrazzano sets jaeger.spec.query.options.prometheus.server-url to the Prometheus server URL managed by Verrazzano, if it exists. To configure an external Prometheus server for your use case, override jaeger.spec.query.options.prometheus.server-url, jaeger.spec.query.options.prometheus.tls.enabled, and jaeger.spec.query.options.prometheus.tls.ca, appropriately, in the Verrazzano custom resource. For more details, see the Jaeger documentation.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              spec:
                query:
                  metricsStorage:
                    type: prometheus

Disable default Jaeger instance creation

If you have a custom Jaeger instance and want to disable the default Jaeger instance created by Verrazzano, then use the following Verrazzano custom resource:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-jaeger
spec:
  profile: prod
  components:
    jaegerOperator:
       enabled: true
       overrides:
        - values:
            jaeger:
              create: false

Jaeger Operator Helm chart values that you cannot override

The following Jaeger Operator Helm values are not supported to be overridden in the Verrazzano custom resource:

nameOverride
fullnameOverride
serviceAccount.name
ingress.enabled
jaeger.spec.storage.dependencies.enabled

If you try to override these Helm values in the Verrazzano custom resource, the request will be rejected and an error message returned.

NOTE: Verrazzano does not support Jaeger Spark dependencies and hence, the Helm chart value jaeger.spec.storage.dependencies.enabled, which is set to false for the Jaeger instance managed by Verrazzano, cannot be overridden.

Configure the Istio mesh to use Jaeger tracing

You can view Istio mesh traffic by enabling Istio’s distributed tracing integration. Traces from the Istio mesh provide observability on application traffic that passes through Istio’s ingress and egress gateways.

Istio tracing is disabled by default. To turn on traces, customize your Istio component like the following example:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true

After enabling tracing, Istio will automatically configure itself with the the Jaeger instance managed by Verrazzano in your cluster, and Istio-injected pods will begin exporting traces to Jaeger.

To export traces to a different Jaeger instance than the one managed by Verrazzano, set meshConfig.defaultConfig.tracing.zipkin.address to the intended Jaeger Collector URL. Any new Istio-injected pods will begin exporting traces to the newly configured Jaeger instance. Existing pods require a restart to pull the new Istio configuration and start sending traces to the newly configured Jaeger instance.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true
                defaultConfig:
                  tracing:
                    zipkin:
                      address: <address:port of your Jaeger collector service>

Istio’s default sampling rate is 1%, meaning 1 in 100 requests will be traced in Jaeger. If you want a different sampling rate, configure your desired rate using the meshConfig.defaultConfig.tracing.sampling Istio installation argument.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: prod
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              meshConfig:
                enableTracing: true
                defaultConfig:
                  tracing:
                    sampling: 25.0

Manage Jaeger indices in OpenSearch

To clean old Jaeger data from OpenSearch, Verrazzano uses the index management provided by Jaeger. By default, a cron job with the following default values is created to clean old traces. To configure it to your use case, override the following Jaeger spec values in the Verrazzano custom resource with your desired values.

storage:
  type: opensearch
  esIndexCleaner:
    enabled: true                                 // turn the cron job deployment on and off
    numberOfDays: 7                               // number of days to wait before deleting a record
    schedule: "55 23 * * *"                       // cron expression for it to run

Configure applications to export traces to Jaeger

The Jaeger agent sidecar is injected to application pods by the "sidecar.jaegertracing.io/inject": "true" annotation. You may apply this annotation to namespaces or pod controllers, such as Deployments. The subsequent snippet shows how to annotate an OAM Component for Jaeger agent injection.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-component
spec:
  workload:
    apiVersion: core.oam.dev/v1alpha2
    kind: ContainerizedWorkload
    metadata:
      name: example-workload
      annotations:
        # The component's Deployment will carry the Jaeger annotation.
        "sidecar.jaegertracing.io/inject": "true"

If you have multiple Jaeger instances in your cluster, specify the name of the Jaeger instance to which you intend to send the traces, as a value for the annotation sidecar.jaegertracing.io/inject. For more details, see the Jaeger documentation.

NOTE: Using the Jaeger agent is not supported in Helidon 3.x. To use Jaeger tracing, the Helidon application should connect directly to the Jaeger collector. See the following example YAML file, where "TRACING_HOST" is set to "jaeger-operator-jaeger-collector.verrazzano-monitoring" and "TRACING_PORT" to "9411". For Jaeger tracing in a multicluster Verrazzano environment, set the "TRACING_HOST" to "jaeger-verrazzano-managed-cluster-collector.verrazzano-monitoring.svc.cluster.local".

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
        version: v1
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          containers:
            - name: hello-helidon-container
              image: "ghcr.io/verrazzano/example-helidon-greet-app-v1:1.0.0-1-20220513221156-7da0d32"
              env:
                - name: "TRACING_SERVICE"
                  value: "hello-helidon"
                - name: "TRACING_PORT"
                  value: "9411"
                - name: "TRACING_HOST"
                  value: "jaeger-operator-jaeger-collector.verrazzano-monitoring"
              ports:
                - containerPort: 8080
                  name: http

View traces on the Jaeger console

After the installation has completed, you can use the Verrazzano Jaeger console to view the traces. For information on how to get the Verrazzano Jaeger console URL and credentials, see Access Verrazzano.

Use Jaeger tracing in a multicluster Verrazzano environment

If the Jaeger Operator component is enabled in the managed cluster, after successful registration with the admin cluster, a Jaeger collector service runs in the managed cluster, which exports the traces to the OpenSearch storage configured in the admin cluster.

NOTE: Traces are exported to the admin cluster only when the Jaeger instance in the admin cluster is configured with the OpenSearch storage.

Listing Jaeger resources in the managed cluster shows output similar to the following.

$ kubectl get jaegers -n verrazzano-monitoring

#sample output
NAME                                STATUS    VERSION   STRATEGY     STORAGE         AGE
jaeger-verrazzano-managed-cluster   Running   1.34.1    production   opensearch      11m

Configure the Istio mesh in a managed cluster to export Jaeger traces to the admin cluster

To export the Istio mesh traces in the managed cluster to the admin cluster, set meshConfig.defaultConfig.tracing.zipkin.address to the Jaeger Collector URL created in the managed cluster that exports the traces to the OpenSearch storage configured in the admin cluster.

Configure the Istio mesh on the managed cluster at the time of the Verrazzano installation, as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: verrazzano
spec:
  profile: managed-cluster
  components:
    jaegerOperator:
      enabled: true
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            meshConfig:
              enableTracing: true
              defaultConfig:
                tracing:
                  zipkin:
                    address: jaeger-verrazzano-managed-cluster-collector.verrazzano-monitoring.svc.cluster.local.:9411

View the managed cluster traces

You can see the managed cluster traces from the Jaeger console in the admin cluster only. To find the Jaeger console URL for your admin cluster, follow the instructions for Accessing Verrazzano.

The spans include the Process tag verrazzano_cluster, which has the name of the managed cluster. To see the traces for the managed cluster only, search based on the tag verrazzano_cluster=<managed cluster name>.

Sample output of Jager console screens

Additional information

Read the blog, Jaeger, Fluentd, and OpenSearch with Verrazzano, for a detailed look at how these Verrazzano components work together to provide a complete observability stack.

Docs: Monitor Components in Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus and Grafana components. Metrics sources expose system and application metrics. The Prometheus components retrieve and store the metrics and Grafana provides dashboards to visualize them.

Metrics sources

The following sections describe metrics sources that Verrazzano provides for OAM and standard Kubernetes applications.

OAM

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.
```
$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list
```
Get the user name and password used to access the metrics source from the corresponding secret.
```
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo
```
Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Standard Kubernetes workloads

Verrazzano supports enabling metric sources for Kubernetes workloads deployed without OAM Components. To enable metrics for Kubernetes workloads, you must create a Service Monitor or Pod Monitor, as applicable. For details on Service Monitor and Pod Monitor, refer to the Prometheus Operator documentation.

When creating the Service Monitor or Pod Monitor for your workload, include the label release, with the value prometheus-operator on the monitor resource.

Verify metrics collection

To verify that the metrics are being collected for your workload, follow these steps.

Access the Prometheus console.
From the console, use the navigation bar to access Status/Targets.
On this page, you will see a target name with this formatting: <monitor-type>/<workload-namespace>_<workload-name>_<workload-type>, where monitor-type may be serviceMonitor or podMonitor.
Copy this job name from the target labels for use in future queries.
Verify that the State of this target is UP.
Next, use the navigation bar to access the Graph.
Here, use the job name you copied to construct this expression: {job="<job_name>"}
Use the graph to run this expression and verify that you see application metrics appear.

Metrics Traits use Service Monitors which require Services for metrics collection. If you are unable to verify metrics collection, you might need to manually create a Service for the workload.

For more information on Prometheus solutions, see Troubleshooting Prometheus.

Legacy workloads

Standard Kubernetes workloads that were metrics sources in earlier versions of Verrazzano (1.3.x or earlier), will continue to be supported when upgrading to later versions of Verrazzano.

For workloads that used the legacy default metrics template, Verrazzano will create a Service Monitor in the workload’s namespace to ensure that metrics continue to be scraped. You can make any ongoing changes to the metrics source configuration by editing the Service Monitor.

For workloads that used a legacy custom metrics template, Verrazzano will configure the Prometheus Operator to ensure that metrics continue to be scraped.

Metrics server

Verrazzano installs the Prometheus Operator in the verrazzano-monitoring namespace.
A single Prometheus pod is created by Prometheus Operator in the same namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following:

A Kubernetes cluster and a compatible kubectl.
dev profile - Each node in the cluster should contain at least two CPUs, 8 GB RAM, and 30 GB of disk storage. The entire cluster requires at least six CPUs, 48 GB RAM, and 100 GB of disk storage. In addition, about 52 GB of storage is required for the persistent volumes.
prod profile - Each node in the cluster should contain at least two CPUs, 24 GB RAM, and 30 GB of disk storage. The entire cluster requires at least eight CPUs, 64 GB RAM, and 150 GB of disk storage. In addition, about 450 GB of storage is required for the persistent volumes.

The persistent volumes are provisioned using the default StorageClass in the cluster. In case of a local disk based PV provisioner, each node in the cluster should have a minimum of 80 GB of disk storage for both of the profiles.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster. Also, depending on the resource requirements of the applications you deploy, the configurations previously suggested may or may not be sufficient.

Supported hardware

Verrazzano requires x86-64; other architectures are not supported.

Supported software versions

Verrazzano supports the following software versions.

Kubernetes

You can install Verrazzano on the following Kubernetes versions.

Verrazzano	Release Date	Latest Patch Release	Latest Patch Release Date	End of Error Correction*	Kubernetes Versions
1.5	2023-02-15	1.5.3	2023-05-09	2024-02-28**	1.21, 1.22, 1.23, 1.24
1.4	2022-09-30	1.4.4	2023-03-15	2023-10-31	1.21, 1.22, 1.23, 1.24
1.3	2022-05-24	1.3.8	2022-11-17	2023-05-31	1.21, 1.22, 1.23
1.2	2022-03-14	1.2.2	2022-05-10	2022-11-30	1.19, 1.20, 1.21
1.1	2021-12-16	1.1.2	2022-03-09	2022-09-30	1.19, 1.20, 1.21
1.0	2021-08-02	1.0.4	2021-12-20	2022-06-30	1.18, 1.19, 1.20

*End of error correction for Verrazzano releases.
**Projected date. Actual date will be determined when the next minor or major release is available.

For more information, see Kubernetes Release Documentation.
For platform specific details, see Verrazzano platform setup.

WebLogic Server

The supported versions of WebLogic Server are dependent on the WebLogic Kubernetes Operator version. See the WebLogic Server versions supported here.

Coherence

The supported versions of Coherence are dependent on the Coherence Operator version. See the Coherence versions supported here.

Helidon

Verrazzano supports all versions of Helidon. For more information, see Helidon and Helidon Commercial Offerings.

Installed components

Verrazzano installs a curated set of open source components. The following table lists each component, its version, and a brief description.

Component	Version	Description
Alertmanager	0.24.0	Handles alerts sent by client applications, such as the Prometheus server.
Argo CD	2.5.3	A declarative, GitOps continuous delivery tool for Kubernetes.
cert-manager	1.9.1	Automates the management and issuance of TLS certificates.
Coherence Operator	3.2.9	Assists with deploying and managing Coherence clusters.
ExternalDNS	0.12.2	Synchronizes exposed Kubernetes Services and ingresses with DNS providers.
Fluentd	1.14.5	Collects logs and sends them to OpenSearch.
Grafana	7.5.17	Tool to help you examine, analyze, and monitor metrics.
Istio	1.15.3	Service mesh that layers transparently onto existing distributed applications.
Jaeger	1.42.0	Distributed tracing system for monitoring and troubleshooting distributed systems.
Jaeger Operator	1.42.0	Provides management for Jaeger tools.
Keycloak	20.0.1	Provides single sign-on with Identity and Access Management.
Kiali	1.57.1	Management console for the Istio service mesh.
kube-state-metrics	2.6.0	Provides metrics about the state of Kubernetes API objects.
MySQL	8.0.32	Open source relational database management system used by Keycloak.
MySQL Operator	8.0.32-2.0.8	Operator for managing MySQL InnoDB Cluster setups inside a Kubernetes cluster.
NGINX Ingress Controller	1.3.1	Traffic management solution for cloud‑native applications in Kubernetes.
Node Exporter	1.3.1	Prometheus exporter for hardware and OS metrics.
OAM Kubernetes Runtime	0.3.3	Plug-in for implementing the Open Application Model (OAM) control plane with Kubernetes.
OpenSearch	2.3.0	Provides a distributed, multitenant-capable full-text search engine.
OpenSearch Dashboards	2.3.0	Provides search and data visualization capabilities for data indexed in OpenSearch.
Prometheus	2.38.0	Provides event monitoring and alerting.
Prometheus Adapter	0.10.0	Provides metrics in support of pod autoscaling.
Prometheus Operator	0.59.1	Provides management for Prometheus monitoring tools.
Prometheus Pushgateway	1.4.2	Allows ephemeral and batch jobs to expose their metrics to Prometheus.
Rancher	2.7.2	Manages multiple Kubernetes clusters.
Rancher Backup Operator	2.1.3	Manages backup and restore of Rancher configurations and data.
Velero	1.9.1	Manages backup and restore of Kubernetes configurations and data.
WebLogic Kubernetes Operator	4.0.6	Assists with deploying and managing WebLogic domains.
WebLogic Monitoring Exporter	2.1.3	Exports Prometheus-compatible metrics from WebLogic instances.

Docs: Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus is a system for monitoring cloud native applications and is used by Verrazzano to monitor applications. Prometheus is used in Verrazzano to collect system performance metrics and applications deployed or managed by Verrazzano. Prometheus analysis the metrics and provide a visualization using Grafana.

Verrazzano installs Prometheus components, including Prometheus Operator and Prometheus, using the kube-prometheus-stack Helm chart. You can customize the installation configuration using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the number of Prometheus replicas.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            prometheus:
              prometheusSpec:
                replicas: 3

To enable Alertmanager, use the following Verrazzano custom resource:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            alertmanager:
              enabled: true
              alertmanagerSpec:
                podMetadata:
                  annotations:
                    sidecar.istio.io/inject: "false"

For more information about setting component overrides, see Customizing the Chart Before Installing.

After you have enabled Alertmanager, you can deploy alert rules to get proactive alerts. To create a TestAlertRule, run the following command.

kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus-operator
  name: prometheus-operator-test
  namespace: verrazzano-monitoring
spec:
  groups:
    - name: test
      rules:
        - alert: TestAlertRule
          annotations:
            description: Test alert rule
            runbook_url: test-runbook-url
            summary: Test alert rule
          expr: |-
            prometheus_config_last_reload_successful{job="prometheus-operator-kube-p-prometheus",namespace="verrazzano-monitoring"} == 0            
          for: 10m
          labels:
            severity: critical
EOF

For more information, see Deploying Prometheus rules.

Docs: Synchronize Managed Clusters Registration

Mon, 01 Jan 0001 00:00:00 +0000

You can synchronize cluster registration across Verrazzano by enabling syncClusters in the Verrazzano custom resource. Any clusters imported in the Rancher console will be synchronized across the rest of Verrazzano, including in Verrazzano managed cluster resources, Rancher, and Argo CD. Additionally, you can use a label selector to filter the clusters you want registered.

Register managed clusters automatically

The following illustrates an admin cluster Verrazzano resource that enables syncClusters.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: admin
spec:
  profile: prod
  components:
    clusterOperator:
      overrides:
      - values:
          syncClusters:
            enabled: true

When enabled is set to true, Verrazzano will synchronize the registration of clusters across Verrazzano, Rancher, and Argo CD.
If enabled is set to false (the default), then Verrazzano will not synchronize the registration of clusters across Verrazzano.

Filter cluster selection

Optionally, to determine which clusters imported into Rancher will be automatically registered by Verrazzano, you can provide a label selector in the Verrazzano resource. This lets you filter which clusters you want to be synchronized.

The following illustrates an admin cluster Verrazzano resource that supports cluster label selection.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: admin
spec:
  profile: prod
  components:
    clusterOperator:
      overrides:
      - values:
          syncClusters:
            enabled: true
            clusterSelector:
              matchExpressions:
              - key: verrazzanomulticluster
                operator: In
                values: [supported]

The syncClusters enabled field must be set to true to use cluster label selection. Verrazzano will automatically register all clusters with labels that match the clusterSelector field.

The clusterSelector field is optional.
If enabled is set to true and the clusterSelector field is omitted, then all clusters imported into Rancher will be automatically registered.

When you import a cluster into Rancher, you can provide a label and value for the cluster. If the label matches this label selector, then the cluster will be synchronized.

Docs: Upgrade Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano installation consists of a stack of components, such as cert-manager, where each component has a specific release version that may be different from the overall Verrazzano version. The Verrazzano platform operator knows the versions of each component associated with the Verrazzano version. When you perform the initial Verrazzano installation, the appropriate version of each component is installed by the platform operator. Post installation, it may be necessary to update one or more of the component images or Helm charts. This update is also handled by the platform operator and is called an upgrade. Currently, Verrazzano does only patch-level upgrades, where a helm upgrade command can be issued for the component. Typically, patch-level upgrades simply replace component images with newer versions.

Application and system pod restarts

If Verrazzano has a new version of Istio, then all the pods with Istio proxy sidecars need to be restarted. This is done so that the new version of the proxy sidecar can be injected into the pods. All Verrazzano pods containing Istio proxy sidecars will be restarted. This includes Verrazzano system pods, such as the NGINX Ingress Controller, along with Verrazzano applications. For WebLogic workloads, Verrazzano will shut down every domain, do the upgrade, then restart every domain. For all other workloads, Verrazzano will perform a rolling restart when the upgrade has completed. There is no user involvement related to restarting applications; it is done automatically during upgrade.

Upgrade steps

It is important to distinguish between updating the Verrazzano platform operator versus upgrading the Verrazzano installation. The platform operator contains the newer component charts and image versions, so it must be updated prior to upgrading the installation. Updating the platform operator has no effect on an existing installation until you initiate the Verrazzano installation upgrade. Currently, there is no way to roll back either the platform operator update or the Verrazzano installation upgrade.

You can upgrade Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

NOTE

For optimal functionality, be sure to install or upgrade the CLI version to match the Verrazzano version to which you are upgrading.

Upgrade Verrazzano using the CLI

In one simple step, you can upgrade to a specified version of Verrazzano using this command.

Update the Verrazzano resource to the desired version.

To update to the latest version (default):
```
$ vz upgrade
```
To update to a specific version, where <version> is the desired version:
```
$ vz upgrade --version <version>
```
Wait for the upgrade to complete. Upgrade logs will be streamed to the command window until the upgrade has completed or until the default timeout (30m) has been reached.

Upgrade using kubectl

Upgrading an existing Verrazzano installation is a two-step process:

Update the Verrazzano platform operator to the Verrazzano release version to which you want to upgrade.
Upgrade the Verrazzano installation.

Update the Verrazzano platform operator

In order to upgrade an existing Verrazzano installation, you must first update the Verrazzano platform operator.

Update the Verrazzano platform operator.

NOTE: If you are using a private container registry, then to update the platform operator, follow the instructions at Use a Private Registry.

To update to the latest version:
```
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/master/verrazzano-platform-operator.yaml
```
To update to a specific version, where <version> is the desired version:
```
# To update to version v1.4.0:
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/<version>/verrazzano-platform-operator.yaml
```
```
# To update to a version prior to v1.4.0:
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/<version>/operator.yaml
```
Wait for the deployment to complete.
```
$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
```
```
# Expected response
deployment "verrazzano-platform-operator" successfully rolled out
```
Confirm that the operator pod is correctly defined and running.
```
$ kubectl -n verrazzano-install get pods
```
```
# Sample output
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s
```

Upgrade Verrazzano

To upgrade the Verrazzano installation, you need to change the version of your installed Verrazzano resource to the version supported by the Verrazzano platform operator.

NOTE: You may only change the version field during an upgrade; changes to other fields or component configurations are not supported at this time.

In one simple step, you can upgrade to a specified version of Verrazzano using this command.

$ kubectl patch vz example-verrazzano -p '{"spec":{"version":"master"}}' --type=merge

Alternatively, you can upgrade the Verrazzano installation using the following steps.

Update the Verrazzano resource to the desired version.

To upgrade the Verrazzano components, you must update the version field in your Verrazzano resource spec to match the version supported by the platform operator to which you upgraded and apply it to the cluster.

The value of the version field in the resource spec must be a Semantic Versioning value corresponding to a valid Verrazzano release version.

To update the resource, do one of the following:

a. Edit the YAML file you used to install Verrazzano and set the version field to the latest version.

For example, to upgrade to master, your YAML file should be edited to add or update the version field.
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
   profile: dev
   version: master
```
Then, apply the resource to the cluster (if you have not edited the resource in-place using kubectl edit).
```
$ kubectl apply -f example-verrazzano.yaml
```
b. Edit the Verrazzano resource directly using kubectl and set the version field directly, for example:
```
$ kubectl edit verrazzano example-verrazzano
# In the resource editor, add or update the version field to "version: master", then save.
```
Wait for the upgrade to complete.
```
$ kubectl wait \
    --timeout=30m \
    --for=condition=UpgradeComplete verrazzano/example-verrazzano
```

Verify the upgrade

Check that all the pods in the verrazzano-system namespace are in the Running state. While the upgrade is in progress, you may see some pods terminating and restarting as newer versions of components are applied, for example:

$ kubectl get pods -n verrazzano-system

# Sample output
coherence-operator-866798c99d-r69xt                1/1     Running   1          43m
fluentd-f9fbv                                      2/2     Running   0          38m
fluentd-n79c4                                      2/2     Running   0          38m
fluentd-xslzw                                      2/2     Running   0          38m
oam-kubernetes-runtime-56cdb56c98-wn2mb            1/1     Running   0          43m
verrazzano-application-operator-7c95ddd5b5-7xzmn   1/1     Running   0          42m
verrazzano-authproxy-594d8c8dcd-llmlr              2/2     Running   0          38m
verrazzano-console-74dbf97fdf-zxvvn                2/2     Running   0          38m
verrazzano-monitoring-operator-6fcf8484fd-gfkhs    1/1     Running   0          38m
verrazzano-operator-66c8566f95-8lbs6               1/1     Running   0          38m
vmi-system-os-master-0                             2/2     Running   0          38m
vmi-system-grafana-799d79648d-wsdp4                2/2     Running   0          38m
vmi-system-kiali-574c6dd94d-f49jv                  2/2     Running   0          41m
vmi-system-opensearchDashboards-77f8d998f4-zzvqr   2/2     Running   0          38m
weblogic-operator-7b447fdb47-wlw64                 2/2     Running   0          42m

Check that the pods in your application namespaces are ready, for example:

$ kubectl get pods -n todo-list

# Sample output
NAME                     READY   STATUS    RESTARTS   AGE
mysql-67575d8954-d4vkm   2/2     Running   0          39h
tododomain-adminserver   4/4     Running   0          39h

Upgrade failures

In Verrazzano 1.3 and later, upgrade will continue to run until it succeeds or until you delete the Verrazzano CR. In previous versions, upgrade could fail and transition to the UpgradeFailed state. If that happens, and you updated the Verrazzano platform operator to 1.3+, then the Verrazzano CR will transition to UpgradePaused. To continue with the upgrade, you must change the CR version to the current version of the Verrazzano platform operator. The following steps illustrate this scenario:

You install Verrazzano 1.1.2.
You upgrade to 1.2.0 by changing the Verrazzano CR version field to v1.2.0.
- For some reason, the upgrade failed and the Verrazzano CR state transitions to UpgradeFailed.
You update the Verrazzano platform operator to 1.3.0.
- The Verrazzano CR state transitions to UpgradePaused.
You change the Verrazzano CR version field to v1.3.0.
- The Verrazzano CR state transitions to Upgrading and stays in that state until it completes, then it transitions to UpgradeComplete.

To see detailed progress of the upgrade, view the logs with the following command.

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

For additional troubleshooting help, see Analysis Advice.

Docs: Verrazzano Analysis Tools

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides the vz analyze command-line tool, which assists in troubleshooting issues in your environment. You can use it to analyze a cluster as well as, to analyze a cluster snapshot captured by the vz bug-report tool. For detailed information about vz bug-report, see here.

The vz analyze command-line tool analyzes the cluster or a cluster snapshot, reports the issues found, and prescribes related actions to take. Users, developers, and Continuous Integration (CI) can use this tooling to quickly identify the root cause of encountered problems, determine mitigation actions, and provide a sharable report with other users or tooling.

Set up the CLI tool

To set up the vz command-line tool, follow the steps here.

Analyze clusters

To analyze a Kubernetes cluster:

$ vz analyze

Analyze cluster snapshots

Use the vz bug-report tool to capture a cluster snapshot.

To create a bug report in a TAR file named my-bug-report.tar.gz and extract it to a directory my-cluster-snapshot:
```
$ vz bug-report my-bug-report.tar.gz
  mkdir my-cluster-snapshot
  tar -xvf my-bug-report.tar.gz -C my-cluster-snapshot
```
Use the vz analyze tool to analyze the cluster snapshot.

To perform an analysis of the cluster snapshot under my-cluster-snapshot:
```
$ vz analyze --capture-dir my-cluster-snapshot
```

Use the vz analyze tool to analyze multiple snapshots

The vz analyze tool will find and analyze all cluster snapshot directories found under a specified root directory. This lets you create a directory to hold the cluster snapshots of related clusters in subdirectories, which the tool can then analyze.

For example:

my-cluster-snapshots
    CAPTURE_DIR-1
        cluster-snapshot
            ...
    CAPTURE_DIR-2
        cluster-snapshot
            ...

To perform an analysis of the clusters under my-cluster-snapshots:

$ vz analyze --capture-dir my-cluster-snapshots

Usage information

Use the following syntax to run vz commands from your terminal window.

$ vz analyze [flags]

Available options

Command	Definition
`--capture-dir string`	Directory holding the captured data.
`-h, --help`	Help for the `vz analyze` command.
`--report-file string`	Name of the report output file. (Default `stdout`)
`--report-format string`	The format of the report output. Valid report formats are “summary” and “detailed”. (Default “summary”)
`-v, --verbose`	Enable verbose output.

Available flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`--kubeconfig string`	Path to the kubeconfig file to use.

Docs: Verrazzano and the Open Application Model

Mon, 01 Jan 0001 00:00:00 +0000

Open Application Model (OAM) is a runtime-agnostic specification for defining cloud native applications; it allows developers to focus on the application instead of the complexities of a particular runtime infrastructure. OAM provides the specification for several file formats and rules for a runtime to interpret. Verrazzano uses OAM to enable the definition of a composite application abstraction and makes OAM constructs available within a VerrazzanoApplication YAML file. Verrazzano provides the flexibility to combine what you want into a multicloud enablement. It uses the VerrazzanoApplication as a means to encapsulate a set of components, scopes, and traits, and deploy them on a selected cluster.

OAM’s workload concept makes it easy to use many different workload types. Verrazzano includes specific workload types with special handling to deploy and manage those types, such as WebLogic, Coherence, and Helidon. OAM’s flexibility lets you create a grouping that is managed as a unit, although each component can be scaled or updated independently.

How does OAM work?

OAM has five core concepts:

Workloads - Declarations of the kinds of resources supported by the platform and the OpenAPI schema for that resource. Most Kubernetes CRDs can be exposed as workloads. Standard Kubernetes resource types can also be used (for example, Deployment, Service, Pod, ConfigMap).
Components - Wrap a workload resource’s specification data within OAM specific metadata.
Application Configurations - Describe a collection of components that comprise an application. This is also where customization (such as, environmental) of each component is done. Customization is achieved using scopes and traits.
Scopes - Apply customization to several components.
Traits - Apply customization to a single component.

Docs: Verrazzano in a Multicluster Environment

Mon, 01 Jan 0001 00:00:00 +0000

Review the following key concepts to understand multicluster Verrazzano.

Admin cluster - A Kubernetes cluster that serves as the central management point for deploying and monitoring applications in managed clusters.
Managed clusters - A Kubernetes cluster that has the following characteristics:
- It is registered with an admin cluster with a unique name.
- Verrazzano multicluster applications may be deployed to the managed cluster from the admin cluster.
- Logs and metrics for Verrazzano system components and Verrazzano multicluster applications deployed on the managed cluster are viewable from the admin cluster.
Verrazzano multicluster resources - Custom Kubernetes resources defined by Verrazzano.
- Each multicluster resource serves as a wrapper for an underlying resource type.
- A multicluster resource allows the placement of the underlying resource to be specified as a list of names of the clusters in which the resource must be placed.

For more details, see here.

Docs: Verrazzano Projects

Mon, 01 Jan 0001 00:00:00 +0000

A project provides a way to group application namespaces that are owned or administered by the same user or group of users. When creating a project, you can specify the subjects: users, groups, or service accounts, that are to be granted access to the namespaces governed by the project. Two types of subjects may be specified:

Project admins, who have both read and write access to the project’s namespaces.
Project monitors, who have read-only access to the project’s namespaces.

For more information, see Projects.

Docs: Argo CD

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides an Argo CD deployment that automates the deployment of the desired application states in the specified target environments. There are scenarios where you may want to back up your Argo CD data and restore it.

Verrazzano uses Velero to facilitate backing up and restoring Argo CD data.

Velero operator prerequisites

Before proceeding with an Argo CD backup or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore Argo CD, you must first enable Velero.

The following configuration shows you how to enable velero with a prod installation profile.
```
  $ kubectl apply -f -<<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: prod
      components:
        velero:
          enabled: true
EOF
```
After Verrazzano installation is completed, verify that the Velero pods are running in the verrazzano-backup namespace.
```
$ kubectl get pod -n verrazzano-backup
```
```
# Sample of pods running after enabling the velero component
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h
```
Velero requires a secret to communicate with the S3 compatible object store, so we create a backup-secret.txt file, which has the object store credentials.
```
[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>
```
In the namespace verrazzano-backup, create a Kubernetes secret, for example verrazzano-backup-creds.
```
$ kubectl create secret generic -n verrazzano-backup <secret-name> --from-file=<key>=<full_path_to_creds_file>
```
The following is an example:
```
$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt
```
NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.
Create a BackupStorageLocation resource, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example, which uses Oracle Cloud Object Storage. For more information, see Backup Storage Location in the Velero documentation.
```
 $ kubectl apply -f -<<EOF
   apiVersion: velero.io/v1
   kind: BackupStorageLocation
   metadata:
     name: verrazzano-backup-location
     namespace: verrazzano-backup
   spec:
     provider: aws
     objectStorage:
       bucket: example-verrazzano
       prefix: backup-demo
     credential:
       name: verrazzano-backup-creds
       key: cloud
     config:
       region: us-phoenix-1
       s3ForcePathStyle: "true"
       s3Url: https://<myobjstorenamespace>.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```

Argo CD backup using Velero

If you created applications with resources running in different namespaces, other than argocd, then based on the following criteria you can back up and restore Argo CD:

If applications running in different namespaces use persistent volumes, then you can back up the namespace where the applications are running with the PV.
If applications running in different namespaces do not use persistent storage, then:
- Take a backup of all the namespaces where the application is running by specifying the namespaces as a list.
- Take a backup of only the argocd namespace, create all the namespaces of different applications, and then restore from the backup.

The following example shows a sample Velero Backup API resource that you can create to initiate an Argo CD backup.

  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: verrazzano-argocd-backup
    namespace: verrazzano-backup
  spec:
    includedNamespaces:
    - argocd
    defaultVolumesToRestic: false
    storageLocation:  verrazzano-backup-location
EOF

The preceding example backs up the Argo CD components:

The defaultVolumesToRestic is set to false so that Velero ignores the associated PVCs.
If the deployed applications refer to a database or persistent volumes, then you need to manually create a backup.

Argo CD backup status

# To display the status of the backup, run the following command
$ kubectl get backup.velero.io -n verrazzano-backup verrazzano-argocd-backup -o yaml

# Sample output showing just the status portion of the backup
status:
  completionTimestamp: ...
  expiration: ...
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 232
    totalItems: 232
  startTimestamp: ...
  version: 1

Argo CD scheduled backups

Velero supports a Schedule API that is a repeatable request that is sent to the Velero server to perform a backup for a given cron notation. After the Schedule object is created, the Velero server will start the backup process. Then, it will wait for the next valid point in the given cron expression and run the backup process on a repeating basis.

Argo CD restore using Velero

To initiate an Argo CD restore operation, first delete the existing Argo CD running on the system and all related data. Make sure that the data you are deleting is not needed. The restore operation will only restore data from the specified backup, and any additional data since that backup will be destroyed by the deletion.

If you restore Argo CD to a different cluster, create a Velero BackupStorageLocation resource in the new cluster that points to the same backup storage location configured in the original cluster. This ensures that the Velero resources created by the original cluster’s backup are automatically synced to the new cluster. Once the sync completes, you will be able to access the backup from the original cluster on the new cluster. It is recommended to configure the BackupStorageLocation on the new cluster as read-only by setting accessMode to ReadOnly in the BackupStorageLocation spec. This ensures that the backup in the object store is not modified from the new cluster. For more information, see Backup Storage Location in the Velero documentation.

Delete the Argo CD components and the namespace.
```
# These are sample commands to demonstrate the Argo CD restore process
$ kubectl delete sts -l app.kubernetes.io/part-of=argocd -n argocd
$ kubectl delete deploy -l app.kubernetes.io/part-of=argocd -n argocd
$ kubectl delete pvc -n argocd <pvc name>
$ kubectl delete ns argocd
```
To perform an Argo CD restore operation, you can invoke the following example Velero Restore API object.

NOTE: For Argo CD, includedNamespaces should list all the namespaces across which the applications are deployed.
```
 $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: verrazzano-argocd-restore
    namespace: verrazzano-backup
  spec:
    backupName: verrazzano-argocd-backup
    includedNamespaces:
    - argocd
EOF
```
The preceding example will restore an Argo CD cluster from an existing backup.
Wait for all the Argo CD pods to be in the RUNNING state.
```
  $ kubectl wait -n argocd --for=condition=ready pod -l app.kubernetes.io/instance=argocd
```
Sample output:
```
    pod/argocd-application-controller-0 condition met
    pod/argocd-applicationset-controller-8489bfbb8-4f686 condition met
    pod/argocd-notifications-controller-c4f5c9684-8qzl8 condition met
    pod/argocd-redis-548968fdd9-4jcrf condition met
    pod/argocd-repo-server-5889c8cc68-5n8j6 condition met
    pod/argocd-server-67b6994987-j9z99 condition met
```

Argo CD restore status

# To display the status of the restore, run the following command
$ kubectl get restore -n verrazzano-backup verrazzano-argocd-restore -o yaml

# Sample output showing just the status portion of the restore
status:
  completionTimestamp: ...
  phase: Completed
  progress:
    itemsRestored: 75
    totalItems: 75
  startTimestamp: ...

Docs: Bug Reports

Mon, 01 Jan 0001 00:00:00 +0000

Use the vz bug-report tool to selectively capture cluster information and create an archive (*.tar.gz) file to help diagnose problems. The archive file helps development and support teams analyze issues and provide recommendations. If a failure occurs while Verrazzano is being installed, upgraded, or uninstalled using the Verrazzano CLI, then the archive file is generated automatically.

CLI setup

To set up the vz command-line tool, follow the steps here.

Use the vz bug-report tool

To create a bug report in a TAR file named my-bug-report.tar.gz:

$ vz bug-report --report-file my-bug-report.tar.gz

We suggest that you review the contents of the bug report before sharing it with support and development teams.

Usage information

Use the following syntax to run vz commands from your terminal window.

$ vz bug-report [flags]

Available options

Command	Definition
`-h, --help`	Help for the `vz bug-report` command.
`-i, --include-namespaces strings`	A comma-separated list of namespaces, in addition to the ones collected by default (system namespaces), for collecting cluster information. This flag can be specified multiple times, such as `--include-namespaces ns1 --include-namespaces ns...`
`-r, --report-file string`	The report file created by the `vz bug-report` command, as a `*.tar.gz` file. Defaults to `bug-report.tar.gz` in the current directory.
`-l --include-logs`	Include logs from the pods in one or more namespaces; this is specified along with the `--include-namespaces` flag.
`-d --duration duration`	The time period during which the logs are collected in seconds, minutes, and hours.
`-v, --verbose`	Enable verbose output.

Available flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`--kubeconfig string`	Path to the kubeconfig file to use.

Examples

Create a bug report file, bugreport.tar.gz, by collecting data from the cluster:
```
$ vz bug-report --report-file bugreport.tar.gz
```
When --report-file is not provided, the command creates bug-report.tar.gz in the current directory.
Create a bug report file, bugreport.tar.gz, including the additional namespace ns1 from the cluster:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1
```
The flag --include-namespaces accepts comma-separated values and can be specified multiple times. For example, the following commands create a bug report by including the additional namespaces ns1, ns2, and ns3:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2,ns3
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-namespaces ns3
```
Use the --include-logs flag to collect the logs from the pods in one or more namespaces, by specifying the --include-namespaces flag. For example, the following command creates a bug report by including the logs from the additional namespaces ns1 and ns2:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs
```
The --duration flag collects logs for the specified time period. The default value is zero (0), which collects the complete pod log. You can specify seconds, minutes, and hours. For example, the following commands create bug reports by including the logs from the additional namespaces ns1 and ns2 during the specified periods of time:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 5m
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 2h
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 300s
```
The values specified for the flag --include-namespaces are case-sensitive.

Docs: Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain.
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self-signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.
Save your signing key pair as a Kubernetes secret.
```
$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key
```
Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure a cert-manager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: Default User Accounts

Mon, 01 Jan 0001 00:00:00 +0000

During installation, Verrazzano generates several default accounts.

System	Account	Secret	Secret Namespace	Description
Keycloak	keycloakadmin	`keycloak-http`	`keycloak`	Keycloak root user: full administrative privileges for Keycloak.
Keycloak	verrazzano	`verrazzano`	`verrazzano-system`	Verrazzano root user: can manage the verrazzano-system realm in Keycloak, including managing users in that realm. This user is a member of the verrazzano-admins group, and, if default role bindings are used, has the verrazzano-admin role.
Rancher	admin	`rancher-admin-secret`	`cattle-system`	Rancher root user: full administrative privileges for Rancher.

Docs: Hello World Helidon

Mon, 01 Jan 0001 00:00:00 +0000

The Hello World Helidon example is a Helidon-based service that returns a “Hello World” response when invoked. The example application is specified using Open Application Model (OAM) component and application configuration YAML files, and then deployed by applying those files.

The example application has two endpoints, which differ in configuration source:

/greet- uses a microprofile properties file. Deploy this application by using the instructions here.
/config- uses a Kubernetes ConfigMap. Deploy this application by using the instructions here.

For more information and the code of this application, see the Verrazzano examples.

Docs: Installation Profiles

Mon, 01 Jan 0001 00:00:00 +0000

This document describes built-in configuration profiles that you can use to simplify a Verrazzano installation. An installation profile is a well-known configuration of Verrazzano settings that can be referenced by name, which then can be customized as needed.

The following table describes the Verrazzano installation profiles.

Profile	Description	Characteristics
`prod`	Full installation, production configuration.	Default profile: - Full installation. - Persistent storage. - Production OpenSearch cluster topology.
`dev`	Development or evaluation configuration.	Lightweight installation: - For evaluation purposes. - No persistence. - Single-node OpenSearch cluster topology.
`managed-cluster`	A specialized installation for managed clusters in a multicluster topology.	Minimal installation for a managed cluster: - Clusters must be registered with an admin cluster to use multicluster features.
`none`	A basic configuration is installed. No components are enabled.	Simplest installation with no components enabled by default.

Use an installation profile

To specify an installation profile when installing Verrazzano, set the profile name in the profile field of your Verrazzano custom resource.

For example, to use the dev profile:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev

To use a different profile, replace dev with prod, managed-cluster or none.

Customize an installation profile

Regardless of the profile, you can override the profile settings for any component. The following example uses a customized dev profile to configure a small 8 Gi persistent volume for the MySQL instance used by Keycloak to provide more stability for the Keycloak service.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-dev-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 8Gi

For details on how to customize Verrazzano components, see Customize an Installation.

Profile configurations

The following table lists the Verrazzano components that are enabled by default with each profile. Note that you can customize any Verrazzano installation regardless of the profile.

Component	dev	prod	managed-cluster
`applicationOperator`	✔️	✔️	✔️
`argoCD`
`authProxy`	✔️	✔️	✔️
`certManager`	✔️	✔️	✔️
`clusterOperator`	✔️	✔️	✔️
`coherenceOperator`	✔️	✔️	✔️
`console`	✔️	✔️
`dns`	✔️	✔️	✔️
`fluentd`	✔️	✔️	✔️
`grafana`	✔️	✔️
`ingressNGINX`	✔️	✔️	✔️
`istio`	✔️	✔️	✔️
`jaegerOperator`
`keycloak`	✔️	✔️
`kiali`	✔️	✔️
`kubeStateMetrics`
`mySQLOperator`	✔️	✔️	✔️
`oam`	✔️	✔️	✔️
`opensearch`	✔️	✔️
`opensearchDashboards`	✔️	✔️
`prometheus`	✔️	✔️	✔️
`prometheusAdapter`
`prometheusNodeExporter`	✔️	✔️	✔️
`prometheusOperator`	✔️	✔️	✔️
`prometheusPushgateway`
`rancher`	✔️	✔️
`rancherBackup`
`velero`
`weblogicOperator`	✔️	✔️	✔️

Prometheus and Grafana configurations

The following table describes the Prometheus and Grafana configurations in each profile.

Profile	Prometheus	Grafana
`prod`	One replica (128 MB memory, 50 Gi storage)	One replica (48 MB memory, 50 Gi storage)
`dev`	One replica (128 MB memory, ephemeral storage)	One replica (48 MB memory, ephemeral storage)
`managed-cluster`	One replica (128 MB memory, 50 Gi storage)	Not installed
`none`	One replica (128 MB memory, ephemeral storage)	Not installed

OpenSearch Dashboards and OpenSearch configurations

The following table describes the OpenSearch Dashboards and OpenSearch cluster topology in each profile.

Profile	OpenSearch	OpenSearch Dashboards
`prod`	Three master replicas (1.4 Gi memory, 50 Gi storage each) One ingest replica (2.5 Gi memory, no storage) Three data replicas (4.8 Gi memory, 50 Gi storage each)	One replica (192 MB memory, ephemeral storage)
`dev`	One master/data/ingest replica (1 Gi memory, ephemeral storage)	One replica (192 MB memory, ephemeral storage)
`managed-cluster`	Not installed	Not installed
`none`	Not installed	Not installed

NOTE

OpenSearch containers are configured to use 75% of the configured request memory for the Java min/max heap settings.

Profile-independent defaults

The following table shows the settings for components that are profile-independent (consistent across all profiles unless overridden).

Component	Default
DNS	Wildcard DNS provider nip.io.
Certificates	Uses the cert-manager self-signed ClusterIssuer for certificates.
Ingress-type	Defaults to `LoadBalancer` service type for the ingress.

For details on how to customize Verrazzano components, see Customize an Installation.

Docs: Keycloak

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano stores user login information in Keycloak. In turn, Keycloak uses MySQL as a back end to store all persistent data. This document shows you how to back up persistent data stored in MySQL from the original cluster and restore it in a new cluster. If you are restoring data to the same cluster, then the terms original cluster and new cluster refer to same cluster.

MySQL Operator prerequisites

MySQL is deployed using the MySQL Operator for Kubernetes. Apart from managing the life cycle of MySQL instances, MySQL Operator provides the capability to back up and restore data using an Amazon S3 compatible object storage.

Before proceeding with a MySQL back up or restore operation, keep the following details handy:

Object storage bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object storage solution.
Object storage prefix name. This will be a child folder under the bucket, which the backup component creates.
Object storage region name.
Object storage signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object storage; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure (OCI), you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

The following example creates a secret mysql-backup-secret in the namespace keycloak. The instructions in this document back up data from MySQL to an Oracle Cloud Object Storage bucket and restore it from there.

MySQL Operator requires a secret to communicate with the S3 compatible object storage, so we create a backup-secret.txt file, which has the object storage credentials.
```
[default]
aws_access_key_id=<object storage access key>
aws_secret_access_key=<object storage secret key>
```
MySQL Operator requires the region name where the bucket is created, so we create a backup-region.txt file, which contains the region information. The following is an example of a backup-region.txt file indicating that the object storage is created in region us-phoenix-1:
```
[default]
region=us-phoenix-1
```
In the namespace keycloak, create a Kubernetes secret, for example mysql-backup-secret.
```
$ kubectl create secret generic --namespace <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file> --from-file=<key>=<full_path_to_config_file>
```
The following is an example to create a Kubernetes secret consisting of credentials to connect to OCI Object Storage.
```
$ kubectl create secret generic --namespace keycloak mysql-backup-secret --from-file=credentials=backup-secret.txt --from-file=config=backup-region.txt
```

NOTE

The secret must be created in the namespace keycloak.
To restore Keycloak on a new cluster, create the secret in the namespace, keycloak, in the new cluster.
To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.

MySQL Operator backup

To initiate a MySQL backup on the original cluster, create the following example custom resource YAML file that uses an OCI Object Storage as a back end. The operator uses the secret referenced in spec.backupProfile.dumpInstance.storage.s3.config to authenticate with the OCI Object Storage.
```
$ kubectl apply -f - <<EOF
    apiVersion: mysql.oracle.com/v2
    kind: MySQLBackup
    metadata:
      name: <backup name>
      namespace: keycloak
    spec:
      clusterName: mysql
      backupProfile:
        name: <backupProfileName>
        dumpInstance:
          storage:
            s3:
              bucketName: <The Object Storage bucket. See the MySQL Operator prerequisites section.>
              config: <Kubernetes secret name. See the MySQL Operator prerequisites section.>
              endpoint: < OCI S3 Object Storage endpoint.>
              prefix: <The prefix name. This folder will be automatically created.>
              profile: default
EOF
```
NOTE
- The config value is mysql-backup-secret, which is the name of the secret that you created previously in the keycloak namespace.
- The clustername has to be mysql.
- The namespace has to be keycloak.
- The profile value is the profile for the security credentials. In this case, it is default.
The following is an example of a MySQLBackup resource to initiate a MySQL backup:
```
$ kubectl apply -f - <<EOF
    apiVersion: mysql.oracle.com/v2
    kind: MySQLBackup
    metadata:
      name: mysql-backup
      namespace: keycloak
    spec:
      clusterName: mysql
      backupProfile:
        name: mysqlOneTime
        dumpInstance:
          storage:
            s3:
              bucketName: mysql-bucket
              config: mysql-backup-secret
              endpoint: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
              prefix: mysql-test
              profile: default
EOF
```
Confirm that the backup operation is complete. Run the following command on the original cluster and ensure that the STATUS is Completed.
```
$ kubectl get MySQLBackup --namespace keycloak
```
```
# Sample output
NAME           CLUSTER   STATUS      OUTPUT                         AGE
mysql-backup   mysql     Completed   mysql-backup-20221025-180836   119s
```
A successful backup of MySQL creates a backup folder in the object storage. Make note of the backup folder prefix name that the MySQL backup created on the original cluster.
```
$ kubectl get mysqlbackup --namespace keycloak <mysql-backup-name> -o jsonpath={.status.output}
```
The following is an example:
```
$ kubectl get mysqlbackup --namespace keycloak mysql-backup -o jsonpath={.status.output}
mysql-backup-20221025-180836
```
Back up MySQL Helm chart and values.

Back up the values in the MySQL Helm chart, in the original cluster to a file, mysql-values.yaml .
```
$ helm get values --namespace keycloak mysql > mysql-values.yaml
```
MySQL Helm charts are present inside the Verrazzano platform operator. Retrieve the charts from the original cluster to a local directory.

The following example retrieves the MySQL charts to a directory mysql-charts under the current directory. In order to avoid data corruption, ensure that the directory, mysql-charts, doesn’t already exist under the current directory.
```
$ kubectl cp --namespace verrazzano-install \
    $(kubectl get pod --namespace verrazzano-install -l app=verrazzano-platform-operator \
    -o custom-columns=:metadata.name --no-headers):platform-operator/thirdparty/charts/mysql \
    -c verrazzano-platform-operator mysql-charts/
```

Scheduled backups

You can also implement schedules for running MYSQL backups. For more information, see the Handling MySQL Backups section, “A PersistentVolumeClaim Scheduled Backup Example.”

MySQL Operator restore

Before you begin, read the MySQL Operator prerequisites. In addition, you must have at least one healthy backup before starting a restore operation.

To initiate a MySQL restore operation from an existing backup, you need to recreate the MySQL cluster. Use the following steps for a successful MySQL restore operation:

Delete the MySQL pods and PersistentVolumeClaim from the system on the new cluster.
```
$ helm delete mysql --namespace keycloak
$ kubectl delete pvc --namespace keycloak -l tier=mysql
```
Start a MySQL restore operation by installing the Helm chart by using the chart from the original cluster.
```
 $ helm install mysql <path to directory mysql-charts, where original charts are extracted> \
         --namespace keycloak \
         --set initDB.dump.name=<dump-name> \
         --set initDB.dumpOptions.loadUsers=true \
         --set initDB.dump.s3.profile=default \
         --set initDB.dump.s3.prefix=<prefixName/backup folder name> \
         --set initDB.dump.s3.bucketName=<OCI bucket name> \
         --set initDB.dump.s3.config=<Kubernetes secret name, see MySQL Operator prerequisites section.> \
         --set initDB.dump.s3.endpoint=<OCI S3 endpoint> \
         --values <mysql values file>
```
The following is an example:
```
 $ helm install mysql mysql-charts \
         --namespace keycloak \
         --set initDB.dump.name=alpha \
         --set initDB.dump.s3.profile=default \
         --set initDB.dump.s3.prefix=mysql-test/mysql-backup-20221025-180836 \
         --set initDB.dump.s3.bucketName=mysql-bucket \
         --set initDB.dump.s3.config=mysql-backup-secret \
         --set initDB.dump.s3.endpoint=https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com \
         --values mysql-values.yaml
```
After performing the restore command, wait for the MySQL cluster to be online. Ensure that the STATUS is ONLINE and the count under ONLINE matches the INSTANCES.
```
$ kubectl get innodbclusters --namespace keycloak mysql
```
```
# Sample output
NAME    STATUS   ONLINE   INSTANCES   ROUTERS   AGE
mysql   ONLINE   3        3           3         2m23s
```
Wait for all the MySQL pods to be in the RUNNING state.
```
$ kubectl wait --namespace keycloak --for=condition=ready pod -l tier=mysql --timeout=600s
```
```
# Sample output
pod/mysql-0 condition met
pod/mysql-1 condition met
pod/mysql-2 condition met
pod/mysql-router-746d9d75c7-6pc5p condition met
pod/mysql-router-746d9d75c7-bhrkw condition met
pod/mysql-router-746d9d75c7-t8bhb condition met
```
At this point, the MySQL cluster has been restored successfully from the backup, along with the PersistentVolumeClaim that was deleted previously.
If you are restoring Keycloak on a new cluster, then update the Keycloak secret.

On the original cluster, if you are restoring Keycloak on a new cluster, then run the following command for the keycloak-http secret in keycloak namespace:
```
 $ kubectl get secret --namespace keycloak keycloak-http -o jsonpath={.data.password}; echo
```
On the new cluster, replace the existing password value with the value displayed from the previous command.
```
 kubectl patch secret keycloak-http --namespace keycloak -p '{"data": {"password": "<password displayed in the step above>"}}'
```
Restart the Keycloak pods.

The removal and recreation of the MySQL cluster may bring down the Keycloak pods because MySQL goes offline during the restore operation. Run the following commands to restart the Keycloak pods:
```
 KEYCLOAK_REPLICAS=$(kubectl get sts --namespace keycloak keycloak -o custom-columns=:status.replicas --no-headers)
 kubectl scale sts --namespace keycloak keycloak --replicas=0
 kubectl scale sts --namespace keycloak keycloak --replicas=${KEYCLOAK_REPLICAS}
 kubectl wait --namespace keycloak --for=condition=ready pod -l app.kubernetes.io/instance=keycloak --timeout=600s
```

Update Verrazzano secrets in the new cluster

The following steps are applicable only if you are restoring Keycloak on a new cluster.

After you complete the MySQL restore operation, the password for the following secrets in the verrazzano-system namespace must be updated in the new cluster:

verrazzano
verrazzano-es-internal
verrazzano-prom-internal

On the original cluster, run the following command for the verrazzano secret:
```
 $ kubectl get secret --namespace verrazzano-system verrazzano -o jsonpath={.data.password}; echo
```
On the new cluster, replace the existing password value with the value displayed in step 1.
```
 kubectl patch secret verrazzano --namespace verrazzano-system -p '{"data": {"password": "<password displayed in step 1>"}}'
```
Repeat steps 1 and 2 for the verrazzano-es-internal and verrazzano-prom-internal secrets.
Restart the fluentd pods in the new cluster to use the original cluster password to connect to OpenSearch.
```
 $ kubectl delete pod -l app=fluentd --namespace verrazzano-system
```

Docs: Kind

Mon, 01 Jan 0001 00:00:00 +0000

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. Follow these instructions to prepare a Kind cluster for running Verrazzano.

NOTE

Kind is not recommended for use on macOS and Windows because the Docker network is not directly exposed to the host.

Prerequisites

Install Docker
Install Kind

Prepare the Kind cluster

To prepare the Kind cluster for use with Verrazzano, you must create the cluster and then install and configure MetalLB in that cluster.

You can create the Kind cluster in two ways: with or without image caching; image caching can speed up your installation time.

Create a Kind cluster

Kind images are prebuilt for each release. To find images suitable for a given release, check the release notes for your Kind version (check with kind version). There you’ll find a complete listing of images created for a Kind release.

The following example references a Kubernetes v1.21.1-based image built for Kind v0.11.1. Replace that image with one suitable for the Kind release you are using. For the supported Kubernetes versions, see the listing here.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

Create a Kind cluster with image caching

While developing or experimenting with Verrazzano, you might destroy and re-create your Kind cluster multiple times. To speed up Verrazzano installation, follow these steps to ensure that the image cache used by containerd inside a Kind cluster, is preserved across clusters. Subsequent installations will be faster because they will not need to pull the images again.

1. Create a named Docker volume that will be used for the image cache and note its mountPoint path. In this example, the volume is named containerd.

$ docker volume create containerd

$ docker volume inspect containerd

# Sample output
{
    "CreatedAt": "2021-01-11T16:27:47Z",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/containerd/_data",
    "Name": "containerd",
    "Options": {},
    "Scope": "local"
}

2. Specify the mountPoint path obtained, as the hostPath under extraMounts in your Kind configuration file, with a containerPath of /var/lib/containerd, which is the default containerd image caching location inside the Kind container. An example of the modified Kind configuration is shown in the following create cluster command.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
    extraMounts:
      - hostPath: /var/lib/docker/volumes/containerd/_data
        containerPath: /var/lib/containerd #This is the location of the image cache inside the Kind container
EOF

NOTE: When using a private container registry or experiencing rate-limiting of container image pulls, refer to the Kind documentation.

Install and configure MetalLB

By default, Kind does not provide an implementation of network load balancers (Services of type LoadBalancer). MetalLB offers a network load balancer implementation.

To install MetalLB:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
$ kubectl create secret generic \
    -n metallb-system memberlist \
    --from-literal=secretkey="$(openssl rand -base64 128)"
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

Wait for MetalLB to be ready, as shown:

$ kubectl get pods -n metallb-system -L app=metallb
NAME                         READY   STATUS    RESTARTS       AGE   APP=METALLB
controller-d9d8c78b6-hhplg   1/1     Running   1 (3d4h ago)   9d    
speaker-4lqpg                1/1     Running   1 (3d4h ago)   9d

For further details, see the MetalLB installation guide.

MetalLB is idle until configured. Configure MetalLB in Layer 2 mode and give it control over a range of IP addresses in the kind Docker network. In versions v0.7.0 and earlier, Kind uses Docker’s default bridge network; in versions v0.8.0 and later, it creates its own bridge network in Kind.

To determine the subnet of the kind Docker network in Kind v0.8.0 and later:

$ docker inspect kind | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.18.0.0/16

To determine the subnet of the kind Docker network in Kind v0.7.0 and earlier:

$ docker inspect bridge | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.17.0.0/16

For use by MetalLB, assign a range of IP addresses at the end of the kind network’s subnet CIDR range.

$ kubectl apply -f - <<-EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: my-ip-space
      protocol: layer2
      addresses:
      - 172.18.0.230-172.18.0.250
EOF

Next steps

To continue, see the Installation Guide.

Docs: Multicluster and Verrazzano Project

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

clusters.verrazzano.io/v1alpha1

clusters.verrazzano.io/v1alpha1

Resource Types:

MultiClusterApplicationConfiguration

MultiClusterApplicationConfiguration specifies the multicluster application API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterApplicationConfiguration

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterApplicationConfigurationSpec

The desired state of a multicluster application resource.

`placement` Placement	Clusters in which the application is to be created.
`secrets` []string	(Optional) List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.
`template` ApplicationConfigurationTemplate	Template containing the metadata and spec for an OAM applicationConfiguration resource.

status
MultiClusterResourceStatus

The observed state of a multicluster application resource.

MultiClusterComponent

MultiClusterComponent specifies the MultiCluster Component API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterComponent

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterComponentSpec

The desired state of a MultiCluster Component resource.

`placement` Placement	Clusters in which the component is to be created.
`template` ComponentTemplate	Template containing the metadata and spec for an OAM component.

status
MultiClusterResourceStatus

The observed state of a MultiCluster Component resource.

MultiClusterConfigMap

MultiClusterConfigMap specifies the MultiCluster ConfigMap API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterConfigMap

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterConfigMapSpec

The desired state of a MultiCluster ConfigMap resource.

`placement` Placement	Clusters in which the ConfigMap is to be created.
`template` ConfigMapTemplate	The embedded Kubernetes ConfigMap.

status
MultiClusterResourceStatus

The observed state of a MultiCluster ConfigMap resource.

MultiClusterSecret

MultiClusterSecret specifies the MultiCluster Secret API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterSecret

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterSecretSpec

The desired state of a MultiCluster Secret resource.

`placement` Placement	Clusters in which the secret is to be created.
`template` SecretTemplate	The embedded Kubernetes secret.

status
MultiClusterResourceStatus

The observed state of a MultiCluster Secret resource.

VerrazzanoProject

VerrazzanoProject specifies the Verrazzano Projects API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string VerrazzanoProject

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoProjectSpec

The desired state of a Verrazzano Project resource.

`placement` Placement	Clusters on which the namespaces are to be created.
`template` ProjectTemplate	The project template.

status
MultiClusterResourceStatus

The observed state of a Verrazzano Project resource.

ApplicationConfigurationTemplate

(Appears on: MultiClusterApplicationConfigurationSpec)

ApplicationConfigurationTemplate has the metadata and embedded spec of the OAM applicationConfiguration resource.

Field Description

metadata
EmbeddedObjectMeta

Metadata describing the application.

spec
OAM core/v1alpha2.ApplicationConfigurationSpec

The embedded OAM application specification.

components
[]OAM core/v1alpha2.ApplicationConfigurationComponent

Components of which this ApplicationConfiguration consists. Each component will be used to instantiate a workload.

Cluster

(Appears on: Placement)

Cluster contains the name of a single cluster.

Field	Description
`name` string	The name of a cluster.

ClusterLevelStatus

(Appears on: MultiClusterResourceStatus)

ClusterLevelStatus describes the status of the multicluster resource in a specific cluster.

Field	Description
`lastUpdateTime` string	Last update time of the resource state in this cluster.
`message` string	Message details about the status in this cluster.
`name` string	Name of the cluster.
`state` StateType	State of the resource in this cluster.

ComponentTemplate

(Appears on: MultiClusterComponentSpec)

ComponentTemplate has the metadata and embedded spec of the OAM component.

Field Description

metadata
EmbeddedObjectMeta

Metadata describing the component.

spec
OAM core/v1alpha2.ComponentSpec

The embedded OAM component specification.

`workload` Kubernetes runtime.RawExtension	A Workload that will be created for each ApplicationConfiguration that includes this Component. Workload is an instance of a workloadDefinition. We either use the GVK info or a special “type” field in the workload to associate the content of the workload with its workloadDefinition
`parameters` []OAM core/v1alpha2.ComponentParameter	(Optional) Parameters exposed by this component. ApplicationConfigurations that reference this component may specify values for these parameters, which will in turn be injected into the embedded workload.

Condition

(Appears on: MultiClusterResourceStatus)

Condition describes current state of a multicluster resource.

Field	Description
`lastTransitionTime` string	(Optional) Last time the condition transitioned from one status to another.
`message` string	(Optional) A message with details about the last transition.
`status` Kubernetes core/v1.ConditionStatus	Status of the condition: one of `True`, `False`, or `Unknown`.
`type` ConditionType	Type of condition.

ConditionType (`string` alias)

(Appears on: Condition)

ConditionType identifies the condition of the multicluster resource which can be checked with kubectl wait.

ConfigMapTemplate

(Appears on: MultiClusterConfigMapSpec)

ConfigMapTemplate has the metadata and spec of the Kubernetes ConfigMap.

Field	Description
`binaryData` map[string][]byte	Corresponds to the `binaryData` field of the `struct` ConfigMap defined in types.go.
`data` map[string]string	Corresponds to the `data` field of the `struct` ConfigMap defined in types.go.
`immutable` bool	Corresponds to the `immutable` field of the `struct` ConfigMap defined in types.go.
`metadata` EmbeddedObjectMeta	Metadata describing the ConfigMap.

EmbeddedObjectMeta

(Appears on: ApplicationConfigurationTemplate, ComponentTemplate, ConfigMapTemplate, SecretTemplate)

EmbeddedObjectMeta is metadata describing a resource.

Field	Description
`annotations` map[string]string	(Optional) Annotations for the resource.
`labels` map[string]string	(Optional) Labels for the resource.
`name` string	(Optional) Name of the resource.
`namespace` string	(Optional) Namespace of the resource.

MultiClusterApplicationConfigurationSpec

(Appears on: MultiClusterApplicationConfiguration)

MultiClusterApplicationConfigurationSpec defines the desired state of a multicluster application.

Field	Description
`placement` Placement	Clusters in which the application is to be created.
`secrets` []string	(Optional) List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.
`template` ApplicationConfigurationTemplate	Template containing the metadata and spec for an OAM applicationConfiguration resource.

MultiClusterComponentSpec

(Appears on: MultiClusterComponent)

MultiClusterComponentSpec defines the desired state of a MultiCluster Component.

Field	Description
`placement` Placement	Clusters in which the component is to be created.
`template` ComponentTemplate	Template containing the metadata and spec for an OAM component.

MultiClusterConfigMapSpec

(Appears on: MultiClusterConfigMap)

MultiClusterConfigMapSpec defines the desired state of a MultiCluster ConfigMap.

Field	Description
`placement` Placement	Clusters in which the ConfigMap is to be created.
`template` ConfigMapTemplate	The embedded Kubernetes ConfigMap.

MultiClusterResourceStatus

(Appears on: MultiClusterApplicationConfiguration, MultiClusterComponent, MultiClusterConfigMap, MultiClusterSecret, VerrazzanoProject)

MultiClusterResourceStatus is the runtime status of a multicluster resource.

Field Description

Field	Description
`clusters` []ClusterLevelStatus	Status information for each cluster.
`conditions` []Condition	The current state of a multicluster resource.
`state` StateType	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Failed`: deployment to cluster failed `Pending`: deployment to cluster is in progress `Succeeded`: deployment to cluster successfully completed

clusters
[]ClusterLevelStatus

Status information for each cluster.

conditions
[]Condition

The current state of a multicluster resource.

state
StateType

The state of the multicluster resource. State values are case-sensitive and formatted as follows:

Failed: deployment to cluster failed
Pending: deployment to cluster is in progress
Succeeded: deployment to cluster successfully completed

MultiClusterSecretSpec

(Appears on: MultiClusterSecret)

MultiClusterSecretSpec defines the desired state of a MultiCluster Secret.

Field	Description
`placement` Placement	Clusters in which the secret is to be created.
`template` SecretTemplate	The embedded Kubernetes secret.

NamespaceTemplate

(Appears on: ProjectTemplate)

NamespaceTemplate contains the metadata and specification of a Kubernetes namespace.

Field Description

metadata
Kubernetes meta/v1.ObjectMeta (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
Kubernetes core/v1.NamespaceSpec

The specification of a namespace.

finalizers
[]Kubernetes core/v1.FinalizerName

(Optional)

Finalizers is an opaque list of values that must be empty to permanently remove object from storage. More info: https://kubernetes.io/docs/tasks/administer-cluster/namespaces/

NetworkPolicyTemplate

(Appears on: ProjectTemplate)

NetworkPolicyTemplate contains the metadata and specification of a Kubernetes NetworkPolicy.

NOTE

To add an application NetworkPolicy, see NetworkPolicies for applications.

Field Description

metadata
Kubernetes meta/v1.ObjectMeta (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
Kubernetes networking/v1.NetworkPolicySpec

The specification of a network policy.

`podSelector` Kubernetes meta/v1.LabelSelector	Selects the pods to which this NetworkPolicy object applies. The array of ingress rules is applied to any pods selected by this field. Multiple network policies can select the same set of pods. In this case, the ingress rules for each are combined additively. This field is NOT optional and follows standard label selector semantics. An empty podSelector matches all pods in this namespace.
`ingress` []Kubernetes networking/v1.NetworkPolicyIngressRule	(Optional) List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod’s local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default)
`egress` []Kubernetes networking/v1.NetworkPolicyEgressRule	(Optional) List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
`policyTypes` []Kubernetes networking/v1.PolicyType	(Optional) List of rule types that the NetworkPolicy relates to. Valid options are [“Ingress”], [“Egress”], or [“Ingress”, “Egress”]. If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ “Egress” ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include “Egress” (since such a policy would not include an Egress section and would otherwise default to just [ “Ingress” ]). This field is beta-level in 1.8

Placement

(Appears on: MultiClusterApplicationConfigurationSpec, MultiClusterComponentSpec, MultiClusterConfigMapSpec, MultiClusterSecretSpec, VerrazzanoProjectSpec)

Placement contains the name of each cluster where a resource will be located.

Field	Description
`clusters` []Cluster	List of clusters.

ProjectTemplate

(Appears on: VerrazzanoProjectSpec)

ProjectTemplate contains the list of namespaces to create and the optional security configuration for each namespace.

Field	Description
`namespaces` []NamespaceTemplate	The list of application namespaces to create for this project.
`networkPolicies` []NetworkPolicyTemplate	(Optional) Network policies applied to namespaces in the project.
`security` SecuritySpec	(Optional) The project security configuration.

SecretTemplate

(Appears on: MultiClusterSecretSpec)

SecretTemplate has the metadata and spec of the Kubernetes Secret.

Field	Description
`data` map[string][]byte	Corresponds to the data field of the struct Secret defined in types.go.
`metadata` EmbeddedObjectMeta	Metadata describing the secret.
`stringData` map[string]string	Corresponds to the `stringData` field of the `struct` Secret defined in types.go.
`type` Kubernetes core/v1.SecretType	The type of secret.

SecuritySpec

(Appears on: ProjectTemplate)

SecuritySpec defines the security configuration for a Verrazzano Project.

Field	Description
`projectAdminSubjects` []Kubernetes rbac/v1.Subject	(Optional) The subjects to bind to the `verrazzano-project-admin` role.
`projectMonitorSubjects` []Kubernetes rbac/v1.Subject	(Optional) The subjects to bind to the `verrazzano-project-monitoring` role.

StateType (`string` alias)

(Appears on: ClusterLevelStatus, MultiClusterResourceStatus)

StateType identifies the state of a multicluster resource.

VerrazzanoProjectSpec

(Appears on: VerrazzanoProject)

VerrazzanoProjectSpec defines the desired state of a Verrazzano Project.

Field	Description
`placement` Placement	Clusters on which the namespaces are to be created.
`template` ProjectTemplate	The project template.

Generated with gen-crd-api-reference-docs

Docs: Network Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano manages and secures network traffic between Verrazzano system components and deployed applications. Verrazzano does not manage or secure traffic for the Kubernetes cluster itself, or for non-Verrazzano services or applications running in the cluster. Traffic is secured at two levels in the network stack:

ISO Layer 3/4: Using NetworkPolicies to control IP access to Pods.
ISO Layer 6: Using TLS and mutual TLS authentication (mTLS) to provide authentication, confidentiality, and integrity for connections within the cluster and for external connections.

NetworkPolicies

By default, all Pods in a Kubernetes cluster have network access to all other Pods in the cluster. Kubernetes has a NetworkPolicy resource that provides network level 3 and 4 security for Pods, restricting both ingress and egress IP traffic for a set of Pods in a namespace. Verrazzano configures all system components with NetworkPolicies to control ingress. Egress is not restricted.

NOTE: A NetworkPolicy resource needs a NetworkPolicy controller to implement the policy, otherwise the policy has no effect. You must install a Kubernetes Container Network Interface (CNI) plug-in that provides a NetworkPolicy controller, such as Calico, before installing Verrazzano, or else the policies are ignored.

NetworkPolicies for system components

Verrazzano installs a set of NetworkPolicies for system components to control ingress into the Pods. A policy is scoped to a namespace and uses selectors to specify the Pods that the policy applies to, along with the ingress and egress rules. For example, the following policy applies to the Verrazzano API Pod in the verrazzano-system namespace. This policy allows network traffic from NGINX Ingress Controller on port 8775 and from Prometheus on port 15090. No other Pods can reach those ports or any other ports of the Verrazzano API Pod. Notice that namespace selectors need to be used; the NetworkPolicy resource does not support specifying the namespace name.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
...
spec:
  PodSelector:
    matchLabels:
      app: verrazzano-api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: ingress-nginx
      PodSelector:
        matchLabels:
          app.kubernetes.io/instance: ingress-controller
    ports:
    - port: 8775
      protocol: TCP
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      PodSelector:
        matchLabels:
          app: system-prometheus
    ports:
    - port: 15090
      protocol: TCP

The following table shows all of the ingresses that allow network traffic into system components. The ports shown are Pod ports, which is what NetworkPolicies require.

Component	Pod Port	From	Description
Argo CD	443	NGINX Ingress	Access from external client
Argo CD	8080	Argo CD Server, Internal	Argo CD Server data port
cert-manager	9402	Prometheus	Prometheus scraping
Coherence Operator	9443	Prometheus	Webhook entrypoint
Istio control plane	15012	Envoy	Envoy access to `istiod`
Istio control plane	15014	Prometheus	Prometheus scraping.
Istio control plane	15017	Kubernetes API Server	Webhook entrypoint
Istio egress gateway	8443	Mesh services	Application egress
Istio egress gateway	15090	Prometheus	Prometheus scraping
Istio ingress gateway	8443	External	Application ingress
Istio ingress gateway	15090	Prometheus	Prometheus scraping
Keycloak	8080	NGINX Ingress	Access from external client
Keycloak	15090	Prometheus	Prometheus scraping
MySql	15090	Prometheus	Prometheus scraping
MySql	3306	Keycloak	Keycloak datastore
Node exporter	9100	Prometheus	Prometheus scraping
OpenSearch	8775	Fluentd	Access from Fluentd
OpenSearch	9200	OpenSearch Dashboards, Internal	OpenSearch data port
OpenSearch	9300	Internal	OpenSearch cluster port
OpenSearch	15090	Prometheus	Envoy metrics scraping
OpenSearch	8775	NGINX Ingress	Access from external client
Prometheus	8775	NGINX Ingress	Access from external client
Prometheus	9090	Grafana	Access for Grafana console
Rancher	80	NGINX Ingress	Access from external client
Rancher	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Application Operator	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Authentication Proxy	8775	NGINX Ingress	Access from external client
Verrazzano Authentication Proxy	15090	Prometheus	Prometheus scraping
Verrazzano Console	8000	NGINX Ingress	Access from external client
Verrazzano Console	15090	Prometheus	Prometheus scraping
Verrazzano Platform Operator	9443	Kubernetes API Server	Webhook entrypoint

NetworkPolicies for applications

By default, applications do not have NetworkPolicies that restrict ingress into the application or egress from it. You can configure them for the application namespaces using the NetworkPolicy section of a Verrazzano project.

NOTE

Verrazzano requires specific ingress to and egress from application pods. If you add a NetworkPolicy for your application namespace or pods, you must add an additional policy to ensure that Verrazzano still has the required access it needs. The ingress policy is needed only if you restrict ingress. Likewise, the egress policy is needed only if you restrict egress. The following are the ingress and egress NetworkPolicies:

Ingress NetworkPolicies

  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-ingressgateway
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: system-prometheus
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: weblogic-operator

Egress NetworkPolicies

  egress:
  - ports:
    - port: 15012
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-egressgateway
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: kube-system
  - ports:
    - port: 8000
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator

NetworkPolicies for Envoy sidecar proxies

As mentioned, Envoy sidecar proxies run in both system component pods and application pods. Each proxy sends requests to the Istio control plane pod, istiod, for a variety of reasons. During installation, Verrazzano creates a NetworkPolicy named istiod-access in the istio-system namespace to give ingress to system component and application sidecar proxies.

Mutual TLS authentication (mTLS)

Istio can be enabled to use mTLS between services in the mesh, and also between the Istio gateways and Envoy sidecar proxies. There are various options to customize mTLS usage, for example it can be disabled on a per-port level. The Istio control plane, Istiod, is a CA and provides key and certificate rotation for the Envoy proxies, both gateways and sidecars.

Verrazzano configures Istio to have strict mTLS for the mesh. All components and applications put into the mesh will use mTLS, with the exception of Coherence clusters, which are not in the mesh. Also, all traffic between the Istio ingress gateway and mesh sidecars use mTLS, and the same is true between the proxy sidecars and the egress gateway.

Verrazzano sets up mTLS during installation with the PeerAuthentication resource as follows:

apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  ...
  spec:
    mtls:
      mode: STRICT

TLS

TLS is used by external clients to access the cluster, both through the NGINX Ingress Controller and the Istio ingress gateway. The certificate used by these TLS connections vary; see Verrazzano security for details. All TLS connections are terminated at the ingress proxy. Traffic between the two proxies and the internal cluster Pods always uses mTLS, because those Pods are all in the Istio mesh.

Istio mesh

Istio provides extensive security protection for both authentication and authorization, as described in Istio Security. Access control and mTLS are two security features that Verrazzano configures. These security features are available in the context of a service mesh.

A service mesh is an infrastructure layer that provides certain capabilities like security, observability, load balancing, and such, for services. Istio defines a service mesh here. In the context of Istio on Kubernetes, a service in the mesh is a Kubernetes Service. Consider the Bob’s Books example application, which has several OAM Components defined. At runtime, there is a Kubernetes Service for each component, and each Service is in the mesh, with one or more Pods associated with the service. All services in the mesh have an Envoy proxy in front of their Pods, intercepting network traffic to and from the Pod. In Kubernetes, that proxy happens to be a sidecar running in each Pod.

There are various ways to put a service in the mesh. Verrazzano uses the namespace label, istio-injection: enabled, to designate that all Pods in a given namespace are in the mesh. When a Pod is created in that namespace, the Istio control plane mutating webhook, changes the Pod spec to add the Envoy proxy sidecar container, causing the Pod to be in the mesh.

Disabling sidecar injection

In certain cases, Verrazzano needs to disable sidecar injection for specific Pods in a namespace. This is done in two ways: first, during installation, Verrazzano modifies the istio-sidecar-injector ConfigMap using a Helm override file for the Istio chart. This excludes several components from the mesh, such as the Verrazzano application operator. Second, certain Pods, such as Coherence Pods, are labeled at runtime with sidecar.istio.io/inject="false" to exclude them from the mesh.

Components in the mesh

The following Verrazzano components are in the mesh and use mTLS for all service to service communication.

Argo CD
Fluentd
Grafana
Kiali
Keycloak
MySQL
NGINX Ingress Controller
OpenSearch
OpenSearch Dashboards
Prometheus
Verrazzano Authentication Proxy
Verrazzano Console
WebLogic Kubernetes Operator

Some of these components, have mesh-related details that are worth noting, as described in the following sections.

NGINX

The NGINX Ingress Controller listens for HTTPS traffic, and provides ingress into the cluster. NGINX is configured to do TLS termination of client connections. All traffic from NGINX to the mesh services use mTLS, which means that traffic is fully encrypted from the client to the target back-end services.

Keycloak and MySQL

Keycloak and MySQL are also in the mesh and use mTLS for network traffic. Because all of the components that use Keycloak are in the mesh, there is end to end mTLS security for all identity management handled by Keycloak. The following components access Keycloak:

Verrazzano Authentication Proxy
Verrazzano Console
OpenSearch
Prometheus
Grafana
Kiali
OpenSearch Dashboards

Prometheus

Although Prometheus is in the mesh, it is configured to use the Envoy sidecar and mTLS only when communicating with Keycloak. All the traffic related to scraping metrics, bypasses the sidecar proxy, doesn’t use the service IP address, but rather connects to the scrape target using the Pod IP address. If the scrape target is in the mesh, then HTTPS is used; otherwise, HTTP is used. For Verrazzano multicluster, Prometheus also connects from the admin cluster to the Prometheus server in the managed cluster by using the managed cluster NGINX Ingress, using HTTPS. Prometheus is in the managed cluster and never establishes connections to targets outside the cluster.

Because Prometheus is in the mesh, additional configuration is done to allow the Envoy sidecar to be bypassed when scraping Pods. This is done with the Prometheus Pod annotation traffic.sidecar.istio.io/includeOutboundIPRanges: <keycloak-service-ip>. This causes traffic bound for Keycloak to go through the Envoy sidecar, and all other traffic to bypass the sidecar.

WebLogic Kubernetes Operator

Applications in the mesh

Before you create a Verrazzano application, you should decide if it should be in the mesh. You control sidecar injection, for example, mesh inclusion, by labeling the application namespace with istio-injection=enabled or istio-injection=disabled. By default, applications will not be put in the mesh if that label is missing. If your application uses a Verrazzano project, then Verrazzano will label the namespaces in the project to enable injection. If the application is in the mesh, then mTLS will be used. You can change the PeerAuthentication mTLS mode as desired if you don’t want strict mTLS. Also, if you need to add mTLS port exceptions, you can do this with DestinationRules or by creating another PeerAuthentication resource in the application namespace. Consult the Istio documentation for more information.

WebLogic

When the WebLogic Kubernetes Operator creates a domain, it needs to communicate with the Pods in the domain. Verrazzano puts the operator in the mesh so that it can communicate with the domain Pods using mTLS. Because of that, the WebLogic domain must be created in the mesh. Also, because mTLS is used, do not configure WebLogic to use TLS. If you want to use a custom certificate for your application, you can specify that in the ApplicationConfiguration, but that TLS connection will be terminated at the Istio ingress gateway, which you configure using a Verrazzano IngressTrait.

Coherence

Coherence clusters are represented by the Coherence resource, and are not in the mesh. When Verrazzano creates a Coherence cluster in a namespace that is annotated to do sidecar injection, it disables injection of the Coherence resource using the sidecar.istio.io/inject="false" label shown previously. Furthermore, Verrazzano will create a DestinationRule in the application namespace to disable mTLS for the Coherence extend port 9000. This allows a service in the mesh to call the Coherence extend proxy. For an example, see Bobs Books.

Here is an example of a DestinationRule created for the Bob’s Books application which includes a Coherence cluster.

API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
...
Spec:
  Host:  *.bobs-books.svc.cluster.local
  Traffic Policy:
    Port Level Settings:
      Port:
        Number:  9000
      Tls:
    Tls:
      Mode:  ISTIO_MUTUAL

Istio access control

Istio lets you control access to your workload in the mesh using the AuthorizationPolicy resource. This lets you control which services or Pods can access your workloads. Some of these options require mTLS; for more information, see Authorization Policy.

Verrazzano always creates AuthorizationPolicies for applications but never for system components. During application deployment, Verrazzano creates the policy in the application namespace and configures it to allow access from the following:

Other Pods in the application
Istio ingress gateway
Prometheus scraper

This prevents other Pods in the cluster from gaining network access to the application Pods. Istio uses a service identity to determine the identity of the request’s origin; for Kubernetes this identity is a service account. Verrazzano creates a per-application AuthorizationPolicy as follows:

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
...
spec:
  rules:
    - from:
    - source:
  principals:
    - cluster.local/ns/sales/sa/greeter
    - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
    - cluster.local/ns/verrazzano-system/sa/verrazzano-monitoring-operator

WebLogic domain access

For WebLogic applications, the WebLogic Kubernetes Operator must have access to the domain Pods for two reasons. First, it must access the domain servers to get health status; second, it must inject configuration into the Monitoring Exporter sidecar running in the domain server Pods. When a WebLogic domain is created, Verrazzano adds an additional source, cluster.local/ns/verrazzano-system/sa/weblogic-operator-sa to the principals section to permit that access.

Docs: Node Failure Guide

Mon, 01 Jan 0001 00:00:00 +0000

A Node failure can occur for many reasons, including hardware failures and network outages. This guide provides information about what to expect when a Node failure occurs and how to recover from a Node failure. Recovery depends on the Storage Provisioner and the type of storage that you use.

NOTE

This guide assumes that the storage provided in the cluster is physically separate from the Node and is recoverable. It does not apply to a local storage on the Node.

What to expect

By default, when a Node fails:

It may take up to a minute for the failure to reflect in the Kubernetes API server and update the Node status to NotReady.
After about five minutes of the Node status being NotReady, the status of the Pods on that Node will be changed to Unknown or NodeLost.
The status of the Pods with controllers, like Daemonsets, Statefulsets, and Deployments, will be changed to Terminating.

NOTE: Pods without a controller, started with a PodSpec, will not be terminated. They must be manually deleted and recreated.
New Pods will start on the Nodes that remain with Ready status.

NOTE: Statefulsets are a special case. The Statefulset controller maintains an ordinal list of Pods, one each for a given name. The Statefulset controller will not start a new Pod with the name of an existing Pod.
The Pods that have associated Persistent Volumes of type ReadWriteOnce, do not become Ready. This is because the Pods try to attach to the existing volumes that are still attached to the old Pod, which is still Terminating. This happens because, at a given point, the Persistent Volumes of type ReadWriteOnce, can be associated only with a single Node, and the new Pod resides on another Node.
If multiple Availability Domains are used in the Kubernetes cluster and the failed Node is the last in that Availability Domain, then the existing volumes will no longer be reachable by new Pods in a separate Availability Domain.

About recovery

After a Node fails, if the Node can be recovered within five minutes, then the Pods will return to a Running state. If the Node is not recovered after five minutes, then the Pods will complete termination and are deleted from the Kubernetes API server. New Pods that have Persistent Volumes of type ReadWriteOnce, will now be able to mount the Persistent Volumes and change to Running.

If a Node cannot be recovered and is replaced, then deleting the Node from the Kubernetes API server will terminate the old Pods and release the Persistent Volumes of type ReadWriteOnce, to be mounted by any new Pods.

If multiple Availability Domains are used in the Kubernetes cluster, then the replacement Node should be added to the same Availability Domain that the deleted Node occupied. This allows the Pods to be scheduled on the replacement Node that can reach the Persistent Volumes in that Availability Domain, and then the Pod status is changed to Running.

Do not forcefully delete Pods or Persistent Volumes in a failed Node that you plan to recover or replace. If you force delete Pods or Persistent Volumes in a failed Node, it may lead to loss of data and, in the case of Statefulsets, it may lead to split-brain scenarios. For more information about Statefulsets, see Force Delete StatefulSet Pods in the Kubernetes documentation.

You can force delete Pods and Persistent Volumes when a failed Node cannot be recovered or replaced in the same Availability Domain as the original Node.

Docs: OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides a ready-to-use, OpenSearch deployment that gives you access to all the log messages from various microservices running on the platform. There are scenarios where you may want to back up your OpenSearch data and restore it.

Verrazzano uses Velero to facilitate backing up and restoring OpenSearch data.

Velero operator prerequisites

Before proceeding with an OpenSearch backup or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore OpenSearch, you must first enable Velero.

The following configuration shows you how to enable velero with a prod installation profile.
```
  $ kubectl apply -f -<<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: prod
      components:    
        velero:
          enabled: true
EOF
```

NOTE: The OpenSearch back up and restore operation is supported only on prod installation profiles with a multinode OpenSearch configuration.

After Velero is enabled, verify that the Velero pods are running in the verrazzano-backup namespace.
```
# Sample of pods running after enabling the velero component
$ kubectl get pod -n verrazzano-backup
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h
```
Velero requires a secret to communicate with the S3 compatible object store, so we create a backup-secret.txt file, which has the object store credentials.
```
[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>
```
In the namespace verrazzano-backup, create a Kubernetes secret, for example verrazzano-backup-creds.
```
$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file>
```
The following is an example:
```
$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt
```
NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.
Create a BackupStorageLocation resource, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example. For more information, see Backup Storage Location in the Velero documentation.
```
 $ kubectl apply -f -<<EOF
 apiVersion: velero.io/v1
 kind: BackupStorageLocation
 metadata:
   name: verrazzano-backup-location
   namespace: verrazzano-backup
 spec:
   provider: aws
   objectStorage:
     bucket: example-verrazzano
     prefix: backup-demo
   credential:
     name: verrazzano-backup-creds
     key: cloud
   config:
     region: us-phoenix-1
     s3ForcePathStyle: "true"
     s3Url: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```

OpenSearch backup Using Velero

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero while invoking a backup. Due to the nature of transient data handled by OpenSearch, the hook invokes the OpenSearch snapshot APIs to back up data streams appropriately, thereby ensuring that there is no loss of data and avoids data corruption as well.

The following example shows a sample Velero Backup API resource that you can create to initiate an OpenSearch backup.

  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: verrazzano-opensearch-backup
    namespace: verrazzano-backup
  spec:
    includedNamespaces:
      - verrazzano-system
    labelSelector:
      matchLabels:
        verrazzano-component: opensearch
    defaultVolumesToRestic: false
    storageLocation:  verrazzano-backup-location
    hooks:
      resources:
        - name: opensearch-backup-test
          includedNamespaces:
            - verrazzano-system
          labelSelector:
            matchLabels:
              statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
          post:                           
            - exec:
                container: es-master
                command:
                  - /usr/share/opensearch/bin/verrazzano-backup-hook
                  - -operation
                  - backup
                  - -velero-backup-name
                  - verrazzano-opensearch-backup
                onError: Fail
                timeout: 10m
EOF

The preceding example backs up the OpenSearch components:

In this case, you are not backing up the PersistentVolumes directly, rather running a hook that invokes the OpenSearch APIs to take a snapshot of the data.
The defaultVolumesToRestic is set to false so that Velero ignores the associated PVCs.
In this case, the hook can be pre or post.
The command used in the hook requires an operation flag and the Velero backup name as an input.
The container on which the hook needs to be run defaults to the first container in the pod. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

After the backup is processed, you can see the hook logs using the velero backup logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch backup logs

# To display the logs from the backup, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-backup-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

OpenSearch scheduled backups

OpenSearch restore using Velero

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero to perform a restore operation. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to restore data streams appropriately, thereby ensuring there is no loss of data and avoids data corruption as well.

If you restore OpenSearch to a different cluster, create a Velero BackupStorageLocation resource in the new cluster that points to the same backup storage location configured in the original cluster. This ensures that the Velero resources created by the original cluster’s backup are automatically synced to the new cluster. Once the sync completes, you will be able to access the backup from the original cluster on the new cluster. It is recommended to configure the BackupStorageLocation on the new cluster as read-only by setting accessMode to ReadOnly in the BackupStorageLocation spec. This ensures that the backup in the object store is not modified from the new cluster. For more information, see Backup Storage Location in the Velero documentation.

To initiate an OpenSearch restore operation, first delete the existing OpenSearch cluster running on the system and all related data.

Scale down the Verrazzano Monitoring Operator. This is required because the operator manages the life cycle of the OpenSearch cluster, so scaling it down to zero ensures that it does not interfere with the restore operation. The restore operation also ensures that this operator is scaled back up to return the system to its previous state.
```
 $ kubectl scale deploy -n verrazzano-system verrazzano-monitoring-operator --replicas=0
```
Delete the OpenSearch components.
```
# These are sample commands to demonstrate the OpenSearch restore process
$ kubectl delete sts -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete deploy -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete pvc -n verrazzano-system -l verrazzano-component=opensearch
```
To perform an OpenSearch restore operation, you can invoke the following example Velero Restore API object.
```
  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: verrazzano-opensearch-restore
    namespace: verrazzano-backup
  spec:
    backupName: verrazzano-opensearch-backup
    includedNamespaces:
      - verrazzano-system
    labelSelector:
      matchLabels:
        verrazzano-component: opensearch
    restorePVs: false
    hooks:
      resources:
      - name: opensearch-test
        includedNamespaces:
        - verrazzano-system       
        labelSelector:
          matchLabels:            
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        postHooks:
        - exec:
            container: es-master
            command:
            - /usr/share/opensearch/bin/verrazzano-backup-hook
            - -operation
            - restore
            - -velero-backup-name
            - verrazzano-opensearch-backup
            waitTimeout: 30m
            execTimeout: 30m
            onError: Fail
EOF
```
The preceding example will restore an OpenSearch cluster from an existing backup.
- In this case, you are not restoring PersistentVolumes directly, rather running a hook that invokes the OpenSearch APIs to restore them from an existing snapshot of the data.
- The restorePVs is set to false so that Velero ignores restoring PVCs.
- The command used in the hook requires an -operation flag and the Velero backup name as an input.
- The postHook will invoke the OpenSearch APIs that restore the snapshot data.
- The container on which the hook needs to be run defaults to the first container in the pod. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.
NOTE: The hook needs to be a postHook because it must be applied after the Kubernetes objects are restored.
Wait for all the OpenSearch pods to be in the RUNNING state.
```
  $ kubectl wait -n verrazzano-system --for=condition=ready pod -l verrazzano-component=opensearch --timeout=600s
    pod/vmi-system-es-data-0-6f49bdf6f5-fc6mz condition met
    pod/vmi-system-es-data-1-8f8785994-4pr7n condition met
    pod/vmi-system-es-data-2-d5f569d98-q8p2v condition met
    pod/vmi-system-es-ingest-6ddd86b9b6-fpl6j condition met
    pod/vmi-system-es-ingest-6ddd86b9b6-jtmrh condition met
    pod/vmi-system-es-master-0 condition met
    pod/vmi-system-es-master-1 condition met
    pod/vmi-system-es-master-2 condition met
```

After the restore operation is processed, you can see the hook logs using the velero restore logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch restore logs

# To display the logs from the restore, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-restore-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

Docs: Rancher

Mon, 01 Jan 0001 00:00:00 +0000

Rancher maintains many configurations, like user credentials and cluster credentials, as ConfigMaps and namespace values. The rancher-backup Operator provides a seamless way to back up and restore Rancher installations, configuration, and data.

rancher-backup Operator prerequisites

Before proceeding with a Rancher back up or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore Rancher, you must first enable rancherBackup.

The following configuration shows you how to enable rancherBackup.
```
$ kubectl apply -f -<<EOF
 apiVersion: install.verrazzano.io/v1beta1
  kind: Verrazzano
 metadata:
    name: example-verrazzano
  spec:
   profile: dev
    components:    
     rancherBackup:
        enabled: true
EOF
```
For rancher-backup, the pods will be created in the cattle-resources-system namespace.
```
  # Sample of pods running after enabling the rancherBackup component

  $ kubectl get pod -n cattle-resources-system
  NAME                              READY   STATUS    RESTARTS   AGE
  rancher-backup-5c4b985697-xw7md   1/1     Running   0          2d4h
```
Rancher requires a secret to communicate with the S3 compatible object store. So, in the namespace verrazzano-backup, create a Kubernetes secret rancher-backup-creds.
```
 $ kubectl create secret generic -n <backup-namespace> <secret-name> --from-literal=accessKey=<accesskey> --from-literal=secretKey=<secretKey>
```
The following is an example:
```
 $ kubectl create secret generic -n verrazzano-backup rancher-backup-creds --from-literal=accessKey="s5VLpXwa0xNZQds4UTVV" --from-literal=secretKey="nFFpvyxpQvb0dIQovsl0"
```

Rancher backup

The rancher-backup Operator creates the backup file, in *.tar.gz format, on the S3 compatible object store.

To initiate a Rancher backup, create the following example custom resource YAML file that uses an Amazon S3 compatible object store as a back end. The operator uses the credentialSecretNamespace value to determine where to look for the Amazon S3 backup secret.
```
 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Backup
   metadata:
     name: <rancher-backup-name>
   spec:
     storageLocation:
       s3:
         credentialSecretName: <rancher backup credential name>
         credentialSecretNamespace: <namespace where credential object was created>
         bucketName: <object store bucket. This must be exist as noted in pre-requisites section>
         folder: <folder name. This folder will be auto created>
         region: <region name where bucket exists>
         endpoint: <object store endpoint configuration>
     resourceSetName: rancher-resource-set
EOF
```

NOTE: In Step 3. of the example in the prerequisites section, you created the secret in the verrazzano-backup namespace.

The following is an example:

 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Backup
   metadata:
     name: rancher-backup-test
   spec:
      storageLocation:
       s3:
         credentialSecretName: rancher-backup-creds
         credentialSecretNamespace: verrazzano-backup
         bucketName: myvz-bucket
         folder: rancher-backup
         region: us-phoenix-1
         endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
     resourceSetName: rancher-resource-set
EOF

The *.tar.gz file is stored in a location configured in the storageLocation field. When the backup is complete, then the rancher-backup Operator creates a file on the S3 compatible object store.

You can retrieve the backed up file name, as shown:
```
 $ kubectl get backups.resources.cattle.io rancher-backup-test
   NAME                 LOCATION   TYPE       LATEST-BACKUP                                                                     RESOURCESET            AGE   STATUS
   rancher-backup-test             One-time   rancher-615034-957d182d-44cb-4b81-bbe0-466900049124-2022-11-14T16-42-28Z.tar.gz   rancher-resource-set   54s   Completed
```

Rancher scheduled backups

To implement scheduled Rancher backups, see Backup Configuration in the Rancher documentation.

Rancher restore

During the restore operation, Rancher ensures that it recreates all the CRDs related to Rancher and configurations. Restoring Rancher is done by creating a custom resource that indicates to rancherBackup to start the restore process.

To initiate a Rancher restore operation, create the following example custom resource YAML file. When a Restore custom resource is created, the operator accesses the backup *.tar.gz file specified and restores the application data from that file.
```
 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Restore
   metadata:
     name: s3-restore
   spec:
     backupFilename: rancher-615034-957d182d-44cb-4b81-bbe0-466900049124-2022-11-14T16-42-28Z.tar.gz
     storageLocation:
       s3:
         credentialSecretName: rancher-backup-creds
         credentialSecretNamespace: verrazzano-backup
         bucketName: myvz-bucket
         folder: rancher-backup
         region: us-phoenix-1
         endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```
The rancher-backup Operator scales down the Rancher deployment during the restore operation and scales it back up after the restoration completes.

Resources are restored in this order:
1. Custom Resource Definitions (CRDs)
2. Cluster-scoped resources
3. Namespace resources
NOTE: The backupFilename is retrieved from the Rancher backup created previously.
Wait for all the Rancher pods to be in the RUNNING state.
```
$ kubectl wait -n cattle-system --for=condition=ready pod -l app=rancher --timeout=600s
  pod/rancher-69976cffc6-bbx4p condition met
  pod/rancher-69976cffc6-fr75t condition met
  pod/rancher-69976cffc6-pcdf2 condition met
```

Docs: Register Managed Clusters Using kubectl

Mon, 01 Jan 0001 00:00:00 +0000

To register managed clusters using the VerrazzanoManagedCluster resource, complete the following steps:

Create the environment variables, KUBECONFIG_ADMIN, KUBECONTEXT_ADMIN, KUBECONFIG_MANAGED1, and KUBECONTEXT_MANAGED1, and point them to the kubeconfig files and contexts for the admin and managed cluster, respectively. You will use these environment variables in subsequent steps when registering the managed cluster. The following shows an example of how to set these environment variables.
```
$ export KUBECONFIG_ADMIN=/path/to/your/adminclusterkubeconfig
$ export KUBECONFIG_MANAGED1=/path/to/your/managedclusterkubeconfig

# Lists the contexts in each kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN config get-contexts -o=name
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 config get-contexts -o=name

# Choose the right context name for your admin and managed clusters from the output shown and set the KUBECONTEXT
# environment variables
$ export KUBECONTEXT_ADMIN=<admin-cluster-context-name>
$ export KUBECONTEXT_MANAGED1=<managed-cluster-context-name>
```
To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster resource on the admin cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster resource"
EOF
```
Wait for the VerrazzanoManagedCluster resource to reach the Ready status. At that point, it will have generated a YAML file that must be applied on the managed cluster to complete the registration process.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc
```
Export the YAML file created to register the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml
```
Apply the registration file exported in the previous step, on the managed cluster.
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register.yaml

# After the command succeeds, you may delete the register.yaml file
$ rm register.yaml
```

Docs: Thanos

Mon, 01 Jan 0001 00:00:00 +0000

Thanos is a group of components that seamlessly integrate with Prometheus to monitor your applications. You can enable and configure Thanos components with Verrazzano and use long-term storage to store metrics. By using the Thanos console, you can query for metrics across all Prometheus instances and long-term storage. Thanos also makes it easier to scale Prometheus horizontally and obtain a global view of data from multiple Prometheus servers.

Advantages of using Thanos:

Long-term metrics retention
High availability
Easy backup for metrics
Efficient data access

For more information on Thanos, see the Thanos website.

Thanos components

Verrazzano currently supports the following Thanos components:

Components	Description
Sidecar	Container that resides in the Prometheus pod. It connects to Prometheus, reads its data for queries, and uploads it to long-term storage.
Store Gateway	Serves metrics from long-term storage.
Query	Implements Prometheus API to aggregate data from the underlying components and provides a user interface for querying across all Prometheus stores.
Query Frontend	Implements Prometheus API and proxies it to Query while caching the response and optionally splits queries.

Enable Thanos

To enable Thanos in Verrazzano, add the following:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  components:
    prometheusOperator:
      enabled: true
      overrides:
      - values:
          prometheus:
            thanos:
              integration: sidecar
    thanos:
      enabled: true

Enable long-term storage

Optionally, to enable long-term storage of metrics complete the following steps:

Step 1: Create a YAML configuration file

Create a storage.yaml file using OCI object storage.

type: OCI
config:
  provider: "raw"
  bucket: "<bucket name>"
  compartment_ocid: "<bucket compartment OCID>"
  tenancy_ocid: "<tenancy OCID>"
  user_ocid: "<user OCID>"
  region: "<region>"
  fingerprint: ""
  privatekey: ""

Step 2: Create a secret

Create the secret for object storage configuration.

The following example uses the file name storage.yaml. The Thanos Store Gateway requires the key in the secret to be objstore.yml.

$ kubectl create namespace verrazzano-monitoring
$ kubectl create secret generic objstore-config -n verrazzano-monitoring --from-file=objstore.yml=storage.yaml

Step 3: Enable storage and Thanos Store Gateway

The following example enables storage, creates the required secret, and enables Thanos Store Gateway. It also configures the Thanos Sidecar to write to object storage and the Store Gateway to read from object storage.

Note: objstore-config is the secret that you created in Step 2.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  components:
    prometheusOperator:
      enabled: true
      overrides:
      - values:
          prometheus:
            thanos:
              integration: sidecar
            prometheusSpec:
              thanos:
                objectStorageConfig:
                  name: objstore-config
                  key: objstore.yml
    thanos:
      enabled: true
      overrides:
      - values:
          existingObjstoreSecret: objstore-config
          storegateway:
            enabled: true

Access the Thanos Query console

You can access the Thanos Query console using the instructions at Get the consoles URLs.

Docs: Traits and Workloads

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

oam.verrazzano.io/v1alpha1

oam.verrazzano.io/v1alpha1

Resource Types:

IngressTrait

IngressTrait specifies the ingress traits API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string IngressTrait

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
IngressTraitSpec

`rules` []IngressRule	A list of ingress rules for an ingress trait.
`tls` IngressSecurity	(Optional) The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

status
IngressTraitStatus

The observed state of an ingress trait and related resources.

LoggingTrait

LoggingTrait specifies the logging traits API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string LoggingTrait

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
LoggingTraitSpec

`imagePullPolicy` string	(Optional) The optional image pull policy for the Fluentd image provided by the user.
`loggingConfig` string	The configuration provided by the user for the Fluentd configuration that consists of fluentd.conf: `<source>\n ... and so on ...\n`.
`loggingImage` string	(Optional) The name of the custom Fluentd image.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

status
LoggingTraitStatus

The observed state of a logging trait and related resources.

MetricsTrait

MetricsTrait specifies the metrics trait API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string MetricsTrait

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MetricsTraitSpec

`enabled` bool	(Optional) Specifies whether metrics collection is enabled. Defaults to `true`.
`path` string	(Optional) The HTTP path for the related metrics endpoint. Defaults to `/metrics`.
`port` int	(Optional) The HTTP port for the related metrics trait. Defaults to `8080`.
`ports` []PortSpec	(Optional) The HTTP endpoints for the related metrics.
`scraper` string	(Optional) The Prometheus deployment used to scrape the related metrics endpoints. By default, the Verrazzano-supplied Prometheus component is used to scrape the endpoint.
`secret` string	(Optional) The name of an opaque secret (for example, `username` and `password`) within the workload’s namespace for metrics endpoint access.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

status
MetricsTraitStatus

The observed state of a metrics trait and related resources.

VerrazzanoCoherenceWorkload

VerrazzanoCoherenceWorkload specifies the Verrazzano Coherence workload API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string VerrazzanoCoherenceWorkload

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoCoherenceWorkloadSpec

The desired state of a Verrazzano Coherence workload.

template
Kubernetes runtime.RawExtension

The metadata and spec for the underlying Coherence resource.

status
VerrazzanoCoherenceWorkloadStatus

The observed state of a Verrazzano Coherence workload.

VerrazzanoHelidonWorkload

VerrazzanoHelidonWorkload specifies the Verrazzano Helidon workload API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string VerrazzanoHelidonWorkload

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoHelidonWorkloadSpec

The desired state of a Verrazzano Helidon workload.

`deploymentTemplate` DeploymentTemplate	An embedded Helidon application deployment.
`serviceTemplate` ServiceTemplate	An embedded Helidon application service.

status
VerrazzanoHelidonWorkloadStatus

The observed state of a Verrazzano Helidon workload.

VerrazzanoWebLogicWorkload

VerrazzanoWebLogicWorkload specifies the Verrazzano WebLogic workload API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string VerrazzanoWebLogicWorkload

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoWebLogicWorkloadSpec

The desired state of a Verrazzano WebLogic workload.

`template` VerrazzanoWebLogicWorkloadTemplate	The metadata and spec for the underlying Domain resource.
`clusters` []VerrazzanoWebLogicWorkloadTemplate

status
VerrazzanoWebLogicWorkloadStatus

The observed state of a Verrazzano WebLogic workload.

AuthorizationPolicy

(Appears on: IngressPath)

AuthorizationPolicy defines the set of rules for authorizing a request.

Field	Description
`rules` []github.com/verrazzano/verrazzano/application-operator/apis/oam/v1alpha1.AuthorizationRule*	Rules are used to match requests from request principals to specific paths given an optional list of conditions.

AuthorizationRule

AuthorizationRule matches requests from a list of request principals that access a specific path subject to a list of conditions.

Field	Description
`from` AuthorizationRuleFrom	Specifies the request principals for access to a request. An asterisk (*) will match when the value is not empty, for example, if any request principal is found in the request.
`when` []github.com/verrazzano/verrazzano/application-operator/apis/oam/v1alpha1.AuthorizationRuleCondition*	(Optional) Specifies a list of additional conditions for access to a request.

AuthorizationRuleCondition

AuthorizationRuleCondition provides additional required attributes for authorization.

Field	Description
`key` string	The name of a request attribute.
`values` []string	A list of allowed values for the attribute.

AuthorizationRuleFrom

(Appears on: AuthorizationRule)

AuthorizationRuleFrom includes a list of request principals.

Field	Description
`requestPrincipals` []string	Specifies the request principals for access to a request.

DeploymentTemplate

(Appears on: VerrazzanoHelidonWorkloadSpec)

DeploymentTemplate specifies the metadata and pod spec of a Helidon workload.

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	Metadata about a Helidon application. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`podSpec` Kubernetes core/v1.PodSpec	The pod spec of a Helidon application.
`selector` Kubernetes meta/v1.LabelSelector	(Optional) Label selector of a Helidon application.
`strategy` Kubernetes apps/v1.DeploymentStrategy	(Optional) The replacement strategy of a Helidon application.

IngressDestination

(Appears on: IngressRule)

IngressDestination specifies a specific destination host and port for the ingress paths.

NOTE

If there are multiple ports defined for a service, then the destination port must be specified OR the service port name must have the prefix http.

Field	Description
`host` string	(Optional) Destination host.
`httpCookie` IngressDestinationHTTPCookie	(Optional) Session affinity cookie.
`port` uint32	(Optional) Destination port.

IngressDestinationHTTPCookie

(Appears on: IngressDestination)

IngressDestinationHTTPCookie specifies a session affinity cookie for an ingress trait.

Field	Description
`name` string	(Optional) The name of the HTTP cookie.
`path` string	(Optional) The path of the HTTP cookie.
`ttl` time.Duration	(Optional) The lifetime of the HTTP cookie (in seconds).

IngressPath

(Appears on: IngressRule)

IngressPath specifies a specific path to be exposed for an ingress trait.

Field Description

path
string

(Optional)

If no path is provided, then it defaults to forward slash (/).

pathType
string

(Optional)

Path type values are case-sensitive and formatted as follows:

exact: exact string match
prefix: prefix-based match
regex: regex-based match

Defaults to prefix if path specified is /; otherwise, defaults to exact.

authorizationPolicy
AuthorizationPolicy

(Optional)

Defines the set of rules for authorizing a request.

IngressRule

(Appears on: IngressTraitSpec)

IngressRule specifies a rule for an ingress trait.

Field	Description
`destination` IngressDestination	(Optional) The destination host and port for the ingress paths.
`hosts` []string	(Optional) One or more hosts exposed by the ingress trait. Wildcard hosts or hosts that are empty are filtered out. If there are no valid hosts provided, then a DNS host name is automatically generated and used.
`paths` []IngressPath	The paths to be exposed for an ingress trait.

IngressSecurity

(Appears on: IngressTraitSpec)

IngressSecurity specifies the secret containing the certificate securing the transport for an ingress trait.

Field	Description
`secretName` string	The name of a secret containing the certificate securing the transport. The specification of a secret here implies that a certificate was created for specific hosts, as specified in an IngressRule.

IngressTraitSpec

(Appears on: IngressTrait)

IngressTraitSpec specifies the desired state of an ingress trait.

Field	Description
`rules` []IngressRule	A list of ingress rules for an ingress trait.
`tls` IngressSecurity	(Optional) The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

IngressTraitStatus

(Appears on: IngressTrait)

IngressTraitStatus specifies the observed state of an ingress trait and related resources.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this ingress trait.

resources
[]OAM common/v1.TypedReference

The resources managed by this ingress trait.

LoggingTraitSpec

(Appears on: LoggingTrait)

LoggingTraitSpec specifies the desired state of a logging trait.

Field	Description
`imagePullPolicy` string	(Optional) The optional image pull policy for the Fluentd image provided by the user.
`loggingConfig` string	The configuration provided by the user for the Fluentd configuration that consists of fluentd.conf: `<source>\n ... and so on ...\n`.
`loggingImage` string	(Optional) The name of the custom Fluentd image.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

LoggingTraitStatus

(Appears on: LoggingTrait)

LoggingTraitStatus specifies the observed state of a logging trait and related resources.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this logging trait.

resources
[]OAM common/v1.TypedReference

The resources managed by this logging trait.

MetricsTraitSpec

(Appears on: MetricsTrait)

MetricsTraitSpec specifies the desired state of a metrics trait.

Field	Description
`enabled` bool	(Optional) Specifies whether metrics collection is enabled. Defaults to `true`.
`path` string	(Optional) The HTTP path for the related metrics endpoint. Defaults to `/metrics`.
`port` int	(Optional) The HTTP port for the related metrics trait. Defaults to `8080`.
`ports` []PortSpec	(Optional) The HTTP endpoints for the related metrics.
`scraper` string	(Optional) The Prometheus deployment used to scrape the related metrics endpoints. By default, the Verrazzano-supplied Prometheus component is used to scrape the endpoint.
`secret` string	(Optional) The name of an opaque secret (for example, `username` and `password`) within the workload’s namespace for metrics endpoint access.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

MetricsTraitStatus

(Appears on: MetricsTrait)

MetricsTraitStatus defines the observed state of a metrics trait and related resources.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this metrics trait.

resources
[]QualifiedResourceRelation

Related resources affected by this metrics trait.

PortSpec

(Appears on: MetricsTraitSpec)

PortSpec defines an HTTP port and path combination.

Field	Description
`path` string	(Optional) The HTTP path for the related metrics endpoint. Defaults to `/metrics`.
`port` int	(Optional) The HTTP port for the related metrics trait. Defaults to `8080`.

QualifiedResourceRelation

(Appears on: MetricsTraitStatus, VerrazzanoHelidonWorkloadStatus)

QualifiedResourceRelation identifies a specific related resource.

Field	Description
`apiversion` string	API version of the related resource.
`kind` string	Kind of the related resource.
`name` string	Name of the related resource.
`namespace` string	Namespace of the related resource.
`role` string	Role of the related resource, for example, `Deployment`.

ServiceTemplate

(Appears on: VerrazzanoHelidonWorkloadSpec)

ServiceTemplate specifies the metadata and pod spec of a Helidon workload.

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	(Optional) Metadata about a Helidon application. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`serviceSpec` Kubernetes core/v1.ServiceSpec	(Optional) The service spec of a Helidon application.

VerrazzanoCoherenceWorkloadSpec

(Appears on: VerrazzanoCoherenceWorkload)

VerrazzanoCoherenceWorkloadSpec wraps a Coherence resource.

Field	Description
`template` Kubernetes runtime.RawExtension	The metadata and spec for the underlying Coherence resource.

VerrazzanoCoherenceWorkloadStatus

(Appears on: VerrazzanoCoherenceWorkload)

VerrazzanoCoherenceWorkloadStatus defines the observed state of a Verrazzano Coherence workload.

Field	Description
`lastGeneration` string	The last generation of the Verrazzano Coherence workload that was reconciled.
`lastRestartVersion` string	The last value of the `verrazzano.io/restart-version` annotation.

VerrazzanoHelidonWorkloadSpec

(Appears on: VerrazzanoHelidonWorkload)

VerrazzanoHelidonWorkloadSpec wraps a Helidon application deployment and service.

Field	Description
`deploymentTemplate` DeploymentTemplate	An embedded Helidon application deployment.
`serviceTemplate` ServiceTemplate	An embedded Helidon application service.

VerrazzanoHelidonWorkloadStatus

(Appears on: VerrazzanoHelidonWorkload)

VerrazzanoHelidonWorkloadStatus defines the observed state of Verrazzano Helidon workload.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this Verrazzano Helidon workload.

resources
[]QualifiedResourceRelation

The resources managed by this Verrazzano Helidon workload.

VerrazzanoWebLogicWorkloadSpec

(Appears on: VerrazzanoWebLogicWorkload)

VerrazzanoWebLogicWorkloadSpec wraps a WebLogic resource. The WebLogic domain specified in the template must contain a spec field and it may include a metadata field.

Field	Description
`template` VerrazzanoWebLogicWorkloadTemplate	The metadata and spec for the underlying Domain resource.
`clusters` []VerrazzanoWebLogicWorkloadTemplate

VerrazzanoWebLogicWorkloadStatus

(Appears on: VerrazzanoWebLogicWorkload)

VerrazzanoWebLogicWorkloadStatus defines the observed state of a Verrazzano WebLogic workload.

Field	Description
`lastGeneration` string	The last generation of the Verrazzano WebLogic workload that was reconciled.
`lastRestartVersion` string	The last value of the `verrazzano.io/restart-version` annotation.
`lastLifecycleAction` string	The last value of the `verrazzano.io/lifecycle-action`.

VerrazzanoWebLogicWorkloadTemplate

(Appears on: VerrazzanoWebLogicWorkloadSpec)

Field Description

apiVersion
string

metadata
Kubernetes runtime.RawExtension

spec
Kubernetes runtime.RawExtension

-
[]byte

Raw is the underlying serialization of this object.

TODO: Determine how to detect ContentType and ContentEncoding of ‘Raw’ data.

-
Kubernetes runtime.Object

Object can hold a representation of this extension - useful for working with versioned structs.

Generated with gen-crd-api-reference-docs

Docs: Perform the Uninstall

Mon, 01 Jan 0001 00:00:00 +0000

You can uninstall Verrazzano using the Verrazzano CLI or with kubectl.

See the following respective sections:

Uninstall using the Verrazzano CLI

Uninstall Verrazzano.
```
 $ vz uninstall
```
Wait for the uninstall operation to complete. The uninstall logs from the Verrazzano platform operator will be streamed to the command window until the uninstall operation has completed or until the default timeout (20m) has been reached.

The following is an example of the output:
```
Uninstalling Verrazzano
2022-11-22T16:31:20.377Z info Reconciling Verrazzano resource default/verrazzano, generation 2, version 1.4.2
2022-11-22T16:31:20.377Z info Deleting Verrazzano installation
2022-11-22T16:31:20.418Z info Uninstalling components
2022-11-22T16:31:20.418Z info Uninstalling Verrazzano default/verrazzano
...
```

Uninstall using kubectl

To delete a Verrazzano installation, delete the Verrazzano custom resource you used to install it into your cluster.

The following example starts a deletion of a Verrazzano installation in the background and then uses the kubectl logs -f command to tail the output of the pod performing the uninstall operation.

Get the name of the Verrazzano custom resource.
```
$ MYVZ=$(kubectl  get vz -o jsonpath="{.items[0].metadata.name}")
```
Delete the Verrazzano custom resource. After the custom resource is deleted, the Verrazzano platform operator completes the uninstall process by removing all of the Verrazzano-related components and resources, which can take some time.
```
$ kubectl delete verrazzano $MYVZ
```
To view the logs during deletion, use the following command:
```
$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'
```

For troubleshooting help, see Analysis Advice.

Docs: Verify Multicluster Installation

Mon, 01 Jan 0001 00:00:00 +0000

Verify that managed cluster registration has completed

You can perform all the verification steps on the admin cluster.

Verify that the managed cluster can connect to the admin cluster. View the status of the VerrazzanoManagedCluster resource on the admin cluster, and check whether the lastAgentConnectTime, prometheusHost, and apiUrl fields are populated. This may take up to two minutes after completing the registration steps.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get vmc managed1 -n verrazzano-mc -o yaml

# Sample output showing the status field
spec:
  ....
  ....
status:
  apiUrl: https://verrazzano.default.172.18.0.211.nip.io
  conditions:
  - lastTransitionTime: "2021-07-07T15:49:43Z"
    message: Ready
    status: "True"
    type: Ready
  lastAgentConnectTime: "2021-07-16T14:47:25Z"
  prometheusHost: prometheus.vmi.system.default.172.18.0.211.nip.io
```
Verify that the managed cluster is successfully registered with Rancher. When you perform the registration steps, Verrazzano also registers the managed cluster with Rancher. View the Rancher console on the admin cluster. If the registration with Rancher was successful, then your cluster will be listed in Rancher’s list of clusters, and will be in the Active state. You can find the Rancher console URL for your cluster by following the instructions for Accessing Verrazzano.

Verify that managed cluster metrics are being collected

Verify that the admin cluster is collecting metrics from the managed cluster. The Prometheus output will include records that contain the name of the Verrazzano cluster (labeled as verrazzano_cluster).

You can find the Prometheus console URL for your cluster by following the instructions for Accessing Verrazzano. Run a query for a metric (for example, node_disk_io_time_seconds_total).

Sample output of a Prometheus query

An alternative approach to using the Prometheus console is to query metrics from the command line. Here is an example of how to obtain Prometheus metrics from the command line. Search the output of the query for responses that have the verrazzano_cluster field set to the name of the managed cluster.

# On the admin cluster
$ prometheusUrl=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.prometheusUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} "${prometheusUrl}/api/v1/query?query=node_disk_io_time_seconds_total"

Verify that managed cluster logs are being collected

Verify that the admin cluster is collecting logs from the managed cluster. The output will include records which have the name of the managed cluster in the cluster_name field.

You can find the OpenSearch Dashboards URL for your cluster by following the instructions for Accessing Verrazzano. Searching the verrazzano-system data stream for log records with the cluster_name set to the managed cluster name yields logs for the managed cluster.

Sample output of a OpenSearch Dashboards screen

An alternative approach to using the OpenSearch Dashboards is to query OpenSearch from the command line. Here is an example of how to obtain log records from the command line. Search the output of the query for responses that have the cluster_name field set to the name of the managed cluster.

# On the admin cluster
$ OS_URL=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.openSearchUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} -X POST -H 'kbn-xsrf: true' "${OS_URL}/verrazzano-system/_search?size=25"

Docs: Verrazzano Managed Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

clusters.verrazzano.io/v1alpha1

clusters.verrazzano.io/v1alpha1

Resource Types:

VerrazzanoManagedCluster

VerrazzanoManagedCluster

VerrazzanoManagedCluster specifies the Verrazzano Managed Cluster API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string VerrazzanoManagedCluster

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoManagedClusterSpec

The desired state of a Verrazzano Managed Cluster resource.

`caSecret` string	The name of a Secret that contains the CA certificate of the managed cluster. This is used to configure the admin cluster to scrape metrics from the Prometheus endpoint on the managed cluster. See the pre-registration instructions for how to create this Secret.
`description` string	(Optional) The description of the managed cluster.
`managedClusterManifestSecret` string	(Optional) The name of the Secret containing the generated YAML manifest file to be applied by the user to the managed cluster. This field is managed by a Verrazzano Kubernetes operator.
`serviceAccount` string	(Optional) The name of the ServiceAccount that was generated for the managed cluster. This field is managed by a Verrazzano Kubernetes operator.

status
VerrazzanoManagedClusterStatus

The observed state of a Verrazzano Managed Cluster resource.

ArgoCDRegistration

(Appears on: VerrazzanoManagedClusterStatus)

ArgoCDRegistration defines the Argo CD registration state for a managed cluster.

Field	Description
`status` ArgoCDRegistrationStatus	The status of the ArgoCD registration.
`lastSetTimestamp` Kubernetes meta/v1.Time	(Optional) The timestamp of last status set.
`message` string	(Optional) Supporting message related to the Argo CD registration status.

ArgoCDRegistrationStatus (`string` alias)

(Appears on: ArgoCDRegistration)

ArgoCDRegistrationStatus identifies the status of an Argo CD registration.

Condition

(Appears on: VerrazzanoManagedClusterStatus)

Condition describes a condition that occurred on the Verrazzano Managed Cluster.

Field	Description
`lastTransitionTime` Kubernetes meta/v1.Time	(Optional) Last time the condition transitioned from one status to another.
`message` string	(Optional) A message with details about the last transition.
`status` Kubernetes core/v1.ConditionStatus	Status of the condition: one of `True`, `False`, or `Unknown`.
`type` ConditionType	The condition of the multicluster resource which can be checked with a `kubectl wait` command. Condition values are case-sensitive and formatted as follows: `Ready`: the VerrazzanoManagedCluster is ready to be used and all resources needed have been generated.

ConditionType (`string` alias)

(Appears on: Condition)

ConditionType identifies the condition of the Verrazzano Managed Cluster which can be checked with kubectl wait.

RancherRegistration

(Appears on: VerrazzanoManagedClusterStatus)

RancherRegistration defines the Rancher registration state for a managed cluster.

Field	Description
`clusterID` string	The Rancher cluster ID for this cluster.
`message` string	(Optional) A supporting message related to the Rancher registration status.
`status` RancherRegistrationStatus	The status of the Rancher registration.

RancherRegistrationStatus (`string` alias)

(Appears on: RancherRegistration)

RancherRegistrationStatus identifies the status of a Rancher registration.

StateType (`string` alias)

(Appears on: VerrazzanoManagedClusterStatus)

StateType identifies the state of the Verrazzano Managed Cluster.

VerrazzanoManagedClusterSpec

(Appears on: VerrazzanoManagedCluster)

VerrazzanoManagedClusterSpec defines the desired state of a Verrazzano Managed Cluster.

Field	Description
`caSecret` string	The name of a Secret that contains the CA certificate of the managed cluster. This is used to configure the admin cluster to scrape metrics from the Prometheus endpoint on the managed cluster. See the pre-registration instructions for how to create this Secret.
`description` string	(Optional) The description of the managed cluster.
`managedClusterManifestSecret` string	(Optional) The name of the Secret containing the generated YAML manifest file to be applied by the user to the managed cluster. This field is managed by a Verrazzano Kubernetes operator.
`serviceAccount` string	(Optional) The name of the ServiceAccount that was generated for the managed cluster. This field is managed by a Verrazzano Kubernetes operator.

VerrazzanoManagedClusterStatus

(Appears on: VerrazzanoManagedCluster)

VerrazzanoManagedClusterStatus defines the observed state of a Verrazzano Managed Cluster.

Field	Description
`apiUrl` string	The Verrazzano API server URL for this managed cluster.
`conditions` []Condition	The current state of this managed cluster.
`lastAgentConnectTime` Kubernetes meta/v1.Time	The last time the agent from this managed cluster connected to the admin cluster.
`prometheusHost` string	The Prometheus host for this managed cluster.
`thanosQueryStore` string	The Thanos Query Store API host name for this managed cluster.
`rancherRegistration` RancherRegistration	The state of Rancher registration for this managed cluster.
`argoCDRegistration` ArgoCDRegistration	The state of ArgoCD registration for this managed cluster.
`state` StateType	The state of this managed cluster.

Generated with gen-crd-api-reference-docs

Docs: Verrazzano v1alpha1

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

install.verrazzano.io/v1alpha1

install.verrazzano.io/v1alpha1

Resource Types:

Verrazzano

Verrazzano

Verrazzano specifies the Verrazzano API.

Field Description

apiVersion
string install.verrazzano.io/v1alpha1

kind
string Verrazzano

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoSpec

`components` ComponentSpec	(Optional) The Verrazzano components.
`defaultVolumeSource` Kubernetes core/v1.VolumeSource	(Optional) Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.
`environmentName` string	(Optional) Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.
`profile` ProfileType	(Optional) The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.
`security` SecuritySpec	(Optional) Security specifies Verrazzano security configuration.
`version` string	(Optional) The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.
`volumeClaimSpecTemplates` []VolumeClaimSpecTemplate	(Optional) Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.

status
VerrazzanoStatus

Acme

(Appears on: Certificate)

Acme identifies the ACME cert issuer.

Field	Description
`emailAddress` string	(Optional) Email address of the user.
`environment` string	(Optional) Environment.
`provider` ProviderType	Name of the Acme provider.

ApplicationOperatorComponent

(Appears on: ComponentSpec)

ApplicationOperatorComponent specifies the Application Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then Application Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

List of Overrides for the default values.yaml file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.

ArgoCDComponent

(Appears on: ComponentSpec)

ArgoCDComponent specifies the Argo CD configuration.

Field Description

enabled
bool

(Optional)

If true, then Argo CD will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

AuthProxyComponent

(Appears on: ComponentSpec)

AuthProxyComponent specifies the AuthProxy configuration.

Field Description

enabled
bool

(Optional)

If true, then AuthProxy will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

kubernetes
AuthProxyKubernetesSection

(Optional)

The Kubernetes resources that can be configured for AuthProxy.

AuthProxyKubernetesSection

(Appears on: AuthProxyComponent)

AuthProxyKubernetesSection specifies the Kubernetes resources that can be customized for AuthProxy.

Field	Description
`CommonKubernetesSpec` CommonKubernetesSpec	(Members of `CommonKubernetesSpec` are embedded into this type.)

CA

(Appears on: Certificate)

CA identifies the Certificate Authority cert issuer.

Field	Description
`clusterResourceNamespace` string	The secret namespace.
`secretName` string	The secret name.

CAPIComponent

(Appears on: ComponentSpec)

CAPIComponent specifies the CAPI configuration.

Field	Description
`enabled` bool	(Optional) If true, then CAPI Providers will be installed.

CertManagerComponent

(Appears on: ComponentSpec)

CertManagerComponent specifies the cert-manager configuration.

Field Description

certificate
Certificate

(Optional)

The certificate configuration.

enabled
bool

(Optional)

If true, then cert-manager will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

Certificate

(Appears on: CertManagerComponent)

Certificate represents the type of cert issuer for an installation.

Field	Description
`acme` Acme	(Optional) The ACME configuration. Either `acme` or `ca` must be specified.
`ca` CA	(Optional) The ACME configuration. Either `acme` or `ca` must be specified.

ClusterOperatorComponent

(Appears on: ComponentSpec)

ClusterOperatorComponent specifies the Cluster Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then the Cluster Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

CoherenceOperatorComponent

(Appears on: ComponentSpec)

CoherenceOperatorComponent specifies the Coherence Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then Coherence Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

CommonKubernetesSpec

(Appears on: AuthProxyKubernetesSection, IstioKubernetesSection)

Kubernetes resources that are common to a subgroup of components.

Field	Description
`replicas` uint32	(Optional) Specifies the number of pod instances to run.
`affinity` Kubernetes core/v1.Affinity	(Optional) Specifies the group of affinity scheduling rules.

CompStateType (`string` alias)

(Appears on: ComponentStatusDetails)

CompStateType identifies the state of a component.

ComponentAvailability (`string` alias)

(Appears on: ComponentStatusDetails)

ComponentAvailability identifies the availability of a Verrazzano Component.

ComponentSpec

(Appears on: VerrazzanoSpec)

ComponentSpec contains a set of components used by Verrazzano.

Field	Description
`applicationOperator` ApplicationOperatorComponent	(Optional) The Application Operator component configuration.
`argoCD` ArgoCDComponent	(Optional) The Argo CD component configuration.
`authProxy` AuthProxyComponent	(Optional) The AuthProxy component configuration.
`capi` CAPIComponent	(Optional) The CAPI component configuration.
`certManager` CertManagerComponent	(Optional) The cert-manager component configuration.
`clusterOperator` ClusterOperatorComponent	(Optional) The Cluster Operator component configuration.
`coherenceOperator` CoherenceOperatorComponent	(Optional) The Coherence Operator component configuration.
`console` ConsoleComponent	(Optional) The Verrazzano Console component configuration.
`dns` DNSComponent	(Optional) The DNS component configuration.
`elasticsearch` ElasticsearchComponent	(Optional) The Elasticsearch component configuration.
`fluentd` FluentdComponent	(Optional) The Fluentd component configuration.
`grafana` GrafanaComponent	(Optional) The Grafana component configuration.
`ingress` IngressNginxComponent	(Optional) The ingress NGINX component configuration.
`istio` IstioComponent	(Optional) The Istio component configuration.
`jaegerOperator` JaegerOperatorComponent	(Optional) The Jaeger Operator component configuration.
`kiali` KialiComponent	(Optional) The Kiali component configuration.
`kibana` KibanaComponent	(Optional) The Kibana component configuration.
`keycloak` KeycloakComponent	(Optional) The Keycloak component configuration.
`kubeStateMetrics` KubeStateMetricsComponent	(Optional) The kube-state-metrics component configuration.
`mySQLOperator` MySQLOperatorComponent	(Optional) The MySQL Operator component configuration.
`oam` OAMComponent	(Optional) The OAM component configuration.
`prometheus` PrometheusComponent	(Optional) The Prometheus component configuration.
`prometheusAdapter` PrometheusAdapterComponent	(Optional) The Prometheus Adapter component configuration.
`prometheusNodeExporter` PrometheusNodeExporterComponent	(Optional) The Prometheus Node Exporter component configuration.
`prometheusOperator` PrometheusOperatorComponent	(Optional) The Prometheus Operator component configuration.
`prometheusPushgateway` PrometheusPushgatewayComponent	(Optional) The Prometheus Pushgateway component configuration.
`rancher` RancherComponent	(Optional) The Rancher component configuration.
`rancherBackup` RancherBackupComponent	(Optional) The rancherBackup component configuration.
`thanos` ThanosComponent	(Optional) The Thanos component configuration.
`velero` VeleroComponent	(Optional) The Velero component configuration.
`verrazzano` VerrazzanoComponent	(Optional) The Verrazzano component configuration.
`weblogicOperator` WebLogicOperatorComponent	(Optional) The WebLogic Kubernetes Operator component configuration.

ComponentStatusDetails

ComponentStatusDetails defines the observed state of a component.

Field	Description
`available` ComponentAvailability	Whether a component is available for use.
`conditions` []Condition	Information about the current state of a component.
`lastReconciledGeneration` int64	The generation of the last Verrazzano resource the Component was successfully reconciled against.
`name` string	Name of the component.
`reconcilingGeneration` int64	The generation of the Verrazzano resource the Component is currently being reconciled against.
`state` CompStateType	The state of a component.
`version` string	The version of a component.

ComponentStatusMap (`map[string]*github.com/verrazzano/verrazzano/platform-operator/apis/verrazzano/v1alpha1.ComponentStatusDetails` alias)

(Appears on: VerrazzanoStatus)

ComponentStatusMap is a map of components status details.

ComponentValidator

Condition

(Appears on: ComponentStatusDetails, VerrazzanoStatus)

Condition describes the current state of an installation.

Field	Description
`lastTransitionTime` string	Last time the condition transitioned from one status to another.
`message` string	Human readable message indicating details about the last transition.
`status` Kubernetes core/v1.ConditionStatus	Status of the condition: one of `True`, `False`, or `Unknown`.
`type` ConditionType	Type of condition.

ConditionType (`string` alias)

(Appears on: Condition)

ConditionType identifies the condition of the install, uninstall, or upgrade, which can be checked with kubectl wait.

ConsoleComponent

(Appears on: ComponentSpec)

ConsoleComponent specifies the Verrazzano Console configuration.

Field Description

enabled
bool

(Optional)

If true, then the Verrazzano Console will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

DNSComponent

(Appears on: ComponentSpec)

DNSComponent specifies the DNS configuration.

Field	Description
`external` External	(Optional) External DNS configuration.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`oci` OCI	(Optional) Oracle Cloud Infrastructure DNS configuration.
`wildcard` Wildcard	(Optional) Wildcard DNS configuration. This is the default with a domain of nip.io.

DatabaseInfo

(Appears on: GrafanaComponent)

DatabaseInfo specifies the database connection information for the Grafana DB instance.

Field	Description
`host` string	The host of the database.
`name` string	The name of the database.

ElasticsearchComponent

(Appears on: ComponentSpec)

ElasticsearchComponent specifies the Elasticsearch configuration.

Field	Description
`enabled` bool	(Optional) If true, then OpenSearch will be installed.
`installArgs` []InstallArgs	(Optional) A list of values to use during the OpenSearch installation. Each argument is specified as either a `name/value` or `name/valueList` pair. For sample usage, see Customize OpenSearch.
`nodes` []OpenSearchNode	(Optional) A list of OpenSearch node groups.
`policies` []VMO /vmcontroller/v1.IndexManagementPolicy	A list of Index State Management policies to enable on OpenSearch.
`plugins` VMO /vmcontroller/v1.OpenSearchPlugins	(Optional) Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image
`disableDefaultPolicy` bool	To disable the default ISM policies.

External

(Appears on: DNSComponent)

External DNS type.

Field	Description
`suffix` string	The suffix for DNS names.

FluentdComponent

(Appears on: ComponentSpec)

FluentdComponent specifies the Fluentd configuration.

Field	Description
`elasticsearchSecret` string	(Optional) The secret containing the credentials for connecting to OpenSearch. This secret needs to be created in the `verrazzano-install` namespace prior to creating the Verrazzano custom resource. Specify the OpenSearch login credentials in the `username` and `password` fields in this secret. Specify the CA for verifying the OpenSearch certificate in the `ca-bundle` field, if applicable. The default `verrazzano` is the secret for connecting to the VMI OpenSearch.
`elasticsearchURL` string	(Optional) The target OpenSearch URLs. Specify this option in this format. The default `http://vmi-system-es-ingest-oidc:8775` is the VMI OpenSearch URL.
`enabled` bool	(Optional) If true, then Fluentd will be installed.
`extraVolumeMounts` []VolumeMount	(Optional) A list of host path volume mounts, in addition to `/var/log`, into the Fluentd DaemonSet. The Fluentd component collects log files in the `/var/log/containers` directory of Kubernetes worker nodes. The `/var/log/containers` directory may contain symbolic links to files located outside the `/var/log` directory. If the host path directory containing the log files is located outside `/var/log`, the Fluentd DaemonSet must have the volume mount of that directory to collect the logs.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`oci` OciLoggingConfiguration	(Optional) The Oracle Cloud Infrastructure Logging configuration.

GrafanaComponent

(Appears on: ComponentSpec)

GrafanaComponent specifies the Grafana configuration.

Field	Description
`database` DatabaseInfo	(Optional) The information to configure a connection to an external Grafana database.
`enabled` bool	(Optional) If true, then Grafana will be installed.
`replicas` int32	(Optional) The number of pods to replicate. The default is `1`.
`smtp` VMO /vmcontroller/v1.SMTPInfo	(Optional) The SMTP notification settings.

IngressNginxComponent

(Appears on: ComponentSpec)

IngressNginxComponent specifies the ingress-nginx configuration.

Field	Description
`enabled` bool	(Optional) If true, then ingress NGINX will be installed.
`ingressClassName` string	(Optional) Name of the ingress class used by the ingress controller. Defaults to `verrazzano-nginx`.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`nginxInstallArgs` []InstallArgs	(Optional) Arguments for installing NGINX.
`ports` []Kubernetes core/v1.ServicePort	(Optional) The list of port configurations used by the ingress.
`type` IngressType	(Optional) The ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the ingress type is `NodePort`, then a valid and accessible IP address must be specified using the `controller.service.externalIPs` key in NGINXInstallArgs. For sample usage, see External Load Balancers.

IngressType (`string` alias)

(Appears on: IngressNginxComponent, IstioIngressSection)

IngressType is the type of ingress.

InstallArgs

(Appears on: ElasticsearchComponent, IngressNginxComponent, IstioComponent, KeycloakComponent, MySQLComponent, VerrazzanoComponent)

InstallArgs identifies a name/value or name/value list needed for the install. Value and ValueList cannot both be specified.

Field	Description
`name` string	Name of the install argument.
`value` string	(Optional) Value for the named install argument.
`setString` bool	(Optional) If the value is a literal string.
`valueList` []string	(Optional) List of values for the named install argument.

InstallOverrides

(Appears on: ApplicationOperatorComponent, ArgoCDComponent, AuthProxyComponent, CertManagerComponent, ClusterOperatorComponent, CoherenceOperatorComponent, ConsoleComponent, DNSComponent, FluentdComponent, IngressNginxComponent, IstioComponent, JaegerOperatorComponent, KeycloakComponent, KialiComponent, KubeStateMetricsComponent, MySQLComponent, MySQLOperatorComponent, OAMComponent, PrometheusAdapterComponent, PrometheusNodeExporterComponent, PrometheusOperatorComponent, PrometheusPushgatewayComponent, RancherBackupComponent, RancherComponent, ThanosComponent, VeleroComponent, VerrazzanoComponent, WebLogicOperatorComponent)

InstallOverrides are used to pass installation overrides to components.

Field	Description
`monitorChanges` bool	(Optional) If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.
`overrides` []Overrides	(Optional) List of overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. Invalid override values will be ignored.

InstanceInfo

(Appears on: VerrazzanoStatus)

InstanceInfo details of installed Verrazzano instance maintained in status field.

Field	Description
`argoCDUrl` string	The Argo CD UI URL for this Verrazzano installation.
`consoleUrl` string	The Console URL for this Verrazzano installation.
`elasticUrl` string	The OpenSearch URL for this Verrazzano installation.
`grafanaUrl` string	The Grafana URL for this Verrazzano installation.
`jaegerUrl` string	The Jaeger UI URL for this Verrazzano installation.
`keyCloakUrl` string	The KeyCloak URL for this Verrazzano installation.
`kialiUrl` string	The Kiali URL for this Verrazzano installation.
`kibanaUrl` string	The OpenSearch Dashboards URL for this Verrazzano installation.
`prometheusUrl` string	The Prometheus URL for this Verrazzano installation.
`rancherUrl` string	The Rancher URL for this Verrazzano installation.
`thanosQueryUrl` string	The Thanos Query URL for this Verrazzano installation. The Thanos Query ingress gets forwarded to the Thanos Query Frontend service.

IstioComponent

(Appears on: ComponentSpec)

IstioComponent specifies the Istio configuration.

Field	Description
`egress` IstioEgressSection	(Optional) The Istio egress gateway configuration.
`enabled` bool	(Optional) If true, then Istio will be installed.
`ingress` IstioIngressSection	(Optional) The Istio ingress gateway configuration.
`injectionEnabled` bool	(Optional) Istio sidecar injection enabled for installed components. Default is `true`.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for default IstioOperator. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here Passing through an invalid IstioOperator resource will result in an error.
`istioInstallArgs` []InstallArgs	(Optional) Arguments for installing Istio.

IstioEgressSection

(Appears on: IstioComponent)

IstioEgressSection specifies the specific configuration options available for the Istio egress gateways.

Field	Description
`kubernetes` IstioKubernetesSection	(Optional) The Kubernetes resources that can be configured for an Istio egress gateway.

IstioIngressSection

(Appears on: IstioComponent)

IstioIngressSection specifies the specific configuration options available for the Istio ingress gateways.

Field	Description
`type` IngressType	(Optional) The Istio ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the Istio ingress type is `NodePort`, then a valid and accessible IP address must be specified using the `gateways.istio-ingressgateway.externalIPs` key in IstioInstallArgs. For sample usage, see External Load Balancers.
`ports` []Kubernetes core/v1.ServicePort	(Optional) The list port configurations used by the Istio ingress.
`kubernetes` IstioKubernetesSection	(Optional) The Kubernetes resources that can be configured for an Istio ingress gateway.

IstioKubernetesSection

(Appears on: IstioEgressSection, IstioIngressSection)

IstioKubernetesSection specifies the Kubernetes resources that can be customized for Istio.

Field	Description
`CommonKubernetesSpec` CommonKubernetesSpec	(Members of `CommonKubernetesSpec` are embedded into this type.)

JaegerOperatorComponent

(Appears on: ComponentSpec)

JaegerOperatorComponent specifies the Jaeger Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then Jaeger Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

KeycloakComponent

(Appears on: ComponentSpec)

KeycloakComponent specifies the Keycloak configuration.

Field	Description
`enabled` bool	(Optional) If true, then Keycloak will be installed.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`keycloakInstallArgs` []InstallArgs	(Optional) Arguments for installing Keycloak.
`mysql` MySQLComponent	(Optional) Contains the MySQL component configuration needed for Keycloak.

KialiComponent

(Appears on: ComponentSpec)

KialiComponent specifies the Kiali configuration.

Field Description

enabled
bool

(Optional)

If true, then Kiali will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

KibanaComponent

(Appears on: ComponentSpec)

KibanaComponent specifies the Kibana configuration.

Field	Description
`enabled` bool	(Optional) If true, then OpenSearch Dashboards will be installed.
`replicas` int32	The number of pods to replicate. The default is `1`.
`plugins` VMO /vmcontroller/v1.OpenSearchDashboardsPlugins	(Optional) Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch-Dashboard image

KubeStateMetricsComponent

(Appears on: ComponentSpec)

KubeStateMetricsComponent specifies the kube-state-metrics configuration.

Field Description

enabled
bool

(Optional)

If true, then kube-state-metrics will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

MySQLComponent

(Appears on: KeycloakComponent)

MySQLComponent specifies the MySQL configuration.

Field Description

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

mysqlInstallArgs
[]InstallArgs

(Optional)

Arguments for installing MySQL.

volumeSource
Kubernetes core/v1.VolumeSource

(Optional)

Defines the type of volume to be used for persistence for Keycloak/MySQL, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the claimName must reference the name of a VolumeClaimSpecTemplate declared in the volumeClaimSpecTemplates section.

MySQLOperatorComponent

(Appears on: ComponentSpec)

MySQLOperatorComponent specifies the MySQL Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then MySQL Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

OAMComponent

(Appears on: ComponentSpec)

OAMComponent specifies the OAM configuration.

Field Description

enabled
bool

(Optional)

If true, then OAM will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

OCI

(Appears on: DNSComponent)

OCI DNS type.

Field	Description
`dnsScope` string	(Optional) Scope of the Oracle Cloud Infrastructure DNS zone (`PRIVATE`, `GLOBAL`). If not specified, then defaults to `GLOBAL`.
`dnsZoneCompartmentOCID` string	The Oracle Cloud Infrastructure DNS compartment OCID.
`dnsZoneOCID` string	The Oracle Cloud Infrastructure DNS zone OCID.
`dnsZoneName` string	Name of Oracle Cloud Infrastructure DNS zone.
`ociConfigSecret` string	Name of the Oracle Cloud Infrastructure configuration secret. Generate a secret based on the Oracle Cloud Infrastructure configuration profile you want to use. You can specify a profile other than `DEFAULT` and specify the secret name. See instructions by running `./install/create_oci_config_secret.sh`.

OciLoggingConfiguration

(Appears on: FluentdComponent)

OciLoggingConfiguration is the Oracle Cloud Infrastructure logging configuration for Fluentd.

Field	Description
`apiSecret` string	(Optional) The name of the secret containing the Oracle Cloud Infrastructure API configuration and private key.
`defaultAppLogId` string	The OCID of the Oracle Cloud Infrastructure Log that will collect application logs.
`systemLogId` string	The OCID of the Oracle Cloud Infrastructure Log that will collect system logs.

OpenSearchNode

(Appears on: ElasticsearchComponent)

OpenSearchNode specifies a node group in the OpenSearch cluster.

Field	Description
`name` string	Name of the node group.
`replicas` int32	(Optional) Node group replica count.
`resources` Kubernetes core/v1.ResourceRequirements	(Optional) Kubernetes container resources for nodes in the node group.
`roles` []VMO /vmcontroller/v1.NodeRole	Role or roles that nodes in the group will assume: may be `master`, `data`, and/or `ingest`.
`storage` OpenSearchNodeStorage	(Optional) Storage settings for the node group.
`javaOpts` string	(Optional) JavaOpts settings for the OpenSearch JVM.

OpenSearchNodeStorage

(Appears on: OpenSearchNode)

Field	Description
`size` string	Node group storage size expressed as a Quantity.

Overrides

(Appears on: InstallOverrides)

Overrides identifies overrides for a component.

Field	Description
`configMapRef` Kubernetes core/v1.ConfigMapKeySelector	(Optional) Selector for ConfigMap containing override data. For sample usage, see ConfigMapRef.
`secretRef` Kubernetes core/v1.SecretKeySelector	(Optional) Selector for Secret containing override data. For sample usage, see SecretRef.
`values` Kubernetes apiextensions/v1.JSON	(Optional) Configure overrides using inline YAML. For sample usage, see Values.

ProfileType (`string` alias)

(Appears on: VerrazzanoSpec)

ProfileType is the type of installation profile.

PrometheusAdapterComponent

(Appears on: ComponentSpec)

PrometheusAdapterComponent specifies the Prometheus Adapter configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Adaptor will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

PrometheusComponent

(Appears on: ComponentSpec)

PrometheusComponent specifies the Prometheus configuration.

Field	Description
`enabled` bool	(Optional) If true, then Prometheus will be installed. This is a legacy setting; the preferred way to configure Prometheus is using the PrometheusOperatorComponent.

PrometheusNodeExporterComponent

(Appears on: ComponentSpec)

PrometheusNodeExporterComponent specifies the Prometheus Node Exporter configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Node Exporter will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

PrometheusOperatorComponent

(Appears on: ComponentSpec)

PrometheusOperatorComponent specifies the Prometheus Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

PrometheusPushgatewayComponent

(Appears on: ComponentSpec)

PrometheusPushgatewayComponent specifies the Prometheus Pushgateway configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Pushgateway will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

ProviderType (`string` alias)

(Appears on: Acme)

ProviderType identifies Acme provider type.

RancherBackupComponent

(Appears on: ComponentSpec)

RancherBackupComponent specifies the rancherBackup configuration.

Field Description

enabled
bool

(Optional)

If true, then rancherBackup will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

RancherComponent

(Appears on: ComponentSpec)

RancherComponent specifies the Rancher configuration.

Field Description

enabled
bool

(Optional)

If true, then Rancher will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

keycloakAuthEnabled
bool

(Optional)

KeycloakAuthEnabled specifies whether the Keycloak Auth provider is enabled. Default is false.

SecuritySpec

(Appears on: VerrazzanoSpec)

SecuritySpec defines the security configuration for Verrazzano.

Field	Description
`adminSubjects` []Kubernetes rbac/v1.Subject	(Optional) Specifies subjects that should be bound to the verrazzano-admin role.
`monitorSubjects` []Kubernetes rbac/v1.Subject	(Optional) Specifies subjects that should be bound to the verrazzano-monitor role.

ThanosComponent

(Appears on: ComponentSpec)

ThanosComponent specifies the Thanos configuration.

Field Description

enabled
bool

(Optional)

If true, then Thanos will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

VeleroComponent

(Appears on: ComponentSpec)

VeleroComponent specifies the Velero configuration.

Field Description

enabled
bool

(Optional)

If true, then Velero will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

VerrazzanoComponent

(Appears on: ComponentSpec)

VerrazzanoComponent specifies the Verrazzano configuration.

Field Description

enabled
bool

(Optional)

If true, then Verrazzano will be installed.

installArgs
[]InstallArgs

(Optional)

Arguments for installing Verrazzano.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

VerrazzanoSpec

(Appears on: Verrazzano)

VerrazzanoSpec defines the desired state of a Verrazzano resource.

Field	Description
`components` ComponentSpec	(Optional) The Verrazzano components.
`defaultVolumeSource` Kubernetes core/v1.VolumeSource	(Optional) Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.
`environmentName` string	(Optional) Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.
`profile` ProfileType	(Optional) The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.
`security` SecuritySpec	(Optional) Security specifies Verrazzano security configuration.
`version` string	(Optional) The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.
`volumeClaimSpecTemplates` []VolumeClaimSpecTemplate	(Optional) Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.

VerrazzanoStatus

(Appears on: Verrazzano)

VerrazzanoStatus defines the observed state of a Verrazzano resource.

Field	Description
`available` string	The summary of Verrazzano component availability.
`components` ComponentStatusMap	States of the individual installed components.
`conditions` []Condition	The latest available observations of an object’s current state.
`state` VzStateType	State of the Verrazzano custom resource.
`instance` InstanceInfo	The Verrazzano instance information.
`version` string	The version of Verrazzano that is installed.

VolumeClaimSpecTemplate

(Appears on: VerrazzanoSpec)

VolumeClaimSpecTemplate Contains common PVC configurations that can be referenced from Components; these do not actually result in generated PVCs, but can be used to provide common configurations to components that declare a PersistentVolumeClaimVolumeSource.

Field Description

metadata
Kubernetes meta/v1.ObjectMeta

Metadata about the PersistentVolumeClaimSpec template.

Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
Kubernetes core/v1.PersistentVolumeClaimSpec

A PersistentVolumeClaimSpec template that can be referenced by a Component to override its default storage settings for a profile. At present, only a subset of the resources.requests object are honored depending on the component.

`accessModes` []Kubernetes core/v1.PersistentVolumeAccessMode	(Optional) accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
`selector` Kubernetes meta/v1.LabelSelector	(Optional) selector is a label query over volumes to consider for binding.
`resources` Kubernetes core/v1.ResourceRequirements	(Optional) resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources
`volumeName` string	(Optional) volumeName is the binding reference to the PersistentVolume backing this claim.
`storageClassName` string	(Optional) storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1
`volumeMode` Kubernetes core/v1.PersistentVolumeMode	(Optional) volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.
`dataSource` Kubernetes core/v1.TypedLocalObjectReference	(Optional) dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. If the AnyVolumeDataSource feature gate is enabled, this field will always have the same contents as the DataSourceRef field.
`dataSourceRef` Kubernetes core/v1.TypedLocalObjectReference	(Optional) dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any local object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the DataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, both fields (DataSource and DataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. There are two important differences between DataSource and DataSourceRef: * While DataSource only allows two specific types of objects, DataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects. * While DataSource ignores disallowed values (dropping them), DataSourceRef preserves all values, and generates an error if a disallowed value is specified. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled.

VolumeMount

(Appears on: FluentdComponent)

VolumeMount defines a hostPath type volume mount.

Field	Description
`destination` string	(Optional) The destination path on the Fluentd container, defaults to the source host path.
`readOnly` bool	(Optional) Specifies if the volume mount is read-only, defaults to `true`.
`source` string	The source host path.

VzStateType (`string` alias)

(Appears on: VerrazzanoStatus)

VzStateType identifies the state of a Verrazzano installation.

WebLogicOperatorComponent

(Appears on: ComponentSpec)

WebLogicOperatorComponent specifies the WebLogic Kubernetes Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then WebLogic Kubernetes Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

Wildcard

(Appears on: DNSComponent)

Wildcard DNS type.

Field	Description
`domain` string	The type of wildcard DNS domain. For example, nip.io, sslip.io, and such.

Generated with gen-crd-api-reference-docs

Docs: Verrazzano v1beta1

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

install.verrazzano.io/v1beta1

install.verrazzano.io/v1beta1

Resource Types:

Verrazzano

Verrazzano

Verrazzano specifies the Verrazzano API.

Field Description

apiVersion
string install.verrazzano.io/v1beta1

kind
string Verrazzano

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoSpec

`components` ComponentSpec	(Optional) The Verrazzano components.
`defaultVolumeSource` Kubernetes core/v1.VolumeSource	(Optional) Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.
`environmentName` string	(Optional) Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.
`profile` ProfileType	(Optional) The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.
`security` SecuritySpec	(Optional) Security specifies Verrazzano security configuration.
`version` string	(Optional) The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.
`volumeClaimSpecTemplates` []VolumeClaimSpecTemplate	(Optional) Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.

status
VerrazzanoStatus

Acme

(Appears on: Certificate)

Acme identifies the ACME cert issuer.

Field	Description
`emailAddress` string	(Optional) Email address of the user.
`environment` string	(Optional) Environment.
`provider` ProviderType	Name of the Acme provider.

ApplicationOperatorComponent

(Appears on: ComponentSpec)

ApplicationOperatorComponent specifies the Application Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then the Application Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

ArgoCDComponent

(Appears on: ComponentSpec)

ArgoCDComponent specifies the Argo CD configuration.

Field Description

enabled
bool

(Optional)

If true, then Argo CD will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

AuthProxyComponent

(Appears on: ComponentSpec)

AuthProxyComponent specifies the AuthProxy configuration.

Field Description

enabled
bool

(Optional)

If true, then AuthProxy will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

CA

(Appears on: Certificate)

CA identifies the Certificate Authority cert issuer.

Field	Description
`clusterResourceNamespace` string	The secret namespace.
`secretName` string	The secret name.

CAPIComponent

(Appears on: ComponentSpec)

CAPIComponent specifies the CAPI configuration.

Field	Description
`enabled` bool	(Optional) If true, then CAPI Providers will be installed.

CertManagerComponent

(Appears on: ComponentSpec)

CertManagerComponent specifies the cert-manager configuration.

Field Description

certificate
Certificate

(Optional)

The certificate configuration.

enabled
bool

(Optional)

If true, then cert-manager will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

Certificate

(Appears on: CertManagerComponent)

Certificate represents the type of cert issuer for an installation.

Field	Description
`acme` Acme	(Optional) The ACME configuration. Either `acme` or `ca` must be specified.
`ca` CA	(Optional) The ACME configuration. Either `acme` or `ca` must be specified.

ClusterOperatorComponent

(Appears on: ComponentSpec)

ClusterOperatorComponent specifies the Cluster Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then the Cluster Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

CoherenceOperatorComponent

(Appears on: ComponentSpec)

CoherenceOperatorComponent specifies the Coherence Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then the Coherence Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

CompStateType (`string` alias)

(Appears on: ComponentStatusDetails)

CompStateType identifies the state of a component.

ComponentAvailability (`string` alias)

(Appears on: ComponentStatusDetails)

ComponentAvailability identifies the availability of a Verrazzano Component.

ComponentSpec

(Appears on: VerrazzanoSpec)

ComponentSpec contains a set of components used by Verrazzano.

Field	Description
`applicationOperator` ApplicationOperatorComponent	(Optional) The Application Operator component configuration.
`argoCD` ArgoCDComponent	(Optional) The Argo CD component configuration.
`authProxy` AuthProxyComponent	(Optional) The AuthProxy component configuration.
`capi` CAPIComponent	(Optional) The CAPI component configuration.
`certManager` CertManagerComponent	(Optional) The cert-manager component configuration.
`clusterOperator` ClusterOperatorComponent	(Optional) The Cluster Operator component configuration.
`coherenceOperator` CoherenceOperatorComponent	(Optional) The Coherence Operator component configuration.
`console` ConsoleComponent	(Optional) The Verrazzano Console component configuration.
`dns` DNSComponent	(Optional) The DNS component configuration.
`fluentd` FluentdComponent	(Optional) The Fluentd component configuration.
`grafana` GrafanaComponent	(Optional) The Grafana component configuration.
`ingressNGINX` IngressNginxComponent	(Optional) The ingress NGINX component configuration.
`istio` IstioComponent	(Optional) The Istio component configuration.
`jaegerOperator` JaegerOperatorComponent	(Optional) The Jaeger Operator component configuration.
`keycloak` KeycloakComponent	(Optional) The Keycloak component configuration.
`kiali` KialiComponent	(Optional) The Kiali component configuration.
`kubeStateMetrics` KubeStateMetricsComponent	(Optional) The kube-state-metrics component configuration.
`mySQLOperator` MySQLOperatorComponent	(Optional) The MySQL Operator component configuration.
`oam` OAMComponent	(Optional) The OAM component configuration.
`opensearch` OpenSearchComponent	(Optional) The OpenSearch component configuration.
`opensearchDashboards` OpenSearchDashboardsComponent	(Optional) The OpenSearch Dashboards component configuration.
`prometheus` PrometheusComponent	(Optional) The Prometheus component configuration.
`prometheusAdapter` PrometheusAdapterComponent	(Optional) The Prometheus Adapter component configuration.
`prometheusNodeExporter` PrometheusNodeExporterComponent	(Optional) The Prometheus Node Exporter component configuration.
`prometheusOperator` PrometheusOperatorComponent	(Optional) The Prometheus Operator component configuration.
`prometheusPushgateway` PrometheusPushgatewayComponent	(Optional) The Prometheus Pushgateway component configuration.
`rancher` RancherComponent	(Optional) The Rancher component configuration.
`rancherBackup` RancherBackupComponent	(Optional) The rancherBackup component configuration.
`thanos` ThanosComponent	(Optional) The Thanos component configuration.
`velero` VeleroComponent	(Optional) The Velero component configuration.
`verrazzano` VerrazzanoComponent	(Optional) The Verrazzano component configuration.
`weblogicOperator` WebLogicOperatorComponent	(Optional) The WebLogic Kubernetes Operator component configuration.

ComponentStatusDetails

ComponentStatusDetails defines the observed state of a component.

Field	Description
`available` ComponentAvailability	Whether a component is available for use.
`conditions` []Condition	Information about the current state of a component.
`lastReconciledGeneration` int64	The generation of the last Verrazzano resource the Component was successfully reconciled against.
`name` string	Name of the component.
`reconcilingGeneration` int64	The generation of the Verrazzano resource the Component is currently being reconciled against.
`state` CompStateType	The state of a component.
`version` string	The version of a component.

ComponentStatusMap (`map[string]*github.com/verrazzano/verrazzano/platform-operator/apis/verrazzano/v1beta1.ComponentStatusDetails` alias)

(Appears on: VerrazzanoStatus)

ComponentStatusMap is a map of components status details.

ComponentValidator

Condition

(Appears on: ComponentStatusDetails, VerrazzanoStatus)

Condition describes the current state of an installation.

Field	Description
`lastTransitionTime` string	Last time the condition transitioned from one status to another.
`message` string	Human readable message indicating details about the last transition.
`status` Kubernetes core/v1.ConditionStatus	Status of the condition: one of `True`, `False`, or `Unknown`.
`type` ConditionType	Type of condition.

ConditionType (`string` alias)

(Appears on: Condition)

ConditionType identifies the condition of the install, uninstall, or upgrade, which can be checked with kubectl wait.

ConsoleComponent

(Appears on: ComponentSpec)

ConsoleComponent specifies the Verrazzano Console configuration.

Field Description

enabled
bool

(Optional)

If true, then Verrazzano Console will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

DNSComponent

(Appears on: ComponentSpec)

DNSComponent specifies the DNS configuration.

Field	Description
`external` External	(Optional) External DNS configuration.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`oci` OCI	(Optional) Oracle Cloud Infrastructure DNS configuration.
`wildcard` Wildcard	(Optional) Wildcard DNS configuration. This is the default with a domain of nip.io.

DatabaseInfo

(Appears on: GrafanaComponent)

DatabaseInfo specifies the database connection information for the Grafana DB instance.

Field	Description
`host` string	The host of the database.
`name` string	The name of the database.

External

(Appears on: DNSComponent)

External DNS type.

Field	Description
`suffix` string	The suffix for DNS names.

FluentdComponent

(Appears on: ComponentSpec)

FluentdComponent specifies the Fluentd configuration.

Field	Description
`enabled` bool	(Optional) If true, then Fluentd will be installed.
`extraVolumeMounts` []VolumeMount	(Optional) A list of host path volume mounts, in addition to `/var/log`, into the Fluentd DaemonSet. The Fluentd component collects log files in the `/var/log/containers` directory of Kubernetes worker nodes. The `/var/log/containers` directory may contain symbolic links to files located outside the `/var/log` directory. If the host path directory containing the log files is located outside `/var/log`, the Fluentd DaemonSet must have the volume mount of that directory to collect the logs.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`oci` OciLoggingConfiguration	(Optional) The Oracle Cloud Infrastructure Logging configuration.
`opensearchSecret` string	(Optional) The secret containing the credentials for connecting to OpenSearch. This secret needs to be created in the `verrazzano-install` namespace prior to creating the Verrazzano custom resource. Specify the OpenSearch login credentials in the `username` and `password` fields in this secret. Specify the CA for verifying the OpenSearch certificate in the `ca-bundle` field, if applicable. The default `verrazzano` is the secret for connecting to the VMI OpenSearch.
`opensearchURL` string	(Optional) The target OpenSearch URLs. Specify this option in this format. The default `http://vmi-system-es-ingest-oidc:8775` is the VMI OpenSearch URL.

GrafanaComponent

(Appears on: ComponentSpec)

GrafanaComponent specifies the Grafana configuration.

Field	Description
`database` DatabaseInfo	(Optional) The information to configure a connection to an external Grafana database.
`enabled` bool	(Optional) If true, then Grafana will be installed.
`replicas` int32	(Optional) The number of pods to replicate. The default is `1`.
`smtp` VMO /vmcontroller/v1.SMTPInfo	(Optional) The SMTP notification settings.

IngressNginxComponent

(Appears on: ComponentSpec)

IngressNginxComponent specifies the ingress NGINX configuration.

Field	Description
`enabled` bool	(Optional) If true, then ingress NGINX will be installed.
`ingressClassName` string	(Optional) Name of the ingress class used by the ingress controller. Defaults to `verrazzano-nginx`.
`InstallOverrides` InstallOverrides	(Members of `InstallOverrides` are embedded into this type.) (Optional) List of Overrides for the default `values.yaml` file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here and invalid values will be ignored.
`ports` []Kubernetes core/v1.ServicePort	(Optional) The list of port configurations used by the ingress.
`type` IngressType	(Optional) The ingress type. Valid values are `LoadBalancer` and `NodePort`. The default value is `LoadBalancer`. If the ingress type is `NodePort`, then a valid and accessible IP address must be specified using the `controller.service.externalIPs` key in the InstallOverrides. For sample usage, see External Load Balancers.

IngressType (`string` alias)

(Appears on: IngressNginxComponent)

IngressType is the type of ingress.

InstallOverrides

InstallOverrides are used to pass installation overrides to components.

Field	Description
`monitorChanges` bool	(Optional) If false, then Verrazzano updates will ignore any configuration changes to this component. Defaults to `true`.
`overrides` []Overrides	(Optional) List of overrides for the default values.yaml file for the component Helm chart. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. Invalid override values will be ignored.

InstanceInfo

(Appears on: VerrazzanoStatus)

InstanceInfo details of the installed Verrazzano instance maintained in status field.

Field	Description
`argoCDUrl` string	The Argo CD UI URL for this Verrazzano installation.
`consoleUrl` string	The Console URL for this Verrazzano installation.
`grafanaUrl` string	The Grafana URL for this Verrazzano installation.
`jaegerUrl` string	The Jaeger UI URL for this Verrazzano installation.
`keyCloakUrl` string	The KeyCloak URL for this Verrazzano installation.
`kialiUrl` string	The Kiali URL for this Verrazzano installation.
`openSearchDashboardsUrl` string	The OpenSearch Dashboards URL for this Verrazzano installation.
`openSearchUrl` string	The OpenSearch URL for this Verrazzano installation.
`prometheusUrl` string	The Prometheus URL for this Verrazzano installation.
`rancherUrl` string	The Rancher URL for this Verrazzano installation.
`thanosQueryUrl` string	The Thanos Query URL for this Verrazzano installation. The Thanos Query ingress gets forwarded to the Thanos Query Frontend service.

IstioComponent

(Appears on: ComponentSpec)

IstioComponent specifies the Istio configuration.

Field Description

enabled
bool

(Optional)

If true, then Istio will be installed.

injectionEnabled
bool

(Optional)

Istio sidecar injection enabled for installed components. Default is true.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

List of Overrides for default IstioOperator. Overrides are merged together, but in the event of conflicting fields, the last override in the list takes precedence over any others. You can find all possible values here Passing through an invalid IstioOperator resource will result in an error.

JaegerOperatorComponent

(Appears on: ComponentSpec)

JaegerOperatorComponent specifies the Jaeger Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then Jaeger Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

KeycloakComponent

(Appears on: ComponentSpec)

KeycloakComponent specifies the Keycloak configuration.

Field Description

enabled
bool

(Optional)

If true, then Keycloak will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

mysql
MySQLComponent

(Optional)

Contains the MySQL component configuration needed for Keycloak.

KialiComponent

(Appears on: ComponentSpec)

KialiComponent specifies the Kiali configuration.

Field Description

enabled
bool

(Optional)

If true, then Kiali will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

KubeStateMetricsComponent

(Appears on: ComponentSpec)

KubeStateMetricsComponent specifies the kube-state-metrics configuration.

Field Description

enabled
bool

(Optional)

If true, then kube-state-metrics will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

MySQLComponent

(Appears on: KeycloakComponent)

MySQLComponent specifies the MySQL configuration.

Field Description

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

volumeSource
Kubernetes core/v1.VolumeSource

(Optional)

MySQLOperatorComponent

(Appears on: ComponentSpec)

MySQLOperatorComponent specifies the MySQL Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then MySQL Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

OAMComponent

(Appears on: ComponentSpec)

OAMComponent specifies the OAM configuration.

Field Description

enabled
bool

(Optional)

If true, then OAM will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

OCI

(Appears on: DNSComponent)

OCI DNS type.

Field	Description
`dnsScope` string	(Optional) Scope of the Oracle Cloud Infrastructure DNS zone (`PRIVATE`, `GLOBAL`). If not specified, then defaults to `GLOBAL`.
`dnsZoneCompartmentOCID` string	The Oracle Cloud Infrastructure DNS compartment OCID.
`dnsZoneOCID` string	The Oracle Cloud Infrastructure DNS zone OCID.
`dnsZoneName` string	Name of Oracle Cloud Infrastructure DNS zone.
`ociConfigSecret` string	Name of the Oracle Cloud Infrastructure configuration secret. Generate a secret based on the Oracle Cloud Infrastructure configuration profile you want to use. You can specify a profile other than `DEFAULT` and specify the secret name. See instructions by running `./install/create_oci_config_secret.sh`.

OciLoggingConfiguration

(Appears on: FluentdComponent)

OciLoggingConfiguration is the Oracle Cloud Infrastructure logging configuration for Fluentd.

Field	Description
`apiSecret` string	(Optional) The name of the secret containing the Oracle Cloud Infrastructure API configuration and private key.
`defaultAppLogId` string	The OCID of the Oracle Cloud Infrastructure Log that will collect application logs.
`systemLogId` string	The OCID of the Oracle Cloud Infrastructure Log that will collect system logs.

OpenSearchComponent

(Appears on: ComponentSpec)

OpenSearchComponent specifies the OpenSearch configuration.

Field	Description
`enabled` bool	(Optional) If true, then OpenSearch will be installed.
`nodes` []OpenSearchNode	(Optional) A list of OpenSearch node groups. For sample usage, see Customize OpenSearch.
`policies` []VMO /vmcontroller/v1.IndexManagementPolicy	(Optional) A list of Index State Management policies to enable on OpenSearch.
`plugins` VMO /vmcontroller/v1.OpenSearchPlugins	(Optional) Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
`disableDefaultPolicy` bool	To disable the default ISM policies.

OpenSearchDashboardsComponent

(Appears on: ComponentSpec)

OpenSearchDashboardsComponent specifies the OpenSearch Dashboards configuration.

Field	Description
`enabled` bool	(Optional) If true, then OpenSearch Dashboards will be installed.
`replicas` int32	The number of pods to replicate. The default is `1`.
`plugins` VMO /vmcontroller/v1.OpenSearchDashboardsPlugins	(Optional) Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch-Dashboard image

OpenSearchNode

(Appears on: OpenSearchComponent)

OpenSearchNode specifies a node group in the OpenSearch cluster.

Field	Description
`name` string	Name of the node group.
`replicas` int32	(Optional) Node group replica count.
`resources` Kubernetes core/v1.ResourceRequirements	(Optional) Kubernetes container resources for nodes in the node group.
`roles` []VMO /vmcontroller/v1.NodeRole	Role or roles that nodes in the group will assume: may be `master`, `data`, and/or `ingest`.
`storage` OpenSearchNodeStorage	(Optional) Storage settings for the node group.
`javaOpts` string	(Optional) JavaOpts settings for the OpenSearch JVM.

OpenSearchNodeStorage

(Appears on: OpenSearchNode)

Field	Description
`size` string	Node group storage size expressed as a Quantity.

Overrides

(Appears on: InstallOverrides)

Overrides identifies overrides for a component.

Field	Description
`configMapRef` Kubernetes core/v1.ConfigMapKeySelector	(Optional) Selector for ConfigMap containing override data. For sample usage, see ConfigMapRef.
`secretRef` Kubernetes core/v1.SecretKeySelector	(Optional) Selector for Secret containing override data. For sample usage, see SecretRef.
`values` Kubernetes apiextensions/v1.JSON	(Optional) Configure overrides using inline YAML. For sample usage, see Values.

ProfileType (`string` alias)

(Appears on: VerrazzanoSpec)

ProfileType is the type of installation profile.

PrometheusAdapterComponent

(Appears on: ComponentSpec)

PrometheusAdapterComponent specifies the Prometheus Adapter configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Adaptor will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

PrometheusComponent

(Appears on: ComponentSpec)

PrometheusComponent specifies the Prometheus configuration.

Field	Description
`enabled` bool	(Optional) If true, then Prometheus will be installed. This is a legacy setting; the preferred way to configure Prometheus is using the PrometheusOperatorComponent.

PrometheusNodeExporterComponent

(Appears on: ComponentSpec)

PrometheusNodeExporterComponent specifies the Prometheus Node Exporter configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Node Exporter will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

PrometheusOperatorComponent

(Appears on: ComponentSpec)

PrometheusOperatorComponent specifies the Prometheus Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

PrometheusPushgatewayComponent

(Appears on: ComponentSpec)

PrometheusPushgatewayComponent specifies the Prometheus Pushgateway configuration.

Field Description

enabled
bool

(Optional)

If true, then Prometheus Pushgateway will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

ProviderType (`string` alias)

(Appears on: Acme)

ProviderType identifies Acme provider type.

RancherBackupComponent

(Appears on: ComponentSpec)

RancherBackupComponent specifies the rancherBackup configuration.

Field Description

enabled
bool

(Optional)

If true, then rancherBackup will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

RancherComponent

(Appears on: ComponentSpec)

RancherComponent specifies the Rancher configuration.

Field Description

enabled
bool

(Optional)

If true, then Rancher will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

keycloakAuthEnabled
bool

(Optional)

KeycloakAuthEnabled specifies whether the Keycloak Auth provider is enabled. Default is false.

SecuritySpec

(Appears on: VerrazzanoSpec)

SecuritySpec defines the security configuration for Verrazzano.

Field	Description
`adminSubjects` []Kubernetes rbac/v1.Subject	(Optional) Specifies subjects that should be bound to the verrazzano-admin role.
`monitorSubjects` []Kubernetes rbac/v1.Subject	(Optional) Specifies subjects that should be bound to the verrazzano-monitor role.

ThanosComponent

(Appears on: ComponentSpec)

ThanosComponent specifies the Thanos configuration.

Field Description

enabled
bool

(Optional)

If true, then Thanos will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

VeleroComponent

(Appears on: ComponentSpec)

VeleroComponent specifies the Velero configuration.

Field Description

enabled
bool

(Optional)

If true, then Velero will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

VerrazzanoComponent

(Appears on: ComponentSpec)

VerrazzanoComponent specifies the Verrazzano configuration.

Field Description

enabled
bool

(Optional)

If true, then Verrazzano will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

VerrazzanoSpec

(Appears on: Verrazzano)

VerrazzanoSpec defines the desired state of Verrazzano resource.

Field	Description
`components` ComponentSpec	(Optional) The Verrazzano components.
`defaultVolumeSource` Kubernetes core/v1.VolumeSource	(Optional) Defines the type of volume to be used for persistence for all components unless overridden, and can be one of either EmptyDirVolumeSource or PersistentVolumeClaimVolumeSource. If PersistentVolumeClaimVolumeSource is declared, then the `claimName` must reference the name of an existing `VolumeClaimSpecTemplate` declared in the `volumeClaimSpecTemplates` section.
`environmentName` string	(Optional) Name of the installation. This name is part of the endpoint access URLs that are generated. The default value is `default`.
`profile` ProfileType	(Optional) The installation profile to select. Valid values are `prod` (production), `dev` (development), and `managed-cluster`. The default is `prod`.
`security` SecuritySpec	(Optional) Security specifies Verrazzano security configuration.
`version` string	(Optional) The version to install. Valid versions can be found here. Defaults to the current version supported by the Verrazzano platform operator.
`volumeClaimSpecTemplates` []VolumeClaimSpecTemplate	(Optional) Defines a named set of PVC configurations that can be referenced from components to configure persistent volumes.

VerrazzanoStatus

(Appears on: Verrazzano)

VerrazzanoStatus defines the observed state of a Verrazzano resource.

Field	Description
`available` string	The summary of Verrazzano component availability.
`components` ComponentStatusMap	States of the individual installed components.
`conditions` []Condition	The latest available observations of an object’s current state.
`state` VzStateType	State of the Verrazzano custom resource.
`instance` InstanceInfo	The Verrazzano instance info.
`version` string	The version of Verrazzano that is installed.

VolumeClaimSpecTemplate

(Appears on: VerrazzanoSpec)

VolumeClaimSpecTemplate Contains common PVC configuration that can be referenced from Components; these do not actually result in generated PVCs, but can be used to provide common configuration to components that declare a PersistentVolumeClaimVolumeSource.

Field Description

metadata
Kubernetes meta/v1.ObjectMeta

Metadata about the PersistentVolumeClaimSpec template.

Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
Kubernetes core/v1.PersistentVolumeClaimSpec

`accessModes` []Kubernetes core/v1.PersistentVolumeAccessMode	(Optional) accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
`selector` Kubernetes meta/v1.LabelSelector	(Optional) selector is a label query over volumes to consider for binding.
`resources` Kubernetes core/v1.ResourceRequirements	(Optional) resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources
`volumeName` string	(Optional) volumeName is the binding reference to the PersistentVolume backing this claim.
`storageClassName` string	(Optional) storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1
`volumeMode` Kubernetes core/v1.PersistentVolumeMode	(Optional) volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.
`dataSource` Kubernetes core/v1.TypedLocalObjectReference	(Optional) dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. If the AnyVolumeDataSource feature gate is enabled, this field will always have the same contents as the DataSourceRef field.
`dataSourceRef` Kubernetes core/v1.TypedLocalObjectReference	(Optional) dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any local object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the DataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, both fields (DataSource and DataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. There are two important differences between DataSource and DataSourceRef: * While DataSource only allows two specific types of objects, DataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects. * While DataSource ignores disallowed values (dropping them), DataSourceRef preserves all values, and generates an error if a disallowed value is specified. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled.

VolumeMount

(Appears on: FluentdComponent)

VolumeMount defines a hostPath type Volume mount.

Field	Description
`destination` string	(Optional) The destination path on the Fluentd Container, defaults to the source host path.
`readOnly` bool	(Optional) Specifies if the volume mount is read-only, defaults to `true`.
`source` string	The source host path.

VzStateType (`string` alias)

(Appears on: VerrazzanoStatus)

VzStateType identifies the state of a Verrazzano installation.

WebLogicOperatorComponent

(Appears on: ComponentSpec)

WebLogicOperatorComponent specifies the WebLogic Kubernetes Operator configuration.

Field Description

enabled
bool

(Optional)

If true, then the WebLogic Kubernetes Operator will be installed.

InstallOverrides
InstallOverrides

(Members of InstallOverrides are embedded into this type.)

(Optional)

Wildcard

(Appears on: DNSComponent)

Wildcard DNS type.

Field	Description
`domain` string	The type of wildcard DNS domain. For example, nip.io, sslip.io, and such.

Generated with gen-crd-api-reference-docs

Docs: Deregister a Managed Cluster

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: The following procedure is for a cluster in which Rancher is enabled on the admin cluster. If Rancher is not enabled, then additional steps are required to deregister a managed cluster, see Deregister a managed cluster without Rancher.

If you want to deregister a managed cluster because you no longer want it to be part of a Verrazzano multicluster environment, then log in to the Rancher console and delete the managed cluster. To delete a cluster in Rancher, see What if I don’t want my registered cluster managed by Rancher? This results in the cluster being deregistered from Verrazzano. The associated VerrazzanoManagedCluster resource will be automatically deleted, and, if present, then the Argo CD registration of the managed cluster also will be removed.

Alternatively, you can deregister a managed cluster by deleting the VerrazzanoManagedCluster resource. This will result in the automatic cleanup of the Rancher cluster, as well as the Argo CD registration, if it is present.

# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    delete vmc -n verrazzano-mc managed1

NOTE: Even after deregistration, any applications that you deployed previously to the managed cluster will continue running on that cluster.

Docs: DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Wildcard DNS Services

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a host name with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    dns:
      wildcard:
        domain: sslip.io

Oracle Cloud Infrastructure DNS

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.
You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI.
```
  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29
```
If you haven’t already uploaded your API signing public key through the console, follow the instructions in this section, How to upload the public key.

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principals require some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI configuration file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script.

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/master/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options.

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev].

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals, all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal.

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS.

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/master/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

Custom DNS

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for host names in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: Fluentd

Mon, 01 Jan 0001 00:00:00 +0000

Issue: Fluentd’s inability to push logs to OpenSearch due to a permission issue when SELinux is in enforcing mode.

If you are unable to see logs in OpenSearch, it may be due to a permission issue in Fluentd when SELinux is in enforcing mode, which prevents Fluentd from pushing the logs to OpenSearch.

To troubleshoot the Fluentd permission issue, follow these steps:

Check Fluentd pod logs.
- Identify the Fluentd pod related to the Verrazzano installation.
- View the logs of the Fluentd pod using the following command:
```
$ kubectl logs <fluentd-pod-name> -n verrazzano-system
```

Check for an error description.

Look for an error message in the Fluentd pod logs.
If you see an error with the following description, then follow the instructions in Step 3. Resolve permission issue.

unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/vz-fluentd-containers.log.pos", it indicates a permission issue. Fluentd doesn't have enough privilege to write `.pos` file.

Resolve permission issue.
- The issue occurs when SELinux is in enforcing mode on the worker nodes and Fluentd does not have the appropriate SELinux context to have read/write access to the logs directory.
- Check if SELinux is in enforcing mode by running this command on the worker nodes:
```
$ sudo getenforce
```
- If SELinux is in enforcing mode, then follow these steps:
  - To fix this issue, you need to override the default SELinux option in the Verrazzano Custom Resource.
  - Edit the Verrazzano CR and add the necessary SELinux options to provide read/write access to the logs directory in the Fluentd section. For example:
    spec: components: fluentd: overrides: - values: seLinuxOptions: type: spc_t
    NOTE
    The spc_t SELinux context is very permissive in that it gives the pod full access to the node on which it is running. If you don’t want the Fluentd pod to have the spc_t context, consider creating a custom SELinux context type with only the required privileges on all the worker nodes instead of using spc_t.
Verification.
- Verify that Fluentd is able to read and push the logs to OpenSearch by reviewing the Fluentd logs.
- Verify that the logs are visible in OpenSearch by accessing the OpenSearch dashboard or using relevant search queries.

Docs: Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Grafana provides visualization for your Prometheus metric data.

Single pod per cluster.
Named vmi-system-grafana-* in the verrazzano-system namespace.
Provides dashboards for metrics visualization.

To access Grafana:

Get the host name from the Grafana ingress.
```
$ kubectl get ingress vmi-system-grafana -n verrazzano-system

# Sample output
NAME                 CLASS    HOSTS                                              ADDRESS          PORTS     AGE
vmi-system-grafana   <none>   grafana.vmi.system.default.123.456.789.10.nip.io   123.456.789.10   80, 443   26h
```
Get the password for the user verrazzano.
```
$ kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo
```
Access Grafana in a browser using the host name.
Log in using the verrazzano user and the password.

Grafana dashboards

Verrazzano installs a number of preconfigured dashboards in Grafana. These dashboards are organized logically into folders. To view the folders, locate the Dashboards icon and select Manage.

Within each folder, you will find one or more preconfigured dashboards. The dashboards provide a visual representation of metrics for monitoring your applications, the Kubernetes clusters where your applications are running, and Verrazzano itself. You can add to these dashboards, and you can copy and modify the preconfigured dashboards.

Here are the folders and dashboard descriptions:

Coherence - A set of dashboards for monitoring Coherence applications. They provide metrics from Coherence clusters, Caches, Elastic Data, Federation Details, HTTP Servers, Kubernetes, Machines, Cluster Members, Persistence Summary, Proxy Servers, and Coherence Services.
Helidon - For Helidon workloads, provides the JVM details for Helidon applications, such as Status, Heap Usage, JVM Heap Sizes, Thread Count, HTTP Requests, and such.
Istio - A set of dashboards for monitoring the Verrazzano Istio service mesh and the workloads running in the mesh.
JVM - For monitoring applications deployed in Verrazzano.
NGINX - For monitoring the Verrazzano system ingress controller and the ingresses created using the ingress controller. Metrics include Controller Request Volume, Controller Connections, Controller Success Rate, Config Reloads, Network Pressure, and such.
OpenSearch - Provides metrics from OpenSearch clusters, such as Cluster Health, Shard Details, Index Details, JVM Metrics, and such.
Prometheus Operator - Provides metrics for the cluster compute resources, such as CPU Utilization, Memory Limits, and such.
WebLogic - For WebLogic workloads, provides WebLogic Server runtime metrics to monitor and diagnose the runtime deployment of WebLogic Server.

In addition, there are the following preconfigured Grafana Verrazzano dashboards. These dashboards are located in Verrazzano Application, Verrazzano Monitoring, and Verrazzano System folders:

Verrazzano Application contains the Application Status dashboard. This dashboard provides information on the overall health and performance of OAM applications that are deployed in Verrazzano and the details of the running pods. You can filter this information based on cluster, application, and component.
- When you deploy a new application in Verrazzano, the application is automatically added to the dashboard and becomes available in the list of Applications. You can select the specific application that you want to monitor and select which application metrics to view. Metrics are available in three rows:
  - Summary provides CPU and memory usage and pod status.
  - Storage provides disk and PVC (Persistent Volume Claims) usage.
  - Requests provides HTTP requests and request metrics.
Verrazzano Monitoring contains dashboards for monitoring the Kubernetes clusters where Verrazzano is running:
- alertmanager-mixin displays alerts created by Alertmanager, including the successful and invalid notifications sent by Alertmanager.
- etcd-mixin provides cluster metrics and shows additional information, such as active streams, RPC rate, database size, memory usage, client traffic, peer traffic, and such.
- kubernetes-mixin provides CPU, memory and network metrics based on cluster, pod, workload type, and nodes; network bandwidth based on cluster, pod, workload type, and nodes; persistent volume usage; and metrics for proxy and scheduler.
- node-exporter-mixin provides node level status for CPU, memory, disk, and network usage.

Verrazzano System has a set of dashboards that provide information on the health of all Verrazzano system components and the resource usage of Kubernetes resources across clusters.
- Resource Usage Detailed gives a detailed view of the resource usage for each of the Verrazzano system components, such as the application operator, cluster operator, monitoring operator, and platform operator. You can filter information based on cluster, component, and so on.
- Resource Usage Summary gives a summary of the resource usage that can be viewed for all components or for a specific component. You can filter information based on cluster, component, and so on. Metrics are available in three rows:
  - Summary provides CPU and memory usage and pod status.
  - Storage provides disk and PVC (Persistent Volume Claims) usage.
  - Requests provides HTTP requests and request metrics.
- System Health gives the health for all Verrazzano system components that can be viewed for local clusters or any other registered clusters.

Dynamic dashboard discovery and portability

Verrazzano enables dynamic discovery of Grafana dashboards. Using this feature, you can package the dashboard configuration and settings as a Kubernetes ConfigMap, and then deploy it individually or along with your application.

To configure a dashboard as a ConfigMap:

Create a dashboard as a JSON file. Or, you can export an existing Grafana dashboard as a JSON file. See Manage Dashboards.
Create a ConfigMap on the same cluster running Grafana that includes:
- The label, grafana_dashboard:"1", so that Grafana discovers the dashboard.
- The JSON code that defines and configures the dashboard (the dashboard JSON from step 1).
- Optionally, a k8s-sidecar-target-directory annotation, which specifies the Grafana folder name in which you want the dashboard to appear. The value of the annotation is the Grafana folder name. If not specified, the dashboard will appear in the default Grafana folder.

Here is an example dashboard ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    k8s-sidecar-target-directory: My App Dashboards
  labels:
    grafana_dashboard: "1"
  name: app-dashboard
  namespace: app
data:
  app_dashboard.json: |-
    {
      "title": "My App Dashboard",
      "uid": "Q4Bkkx",
      "version": 2,
      "panels": [
        {
          ...
        }
      ...
    }

After the ConfigMap is created, the dashboard gets loaded in Grafana.

Alternatively, you can create a ConfigMap from a file using kubectl create configmap, or generate ConfigMap from files. See Create a ConfigMap.

You can use dashboard ConfigMaps to deploy custom dashboards; these dashboards can be modified but cannot be saved in Grafana. If you want to modify the dashboard, then either update the ConfigMap or save the dashboard to a new location from the Grafana console.

Docs: Install Multicluster Verrazzano Without Rancher

Mon, 01 Jan 0001 00:00:00 +0000

Rancher is recommended for Verrazzano multicluster installations. However, if Rancher is not enabled on the admin cluster, then registration requires you to perform a few additional steps. The following instructions show you how to register a managed cluster when Rancher is not enabled in Verrazzano.

Prerequisites

Before you begin, make sure you have completed the Prerequisites and Install Verrazzano, as described in Install Multicluster Verrazzano.

Preregistration setup

Before registering the managed cluster, first you will need to set up the following items:

A Secret containing the managed cluster’s CA certificate. Note that the cacrt field in this secret can be empty only if the managed cluster uses a well-known CA. This CA certificate is used by the admin cluster to scrape metrics from the managed cluster, for both applications and Verrazzano components.
A ConfigMap containing the externally reachable address of the admin cluster. This will be provided to the managed cluster during registration so that it can connect to the admin cluster.

Follow these preregistration setup steps.

If needed for the admin cluster, then obtain the managed cluster’s CA certificate. The admin cluster scrapes metrics from the managed cluster’s Prometheus endpoint. If the managed cluster Verrazzano installation uses self-signed certificates, then the admin cluster will need the managed cluster’s CA certificate to make an https connection.
- Depending on whether the Verrazzano installation on the managed cluster uses self-signed certificates or certificates signed by a well-known certificate authority, choose the appropriate instructions:
  - Well-known CA
  - Self-signed certificates
- If you are unsure what type of certificates are used, use the following instructions.
  - To check the ca.crt field of the verrazzano-tls secret in the verrazzano-system namespace on the managed cluster:
    # On the managed cluster $ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \ -n verrazzano-system get secret verrazzano-tls -o jsonpath='{.data.ca\.crt}'
    If this value is empty, then your managed cluster is using certificates signed by a well-known certificate authority. Otherwise, your managed cluster is using self-signed certificates.
    
    Well-known CA
    
    In this case, no additional configuration is necessary.
    
    Self-signed certificates
    
    If the managed cluster certificates are self-signed, create a file called managed1.yaml containing the CA certificate of the managed cluster as the value of the cacrt field. In the following commands, the managed cluster’s CA certificate is saved in an environment variable called MGD_CA_CERT. Then use the --dry-run option of the kubectl command to generate the managed1.yaml file.
    # On the managed cluster $ export MGD_CA_CERT=$(kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \ get secret verrazzano-tls \ -n verrazzano-system \ -o jsonpath="{.data.ca\.crt}" | base64 --decode) $ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \ create secret generic "ca-secret-managed1" \ -n verrazzano-mc \ --from-literal=cacrt="$MGD_CA_CERT" \ --dry-run=client \ -o yaml > managed1.yaml
    Create a Secret on the admin cluster that contains the CA certificate for the managed cluster. This secret will be used for scraping metrics from the managed cluster. The managed1.yaml file that was created in the previous step provides input to this step.
    # On the admin cluster $ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \ apply -f managed1.yaml # After the command succeeds, you may delete the managed1.yaml file $ rm managed1.yaml
Use the following instructions to obtain the Kubernetes API server address for the admin cluster. This address must be accessible from the managed cluster.
- Most Kubernetes clusters
- Kind clusters
Most Kubernetes clusters

For most types of Kubernetes clusters, except for Kind clusters, you can find the externally accessible API server address of the admin cluster from its kubeconfig file.
```
# View the information for the admin cluster in your kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN config view --minify

# Sample output
apiVersion: v1
kind: Config
clusters:
- cluster:
  certificate-authority-data: DATA+OMITTED
  server: https://11.22.33.44:6443
  name: my-admin-cluster
contexts:
....
....
```
In the output of this command, you will find the URL of the admin cluster API server in the server field. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
$ export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```
Kind clusters

Kind clusters run within a Docker container. If your admin and managed clusters are Kind clusters, then the API server address of the admin cluster in its kubeconfig file is typically a local address on the host machine, which will not be accessible from the managed cluster. Use the kind command to obtain the internal kubeconfig of the admin cluster, which will contain a server address accessible from other Kind clusters on the same machine, and therefore in the same Docker network.
```
$ kind get kubeconfig --internal --name <your-admin-cluster-name> | grep server
```
In the output of this command, you can find the URL of the admin cluster API server in the server field. Set the value of the ADMIN_K8S_SERVER_ADDRESS variable to this URL.
```
$ export ADMIN_K8S_SERVER_ADDRESS=<the server address from the config output>
```
On the admin cluster, create a ConfigMap that contains the externally accessible admin cluster Kubernetes server address found in the previous step. To be detected by Verrazzano, this ConfigMap must be named verrazzano-admin-cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
apply -f <<EOF -
apiVersion: v1
kind: ConfigMap
metadata:
  name: verrazzano-admin-cluster
  namespace: verrazzano-mc
data:
  server: "${ADMIN_K8S_SERVER_ADDRESS}"
EOF
```

Registration steps

Perform the first three registration steps on the admin cluster, and the last step, on the managed cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster

To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster resource on the admin cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster object"
  caSecret: ca-secret-managed1
EOF
```
Wait for the VerrazzanoManagedCluster resource to reach the Ready status. At that point, it will have generated a YAML file that must be applied on the managed cluster to complete the registration process.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc
```
Export the YAML file created to register the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml
```

On the managed cluster

Apply the registration file exported in the previous step, on the managed cluster.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register.yaml

# After the command succeeds, you may delete the register.yaml file
$ rm register.yaml

After this step, the managed cluster will begin connecting to the admin cluster periodically. When the managed cluster connects to the admin cluster, it will update the Status field of the VerrazzanoManagedCluster resource for this managed cluster, with the following information:

The timestamp of the most recent connection made from the managed cluster, in the lastAgentConnectTime status field.
The host address of the Prometheus instance running on the managed cluster, in the prometheusHost status field. Then this is used by the admin cluster to scrape metrics from the managed cluster.
The API address of the managed cluster, in the apiUrl status field. This is used by the admin cluster’s authentication proxy to route incoming requests for managed cluster information, to the managed cluster’s authentication proxy.

Verify that managed cluster registration has completed

After these steps have been completed, return to Verify that managed cluster registration has completed.

Deregister a managed cluster without Rancher

NOTE: The following procedure is for a Verrazzano multicluster environment in which Rancher is not enabled on the admin cluster. If Rancher is enabled, see Deregister a managed cluster.

If you want to deregister a managed cluster because you no longer want it to be part of a Verrazzano multicluster environment, then complete the following steps.

On the admin cluster

Export the YAML file created to register the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml
```
Delete the VerrazzanoManagedCluster resource on the admin cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    delete vmc -n verrazzano-mc managed1
```

On the managed cluster

Delete the resources in the registration file exported in the previous step, on the managed cluster.

# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    delete -f register.yaml
# After the command succeeds, you may delete the register.yaml file
$ rm register.yaml

Docs: Keycloak and SSO

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano can be deployed to a number of different hosted and on-premises Kubernetes environments. Particularly in hosted environments, it may not be possible to choose the authentication providers configured for the Kubernetes API server, and Verrazzano may have no ability to view, manage, or authenticate users.

Verrazzano installs Keycloak to provide a common user store across all Kubernetes environments. The Verrazzano admin user can create and manage user accounts in Keycloak, and Verrazzano can authenticate and authorize Keycloak users.

Also, you can configure Keycloak to delegate authentication to an external user store, such as Active Directory or an LDAP server.

Because Keycloak is not configured as an authentication provider for the Kubernetes API, authenticating Keycloak users to Kubernetes requires the use of a proxy that impersonates Keycloak users when making Kubernetes API requests. For more information about the Verrazzano authentication proxy, see Verrazzano Proxies.

Keycloak is also used when authenticating to the Verrazzano Console and the various Verrazzano Monitoring Instance (VMI) logging and metrics consoles. The Verrazzano Console uses the OpenID Connect (OIDC) PKCE flow to authenticate users against Keycloak and obtain ID and access tokens. Authentication for VMI consoles is provided by the Verrazzano authentication proxy, which also uses PKCE to authenticate users, validates the resulting tokens, and authorizes incoming requests. For more information about the Verrazzano authentication proxy, see Verrazzano Proxies.

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano logging stack consists of Fluentd, OpenSearch, and OpenSearch Dashboards components.

Fluentd: a log aggregator that collects, processes, and formats logs from Kubernetes clusters.
OpenSearch: a scalable search and analytics engine for storing Kubernetes logs.
OpenSearch Dashboards: a visualization layer that provides a user interface to query and visualize collected logs.

As shown in the following diagram, logs written to stdout by a container running on Kubernetes are picked up by the kubelet service running on that node and written to /var/log/containers.

Fluentd sidecar

For components with multiple log streams or that cannot log to stdout, Verrazzano deploys a Fluentd sidecar which parses and translates the log stream. The resulting log is sent to stdout of the sidecar container and then written to /var/log/containers by the kubelet service.

For example, in a WebLogic deployment, AdminServer.log is consumed, translated, and written to stdout by the Fluentd sidecar. You can view these logs using kubectl on the container named fluentd-stdout-sidecar.

$ kubectl logs tododomain-adminserver \
   -n todo-list \
   -c fluentd-stdout-sidecar

The Verrazzano Fluentd Docker image comes with these plug-ins:

The Verrazzano Fluentd Docker image also has two local default plug-ins, kubernetes_parser and kubernetes_multiline_parser. These plug-ins help to parse Kubernetes management log files.

Here are example use cases for these plug-ins:

# ---- fluentd.conf ----
# kubernetes parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>

# kubernetes multi-line parser
<source>
  @type tail
  path ./kubelet.log
  read_from_head yes
  tag kubelet
  <parse>
     @type multiline_kubernetes
  </parse>
</source>
# ----   EOF      ----

Fluentd DaemonSet

Verrazzano deploys a Fluentd DaemonSet which runs one Fluentd replica per node in the verrazzano-system namespace. Each instance pulls logs from the node’s /var/log/containers directory and writes them to the target OpenSearch data stream. Verrazzano system applications receive special handling, and write their logs to the verrazzano-system data stream. Verrazzano application logs are exported to a data stream based on the application’s namespace, following this format: verrazzano-application-<application namespace>.

For example, vmi-system-opensearchDashboards logs written to /var/log/containers will be pulled by Fluentd and written to OpenSearch. The logs are exported to the verrazzano-system data stream, because vmi-system-opensearchDashboards is a Verrazzano system application. For a non-system application, if it is in the myapp namespace, then its logs will be exported to the verrazzano-application-myapp data stream.

OpenSearch

Verrazzano creates an OpenSearch cluster as the store and search engine for the logs processed by Fluentd. Records written by Fluentd can be queried using the OpenSearch REST API.

For example, you can use curl to get all of the OpenSearch data streams. First, you must get the password for the verrazzano user and the host for the Verrazzano Monitoring Instance (VMI) OpenSearch.

$ PASS=$(kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo)
$ HOST=$(kubectl get ingress \
    -n verrazzano-system vmi-system-os-ingest \
    -o jsonpath={.spec.rules[0].host})

$ curl -ik \
   --user verrazzano:$PASS https://$HOST/_data_stream

To see all of the records for a specific data stream, do the following:

$ DATA_STREAM=verrazzano-application-todo-list

$ curl -ik \
    --user verrazzano:$PASS https://$HOST/$DATA_STREAM/_search?q=message:*

Verrazzano provides support for Installation Profiles. The production profile (prod), which is the default, provides a 3-node OpenSearch and persistent storage for the VMI. The development profile (dev) provides a single node OpenSearch and no persistent storage for the VMI. The managed-cluster profile does not install OpenSearch or OpenSearch Dashboards in the local cluster; all logs are forwarded to the admin cluster’s OpenSearch instance.

If you want the logs sent to an external OpenSearch, instead of the default VMI OpenSearch, specify opensearchURL and opensearchSecret in the Fluentd Component configuration in your Verrazzano custom resource.

The following is an example of a Verrazzano custom resource to send the logs to the OpenSearch endpoint https://external-os.default.172.18.0.231.nip.io.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: default
spec:
  components:
    fluentd:
      opensearchURL: https://external-os.default.172.18.0.231.nip.io
      opensearchSecret: external-os-secret

For information on OpenSearch, see the Customize OpenSearch.

OpenSearch Dashboards

OpenSearch Dashboards is a visualization dashboard for the content indexed on an OpenSearch cluster. Verrazzano creates a OpenSearch Dashboards deployment to provide a user interface for querying and visualizing the log data collected in OpenSearch.

To access the OpenSearch Dashboards, read Access Verrazzano.

To see the records of an OpenSearch index or data stream through OpenSearch Dashboards, create an index pattern to filter for records under the desired data stream or index.

For example, to see the log records of a WebLogic application deployed to the todo-list namespace, create an index pattern of verrazzano-application-todo-*.

Docs: Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

This document describes some common problems you might encounter when using multicluster Verrazzano, and how to troubleshoot them.

If you created multicluster resources in the admin cluster, and specified a placement value in a managed cluster, then those resources will get created in that managed cluster. If they do not get created in the managed cluster, then use the following steps to troubleshoot:

Verify that the managed cluster is registered correctly and can connect to the admin cluster.
Verify that the VerrazzanoProject for the resource’s namespace, also has a placement in that managed cluster.
Check the multicluster resource’s status field on the admin cluster to know what the status of that resource is on each managed cluster to which it is targeted.

If you update the DNS of the admin cluster and notice that the managed cluster status is unavailable in the Rancher console, along with the error x509: certificate is valid for <rancher new url>, not <rancher old url> seen in the cattle-cluster-agent (Rancher Agent) logs on the managed cluster, then re-register the managed cluster, as described here.

Verify managed cluster registration and connectivity

You can verify that a managed cluster was successfully registered with an admin cluster by viewing the corresponding VerrazzanoManagedCluster (VMC) resource on the admin cluster. For example, to verify that a managed cluster named managed1 was successfully registered:

# on the admin cluster
$ kubectl get verrazzanomanagedcluster managed1 \
    -n verrazzano-mc \
    -o yaml

Partial sample output from the previous command.

  status:
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:27Z"
      message: Ready
      status: "True"
      type: Ready
    lastAgentConnectTime: "2021-06-22T21:06:04Z"
    ... other fields ...

Check the lastAgentConnectTime in the status of the VMC resource. This is the last time at which the managed cluster connected to the admin cluster. If this value is not present, or is not recent (within the last three minutes), then the managed cluster named managed1 cannot successfully connect to the admin cluster. This could be due to several reasons:

The managed cluster registration process step of applying the registration YAML on the managed cluster, was not completed. For the complete setup instructions, see here.
The managed cluster does not have network connectivity to the admin cluster. The managed cluster will attempt to connect to the admin cluster at regular intervals, and any errors will be reported in the verrazzano-application-operator pod’s log on the managed cluster. View the logs using the following command:
```
# on the managed cluster
$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator
```

If these logs reveal that there is a connectivity issue, then in the case of an installation that includes Rancher on the admin cluster, there may have been a problem with Verrazzano pushing registration details or updates to the managed cluster. Try exporting and applying the registration manifest to the managed cluster as shown:

# on the admin cluster
     kubectl get secret \
     -n verrazzano-mc verrazzano-cluster-managed1-manifest \
     -o jsonpath={.data.yaml} | base64 --decode > register.yaml

# on the managed cluster
     kubectl apply -f register.yaml

NOTE: If your installation disabled Rancher on the admin cluster, then check the admin cluster Kubernetes server address that you provided during registration and ensure that it is correct, and that it is reachable from the managed cluster. If it is incorrect, then you will need to repeat the managed cluster registration process described in the setup instructions here.

Verify VerrazzanoProject placement

For Verrazzano to create an application namespace in a managed cluster, that namespace must be part of a VerrazzanoProject that:

Includes that namespace.
Has a placement value that includes that managed cluster.

View the details of the project that corresponds to your application’s namespace. In the example command that follows, the project name is assumed to be myproject. All projects are expected to be created in the verrazzano-mc namespace.

# on the admin cluster
$ kubectl get verrazzanoproject myproject \
    -n verrazzano-mc \
    -o yaml

The following partial sample output is for a project that will result in the namespace mynamespace being created on the managed cluster managed1.

spec:
  placement:
    clusters:
    - name: managed1
  template:
    namespaces:
    - metadata:
        name: mynamespace
....other fields....

Check the multicluster resource status

On the admin cluster, each multicluster resource’s status field is updated with the status of the underlying resource on each managed cluster in which it is placed.

The following example command shows how to view the status of a MultiClusterApplicationConfiguration named myapp, in the namespace mynamespace, that has a placement value that includes the managed cluster managed1.

$ kubectl get multiclusterapplicationconfiguration myapp \
    -n mynamespace \
    -o yaml

The status of the underlying resource in each cluster specified in the placement is shown in the following partial sample output.

  status:
    clusters:
    - lastUpdateTime: "2021-06-22T21:05:04Z"
      message: OAM Application Configuration created
      name: managed1
      state: Succeeded
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:58Z"
      message: OAM Application Configuration created
      status: "True"
      type: DeployComplete
    state: Succeeded

The status message contains additional information on the operation’s success or failure.

Re-register the managed cluster

Perform the following steps to re-register the managed cluster with the admin cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster, export the register YAML file newly created on the admin cluster to re-register the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register_new.yaml
```
On the managed cluster, apply the registration file exported in the previous step.
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register_new.yaml

# After the command succeeds, you may delete the register_new.yaml file
$ rm register_new.yaml
```
On the admin cluster, run kubectl patch clusters.management.cattle.io to trigger redeployment of the Rancher agent on the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get clusters.management.cattle.io

# Sample output
NAME      AGE
c-mzb2h   4h48m
local     4h56m

$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    patch clusters.management.cattle.io <the managed cluster name from the above output> \
    -p '{"status":{"agentImage":"dummy"}}' --type merge

# Sample output
cluster.management.cattle.io/c-mzb2h patched
```

Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: Application Deployment Guide

Overview

What you need

Application development

Application deployment

Verrazzano components

Verrazzano application configurations

Deploy the application

Verify the deployment

Explore the application

Access the application’s logs

Access the application’s metrics

Suppress Kiali console warnings

Remove the application

Docs: Application Deployment

Review oam-kubernetes-runtime operator status

Review verrazzano-application-operator operator status

Review oam-kubernetes-runtime operator logs

Review verrazzano-application-operator logs

Review generated workload resources

Review generated Trait resources

Check for RBAC privilege issues

Check resource owners

Check related resources

Docs: Application Security

Keycloak

NOTE

Network security

NOTE

Pod security

Specify a non-root user in the container image

Specify security settings for the Pod

Helidon pod security

Pod security for ContainerizedWorkload applications

Pod security for applications using standard Kubernetes resources

Docs: Argo CD

Before you begin

Configure repositories

Deploy applications

Verify the deployed application

Undeploy applications

Docs: AuthProxy

Docs: Bob's Books

Before you begin

Overview

Deploy the application

NOTE

Access the applications using the WebLogic Server Administration Console

NOTE

Access bobs-bookstore

Access bobbys-front-end

Verify the deployed application

Undeploy the application

Docs: Considerations

Docs: Generic Kubernetes

Prepare for the generic install

NOTE

Customizations

Next steps

Docs: Install Multicluster

Prerequisites

NOTE

Install Verrazzano

Register managed clusters using the console

Next steps

NOTE

Docs: Install with CLI

Prerequisites

NOTE

Prepare for the installation

Perform the installation

Install Verrazzano

Verify the installation

Next steps

Docs: Install with kubectl

Prerequisites

NOTE

Prepare for the installation

Install the Verrazzano platform operator