Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: About Applications in Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses the Open Application Model (OAM) specification to provide a layered approach to describing and deploying applications. OAM is a specification developed within the Cloud Native Computing Foundation (CNCF). Verrazzano is compliant with the OAM specification version 0.2.1.

An ApplicationConfiguration is a composition of Components. Components encapsulate application implementation details. Application deployers apply Traits and Scopes to customize the Components for the environment.

The OAM specification supports extensibility. The behavior of the platform can be extended by adding OAM compliant definitions and controllers. Specifically, new workload, Trait, and Scope definitions can be added. These definitions can be referenced by Components and application configurations and are processed by custom controllers.

Application configurations

An ApplicationConfiguration is a collection of references to Components. A set of Traits and Scopes can be applied to each Component reference. The platform uses these Components, Traits, and Scopes to generate the final application resources during deployment.

The following sample shows the high level structure of an ApplicationConfiguration.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
...
spec:
  components:
    - componentName: example-component-1
      traits:
        ...
      scopes:
        ...
    - componentName: example-component-2
        ...

Components

A Component wraps the content of a workload. The platform extracts the workload during deployment and creates new resources that result from the application of Traits and Scopes. Verrazzano and the OAM specification provide several workloads, for example VerrazzanoHelidonWorkload and ContainerizedWorkload. The workloads also can be any Kubernetes resource. For some Kubernetes resources, the oam-kubernetes-runtime operator may need to be granted additional permission.

A Component can also be parameterized; this allows the workload content to be customized when referenced within an ApplicationConfiguration. See the OAM specification for details.

The following sample shows the high level structure of a Component.

apiVersion: core.oam.dev/v1alpha2
kind: Component
...
spec:
  workload:
    ...
  parameters:
    ...

For information about Traits, see Verrazzano Application Traits.
For information about Workloads, see Verrazzano Application Workloads.

Deployment

An application deployment occurs in Verrazzano through a number of Kubernetes controllers, reading and writing various resources. Each controller processes application resources, and generates or updates other related resources. Different types of controllers process different levels of application resources.

The ApplicationConfiguration controller processes ApplicationConfiguration and Component resources. This controller extracts and stores a workload for each Component referenced within ApplicationConfiguration resources. Verrazzano implements the ApplicationConfiguration controller within the oam-kubernetes-runtime operator. Similarly, the ApplicationConfiguration controller extracts and stores Trait resources associated with Component resources in the ApplicationConfiguration.

The workload controllers process workload resources created by the ApplicationConfiguration controller, for example ContainerizedWorkload or VerrazzanoWebLogicWorkload. This controller processes these workload resources and generates more specific runtime resources. For example, the ContainerizedWorkload controller processes a ContainerizedWorkload resource and generates a Deployment resource. The VerrazzanoWebLogicWorkload controller processes a VerrazzanoWebLogicWorkload resource and generates a Domain resource. These controllers may take into account Traits and Scopes that are applied to the workload’s Component references within the ApplicationConfiguration. Verrazzano implements these workload controllers in two operators. Verrazzano specific workloads, for example VerrazzanoHelidonWorkload, are processed by a controller within the verrazzano-application-operator. Workloads defined by OAM, for example ContainerizedWorkload, are processed by a controller with the oam-kubernetes-runtime operator.

The Trait controllers process Trait resources created by the ApplicationConfiguration controller, for example MetricsTrait. The ApplicationConfiguration controller records the Component to which it was applied within each extracted Trait. The Trait controllers process extracted Trait resources, and generate or update other related resources. For example, the IngressTrait controller within the verrazzano-application-operator processes IngressTrait resources and generates related Gateway and VirtualService resources. The same operator contains a MetricsTrait controller which processes MetricsTrait resources and adds annotations to related resources such as Deployments.

Scope controllers read Scope resources updated by the ApplicationConfiguration controller during deployment. The ApplicationConfiguration controller updates the Scope resources with references to each Component to which the Scope is applied.

The following diagram shows the relationships between the resources and controllers described previously.

The following diagram, based on the hello-helidon example, shows the processing of resources from a Kubernetes operator perspective. Controllers within the oam-kubernetes-runtime process the ApplicationConfiguration and Component resources and generate VerrazzanoHelidonWorkload and IngressTrait. Then controllers within the verrazzano-application-operator process the VerrazzanoHelidonWorkload and IngressTrait resources to generate Deployment, VirtualService, and other resources.

Docs: Analyze Verrazzano Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides the vz analyze command-line tool, which assists in troubleshooting issues in your environment. You can use it to analyze a cluster as well as, to analyze a cluster snapshot captured by the vz bug-report tool. For detailed information about vz bug-report, see here.

The vz analyze command-line tool analyzes the cluster or a cluster snapshot, reports the issues found, and prescribes related actions to take. Users, developers, and Continuous Integration (CI) can use this tooling to quickly identify the root cause of encountered problems, determine mitigation actions, and provide a sharable report with other users or tooling.

Set up the CLI tool

To set up the vz command-line tool, follow the steps here.

Analyze clusters

To analyze a Kubernetes cluster:

$ vz analyze

Analyze cluster snapshots

Use the vz bug-report tool to capture a cluster snapshot.

To create a bug report in a TAR file named my-bug-report.tar.gz and extract it to a directory my-cluster-snapshot:
```
$ vz bug-report my-bug-report.tar.gz
  mkdir my-cluster-snapshot
  tar -xvf my-bug-report.tar.gz -C my-cluster-snapshot
```
Use the vz analyze tool to analyze the cluster snapshot.

To perform an analysis of the cluster snapshot under my-cluster-snapshot:
```
$ vz analyze --capture-dir my-cluster-snapshot
```

Use the vz analyze tool to analyze multiple snapshots

The vz analyze tool will find and analyze all cluster snapshot directories found under a specified root directory. This lets you create a directory to hold the cluster snapshots of related clusters in subdirectories, which the tool can then analyze.

For example:

my-cluster-snapshots
    CAPTURE_DIR-1
        cluster-snapshot
            ...
    CAPTURE_DIR-2
        cluster-snapshot
            ...

To perform an analysis of the clusters under my-cluster-snapshots:

$ vz analyze --capture-dir my-cluster-snapshots

Usage information

Use the following syntax to run vz commands from your terminal window.

$ vz analyze [flags]

Available options

Command	Definition
`--capture-dir string`	Directory holding the captured data.
`-h, --help`	Help for the `vz analyze` command.
`--report-file string`	Name of the report output file. (Default `stdout`)
`--report-format string`	The format of the report output. Valid report formats are “summary” and “detailed”. (Default “summary”)
`-v, --verbose`	Enable verbose output.

Available flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`--kubeconfig string`	Path to the kubeconfig file to use.

Docs: Application Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides the following support.

Keycloak

Applications can use the Verrazzano Keycloak server as an Identity Provider. Keycloak supports SAML 2.0 and OpenID Connect (OIDC) authentication and authorization flows. Verrazzano does not provide any explicit integrations for applications.

NOTE

If using Keycloak for application authentication and authorization, create a new realm to contain application users and clients. Do not use the verrazzano-system realm, or the default (Keycloak system) realm. The Keycloak root user account (keycloakadmin) has privileges to create realms.

Network security

Verrazzano uses Istio to authenticate and authorize incoming network connections for applications. Verrazzano also provides support for configuring Kubernetes NetworkPolicy on Verrazzano projects. NetworkPolicy rules control where network connections can be made.

NOTE

Enforcement of NetworkPolicy requires that a Kubernetes Container Network Interface (CNI) provider, such as Calico, be configured for the cluster.

For more information on how Verrazzano secures network traffic, see Network Security.

Pod security

By default, all containers within a pod run as root (UID 0) within the container. Most applications do not require this level of access and doing so is considered a security risk.

It is recommended that applications attempt to meet the requirements of the Kubernetes restricted Pod Security Standard. This essentially means running the container within a pod as a non-root user with minimal capabilities, and without the ability to escalate privileges. Each container image also should define a non-root user identity that the container process will run, as by default, for added security.

In the Kubernetes Pod specification, there is a Pod SecurityContext for defining security at the pod level and a Container SecurityContext used to define security for containers. Some fields are common between the two security contexts, and others are unique. For details, see the API specifications for each. Where there is overlap, settings defined at the container level override settings defined at the pod level.

The following sections describe implementing these standards in more detail.

Specify a non-root user in the container image

Unless otherwise specified, all containers run as the root user. It is recommended that each container image build explicitly creates an unprivileged, non-root user and group, and then uses that with the USER instruction in the Dockerfile for the container.

To achieve this, modify the container’s image build and use the USER <UID> instruction.

For example:

# Run as user 1000
USER 1000

This will make the process within the container run as UID 1000. Even if there is no entry in /etc/passwd matching the UID declared, the container will run as the specified UID with minimal privileges.

For example, this is illustrated by a running image using the kubectl run command with the defaults:

% kubectl run -it --rm myol --image=ghcr.io/oracle/oraclelinux:7-slim --restart=Never -- bash
If you don't see a command prompt, try pressing enter.
bash-4.2# whoami
root
bash-4.2# id
uid=0(root) gid=0(root) groups=0(root)
bash-4.2#

To run the same image as a non-root user, you can override the default user and group for the container process, as shown:

% kubectl run -it --rm myol --image=ghcr.io/oracle/oraclelinux:7-slim --restart=Never --overrides='{ "spec": { "securityContext": { "runAsUser": 1000, "runAsGroup": 1000, "runAsNonRoot": true } } }' -- bash
If you don't see a command prompt, try pressing enter.
bash-4.2$
bash-4.2$ whoami
whoami: cannot find name for user ID 1000
bash-4.2$ id
uid=1000 gid=1000 groups=1000

In the second example, the container is running as UID 1000 with a GID of 1000. Running whoami from within the container returns an error because USER 1000 is not defined in /etc/passwd, but running the id command from the shell shows that the container process is indeed running as the desired UID (1000).

Specify security settings for the Pod

By default, containers within Kubernetes pods run as the image default user, which in turn defaults to the root user (UID 0).

You can use the pod and container securityContext fields to force containers within a pod to run as non-root and prevent the container from acquiring escalated privileges. These will override any USER setting within the image.

apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
    ...
    spec:
      # Define a security context for all containers in the pod
      securityContext:
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: some-container
        ...
        # Define a security context for the container; settings defined here have precedence over the pod securityContext
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
      ...

As mentioned previously, where there is overlap between the pod and container security settings, the settings defined at the container level override settings defined at the pod level.

Helidon pod security

The following YAML shows how to explicitly specify the pod security context for a Helidon application. With these settings, the Helidon application will meet the requirements of the Kubernetes restricted Pod Security Standard.

Note that the runAsUser 2000 UID does not exist in the container, as described previously.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: hello-helidon-component
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    metadata:
      name: hello-helidon-workload
      labels:
        app: hello-helidon
        version: v1
    spec:
      deploymentTemplate:
        metadata:
          name: hello-helidon-deployment
        podSpec:
          securityContext:
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: hello-helidon-container
...
              securityContext:
                runAsNonRoot: true
                runAsGroup: 2000
                runAsUser: 2000
                privileged: false
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL

Pod security for ContainerizedWorkload applications

The only means for controlling pod security for the ContainerizedWorkload type is to specify a non-root user, using the USER instruction in the container image build, as described in this section, Specify a non-root user in the container image.

Pod security for applications using standard Kubernetes resources

You can deploy applications using standard Kubernetes resources. You configure security for these resources as you typically would for any Kubernetes Deployment resource.

For example:

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: example-deployment
spec:
  workload:
    kind: Deployment
    apiVersion: apps/v1
    name: oam-kube-dep
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: oam-kube-app
      template:
        metadata:
          labels:
            app: oam-kube-app
        spec:
          securityContext:
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: oam-kube-cnt
              image: hashicorp/http-echo
              args:
                - "-text=hello"
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                    - ALL
                privileged: false

Docs: Bob's Books

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Install Verrazzano by following the installation instructions.
To download the example image, you must first accept the license agreement.
- In a browser, navigate to https://container-registry.oracle.com/ and sign in.
- Search for example-bobbys-coherence, example-bobbys-front-end, example-bobs-books-order-manager, example-roberts-coherence, and weblogic.
- For each one:
  - Select the image name in the results.
  - From the drop-down menu, select your language and click Continue.
  - Then read and accept the license agreement.
NOTE: The Bob’s Books example application deployment files are contained in the Verrazzano project located at <VERRAZZANO_HOME>/examples/bobs-books, where <VERRAZZANO_HOME> is the root of the Verrazzano project.

Overview

Bob’s Books consists of three main parts:

A back-end “order processing” application, which is a Java EE application with REST services and a very simple JSP UI, which stores data in a MySQL database. This application runs on WebLogic Server.
A front-end web store “Robert’s Books”, which is a general book seller. This is implemented as a Helidon microservice, which gets book data from Coherence, uses a Coherence cache store to persist data for the order manager, and has a React web UI.
A front-end web store “Bobby’s Books”, which is a specialty children’s book store. This is implemented as a Helidon microservice, which gets book data from a (different) Coherence cache store, interfaces directly with the order manager, and has a JSF web UI running on WebLogic Server.

For more information and the source code of this application, see the Verrazzano Examples.

Deploy the application

NOTE

To run this application in the default namespace:

$ kubectl label namespace default verrazzano-managed=true istio-injection=enabled

If you chose the default namespace, you can skip Step 1 and ignore the -n option in the rest of the commands.

Create a namespace for the example and add a label identifying the namespace as managed by Verrazzano.
```
 $ kubectl create namespace bobs-books
 $ kubectl label namespace bobs-books verrazzano-managed=true istio-injection=enabled
```
Create a docker-registry secret to enable pulling the example image from the registry.
```
$ kubectl create secret docker-registry bobs-books-repo-credentials \
        --docker-server=container-registry.oracle.com \
        --docker-username=YOUR_REGISTRY_USERNAME \
        --docker-password=YOUR_REGISTRY_PASSWORD \
        --docker-email=YOUR_REGISTRY_EMAIL \
        -n bobs-books
```
Replace YOUR_REGISTRY_USERNAME, YOUR_REGISTRY_PASSWORD, and YOUR_REGISTRY_EMAIL with the values you use to access the registry.
Create secrets for the WebLogic domains:
```
  # Replace the values of the WLS_USERNAME and WLS_PASSWORD environment variables as appropriate.
  $ export WLS_USERNAME=<username>
  $ export WLS_PASSWORD=<password>
  $ kubectl create secret generic bobbys-front-end-weblogic-credentials \
      --from-literal=password=$WLS_PASSWORD \
      --from-literal=username=$WLS_USERNAME \
      -n bobs-books

  $ kubectl create secret generic bobs-bookstore-weblogic-credentials \
      --from-literal=password=$WLS_PASSWORD \
      --from-literal=username=$WLS_USERNAME \
      -n bobs-books

  $ kubectl create secret generic mysql-credentials \
      --from-literal=username=$WLS_USERNAME \
      --from-literal=password=$WLS_PASSWORD \
      --from-literal=url=jdbc:mysql://mysql.bobs-books.svc.cluster.local:3306/books \
      -n bobs-books
```
Note that the example application is preconfigured to use specific secret names. For the source code of this application, see the Verrazzano Examples. If you want to use secret names that are different from what is specified in the source code, you will need to update the corresponding YAML file and rebuild the Docker images for the example application.
To deploy the application, apply the example resources.
```
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-comp.yaml -n bobs-books
$ kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-app.yaml -n bobs-books
```
Wait for all of the pods in the Bob’s Books example application to be ready. You can monitor their progress by listing the pods and inspecting the output, or you can use the kubectl wait command.

You may need to repeat the kubectl wait command several times before it is successful. The WebLogic Server and Coherence pods may take a while to be created and Ready.
```
$ kubectl get pods -n bobs-books

# -or- #

$ kubectl wait \
    --for=condition=Ready pods \
    --all -n bobs-books \
    --timeout=600s
```
Get the EXTERNAL_IP address of the istio-ingressgateway service.
```
$ ADDRESS=$(kubectl get service \
    -n istio-system istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo $ADDRESS

# Sample output
11.22.33.44
```
Get the generated host name for the application.
```
$ HOST=$(kubectl get gateways.networking.istio.io bobs-books-bobs-books-gw \
    -n bobs-books \
    -o jsonpath='{.spec.servers[0].hosts[0]}')
$ echo $HOST

# Sample output
bobs-books.bobs-books.11.22.33.44.nip.io
```
Access the application. To access the application in a browser, you will need to do one of the following:
- Option 1: If you are using nip.io, then you can access the application using the generated host name. For example:
  - Robert’s Books UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/.
  - Bobby’s Books UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/bobbys-front-end/.
  - Bob’s order manager UI at https://bobs-books.bobs-books.11.22.33.44.nip.io/bobs-bookstore-order-manager/orders.
- Option 2: Temporarily, modify the /etc/hosts file (on Mac or Linux) or c:\Windows\System32\Drivers\etc\hosts file (on Windows 10), to add an entry mapping the host used by the application to the external IP address assigned to your gateway. For example:
```
11.22.33.44 bobs-books.example.com
```
  Then, you can use a browser to access the application, as shown:
  - Robert’s Books UI at https://bobs-books.example.com/.
  - Bobby’s Books UI at https://bobs-books.example.com/bobbys-front-end/.
  - Bob’s order manager UI at https://bobs-books.example.com/bobs-bookstore-order-manager/orders.
- Option 3: Alternatively, point your own DNS name to the load balancer’s external IP address. In this case, you would need to have edited the bobs-books-app.yaml file to use the appropriate values under the hosts section for the application (such as your-roberts-books-host.your.domain), before deploying the application. Then, you can use a browser to access the application, as shown:
  - Robert’s Books UI at https://<your-roberts-books-host.your.domain>/.
  - Bobby’s Books UI at https://<your-bobbys-books-host.your.domain>/bobbys-front-end/.
  - Bob’s order manager UI at https://<your-bobs-orders-host.your.domain>/.

Access the applications using the WebLogic Server Administration Console

Use the WebLogic Server Administration Console to access the applications as follows.

NOTE

It is recommended that the WebLogic Server Administration Console not be exposed publicly.

Access bobs-bookstore

Set up port forwarding.
```
$ kubectl port-forward pods/bobs-bookstore-adminserver 7001:7001 -n bobs-books
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

Access bobbys-front-end

Set up port forwarding.
```
$ kubectl port-forward pods/bobbys-front-end-adminserver 7001:7001 -n bobs-books
```
NOTE: If you are using the Oracle Cloud Infrastructure Cloud Shell to run kubectl, in order to access the WebLogic Server Administration Console using port forwarding, you will need to run kubectl on another machine.
Access the WebLogic Server Administration Console from your browser.
```
http://localhost:7001/console
```

Verify the deployed application

Verify that the application configuration, domains, Coherence resources, and ingress trait all exist.
```
$ kubectl get ApplicationConfiguration -n bobs-books
$ kubectl get Domain -n bobs-books
$ kubectl get Coherence -n bobs-books
$ kubectl get IngressTrait -n bobs-books
```
Verify that the service pods are successfully created and transition to the READY state. Note that this may take a few minutes and that you may see some of the services terminate and restart.
```
$ kubectl get pods -n bobs-books

# Sample output
NAME                                                READY   STATUS    RESTARTS   AGE
bobbys-helidon-stock-application-868b5965c8-dk2xb   3/3     Running   0          19h
bobbys-coherence-0                                  2/2     Running   0          19h
bobbys-front-end-adminserver                        3/3     Running   0          19h
bobbys-front-end-managed-server1                    3/3     Running   0          19h
bobs-bookstore-adminserver                          3/3     Running   0          19h
bobs-bookstore-managed-server1                      3/3     Running   0          19h
mysql-669665fb54-9m8wq                              2/2     Running   0          19h
robert-helidon-96997fcd5-kzjkf                      3/3     Running   0          19h
robert-helidon-96997fcd5-nlswm                      3/3     Running   0          19h
roberts-coherence-0                                 2/2     Running   0          17h
roberts-coherence-1                                 2/2     Running   0          17h
```

Undeploy the application

To undeploy the application, delete the Bob’s Books OAM resources.
```
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-app.yaml -n bobs-books
$ kubectl delete -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/bobs-books/bobs-books-comp.yaml -n bobs-books
```
Delete the namespace bobs-books after the application pods are terminated. The secrets created for the WebLogic domain also will be deleted.
```
$ kubectl delete namespace bobs-books
```

Docs: Cluster API

Mon, 01 Jan 0001 00:00:00 +0000

The Cluster API (CAPI) project was developed as a sub-project of Kubernetes and strives to standardize a set of Kubernetes-style APIs for cluster management. External organizations can then build upon these standard APIs to develop custom cluster management solutions.

Learn more about CAPI at Kubernetes Cluster API Documentation.

Verrazzano incorporates CAPI functionality through the clusterAPI component, which provides the ability to quickly design and deploy clusters and then continue managing your clusters throughout their life cycle, all from within Verrazzano.

NOTE

The terminology around clusters differs between CAPI and Verrazzano though the underlying concepts are the same. What CAPI calls Management and Workload clusters are equivalent to admin and managed clusters, respectively, in Verrazzano.

CAPI splits cluster management responsibilities across three main components, which it calls providers:

Infrastructure providers standardize the host environment by provisioning any infrastructure or computational resources required by the cluster or machine.
Bootstrap providers streamline the node creation process by converting servers into Kubernetes nodes.
Control plane providers work with the Kubernetes API to regulate your clusters, ensuring that they always strive toward a desired state.

Currently, Verrazzano supports the CAPI provider for Oracle Cloud Native Environment (CAPOCNE) which bundles a bootstrap and a control plane provider together and works with the CAPOCI infrastructure provider offered by Oracle Cloud Infrastructure (OCI).

During the setup process, the bootstrap provider converts a cluster into an admin cluster - a Kubernetes cluster that controls any other, subordinate or managed clusters. It generates certificates, starts and manages the creation of additional nodes, and handles the addition of control plane and worker nodes to the cluster.

Next, a CAPI infrastructure provider will provision the first instance on the cloud provider and generate a provider ID, a unique identifier that any future nodes and clusters will use to associate with the instance. It will also create a kubeconfig file. The first control plane node is ready after these are created.

After the admin cluster is up and running, you can use the clusterAPI component to create additional managed clusters.

Docs: Create a Kubernetes Deployment and Service

Mon, 01 Jan 0001 00:00:00 +0000

Create a namespace for the example application and add labels identifying the namespace as managed by Verrazzano and enabled for Istio.
```
$ kubectl create namespace hello-helidon
$ kubectl label namespace hello-helidon verrazzano-managed=true istio-injection=enabled
```
Deploy the Hello Helidon Greet application.
```
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    description: Hello Helidon application
    version: v1.0.0
  labels:
    app: hello-helidon
  name: hello-helidon-deployment
  namespace: hello-helidon
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: hello-helidon
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        verrazzano.io/metricsEnabled: "true" # This annotation is used in the ServiceMonitor YAML to specify whether metrics are enabled.
        verrazzano.io/metricsPath: /metrics  # This annotation is used in the ServiceMonitor YAML to replace the metrics path.
        verrazzano.io/metricsPort: "8080"    # This annotation is used in the ServiceMonitor YAML to replace the metrics port.
      labels:
        app: hello-helidon # This label is used in the Service YAML in the selector.
    spec:
      containers:
      - image: ghcr.io/verrazzano/example-helidon-greet-app-v1:1.0.0-1-20230126194830-31cd41f
        imagePullPolicy: IfNotPresent
        name: hello-helidon-container
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
```
Access the Hello Helidon Greet application inside the Kubernetes cluster.
```
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-helidon
  name: hello-helidon-deployment
  namespace: hello-helidon
spec:
  ports:
  - name: hello-helidon-container-8080
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: hello-helidon
```

Docs: Customize AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

Path to Field	Description
`spec.components.authProxy.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.authProxy.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes. spec: components: authProxy: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - verrazzano-authproxy topologyKey: kubernetes.io/hostname

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authProxy:
      overrides:
      - values:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

Docs: Install Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Before you begin, read this document, Verrazzano in a multicluster environment.
To set up a multicluster Verrazzano environment, you will need two or more Kubernetes clusters. One of these clusters will be the admin cluster; the others will be managed clusters. For instructions on preparing Kubernetes platforms for installing Verrazzano, see Platform Setup.

NOTE

If Rancher is not enabled, then refer to Advanced Multicluster Installation because additional steps are required to register a managed cluster.

The following instructions assume an admin cluster and a single managed cluster. For each additional managed cluster, simply repeat the managed cluster set up and registration instructions.

Set up the admin cluster

One of your clusters will be designated as the admin cluster. This cluster will serve as the central management point for your managed clusters. On the admin cluster, install Verrazzano using the dev or prod profile.

For detailed instructions on how to install and customize Verrazzano on a Kubernetes cluster using a specific profile, see the Installation Guide and Installation Profiles.

Set up the managed cluster

The second cluster will be a managed cluster. On this cluster, install Verrazzano using the managed-cluster profile. The managed-cluster profile contains only the components that are required for a full-featured Verrazzano managed cluster.
NOTE: You also can use the dev or prod profile, which will install components not required for a managed cluster.

For managed cluster configurations that have a smaller footprint than the managed-cluster profile, with correspondingly limited features, see Minimal managed cluster configurations.

Register the managed cluster

Register the managed clusters in a multicluster environment by following a registration method found here. After registration is completed, your Verrazzano multicluster environment is ready to use.

Minimal managed cluster configurations

You can use minimal Verrazzano managed cluster configurations in cases where managed clusters need to have a smaller footprint, such as in edge clusters. Note that using smaller managed cluster configurations results in limited multicluster features.

These examples use Verrazzano with profile: none, which means that no Verrazzano components are enabled by default, and then individually enables the specific Verrazzano components needed.

Minimum configuration

To install Verrazzano on a managed cluster with the smallest possible configuration and the fewest features, you can use the minimal.yaml file.

In this configuration:

No metrics are collected for the managed cluster.
You can view and interact with the managed cluster in the Verrazzano Dashboard on the admin cluster.
If you have enabled the argoCD component on the admin cluster, you can use ArgoCD GitOps to distribute applications from the admin cluster to the managed cluster.

Observability using Prometheus

Use the minimal-prometheus.yaml file to install Verrazzano on the managed cluster, with the minimum footprint needed to get Prometheus federation of metrics.

In this configuration:

The features of the minimum configuration will be available.
Metrics will be collected on the managed cluster.
Managed cluster metrics will be available in Prometheus on the admin cluster.

Observability using Thanos

Use the minimal-thanos.yaml file to install Verrazzano on the managed cluster, with the minimum footprint needed to use Thanos to collect metrics.

To use Thanos to get managed cluster metrics, you also must enable Thanos on the admin cluster. For details, and additional Thanos configuration information, see the documentation on how to enable and configure Thanos.

In this configuration:

The features of the minimum configuration are available.
Metrics will be collected on the managed cluster.
Managed cluster metrics will be available in Thanos on the admin cluster.

Docs: Install With CLI

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster using the CLI.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster.

Prepare for the installation

Before installing Verrazzano, see instructions on preparing Kubernetes platforms and installing the Verrazzano CLI (optional). Make sure that you have a valid kubeconfig file pointing to the Kubernetes cluster that you want to use for installing Verrazzano.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For instructions on how to install a CNI plug-in, see the documentation for your Kubernetes cluster.

Perform the installation

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations. When applying the Verrazzano custom resource, the Verrazzano CLI deploys and installs the Verrazzano platform operator; you need only to install Verrazzano as described in the following section.

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information, see Installation Profiles.

This document shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE: Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For an overview of how to configure Verrazzano, see Modify Verrazzano Installations. For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see Customizing DNS for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands.

Install Verrazzano with its dev profile.
```
$ vz install -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: verrazzano-storage
  volumeClaimSpecTemplates:
    - metadata:
        name: verrazzano-storage
      spec:
        resources:
          requests:
            storage: 2Gi
EOF
```

This command installs the Verrazzano platform operator and applies the Verrazzano custom resource.

Wait for the installation to complete. Installation logs will be streamed to the command window until the installation has completed or until the default timeout (30m) has been reached.

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

Next steps

Verify the installed Verrazzano using the vz CLI. See Verify using CLI.

Docs: Install With kubectl

Mon, 01 Jan 0001 00:00:00 +0000

The following instructions show you how to install Verrazzano in a single Kubernetes cluster using kubectl.

Prerequisites

Find the Verrazzano prerequisite requirements here.
Review the list of the software versions supported and installed by Verrazzano.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster.

Prepare for the installation

You can install Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

Install the Verrazzano platform operator

Verrazzano provides a platform operator to manage the life cycle of Verrazzano installations. Using the Verrazzano custom resource, you can install, uninstall, and upgrade Verrazzano installations.

To install the Verrazzano platform operator:

Deploy the Verrazzano platform operator.
```
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/master/verrazzano-platform-operator.yaml
```
Wait for the deployment to complete.
```
$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
```
```
# Expected response
deployment "verrazzano-platform-operator" successfully rolled out
```
Confirm that the operator pod is correctly defined and running.
```
 $ kubectl -n verrazzano-install get pods
```
```
 # Sample output
 NAME                                            READY   STATUS    RESTARTS   AGE
 verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s
```

Perform the installation

Verrazzano supports the following installation profiles: development (dev), production (prod), and managed cluster (managed-cluster). For more information, see Installation Profiles.

This document shows how to create a basic Verrazzano installation using:

The development (dev) installation profile
Wildcard-DNS, where DNS is provided by nip.io (the default)

NOTE: Because the dev profile installs self-signed certificates, when installing Verrazzano on macOS, you might see: Your connection is not private. For a workaround, see this FAQ.

For a complete description of Verrazzano configuration options, see the Verrazzano Custom Resource Definition.

To use other DNS options, see Customzing DNS for more details.

Install Verrazzano

To create a Verrazzano installation as described in the previous section, run the following commands.

$ kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: ${VZ_PROFILE:-dev}
EOF

$ kubectl wait \
  --timeout=20m \
  --for=condition=InstallComplete verrazzano/example-verrazzano

NOTE: If you are installing Verrazzano in an Azure Kubernetes Service (AKS) cluster, add the following annotation for ingressNGINX to access Verrazzano endpoints on Azure.

$ kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingressNGINX:
      overrides:
      - values:
          controller:
            service:
              annotations:
                service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
    dns:
      wildcard:
        domain: nip.io
EOF

To use a different profile with the previous example, set the VZ_PROFILE environment variable to the name of the profile you want to install.

If an error occurs, check the log output of the installation. You can view the logs with the following command.

  $ kubectl logs -n verrazzano-install \
      -f $(kubectl get pod \
      -n verrazzano-install \
      -l app=verrazzano-platform-operator \
      -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
      | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

Next steps

Verify the installed Verrazzano using kubectl. See Verify using CLI.

Docs: Installed Software

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano installs a curated set of open source software. The following table lists the software, its version, and a brief description.

Software	Version	Description
Argo CD	2.7.2	A declarative, GitOps continuous delivery tool for Kubernetes.
cert-manager	1.9.1	Automates the management and issuance of TLS certificates.
cert-manager-webhook-oci	0.1.0	Cert-Manager webhook for Oracle Clould Infrastructure DNS.
Cluster API	1.4.2	Specifies the Cluster API configuration.
Coherence Operator	3.2.10	Assists with deploying and managing Coherence clusters.
ExternalDNS	0.12.2	Synchronizes exposed Kubernetes Services and ingresses with DNS providers.
Fluentd	1.14.5	Collects logs and sends them to OpenSearch.
Fluent Operator	2.2.0	Provides management for FluentBit and Fluentd in a Kubernetes environment.
Grafana	7.5.17	Tool to help you examine, analyze, and monitor metrics.
Istio	1.15.3	Service mesh that layers transparently onto existing distributed applications.
Jaeger	1.42.0	Distributed tracing system for monitoring and troubleshooting distributed systems.
Jaeger Operator	1.42.0	Provides management for Jaeger tools.
Keycloak	20.0.1	Provides single sign-on with Identity and Access Management.
Kiali	1.57.1	Management console for the Istio service mesh.
kube-state-metrics	2.8.2	Provides metrics about the state of Kubernetes API objects.
MySQL	8.0.32	Open source relational database management system used by Keycloak.
MySQL Operator	8.0.32-2.0.8	Operator for managing MySQL InnoDB Cluster setups inside a Kubernetes cluster.
NGINX Ingress Controller	1.7.1	Traffic management solution for cloud‑native applications in Kubernetes.
Node Exporter	1.3.1	Prometheus exporter for hardware and OS metrics.
OAM Kubernetes Runtime	0.3.3	Plug-in for implementing the Open Application Model (OAM) control plane with Kubernetes.
OpenSearch	2.3.0	Provides a distributed, multitenant-capable full-text search engine.
OpenSearch Dashboards	2.3.0	Provides search and data visualization capabilities for data indexed in OpenSearch.
Prometheus	2.44.0	Provides event monitoring and alerting.
Prometheus Adapter	0.10.0	Provides metrics in support of pod autoscaling.
Prometheus Operator	0.64.1	Provides management for Prometheus monitoring tools.
Prometheus Pushgateway	1.4.2	Allows ephemeral and batch jobs to expose their metrics to Prometheus.
Rancher	2.7.3	Manages multiple Kubernetes clusters.
Rancher Backup Operator	3.1.0	Manages backup and restore of Rancher configurations and data.
Thanos	0.30.2	Integrates with Prometheus to access and store metrics data.
Velero	1.9.1	Manages backup and restore of Kubernetes configurations and data.
WebLogic Kubernetes Operator	4.1.0	Assists with deploying and managing WebLogic domains.
WebLogic Monitoring Exporter	2.1.4	Exports Prometheus-compatible metrics from WebLogic instances.

Docs: Logging Components in Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano logging stack consists of Fluentd, OpenSearch, and OpenSearch Dashboards components.

Fluentd: a log aggregator that collects, processes, and formats logs from Kubernetes clusters.
OpenSearch: a scalable search and analytics engine for storing Kubernetes logs.
OpenSearch Dashboards: a visualization layer that provides a user interface to query and visualize collected logs.

As shown in the following diagram, logs written to stdout by a container running on Kubernetes are picked up by the kubelet service running on that node and written to /var/log/containers.

Docs: Multicluster and Verrazzano Project

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

clusters.verrazzano.io/v1alpha1

clusters.verrazzano.io/v1alpha1

Resource Types:

MultiClusterApplicationConfiguration

MultiClusterApplicationConfiguration specifies the multicluster application API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterApplicationConfiguration

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterApplicationConfigurationSpec

The desired state of a multicluster application resource.

`placement` Placement	Clusters in which the application is to be created.
`secrets` []string	(Optional) List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.
`template` ApplicationConfigurationTemplate	Template containing the metadata and spec for an OAM applicationConfiguration resource.

status
MultiClusterResourceStatus

The observed state of a multicluster application resource.

MultiClusterComponent

MultiClusterComponent specifies the MultiCluster Component API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterComponent

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterComponentSpec

The desired state of a MultiCluster Component resource.

`placement` Placement	Clusters in which the component is to be created.
`template` ComponentTemplate	Template containing the metadata and spec for an OAM component.

status
MultiClusterResourceStatus

The observed state of a MultiCluster Component resource.

MultiClusterConfigMap

MultiClusterConfigMap specifies the MultiCluster ConfigMap API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterConfigMap

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterConfigMapSpec

The desired state of a MultiCluster ConfigMap resource.

`placement` Placement	Clusters in which the ConfigMap is to be created.
`template` ConfigMapTemplate	The embedded Kubernetes ConfigMap.

status
MultiClusterResourceStatus

The observed state of a MultiCluster ConfigMap resource.

MultiClusterSecret

MultiClusterSecret specifies the MultiCluster Secret API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string MultiClusterSecret

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MultiClusterSecretSpec

The desired state of a MultiCluster Secret resource.

`placement` Placement	Clusters in which the secret is to be created.
`template` SecretTemplate	The embedded Kubernetes secret.

status
MultiClusterResourceStatus

The observed state of a MultiCluster Secret resource.

VerrazzanoProject

VerrazzanoProject specifies the Verrazzano Projects API.

Field Description

apiVersion
string clusters.verrazzano.io/v1alpha1

kind
string VerrazzanoProject

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoProjectSpec

The desired state of a Verrazzano Project resource.

`placement` Placement	Clusters on which the namespaces are to be created.
`template` ProjectTemplate	The project template.

status
MultiClusterResourceStatus

The observed state of a Verrazzano Project resource.

ApplicationConfigurationTemplate

(Appears on:MultiClusterApplicationConfigurationSpec)

ApplicationConfigurationTemplate has the metadata and embedded spec of the OAM applicationConfiguration resource.

Field Description

metadata
EmbeddedObjectMeta

Metadata describing the application.

spec
OAM core/v1alpha2.ApplicationConfigurationSpec

The embedded OAM application specification.

components
[]OAM core/v1alpha2.ApplicationConfigurationComponent

Components of which this ApplicationConfiguration consists. Each component will be used to instantiate a workload.

Cluster

(Appears on:Placement)

Cluster contains the name of a single cluster.

Field	Description
`name` string	The name of a cluster.

ClusterLevelStatus

(Appears on:MultiClusterResourceStatus)

ClusterLevelStatus describes the status of the multicluster resource in a specific cluster.

Field	Description
`lastUpdateTime` string	Last update time of the resource state in this cluster.
`message` string	Message details about the status in this cluster.
`name` string	Name of the cluster.
`state` StateType	State of the resource in this cluster.

ComponentTemplate

(Appears on:MultiClusterComponentSpec)

ComponentTemplate has the metadata and embedded spec of the OAM component.

Field Description

metadata
EmbeddedObjectMeta

Metadata describing the component.

spec
OAM core/v1alpha2.ComponentSpec

The embedded OAM component specification.

`workload` Kubernetes runtime.RawExtension	A Workload that will be created for each ApplicationConfiguration that includes this Component. Workload is an instance of a workloadDefinition. We either use the GVK info or a special “type” field in the workload to associate the content of the workload with its workloadDefinition
`parameters` []OAM core/v1alpha2.ComponentParameter	(Optional) Parameters exposed by this component. ApplicationConfigurations that reference this component may specify values for these parameters, which will in turn be injected into the embedded workload.

Condition

(Appears on:MultiClusterResourceStatus)

Condition describes current state of a multicluster resource.

Field	Description
`lastTransitionTime` string	(Optional) Last time the condition transitioned from one status to another.
`message` string	(Optional) A message with details about the last transition.
`status` Kubernetes core/v1.ConditionStatus	Status of the condition: one of `True`, `False`, or `Unknown`.
`type` ConditionType	Type of condition.

ConditionType (`string` alias)

(Appears on:Condition)

ConditionType identifies the condition of the multicluster resource which can be checked with kubectl wait.

Value	Description
"DeployComplete"	DeployComplete means deployment to the specified cluster completed successfully.
"DeployFailed"	DeployFailed means the deployment to the specified cluster has failed.
"DeployPending"	DeployPending means deployment to the specified cluster is in progress.

ConfigMapTemplate

(Appears on:MultiClusterConfigMapSpec)

ConfigMapTemplate has the metadata and spec of the Kubernetes ConfigMap.

Field	Description
`binaryData` map[string][]byte	Corresponds to the `binaryData` field of the `struct` ConfigMap defined in types.go.
`data` map[string]string	Corresponds to the `data` field of the `struct` ConfigMap defined in types.go.
`immutable` bool	Corresponds to the `immutable` field of the `struct` ConfigMap defined in types.go.
`metadata` EmbeddedObjectMeta	Metadata describing the ConfigMap.

EmbeddedObjectMeta

(Appears on:ApplicationConfigurationTemplate, ComponentTemplate, ConfigMapTemplate, SecretTemplate)

EmbeddedObjectMeta is metadata describing a resource.

Field	Description
`annotations` map[string]string	(Optional) Annotations for the resource.
`labels` map[string]string	(Optional) Labels for the resource.
`name` string	(Optional) Name of the resource.
`namespace` string	(Optional) Namespace of the resource.

MultiClusterApplicationConfigurationSpec

(Appears on:MultiClusterApplicationConfiguration)

MultiClusterApplicationConfigurationSpec defines the desired state of a multicluster application.

Field	Description
`placement` Placement	Clusters in which the application is to be created.
`secrets` []string	(Optional) List of secrets used by the application. These secrets must be created in the application’s namespace before deploying a MultiClusterApplicationConfiguration resource.
`template` ApplicationConfigurationTemplate	Template containing the metadata and spec for an OAM applicationConfiguration resource.

MultiClusterComponentSpec

(Appears on:MultiClusterComponent)

MultiClusterComponentSpec defines the desired state of a MultiCluster Component.

Field	Description
`placement` Placement	Clusters in which the component is to be created.
`template` ComponentTemplate	Template containing the metadata and spec for an OAM component.

MultiClusterConfigMapSpec

(Appears on:MultiClusterConfigMap)

MultiClusterConfigMapSpec defines the desired state of a MultiCluster ConfigMap.

Field	Description
`placement` Placement	Clusters in which the ConfigMap is to be created.
`template` ConfigMapTemplate	The embedded Kubernetes ConfigMap.

MultiClusterResourceStatus

(Appears on:MultiClusterApplicationConfiguration, MultiClusterComponent, MultiClusterConfigMap, MultiClusterSecret, VerrazzanoProject)

MultiClusterResourceStatus is the runtime status of a multicluster resource.

Field Description

Field	Description
`clusters` []ClusterLevelStatus	Status information for each cluster.
`conditions` []Condition	The current state of a multicluster resource.
`state` StateType	The state of the multicluster resource. State values are case-sensitive and formatted as follows: `Failed`: deployment to cluster failed `Pending`: deployment to cluster is in progress `Succeeded`: deployment to cluster successfully completed

clusters
[]ClusterLevelStatus

Status information for each cluster.

conditions
[]Condition

The current state of a multicluster resource.

state
StateType

The state of the multicluster resource. State values are case-sensitive and formatted as follows:

Failed: deployment to cluster failed
Pending: deployment to cluster is in progress
Succeeded: deployment to cluster successfully completed

MultiClusterSecretSpec

(Appears on:MultiClusterSecret)

MultiClusterSecretSpec defines the desired state of a MultiCluster Secret.

Field	Description
`placement` Placement	Clusters in which the secret is to be created.
`template` SecretTemplate	The embedded Kubernetes secret.

NamespaceTemplate

(Appears on:ProjectTemplate)

NamespaceTemplate contains the metadata and specification of a Kubernetes namespace.

Field Description

metadata
Kubernetes meta/v1.ObjectMeta (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
Kubernetes core/v1.NamespaceSpec

The specification of a namespace.

finalizers
[]Kubernetes core/v1.FinalizerName

(Optional)

Finalizers is an opaque list of values that must be empty to permanently remove object from storage. More info: https://kubernetes.io/docs/tasks/administer-cluster/namespaces/

NetworkPolicyTemplate

(Appears on:ProjectTemplate)

NetworkPolicyTemplate contains the metadata and specification of a Kubernetes NetworkPolicy.

NOTE

To add an application NetworkPolicy, see NetworkPolicies for applications.

Field Description

metadata
Kubernetes meta/v1.ObjectMeta (Optional) Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
Kubernetes networking/v1.NetworkPolicySpec

The specification of a network policy.

`podSelector` Kubernetes meta/v1.LabelSelector	Selects the pods to which this NetworkPolicy object applies. The array of ingress rules is applied to any pods selected by this field. Multiple network policies can select the same set of pods. In this case, the ingress rules for each are combined additively. This field is NOT optional and follows standard label selector semantics. An empty podSelector matches all pods in this namespace.
`ingress` []Kubernetes networking/v1.NetworkPolicyIngressRule	(Optional) List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod’s local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default)
`egress` []Kubernetes networking/v1.NetworkPolicyEgressRule	(Optional) List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
`policyTypes` []Kubernetes networking/v1.PolicyType	(Optional) List of rule types that the NetworkPolicy relates to. Valid options are [“Ingress”], [“Egress”], or [“Ingress”, “Egress”]. If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ “Egress” ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include “Egress” (since such a policy would not include an Egress section and would otherwise default to just [ “Ingress” ]). This field is beta-level in 1.8

Placement

(Appears on:MultiClusterApplicationConfigurationSpec, MultiClusterComponentSpec, MultiClusterConfigMapSpec, MultiClusterSecretSpec, VerrazzanoProjectSpec)

Placement contains the name of each cluster where a resource will be located.

Field	Description
`clusters` []Cluster	List of clusters.

ProjectTemplate

(Appears on:VerrazzanoProjectSpec)

ProjectTemplate contains the list of namespaces to create and the optional security configuration for each namespace.

Field	Description
`namespaces` []NamespaceTemplate	The list of application namespaces to create for this project.
`networkPolicies` []NetworkPolicyTemplate	(Optional) Network policies applied to namespaces in the project.
`security` SecuritySpec	(Optional) The project security configuration.

SecretTemplate

(Appears on:MultiClusterSecretSpec)

SecretTemplate has the metadata and spec of the Kubernetes Secret.

Field	Description
`data` map[string][]byte	Corresponds to the data field of the struct Secret defined in types.go.
`metadata` EmbeddedObjectMeta	Metadata describing the secret.
`stringData` map[string]string	Corresponds to the `stringData` field of the `struct` Secret defined in types.go.
`type` Kubernetes core/v1.SecretType	The type of secret.

SecuritySpec

(Appears on:ProjectTemplate)

SecuritySpec defines the security configuration for a Verrazzano Project.

Field	Description
`projectAdminSubjects` []Kubernetes rbac/v1.Subject	(Optional) The subjects to bind to the `verrazzano-project-admin` role.
`projectMonitorSubjects` []Kubernetes rbac/v1.Subject	(Optional) The subjects to bind to the `verrazzano-project-monitoring` role.

StateType (`string` alias)

(Appears on:ClusterLevelStatus, MultiClusterResourceStatus)

StateType identifies the state of a multicluster resource.

Value	Description
"Failed"	Failed is the state when deploy to specified cluster has failed.
"Pending"	Pending is the state when deploy to specified cluster is in progress.
"Succeeded"	Succeeded is the state when deploy to specified cluster is completed.

VerrazzanoProjectSpec

(Appears on:VerrazzanoProject)

VerrazzanoProjectSpec defines the desired state of a Verrazzano Project.

Field	Description
`placement` Placement	Clusters on which the namespaces are to be created.
`template` ProjectTemplate	The project template.

Generated with gen-crd-api-reference-docs on git commit 238db2c3e.

Docs: Multicluster Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano may be installed in a multicluster environment, consisting of an admin cluster and optionally, one or more managed clusters.

The admin cluster is a central point from which Verrazzano applications in managed clusters can be deployed and monitored.
Managed clusters are registered with an admin cluster.
MultiClusterApplicationConfiguration resources are used to target applications to any cluster in a multicluster Verrazzano environment.

The following diagram shows a high-level overview of how multicluster Verrazzano works. For a more detailed view, see the diagram here.

Admin cluster

A Verrazzano admin cluster is a central management point for:

Deploying and undeploying applications to the managed clusters registered with the admin cluster.
Viewing logs and metrics for both Verrazzano Components and applications that reside in the managed clusters.

You may register one or more managed clusters with the admin cluster by creating a VerrazzanoManagedCluster resource in the verrazzano-mc namespace of an admin cluster.

NOTE: The admin cluster has a fully functional Verrazzano installation. You can locate applications on the admin cluster as well as on managed clusters.

Managed clusters

A Verrazzano managed cluster has a minimal footprint of Verrazzano, installed using the managed-cluster installation profile. A managed cluster has the following additional characteristics:

It is registered with an admin cluster with a unique name.
Logs for Verrazzano system Components and Verrazzano multicluster applications are sent to OpenSearch running on the admin cluster, and are viewable from that cluster.
A Verrazzano MultiClusterApplicationConfiguration Kubernetes resource, created on the admin cluster, will be retrieved and deployed to a managed cluster if all of the following are true:
- The MultiClusterApplicationConfiguration is in a namespace governed by a VerrazzanoProject.
- The VerrazzanoProject has a placement value that includes this managed cluster.
- The MultiClusterApplicationConfiguration itself has a placement value that includes this managed cluster.

Verrazzano multicluster applications

Verrazzano includes a MultiClusterApplicationConfiguration resource definition for applications that may be targeted for placement in one or more clusters.

A MultiClusterApplicationConfiguration is a wrapper for an ApplicationConfiguration, and additionally allows the placement of the underlying resource to be specified as a list of names of the clusters in which the ApplicationConfiguration must be placed.
MultiClusterApplicationConfiguration resources, along with their associated Component and Secret resources, are created in the admin cluster, in a namespace that is part of a VerrazzanoProject, and targeted for placement in either the local admin cluster or a remote managed cluster.
A multicluster application is considered part of a VerrazzanoProject if it is in a namespace that is governed by that VerrazzanoProject.

Managed cluster registration

A managed cluster may be registered with an admin cluster using a two-step process:

Step 1: Create a VerrazzanoManagedCluster resource in the verrazzano-mc namespace of the admin cluster.

Step 2: Retrieve the Kubernetes manifest file generated in the VerrazzanoManagedCluster resource and apply it on the managed cluster to complete the registration.

When a managed cluster is registered, the following will happen:

After both steps of the registration are complete, the managed cluster begins polling the admin cluster for VerrazzanoProject resources and MultiClusterApplicationConfiguration resources, which specify a placement in this managed cluster.
- Any VerrazzanoProject resources placed in this managed cluster are retrieved, and the corresponding namespaces and security permissions (RoleBindings) are created in the managed cluster.
- Any MultiClusterApplicationConfigurations that are placed in this managed cluster, and are in a VerrazzanoProject that is also placed in this managed cluster, are retrieved, and created or updated on the managed cluster. The underlying ApplicationConfiguration represented by the MultiClusterApplicationConfiguration is unwrapped, and created or updated on the managed cluster. The managed cluster namespace of the MultiClusterApplicationConfiguration and its underlying ApplicationConfiguration match the admin cluster namespace of the MultiClusterApplicationConfiguration.
- Any Component and Secret resources referenced by the retrieved MultiClusterApplicationConfigurations, are also retrieved and created on the managed cluster.
When the managed cluster connects to the admin cluster, it updates the VerrazzanoManagedCluster resource for this managed cluster with:
- The endpoint URL that the admin cluster should use to scrape Prometheus metrics from the managed cluster.
- The date and time of the most recent successful connection from the managed cluster to the admin cluster.
For MultiClusterApplicationConfigurations retrieved and unwrapped on a managed cluster, the application logs are sent to OpenSearch on the admin cluster, and may be viewed from the Verrazzano-installed OpenSearch Dashboards on the admin cluster. Likewise, application metrics will be scraped by the admin cluster and available from Verrazzano-installed Prometheus on the admin cluster.

Detailed view of multicluster Verrazzano

This diagram shows a detailed view of how multicluster Verrazzano works.

For more information, see the API Documentation for the resources described here.

Deploy multicluster applications

For an example of how to deploy an application in a multicluster environment, see Multicluster ToDo List.

Docs: Prepare to Upgrade Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano installation consists of a stack of components, such as cert-manager, where each component has a specific release version that may be different from the overall Verrazzano version. The Verrazzano platform operator knows the versions of each component associated with the Verrazzano version. When you perform the initial Verrazzano installation, the appropriate version of each component is installed by the platform operator. Post installation, it may be necessary to update one or more of the component images or Helm charts. This update is also handled by the platform operator and is called an upgrade. Currently, Verrazzano does only patch-level upgrades, where a helm upgrade command can be issued for the component. Typically, patch-level upgrades simply replace component images with newer versions.

Application and system pod restarts

If Verrazzano has a new version of Istio, then all the pods with Istio proxy sidecars need to be restarted. This is done so that the new version of the proxy sidecar can be injected into the pods. All Verrazzano system pods containing Istio proxy sidecars will be restarted. Additionally, OAM application workloads with old versions of the Istio proxy sidecar will be restarted as well. Verrazzano will perform a rolling restart of all OAM workloads, except WebLogic workloads with an Istio proxy sidecar that is out of date by two minor versions or more. For these workloads, Verrazzano will stop the WebLogic domain prior to upgrade and restart it after the upgrade has been completed. This may result in downtime during upgrade for these WebLogic workloads. There is no user involvement related to restarting OAM applications; it is done automatically during upgrade.

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following:

A Kubernetes cluster and a compatible kubectl.
Two load balancers. Note that in a Verrazzano multicluster environment, two load balancers are required for each cluster.
dev profile - Each node in the cluster should contain at least two CPUs, 16 GB RAM, and 100 GB of disk storage. The entire cluster requires at least six CPUs, 48 GB RAM, and 100 GB of disk storage. In addition, about 52 GB of storage is required for the persistent volumes.
prod profile - Each node in the cluster should contain at least four CPUs, 32 GB RAM, and 100 GB of disk storage. The entire cluster requires at least eight CPUs, 64 GB RAM, and 150 GB of disk storage. In addition, about 450 GB of storage is required for the persistent volumes.

The storageClassName is not available for all the components where you configure persistent volumes, therefore, when it is not specified, the default StorageClass from the cluster is used.

NOTE

To avoid conflicts with Verrazzano system components, we recommend installing Verrazzano into an empty cluster. Also, depending on the resource requirements of the applications you deploy, the configurations previously suggested may or may not be sufficient.

Supported hardware

Verrazzano requires x86-64; other architectures are not supported.

Supported software versions

Verrazzano supports the following software versions.

Kubernetes

You can install Verrazzano on the following Kubernetes versions.

Verrazzano	Kubernetes Versions
1.6	1.24, 1.25, 1.26
1.5	1.21, 1.22, 1.23, 1.24
1.4	1.21, 1.22, 1.23, 1.24
1.3	1.21, 1.22, 1.23
1.2	1.19, 1.20, 1.21
1.1	1.19, 1.20, 1.21
1.0	1.18, 1.19, 1.20

For more information, see Kubernetes Release Documentation.
For platform specific details, see Verrazzano platform setup.

WebLogic Server

The supported versions of WebLogic Server are dependent on the WebLogic Kubernetes Operator version. See the WebLogic Server versions supported here.

Coherence

The supported versions of Coherence are dependent on the Coherence Operator version. See the Coherence versions supported here.

Helidon

Verrazzano supports all versions of Helidon. For more information, see Helidon and Helidon Commercial Offerings.

Verrazzano installed software

For a detailed list of Verrazzano installed software, see here.

Docs: Understand Application Tracing Components in Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides Jaeger, a distributed tracing system used for monitoring and troubleshooting microservices.

About distributed tracing

Distributed tracing is a method for tracking application behavior as it propagates through distributed systems. You can track application requests as they flow from front end devices to back end services and databases.

Distributed tracing is useful for tracking and observability, in general, however, it can also help you identify the exact line of occurrence of a error in a complex architecture. You can use distributed tracing to troubleshoot requests that exhibit high latency or errors.

With distributed tracing, the application transactions are captured using request and response headers. A trace header gets added from the original request to subsequent requests and thus creating a link through out the entire transaction that can be traced back to the origin. A single trace typically shows the activity for an individual transaction or request within the application being monitored, from the browser or mobile device, down through to the database and back.

In distributed tracing, a single trace contains a series of tagged time intervals called spans. A span can be thought of as a single unit of work. Spans have a start and end time, and optionally may include other metadata, like tags or events. Spans have relationships between one another, including parent-child relationships, which are used to show the specific path a particular transaction takes through the numerous services or components that make up the application.

A trace represents an end-to-end request; it’s made up of a single or multiple spans.
A span represents work done by a single-service with time intervals and associated metadata, such as tags and events:
- Tags let you query your traces to filter the results and help with your collaboration and debugging efforts.
- Events will show up as logs associated with the span that added the event.

About Jaeger tracing

As with most other distributed tracing systems, Jaeger works with spans and traces, as defined in the OpenTracing specification.

In Verrazzano, Jaeger is comprised of the following components:

Collector service, which receives traces from clients, processes them, and stores them in a storage back end.
Query service, which exposes the APIs for retrieving traces from storage and hosts the Jaeger console for searching and analyzing traces.
Storage, may be ephemeral or backed by OpenSearch.
- Jaeger can be configured to use one of several choices for storage back ends.
- In Verrazzano, trace records are stored in OpenSearch; OpenSearch must be enabled in Verrazzano for this to work.

Jaeger is very configurable. See the Jaeger documentation for detailed information about Jaeger features and architecture. In Verrazzano, Jaeger does not include the Ingester service, Kafka, or Sparks jobs.

Next steps

Enable Jaeger and customize your Verrazzano Jaeger installation. See configure tracing.
Configure your applications to send traces to Jaeger.
Then, use the Verrazzano Jaeger console to view traces.

Docs: Understand Monitoring Components in Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano metrics stack automates metrics aggregation and consists of Prometheus, Thanos, and Grafana components. Metrics sources expose system and application metrics. The Prometheus and Thanos components retrieve and store the metrics and Grafana provides dashboards to visualize them.

The following is a representation of metrics collection in a single cluster environment.

Metrics sources

The following sections describe metrics sources that Verrazzano provides for OAM and standard Kubernetes applications.

OAM

Metrics sources produce metrics and expose them to the Kubernetes Prometheus system using annotations in the pods. The metrics annotations may differ slightly depending on the resource type. The following is an example of the WebLogic Prometheus-related configuration specified in the todo-list application pod:

$ kubectl describe pod tododomain-adminserver -n todo-list

Annotations:  prometheus.io/path: /wls-exporter/metrics
              prometheus.io/port: 7001
              prometheus.io/scrape: true

For other resource types, such as Coherence or Helidon, the annotations would look similar to this:

Annotations:  verrazzano.io/metricsEnabled: true
              verrazzano.io/metricsPath: /metrics
              verrazzano.io/metricsPort: 8080

To look directly at the metrics that are being made available by the metric source, map the port and then access the path.

For example, for the previous metric source:

Map the port being used to expose the metrics.
```
$ kubectl port-forward tododomain-adminserver 7001:7001 -n todo-list
```
Get the user name and password used to access the metrics source from the corresponding secret.
```
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.username} | base64 \
    --decode; echo
$ kubectl get secret \
    --namespace todo-list tododomain-weblogic-credentials \
    -o jsonpath={.data.password} | base64 \
    --decode; echo
```
Access the metrics at the exported path, using the user name and password retrieved in the previous step.
```
$ curl -u USERNAME:PASSWORD localhost:7001/wls-exporter/metrics
```

Standard Kubernetes workloads

Verrazzano supports enabling metric sources for Kubernetes workloads deployed without OAM Components. To enable metrics for Kubernetes workloads, you must create a Service Monitor or Pod Monitor, as applicable.

Metrics Traits use Service Monitors which require Services for metrics collection.
For details on Service Monitor and Pod Monitor, refer to the Prometheus Operator documentation.

When creating the Service Monitor or Pod Monitor for your workload, include the label release, with the value prometheus-operator on the monitor resource.

Verify metrics collection

To verify that the metrics are being collected for your workload, follow these steps.

Access the Prometheus console.
From the console, use the navigation bar to access Status/Targets.
On this page, you will see a target name with this formatting: <monitor-type>/<workload-namespace>_<workload-name>_<workload-type>, where monitor-type may be serviceMonitor or podMonitor.
Copy this job name from the target labels for use in future queries.
Verify that the State of this target is UP.
Next, use the navigation bar to access the Graph.
Here, use the job name you copied to construct this expression: {job="<job_name>"}
Use the graph to run this expression and verify that you see application metrics appear.

If you are unable to verify metrics collection, you might need to manually create a Service for the workload.

For more information on Prometheus solutions, see Troubleshooting Prometheus.

Legacy workloads

Standard Kubernetes workloads that were metrics sources in earlier versions of Verrazzano (1.3.x or earlier), will continue to be supported when upgrading to later versions of Verrazzano.

For workloads that used the legacy default metrics template, Verrazzano will create a Service Monitor in the workload’s namespace to ensure that metrics continue to be scraped. You can make any ongoing changes to the metrics source configuration by editing the Service Monitor.

For workloads that used a legacy custom metrics template, Verrazzano will configure the Prometheus Operator to ensure that metrics continue to be scraped.

Metrics server

Verrazzano installs the Prometheus Operator in the verrazzano-monitoring namespace.
A single Prometheus pod is created by Prometheus Operator in the same namespace.
Discovers exposed metrics source endpoints.
Scrapes metrics from metrics sources.
Responsible for exposing all metrics.

Docs: Uninstall Considerations

Mon, 01 Jan 0001 00:00:00 +0000

Before uninstalling Verrazzano, you should delete your Verrazzano applications because they may not function properly after the uninstall operation is done.

When you uninstall Verrazzano:

All of the Verrazzano components are uninstalled.
The CRDs installed by Verrazzano are not deleted.
Any applications that were deployed will still exist, but they may not be functional.

Docs: Use the Verrazzano Console

Mon, 01 Jan 0001 00:00:00 +0000

To register a cluster, complete the following steps:

Enable syncClusters.
For information about syncClusters, see Enable syncClusters.
```
kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: admin
spec:
  profile: prod
  components:
    clusterOperator:
      overrides:
      - values:
          syncClusters:
            enabled: true
EOF
```
On the admin cluster, open the Rancher console.
You can find the Rancher console URL for your cluster by following the instructions here.
Select Cluster Management, and then click Import Existing Cluster.
Provide a name for your managed cluster, for example, managed1.
Optional. In your Verrazzano configuration, if you specified a cluster label selector, then under Labels & Annotations provide the label and value for the cluster. The label and value information must match the cluster selection matchExpression in your Verrazzano configuration for selective cluster registration to occur.
Click Create.
On the next screen, follow the on-screen instructions to complete the registration by running the provided command against the managed cluster.

After the cluster reaches the Active state in the console, synchronization with Verrazzano will happen automatically and a VerrazzanoManagedCluster resource will be created in the verrazzano-mc namespace.

Run the following command to view the details and status of Verrazzano’s multicluster initialization operations:

$ kubectl get vmc -n verrazzano-mc <Rancher_cluster_name> -o yaml

For more information, see Registering Existing Clusters in the Rancher documentation.

NOTE: You can also register managed clusters using kubectl, see Register Managed Clusters using kubectl.

Next steps

Verify your multicluster Verrazzano environment set up by following the instructions at Verify Multicluster Installation.
Deploy multicluster example applications. See Examples of using Verrazzano in a multicluster environment.

NOTE

To deregister a managed cluster, see Deregister a Managed Cluster.

Docs: Verify Using CLI

Mon, 01 Jan 0001 00:00:00 +0000

To verify the Verrazzano installation, you can use the vz status command to determine the status of your installation. After a successful installation, Verrazzano should be in the Ready state.

$ vz status

# Sample output for a dev profile install
Verrazzano Status
  Name: example-verrazzano
  Namespace: default
  Profile: prod
  Version: v1.5.1
  State: Ready
  Available Components: 23/23
  Access Endpoints:
    consoleUrl: https://verrazzano.default.10.0.0.1.nip.io
    grafanaUrl: https://grafana.vmi.system.default.10.0.0.1.nip.io
    keyCloakUrl: https://keycloak.default.10.0.0.1.nip.io
    kialiUrl: https://kiali.vmi.system.default.10.0.0.1.nip.io
    openSearchDashboardsUrl: https://osd.vmi.system.default.10.0.0.1.nip.io
    openSearchUrl: https://opensearch.vmi.system.default.10.0.0.1.nip.io
    prometheusUrl: https://prometheus.vmi.system.default.10.0.0.1.nip.io
    rancherUrl: https://rancher.default.10.0.0.1.nip.io

For installation troubleshooting help, see the Analysis Advice.

After the installation has completed, you can use the Verrazzano consoles. For information on how to get the consoles URLs and credentials, see Access Verrazzano.

Next steps

(Optional) Run the example applications located here.

Docs: About Verrazzano Traits

Mon, 01 Jan 0001 00:00:00 +0000

Traits customize Component workloads and generate related resources during deployment. Verrazzano provides several Traits, for example IngressTrait and MetricsTrait. The platform extracts Traits contained within an ApplicationConfiguration during deployment. This processing is similar to the extraction of workload content from Component resources. Note that for some Kubernetes resources, the oam-kubernetes-runtime operator may need to be granted create permission.

A Kubernetes operator, for example verrazzano-application-operator, processes these extracted Traits and may create additional related resources or may alter related workloads. Each Trait implementation will behave differently.

The following sample shows an IngressTrait applied to a referenced Component.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
...
spec:
  components:
    - componentName: example-component
      traits:
        - trait:
            apiVersion: oam.verrazzano.io/v1alpha1
            kind: IngressTrait
            spec:
              rules:
                - paths:
                    - path: "/greet"

Each Trait type optionally can have an associated TraitDefinition. This provides the platform with additional information about the Trait’s schema and workloads to which the Trait can be applied. A TraitDefintion is typically provided by the platform, not an end user.

The Verrazzano platform provides several Trait definitions and implementations:

Docs: About Verrazzano Workloads

Mon, 01 Jan 0001 00:00:00 +0000

Components contain an embedded workload. Verrazzano and the OAM specification provide several workloads, for example VerrazzanoWebLogicWorkload and ContainerizedWorkload. Workloads can also be any Kubernetes resource.

The following sample shows a VerrazzanoHelidonWorkload workload embedded within a Component.

apiVersion: core.oam.dev/v1alpha2
kind: Component
...
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoHelidonWorkload
    spec:
      deploymentTemplate:
        podSpec:
          containers:
            - name: example-container
              image: ...
              ...

A workload can optionally have an associated WorkloadDefinition. This provides the platform with information about the schema of the workload. A WorkloadDefintion is typically provided by the platform, not an end user.

Scopes

Scopes customize Component workloads and generate related resources during deployment. An ApplicationConfiguration contains Scope references instead of the Scope’s content being embedded. The platform will update the Scopes with a reference to each applied Component. This update triggers the related operator to process the Scope.

The following sample shows a reference to a HealthScope named example-health-scope.

apiVersion: core.oam.dev/v1alpha2
kind: ApplicationConfiguration
...
spec:
  components:
    - componentName: example-component
      scopes:
        - scopeRef:
            apiVersion: core.oam.dev/v1alpha2
            kind: HealthScope
            name: example-health-scope
        ...

The following sample shows the configuration details of the referenced HealthScope.

apiVersion: core.oam.dev/v1alpha2
kind: HealthScope
metadata:
  name: example-health-scope
spec:
  probe-method: GET
  probe-endpoint: /health

Each Scope type can optionally have an associated ScopeDefinition. This provides the platform with additional information about processing the Scope:

The Scope’s schema
The workload types to which the Scope can be applied
The field within the Scope used to record related Component references

A ScopeDefintion is typically provided by the platform, not an end user.

The Verrazzano platform provides several workload definitions and implementations:

The VerrazzanoWebLogicWorkload is used for WebLogic workloads. See WebLogic Workload.
The VerrazzanoCoherenceWorkload is used for Coherence workloads. See Coherence Workload.
The VerrazzanoHelidonWorkload is used for Helidon workloads. See Helidon Workload.

OAM ContainerizedWorkload

The ContainerizedWorkload should be used for long-running container workloads which are not covered by the workload types described previously. This workload type is similar to the Deployment workload. It is provided to ensure that OAM can be used for non-Kubernetes deployment environments. See the OAM specification.

Docs: Verrazzano Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Each Verrazzano installation sits on top of a Kubernetes cluster (that can be hosted on a variety of environments). Verrazzano relies on the Kubernetes API to pull information from the cluster to its components and push instructions from its components down to the cluster. Specifically, Verrazzano uses custom resources, extensions of the Kubernetes API to configure Kubernetes environments and provide additional functionality.

Because Verrazzano extends Kubernetes, you can continue to use many of the Kubernetes management tools you’re already familiar with, such as kubectl and kubeadm.

Each Kubernetes cluster must have both a Container Network Interface (CNI) and a Container Storage Interface (CSI) plugin.

Inside a Verrazzano installation

Verrazzano consists of many, discrete components. To coordinate between these components, Verrazzano engages the Istio service mesh - an infrastructure layer that manages the flow of information between services. Most Verrazzano components operate from within the Istio service mesh and rely on it to facilitate fast, reliable, and secure communication between them. Istio also provides useful features such as load balancing, traffic control, observability, and more.

Learn more about how Istio handles traffic and network security in Verrazzano at Networking.

The components within the service mesh can broadly be organized into the following categories:

Backup and restore
Certificate management
Cluster management
GitOps
Identity management
Observability
Verrazzano operators

See Installed Software for a list of the components included with Verrazzano. Depending on which installation profile you choose to apply, some components are not enabled by default.

One Verrazzano component that does not reside within the service mesh is the Verrazzano platform operator, a custom Kubernetes operator that handles the management of Verrazzano itself - including installation (and uninstallation), upgrades, and troubleshooting. The platform operator also works with the service mesh to synchronize actions between the components and Kubernetes clusters.

Application deployment

With Verrazzano, you can manage a variety of application types. Depending on their type, applications are deployed from different spaces within the architecture of Verrazzano.

Applications that conform to the Open Application Model (OAM) specification are deployed within the service mesh. Their close proximity to other components allows for significant automation and configuration benefits.

Other application types are deployed in the Kubernetes cluster. They can still use Verrazzano features but may require additional configuration. Whichever application types you use, they’ll benefit from Verrazzano’s comprehensive management solution.

Multicluster support

You can use Verrazzano in single and multicluster Kubernetes environments. In a multicluster environment, there is an admin cluster, which is the central management point for deploying and monitoring applications, and one or more managed clusters, which look to the admin cluster for configurations.

Verrazzano is installed on every cluster; admin clusters are installed with either a dev or prod profile, while managed clusters use a managed-cluster profile which has fewer components enabled by default and requires registration to an admin cluster. See Installation Profiles for details.

After registration, a managed cluster submits logging and metrics data to its admin cluster, and receives configuration instructions in turn, over HTTPS.

Docs: Argo CD

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides an Argo CD deployment that automates the deployment of the desired application states in the specified target environments. There are scenarios where you may want to back up your Argo CD data and restore it.

Verrazzano uses Velero to facilitate backing up and restoring Argo CD data.

Velero operator prerequisites

Before proceeding with an Argo CD backup or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore Argo CD, you must first enable Velero.

The following configuration shows you how to enable velero with a prod installation profile.
```
  $ kubectl apply -f -<<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: prod
      components:
        velero:
          enabled: true
EOF
```
After Verrazzano installation is completed, verify that the Velero pods are running in the verrazzano-backup namespace.
```
$ kubectl get pod -n verrazzano-backup
```
```
# Sample of pods running after enabling the velero component
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h
```
Velero requires a secret to communicate with the S3 compatible object store, so we create a backup-secret.txt file, which has the object store credentials.
```
[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>
```
In the namespace verrazzano-backup, create a Kubernetes secret, for example verrazzano-backup-creds.
```
$ kubectl create secret generic -n verrazzano-backup <secret-name> --from-file=<key>=<full_path_to_creds_file>
```
The following is an example:
```
$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt
```
NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.
Create a BackupStorageLocation resource, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example, which uses Oracle Cloud Object Storage. For more information, see Backup Storage Location in the Velero documentation.
```
 $ kubectl apply -f -<<EOF
   apiVersion: velero.io/v1
   kind: BackupStorageLocation
   metadata:
     name: verrazzano-backup-location
     namespace: verrazzano-backup
   spec:
     provider: aws
     objectStorage:
       bucket: example-verrazzano
       prefix: backup-demo
     credential:
       name: verrazzano-backup-creds
       key: cloud
     config:
       region: us-phoenix-1
       s3ForcePathStyle: "true"
       s3Url: https://<myobjstorenamespace>.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```

Argo CD backup using Velero

If you created applications with resources running in different namespaces, other than argocd, then based on the following criteria you can back up and restore Argo CD:

If applications running in different namespaces use persistent volumes, then you can back up the namespace where the applications are running with the PV.
If applications running in different namespaces do not use persistent storage, then:
- Take a backup of all the namespaces where the application is running by specifying the namespaces as a list.
- Take a backup of only the argocd namespace, create all the namespaces of different applications, and then restore from the backup.

The following example shows a sample Velero Backup API resource that you can create to initiate an Argo CD backup.

  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: verrazzano-argocd-backup
    namespace: verrazzano-backup
  spec:
    includedNamespaces:
    - argocd
    defaultVolumesToRestic: false
    storageLocation:  verrazzano-backup-location
EOF

The preceding example backs up the Argo CD components:

The defaultVolumesToRestic is set to false so that Velero ignores the associated PVCs.
If the deployed applications refer to a database or persistent volumes, then you need to manually create a backup.

Argo CD backup status

# To display the status of the backup, run the following command
$ kubectl get backup.velero.io -n verrazzano-backup verrazzano-argocd-backup -o yaml

# Sample output showing just the status portion of the backup
status:
  completionTimestamp: ...
  expiration: ...
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 232
    totalItems: 232
  startTimestamp: ...
  version: 1

Argo CD scheduled backups

Velero supports a Schedule API that is a repeatable request that is sent to the Velero server to perform a backup for a given cron notation. After the Schedule object is created, the Velero server will start the backup process. Then, it will wait for the next valid point in the given cron expression and run the backup process on a repeating basis.

Argo CD restore using Velero

To initiate an Argo CD restore operation, first delete the existing Argo CD running on the system and all related data. Make sure that the data you are deleting is not needed. The restore operation will only restore data from the specified backup, and any additional data since that backup will be destroyed by the deletion.

If you restore Argo CD to a different cluster, create a Velero BackupStorageLocation resource in the new cluster that points to the same backup storage location configured in the original cluster. This ensures that the Velero resources created by the original cluster’s backup are automatically synced to the new cluster. Once the sync completes, you will be able to access the backup from the original cluster on the new cluster. It is recommended to configure the BackupStorageLocation on the new cluster as read-only by setting accessMode to ReadOnly in the BackupStorageLocation spec. This ensures that the backup in the object store is not modified from the new cluster. For more information, see Backup Storage Location in the Velero documentation.

Delete the Argo CD components and the namespace.
```
# These are sample commands to demonstrate the Argo CD restore process
$ kubectl delete sts -l app.kubernetes.io/part-of=argocd -n argocd
$ kubectl delete deploy -l app.kubernetes.io/part-of=argocd -n argocd
$ kubectl delete pvc -n argocd <pvc name>
$ kubectl delete ns argocd
```
To perform an Argo CD restore operation, you can invoke the following example Velero Restore API object.

NOTE: For Argo CD, includedNamespaces should list all the namespaces across which the applications are deployed.
```
 $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: verrazzano-argocd-restore
    namespace: verrazzano-backup
  spec:
    backupName: verrazzano-argocd-backup
    includedNamespaces:
    - argocd
EOF
```
The preceding example will restore an Argo CD cluster from an existing backup.
Wait for all the Argo CD pods to be in the RUNNING state.
```
  $ kubectl wait -n argocd --for=condition=ready pod -l app.kubernetes.io/instance=argocd
```
Sample output:
```
    pod/argocd-application-controller-0 condition met
    pod/argocd-applicationset-controller-8489bfbb8-4f686 condition met
    pod/argocd-notifications-controller-c4f5c9684-8qzl8 condition met
    pod/argocd-redis-548968fdd9-4jcrf condition met
    pod/argocd-repo-server-5889c8cc68-5n8j6 condition met
    pod/argocd-server-67b6994987-j9z99 condition met
```

Argo CD restore status

# To display the status of the restore, run the following command
$ kubectl get restore -n verrazzano-backup verrazzano-argocd-restore -o yaml

# Sample output showing just the status portion of the restore
status:
  completionTimestamp: ...
  phase: Completed
  progress:
    itemsRestored: 75
    totalItems: 75
  startTimestamp: ...

Docs: CLI Setup

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano command-line tool, vz, is available for Linux and Mac systems. Download the binary you want from the Releases page.

NOTE

Installing the command-line tool, vz, is optional.
For optimal functionality, install or upgrade the CLI version to match the desired Verrazzano version.

Install the vz CLI

These instructions demonstrate installing the CLI on Linux AMD64 machines.

Download the latest release

 $ curl -LO https://github.com/verrazzano/verrazzano/releases/download/master/verrazzano-1.6.0-linux-amd64.tar.gz

Validate the binary (optional)

Download the vz checksum file.

 $ curl -LO https://github.com/verrazzano/verrazzano/releases/download/master/verrazzano-1.6.0-linux-amd64.tar.gz.sha256

Validate the vz binary against the checksum file.

$ sha256sum -c verrazzano-2.0.0-linux-amd64.tar.gz.sha256

Unpack and copy the vz binary

 $ tar xvf verrazzano-2.0.0-linux-amd64.tar.gz

The following command needs to be run as root.

 $ sudo cp verrazzano-2.0.0/bin/vz /usr/local/bin

Test to ensure that the version you installed is up-to-date

 $ vz version

The resulting output should be similar to the following.

Version: master
BuildDate: 2023-02-12T21:07:26Z
GitCommit: cb0778bbf7a2cd90e1ae8458abd242f9da27a100

Use the vz CLI

Verrazzano provides a command-line tool for managing a Verrazzano environment using the Verrazzano and Kubernetes API.

Common use cases include installing, upgrading, and uninstalling Verrazzano, as well as analyzing failures in a running Verrazzano environment.

Usage information

Use the following syntax to run vz commands from your terminal window.

vz [command] [flags]

Available commands

Command	Definition
`analyze`	Analyze cluster
`bug-report`	Collect information from the cluster to report an issue
`completion`	Generate the autocompletion script for the specified shell
`help`	Help about any command
`install`	Install Verrazzano
`status`	Status of the Verrazzano installation and access endpoints
`uninstall`	Uninstall Verrazzano
`upgrade`	Upgrade Verrazzano
`version`	Verrazzano version information

Available Flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`-h`, `--help`	Help for `vz`.
`--kubeconfig string`	Path to the kubeconfig file to use.

Docs: Customize Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain.
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self-signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.
Save your signing key pair as a Kubernetes secret. The secret must be created in the namespace corresponding to the clusterResourceNamespace used by cert-manager (cert-manager by default).
For more information, see the cert-manager documentation.
```
$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key
```
Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.clusterIssuer.ca in the Verrazzano custom resource:
- spec.components.clusterIssuer.ca.secretName
- (If needed) spec.components.clusterIssuer.clusterResourceNamespace
For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    clusterIssuer:
      clusterResourceNamespace: mynamespace
      ca:
        secretName: myca
```
In this example, mynamespace will be configured as the clusterResourceNamespace for the Verrazzano cert-manager instance when it is installed.

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.clusterIssuer.letsEncrypt field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure the Verrazzano Let’s Encrypt issuer with the following values in the Verrazzano custom resource:

Set the spec.components.clusterIssuer.letsEncrypt.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.clusterIssuer.letsEncrypt.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    clusterIssuer:
      letsEncrypt:
        emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    clusterIssuer:
      letsEncrypt:
        emailAddress: jane.doe@mycompany.com
        environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Use your own cert-manager

You can either use the Verrazzano-provided cert-manager or use your own. To use your own cert-manager, disable the certManager component in the Verrazzano CR as shown:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-cm-example
spec:
  profile: dev
  components:
    certManager:
      enabled: false

The Verrazzano certificates issuer is configured using the spec.components.clusterIssuer component. The Verrazzano default clusterIssuer assumes that cert-manager is installed in the cert-manager namespace, and that the clusterResourceNamespace for cert-manager is cert-manager.

If your cert-manager is installed into a different namespace or uses a separate namespace for the clusterResourceNamespace, you can configure this on the Verrazzano clusterIssuer component.

In the following example, the clusterIssuer is configured to use the namespace my-cm-resources as the clusterResourceNamespace:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-cmres-example
spec:
  profile: dev
  components:
    certManager:
      enabled: false
    clusterIssuer:
      clusterResourceNamespace: my-cm-resources

Use Let’s Encrypt with OCI DNS with your own cert-manager

Verrazzano uses a webhook component to support Let’s Encrypt certificates with OCI DNS. This webhook implements the cert-manager solver webhook pattern to support solving DNS01 challenges for OCI DNS.

In most circumstances, you will not need to configure this because it will automatically deploy when OCI DNS and Let’s Encrypt certificates are in use. However, if you are using your own cert-manager installed in a namespace other than cert-manager and/or the clusterResourceNamespace is something other than cert-manager, you will need to configure this for the webhook.

The following example configures the webhook to use the namespace my-cm for the cert-manager installation namespace and my-cm-resources for the clusterResourceNamespace:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-cm-letsencrypt-example
spec:
  profile: dev
  components:
    certManager:
      enabled: false
    clusterIssuer:
      clusterResourceNamespace: my-cm-resources
      letsEncrypt:
        emailAddress: jane.doe@mycompany.com
        environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com
    certManagerWebhookOCI:
      overrides:
      - values:
          certManager:
            namespace: my-cm
            clusterResourceNamespace: my-cm-resources

Docs: Customize DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Wildcard DNS Services

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a host name with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    dns:
      wildcard:
        domain: sslip.io

Oracle Cloud Infrastructure DNS

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.
You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI.
```
  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29
```
If you haven’t already uploaded your API signing public key through the console, follow the instructions in this section, How to upload the public key.

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principals require some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI configuration file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script.

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/master/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options.

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev].

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals, all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal.

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS.

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/master/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

Custom DNS

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for host names in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: Default User Accounts

Mon, 01 Jan 0001 00:00:00 +0000

During installation, Verrazzano generates several default accounts.

System	Account	Secret	Secret Namespace	Description
Keycloak	keycloakadmin	`keycloak-http`	`keycloak`	Keycloak root user: full administrative privileges for Keycloak.
Keycloak	verrazzano	`verrazzano`	`verrazzano-system`	Verrazzano root user: can manage the verrazzano-system realm in Keycloak, including managing users in that realm. This user is a member of the verrazzano-admins group, and, if default role bindings are used, has the verrazzano-admin role.
Rancher	admin	`rancher-admin-secret`	`cattle-system`	Rancher root user: full administrative privileges for Rancher.

Docs: Verrazzano Features

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano Enterprise Container Platform delivers a unified platform for standardizing application and cluster management across various hosting environments. It’s built from many familiar open source projects, each working in concert on Kubernetes clusters to provide vital container management services.

Getting started with Verrazzano is easy - the basic installation process is simple yet flexible enough to evolve with your requirements.

Unified hosting environment for all application types

The flexibility of Verrazzano ensures that you can deploy both cloud native and traditional applications in on-premises and multicloud environments and manage them all using the same components. Using a consistent set of tools and processes reduces overhead related to training and maintenance, leaving more development resources available to address your business needs.

Verrazzano can even accelerate your approach to modernizing your traditional applications. While cloud native technologies offer many advantages, the labor requirements necessary to convert your existing applications can be intimidating. Verrazzano presents a bridge between the two: developers can immediately start building new applications cloud natively, while simultaneously maintaining and converting their existing traditional applications on a timeline that makes sense, all within the same platform.

Complete application life cycle management

Starting with deployment, Verrazzano supports an application through all phases of its life cycle. It extensively automates the deployment process so you can reproduce successful application deployments reliably, even in multicloud environments.

Verrazzano understands that application management is a continuous and dynamic process where deployment is just the beginning. As an application moves through its life cycle, its management system must be able to adapt to its developing needs. Day 2 operations are given equal consideration so that post-deployment, you can continue to optimize your applications and their clusters over the course of their lifetime.

With management occurring within a single platform, Verrazzano makes it easy to follow the principles of DevOps and GitOps. Coordination between diverse types of users becomes effortless, encouraging quick resolutions and creative innovations.

For a list of the software that make up Verrazzano and what they do, see Installed Software.

Dependable infrastructure management

Verrazzano also recognizes that the direct management of applications is only part of what makes up a useful container management platform: to consistently provide a stable and reliable service, you need to start with a healthy foundation. The infrastructure supporting your applications must be equally well supported.

Verrazzano tracks and receives valuable and timely information on the state of your clusters to ensure they’re operating optimally.

Verrazzano also supports both single and multicluster Kubernertes environments.

For more information on how Verrazzano works in a multicluster environment, see Multicluster.

Comprehensive observability

Verrazzano includes a variety of logging, monitoring, and tracing components so you can make intelligent decisions based on the comprehensive data generated by your application and its environment. Many of the components in this observability stack offer significant automated functionality which means Verrazzano alerts you to problems before you know to look for them.

For more information on the available observability options, see Observability.

Integrated security

Verrazzano protects your applications and other resources with a robust set of security-focused components. It takes advantage of Kubernetes Role-Based Access Control to authorize only the right users to the right resources. Verrazzano also incorporates an identity solution that’s easily integrated with OAuth and LDAP systems so you can streamline user management and implement Single Sign On services, protecting Verrazzano components and your applications.

Furthermore, a single and cohesive platform can reduce potential security gaps that often arise when working across unrelated software components. Verrazzano seals potential breaches between its individual components to minimize any compatibility issues and ensure each component remains on a supported and secure version.

For more information on how Verrazzano protects your applications, see Security.

Docs: Use Fluent Operator and Fluent Bit

Mon, 01 Jan 0001 00:00:00 +0000

Beginning in v1.6.0, Fluent Operator is included in the logging stack and when enabled, the Fluent Operator configures and manages Fluent Bit, a logging agent that runs as a DaemonSet.

Fluent Bit

Fluent Bit is a logging agent that collects, processes, and sends logs from Kubernetes clusters to log stores.

Using Fluent Operator, Verrazzano deploys the Fluent Bit DaemonSet, which runs one Fluent Bit replica per node in the verrazzano-system namespace. Each instance reads logs from the node’s /var/log/containers directory and writes them to a target OpenSearch data store.

In Fluent Bit, the four fundamental types of configurations are:

Input: to collect data from a source.
Filter: to process data that was collected.
Output: to send collected and processed logs to a data store.
Parser: to parse data in a specific format. Inputs and Filters make use of parser configurations.

Fluent Operator

Verrazzano includes Fluent Operator as an optional component. When enabled, the operator is installed in the cluster in the verrazzano-system namespace and creates the Fluent Bit DaemonSet in the same namespace, using the required custom resources. For a list of custom resources that the operator supports to configure Fluent Bit, see Fluent Bit resources. All the CRDs with the prefix Cluster are cluster-wide configurations that you can use to configure all the cluster logs.

Like cluster-wide resources, the operator comes with namespaced resources, which when created will process logs from the namespace in which these resources exist. The namespaced and cluster-wide configurations will run in conjunction and complement each other. Creating a namespaced resource doesn’t override an existing cluster-wide resource.

Enable logging with Fluent Operator

The Verrazzano resource defines two components to configure logging using Fluent Operator:

fluentOperator: When enabled, installs Fluent Operator and configures a Fluent Bit instance running as a DaemonSet in the cluster. The fluentOperator component creates ClusterInput custom resources of type tail and systemd, and a set of ClusterFilters to enrich the collected logs with Kubernetes metadata.
fluentbitOpensearchOutput: When enabled, creates two ClusterOutput resources to send logs from the cluster to OpenSearch. The two ClusterOutput resources are:
- opensearch-system-clusteroutput: A central output sink to send logs coming from namespaces where the Verrazzano components reside to the verrazzano-system data stream in OpenSearch.
- opensearch-application-clusteroutput: Sends logs coming from namespaces that are not system to the verrazzano-application-<namespace_name> data stream.

By default, these two components are disabled and must be enabled in the Verrazzano custom resource. The following is an example of a Verrazzano resource manifest file with the two components enabled.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    fluentOperator:
      enabled: true
    fluentbitOpensearchOutput:
      enabled: true

If Verrazzano is installed and running, then to enable these two components, see Modify Verrazzano Installations.

NOTE: To collect, process, and send logs to OpenSearch, both components must be enabled.

To get the system and application ClusterOutput resources, run the following command:

$ kubectl get cfbo

You will see an opensearch-system-clusteroutput and an opensearch-application-clusteroutput.

Uninstall Fluentd

Fluentd is the default logging agent, which runs as a DaemonSet that collects, processes, and sends logs to log stores. When Verrazzano is installed, Fluentd is installed by default. With the inclusion of Fluent Bit via the Fluent Operator, you now have the option to run either of these components. These two, Fluentd and Fluent Bit, can co-exist, but if your log store is Verrazzano OpenSearch, then you should uninstall Fluentd because both components will send the same logs to Verrazzano OpenSearch resulting in duplicate logs.

NOTE: Fluent Bit does not support sending logs to OCI-Logging. If your log store is OCI-Logging, then continue using Fluentd.

To uninstall Fluentd, use the following Verrazzano resource manifest file.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    fluentOperator:
      enabled: true
    fluentbitOpensearchOutput:
      enabled: true
    fluentd:
      enabled: false

Configure custom cluster-wide resources

If you prefer to create a cluster-wide resource, like a ClusterFilter, to filter all the logs from the cluster in a specific manner, or a ClusterOutput, such that most of the logs get stored in a custom log store, then add the following label under metadata.labels of the custom resource manifest file.

fluentbit.fluent.io/enabled: "true"

The following is an example of ClusterOutput that sends logs from a cluster to an OpenSearch index.

$ kubectl apply -f - <<EOF
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  labels:
    fluentbit.fluent.io/enabled: "true"
  name: example-clusteroutput
spec:
  match: *
  opensearch:
    host: <host-url>
    httpPassword:
      valueFrom:
        secretKeyRef:
          key: <password-key>
          name: <your-secret-name>
    httpUser:
      valueFrom:
        secretKeyRef:
          key: <username-key>
          name: <your-secret-name>
    index: <index-name>
    port: <port-number>
    suppressTypeName: true
EOF

Replace <host-url>, <port-number>, and <index-name> with the appropriate values.
Replace <your-secret-name>, <password-key>, and <username-key> with the appropriate values.

For any ClusterOutput, create the secret containing the user credentials in the verrazzano-system namespace.

Configure custom namespaced resources

To process logs for an application namespace, you must configure namespaced resources.

FluentBitConfig

The Fluent Operator supports configurability at the namespace level that lets you create Fluent Bit configurations for logs from your application namespace.

The FluentBitConfig custom resource is a namespaced resource, which is used by the Fluent Operator to select resources, like Filters, Outputs, and Parsers using label selectors. These resources will be checked in the same namespace as FluentBitConfig. Using label selectors, FluentBitConfig can also select ClusterParser resources.

If you use the namespace level configurability feature of the operator, then you must create a minimum of one FluentBitConfig resource in your namespace. The FluentBitConfig resource should contain the following label under metadata.labels.

fluentbit.verrazzano.io/namespace-config: "verrazzano"

The following is an example of a FluentBitConfig resource to select all Filters, Outputs, and Parsers with the label, label: "foo".

$ kubectl apply -f - <<EOF
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  labels:
    fluentbit.verrazzano.io/namespace-config: "verrazzano"
  name: example-fluentbitconfig
  namespace: <application-namespace>
spec:
  filterSelector:
    matchLabels:
      foo: "bar"
  parserSelector:
    matchLabels:
      foo: "bar"
  outputSelector:
    matchLabels:
      foo: "bar"
EOF

You can set the labels under spec.filterSelector, spec.parserSelector, and spec.outputSelector to any valid label; you just need to create the corresponding Filter, Output and Parser custom resources with that label.

Custom filtering and parsing

The following is an example of a Filter and a Parser to parse logs from the myapp application, running in the my-app namespace, and a FluentBitConfig resource to locate these Filter and Parser resources in the same namespace, where logs emitted have the following format:

2023-05-29T09:53:35.959135345Z stdout F 2023-05-29 09:53:35 +0000 [warn]: #0 got incomplete line before first line from /logs/myapp-0.log: "(thread=DefaultHttpServerThread-1, member=3, up=342.673): Health: checking safe\n"

$ kubectl apply -f - <<EOF
-----
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  labels:
    fluentbit.verrazzano.io/namespace-config: "verrazzano"
  name: myapp-fluentbitconfig
  namespace: my-app
spec:
  filterSelector:
    matchLabels:
      app: "myapp"
  parserSelector:
    matchLabels:
      app: "myapp"
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: Filter
metadata:
  labels:
    app: "myapp"
  name: myapp-filter
  namespace: my-app
spec:
  filters:
  - parser:
      keyName: log
      reserveData: true
      parser: myapp-parser
  match: kube.*
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: Parser
metadata:
  labels:       
    app: "myapp"
  name: myapp-parser
  namespace: my-app
spec:
  regex:
    regex: '/^(?<logtime>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{9}Z) (?<level>[^\s]+)[^:]+: (?<message>.*)$/'
    timeFormat: '%Y-%m-%dT%H:%M:%S.%LZ'
    timeKey: logtime
EOF

This configuration consists of three resources: FluentBitConfig, Filter, and Parser.

FluentBitConfig:

Name: myapp-fluentbitconfig
Namespace: my-app
For accurate filtering and parsing, it uses a filterSelector and parserSelector to match labels with the value myapp.

Filter:

Name: myapp-filter
Namespace: my-app
This Filter is associated with the myapp application.
It applies a Parser, called myapp-parser, to the logs and the Filter is configured to match logs from the kube source.

Parser:

Name: myapp-parser
Namespace: my-app
This Parser defines the regular expression and formatting details for parsing logs. It extracts three fields from the log lines: logtime, level, and message.

After applying this configuration, you will observe that the logs emitted by the myapp application are now parsed according to the regex and enriched with the following fields:

level: stderr
logtime: 2023-05-29T09:53:35.959135345Z
message: Health: checking safe

You can adjust the provided regular expression (regex) in the Parser resource according to the specific format of your logs. Modify it to match the structure and patterns of your application’s log lines.

Custom output

If you are running your own log store, then you can create an Output resource in your application namespace, so that only the logs from your application namespace go to this log store.

The following is an example of an Output resource for the myapp application, running in the my-app namespace, to send logs to a custom OpenSearch cluster, protected by basic authentication.

---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  labels:
    fluentbit.verrazzano.io/namespace-config: "verrazzano"
  name: myapp-fluentbitconfig
  namespace: my-app
spec:
  outputSelector:
    matchLabels:
      app: "myapp"
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: Output
metadata:
  labels:
    app: "myapp"
  name: myapp-output
  namespace: my-app
spec:
  match: kube.*
  opensearch:
    host: <host-url>
    httpPassword:
      valueFrom:
        secretKeyRef:
          key: <password-key>
          name: <your-secret-name>
    httpUser:
      valueFrom:
        secretKeyRef:
          key: <username-key>
          name: <your-secret-name>
    index: <index-name>
    port: <port-number>
    suppressTypeName: true
EOF

Replace <host-url>, <port-number>, and <index-name> with the appropriate values. The secret holding the credentials for the OpenSearch cluster needs to be created in the same namespace as the Output, that is, the application namespace.
Replace <your-secret-name>, <password-key>, and <username-key> with the appropriate values.

For any namespaced Output resources, the secret containing the credentials for the OpenSearch cluster needs to be created in the same namespace as the Output, that is, the application namespace.

Note that with this configuration, your applications logs still will continue to go to the default cluster output as well.

Disable application log collection in Verrazzano OpenSearch

By default, the application ClusterOutput created by Verrazzano ensures that the logs for your applications are sent to the system OpenSearch. If you have configured an Output in your application namespace, your application logs will now be stored in two locations: the system OpenSearch, by using the application ClusterOutput, and the custom destination, by using the Output resource you created in your application namespace.

If you want to opt out of the Verrazzano OpenSearch log collection, then disable the application ClusterOutput by editing your Verrazzano custom resource as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    fluentOperator:
      enabled: true
    fluentbitOpensearchOutput:
      enabled: true
      overrides:
        - values:
            application:
              enabled: false

After updating your Verrazzano custom resource, you will notice that the opensearch-application-clusteroutput ClusterOutput resource will be removed from the cluster. The opensearch-system-clusteroutput will continue to exist and will send the Verrazzano component logs to Verrazzano OpenSearch.

Configure the systemd logs directory

By default, the systemd journal logs directory is set to /var/run/journal. However, depending on the environment, the directory location may vary.

To override the default configuration and set the logs directory to a different path, edit your Verrazzano custom resource as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    fluentOperator:
      enabled: true
      overrides:
        - values:
            fluentbit:
              input:
                systemd:
                  path: <new-path>
    fluentbitOpensearchOutput:
      enabled: true

Check Fluent Bit configurations

View the generated Fluent Bit configuration that the Fluent Operator loads in a secret and mounts as a volume in a Fluent Bit DaemonSet, as follows:

$ kubectl -n verrazzano-system get secrets Fluent Bit-config -ojson | jq '.data."fluent-bit.conf"' | awk -F '"' '{printf $2}' | base64 --decode
$ kubectl -n verrazzano-system get secrets Fluent Bit-config -ojson | jq '.data."parsers.conf"' | awk -F '"' '{printf $2}' | base64 --decode

Docs: Gather Metrics and Collect Logs

Mon, 01 Jan 0001 00:00:00 +0000

Use Prometheus to collect system performance metrics and metrics for applications deployed or managed by Verrazzano. For information, see Prometheus.

Gather metrics

The following is an example of using Prometheus to scrape the metrics endpoint of the Hello Helidon Greet application.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prometheus-operator
  name: hello-helidon
  namespace: hello-helidon
spec:
  endpoints:
  - bearerTokenSecret:
      key: ""
    enableHttp2: false
    path: /metrics
    relabelings:
    - action: replace
      replacement: local
      targetLabel: verrazzano_cluster
    - action: keep
      regex: true;hello-helidon
      sourceLabels:
      - __meta_kubernetes_pod_annotation_verrazzano_io_metricsEnabled
      - __meta_kubernetes_pod_label_app
    - action: replace
      regex: (.+)
      sourceLabels:
      - __meta_kubernetes_pod_annotation_verrazzano_io_metricsPath
      targetLabel: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      sourceLabels:
      - __address__
      - __meta_kubernetes_pod_annotation_verrazzano_io_metricsPort
      targetLabel: __address__
    - action: replace
      regex: (.*)
      replacement: $1
      sourceLabels:
      - __meta_kubernetes_namespace
      targetLabel: namespace
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      sourceLabels:
      - __meta_kubernetes_pod_name
      targetLabel: pod_name
    - action: labeldrop
      regex: (controller_revision_hash)
    - action: replace
      regex: .*/(.*)$
      replacement: $1
      sourceLabels:
      - name
      targetLabel: webapp
    - action: replace
      regex: ;(.*)
      replacement: $1
      separator: ;
      sourceLabels:
      - application
      - app
      targetLabel: application
    scheme: https
    tlsConfig:
      ca: {}
      caFile: /etc/istio-certs/root-cert.pem
      cert: {}
      certFile: /etc/istio-certs/cert-chain.pem
      insecureSkipVerify: true
      keyFile: /etc/istio-certs/key.pem
  namespaceSelector:
    matchNames:
    - hello-helidon
  selector: {}

Wiring for logs

Application logs record events happening in the Kubernetes cluster, which are automatically accessed by Verrazzano when required.

Docs: Get Console URLs

Mon, 01 Jan 0001 00:00:00 +0000

You can access the installation endpoints using the Verrazzano CLI or with kubectl:

Verrazzano CLI

You can get the endpoints for these consoles by issuing the following command and examining the Status.Instance field:

$ vz status

The resulting output is similar to the following:

Verrazzano Status
  Name: verrazzano
  Namespace: default
  Version: 1.5.0
  State: Ready
  Profile: dev
  Available Components: 24/24
  Access Endpoints:
    argoCDUrl: https://argocd.default.11.22.33.44.nip.io
    consoleUrl: https://verrazzano.default.11.22.33.44.nip.io
    grafanaUrl: https://grafana.vmi.system.default.11.22.33.44.nip.io
    jaegerURL: https://jaeger.default.11.22.33.44.nip.io
    keyCloakUrl: https://keycloak.default.11.22.33.44.nip.io
    kialiUrl: https://kiali.vmi.system.default.11.22.33.44.nip.io
    openSearchDashboardsUrl: https://osd.vmi.system.default.11.22.33.44.nip.io
    openSearchUrl: https://opensearch.vmi.system.default.11.22.33.44.nip.io
    prometheusUrl: https://prometheus.vmi.system.default.11.22.33.44.nip.io
    rancherUrl: https://rancher.default.11.22.33.44.nip.io
    thanosQueryUrl: https://thanos-query.default.11.22.33.44.nip.io

kubectl

You can get the endpoints for these consoles by issuing the following command and examining the Status.Instance field:

$ kubectl get vz -o yaml

The resulting output is similar to the following (abbreviated to show only the relevant portions):

  ...
  status:
    conditions:
    - lastTransitionTime: "2021-06-30T03:10:00Z"
      message: Verrazzano install in progress
      status: "True"
      type: InstallStarted
    - lastTransitionTime: "2021-06-30T03:18:33Z"
      message: Verrazzano install completed successfully
      status: "True"
      type: InstallComplete
    instance:
      argoCDUrl: https://argocd.default.11.22.33.44.nip.io
      consoleUrl: https://verrazzano.default.11.22.33.44.nip.io
      grafanaUrl: https://grafana.vmi.system.default.11.22.33.44.nip.io
      keyCloakUrl: https://keycloak.default.11.22.33.44.nip.io
      kialiUrl: https://kiali.vmi.system.default.11.22.33.44.nip.io
      opensearchDashboardsUrl: https://osd.vmi.system.default.11.22.33.44.nip.io
      opensearchUrl: https://opensearch.vmi.system.default.11.22.33.44.nip.io
      prometheusUrl: https://prometheus.vmi.system.default.11.22.33.44.nip.io
      rancherUrl: https://rancher.default.11.22.33.44.nip.io
      thanosQueryUrl: https://thanos-query.default.11.22.33.44.nip.io

If you have jq installed, then you can use the following command to get the instance URLs more directly.

$ kubectl get vz -o jsonpath="{.items[].status.instance}" | jq .

The following is an example of the output:

{
"argoCDUrl": https://argocd.default.11.22.33.44.nip.io
"consoleUrl": "https://verrazzano.default.11.22.33.44.nip.io",
"grafanaUrl": "https://grafana.vmi.system.default.11.22.33.44.nip.io",
"keyCloakUrl": "https://keycloak.default.11.22.33.44.nip.io",
"kialiUrl": "https://kiali.vmi.system.default.11.22.33.44.nip.io",
"opensearchUrl": "https://opensearch.vmi.system.default.11.22.33.44.nip.io",
"opensearchDashboardsUrl": "https://osd.vmi.system.default.11.22.33.44.nip.io",
"prometheusUrl": "https://prometheus.vmi.system.default.11.22.33.44.nip.io",
"rancherUrl": "https://rancher.default.11.22.33.44.nip.io"
"thanosQueryUrl": "https://thanos-query.default.11.22.33.44.nip.io"
}

Docs: Hello World Helidon

Mon, 01 Jan 0001 00:00:00 +0000

The Hello World Helidon example is a Helidon-based service that returns a “Hello World” response when invoked. The example application is specified using Open Application Model (OAM) component and application configuration YAML files, and then deployed by applying those files.

The example application has two endpoints, which differ in configuration source:

/greet- uses a microprofile properties file. Deploy this application by using the instructions here.
/config- uses a Kubernetes ConfigMap. Deploy this application by using the instructions here.

For more information and the code of this application, see the Verrazzano examples.

Docs: Installation Profiles

Mon, 01 Jan 0001 00:00:00 +0000

This document describes built-in configuration profiles that you can use to simplify a Verrazzano installation. An installation profile is a well-known configuration of Verrazzano settings that can be referenced by name, which then can be customized as needed.

The following table describes the Verrazzano installation profiles.

Profile	Description	Characteristics
`prod`	Full installation, production configuration.	Default profile: - Full installation. - Persistent storage. - Production OpenSearch cluster topology.
`dev`	Development or evaluation configuration.	Lightweight installation: - For evaluation purposes. - No persistence. - Single-node OpenSearch cluster topology.
`managed-cluster`	A specialized installation for managed clusters in a multicluster topology.	Minimal installation for a managed cluster: - Clusters must be registered with an admin cluster to use multicluster features.

Use an installation profile

To specify an installation profile when installing Verrazzano, set the profile name in the profile field of your Verrazzano custom resource.

For example, to use the dev profile:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev

To use a different profile, replace dev with prod, or managed-cluster.

Customize an installation profile

Regardless of the profile, you can override the profile settings for any component. The following example uses a customized dev profile to configure a small 8 Gi persistent volume for the MySQL instance used by Keycloak to provide more stability for the Keycloak service.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-dev-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 8Gi

For details on how to customize Verrazzano components, see Modify Verrazzano Installations.

Profile configurations

The following table lists the Verrazzano components that are enabled by default with each profile. Note that you can customize any Verrazzano installation regardless of the profile.

Component	dev	prod	managed-cluster
`applicationOperator`	✔️	✔️	✔️
`argoCD`
`authProxy`	✔️	✔️	✔️
`certManager`	✔️	✔️	✔️
`certManagerWebhookOCI`	️	️
`clusterAPI`	✔️	✔️
`clusterAgent`	✔️	✔️	✔️
`clusterIssuer`	✔️	✔️	✔️
`clusterOperator`	✔️	✔️	✔️
`coherenceOperator`	✔️	✔️	✔️
`console`	✔️	✔️
`dns`	✔️	✔️	✔️
`fluentbitOpensearchOutput`
`fluentd`	✔️	✔️	✔️
`fluentOperator`	️	️	️
`grafana`	✔️	✔️
`ingressNGINX`	✔️	✔️	✔️
`istio`	✔️	✔️	✔️
`jaegerOperator`
`keycloak`	✔️	✔️
`kiali`	✔️	✔️
`kubeStateMetrics`
`mySQLOperator`	✔️	✔️	✔️
`oam`	✔️	✔️	✔️
`opensearch`	✔️	✔️
`opensearchDashboards`	✔️	✔️
`prometheus`	✔️	✔️	✔️
`prometheusAdapter`
`prometheusNodeExporter`	✔️	✔️	✔️
`prometheusOperator`	✔️	✔️	✔️
`prometheusPushgateway`
`rancher`	✔️	✔️
`rancherBackup`
`thanos`
`velero`
`weblogicOperator`	✔️	✔️	✔️

Prometheus and Grafana configurations

The following table describes the Prometheus and Grafana configurations in each profile.

Profile	Prometheus	Grafana
`prod`	One replica (128 MB memory, 50 Gi storage)	One replica (48 MB memory, 50 Gi storage)
`dev`	One replica (128 MB memory, ephemeral storage)	One replica (48 MB memory, ephemeral storage)
`managed-cluster`	One replica (128 MB memory, 50 Gi storage)	Not installed

OpenSearch Dashboards and OpenSearch configurations

The following table describes the OpenSearch Dashboards and OpenSearch cluster topology in each profile.

Profile	OpenSearch	OpenSearch Dashboards
`prod`	Three master replicas (1.4 Gi memory, 50 Gi storage each) One ingest replica (2.5 Gi memory, no storage) Three data replicas (4.8 Gi memory, 50 Gi storage each)	One replica (192 MB memory, ephemeral storage)
`dev`	One master/data/ingest replica (1 Gi memory, ephemeral storage)	One replica (192 MB memory, ephemeral storage)
`managed-cluster`	Not installed	Not installed

NOTE

OpenSearch containers are configured to use 75% of the configured request memory for the Java min/max heap settings.

Profile-independent defaults

The following table shows the settings for components that are profile-independent (consistent across all profiles unless overridden).

Component	Default
DNS	Wildcard DNS provider nip.io.
Certificates	Uses the cert-manager self-signed ClusterIssuer for certificates.
Ingress-type	Defaults to `LoadBalancer` service type for the ingress.

Docs: Keycloak

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano stores user login information in Keycloak. In turn, Keycloak uses MySQL as a back end to store all persistent data. This document shows you how to back up persistent data stored in MySQL from the original cluster and restore it in a new cluster. If you are restoring data to the same cluster, then the terms original cluster and new cluster refer to same cluster.

MySQL Operator prerequisites

MySQL is deployed using the MySQL Operator for Kubernetes. Apart from managing the life cycle of MySQL instances, MySQL Operator provides the capability to back up and restore data using an Amazon S3 compatible object storage.

Before proceeding with a MySQL back up or restore operation, keep the following details handy:

Object storage bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object storage solution.
Object storage prefix name. This will be a child folder under the bucket, which the backup component creates.
Object storage region name.
Object storage signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object storage; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure (OCI), you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

The following example creates a secret mysql-backup-secret in the namespace keycloak. The instructions in this document back up data from MySQL to an Oracle Cloud Object Storage bucket and restore it from there.

MySQL Operator requires a secret to communicate with the S3 compatible object storage, so we create a backup-secret.txt file, which has the object storage credentials.
```
[default]
aws_access_key_id=<object storage access key>
aws_secret_access_key=<object storage secret key>
```
MySQL Operator requires the region name where the bucket is created, so we create a backup-region.txt file, which contains the region information. The following is an example of a backup-region.txt file indicating that the object storage is created in region us-phoenix-1:
```
[default]
region=us-phoenix-1
```
In the namespace keycloak, create a Kubernetes secret, for example mysql-backup-secret.
```
$ kubectl create secret generic --namespace <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file> --from-file=<key>=<full_path_to_config_file>
```
The following is an example to create a Kubernetes secret consisting of credentials to connect to OCI Object Storage.
```
$ kubectl create secret generic --namespace keycloak mysql-backup-secret --from-file=credentials=backup-secret.txt --from-file=config=backup-region.txt
```

NOTE

The secret must be created in the namespace keycloak.
To restore Keycloak on a new cluster, create the secret in the namespace, keycloak, in the new cluster.
To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.

MySQL Operator backup

To initiate a MySQL backup on the original cluster, create the following example custom resource YAML file that uses an OCI Object Storage as a back end. The operator uses the secret referenced in spec.backupProfile.dumpInstance.storage.s3.config to authenticate with the OCI Object Storage.
```
$ kubectl apply -f - <<EOF
    apiVersion: mysql.oracle.com/v2
    kind: MySQLBackup
    metadata:
      name: <backup name>
      namespace: keycloak
    spec:
      clusterName: mysql
      backupProfile:
        name: <backupProfileName>
        dumpInstance:
          storage:
            s3:
              bucketName: <The Object Storage bucket. See the MySQL Operator prerequisites section.>
              config: <Kubernetes secret name. See the MySQL Operator prerequisites section.>
              endpoint: < OCI S3 Object Storage endpoint.>
              prefix: <The prefix name. This folder will be automatically created.>
              profile: default
EOF
```
NOTE
- The config value is mysql-backup-secret, which is the name of the secret that you created previously in the keycloak namespace.
- The clustername has to be mysql.
- The namespace has to be keycloak.
- The profile value is the profile for the security credentials. In this case, it is default.
The following is an example of a MySQLBackup resource to initiate a MySQL backup:
```
$ kubectl apply -f - <<EOF
    apiVersion: mysql.oracle.com/v2
    kind: MySQLBackup
    metadata:
      name: mysql-backup
      namespace: keycloak
    spec:
      clusterName: mysql
      backupProfile:
        name: mysqlOneTime
        dumpInstance:
          storage:
            s3:
              bucketName: mysql-bucket
              config: mysql-backup-secret
              endpoint: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
              prefix: mysql-test
              profile: default
EOF
```
Confirm that the backup operation is complete. Run the following command on the original cluster and ensure that the STATUS is Completed.
```
$ kubectl get MySQLBackup --namespace keycloak
```
```
# Sample output
NAME           CLUSTER   STATUS      OUTPUT                         AGE
mysql-backup   mysql     Completed   mysql-backup-20221025-180836   119s
```
A successful backup of MySQL creates a backup folder in the object storage. Make note of the backup folder prefix name that the MySQL backup created on the original cluster.
```
$ kubectl get mysqlbackup --namespace keycloak <mysql-backup-name> -o jsonpath={.status.output}
```
The following is an example:
```
$ kubectl get mysqlbackup --namespace keycloak mysql-backup -o jsonpath={.status.output}
mysql-backup-20221025-180836
```
Back up MySQL Helm chart and values.

Back up the values in the MySQL Helm chart, in the original cluster to a file, mysql-values.yaml .
```
$ helm get values --namespace keycloak mysql > mysql-values.yaml
```
MySQL Helm charts are present inside the Verrazzano platform operator. Retrieve the charts from the original cluster to a local directory.

The following example retrieves the MySQL charts to a directory mysql-charts under the current directory. In order to avoid data corruption, ensure that the directory, mysql-charts, doesn’t already exist under the current directory.
```
$ kubectl cp --namespace verrazzano-install \
    $(kubectl get pod --namespace verrazzano-install -l app=verrazzano-platform-operator \
    -o custom-columns=:metadata.name --no-headers):platform-operator/thirdparty/charts/mysql \
    -c verrazzano-platform-operator mysql-charts/
```

Scheduled backups

You can also implement schedules for running MYSQL backups. For more information, see the Handling MySQL Backups section, “A PersistentVolumeClaim Scheduled Backup Example.”

MySQL Operator restore

Before you begin, read the MySQL Operator prerequisites. In addition, you must have at least one healthy backup before starting a restore operation.

To initiate a MySQL restore operation from an existing backup, you need to recreate the MySQL cluster. Use the following steps for a successful MySQL restore operation:

Delete the MySQL pods and PersistentVolumeClaim from the system on the new cluster.
```
$ helm delete mysql --namespace keycloak
$ kubectl delete pvc --namespace keycloak -l tier=mysql
```
Start a MySQL restore operation by installing the Helm chart by using the chart from the original cluster.
```
 $ helm install mysql <path to directory mysql-charts, where original charts are extracted> \
         --namespace keycloak \
         --set initDB.dump.name=<dump-name> \
         --set initDB.dumpOptions.loadUsers=true \
         --set initDB.dump.s3.profile=default \
         --set initDB.dump.s3.prefix=<prefixName/backup folder name> \
         --set initDB.dump.s3.bucketName=<OCI bucket name> \
         --set initDB.dump.s3.config=<Kubernetes secret name, see MySQL Operator prerequisites section.> \
         --set initDB.dump.s3.endpoint=<OCI S3 endpoint> \
         --values <mysql values file>
```
The following is an example:
```
 $ helm install mysql mysql-charts \
         --namespace keycloak \
         --set initDB.dump.name=alpha \
         --set initDB.dump.s3.profile=default \
         --set initDB.dump.s3.prefix=mysql-test/mysql-backup-20221025-180836 \
         --set initDB.dump.s3.bucketName=mysql-bucket \
         --set initDB.dump.s3.config=mysql-backup-secret \
         --set initDB.dump.s3.endpoint=https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com \
         --values mysql-values.yaml
```
After performing the restore command, wait for the MySQL cluster to be online. Ensure that the STATUS is ONLINE and the count under ONLINE matches the INSTANCES.
```
$ kubectl get innodbclusters --namespace keycloak mysql
```
```
# Sample output
NAME    STATUS   ONLINE   INSTANCES   ROUTERS   AGE
mysql   ONLINE   3        3           3         2m23s
```
Wait for all the MySQL pods to be in the RUNNING state.
```
$ kubectl wait --namespace keycloak --for=condition=ready pod -l tier=mysql --timeout=600s
```
```
# Sample output
pod/mysql-0 condition met
pod/mysql-1 condition met
pod/mysql-2 condition met
pod/mysql-router-746d9d75c7-6pc5p condition met
pod/mysql-router-746d9d75c7-bhrkw condition met
pod/mysql-router-746d9d75c7-t8bhb condition met
```
At this point, the MySQL cluster has been restored successfully from the backup, along with the PersistentVolumeClaim that was deleted previously.
If you are restoring Keycloak on a new cluster, then update the Keycloak secret.

On the original cluster, if you are restoring Keycloak on a new cluster, then run the following command for the keycloak-http secret in keycloak namespace:
```
 $ kubectl get secret --namespace keycloak keycloak-http -o jsonpath={.data.password}; echo
```
On the new cluster, replace the existing password value with the value displayed from the previous command.
```
 kubectl patch secret keycloak-http --namespace keycloak -p '{"data": {"password": "<password displayed in the step above>"}}'
```
Restart the Keycloak pods.

The removal and recreation of the MySQL cluster may bring down the Keycloak pods because MySQL goes offline during the restore operation. Run the following commands to restart the Keycloak pods:
```
 KEYCLOAK_REPLICAS=$(kubectl get sts --namespace keycloak keycloak -o custom-columns=:status.replicas --no-headers)
 kubectl scale sts --namespace keycloak keycloak --replicas=0
 kubectl scale sts --namespace keycloak keycloak --replicas=${KEYCLOAK_REPLICAS}
 kubectl wait --namespace keycloak --for=condition=ready pod -l app.kubernetes.io/instance=keycloak --timeout=600s
```

Update Verrazzano secrets in the new cluster

The following steps are applicable only if you are restoring Keycloak on a new cluster.

After you complete the MySQL restore operation, the password for the following secrets in the verrazzano-system namespace must be updated in the new cluster:

verrazzano
verrazzano-es-internal
verrazzano-prom-internal

On the original cluster, run the following command for the verrazzano secret:
```
 $ kubectl get secret --namespace verrazzano-system verrazzano -o jsonpath={.data.password}; echo
```
On the new cluster, replace the existing password value with the value displayed in step 1.
```
 kubectl patch secret verrazzano --namespace verrazzano-system -p '{"data": {"password": "<password displayed in step 1>"}}'
```
Repeat steps 1 and 2 for the verrazzano-es-internal and verrazzano-prom-internal secrets.
Restart the fluentd pods in the new cluster to use the original cluster password to connect to OpenSearch.
```
 $ kubectl delete pod -l app=fluentd --namespace verrazzano-system
```

Docs: Node Failure Guide

Mon, 01 Jan 0001 00:00:00 +0000

A Node failure can occur for many reasons, including hardware failures and network outages. This guide provides information about what to expect when a Node failure occurs and how to recover from a Node failure. Recovery depends on the Storage Provisioner and the type of storage that you use.

NOTE

This guide assumes that the storage provided in the cluster is physically separate from the Node and is recoverable. It does not apply to a local storage on the Node.

What to expect

By default, when a Node fails:

It may take up to a minute for the failure to reflect in the Kubernetes API server and update the Node status to NotReady.
After about five minutes of the Node status being NotReady, the status of the Pods on that Node will be changed to Unknown or NodeLost.
The status of the Pods with controllers, like Daemonsets, Statefulsets, and Deployments, will be changed to Terminating.

NOTE: Pods without a controller, started with a PodSpec, will not be terminated. They must be manually deleted and recreated.
New Pods will start on the Nodes that remain with Ready status.

NOTE: Statefulsets are a special case. The Statefulset controller maintains an ordinal list of Pods, one each for a given name. The Statefulset controller will not start a new Pod with the name of an existing Pod.
The Pods that have associated Persistent Volumes of type ReadWriteOnce, do not become Ready. This is because the Pods try to attach to the existing volumes that are still attached to the old Pod, which is still Terminating. This happens because, at a given point, the Persistent Volumes of type ReadWriteOnce, can be associated only with a single Node, and the new Pod resides on another Node.
If multiple Availability Domains are used in the Kubernetes cluster and the failed Node is the last in that Availability Domain, then the existing volumes will no longer be reachable by new Pods in a separate Availability Domain.

About recovery

After a Node fails, if the Node can be recovered within five minutes, then the Pods will return to a Running state. If the Node is not recovered after five minutes, then the Pods will complete termination and are deleted from the Kubernetes API server. New Pods that have Persistent Volumes of type ReadWriteOnce, will now be able to mount the Persistent Volumes and change to Running.

If a Node cannot be recovered and is replaced, then deleting the Node from the Kubernetes API server will terminate the old Pods and release the Persistent Volumes of type ReadWriteOnce, to be mounted by any new Pods.

If multiple Availability Domains are used in the Kubernetes cluster, then the replacement Node should be added to the same Availability Domain that the deleted Node occupied. This allows the Pods to be scheduled on the replacement Node that can reach the Persistent Volumes in that Availability Domain, and then the Pod status is changed to Running.

Do not forcefully delete Pods or Persistent Volumes in a failed Node that you plan to recover or replace. If you force delete Pods or Persistent Volumes in a failed Node, it may lead to loss of data and, in the case of Statefulsets, it may lead to split-brain scenarios. For more information about Statefulsets, see Force Delete StatefulSet Pods in the Kubernetes documentation.

You can force delete Pods and Persistent Volumes when a failed Node cannot be recovered or replaced in the same Availability Domain as the original Node.

Docs: OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides a ready-to-use, OpenSearch deployment that gives you access to all the log messages from various microservices running on the platform. There are scenarios where you may want to back up your OpenSearch data and restore it.

Verrazzano uses Velero to facilitate backing up and restoring OpenSearch data.

Velero operator prerequisites

Before proceeding with an OpenSearch backup or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore OpenSearch, you must first enable Velero.

The following configuration shows you how to enable velero with a prod installation profile.
```
  $ kubectl apply -f -<<EOF
    apiVersion: install.verrazzano.io/v1beta1
    kind: Verrazzano
    metadata:
      name: example-verrazzano
    spec:
      profile: prod
      components:    
        velero:
          enabled: true
EOF
```

NOTE: The OpenSearch back up and restore operation is supported only on prod installation profiles with a multinode OpenSearch configuration.

After Velero is enabled, verify that the Velero pods are running in the verrazzano-backup namespace.
```
# Sample of pods running after enabling the velero component
$ kubectl get pod -n verrazzano-backup
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h
```
Velero requires a secret to communicate with the S3 compatible object store, so we create a backup-secret.txt file, which has the object store credentials.
```
[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>
```
In the namespace verrazzano-backup, create a Kubernetes secret, for example verrazzano-backup-creds.
```
$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file>
```
The following is an example:
```
$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt
```
NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.
Create a BackupStorageLocation resource, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example. For more information, see Backup Storage Location in the Velero documentation.
```
 $ kubectl apply -f -<<EOF
 apiVersion: velero.io/v1
 kind: BackupStorageLocation
 metadata:
   name: verrazzano-backup-location
   namespace: verrazzano-backup
 spec:
   provider: aws
   objectStorage:
     bucket: example-verrazzano
     prefix: backup-demo
   credential:
     name: verrazzano-backup-creds
     key: cloud
   config:
     region: us-phoenix-1
     s3ForcePathStyle: "true"
     s3Url: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```

OpenSearch backup Using Velero

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero while invoking a backup. Due to the nature of transient data handled by OpenSearch, the hook invokes the OpenSearch snapshot APIs to back up data streams appropriately, thereby ensuring that there is no loss of data and avoids data corruption as well.

The following example shows a sample Velero Backup API resource that you can create to initiate an OpenSearch backup.

  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Backup
  metadata:
    name: verrazzano-opensearch-backup
    namespace: verrazzano-backup
  spec:
    includedNamespaces:
      - verrazzano-system
    labelSelector:
      matchLabels:
        verrazzano-component: opensearch
    defaultVolumesToRestic: false
    storageLocation:  verrazzano-backup-location
    hooks:
      resources:
        - name: opensearch-backup-test
          includedNamespaces:
            - verrazzano-system
          labelSelector:
            matchLabels:
              statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
          post:                           
            - exec:
                container: es-master
                command:
                  - /usr/share/opensearch/bin/verrazzano-backup-hook
                  - -operation
                  - backup
                  - -velero-backup-name
                  - verrazzano-opensearch-backup
                onError: Fail
                timeout: 10m
EOF

The preceding example backs up the OpenSearch components:

In this case, you are not backing up the PersistentVolumes directly, rather running a hook that invokes the OpenSearch APIs to take a snapshot of the data.
The defaultVolumesToRestic is set to false so that Velero ignores the associated PVCs.
In this case, the hook can be pre or post.
The command used in the hook requires an operation flag and the Velero backup name as an input.
The container on which the hook needs to be run defaults to the first container in the pod. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

After the backup is processed, you can see the hook logs using the velero backup logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch backup logs

# To display the logs from the backup, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-backup-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

OpenSearch scheduled backups

OpenSearch restore using Velero

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero to perform a restore operation. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to restore data streams appropriately, thereby ensuring there is no loss of data and avoids data corruption as well.

If you restore OpenSearch to a different cluster, create a Velero BackupStorageLocation resource in the new cluster that points to the same backup storage location configured in the original cluster. This ensures that the Velero resources created by the original cluster’s backup are automatically synced to the new cluster. Once the sync completes, you will be able to access the backup from the original cluster on the new cluster. It is recommended to configure the BackupStorageLocation on the new cluster as read-only by setting accessMode to ReadOnly in the BackupStorageLocation spec. This ensures that the backup in the object store is not modified from the new cluster. For more information, see Backup Storage Location in the Velero documentation.

To initiate an OpenSearch restore operation, first delete the existing OpenSearch cluster running on the system and all related data.

Scale down the Verrazzano Monitoring Operator. This is required because the operator manages the life cycle of the OpenSearch cluster, so scaling it down to zero ensures that it does not interfere with the restore operation. The restore operation also ensures that this operator is scaled back up to return the system to its previous state.
```
 $ kubectl scale deploy -n verrazzano-system verrazzano-monitoring-operator --replicas=0
```
Delete the OpenSearch components.
```
# These are sample commands to demonstrate the OpenSearch restore process
$ kubectl delete sts -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete deploy -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete pvc -n verrazzano-system -l verrazzano-component=opensearch
```
To perform an OpenSearch restore operation, you can invoke the following example Velero Restore API object.
```
  $ kubectl apply -f - <<EOF
  apiVersion: velero.io/v1
  kind: Restore
  metadata:
    name: verrazzano-opensearch-restore
    namespace: verrazzano-backup
  spec:
    backupName: verrazzano-opensearch-backup
    includedNamespaces:
      - verrazzano-system
    labelSelector:
      matchLabels:
        verrazzano-component: opensearch
    restorePVs: false
    hooks:
      resources:
      - name: opensearch-test
        includedNamespaces:
        - verrazzano-system       
        labelSelector:
          matchLabels:            
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        postHooks:
        - exec:
            container: es-master
            command:
            - /usr/share/opensearch/bin/verrazzano-backup-hook
            - -operation
            - restore
            - -velero-backup-name
            - verrazzano-opensearch-backup
            waitTimeout: 30m
            execTimeout: 30m
            onError: Fail
EOF
```
The preceding example will restore an OpenSearch cluster from an existing backup.
- In this case, you are not restoring PersistentVolumes directly, rather running a hook that invokes the OpenSearch APIs to restore them from an existing snapshot of the data.
- The restorePVs is set to false so that Velero ignores restoring PVCs.
- The command used in the hook requires an -operation flag and the Velero backup name as an input.
- The postHook will invoke the OpenSearch APIs that restore the snapshot data.
- The container on which the hook needs to be run defaults to the first container in the pod. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.
NOTE: The hook needs to be a postHook because it must be applied after the Kubernetes objects are restored.
Wait for all the OpenSearch pods to be in the RUNNING state.
```
  $ kubectl wait -n verrazzano-system --for=condition=ready pod -l verrazzano-component=opensearch --timeout=600s
    pod/vmi-system-es-data-0-6f49bdf6f5-fc6mz condition met
    pod/vmi-system-es-data-1-8f8785994-4pr7n condition met
    pod/vmi-system-es-data-2-d5f569d98-q8p2v condition met
    pod/vmi-system-es-ingest-6ddd86b9b6-fpl6j condition met
    pod/vmi-system-es-ingest-6ddd86b9b6-jtmrh condition met
    pod/vmi-system-es-master-0 condition met
    pod/vmi-system-es-master-1 condition met
    pod/vmi-system-es-master-2 condition met
```

After the restore operation is processed, you can see the hook logs using the velero restore logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch restore logs

# To display the logs from the restore, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-restore-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

OpenSearch restore in an existing cluster using OpenSearch API

OpenSearch restore using Velero typically is used for disaster recovery scenarios where you need to restore the entire cluster. But, if you want to restore the OpenSearch data within an already existing cluster, then you can use the OpenSearch API.

Assuming that you have previously created a backup using Velero, within the same cluster that would have completed the repository registration, and want to solely restore the OpenSearch data from that specific backup, run the following commands.

To get the registered repositories:

# To see all snapshot repositories
$ GET _snapshot/_all

To get all the snapshots in the registered repository:

# To see all snapshots in a repository
$ GET _snapshot/<backup_repository_name>/_all

To restore the specific snapshot:

# To restore the existing snapshot, run the following command
$ POST _snapshot/<backup_repository_name>/<snapshot-name>/_restore

<backup_repository_name>: The name of the backup repository where your OpenSearch snapshot is stored. Replace <backup_repository_name> with the name of your backup repository.
<snapshot-name>: The name of the OpenSearch snapshot you want to restore. Replace <snapshot-name> with the name of your snapshot.

For more information, see OpenSearch Restore in the OpenSearch documentation.

Docs: Perform the Uninstall

Mon, 01 Jan 0001 00:00:00 +0000

You can uninstall Verrazzano using the Verrazzano CLI or with kubectl.

See the following respective sections:

Uninstall using the Verrazzano CLI

Uninstall Verrazzano.
```
 $ vz uninstall
```
Wait for the uninstall operation to complete. The uninstall logs from the Verrazzano platform operator will be streamed to the command window until the uninstall operation has completed or until the default timeout (20m) has been reached.

The following is an example of the output:
```
Uninstalling Verrazzano
2022-11-22T16:31:20.377Z info Reconciling Verrazzano resource default/verrazzano, generation 2, version 1.4.2
2022-11-22T16:31:20.377Z info Deleting Verrazzano installation
2022-11-22T16:31:20.418Z info Uninstalling components
2022-11-22T16:31:20.418Z info Uninstalling Verrazzano default/verrazzano
...
```

Uninstall using kubectl

To delete a Verrazzano installation, delete the Verrazzano custom resource you used to install it into your cluster.

The following example starts a deletion of a Verrazzano installation in the background and then uses the kubectl logs -f command to tail the output of the pod performing the uninstall operation.

Get the name of the Verrazzano custom resource.
```
$ MYVZ=$(kubectl  get vz -o jsonpath="{.items[0].metadata.name}")
```
Delete the Verrazzano custom resource. After the custom resource is deleted, the Verrazzano platform operator completes the uninstall process by removing all of the Verrazzano-related components and resources, which can take some time.
```
$ kubectl delete verrazzano $MYVZ
```
To view the logs during deletion, use the following command:
```
$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'
```

For troubleshooting help, see Analysis Advice.

Docs: Perform the Verrazzano Upgrade

Mon, 01 Jan 0001 00:00:00 +0000

It is important to distinguish between updating the Verrazzano platform operator versus upgrading the Verrazzano installation. The platform operator contains the newer component charts and image versions, so it must be updated prior to upgrading the installation. Updating the platform operator has no effect on an existing installation until you initiate the Verrazzano installation upgrade. Currently, there is no way to roll back either the platform operator update or the Verrazzano installation upgrade.

You can upgrade Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

NOTE

For optimal functionality, be sure to install or upgrade the CLI version to match the Verrazzano version to which you are upgrading.

Upgrade Verrazzano using the CLI

In one simple step, you can upgrade to a specified version of Verrazzano using this command.

Update the Verrazzano resource to the desired version.

To update to the latest version (default):
```
$ vz upgrade
```
To update to a specific version, where <version> is the desired version:
```
$ vz upgrade --version <version>
```
Wait for the upgrade to complete. Upgrade logs will be streamed to the command window until the upgrade has completed or until the default timeout (30m) has been reached.

Upgrade using kubectl

Upgrading an existing Verrazzano installation is a two-step process:

Update the Verrazzano platform operator to the Verrazzano release version to which you want to upgrade.
Upgrade the Verrazzano installation.

Update the Verrazzano platform operator

In order to upgrade an existing Verrazzano installation, you must first update the Verrazzano platform operator.

Update the Verrazzano platform operator.

NOTE: If you are using a private container registry, then to update the platform operator, follow the instructions at Use a Private Registry.

To update to the latest version:
```
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/master/verrazzano-platform-operator.yaml
```
To update to a specific version, where <version> is the desired version:
```
# To update to the desired version:
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/<version>/verrazzano-platform-operator.yaml
```
Wait for the deployment to complete.
```
$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
```
```
# Expected response
deployment "verrazzano-platform-operator" successfully rolled out
```
Confirm that the operator pod is correctly defined and running.
```
$ kubectl -n verrazzano-install get pods
```
```
# Sample output
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s
```

Upgrade Verrazzano

To upgrade the Verrazzano installation, you need to change the version of your installed Verrazzano resource to the version supported by the Verrazzano platform operator.

NOTE: You may only change the version field during an upgrade; changes to other fields or component configurations are not supported at this time.

In one simple step, you can upgrade to a specified version of Verrazzano using this command.

$ kubectl patch vz example-verrazzano -p '{"spec":{"version":"master"}}' --type=merge

Alternatively, you can upgrade the Verrazzano installation using the following steps.

Update the Verrazzano resource to the desired version.

To upgrade the Verrazzano components, you must update the version field in your Verrazzano resource spec to match the version supported by the platform operator to which you upgraded and apply it to the cluster.

The value of the version field in the resource spec must be a Semantic Versioning value corresponding to a valid Verrazzano release version.

To update the resource, do one of the following:

a. Edit the YAML file you used to install Verrazzano and set the version field to the latest version.

For example, to upgrade to master, your YAML file should be edited to add or update the version field.
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
   profile: dev
   version: master
```
Then, apply the resource to the cluster (if you have not edited the resource in-place using kubectl edit).
```
$ kubectl apply -f example-verrazzano.yaml
```
b. Edit the Verrazzano resource directly using kubectl and set the version field directly, for example:
```
$ kubectl edit verrazzano example-verrazzano
# In the resource editor, add or update the version field to "version: master", then save.
```
Wait for the upgrade to complete.
```
$ kubectl wait \
    --timeout=30m \
    --for=condition=UpgradeComplete verrazzano/example-verrazzano
```

Docs: Prepare an Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the Oracle Cloud Infrastructure installation

Create the OKE cluster using the Oracle Cloud Infrastructure Console or by some other means.
Follow the instructions provided by OKE to download the Kubernetes configuration file for your cluster, and set the following ENV variable:
```
$ export KUBECONFIG=<path to valid Kubernetes config>
```
Optional, if your organization requires the use of a private registry to the Docker images installed by Verrazzano, see Use a Private Registry.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For an example on OKE, see Installing Calico and Setting Up Network Policies.

NOTE

OKE clusters using only the flannel overlay network is supported. OCI VCN-Native Pod Networking will not work.

Next steps

To continue, see the Installation Guide.

Docs: Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus is a system for monitoring cloud native applications and is used by Verrazzano to monitor applications. Prometheus is used in Verrazzano to collect system performance metrics and metrics for applications deployed or managed by Verrazzano. Prometheus analyzes the metrics and provides visualization using Grafana.

Customize Prometheus configuration

Verrazzano installs Prometheus components, including Prometheus Operator and Prometheus, using the kube-prometheus-stack Helm chart. You can customize the installation configuration using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the number of Prometheus replicas.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            prometheus:
              prometheusSpec:
                replicas: 3

For more information about setting component overrides, see Installation Overrides.

For information about all the overrides supported by the kube-prometheus-stack chart in Verrazzano, see values.yaml.

For instructions to customize persistent storage settings for Prometheus, see Customize Persistent Storage.

Configure Alertmanager

To configure Alertmanager to send alerts as SMTP notifications, complete the following steps:

Create a secret named smtp-secret in the verrazzano-monitoring namespace that contains the SMTP server credentials. For example:
```
$ kubectl -n verrazzano-monitoring create secret generic smtp-secret \
 --from-literal=username="<smtp server username>" \
 --from-literal=password="<smtp server password>"
```
Configure the Prometheus Operator component of the Verrazzano custom resource. For example:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            alertmanager:
              enabled: true
              alertmanagerSpec:
                podMetadata:
                  annotations:
                    sidecar.istio.io/inject: "false"
                secrets:
                - smtp-secret
              config:
                global:
                  smtp_auth_password_file: /etc/alertmanager/secrets/smtp-secret/password
                  smtp_auth_username: "<smtp server username>"
                  smtp_from: "<e-mail address used when sending out emails>"
                  smtp_smarthost: "<host or host:port for the smtp server>"
              receivers:
              - email_configs:
                - send_resolved: true
                  to: "<e-mail address of the receiver>"
                name: email-notifications
              route:
                group_by:
                - alertname
                receiver: email-notifications
                routes:
                - matchers:
                  - alertname =~ "InfoInhibitor|Watchdog"
                  receiver: email-notifications
```

For more information about Alertmanager configurations, see the Alertmanager Documentation.

After you have enabled Alertmanager in Step 2, you can deploy alert rules to get proactive alerts. To create a TestAlertRule, run the following command.

$ kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus-operator
  name: prometheus-operator-test
  namespace: verrazzano-monitoring
spec:
  groups:
    - name: test
      rules:
        - alert: TestAlertRule
          annotations:
            description: Test alert rule
            runbook_url: test-runbook-url
            summary: Test alert rule
          expr: |-
            prometheus_config_last_reload_successful{job="prometheus-operator-kube-p-prometheus",namespace="verrazzano-monitoring"} == 0
          for: 10m
          labels:
            severity: critical
EOF

For more information, see Deploying Prometheus Rules.

Configure data retention settings

Verrazzano configures Prometheus with a default data retention setting of 10 days. The rate of metrics data collected depends on many factors, including the number of monitors, the monitor scrape intervals, and the number of metrics returned by each monitor.

When using persistent storage for Prometheus, it is possible to consume all storage. If Prometheus uses all available persistent storage, then queries return no data and new metrics cannot be saved. You can customize the persistent storage settings, and change the data retention days and configure a maximum retention size. When configuring retention size, a good rule of thumb is to set the value to no more than 85 percent of the persistent storage size.

The following example configures Prometheus to store at most three days or 40 GB of metrics data.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-prometheus
spec:
  profile: prod
  components:
    prometheusOperator:
      overrides:
        - values:
            prometheus:
              prometheusSpec:
                retention: 3d
                retentionSize: 40GB

Docs: Rancher

Mon, 01 Jan 0001 00:00:00 +0000

Rancher maintains many configurations, like user credentials and cluster credentials, as ConfigMaps and namespace values. The rancher-backup Operator provides a seamless way to back up and restore Rancher installations, configuration, and data.

rancher-backup Operator prerequisites

Before proceeding with a Rancher back up or restore operation, the following details should be kept handy:

Object store bucket name.
- An Amazon S3 compatible object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - For reference, make a note of the bucket name and tenancy name.
  - For more information about creating a bucket with Object Storage, see Managing Buckets.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region name.
Object store signing key.
- A signing key, which is required to authenticate with the Amazon S3 compatible object store; this is an Access Key/Secret Key pair.
- In Oracle Cloud Infrastructure, you or your administrator creates the Customer Secret Key.
  - An associated Access Key will be generated for the secret key.
  - To create a Customer Secret Key, see Customer Secret Key.

To back up or restore Rancher, you must first enable rancherBackup.

The following configuration shows you how to enable rancherBackup.
```
$ kubectl apply -f -<<EOF
 apiVersion: install.verrazzano.io/v1beta1
  kind: Verrazzano
 metadata:
    name: example-verrazzano
  spec:
   profile: dev
    components:    
     rancherBackup:
        enabled: true
EOF
```
For rancher-backup, the pods will be created in the cattle-resources-system namespace.
```
  # Sample of pods running after enabling the rancherBackup component

  $ kubectl get pod -n cattle-resources-system
  NAME                              READY   STATUS    RESTARTS   AGE
  rancher-backup-5c4b985697-xw7md   1/1     Running   0          2d4h
```
Rancher requires a secret to communicate with the S3 compatible object store. So, in the namespace verrazzano-backup, create a Kubernetes secret rancher-backup-creds.
```
 $ kubectl create secret generic -n <backup-namespace> <secret-name> --from-literal=accessKey=<accesskey> --from-literal=secretKey=<secretKey>
```
The following is an example:
```
 $ kubectl create secret generic -n verrazzano-backup rancher-backup-creds --from-literal=accessKey="s5VLpXwa0xNZQds4UTVV" --from-literal=secretKey="nFFpvyxpQvb0dIQovsl0"
```

Rancher backup

The rancher-backup Operator creates the backup file, in *.tar.gz format, on the S3 compatible object store.

To initiate a Rancher backup, create the following example custom resource YAML file that uses an Amazon S3 compatible object store as a back end. The operator uses the credentialSecretNamespace value to determine where to look for the Amazon S3 backup secret.
```
 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Backup
   metadata:
     name: <rancher-backup-name>
   spec:
     storageLocation:
       s3:
         credentialSecretName: <rancher backup credential name>
         credentialSecretNamespace: <namespace where credential object was created>
         bucketName: <object store bucket. This must be exist as noted in pre-requisites section>
         folder: <folder name. This folder will be auto created>
         region: <region name where bucket exists>
         endpoint: <object store endpoint configuration>
     resourceSetName: rancher-resource-set
EOF
```

NOTE: In Step 3. of the example in the prerequisites section, you created the secret in the verrazzano-backup namespace.

The following is an example:

 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Backup
   metadata:
     name: rancher-backup-test
   spec:
      storageLocation:
       s3:
         credentialSecretName: rancher-backup-creds
         credentialSecretNamespace: verrazzano-backup
         bucketName: myvz-bucket
         folder: rancher-backup
         region: us-phoenix-1
         endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
     resourceSetName: rancher-resource-set
EOF

The *.tar.gz file is stored in a location configured in the storageLocation field. When the backup is complete, then the rancher-backup Operator creates a file on the S3 compatible object store.

You can retrieve the backed up file name, as shown:
```
 $ kubectl get backups.resources.cattle.io rancher-backup-test
   NAME                 LOCATION   TYPE       LATEST-BACKUP                                                                     RESOURCESET            AGE   STATUS
   rancher-backup-test             One-time   rancher-615034-957d182d-44cb-4b81-bbe0-466900049124-2022-11-14T16-42-28Z.tar.gz   rancher-resource-set   54s   Completed
```

Rancher scheduled backups

To implement scheduled Rancher backups, see Backup Configuration in the Rancher documentation.

Rancher restore

During the restore operation, Rancher ensures that it recreates all the CRDs related to Rancher and configurations. Restoring Rancher is done by creating a custom resource that indicates to rancherBackup to start the restore process.

To initiate a Rancher restore operation, create the following example custom resource YAML file. When a Restore custom resource is created, the operator accesses the backup *.tar.gz file specified and restores the application data from that file.
```
 $ kubectl apply -f - <<EOF
   apiVersion: resources.cattle.io/v1
   kind: Restore
   metadata:
     name: s3-restore
   spec:
     backupFilename: rancher-615034-957d182d-44cb-4b81-bbe0-466900049124-2022-11-14T16-42-28Z.tar.gz
     storageLocation:
       s3:
         credentialSecretName: rancher-backup-creds
         credentialSecretNamespace: verrazzano-backup
         bucketName: myvz-bucket
         folder: rancher-backup
         region: us-phoenix-1
         endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
EOF
```
The rancher-backup Operator scales down the Rancher deployment during the restore operation and scales it back up after the restoration completes.

Resources are restored in this order:
1. Custom Resource Definitions (CRDs)
2. Cluster-scoped resources
3. Namespace resources
NOTE: The backupFilename is retrieved from the Rancher backup created previously.
Wait for all the Rancher pods to be in the RUNNING state.
```
$ kubectl wait -n cattle-system --for=condition=ready pod -l app=rancher --timeout=600s
  pod/rancher-69976cffc6-bbx4p condition met
  pod/rancher-69976cffc6-fr75t condition met
  pod/rancher-69976cffc6-pcdf2 condition met
```

Docs: Traits and Workloads

Mon, 01 Jan 0001 00:00:00 +0000

Packages:

oam.verrazzano.io/v1alpha1

oam.verrazzano.io/v1alpha1

Resource Types:

IngressTrait

IngressTrait specifies the ingress traits API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string IngressTrait

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
IngressTraitSpec

`rules` []IngressRule	A list of ingress rules for an ingress trait.
`tls` IngressSecurity	(Optional) The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

status
IngressTraitStatus

The observed state of an ingress trait and related resources.

LoggingTrait

LoggingTrait specifies the logging traits API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string LoggingTrait

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
LoggingTraitSpec

`imagePullPolicy` string	(Optional) The optional image pull policy for the Fluentd image provided by the user.
`loggingConfig` string	The configuration provided by the user for the Fluentd configuration that consists of fluentd.conf: `<source>\n ... and so on ...\n`.
`loggingImage` string	(Optional) The name of the custom Fluentd image.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

status
LoggingTraitStatus

The observed state of a logging trait and related resources.

MetricsTrait

MetricsTrait specifies the metrics trait API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string MetricsTrait

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
MetricsTraitSpec

`enabled` bool	(Optional) Specifies whether metrics collection is enabled. Defaults to `true`.
`path` string	(Optional) The HTTP path for the related metrics endpoint. Defaults to `/metrics`.
`port` int	(Optional) The HTTP port for the related metrics trait. Defaults to `8080`.
`ports` []PortSpec	(Optional) The HTTP endpoints for the related metrics.
`scraper` string	(Optional) The Prometheus deployment used to scrape the related metrics endpoints. By default, the Verrazzano-supplied Prometheus component is used to scrape the endpoint.
`secret` string	(Optional) The name of an opaque secret (for example, `username` and `password`) within the workload’s namespace for metrics endpoint access.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

status
MetricsTraitStatus

The observed state of a metrics trait and related resources.

VerrazzanoCoherenceWorkload

VerrazzanoCoherenceWorkload specifies the Verrazzano Coherence workload API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string VerrazzanoCoherenceWorkload

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoCoherenceWorkloadSpec

The desired state of a Verrazzano Coherence workload.

template
Kubernetes runtime.RawExtension

The metadata and spec for the underlying Coherence resource.

status
VerrazzanoCoherenceWorkloadStatus

The observed state of a Verrazzano Coherence workload.

VerrazzanoHelidonWorkload

VerrazzanoHelidonWorkload specifies the Verrazzano Helidon workload API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string VerrazzanoHelidonWorkload

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoHelidonWorkloadSpec

The desired state of a Verrazzano Helidon workload.

`deploymentTemplate` DeploymentTemplate	An embedded Helidon application deployment.
`serviceTemplate` ServiceTemplate	An embedded Helidon application service.

status
VerrazzanoHelidonWorkloadStatus

The observed state of a Verrazzano Helidon workload.

VerrazzanoWebLogicWorkload

VerrazzanoWebLogicWorkload specifies the Verrazzano WebLogic workload API.

Field Description

apiVersion
string oam.verrazzano.io/v1alpha1

kind
string VerrazzanoWebLogicWorkload

metadata
Kubernetes meta/v1.ObjectMeta Refer to the Kubernetes API documentation for the fields of the metadata field.

spec
VerrazzanoWebLogicWorkloadSpec

The desired state of a Verrazzano WebLogic workload.

`template` VerrazzanoWebLogicWorkloadTemplate	The metadata and spec for the underlying Domain resource.
`clusters` []VerrazzanoWebLogicWorkloadTemplate

status
VerrazzanoWebLogicWorkloadStatus

The observed state of a Verrazzano WebLogic workload.

AuthorizationPolicy

(Appears on:IngressPath)

AuthorizationPolicy defines the set of rules for authorizing a request.

Field	Description
`rules` []AuthorizationRule	Rules are used to match requests from request principals to specific paths given an optional list of conditions.

AuthorizationRule

(Appears on:AuthorizationPolicy)

AuthorizationRule matches requests from a list of request principals that access a specific path subject to a list of conditions.

Field	Description
`from` AuthorizationRuleFrom	Specifies the request principals for access to a request. An asterisk (*) will match when the value is not empty, for example, if any request principal is found in the request.
`when` []AuthorizationRuleCondition	(Optional) Specifies a list of additional conditions for access to a request.

AuthorizationRuleCondition

(Appears on:AuthorizationRule)

AuthorizationRuleCondition provides additional required attributes for authorization.

Field	Description
`key` string	The name of a request attribute.
`values` []string	A list of allowed values for the attribute.

AuthorizationRuleFrom

(Appears on:AuthorizationRule)

AuthorizationRuleFrom includes a list of request principals.

Field	Description
`requestPrincipals` []string	Specifies the request principals for access to a request.

DeploymentTemplate

(Appears on:VerrazzanoHelidonWorkloadSpec)

DeploymentTemplate specifies the metadata and pod spec of a Helidon workload.

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	Metadata about a Helidon application. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`podSpec` Kubernetes core/v1.PodSpec	The pod spec of a Helidon application.
`selector` Kubernetes meta/v1.LabelSelector	(Optional) Label selector of a Helidon application.
`strategy` Kubernetes apps/v1.DeploymentStrategy	(Optional) The replacement strategy of a Helidon application.

IngressDestination

(Appears on:IngressRule)

IngressDestination specifies a specific destination host and port for the ingress paths.

NOTE

If there are multiple ports defined for a service, then the destination port must be specified OR the service port name must have the prefix http.

Field	Description
`host` string	(Optional) Destination host.
`httpCookie` IngressDestinationHTTPCookie	(Optional) Session affinity cookie.
`port` uint32	(Optional) Destination port.

IngressDestinationHTTPCookie

(Appears on:IngressDestination)

IngressDestinationHTTPCookie specifies a session affinity cookie for an ingress trait.

Field	Description
`name` string	(Optional) The name of the HTTP cookie.
`path` string	(Optional) The path of the HTTP cookie.
`ttl` time.Duration	(Optional) The lifetime of the HTTP cookie (in seconds).

IngressPath

(Appears on:IngressRule)

IngressPath specifies a specific path to be exposed for an ingress trait.

Field Description

path
string

(Optional)

If no path is provided, then it defaults to forward slash (/).

pathType
string

(Optional)

Path type values are case-sensitive and formatted as follows:

exact: exact string match
prefix: prefix-based match
regex: regex-based match

Defaults to prefix if path specified is /; otherwise, defaults to exact.

authorizationPolicy
AuthorizationPolicy

(Optional)

Defines the set of rules for authorizing a request.

IngressRule

(Appears on:IngressTraitSpec)

IngressRule specifies a rule for an ingress trait.

Field	Description
`destination` IngressDestination	(Optional) The destination host and port for the ingress paths.
`hosts` []string	(Optional) One or more hosts exposed by the ingress trait. Wildcard hosts or hosts that are empty are filtered out. If there are no valid hosts provided, then a DNS host name is automatically generated and used.
`paths` []IngressPath	The paths to be exposed for an ingress trait.

IngressSecurity

(Appears on:IngressTraitSpec)

IngressSecurity specifies the secret containing the certificate securing the transport for an ingress trait.

Field	Description
`secretName` string	The name of a secret containing the certificate securing the transport. The specification of a secret here implies that a certificate was created for specific hosts, as specified in an IngressRule.

IngressTraitSpec

(Appears on:IngressTrait)

IngressTraitSpec specifies the desired state of an ingress trait.

Field	Description
`rules` []IngressRule	A list of ingress rules for an ingress trait.
`tls` IngressSecurity	(Optional) The security parameters for an ingress trait. This is required only if specific hosts are given in an IngressRule.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

IngressTraitStatus

(Appears on:IngressTrait)

IngressTraitStatus specifies the observed state of an ingress trait and related resources.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this ingress trait.

resources
[]OAM common/v1.TypedReference

The resources managed by this ingress trait.

LoggingTraitSpec

(Appears on:LoggingTrait)

LoggingTraitSpec specifies the desired state of a logging trait.

Field	Description
`imagePullPolicy` string	(Optional) The optional image pull policy for the Fluentd image provided by the user.
`loggingConfig` string	The configuration provided by the user for the Fluentd configuration that consists of fluentd.conf: `<source>\n ... and so on ...\n`.
`loggingImage` string	(Optional) The name of the custom Fluentd image.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

LoggingTraitStatus

(Appears on:LoggingTrait)

LoggingTraitStatus specifies the observed state of a logging trait and related resources.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this logging trait.

resources
[]OAM common/v1.TypedReference

The resources managed by this logging trait.

MetricsTraitSpec

(Appears on:MetricsTrait)

MetricsTraitSpec specifies the desired state of a metrics trait.

Field	Description
`enabled` bool	(Optional) Specifies whether metrics collection is enabled. Defaults to `true`.
`path` string	(Optional) The HTTP path for the related metrics endpoint. Defaults to `/metrics`.
`port` int	(Optional) The HTTP port for the related metrics trait. Defaults to `8080`.
`ports` []PortSpec	(Optional) The HTTP endpoints for the related metrics.
`scraper` string	(Optional) The Prometheus deployment used to scrape the related metrics endpoints. By default, the Verrazzano-supplied Prometheus component is used to scrape the endpoint.
`secret` string	(Optional) The name of an opaque secret (for example, `username` and `password`) within the workload’s namespace for metrics endpoint access.
`workloadRef` OAM common/v1.TypedReference	The WorkloadReference of the workload to which this trait applies. This value is populated by the OAM runtime when an ApplicationConfiguration resource is processed. When the ApplicationConfiguration is processed, a trait and a workload resource are created from the content of the ApplicationConfiguration. The WorkloadReference is provided in the trait by OAM to ensure that the trait controller can find the workload associated with the component containing the trait within the original ApplicationConfiguration.

MetricsTraitStatus

(Appears on:MetricsTrait)

MetricsTraitStatus defines the observed state of a metrics trait and related resources.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this metrics trait.

resources
[]QualifiedResourceRelation

Related resources affected by this metrics trait.

PortSpec

(Appears on:MetricsTraitSpec)

PortSpec defines an HTTP port and path combination.

Field	Description
`path` string	(Optional) The HTTP path for the related metrics endpoint. Defaults to `/metrics`.
`port` int	(Optional) The HTTP port for the related metrics trait. Defaults to `8080`.

QualifiedResourceRelation

(Appears on:MetricsTraitStatus, VerrazzanoHelidonWorkloadStatus)

QualifiedResourceRelation identifies a specific related resource.

Field	Description
`apiversion` string	API version of the related resource.
`kind` string	Kind of the related resource.
`name` string	Name of the related resource.
`namespace` string	Namespace of the related resource.
`role` string	Role of the related resource, for example, `Deployment`.

ServiceTemplate

(Appears on:VerrazzanoHelidonWorkloadSpec)

ServiceTemplate specifies the metadata and pod spec of a Helidon workload.

Field	Description
`metadata` Kubernetes meta/v1.ObjectMeta	(Optional) Metadata about a Helidon application. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`serviceSpec` Kubernetes core/v1.ServiceSpec	(Optional) The service spec of a Helidon application.

VerrazzanoCoherenceWorkloadSpec

(Appears on:VerrazzanoCoherenceWorkload)

VerrazzanoCoherenceWorkloadSpec wraps a Coherence resource.

Field	Description
`template` Kubernetes runtime.RawExtension	The metadata and spec for the underlying Coherence resource.

VerrazzanoCoherenceWorkloadStatus

(Appears on:VerrazzanoCoherenceWorkload)

VerrazzanoCoherenceWorkloadStatus defines the observed state of a Verrazzano Coherence workload.

Field	Description
`lastGeneration` string	The last generation of the Verrazzano Coherence workload that was reconciled.
`lastRestartVersion` string	The last value of the `verrazzano.io/restart-version` annotation.

VerrazzanoHelidonWorkloadSpec

(Appears on:VerrazzanoHelidonWorkload)

VerrazzanoHelidonWorkloadSpec wraps a Helidon application deployment and service.

Field	Description
`deploymentTemplate` DeploymentTemplate	An embedded Helidon application deployment.
`serviceTemplate` ServiceTemplate	An embedded Helidon application service.

VerrazzanoHelidonWorkloadStatus

(Appears on:VerrazzanoHelidonWorkload)

VerrazzanoHelidonWorkloadStatus defines the observed state of Verrazzano Helidon workload.

Field Description

ConditionedStatus
OAM common/v1.ConditionedStatus

(Members of ConditionedStatus are embedded into this type.)

Reconcile status of this Verrazzano Helidon workload.

resources
[]QualifiedResourceRelation

The resources managed by this Verrazzano Helidon workload.

VerrazzanoWebLogicWorkloadSpec

(Appears on:VerrazzanoWebLogicWorkload)

VerrazzanoWebLogicWorkloadSpec wraps a WebLogic resource. The WebLogic domain specified in the template must contain a spec field and it may include a metadata field.

Field	Description
`template` VerrazzanoWebLogicWorkloadTemplate	The metadata and spec for the underlying Domain resource.
`clusters` []VerrazzanoWebLogicWorkloadTemplate

VerrazzanoWebLogicWorkloadStatus

(Appears on:VerrazzanoWebLogicWorkload)

VerrazzanoWebLogicWorkloadStatus defines the observed state of a Verrazzano WebLogic workload.

Field	Description
`lastGeneration` string	The last generation of the Verrazzano WebLogic workload that was reconciled.
`lastRestartVersion` string	The last value of the `verrazzano.io/restart-version` annotation.
`lastLifecycleAction` string	The last value of the `verrazzano.io/lifecycle-action`.

VerrazzanoWebLogicWorkloadTemplate

(Appears on:VerrazzanoWebLogicWorkloadSpec)

Field Description

apiVersion
string

metadata
Kubernetes runtime.RawExtension

spec
Kubernetes runtime.RawExtension

-
[]byte

Raw is the underlying serialization of this object.

TODO: Determine how to detect ContentType and ContentEncoding of ‘Raw’ data.

-
Kubernetes runtime.Object

Object can hold a representation of this extension - useful for working with versioned structs.

Generated with gen-crd-api-reference-docs on git commit 238db2c3e.

Docs: Use Kubernetes Custom Resources

Mon, 01 Jan 0001 00:00:00 +0000

To register managed clusters using the VerrazzanoManagedCluster resource, complete the following steps:

Create the environment variables, KUBECONFIG_ADMIN, KUBECONTEXT_ADMIN, KUBECONFIG_MANAGED1, and KUBECONTEXT_MANAGED1, and point them to the kubeconfig files and contexts for the admin and managed cluster, respectively. You will use these environment variables in subsequent steps when registering the managed cluster. The following shows an example of how to set these environment variables.
```
$ export KUBECONFIG_ADMIN=/path/to/your/adminclusterkubeconfig
$ export KUBECONFIG_MANAGED1=/path/to/your/managedclusterkubeconfig

# Lists the contexts in each kubeconfig file
$ kubectl --kubeconfig $KUBECONFIG_ADMIN config get-contexts -o=name
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 config get-contexts -o=name

# Choose the right context name for your admin and managed clusters from the output shown and set the KUBECONTEXT
# environment variables
$ export KUBECONTEXT_ADMIN=<admin-cluster-context-name>
$ export KUBECONTEXT_MANAGED1=<managed-cluster-context-name>
```
To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster resource on the admin cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster resource"
EOF
```
Wait for the VerrazzanoManagedCluster resource to reach the Ready status. At that point, it will have generated a YAML file that must be applied on the managed cluster to complete the registration process.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc
```
Export the YAML file created to register the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml
```
Apply the registration file exported in the previous step, on the managed cluster.
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register.yaml

# After the command succeeds, you may delete the register.yaml file
$ rm register.yaml
```

Docs: Use the Bug Report Tool

Mon, 01 Jan 0001 00:00:00 +0000

Use the vz bug-report tool to selectively capture cluster information and create an archive (*.tar.gz) file to help diagnose problems. The archive file helps development and support teams analyze issues and provide recommendations. If a failure occurs while Verrazzano is being installed, upgraded, or uninstalled using the Verrazzano CLI, then the archive file is generated automatically.

CLI setup

To set up the vz command-line tool, follow the steps here.

Use the vz bug-report tool

To create a bug report in a TAR file named my-bug-report.tar.gz:

$ vz bug-report --report-file my-bug-report.tar.gz

We suggest that you review the contents of the bug report before sharing it with support and development teams.

Usage information

Use the following syntax to run vz commands from your terminal window.

$ vz bug-report [flags]

Available options

Command	Definition
`-h, --help`	Help for the `vz bug-report` command.
`-i, --include-namespaces strings`	A comma-separated list of namespaces, in addition to the ones collected by default (system namespaces), for collecting cluster information. This flag can be specified multiple times, such as `--include-namespaces ns1 --include-namespaces ns...`
`-r, --report-file string`	The report file created by the `vz bug-report` command, as a `*.tar.gz` file. Defaults to `bug-report.tar.gz` in the current directory.
`-l --include-logs`	Include logs from the pods in one or more namespaces; this is specified along with the `--include-namespaces` flag.
`-d --duration duration`	The time period during which the logs are collected in seconds, minutes, and hours.
`-v, --verbose`	Enable verbose output.

Available flags

These flags apply to all the commands.

Flag	Definition
`--context string`	The name of the kubeconfig file context to use.
`--kubeconfig string`	Path to the kubeconfig file to use.

Examples

Create a bug report file, bugreport.tar.gz, by collecting data from the cluster:
```
$ vz bug-report --report-file bugreport.tar.gz
```
When --report-file is not provided, the command creates bug-report.tar.gz in the current directory.
Create a bug report file, bugreport.tar.gz, including the additional namespace ns1 from the cluster:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1
```
The flag --include-namespaces accepts comma-separated values and can be specified multiple times. For example, the following commands create a bug report by including the additional namespaces ns1, ns2, and ns3:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2,ns3
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-namespaces ns3
```
Use the --include-logs flag to collect the logs from the pods in one or more namespaces, by specifying the --include-namespaces flag. For example, the following command creates a bug report by including the logs from the additional namespaces ns1 and ns2:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs
```
The --duration flag collects logs for the specified time period. The default value is zero (0), which collects the complete pod log. You can specify seconds, minutes, and hours. For example, the following commands create bug reports by including the logs from the additional namespaces ns1 and ns2 during the specified periods of time:
```
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 5m
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 2h
$ vz bug-report --report-file bugreport.tgz --include-namespaces ns1,ns2 --include-logs --duration 300s
```
The values specified for the flag --include-namespaces are case-sensitive.

Docs: Verify Managed Cluster Registration

Mon, 01 Jan 0001 00:00:00 +0000

Verify that managed cluster registration has completed

You can perform all the verification steps on the admin cluster.

Verify that the managed cluster can connect to the admin cluster. View the status of the VerrazzanoManagedCluster resource on the admin cluster, and check whether the lastAgentConnectTime, prometheusHost, and apiUrl fields are populated. This may take up to two minutes after completing the registration steps.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get vmc managed1 -n verrazzano-mc -o yaml

# Sample output showing the status field
spec:
  ....
  ....
status:
  apiUrl: https://verrazzano.default.172.18.0.211.nip.io
  conditions:
  - lastTransitionTime: "2023-06-16T07:00:00Z"
    message: Ready
    status: "True"
    type: Ready
  lastAgentConnectTime: "2023-06-16T07:25:16Z"
  prometheusHost: prometheus.vmi.system.default.172.18.0.211.nip.io
  rancherRegistration:
    clusterID: c-xqstg
    message: Registration of managed cluster completed successfully for cluster managed with ID c-xqstg
  status: Completed
  state: Active
```
Verify that the managed cluster is successfully registered with Rancher. When you perform the registration steps, Verrazzano also registers the managed cluster with Rancher. View the Rancher console on the admin cluster. If the registration with Rancher was successful, then your cluster will be listed in Rancher’s list of clusters, and will be in the Active state. You can find the Rancher console URL for your cluster by following the instructions for Accessing Verrazzano.

Verify that managed cluster metrics are being collected

Verify that the admin cluster is collecting metrics from the managed cluster. The Prometheus output will include records that contain the name of the Verrazzano cluster (labeled as verrazzano_cluster).

You can find the Prometheus console URL for your cluster by following the instructions for Accessing Verrazzano. Run a query for a metric (for example, node_disk_io_time_seconds_total).

Sample output of a Prometheus query

An alternative approach to using the Prometheus console is to query metrics from the command line. Here is an example of how to obtain Prometheus metrics from the command line. Search the output of the query for responses that have the verrazzano_cluster field set to the name of the managed cluster.

# On the admin cluster
$ prometheusUrl=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.prometheusUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} "${prometheusUrl}/api/v1/query?query=node_disk_io_time_seconds_total"

Verify that managed cluster logs are being collected

Verify that the admin cluster is collecting logs from the managed cluster. The output will include records which have the name of the managed cluster in the cluster_name field.

You can find the OpenSearch Dashboards URL for your cluster by following the instructions for Accessing Verrazzano. Searching the verrazzano-system data stream for log records with the cluster_name set to the managed cluster name yields logs for the managed cluster.

Sample output of a OpenSearch Dashboards screen

An alternative approach to using the OpenSearch Dashboards is to query OpenSearch from the command line. Here is an example of how to obtain log records from the command line. Search the output of the query for responses that have the cluster_name field set to the name of the managed cluster.

# On the admin cluster
$ OS_URL=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
                 get verrazzano -o jsonpath='{.items[0].status.instance.openSearchUrl}')
$ VZPASS=$(kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
           get secret verrazzano --namespace verrazzano-system \
           -o jsonpath={.data.password} | base64 --decode; echo)
$ curl -k --user verrazzano:${VZPASS} -X POST -H 'kbn-xsrf: true' "${OS_URL}/verrazzano-system/_search?size=25"

Docs: Verify Using CLI kubectl

Mon, 01 Jan 0001 00:00:00 +0000

To verify the Verrazzano installation, you can use kubectl to view the status of the Verrazzano resource. After a successful installation, Verrazzano status should be InstallComplete.

$ kubectl get vz

# Example response
NAME                 AVAILABLE   STATUS            VERSION
example-verrazzano   23/23       InstallComplete   v2.0.0

For installation troubleshooting help, see Analysis Advice.

After the installation has completed, you can use the Verrazzano consoles. For information on how to get the consoles URLs and credentials, see Access Verrazzano.

Next steps

(Optional) Run the example applications located here.

Docs: Certificate Management

Mon, 01 Jan 0001 00:00:00 +0000

Use the following example to secure public access to the Hello Helidon Greet application.

Replace domain name with the Kubernetes cluster domain used in Verrazzano. This ensures that you have a fully-qualified domain name for host entries in the resources.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: hello-helidon-hello-helidon-ingress-cert
  namespace: istio-system # Note the use of the istio-system Namespace.
spec:
  dnsNames:
  - hello-helidon.hello-helidon.<domain name>
  issuerRef:
    kind: ClusterIssuer
    name: verrazzano-cluster-issuer
  secretName: hello-helidon-hello-helidon-ingress-cert-secret

Docs: Coherence Workload

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano application can contain any number of Coherence component workloads, where each workload is a standalone Coherence cluster, independent from other Coherence clusters in the application.

Verrazzano uses the standard Coherence Operator to provision and manage clusters, as documented at Coherence Operator. The Coherence Operator uses a CRD, coherence.oracle.com (Coherence resource), to represent a Coherence cluster. When a Verrazzano application with Coherence is provisioned, Verrazzano configures the default logging and metrics for the Coherence cluster. Logs are sent to OpenSearch and metrics to Prometheus. You can view this telemetry data using the OpenSearch Dashboards and Grafana consoles.

OAM Component

The custom resource YAML file for the Coherence cluster is specified as a VerrazzanoCoherenceWorkload custom resource. In the following example, everything under the spec: section is standard Coherence resource YAML that you would typically use to provision a Coherence cluster. Including this Component reference in your ApplicationConfiguration will result in a new Coherence cluster being provisioned. You can have multiple clusters in the same application with no conflict.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
      ...

Life cycle

With Verrazzano, you manage the life cycle of applications using Component and ApplicationConfiguration resources. Typically, you would modify the Coherence cluster resource to make changes or to do lifecycle operations, like scale in and scale out. However, in the Verrazzano environment, the cluster resource is owned by the Verrazzano application operator and will be reconciled to match the Component workload resource. Therefore, you need to manage the cluster configuration by modifying the resource, either by kubectl edit or applying a new YAML file. Verrazzano will notice that the Component resource changed and will update the Coherence resource as needed.

Provisioning

When you apply the Component YAML file shown previously, Kubernetes will create a component.oam.verrazzano.io resource, but the Coherence cluster will not be created until you create the ApplicationConfiguration resource, which references the Coherence component. When the application is created, Verrazzano creates a Coherence custom resource for each cluster, which is subsequently processed by the Coherence Operator, resulting in a new cluster. After a cluster is created, the Coherence Operator will monitor the Coherence resource to reconcile the state of the cluster. You can add a new Coherence workload to a running application, or remove an existing workload, by modifying the ApplicationConfiguration resource, and adding or removing the Coherence component.

Scaling

Scaling a Coherence cluster is done by modifying the replicas field in the Component resource. Verrazzano will modify the Coherence resource replicas field and the cluster will be scaled accordingly. The following example configuration shows the replicas field that specifies the number of pods in the cluster.

apiVersion: core.oam.dev/v1alpha2
kind: Component
metadata:
  name: orders
  namespace: sockshop
spec:
  workload:
    apiVersion: oam.verrazzano.io/v1alpha1
    kind: VerrazzanoCoherenceWorkload
    spec:
      template:
        metadata:
          name: orders-coh
        spec:
          cluster: SockShop
          replicas: 3
          ...

NOTE: A Coherence cluster provisioned with Verrazzano does not support autoscaling with a Horizontal Pod Autoscaler.

Termination

You can terminate the Coherence cluster by removing the Component from the ApplicationConfiguration or by deleting the ApplicationConfiguration resource entirely.

NOTE

Do not delete the Coherence component if the application is still using it.

Logging

When a Coherence cluster is provisioned, Verrazzano configures it to send logs to OpenSearch. This is done by injecting a Fluentd sidecar configuration into the Coherence resource. The Coherence Operator will create the pod with the Fluentd sidecar. This sidecar periodically copies the Coherence logs from /logs to stdout, enabling the Fluentd DaemonSet in the verrazzano-system namespace to send the logs to OpenSearch. Note that the Fluend sidecar running in the Coherence pod never communicates with OpenSearch or any other network endpoint.

The logs are placed in a per-namespace OpenSearch data stream named verrazzano-application-<namespace>, for example: verrazzano-application-sockshop. All logs from Coherence pods in the same namespace will go into the same data stream, even for different applications. This is standard behavior and there is no way to disable or change it.

Each log record has some Coherence and application fields, along with the log message itself. For example:

 kubernetes.labels.coherenceCluster        SockShop
 kubernetes.labels.app_oam_dev/name        sockshop-appconf
 kubernetes.labels.app_oam_dev/component   orders
 ...

Metrics

Verrazzano uses Prometheus to scrape metrics from Coherence cluster pods. Like logging, metrics scraping is also enabled during provisioning, however, the Coherence resource YAML file must have proper metrics configuration. For details, see Coherence Metrics. In summary, there are two ways to configure the Coherence metrics endpoint. Coherence has a default metrics endpoint that you can enable. If your application serves metrics from its own endpoint, such as a Helidon application, then do not use the native Coherence metrics endpoint. To see the difference, examine the socks-shop and bobs-books examples.

Bobs Books

The bobs-books example uses the default Coherence metrics endpoint, so the configuration must enable this feature, shown in the following metrics section of the roberts-coherence component in the YAML file, bobs-books-comp.yaml.

...
          coherence:
            metrics:
              enabled: true

Sock Shop

The sock-shop example, which is a Helidon application with embedded Coherence, explicitly specifies the metrics port 7001 and doesn’t enable Coherence metrics. Coherence metrics still will be scraped, but not at the default endpoint.

      ports:
        ...
        - name: metrics
          port: 7001
          serviceMonitor:
            enabled: true

Because sock-shop components are not using the default Coherence metrics port, you must add a MetricsTrait section to the ApplicationConfiguration for each component, specifying the metrics port as follows:

    - trait:
        apiVersion: oam.verrazzano.io/v1alpha1
        kind: MetricsTrait
        metadata:
          name: carts-metrics
        spec:
          port: 7001

Prometheus configuration

Prometheus is configured using the Prometheus Operator to scrape application targets. During application deployment, Verrazzano creates or updates Service Monitors based on the MetricsTrait specified in the ApplicationConfiguration. When the application is deleted, Verrazzano removes the Service Monitors so that metrics are no longer collected for it.

Here is an example of the sock-shop Prometheus Service Monitor resource for catalog-coh in the application namespace.
Notice that services with certain labels are targeted. Prometheus Operator will find the Service Monitor and generate the scrape configuration to be used by Prometheus.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  ....
  name: catalog-coh-metrics
  namespace: sockshop
  ....
spec:
  endpoints:
  - bearerTokenSecret:
      key: ""
    port: metrics
    relabelings:
    - action: labeldrop
      regex: (endpoint|instance|job|service)
  namespaceSelector: {}
  selector:
    matchLabels:
      coherenceCluster: SockShop
      coherenceComponent: coherence-service
      coherenceDeployment: catalog-coh
      coherencePort: metrics
      coherenceRole: Catalog

Here are the labels on the corresponding catalog-coh-metrics service.

kind: Service
metadata:
  labels:
    coherenceCluster: SockShop
    coherenceComponent: coherence-service
    coherenceDeployment: catalog-coh
    coherencePort: metrics
    coherenceRole: Catalog
spec:
  ports:
  - name: metrics
    port: 9612
    protocol: TCP
    targetPort: 9612
  ....

Istio integration

Verrazzano ensures that Coherence clusters are not included in an Istio mesh, even if the namespace has the istio-injection: enabled label. This is done by adding the sidecar.istio.io/inject: "false" annotation to the Coherence resource, resulting in Coherence pods being created with that label. However, other application components in the mesh using mutual TLS authentication (mTLS) may need to communicate with Coherence. To handle this case, Verrazzano automatically creates an Istio DestinationRule to disable TLS for the Coherence port. This policy disables mTLS for port 9000, which happens to be used as a Coherence extend port for Bob’s Books.

  trafficPolicy:
    portLevelSettings:
    - port:
        number: 9000
      tls: {}
   ...

Currently, port 9000 is the only port where TLS is disabled, so you need to use this as the Coherence extend port if other components in the mesh access Coherence over the extend protocol.

Verrazzano Enterprise Container Platform – Welcome to Verrazzano

Docs: About Applications in Verrazzano

Application configurations

Components

Deployment

Docs: Analyze Verrazzano Clusters

Set up the CLI tool

Analyze clusters

Analyze cluster snapshots

Use the vz analyze tool to analyze multiple snapshots

Usage information

Available options

Available flags

Docs: Application Security

Keycloak

NOTE

Network security

NOTE

Pod security

Specify a non-root user in the container image

Specify security settings for the Pod

Helidon pod security

Pod security for ContainerizedWorkload applications

Pod security for applications using standard Kubernetes resources

Docs: Bob's Books

Before you begin

Overview

Deploy the application

NOTE

Access the applications using the WebLogic Server Administration Console

NOTE

Access bobs-bookstore

Access bobbys-front-end

Verify the deployed application

Undeploy the application

Docs: Cluster API

NOTE

Docs: Create a Kubernetes Deployment and Service

Docs: Customize AuthProxy

Docs: Install Multicluster Verrazzano

Prerequisites

NOTE

Set up the admin cluster

Set up the managed cluster

Register the managed cluster

Minimal managed cluster configurations

Minimum configuration

Observability using Prometheus

Observability using Thanos

Docs: Install With CLI

Prerequisites

NOTE

Prepare for the installation

Perform the installation

Install Verrazzano

Next steps

Docs: Install With kubectl

Prerequisites

NOTE

Prepare for the installation

Install the Verrazzano platform operator

Perform the installation

Install Verrazzano

Next steps

Docs: Installed Software

Docs: Logging Components in Verrazzano

Docs: Multicluster and Verrazzano Project

clusters.verrazzano.io/v1alpha1

MultiClusterApplicationConfiguration

MultiClusterComponent

MultiClusterConfigMap

MultiClusterSecret

VerrazzanoProject

ApplicationConfigurationTemplate

Cluster

ClusterLevelStatus

ComponentTemplate

Condition

ConditionType (string alias)

ConfigMapTemplate

ConditionType (`string` alias)

StateType (`string` alias)