Verrazzano Enterprise Container Platform – Platform Setup

Docs: Prepare an Oracle Cloud Native Environment Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Install Oracle Cloud Native Environment

Deploy Oracle Cloud Native Environment with the Kubernetes module, following instructions from Oracle Cloud Native Environment: Getting Started.

Install a Kubernetes network load balancer implementation, such as OCI-CCM or MetalLB.
Install a Container Storage Interface Driver, such as OCI-CCM or Gluster.

Notes

The oci-ccm module does not elect a default StorageClass or configure policies for the CSIDrivers that it installs. A reasonable choice is the oci-bv StorageClass with its CSIDriver configured with the File group policy.
```
kubectl patch sc oci-bv -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: blockvolume.csi.oraclecloud.com
spec:
  fsGroupPolicy: File
EOF
```
Unless explicitly configured, the externalip-validation-webhook-service defaults to blocking all external IP addresses in the cluster, which causes the Verrazzano installation to fail because an IP address cannot be assigned to an ingress controller. When this situation occurs, the Verrazzano platform operator logs will contain a message similar to this:
```
admission webhook "validate-externalip.webhook.svc" denied the request: spec.externalIPs:
    Invalid value: "<external IP address>": externalIP specified is not allowed to use
```
To avoid this error, either disable the externalip-validation-webhook-service or configure the service with your load balancer IP addresses prior to installing Verrazzano. For more information, see Enabling Access to all externalIPs.

Examples

Oracle Cloud Infrastructure

The following is an example of Oracle Cloud Infrastructure that can be used to evaluate Verrazzano installed on Oracle Cloud Native Environment. If other environments are used, the capacity and configuration should be similar.

You can use the VCN Wizard of the Oracle Cloud Infrastructure Console to automatically create most of the described network infrastructure. Additional security lists and rules, as detailed in the following sections, need to be added manually. All Classless Inter-Domain Routing (CIDR) values provided are examples and can be customized as required.

Virtual Cloud Network (for example, CIDR 10.0.0.0/16)

Public Subnet for Load Balancer (for example, CIDR 10.0.0.0/24)

Security List / Ingress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type & Code	Description
No	`0.0.0.0/0`	ICMP			3, 4	ICMP errors
No	`10.0.0.0/16`	ICMP			3	ICMP errors
No	`0.0.0.0/0`	TCP	All	22		SSH
No	`0.0.0.0/0`	TCP	All	443		HTTPS load balancer

Security List / Egress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Description
No	`10.0.1.0/24`	TCP	All	22	SSH
No	`10.0.1.0/24`	TCP	All	31443	HTTPS load balancer
No	`10.0.1.0/24`	TCP	All	32443	HTTPS load balancer

Private Subnet for Kubernetes Cluster (for example, CIDR 10.0.1.0/24)

Security List / Ingress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type & Code	Description
No	`0.0.0.0/0`	ICMP			3, 4	ICMP errors
No	`10.0.0.0/16`	ICMP			3	ICMP errors
No	`10.0.0.0/16`	TCP	All	22		SSH
No	`10.0.0.0/24`	TCP	All	31443		HTTPS load balancer
No	`10.0.0.0/24`	TCP	All	32443		HTTPS load balancer
No	`10.0.1.0/24`	TCP	All	2379-2380		Kubernetes etcd
No	`10.0.1.0/24`	TCP	All	6443		Kubernetes API Server
No	`10.0.1.0/24`	TCP	All	6446		MySQL
No	`10.0.1.0/24`	TCP	All	8090-8091		Oracle Cloud Native Environment Platform Agent
No	`10.0.1.0/24`	UDP	All	8472		Flannel
No	`10.0.1.0/24`	TCP	All	10250-10255		Kubernetes Kublet

Security List / Egress Rules

Stateless	Destination	Protocol	Source Ports	Destination Ports	Type and Code	Description
No	`10.0.0.0/16`	TCP				All egress traffic

DHCP Options

DNS Type
Internet and VCN Resolver

Route Tables

Public Subnet Route Table Rules

Destination	Target
`0.0.0.0/0`	Internet Gateway

Private Subnet Route Table Rules

Destination	Target
`0.0.0.0/0`	NAT Gateway
All Oracle Cloud Infrastructure Services	Service Gateway

Compute Instances

The following compute resources adhere to the guidelines provided in Oracle Cloud Native Environment: Getting Started. The attributes indicated (for example, Subnet, RAM, Shape, and Image) are recommendations that have been tested. Other values can be used if required.

Role	Subnet	Suggested RAM	Compatible VM Shape	Compatible VM Image
SSH Jump Host	Public	8 GB	VM.Standard3.Flex	Oracle Linux 7.9
Oracle Cloud Native Environment Operator Host	Private	16 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Control Plane Node	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Worker Node 1	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Worker Node 2	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9
Kubernetes Worker Node 3	Private	32 GB	VM.Standard3.Flex	Oracle Linux 7.9

Next steps

To continue, see the Installation Guide.

Docs: Prepare an Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the Oracle Cloud Infrastructure installation

Create the OKE cluster using the Oracle Cloud Infrastructure Console or by some other means.
Follow the instructions provided by OKE to download the Kubernetes configuration file for your cluster, and set the following ENV variable:
```
$ export KUBECONFIG=<path to valid Kubernetes config>
```
Optional, if your organization requires the use of a private registry to the Docker images installed by Verrazzano, see Use a Private Registry.

NOTE: Verrazzano can create network policies that can be used to limit the ports and protocols that pods use for network communication. Network policies provide additional security but they are enforced only if you install a Kubernetes Container Network Interface (CNI) plug-in that enforces them, such as Calico. For an example on OKE, see Installing Calico and Setting Up Network Policies.

NOTE

OKE clusters using only the flannel overlay network is supported. OCI VCN-Native Pod Networking will not work.

Next steps

To continue, see the Installation Guide.

Docs: Prepare a Generic Kubernetes Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prepare for the generic install

Verrazzano requires that your Kubernetes cluster provides an implementation of network load balancers (Services of type LoadBalancer) for a production environment. If your generic Kubernetes implementation provides this feature, then you can use a default configuration of the Verrazzano custom resource with no customizations and follow the Installation Guide.

NOTE

Remember to not overlap network Classless Inter-Domain Routing (CIDR) blocks when designing and implementing your Kubernetes cluster; proper routing relies on that.

You can install a load balancer, such as MetalLB. This setup requires knowledge of networking both inside and outside your Kubernetes cluster. This would include specifics of your Container Network Interface (CNI) implementation, IP address allocation schemes, and routing that goes beyond the scope of this documentation. For a Kind implementation, see Install and configure MetalLB.

It is possible to use a Kubernetes Service of type NodePort to test aspects of Verrazzano. This requires a good working knowledge of networking and has limited use cases.

Next steps

To continue, see the Installation Guide.

Docs: Prepare a Kind Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Kind is a tool for running local Kubernetes clusters using Docker container “nodes”. Follow these instructions to prepare a Kind cluster for running Verrazzano.

NOTE

Kind is not recommended for use on macOS and Windows because the Docker network is not directly exposed to the host.

Prerequisites

Install Docker
Install Kind

Prepare the Kind cluster

To prepare the Kind cluster for use with Verrazzano, you must create the cluster and then install and configure MetalLB in that cluster.

You can create the Kind cluster in two ways: with or without image caching; image caching can speed up your installation time.

Create a Kind cluster

Kind images are prebuilt for each release. To find images suitable for a given release, check the release notes for your Kind version (check with kind version). There you’ll find a complete listing of images created for a Kind release.

The following example references a Kubernetes v1.26-based image built for Kind v0.20.0. Replace that image with one suitable for the Kind release you are using. For the supported Kubernetes versions, see the listing here.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.26@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
EOF

Create a Kind cluster with image caching

While developing or experimenting with Verrazzano, you might destroy and re-create your Kind cluster multiple times. To speed up Verrazzano installation, follow these steps to ensure that the image cache used by containerd inside a Kind cluster, is preserved across clusters. Subsequent installations will be faster because they will not need to pull the images again.

1. Create a named Docker volume that will be used for the image cache and note its mountPoint path. In this example, the volume is named containerd.

$ docker volume create containerd

$ docker volume inspect containerd

# Sample output
{
    "CreatedAt": "2021-01-11T16:27:47Z",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/containerd/_data",
    "Name": "containerd",
    "Options": {},
    "Scope": "local"
}

2. Specify the mountPoint path obtained, as the hostPath under extraMounts in your Kind configuration file, with a containerPath of /var/lib/containerd, which is the default containerd image caching location inside the Kind container. An example of the modified Kind configuration is shown in the following create cluster command.

$ kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.26@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
    kubeadmConfigPatches:
      - |
        kind: ClusterConfiguration
        apiServer:
          extraArgs:
            "service-account-issuer": "kubernetes.default.svc"
            "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
    extraMounts:
      - hostPath: /var/lib/docker/volumes/containerd/_data
        containerPath: /var/lib/containerd #This is the location of the image cache inside the Kind container
EOF

NOTE: When using a private container registry or experiencing rate-limiting of container image pulls, refer to the Kind documentation.

Install and configure MetalLB

By default, Kind does not provide an implementation of network load balancers (Services of type LoadBalancer). MetalLB offers a network load balancer implementation.

To install MetalLB:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/namespace.yaml
$ kubectl create secret generic \
    -n metallb-system memberlist \
    --from-literal=secretkey="$(openssl rand -base64 128)"
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.11.0/manifests/metallb.yaml

NOTE: When using MetalLB on Kubernetes v1.26, install MetalLB v0.11.0, which you can do with manifest files or a Helm chart.

Wait for MetalLB to be ready, as shown:

$ kubectl get pods -n metallb-system -L app=metallb
NAME                         READY   STATUS    RESTARTS       AGE   APP=METALLB
controller-d9d8c78b6-hhplg   1/1     Running   1 (3d4h ago)   9d    
speaker-4lqpg                1/1     Running   1 (3d4h ago)   9d

For further details, see the MetalLB installation guide.

MetalLB is idle until configured. Configure MetalLB in Layer 2 mode and give it control over a range of IP addresses in the kind Docker network. In versions v0.7.0 and earlier, Kind uses Docker’s default bridge network; in versions v0.8.0 and later, it creates its own bridge network in Kind.

To determine the subnet of the kind Docker network in Kind v0.8.0 and later:

$ docker inspect kind | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.18.0.0/16

To determine the subnet of the kind Docker network in Kind v0.7.0 and earlier:

$ docker inspect bridge | jq '.[0].IPAM.Config[0].Subnet' -r

# Sample output
172.17.0.0/16

For use by MetalLB, assign a range of IP addresses at the end of the kind network’s subnet CIDR range.

$ kubectl apply -f - <<-EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: my-ip-space
      protocol: layer2
      addresses:
      - 172.18.0.230-172.18.0.250
EOF

Next steps

To continue, see the Installation Guide.