Verrazzano Enterprise Container Platform – Prepare an Oracle Cloud Native Environment Cluster

Docs: Configure a VCN for OCNE

Mon, 01 Jan 0001 00:00:00 +0000

Before you can create Oracle Cloud Native Environment (OCNE) clusters on Oracle Cloud Infrastructure (OCI), you’ll need to configure a virtual cloud network (VCN) in your OCI compartment. VCNs are software-defined networks that manage access to your cloud resources.

See Networking Overview in the OCI documentation for more information.

You can use the VCN Wizard in the OCI Console to automatically create most of the required network infrastructure. Additional subnets and security rules (described below) must be added manually.

Within your VCN, you’ll need:

Subnets (with security rules)
Gateways
Route tables

NOTE

In addition to the specifications listed, make sure that the VCN is configured to accept the ports and protocols required by Kubernetes. See Ports and Protocols in the Kubernetes documentation for more information.

Subnets

Subnets are subdivisions within a VCN that help to organize configuration settings. All instances within a subnet use the same route table, security lists, and DHCP options. Subnets can be either public or private. For an OCNE cluster, you’ll need both public and private subnets, with four subnets in total.

See Overview of VCNs and Subnets in the OCI documentation for more information.

Each subnet requires its own set of security rules that establish rules for virtual firewalls. These ingress and egress rules specify the types of traffic (protocol and port) that are allowed in and out of the instances.

See Security Rules in the OCI documentation for more information.

NOTE: You can use either Network Security Groups (NSGs) or security lists to add security rules to your VCN. We recommend using NSGs whenever possible. See Comparison of Security Lists and Network Security Groups in the OCI documentation.

Subnet 1: control plane endpoint

A public subnet for the control plane endpoint that houses an OCI load balancer. The load balancer acts as a reverse proxy for the Kubernetes API server.

In this subnet, create security rules that cover the following traffic:

Egress: control plane traffic
Ingress: external access to the Kubernetes API endpoint
Ingress: ICMP path discovery

Subnet 2: control plane nodes

A private subnet that houses the control plane nodes that run Kubernetes control plane components such as the API Server and the control plane pods.

In this subnet, create security rules that cover the following traffic:

Egress: node internet access
Ingress: east-west traffic, originating from within the VCN
Ingress: control plane endpoint to control plane node access on API endpoint
Ingress: worker nodes to control plane node access on API endpoint
Ingress: ETCD client and peer
Ingress: SSH traffic
Ingress: control plane to control plane kubelet communication
Ingress:
Ingress: Calico rules for control plane and worker nodes for
- BGP
- IP-in-IP

Subnet 3: service load balancers

A public subnet that houses the service load balancers.

In this subnet, create security rules that cover the following traffic:

Egress: service load balancer to NodePort on worker nodes
Ingress: ICMP path discovery
Ingress: HTTP and HTTPS traffic

Subnet 4: worker nodes

A private subnet that houses the worker nodes.

In this subnet, create security rules that cover the following traffic:

Egress: node internet access
Ingress: east-west traffic, originating from within the VCN
Ingress: SSH traffic
Ingress: ICMP path discovery
Ingress: control plane to kubelet on worker nodes
Ingress: worker node to worker node
Ingress: Calico rules for control plane and worker nodes for
- BGP
- IP-in-IP
Ingress: worker nodes to default NodePort ingress

Gateways

Gateways control access from your VCN to other networks. You’ll need to configure three different types of gateways:

You may need to perform some additional configuration to expose the VCN’s subnets directly to the internet. See Access to the Internet in the OCI documentation for details.

Route tables

Route tables send traffic out of the VCN (for example, to the internet, to your on-premises network, or to a peered VCN) using rules that are similar to traditional network route rules.

See VCN Route Tables in the OCI documentation for more information.

For OCNE clusters, you’ll need to create two route tables:

A route table for public subnets that will route stateful traffic to and from the internet gateway. Assign this route table to both public subnets.
A route table for private subnets that will route stateful traffic to and from the NAT and service gateways. Assign this route table to both private subnets.

Docs: Configure NFS Storage

Mon, 01 Jan 0001 00:00:00 +0000

Complete the following steps to configure NFS storage in an Oracle Cloud Native Environment.

Create an OLCNE cluster. See OLCNE cluster.

The cluster must have at least 3 worker nodes.
Create an NFS server. For an example that uses an NFS server on Oracle Linux, see Create an NFS server on Oracle Linux.

a. Install the NFS utility package on the server and client instances.
```
$ sudo dnf install -y nfs-utils
```
b. Create a directory for your shared files. Make sure that the server does not have root ownership.

c. Define the shared directory in /etc/exports with the correct permissions. Make sure to disable root squashing.
```
$ <path to directory> <ip-address/subnet-mask>(rw,sync,no_root_squash,no_subtree_check)
```
d. Set the firewall to allow NFS traffic.
```
$ sudo firewall-cmd --permanent --zone=public --add-service=nfs
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all
```
e. Enable and start the NFS service.
```
$ sudo systemctl enable --now nfs-server
```
Deploy an NFS provisioner to your cluster.

a. Install an NFS client provisioner of your choice. For an example, see Kubernetes NFS Subdir External Provisioner.

b. Add the required Helm repo.
```
$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
```
c. Install the provisioner. Set your storage class as a default and create a service account.
```
$ helm install nfs-test \
   --set nfs.server=<server ip address> \
   --set nfs.path=<path> \
   --set storageClass.name=<name> \
   --set storageClass.defaultClass=true,rbac.create=true \
   --set storageClass.provisionerName=nfsclientprov/nfs \
   --set serviceAccount.create=true \
   --set serviceAccount.name=nfs-svc-acc-nfs nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
```
d. Only one storage class should be listed as the default. If required, edit the other storage classes and delete the following annotation:
```
$ storageclass.kubernetes.io/is-default-class: "true"
```

Docs: Configure Access to a Private Registry

Mon, 01 Jan 0001 00:00:00 +0000

A private Docker registry is called an insecure registry when it is configured for access using a self-signed certificate or over an unencrypted HTTP connection. For example, for the Oracle Cloud Native Environment platform, insecure registries must be configured in /etc/containers/registries.conf as follows on the worker nodes:

 [registries]
    [registries.insecure]
      registries = ["insecure-registry-1:1001/registry1","insecure-registry-2:1001/registry2"]