HA installation on K8s

Highly Available Installation (Kubernetes)Copied!

Netmaker comes with a Helm chart to deploy with High Availability on Kubernetes:

helm repo add netmaker https://gravitl.github.io/netmaker-helm/
helm repo update

RequirementsCopied!

To run HA Netmaker on Kubernetes, your cluster must have the following:

• RWO and RWX Storage Classes

• An Ingress Controller and valid TLS certificates

  This chart can currently generate ingress for: - Nginx Ingress + LetsEncrypt/Cert-Manager

  To generate automatically, make sure one of the two is configured for your cluster

• Ability to set up ingress route for Secure Web Sockets

  Nginx Ingress supports Secure Web Sockets (WSS) by default. If you are not using Nginx Ingress, you must route external traffic from broker.domain to the MQTT service, and provide valid TLS certificates.

  One option is to set up a Load Balancer which routes broker.domain:443 to the MQTT service on port 8883.

  We do not provide guidance beyond this, and recommend using an Ingress Controller that supports websockets.

Furthermore, the chart will by default install and use a postgresql cluster as its datastore:

Example Installations:Copied!

An annotated install command:

helm install netmaker/netmaker --generate-name \ # generate a random id for the deploy
--set baseDomain=nm.example.com \ # the base wildcard domain to use for the netmaker api/dashboard/mq ingress
--set server.replicas=3 \ # number of server replicas to deploy (3 by default)
--set ingress.enabled=true \ # deploy ingress automatically (requires nginx and cert-manager + letsencrypt)
--set ingress.kubernetes.io/ingress.class=nginx \ # ingress class to use
--set ingress.cert-manager.io/cluster-issuer=letsencrypt-prod \ # LetsEncrypt certificate issuer to use
--set postgresql-ha.postgresql.replicaCount=2 \ # number of DB replicas to deploy (default 2)

The below command will install netmaker with two server replicas, a coredns server, and ingress with routes of api.nm.example.com, grpc.nm.example.com, and dashboard.nm.example.com. CoreDNS will be reachable at 10.245.75.75, and will use NFS to share a volume with Netmaker (to configure dns entries).

helm install netmaker/netmaker --generate-name --set baseDomain=nm.example.com \
--set replicas=2 --set ingress.enabled=true --set dns.enabled=true \
--set dns.clusterIP=10.245.75.75 --set dns.RWX.storageClassName=nfs \
--set ingress.className=nginx

The below command will install netmaker with three server replicas (the default), no coredns, and ingress with routes of api.netmaker.example.com, grpc.netmaker.example.com, and dashboard.netmaker.example.com. There will be one UI replica instead of two and one database instance instead of two. Traefik will look for a ClusterIssuer named “le-prod-2” to get valid certificates for the ingress.

helm3 install netmaker/netmaker --generate-name \
--set baseDomain=netmaker.example.com --set postgresql-ha.postgresql.replicaCount=1 \
--set ui.replicas=1 --set ingress.enabled=true \
--set ingress.tls.issuerName=le-prod-2 --set ingress.className=traefik

Recommended Settings:Copied!

This install has some notable exceptions: Ingress must be configured on your cluster, with cluster issuer for TLS certs DNS will be disabled

Below, we discuss the considerations for Ingress, Kernel WireGuard, and DNS.

MQCopied!

The MQ Broker is deployed either with Ingress (Nginx ) preconfigured, or without. If you are using an ingress controller other than Nginx, Netmaker’s MQTT will not be complete. “broker.domain” must reach the MQTT service at port 8883 over WSS (Secure Web Sockets).

IngressCopied!

To run HA Netmaker, you must have ingress installed and enabled on your cluster with valid TLS certificates (not self-signed). If you are running Nginx as your Ingress Controller and LetsEncrypt for TLS certificate management, you can run the helm install with the following settings:

• –set ingress.enabled=true

• –set ingress.annotations.cert-manager.io/cluster-issuer=<your LE issuer name>

If you are not using Nginx and LetsEncrypt, we recommend leaving ingress.enabled=false (default), and then manually creating the ingress objects post-install. You will need three ingress objects with TLS:

• dashboard.<baseDomain>

• api.<baseDomain>

• broker.<baseDomain>

There are some example ingress objects in the kube/example folder.

DNSCopied!

By Default, the helm chart will deploy without DNS enabled. To enable DNS, specify with:

• –set dns.enabled=true

This will require specifying a RWX storage class, e.g.:

• –set dns.RWX.storageClassName=nfs

This will also require specifying a service address for DNS. Choose a valid ipv4 address from the service IP CIDR for your cluster, e.g.:

• –set dns.clusterIP=10.245.69.69

This address will only be reachable from hosts that have access to the cluster service CIDR. It is only designed for use cases related to k8s. If you want a more general-use Netmaker server on Kubernetes for use cases outside of k8s, you will need to do one of the following: bind the CoreDNS service to port 53 on one of your worker nodes and set the COREDNS_ADDRESS equal to the public IP of the worker node. Create a private Network with Netmaker and set the COREDNS_ADDRESS equal to the private address of the host running CoreDNS. For this, CoreDNS will need a node selector and will ideally run on the same host as one of the Netmaker server instances.

ValuesCopied!

To view all options for the chart, please visit the README in the netmaker-helm chart repo here .