These are the build notes for a multi-master, multi-worker K8s cluster (or docker swarm). The intent is to build from scratch a set of machines that provide HA for container access and control as well as HA and scaling for containers. Here's roughly what it might look like:

High Level Concepts

Persistent storage on masters using Gluster for replication (3-way mirror).

Multiple K8s master nodes for basic HA (odd number to better manage cluster splits).

Build from notes/documentation instead of others pre-built scripts (so that I actually learn it).

Research Notes

  • No swap on K8s hosts – See kubeadm issue 53533. I think it's a bit dumb but it is what it is.
  • Disk 0 is OS
  • Disk 1 is a Gluster Brick (could do multiple smaller bricks here, but I don't see a reason this early in the project)
  • Is the Gluster FS shared via NFS or Gluster client?
  • Where does haproxyd/nginx/traefik run?
  • What networks (stack, VLANs/IPs etc) are needed here?
  • Node selector labels for deciding where containers run (workers vs masters) as I suspect running the reverse proxy on the masters might be preferable here

Debian 10.2 NetInstall - Masters

  • 2 vCPU
  • 2 GiB RAM
  • 1x NIC
  • 1x 32 GiB OS
  • 1x 512 GiB Storage

Base OS Install Process

  • Graphical Install 
  • English Language
  • Location Australia
  • American English keyboard
  • Hostname - k8master01, k8master02, k8master03
  • DNS domain name - as appropriate
  • Root passwords set and recorded
  • Local administrator created, password recorded
  • Clock - Timezone, really - New South Wales
  • Manual Partitioning
  • Select SCSI1 (0,0,0) sda
    • Create new partition table
    • Create new 512MB partition, configure as an EFI System Partition (note - this is required because this is a Hyper-V Gen2 VM and boots UEFI, not BIOS) at the beginning of the disk
    • Create new partition for the rest of the disk, configure as Ext4 and mount as / (this should be the default)
    • Commit the partitions
  • Don't scan the CD image
  • Add the local Debian mirror
  • Disable popcon
  • Uncheck the Debian Desktop Environments and Print Server - leave only SSH server and standard system utilities checked

Post-Install Configuration

Edit /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
The loopback network interface
auto lo
iface lo inet loopback
The primary network interface
allow-hotplug eth0
# iface eth0 inet dhcp
iface eth0 inet static
iface eth0 inet6 static
   address 2001:2662:9da5:ae60::221/64
   gateway 2001:2662:9da5:ae60::1

And /etc/resolv.conf:

domain ad.domain.net
search ad.domain.net

And install a bunch of daemons (AD join, Hyper-V support etc)

# apt install hyperv-daemons curl apt-transport-https realmd adcli sssd ntp packagekit sssd-tools cifs-utils sudo dnsutils

That ended up being ~87 packages.

Create a configuration file for your SSSD domain, e.g. /etc/sssd/conf.d/ad.conf:

use_fully_qualified_names = False
fallback_homedir = /home/%d/%u
chpass_provider = ad
ad_hostname = hostname.ad.domain.net
dyndns_update = true
dyndns_refresh_interval = 43200
dyndns_update_ptr = true
dyndns_ttl = 3600
ad_gpo_access_control = disabled

Set permissions to o400 (- r - - - - - - - -) or it is ignored by sssd on startup.

Create the base folder for home directories:

# mkdir /home/ad.domain.net
# chown root:root /home/ad.domain.net
# chmod 755 /home/ad.domain.net/

Update /etc/pam.d/common-session, adding the second line as shown:

session required pam_unix.so
session required pam_mkhomedir.so umask=077 skel=/etc/skel

Restart sssd to effect the changes

# systemctl restart sssd

Enable sudo for Domain Admins (and/or other groups as desired) - this example is very permissive, but it suits the environment:

# echo %domain\\\ admins ALL=\(ALL\) ALL > /etc/sudoers.d/domain-admins

Join the domain with realm:

 # realm join -U Administrator ad.domain.net

According to the documentation at kubernetes.io, hosts using nftables are not compatible, so switch to iptables instead:

# ensure legacy binaries are installed
sudo apt-get install -y iptables arptables ebtables

# switch to legacy versions
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

Next we'll build the Gluster volume.

Side Note: If you build one VM and clone it on Hyper-V, use this script to change the UUID for the machine. K8s apparently needs unique DMI IDs for each node.