These are the build notes for a multi-master, multi-worker K8s cluster. The intent is to build from scratch a set of machines that provide HA for container access and control as well as HA and scaling for containers. Here's roughly what it might look like:
High Level Concepts
Persistent storage on masters using Gluster for replication (3-way mirror).
Multiple K8s master nodes for basic HA (odd number to better manage cluster splits).
Build from notes/documentation instead of others pre-built scripts (so that I actually learn it).
- No swap on K8s hosts – See kubeadm issue 53533. I think it's a bit dumb but it is what it is.
- Disk 0 is OS
- Disk 1 is a Gluster Brick (could do multiple smaller bricks here, but I don't see a reason this early in the project)
- Is the Gluster FS shared via NFS or Gluster client?
- Where does haproxyd/nginx/traefik run?
- What networks (stack, VLANs/IPs etc) are needed here?
- Node selector labels for deciding where containers run (workers vs masters) as I suspect running the reverse proxy on the masters might be preferable here
Debian 10.2 NetInstall - Masters
- 2 vCPU
- 2 GiB RAM
- 1x NIC
- 1x 32 GiB OS
- 1x 512 GiB Storage
Base OS Install Process
- Graphical Install
- English Language
- Location Australia
- American English keyboard
- Hostname - k8master01, k8master02, k8master03
- DNS domain name - as appropriate
- Root passwords set and recorded
- Local administrator created, password recorded
- Clock - Timezone, really - New South Wales
- Manual Partitioning
- Select SCSI1 (0,0,0) sda
- Create new partition table
- Create new 512MB partition, configure as an EFI System Partition (note - this is required because this is a Hyper-V Gen2 VM and boots UEFI, not BIOS) at the Beginning of the disk
- Create new partition for the rest of the disk, configure as Ext4 and mount as / (this should be the default)
- Commit the partitions
- Don't scan the CD image
- Add the local Debian mirror
- Disable popcon
- Uncheck the Debian Desktop Environments and Print Server - leave only SSH server and standard system utilities checked
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
The loopback network interface
iface lo inet loopback
The primary network interface
# iface eth0 inet dhcp
iface eth0 inet static
iface eth0 inet6 static
And install a bunch of daemons (AD join, Hyper-V support etc)
apt install hyperv-daemons curl apt-transport-https realmd adcli sssd ntp packagekit sssd-tools cifs-utils sudo dnsutils
That ended up being ~87 packages.
Create a configuration file for your SSSD domain, e.g. /etc/sssd/conf.d/ad.conf:
use_fully_qualified_names = False
fallback_homedir = /home/%d/%u
chpass_provider = ad
ad_hostname = hostname.ad.domain.net
dyndns_update = true
dyndns_refresh_interval = 43200
dyndns_update_ptr = true
dyndns_ttl = 3600
ad_gpo_access_control = disabled
Set permissions to 0x400 or it is ignored by sssd on startup.
Create the base folder for home directories:
# mkdir /home/ad.domain.net # chown root:root /home/ad.domain.net # chmod 755 /home/ad.domain.net/
Update /etc/pam.d/common-session, adding the second line as shown:
session required pam_unix.so session required pam_mkhomedir.so umask=077 skel=/etc/skel
Restart sssd to effect the changes
# systemctl restart sssd
Enable sudo for Domain Admins (and/or other groups as desired) - this example is very permissive, but it suits the environment:
# echo %domain\\\ admins ALL=\(ALL\) ALL > /etc/sudoers.d/domain-admins
Join the domain with realm:
# realm join -U Administrator ad.domain.net
According to the documentation at kubernetes.io, hosts using nftables are not compatible, so switch to iptables instead:
# ensure legacy binaries are installed sudo apt-get install -y iptables arptables ebtables # switch to legacy versions sudo update-alternatives --set iptables /usr/sbin/iptables-legacy sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy sudo update-alternatives --set arptables /usr/sbin/arptables-legacy sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy
Next we'll build the Gluster volume.
Side Note: If you build one VM and clone it on Hyper-V, use this script to change the UUID for the machine. K8s apparently needs unique DMI IDs for each node.