It took me nearly two hours to get everything back up after booting into the new kernels tonight.
The issue revolves around some recently added ufw rules to improve security. Even though I have explicit rules permitting machines that need to see each others portmapper (rpcbind) to do so, they aren’t working and when ufw is enabled, none of the machines can see each others portmap, this breaks nfs and nis.
So I’ll have to do some further investigation as to why this is the case, but that was the cause of the long downtime.
This affected all eskimo.com services including our Fediverse services.