⚠️ CRITICAL WARNING: ingress-nginx with K3s

The Problem

When using K3s with the built-in svclb (ServiceLB), DO NOT add hostNetwork: true to the ingress-nginx controller.

Why This Happens

K3s automatically deploys svclb-* pods when you create a LoadBalancer service. These svclb pods:

Use hostNetwork: true by design
Bind to ports 80 and 443 on the host

If you also add hostNetwork: true to ingress-nginx-controller:

Both svclb pods AND ingress-nginx pods try to bind to ports 80/443
This causes bind conflicts
External traffic cannot reach the ingress controller
You'll see "connection refused" or routing failures

The Solution

Remove hostNetwork: true from ingress-nginx-controller DaemonSet/Deployment.

# Check current config
kubectl get ds -n ingress-nginx ingress-nginx-controller -o yaml | grep -A5 hostNetwork

# If hostNetwork is true, patch to remove it:
kubectl patch ds -n ingress-nginx ingress-nginx-controller --type='json' -p='[{"op":"remove","path":"/spec/template/spec/hostNetwork"}]'

# Restart pods
kubectl rollout restart ds -n ingress-nginx ingress-nginx-controller

How K3s svclb Works

External Request (port 80/443)
        │
        ▼
┌───────────────────┐
│  svclb-* pod      │  ← hostNetwork: true, binds to 80/443
│  (K3s ServiceLB)  │
└─────────┬─────────┘
          │
          ▼ forwards to service
┌───────────────────────────────┐
│  ingress-nginx-controller svc │  (LoadBalancer type)
│  ClusterIP:10.43.x.x:80/443   │
└─────────┬─────────────────────┘
          │
          ▼
┌───────────────────────────────┐
│  ingress-nginx-controller pod │  ← NO hostNetwork needed
│  Listens on container ports   │
└───────────────────────────────┘

Verification

# Check svclb pods are running
kubectl get pods -A | grep svclb

# Should see:
# kube-system   svclb-ingress-nginx-controller-xxxxx   Running

# Verify ports are accessible
curl -I http://SERVER_IP

# Should get HTTP response from ingress-nginx

If you use NodePort instead of LoadBalancer, svclb pods won't be created
If you disable K3s ServiceLB and use MetalLB, different rules apply
Cloud providers with real LoadBalancers also don't need hostNetwork

Date: 2025-01-18 Issue discovered while deploying FourSat infrastructure

2.7 KiB Raw Blame History