masoodafar-web
83 lines
2.7 KiB
Markdown
83 lines
2.7 KiB
Markdown
# ⚠️ CRITICAL WARNING: ingress-nginx with K3s
|
|
|
|
## The Problem
|
|
|
|
When using **K3s** with the built-in **svclb (ServiceLB)**, DO NOT add `hostNetwork: true` to the ingress-nginx controller.
|
|
|
|
## Why This Happens
|
|
|
|
K3s automatically deploys `svclb-*` pods when you create a `LoadBalancer` service. These svclb pods:
|
|
- Use `hostNetwork: true` by design
|
|
- Bind to ports 80 and 443 on the host
|
|
|
|
If you also add `hostNetwork: true` to ingress-nginx-controller:
|
|
- **Both** svclb pods AND ingress-nginx pods try to bind to ports 80/443
|
|
- This causes bind conflicts
|
|
- External traffic cannot reach the ingress controller
|
|
- You'll see "connection refused" or routing failures
|
|
|
|
## The Solution
|
|
|
|
**Remove `hostNetwork: true`** from ingress-nginx-controller DaemonSet/Deployment.
|
|
|
|
```bash
|
|
# Check current config
|
|
kubectl get ds -n ingress-nginx ingress-nginx-controller -o yaml | grep -A5 hostNetwork
|
|
|
|
# If hostNetwork is true, patch to remove it:
|
|
kubectl patch ds -n ingress-nginx ingress-nginx-controller --type='json' -p='[{"op":"remove","path":"/spec/template/spec/hostNetwork"}]'
|
|
|
|
# Restart pods
|
|
kubectl rollout restart ds -n ingress-nginx ingress-nginx-controller
|
|
```
|
|
|
|
## How K3s svclb Works
|
|
|
|
```
|
|
External Request (port 80/443)
|
|
│
|
|
▼
|
|
┌───────────────────┐
|
|
│ svclb-* pod │ ← hostNetwork: true, binds to 80/443
|
|
│ (K3s ServiceLB) │
|
|
└─────────┬─────────┘
|
|
│
|
|
▼ forwards to service
|
|
┌───────────────────────────────┐
|
|
│ ingress-nginx-controller svc │ (LoadBalancer type)
|
|
│ ClusterIP:10.43.x.x:80/443 │
|
|
└─────────┬─────────────────────┘
|
|
│
|
|
▼
|
|
┌───────────────────────────────┐
|
|
│ ingress-nginx-controller pod │ ← NO hostNetwork needed
|
|
│ Listens on container ports │
|
|
└───────────────────────────────┘
|
|
```
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
# Check svclb pods are running
|
|
kubectl get pods -A | grep svclb
|
|
|
|
# Should see:
|
|
# kube-system svclb-ingress-nginx-controller-xxxxx Running
|
|
|
|
# Verify ports are accessible
|
|
curl -I http://SERVER_IP
|
|
|
|
# Should get HTTP response from ingress-nginx
|
|
```
|
|
|
|
## Related Issues
|
|
|
|
- If you use `NodePort` instead of `LoadBalancer`, svclb pods won't be created
|
|
- If you disable K3s ServiceLB and use MetalLB, different rules apply
|
|
- Cloud providers with real LoadBalancers also don't need hostNetwork
|
|
|
|
---
|
|
|
|
*Date: 2025-01-18*
|
|
*Issue discovered while deploying FourSat infrastructure*
|