Cilium 踩坑总结
测试环境
KVM - hayden@HaydenArchDesktop ~> virsh version
Compiled against library: libvirt 8.6.0
Using library: libvirt 8.6.0
Using API: QEMU 8.6.0
Running hypervisor: QEMU 7.0.0VM OS： root@fedora ~# cat /etc/os-release
NAME=”Fedora Linux”
VERSION=”36 (Thirty Six)”
ID=fedora
VERSION_ID=36KVM 虚拟机两台
- Master： hostname: fedora
- Node： hostname: knode1
kubernetes 版本： v1.24.3
kubernetes 安装方式： kubeadm
docker版本: docker://20.10.17
NOTE: (尝试使用Containerd， 但是翻车了， 控制平面的Pod启动不了， 所以放弃了，遂使用CRI-Dockerd，配置了Docker runtime， Containerd使用默认的参数无法正常的启动， 看起来即使真的升级到了1.24 迁移还是一个问题)
Kernel Version: 5.17.5-300.fc36.x86_64
Helm参数
如果是在KVM启动的虚拟机，可以通过这个安装参数来开启更多功能，但是受限于我的KVM虚拟网卡驱动不能attach xdp 程序， 所以。。。。xdp 加速无法启用，但是其他的高级特性均可开启， 集群状态正常。
helm upgrade -i cilium cilium/cilium \
--namespace kube-system \
--set tunnel=disabled \
--set autoDirectNodeRoutes=true \
--set loadBalancer.mode=dsr \
--set kubeProxyReplacement=strict \
--set enableIPv4Masquerade=false \
--set loadBalancer.algorithm=maglev \
--set devices=enp1s0 \
--set k8sServiceHost=192.168.31.100 \
--set k8sServicePort=6443 \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=trueCilium Cli
touch ./install_cilium_cli.sh
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}Hubble Cli
touch ./install_hubble_client.sh
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}Cilium Cli 安装插件命令
cilium upgrade # 原地升级
# 安装并直接替换kubeproxy
cilium install --version 1.14.1 --set kubeProxyReplacement=true --set=ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16" --set k8sServiceHost=kube3s.liarlee.site --set k8sServicePort=6443
# 升级
hayden@arch ~> cilium upgrade
🔮 Auto-detected Kubernetes kind: K3s
ℹ️ Using Cilium version 1.14.2
🔮 Auto-detected cluster name: default
# 看状态
cilium status
# 使用hubble
cilium hubble port-forward &
cilium hubble enable --ui
hubble observe --since=1m -t l7 -
特性以及状态检查
默认的安装完成之后开启特性如下：
- Kubeproxy Bypass
- Iptables Bypass
- LoadBalancer 算法： Meglav
- LoadBalancer 特性： DSR
- 报文Masquerade 封装： Disabled
- Hubble ： Enable
- 隧道封包： Disabled
下面是 CIlium Status的命令返回结果：
root@fedora ~# cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: OK
\__/¯¯\__/ ClusterMesh: disabled
\__/
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: hubble-relay Running: 1
hubble-ui Running: 1
cilium Running: 2
cilium-operator Running: 2
Cluster Pods: 14/14 managed by Cilium
Image versions hubble-ui quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1
cilium quay.io/cilium/cilium:v1.12.0@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade: 2
cilium-operator quay.io/cilium/operator-generic:v1.12.0@sha256:bb2a42eda766e5d4a87ee8a5433f089db81b72dd04acf6b59fcbb445a95f9410: 2
hubble-relay quay.io/cilium/hubble-relay:v1.12.0@sha256:ca8033ea8a3112d838f958862fa76c8d895e3c8d0f5590de849b91745af5ac4d: 1ds/cilium 中的命令返回结果：
root@fedora:/home/cilium# cilium status
KVStore: Ok Disabled
Kubernetes: Ok 1.24 (v1.24.3) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp1s0 192.168.31.100 (Direct Routing)]
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.12.0 (v1.12.0-9447cd1)
NodeMonitor: Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 3/254 allocated from 10.0.0.0/24,
BandwidthManager: Disabled
Host Routing: BPF
Masquerading: Disabled
Controller Status: 23/23 healthy
Proxy Status: OK, ip 10.0.0.78, 0 redirects active on ports 10000-20000
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 283/4095 (6.91%), Flows/s: 1.62 Metrics: Disabled
Encryption: Disabled
Cluster health: 2/2 reachable (2022-08-23T01:57:33Z)Cilium verbose的详细结果：
root@fedora:/home/cilium# cilium status --verbose
KVStore: Ok Disabled
Kubernetes: Ok 1.24 (v1.24.3) [linux/amd64]
Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement: Strict [enp1s0 192.168.31.100 (Direct Routing)]
Host firewall: Disabled
CNI Chaining: none
Cilium: Ok 1.12.0 (v1.12.0-9447cd1)
NodeMonitor: Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 3/254 allocated from 10.0.0.0/24,
Allocated addresses:
10.0.0.11 (kube-system/coredns-6d4b75cb6d-tbgrr)
10.0.0.201 (health)
10.0.0.78 (router)
BandwidthManager: Disabled
Host Routing: BPF
Masquerading: Disabled
Clock Source for BPF: ktime
Controller Status: 23/23 healthy
Name Last success Last error Count Message
cilium-health-ep 29s ago never 0 no error
dns-garbage-collector-job 34s ago never 0 no error
endpoint-1313-regeneration-recovery never never 0 no error
endpoint-2868-regeneration-recovery never never 0 no error
endpoint-2945-regeneration-recovery never never 0 no error
endpoint-gc 3m34s ago never 0 no error
ipcache-inject-labels 3m30s ago 3m33s ago 0 no error
k8s-heartbeat 4s ago never 0 no error
link-cache 15s ago never 0 no error
metricsmap-bpf-prom-sync 4s ago never 0 no error
resolve-identity-1313 3m29s ago never 0 no error
resolve-identity-2868 3m29s ago never 0 no error
resolve-identity-2945 3m30s ago never 0 no error
sync-endpoints-and-host-ips 30s ago never 0 no error
sync-lb-maps-with-k8s-services 3m30s ago never 0 no error
sync-node-with-ciliumnode (fedora) 3m32s ago 3m33s ago 0 no error
sync-policymap-1313 26s ago never 0 no error
sync-policymap-2868 26s ago never 0 no error
sync-policymap-2945 26s ago never 0 no error
sync-to-k8s-ciliumendpoint (1313) 9s ago never 0 no error
sync-to-k8s-ciliumendpoint (2868) 9s ago never 0 no error
sync-to-k8s-ciliumendpoint (2945) 0s ago never 0 no error
template-dir-watcher never never 0 no error
Proxy Status: OK, ip 10.0.0.78, 0 redirects active on ports 10000-20000
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 377/4095 (9.21%), Flows/s: 1.67 Metrics: Disabled
KubeProxyReplacement Details:
Status: Strict
Socket LB: Enabled
Socket LB Protocols: TCP, UDP
Devices: enp1s0 192.168.31.100 (Direct Routing)
Mode: DSR
Backend Selection: Maglev (Table Size: 16381)
Session Affinity: Enabled
Graceful Termination: Enabled
NAT46/64 Support: Disabled
XDP Acceleration: Disabled
Services:
- ClusterIP: Enabled
- NodePort: Enabled (Range: 30000-32767)
- LoadBalancer: Enabled
- externalIPs: Enabled
- HostPort: Enabled
BPF Maps: dynamic sizing: on (ratio: 0.002500)
Name Size
Non-TCP connection tracking 65536
TCP connection tracking 131072
Endpoint policy 65535
Events 4
IP cache 512000
IP masquerading agent 16384
IPv4 fragmentation 8192
IPv4 service 65536
IPv6 service 65536
IPv4 service backend 65536
IPv6 service backend 65536
IPv4 service reverse NAT 65536
IPv6 service reverse NAT 65536
Metrics 1024
NAT 131072
Neighbor table 131072
Global policy 16384
Per endpoint policy 65536
Session affinity 65536
Signal 4
Sockmap 65535
Sock reverse NAT 65536
Tunnel 65536
Encryption: Disabled
Cluster health: 2/2 reachable (2022-08-23T01:59:33Z)
Name IP Node Endpoints
fedora (localhost) 192.168.31.100 reachable reachable
knode1 192.168.31.101 reachable reachablecilium-health status –probe 之后的结果， 多个节点之间的联通性正常：
root@fedora:/home/cilium# cilium-health status --probe
Probe time: 2022-08-23T02:01:04Z
Nodes:
fedora (localhost):
Host connectivity to 192.168.31.100:
ICMP to stack: OK, RTT=140.825µs
HTTP to agent: OK, RTT=199.54µs
Endpoint connectivity to 10.0.0.201:
ICMP to stack: OK, RTT=128.2µs
HTTP to agent: OK, RTT=263.034µs
knode1:
Host connectivity to 192.168.31.101:
ICMP to stack: OK, RTT=235.981µs
HTTP to agent: OK, RTT=330.706µs
Endpoint connectivity to 10.0.1.251:
ICMP to stack: OK, RTT=177.807µs
HTTP to agent: OK, RTT=275.869µs没有执行Cilium connectivity Test， 这个测试和 1.1.1.1:443 的测试在基本上每次都是失败的。目前原因还不清楚， 没调查，但是集群内的Pod访问外部的网络是完全正常的。
基于AWS EC2 测试的结果（无法关闭 Masquerade 以及 绕过 Iptables）， 与flannel相比，Cilium当前的性能还是少少差一点点，但是跨越公网访问的稳定性更强，HTTP完全没有Failed请求。
没有开启XDP， 原因是我使用的是KVM的虚拟机 ，使用的驱动程序是VirtIO， 看起来这个驱动在我当前版本的虚拟化里面可能需要做一些特殊的设置才能开启， 我没有继续进行测试。
目前也没法对性能做任何的对比测试， 没有条件启动两个集群做对比，可以想见因为底层的响应时间极快，所以没有办法看出差距； 要么就是资源的抢占导致本身的测试结果不够稳定。
我目前还在 EC2 以及 EKS 中调试：
EKS：AWS CNI替换掉之后有自己的问题， 不能开启完全的Iptables Bypass， 由于启用了ENI MODE 就会提供 EndpointRoute， 感觉这已经是BypassIptables， 但是没能验证，不确定。
EC2 部署的 Kubernetes： 集群不能开启HostRouting， 开启之后cilium-health status 无法完成对端节点上面Endpoint的检查， Connection Timeout。也不能完全work。
启用 Wireguard 节点间加密
官方文档: https://docs.cilium.io/en/stable/security/network/encryption/
- 启用 wireguard 节点间通信加密
~> cilium upgrade --version 1.15.4 --set kubeProxyReplacement=true --set bpf.masquerade=true --set encryption.enabled=true --set encryption.type=wireguard - 验证
hayden@arch ~> kubectl -n kube-system exec -ti ds/cilium -- bash Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init) root@haydenzgo:/home/cilium# cilium-dbg status | grep Encryption Encryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey: OvYVIjFwbWphY2DQl5MbJqd48qbGLCyhbqGJaqkz4yM=, Port: 51871, Peers: 1)] root@haydenzgo:/home/cilium#
其他资料
其他公司业务的配置以及测试。
https://www.ebpf.top/post/cilium-standalone-L4LB-XDP-zh/
按照Linux基金会的PPT中描述， KVM 虚拟化条件下也是可以开启XDP的，但是我只是测试， 就不折腾了， 我自己的PC 效果不一定明显。
https://lore.kernel.org/bpf/b41ab0f0-4537-74b5-d7c3-b20ce082bdd6@redhat.com/T/
启用 Bandwidth Manager 和 BBR
# 尝试添加了bbr 算法的选项， 使用如下命令进行变更：
cilium upgrade --version 1.14.5 --set kubeProxyReplacement=true --set=ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16" --set k8sServiceHost=kube3s.liarlee.site --set k8sServicePort=6443 --set bandwidthManager.enabled=true --set bandwidthManager.bbr=true
# 使用如下的命令进行确认：
kubectl -n kube-system exec ds/cilium -- cilium status | grep BandwidthManager本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 Liarlee's Notebook！
