Aben Blog

Introduction#

Recently, I found another way to have fun from the article "Deploying k3s Cluster Across Cloud Vendors", which inspired me to write this article. With the recent Double 11 promotions from major cloud service providers, Tencent Cloud is again offering small instances for just a few bucks for three years. The question arises: each cloud service provider can only purchase one instance at the promotional price, and ultimately our cloud instances will be distributed across different cloud service providers, leading to underutilization. Is it possible to integrate them to output computing power? Of course! That is to use WireGuard to set up a Kubernetes cluster.

Since the version mentioned in the previous article is outdated, I will start based on the latest software version and follow the article's method.

PS: After completing the setup, I realized a hard truth: whether you have something to do or not, you really need to read the official documentation of the software or engage in the official community (like GitHub) more often. The official documentation is already detailed enough, and there are people in the community who have already encountered various pitfalls. If you want to find clarity, read the official documentation more.

Environment Preparation#

I have prepared several cloud instances pre-installed with Ubuntu 20.04 on Tencent Cloud and Vultr. Of course, they can be any cloud service provider's instances, as long as they have public access and can run Linux systems.

Install Docker#

Install WireGuard#

Make sure to install the WireGuard software on each node. The installation details for Ubuntu 20.04 are as follows:

Here, you only need to complete the correct installation of WireGuard; no configuration or startup is required. Other tasks can be left to K3s for network configuration. What we need to do is to make K3s work efficiently and effectively. In fact, K3s has already prepared everything for us; we just need to configure the startup parameters simply.

Building K3s Cluster Across Clouds#

Since my cloud instances are distributed across different cloud service providers, they cannot access each other through the internal network environment provided by the providers. Here, we need to use WireGuard to complete the remote networking. Since K3s has already integrated WireGuard through Flannel, we can easily complete the networking with some simple configurations.

You need to install WireGuard on every node, both server and agents before attempting to leverage the WireGuard flannel backend option. The wireguard backend will be removed from v1.26 in favor of wireguard-native backend natively from Flannel.

Before starting, it is recommended that everyone read the official guide. Only by understanding the background can we clear up the confusion. It mentions that the configuration parameters for different versions of K3s vary, and it is worth noting that starting from v1.26, the startup parameter has changed from flannel-backend: wireguard to flannel-backend: wireguard-native.

We can refer to the deployment commands in the previous article "Kubernetes Getting Started to Practice: Initial Experience with K3s Cluster".

Install K3s Server#

When starting each Server, you need to add the following startup parameters to activate WireGuard:

The complete startup process for K3s Server is as follows:

Install K3s Agent#

When starting each Agent, you need to add the following startup parameters to activate WireGuard:

The complete startup process for K3s Agent is as follows:

About metrics-server Issues#

The metrics-server cannot retrieve metrics because the preferred value for kubelet-preferred-address-types is InternalIP, while the internal IP of the cloud server is an internal IP, which varies across different cloud providers and cannot communicate.

The issue is that metrics-server cannot obtain core metrics such as CPU and memory utilization, requiring manual intervention for configuration. In the newly released version v1.23.14+k3s1, it has been corrected that enabling the flannel-external-ip=true option will dynamically adjust the priority order of -kubelet-preferred-address-types=ExternalIP,InternalIP,Hostname.

Below, I will elaborate on how this feature adjustment affects the K3s cluster:

In versions v1.23.13+k3s1 and older#

Check the default configuration:

You need to modify the manifests of metrics-server by using the following command to edit the manifests online:

Adjust the following execution parameters and save:

After saving, wait for the resources to be rescheduled, and this will allow the metrics-server to use the public IP to communicate with the nodes. Check the core metrics again:

In versions v1.23.14+k3s1 and newer#

• [Release 1.23] Change the priority of address types depending on flannel-external-ip
• [Release 1.23] Change addr types in metrics server

1. Describe pod/metrics-server-, look for ARGS and check those scenarios: -kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname when flannel-external-ip=false
2. Do the same steps just change flannel-external-ip: true and look for -kubelet-preferred-address-types=ExternalIP,InternalIP,Hostname when flannel-external-ip=true

⚠️ Notes ⚠️#

(Don't ask why, just try it)

• Security group firewalls need to allow relevant ports
  • TCP 6443: K3s Server port
  • TCP 10250: metrics-server service port, used for communication between K3s Server and Agent to collect metrics; otherwise, core metrics such as CPU and memory utilization cannot be obtained
  • UDP 51820: Open the default port for flannel-backend: wireguard-native, which is used by Flannel's backend with WireGuard
  • TCP 30000-32767: K8s NodePort range, convenient for external debugging
• Optional startup parameters
  • --tls-san
    • Add other hostnames or IPs as alternative names in the TLS certificate
    • This allows access control and operation of remote clusters through public IP in a public environment
    • Or if deploying multiple servers and using LB for load balancing, you need to retain the public address
  • --disable servicelb
  • --disable traefik
• Disable unused components to save performance
  • Service Load Balancer
    • K3s provides a load balancer called Klipper Load Balancer, which can use available host ports. It allows the creation of LoadBalancer type Service, but does not include the implementation of LB. Some LB services require cloud providers, such as Amazon EC2. In contrast, K3s service LB allows the use of LB services without a cloud provider.
    • To disable the embedded LB, use the --disable servicelb option when starting each Server.
  • Traefik Ingress Controller
    • Traefik is a modern HTTP reverse proxy and load balancer. By default, Traefik is deployed when starting the Server. The Traefik ingress controller will use ports 80 and 443 on the host (i.e., these ports cannot be used for HostPort or NodePort).
    • To disable it, use the --disable traefik option when starting each server.

Verify K3s Cross-Cloud Cluster and Network#

Verify Cross-Cloud Cluster#

Verify Cross-Cloud Network#

Use the built-in CoreDNS, Service, and Pod to debug the network and verify whether the network is reachable between different nodes.

Before starting, quickly create a Service named whoami:

How to test the load balancing effect of the Service?#

Since the IP addresses of Service and Pod are all internal network segments of the Kubernetes cluster, we need to use kubectl exec to enter the Pod (or SSH into any node of the cluster) and then use tools like curl to access the Service.

Thanks to the built-in CoreDNS of the cluster, we can access the corresponding Service and Pod internally via domain names:

• The fully qualified domain name of the Service object is "object.namespace.svc.cluster.local", but often the latter part can be omitted, and just writing "object.namespace" or even "object" is sufficient, as it defaults to the namespace where the object is located (in this case, default)
  • For example, whoami, whoami.default, etc.
• Kubernetes also assigns a domain name to each Pod, in the form of "IP address.namespace.pod.cluster.local", but the . in the IP address needs to be replaced with -
  • For example, 10.42.2.2 corresponds to the domain name 10-42-2-2.default.pod

This way, we no longer need to worry about the IP addresses of Service and Pod objects; we just need to know their names and can access the backend services using DNS.

Access via External Network#

By accessing the Service via the external network, after multiple requests, it can be observed that the main node [http://42.193.XXX.XXX:32064](http://42.193.XXX.XXX:32064) serves as the access entry, successfully load balancing to Pods on different nodes and responding correctly.

Accessing Service CLUSTER-IP from Each Node in the Cluster#

By directly accessing the Service from the nodes, after multiple requests, it can also successfully load balance to Pods on different nodes and respond correctly.

Accessing Service and Pod from Within the Cluster#

Through various network verifications, it is proven that the multi-cloud networking environment using Flannel integrated with WireGuard is functional and can be used confidently.

Reference Links#

• K3s Official Documentation - English
• K3s Official Documentation - Chinese
• https://www.escapelife.site/posts/754ba85c.html
• https://docs.k3s.io/installation/network-options#distributed-hybrid-or-multicloud-cluster
• https://github.com/k3s-io/k3s/issues/5101
• https://www.netmaker.org/blog/deploy-distributed-kubernetes-clusters-with-wireguard-and-netmaker
• https://icloudnative.io/posts/deploy-k3s-cross-public-cloud/
• https://blog.csdn.net/wq1205750492/article/details/124883196
• https://icloudnative.io/posts/use-wireguard-as-kubernetes-cni/
• https://www.wireguard.com/install/
• https://gitee.com/spoto/wireguard
• https://www.inovex.de/de/blog/how-to-set-up-a-k3s-cluster-on-wireguard/
• https://cloud.tencent.com/developer/article/1985806
• ▶️ https://b23.tv/EHKAM7s
• ▶️ https://youtu.be/z2jvlFVU3dw
• ▶️ https://youtu.be/x1IF2XO051U

Original address: https://y0ngb1n.github.io/a/setup-k3s-cluster-multicloud-with-wireguard.html

If you find the content useful, feel free to like and share it with your friends; thank you in advance.

If you want to see updates on subsequent content faster, please hit "like", "share", or "favorite". These free encouragements will influence the update speed of future content.