如何搭建一个简单的K3S集群练手
安装
# 先卸载
/usr/local/bin/k3s-uninstall.sh
rm -rf /etc/rancher/k3s
rm -rf /var/lib/rancher/k3s
rm -rf /var/lib/cni/networks/k8s
# x.x.x.x是你的服务器外网IP
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --tls-san x.x.x.x" sh -
准备dns01验证
现在不用https都不好意思出门，这里假设我们的cf的token已经申请，token为cf_token_for_dns01，邮箱为[email protected]，需要操作的域名为web.your_doamin.com和grafana.your_domain.com
# 创建token的secret
kubectl create secret generic cloudflare-api-token \
--namespace kube-system \
--from-literal=api-token=cf_token_for_dns01
# 创建邮箱的secret
kubectl create secret generic acme-credentials \
--namespace=kube-system \
[email protected]
# 编辑文件
# vi /var/lib/rancher/k3s/server/manifests/traefik.yaml
valuesContent: |-
deployment:
podAnnotations:
prometheus.io/port: "8082"
prometheus.io/scrape: "true"
providers:
kubernetesIngress:
publishedService:
enabled: true
priorityClassName: "system-cluster-critical"
image:
repository: "rancher/mirrored-library-traefik"
tag: "3.3.6"
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
service:
ipFamilyPolicy: "PreferDualStack"
env:
- name: CF_DNS_API_TOKEN
valueFrom:
secretKeyRef:
name: cloudflare-api-token
key: api-token
- name: ACME_EMAIL
valueFrom:
secretKeyRef:
name: acme-credentials
key: email
additionalArguments:
- "--log.level=DEBUG"
- "--certificatesresolvers.cloudflare.acme.email=$(ACME_EMAIL)"
- "--certificatesresolvers.cloudflare.acme.storage=/data/acme.json"
- "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
- "--certificatesresolvers.cloudflare.acme.dnschallenge.delaybeforecheck=30"
- "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
persistence:
enabled: true
如果感觉修改的有问题，可以查看系统里的helmcharts.helm.cattle.io
kubectl get helmcharts.helm.cattle.io -n kube-system traefik -o yaml
修改后k3s会更新traefik，现在系统已经有了自动申请证书的能力！下一步我们部署一个简单nginx，然后为他绑定上域名。
https证书支持
先在web命名空间下创建一个nginx pod
apiVersion: v1
kind: Namespace
metadata:
name: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: web
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine
ports:
- containerPort: 80
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "200m"
memory: "256Mi"
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
namespace: web
spec:
type: ClusterIP
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
ok，我们现在有了一个nginx的workload，现在我们在当前命名空间声明IngressRoute，准备申请证书
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: redirect-to-https
namespace: web
spec:
redirectScheme:
scheme: https
permanent: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: nginx-http
namespace: web
spec:
entryPoints:
- web
routes:
- match: Host(`web.your_doamin.com`)
kind: Rule
middlewares:
- name: redirect-to-https
services:
- name: nginx-service
port: 80
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: nginx-https
namespace: web
spec:
entryPoints:
- websecure
routes:
- match: Host(`web.your_doamin.com`)
kind: Rule
services:
- name: nginx-service
port: 80
tls:
certResolver: cloudflare
现在可以使用https://web.your_domain.com访问你的nginx了！
趁热打铁，我们部署一个Prometheus和Grafana，同时支持https证书，首先，我们安装prometheus-stack-grafana
# vi /var/lib/rancher/k3s/server/manifests/prometheus-grafana-stack.yml
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: prometheus
namespace: kube-system
spec:
chart: prometheus
repo: https://prometheus-community.github.io/helm-charts
createNamespace: true
targetNamespace: monitoring
version: 27.16.0
valuesContent: |-
alertmanager:
enabled: true
server:
persistentVolume:
enabled: true
size: 10Gi
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
kubeStateMetrics:
enabled: true
nodeExporter:
enabled: true
pushgateway:
enabled: false
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: grafana
namespace: kube-system
spec:
chart: grafana
repo: https://grafana.github.io/helm-charts
targetNamespace: monitoring
createNamespace: true
version: 9.2.1
valuesContent: |-
persistence:
enabled: true
size: 5Gi
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 200m
memory: 512Mi
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus-server.monitoring.svc.cluster.local
access: proxy
isDefault: true
adminPassword: admin_pass
initChownData:
enabled: false
安装完成后我们在monitoring下创建IngressRoute和Middleware。
---
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: redirect-to-https
namespace: web
spec:
redirectScheme:
scheme: https
permanent: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: grafana-http
namespace: monitoring
spec:
entryPoints:
- web
routes:
- match: Host(`grafana.your_domain.com`)
kind: Rule
middlewares:
- name: redirect-to-https
services:
- name: prometheus-stack-grafana
port: 80
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: grafana-https
namespace: monitoring
spec:
entryPoints:
- websecure
routes:
- match: Host(`grafana.your_domain.com`)
kind: Rule
services:
- name: prometheus-stack-grafana
port: 80
tls:
certResolver: cloudflare
---
现在可以使用https://grafana.your_domain.com访问你的grafana了！