如何搭建一个简单的K3S集群练手

安装

# 先卸载
/usr/local/bin/k3s-uninstall.sh
rm -rf /etc/rancher/k3s
rm -rf /var/lib/rancher/k3s
rm -rf /var/lib/cni/networks/k8s

# x.x.x.x是你的服务器外网IP
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --tls-san x.x.x.x" sh -

准备dns01验证

现在不用https都不好意思出门，这里假设我们的cf的token已经申请，token为cf_token_for_dns01，邮箱为[email protected]，需要操作的域名为web.your_doamin.com和grafana.your_domain.com

# 创建token的secret
kubectl create secret generic cloudflare-api-token \
  --namespace kube-system \
  --from-literal=api-token=cf_token_for_dns01

# 创建邮箱的secret
kubectl create secret generic acme-credentials \
  --namespace=kube-system \
  [email protected]

# 编辑文件
# vi /var/lib/rancher/k3s/server/manifests/traefik.yaml
valuesContent: |-
  deployment:
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
  providers:
    kubernetesIngress:
      publishedService:
        enabled: true
  priorityClassName: "system-cluster-critical"
  image:
    repository: "rancher/mirrored-library-traefik"
    tag: "3.3.6"
  tolerations:
  - key: "CriticalAddonsOnly"
    operator: "Exists"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  service:
    ipFamilyPolicy: "PreferDualStack"
  env:
    - name: CF_DNS_API_TOKEN
      valueFrom:
        secretKeyRef:
          name: cloudflare-api-token
          key: api-token
    - name: ACME_EMAIL
      valueFrom:
        secretKeyRef:
          name: acme-credentials
          key: email
  additionalArguments:
    - "--log.level=DEBUG"
    - "--certificatesresolvers.cloudflare.acme.email=$(ACME_EMAIL)"
    - "--certificatesresolvers.cloudflare.acme.storage=/data/acme.json"
    - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
    - "--certificatesresolvers.cloudflare.acme.dnschallenge.delaybeforecheck=30"
    - "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
  persistence:
    enabled: true

如果感觉修改的有问题，可以查看系统里的helmcharts.helm.cattle.io

kubectl get helmcharts.helm.cattle.io -n kube-system traefik -o yaml

修改后k3s会更新traefik，现在系统已经有了自动申请证书的能力！下一步我们部署一个简单nginx，然后为他绑定上域名。

https证书支持

先在web命名空间下创建一个nginx pod

apiVersion: v1
kind: Namespace
metadata:
  name: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: web
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

ok，我们现在有了一个nginx的workload，现在我们在当前命名空间声明IngressRoute，准备申请证书

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-to-https
  namespace: web
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: nginx-http
  namespace: web
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`web.your_doamin.com`)
      kind: Rule
      middlewares:
        - name: redirect-to-https
      services:
        - name: nginx-service
          port: 80
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: nginx-https
  namespace: web
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`web.your_doamin.com`)
      kind: Rule
      services:
        - name: nginx-service
          port: 80
  tls:
    certResolver: cloudflare

现在可以使用https://web.your_domain.com访问你的nginx了！

趁热打铁，我们部署一个Prometheus和Grafana，同时支持https证书，首先，我们安装prometheus-stack-grafana

# vi /var/lib/rancher/k3s/server/manifests/prometheus-grafana-stack.yml
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: prometheus
  namespace: kube-system
spec:
  chart: prometheus
  repo: https://prometheus-community.github.io/helm-charts
  createNamespace: true
  targetNamespace: monitoring
  version: 27.16.0
  valuesContent: |-
    alertmanager:
      enabled: true
    server:
      persistentVolume:
        enabled: true
        size: 10Gi
      resources:
        limits:
          cpu: 1
          memory: 2Gi
        requests:
          cpu: 500m
          memory: 1Gi
    kubeStateMetrics:
      enabled: true
    nodeExporter:
      enabled: true
    pushgateway:
      enabled: false
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: grafana
  namespace: kube-system
spec:
  chart: grafana
  repo: https://grafana.github.io/helm-charts
  targetNamespace: monitoring
  createNamespace: true
  version: 9.2.1
  valuesContent: |-
    persistence:
      enabled: true
      size: 5Gi
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 200m
        memory: 512Mi
    datasources:
      datasources.yaml:
        apiVersion: 1
        datasources:
        - name: Prometheus
          type: prometheus
          url: http://prometheus-server.monitoring.svc.cluster.local
          access: proxy
          isDefault: true
    adminPassword: admin_pass
    initChownData:
      enabled: false

安装完成后我们在monitoring下创建IngressRoute和Middleware。

---
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-to-https
  namespace: web
spec:
  redirectScheme:
    scheme: https
    permanent: true
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana-http
  namespace: monitoring
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`grafana.your_domain.com`)
      kind: Rule
      middlewares:
        - name: redirect-to-https
      services:
        - name: prometheus-stack-grafana
          port: 80
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana-https
  namespace: monitoring
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.your_domain.com`)
      kind: Rule
      services:
        - name: prometheus-stack-grafana
          port: 80
  tls:
    certResolver: cloudflare
---

现在可以使用https://grafana.your_domain.com访问你的grafana了！