我们当前开发环境 K3s 使用的是 Traefik 作为 Ingress，生产环境的 K8s 使用的是 Nginx 作为 Ingress，要将 Traefik 替换为 Nginx，并不是因为 Traefix 不如 Nginx，只是想把我们开发环境里的 Ingress 和生产环境的 Ingress 统一下，简化 chart 包的写法。

我们先更新下 K3s，排除掉 Traefik 的安装，这里官方文档就有提到：

curl -sfL https://get.k3s.io | sh -s - --disable traefik

这样安装完的 K3s 是不带 Ingress 的，我们需要额外再安装一下 Nginx Ingress：

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace

执行完这两个核心动作之后，Nginx 就变成 K3s 中的 Ingress 组件了，更完整的操作命令如下：

curl -sfL https://get.k3s.io | sh -s - --disable traefik

crictl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
ctr image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1 k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
ctr image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1 k8s.gcr.io/ingress-nginx/kube-webhook-certgen@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660

crictl pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.2.0
ctr image tag registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.2.0 k8s.gcr.io/ingress-nginx/controller:v1.2.0
ctr image tag registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.2.0 k8s.gcr.io/ingress-nginx/controller@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185


curl -O https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace

vim /u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk

大约在 176 行，行尾增加 -lnnz11

#===========================
#  emdctl
#===========================

$(SYSMANBIN)emdctl:
        $(MK_EMAGENT_NMECTL) -lnnz11

XQuartz 就是 macOS 下的 X11，部分 Linux 程序安装时需使用到图形界面，可以选择这个解决方案。使用 brew 就可以安装上：

brew install xquartz --cask

安装后通过 应用-实用工具-XQuartz 可启动 XQuartz 程序，启动后没有什么窗口效果，此时打开 macOS 上的终端程序，输入命令：

export DISPLAY=:0
ssh -Y 192.168.56.10

此时就会登录到要安装程序的 Linux 主机上，启动一个窗口程序测试一下效果：

xclock

只要配置一下 hostname 属性就可以

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"
  config.vm.hostname = "oracle"
end

先安装 X Windows，再安装 GNOME，最后让系统默认启动图形。

yum groupinstall -y "X Window System"
yum groupinstall -y "GNOME Desktop"
systemctl set-default graphical

先从 https://github.com/k3s-io/k3s/releases 下载 k3s 命令与 k3s-airgap-images-amd64.tar 离线包

wget https://github.com/k3s-io/k3s/releases/download/v1.23.3%2Bk3s1/k3s
wget https://github.com/k3s-io/k3s/releases/download/v1.23.3%2Bk3s1/k3s-airgap-images-amd64.tar
curl -o install.sh https://get.k3s.io
sudo chmod +x k3s
sudo chmod +x install.sh

安装必要的系统环境

sudo yum install -y container-selinux
sudo yum install -y https://rpm.rancher.io/k3s/stable/common/centos/7/noarch/k3s-selinux-0.4-1.el7.noarch.rpm

然后就可以开始准备安装

sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/
sudo cp k3s /usr/local/bin/
INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

安装 nfs 服务

mkdir /data -m 777
vi /etc/exports

/data		*(rw,insecure,no_root_squash)

service nfs restart

安装 k3s 服务

curl -sfL https://get.k3s.io | sh -s - --kube-apiserver-arg "feature-gates=RemoveSelfLink=false"

如果实现已经安装过，需要手动修改启动文件

ExecStart=/usr/local/bin/k3s \
    server \
	'--kube-apiserver-arg' \
	'feature-gates=RemoveSelfLink=false' \

安装 helm

https://github.com/helm/helm/releases

curl -O https://get.helm.sh/helm-v3.8.0-linux-amd64.tar.gz
tar zxvf helm-v3.8.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

安装 nfs-client-provisioner

helm repo add moikot https://moikot.github.io/helm-charts
helm -n kube-system install nfs-client-provisioner moikot/nfs-client-provisioner --version 1.3.0 --set nfs.server=192.168.57.11 --set nfs.path=/data

效果测试

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install redis bitnami/redis --set global.storageClass=nfs-client

brew install android-platform-tools

安装后即可执行 adb 命令

$ adb devices
* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached

brew install seafile-client

curl -sfL https://get.k3s.io | sh -

[INFO] Finding release for channel stable
[INFO] Using v1.22.5+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.22.5+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s