Sen Blog

Appearance

Operator ​

什么是Operator ​

Kubernetes operator 是一种封装、部署、管理 kubernetes 应用的方法。它是 Kubernetes 的扩展软件，利用自定义资源管理应用及组件。operator 所有的操作都是调用 Kubernetes Apiserver 的接口，所以本质上它也是 Apiserver 的客户端软件。

Operator 搭建方式有两种 1、kubernetes-sigs/kubebuilder 2、 operator-framework/operator-sdk 本文采用kubeduiler方式。

环境搭建 ​

Centos 安装Golang环境 ​

安装wget ​

bash
yum install -y wget

下载golang ​

汝可使用 GoLang官方网站 查阅所需版本.

bash
wget https://dl.google.com/go/go1.20.6.linux-amd64.tar.gz

解压 ​

bash
tar zxvf go1.20.6.linux-amd64.tar.gz -C /usr/local

配置环境变量 ​

编辑 vim /etc/profile 在末尾添加如下配置.

bash
$ vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
$ source /etc/profile

使用 source /etc/profile 生效.

查看版本 ​

bash
[root@master ~]# go version
go version go1.20.6 linux/amd64

安装GCC ​

安装gcc避免出现未知错误

bash
yum install -y gcc

安装kubebuilder ​

GitHub 地址：kubernetes-sigs/kubebuilder

下载 ​

bash
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)

配置命令 ​

bash
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

查看版本 ​

bash
[root@master tomato-operator]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.11.1", KubernetesVendor:"1.27.1", GitCommit:"1dc8ed95f7cc55fef3151f749d3d541bec3423c9", BuildDate:"2023-07-03T13:10:56Z", GoOs:"linux", GoArch:"amd64"}

创建Operator ​

搭建tomato-operator工程 ​

bash
[root@master operator]# mkdir tomato-operator
[root@master operator]# cd tomato-operator/
[root@master tomato-operator]# kubebuilder init --domain senhao.top --repo github.com/senhao-xu/tomato-operator --project-name tomato-operator
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.15.0
go: downloading sigs.k8s.io/controller-runtime v0.15.0
go: downloading k8s.io/apimachinery v0.27.2
go: downloading k8s.io/client-go v0.27.2
go: downloading k8s.io/utils v0.0.0-20230209194617-a36077c30491
go: downloading github.com/go-logr/logr v1.2.4
go: downloading k8s.io/klog/v2 v2.90.1
go: downloading github.com/prometheus/client_golang v1.15.1
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/google/gofuzz v1.1.0
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3
go: downloading k8s.io/component-base v0.27.2
go: downloading golang.org/x/net v0.10.0
go: downloading github.com/imdario/mergo v0.3.6
go: downloading golang.org/x/term v0.8.0
go: downloading github.com/evanphx/json-patch/v5 v5.6.0
go: downloading gomodules.xyz/jsonpatch/v2 v2.3.0
go: downloading k8s.io/api v0.27.2
go: downloading github.com/evanphx/json-patch v4.12.0+incompatible
go: downloading k8s.io/apiextensions-apiserver v0.27.2
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/prometheus/client_model v0.4.0
go: downloading github.com/prometheus/common v0.42.0
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading golang.org/x/oauth2 v0.5.0
go: downloading golang.org/x/time v0.3.0
go: downloading k8s.io/kube-openapi v0.0.0-20230501164219-8b0f38b5fd1f
go: downloading github.com/golang/protobuf v1.5.3
go: downloading github.com/google/gnostic v0.5.7-v3refs
go: downloading golang.org/x/sys v0.8.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/fsnotify/fsnotify v1.6.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.2.0
go: downloading github.com/prometheus/procfs v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/google/go-cmp v0.5.9
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.9.0
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
go: downloading github.com/emicklei/go-restful/v3 v3.9.0
go: downloading github.com/go-openapi/swag v0.22.3
go: downloading github.com/go-openapi/jsonreference v0.20.1
go: downloading github.com/go-openapi/jsonpointer v0.19.6
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/josharian/intern v1.0.0
Update dependencies:
$ go mod tidy
go: downloading github.com/onsi/gomega v1.27.7
go: downloading github.com/go-logr/zapr v1.2.4
go: downloading go.uber.org/zap v1.24.0
go: downloading github.com/onsi/ginkgo/v2 v2.9.5
go: downloading github.com/stretchr/testify v1.8.1
go: downloading go.uber.org/goleak v1.2.1
go: downloading go.uber.org/atomic v1.7.0
go: downloading go.uber.org/multierr v1.6.0
go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/kr/pretty v0.3.1
go: downloading github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
go: downloading golang.org/x/tools v0.9.1
go: downloading github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/rogpeppe/go-internal v1.10.0
go: downloading github.com/benbjohnson/clock v1.1.0
Next: define a resource with:
$ kubebuilder create api
[root@master tomato-operator]# ls
cmd  config  Dockerfile  go.mod  go.sum  hack  Makefile  PROJECT  README.md

创建Api ​

bash
[root@master tomato-operator]# kubebuilder create api --group tomato --version v1 --kind Nginx
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/guestbook_types.go
api/v1/groupversion_info.go
internal/controller/suite_test.go
internal/controller/guestbook_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
mkdir -p /root/operator/tomato-operator/bin
test -s /root/operator/tomato-operator/bin/controller-gen && /root/operator/tomato-operator/bin/controller-gen --version | grep -q v0.12.0 || \
GOBIN=/root/operator/tomato-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
/root/operator/tomato-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

此时operator初始化完成，可以开发自定义逻辑。

生成CRD资源 ​

bash
[root@master tomato-operator]# make manifests
/root/operator/tomato-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

安装CRD ​

bash
[root@master tomato-operator]# make install
/root/operator/tomato-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
test -s /root/operator/tomato-operator/bin/kustomize || GOBIN=/root/operator/tomato-operator/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.0.1
go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.0.1
go: downloading github.com/spf13/cobra v1.4.0
go: downloading sigs.k8s.io/kustomize/api v0.13.2
go: downloading sigs.k8s.io/kustomize/cmd/config v0.11.1
go: downloading sigs.k8s.io/kustomize/kyaml v0.14.1
go: downloading golang.org/x/text v0.6.0
go: downloading github.com/go-errors/errors v1.4.2
go: downloading k8s.io/kube-openapi v0.0.0-20230109183929-3758b55a6596
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/xlab/treeprint v1.1.0
go: downloading github.com/evanphx/json-patch v4.11.0+incompatible
go: downloading google.golang.org/protobuf v1.28.0
go: downloading github.com/google/go-cmp v0.5.5
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/golang/protobuf v1.5.2
go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
/root/operator/tomato-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.senhao.top created

镜像构建 ​

构建速度与网络及主机配置有关

bash
[root@master tomato-operator]# go mod vendor
[root@master tomato-operator]# make docker-build
test -s /root/operator/tomato-operator/bin/controller-gen && /root/operator/tomato-operator/bin/controller-gen --version | grep -q v0.12.0 || \
GOBIN=/root/operator/tomato-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
/root/operator/tomato-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/operator/tomato-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
test -s /root/operator/tomato-operator/bin/setup-envtest || GOBIN=/root/operator/tomato-operator/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
go: downloading sigs.k8s.io/controller-runtime/tools/setup-envtest v0.0.0-20230804180024-dbcf380a24f5
go: downloading sigs.k8s.io/controller-runtime v0.15.1
go: downloading github.com/go-logr/zapr v1.2.0
go: downloading github.com/go-logr/logr v1.2.0
go: downloading github.com/spf13/afero v1.6.0
go: downloading go.uber.org/zap v1.19.1
go: downloading golang.org/x/text v0.3.7
KUBEBUILDER_ASSETS="/root/operator/tomato-operator/bin/k8s/1.27.1-linux-amd64" go test ./... -coverprofile cover.out
?   	github.com/senhao-xu/tomato-operator/api/v1	[no test files]
?   	github.com/senhao-xu/tomato-operator/cmd	[no test files]
ok  	github.com/senhao-xu/tomato-operator/internal/controller	0.035s	coverage: 0.0% of statements
docker build -t controller:latest .
Sending build context to Docker daemon  50.44MB
Step 1/16 : FROM golang:1.20 as builder
1.20: Pulling from library/golang
785ef8b9b236: Pull complete 
5a6dad8f55ae: Pull complete 
bd36c7bfe5f4: Pull complete 
9e59e6b803ed: Pull complete 
f9e56f8fc718: Pull complete 
5e649a4c4e0b: Pull complete 
Digest: sha256:bc5f0b5e43282627279fe5262ae275fecb3d2eae3b33977a7fd200c7a760d6f1
Status: Downloaded newer image for golang:1.20
 ---> 99f8cec02f0a
Step 2/16 : ARG TARGETOS
 ---> Running in 8be3ea60cba7
Removing intermediate container 8be3ea60cba7
 ---> af8a3c590d45
Step 3/16 : ARG TARGETARCH
 ---> Running in b6619762ed4e
Removing intermediate container b6619762ed4e
 ---> dbb76082e494
Step 4/16 : WORKDIR /workspace
 ---> Running in 13d5d1e31090
Removing intermediate container 13d5d1e31090
 ---> f82b58f0576f
Step 5/16 : COPY go.mod go.mod
 ---> 43c398399971
Step 6/16 : COPY go.sum go.sum
 ---> 99c9edba860b
Step 7/16 : RUN go mod download
 ---> Running in 24c88acd3a97
Removing intermediate container 24c88acd3a97
 ---> 64cbccfb865c
Step 8/16 : COPY cmd/main.go cmd/main.go
 ---> 4f23b9f40dd5
Step 9/16 : COPY api/ api/
 ---> 274c255448fe
Step 10/16 : COPY internal/controller/ internal/controller/
 ---> 2fe66fd2015e
Step 11/16 : RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go
 ---> Running in 252c942e5693
Removing intermediate container 252c942e5693
 ---> 3744185dde3a
Step 12/16 : FROM gcr.io/distroless/static:nonroot
nonroot: Pulling from distroless/static
a7ca0d9ba68f: Pull complete 
fe5ca62666f0: Pull complete 
b02a7525f878: Pull complete 
fcb6f6d2c998: Pull complete 
e8c73c638ae9: Pull complete 
1e3d9b7d1452: Pull complete 
4aa0ea1413d3: Pull complete 
7c881f9ab25e: Pull complete 
5627a970d25e: Pull complete 
Digest: sha256:9ecc53c269509f63c69a266168e4a687c7eb8c0cfd753bd8bfcaa4f58a90876f
Status: Downloaded newer image for gcr.io/distroless/static:nonroot
 ---> ea2dc773e32d
Step 13/16 : WORKDIR /
 ---> Running in 60ce6bf75120
Removing intermediate container 60ce6bf75120
 ---> e0cacab08108
Step 14/16 : COPY --from=builder /workspace/manager .
 ---> 255a1aabcbd5
Step 15/16 : USER 65532:65532
 ---> Running in ef6d9094a55e
Removing intermediate container ef6d9094a55e
 ---> 1ec20a1d73d9
Step 16/16 : ENTRYPOINT ["/manager"]
 ---> Running in 7ac70ab5cfe2
Removing intermediate container 7ac70ab5cfe2
 ---> 6b17070a25f3
Successfully built 6b17070a25f3
Successfully tagged controller:latest

部署 ​

bash
[root@master tomato-operator]# make deploy
test -s /root/operator/tomato-operator/bin/controller-gen && /root/operator/tomato-operator/bin/controller-gen --version | grep -q v0.12.0 || \
GOBIN=/root/operator/tomato-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
/root/operator/tomato-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/operator/tomato-operator/bin/kustomize version is not expected v5.0.1. Removing it before installing.
test -s /root/operator/tomato-operator/bin/kustomize || GOBIN=/root/operator/tomato-operator/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.0.1
cd config/manager && /root/operator/tomato-operator/bin/kustomize edit set image controller=controller:latest
/root/operator/tomato-operator/bin/kustomize build config/default | kubectl apply -f -
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/tomato-operator-system created
customresourcedefinition.apiextensions.k8s.io/nginxes.tomato.senhao.top unchanged
serviceaccount/tomato-operator-controller-manager created
role.rbac.authorization.k8s.io/tomato-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/tomato-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/tomato-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/tomato-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/tomato-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/tomato-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/tomato-operator-proxy-rolebinding created
service/tomato-operator-controller-manager-metrics-service created
deployment.apps/tomato-operator-controller-manager created

卸载 ​

bash
[root@master tomato-operator]# make uninstall
test -s /root/operator/tomato-operator/bin/controller-gen && /root/operator/tomato-operator/bin/controller-gen --version | grep -q v0.12.0 || \
GOBIN=/root/operator/tomato-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
/root/operator/tomato-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/operator/tomato-operator/bin/kustomize version is not expected v5.0.1. Removing it before installing.
test -s /root/operator/tomato-operator/bin/kustomize || GOBIN=/root/operator/tomato-operator/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.0.1
/root/operator/tomato-operator/bin/kustomize build config/crd | kubectl delete --ignore-not-found=false -f -
customresourcedefinition.apiextensions.k8s.io "nginxes.tomato.senhao.top" deleted

常见错误 ​

exit status 2 ​

修复方法： 1、 安装gcc 2、使用 go mod tidy 检测go版本是否为对应版本

bash
[root@master tomato-operator]# kubebuilder create api --group webapp --version v1 --kind Guestbook
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/guestbook_types.go
api/v1/groupversion_info.go
internal/controller/suite_test.go
internal/controller/guestbook_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
mkdir -p /root/operator/tomato-operator/bin
test -s /root/operator/tomato-operator/bin/controller-gen && /root/operator/tomato-operator/bin/controller-gen --version | grep -q v0.12.0 || \
GOBIN=/root/operator/tomato-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.12.0
# crypto/internal/nistec
/usr/local/go/src/crypto/internal/nistec/p256_ordinv.go:14:6: p256OrdMul redeclared in this block
	/usr/local/go/src/crypto/internal/nistec/p256_asm_ordinv.go:14:6: other declaration of p256OrdMul
/usr/local/go/src/crypto/internal/nistec/p256_ordinv.go:19:6: p256OrdSqr redeclared in this block
	/usr/local/go/src/crypto/internal/nistec/p256_asm_ordinv.go:19:6: other declaration of p256OrdSqr
/usr/local/go/src/crypto/internal/nistec/p256_ordinv.go:21:6: P256OrdInverse redeclared in this block
	/usr/local/go/src/crypto/internal/nistec/p256_asm_ordinv.go:21:6: other declaration of P256OrdInverse
make: *** [/root/operator/tomato-operator/bin/controller-gen] Error 1
Error: failed to create API: unable to run post-scaffold tasks of "base.go.kubebuilder.io/v4": exit status 2

参考文章： 使用Go开发Kubernetes Operator

Copyright © 2024 Sen Blog; ICP License: 京ICP备2021037002号-1