Kubernetes Operator开发与StatefulSet有状态应用管理

背景与挑战

在Kubernetes生态中，有状态应用（如数据库、消息队列、缓存系统）的管理一直是最具挑战性的领域之一。与传统无状态应用相比，有状态应用需要确保：

• 数据持久性：确保数据不因Pod重启或迁移而丢失
• 稳定标识：维护固定的网络标识和存储映射关系
• 有序操作：部署、扩展、升级等操作需要严格的顺序控制
• 状态一致性：保证集群状态的一致性和数据完整性

以MySQL数据库集群为例，传统Deployment无法满足以下需求：
- 主从节点的有序启动和配置
- 自动故障转移和数据同步
- 备份恢复的自动化管理
- 配置变更的滚动更新

实现方案

基于Operator模式我们开发了MySQL集群管理Operator，实现有状态应用的自动化管理

• CRD（CustomResourceDefinition）​：定义MySQLCluster资源类型，封装集群规格、备份策略等配置
• 控制器逻辑​​：持续调和实际状态与期望状态，处理节点故障转移、数据备份等复杂操作
• 初始化与配置管理​​：自动处理MySQL系统初始化、主从复制配置和参数优化

apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
metadata:
  name: mycluster
spec:
  instances: 3
  secretName: my-secret
  tlsUseSelfSigned: true
  router:
    instances: 1
  podSpec:
    resources:
      requests:
        memory: "2Gi"
        cpu: "1000m"
      limits:
        memory: "4Gi" 
        cpu: "2000m"
    # 关键：Pod反亲和性确保节点分布
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
   

INSTALLATION (MANUAL)

INSTALL PACKAGES WITH CEPHADM

prepare env

yum install python3 yum-utils -y
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce docker-ce-cli containerd.io
systemctl enable docker && service docker start
systemctl disable firewalld &&  service firewalld stop

Download the cephadm script

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm
chmod +x cephadm

Configure the Ceph repository based on the release name:

./cephadm add-repo --release Octopus
#https://docs.ceph.com/en/latest/releases/index.html

Install the appropriate packages. You can install them using your package management tool (e.g., APT, Yum) directly, or you can also use the cephadm wrapper. For example:

sudo rpm --import 'https://download.ceph.com/keys/release.asc'
./cephadm install
./cephadm install ceph-common

BOOTSTRAP A NEW CLUS

Operation

Copy file between two esxis

Check if SSH client service is enabled. Will return 'False' if 'Disabled'.

esxcli network firewall ruleset list --ruleset-id sshClient

Then enable SSH client.

esxcli network firewall ruleset set --ruleset-id sshClient --enabled=true

List disk

df -hl

copy file between esxi

scp -r workspace/ 192.168.0.111:/vmfs/volumes/datastore1

unregister vm

vim-cmd vmsvc/getallvms
vim-cmd vmsvc/unregister 13

MicroK8s is the smallest, fastest, fully-conformant Kubernetes that tracks upstream releases and makes clustering trivial. MicroK8s is great for offline development, prototyping, and testing.

Install MicroK8s

Requirements

• Operating system which supports snapd
• At least 20G of disk space and 4G of memory are recommended
• An internet connection

For Centos7

service firewall stop && systemctl disable firewalld
yum install epel-release
yum install snapd
systemctl enable --now snapd.socket
sudo ln -s /var/lib/snapd/snap /snap

Install MicroK8s

MicroK8s will install a minimal, lightweight Kubernetes you can run and use on practically any machine. It can be installed with a snap:

snap install microk8s --classic --channel=1.21/stable
snap install microk8s --classic --channel=1.19/stable
#from 1.21 to 1.19
snap refresh --classic --channel=1.19/stable microk8s

All the currently available channels are shown if you run

snap info m

Install

Debian / Ubuntu:

apt-get install python-pip
pip install shadowsocks

CentOS:

yum install python-setuptools && easy_install pip
#curl -o get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py
pip install shadowsocks

Windows:

See [Install Server on Windows]

Usage

ssserver -p 443 -k password -m aes-256-cfb

To run in the background:

sudo ssserver -p 443 -k password -m aes-256-cfb --user nobody -d start
ssserver --log-file /var/log/shadowsocks-12300.log --pid-file /var/run/shadowsocks-12300.pid --user nobody -p 12300 -k password -m aes-256-cfb -s 0.0.0.0 -d start

To stop:

sudo ssserver -d stop

To check the log:

sudo less /var/log/shadowsocks.log

Check all the options via -h. You can also use a [Configuration] file
instead.

Client

• [Windows] / [OS X]
• [Android] / [iOS]
• [OpenWRT]

Use GUI clients on your local PC/phones. Check the README of your client
for more information.

Documenta

Install Jdk

参考Jdk安装

Configure SSH

参考SSH免密码登陆

Install Ntp

yum install ntp
vi /etc/ntp.conf
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst
systemctl enable ntpd
systemctl start ntpd

Install Hadoop

Download Hadoop

wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.5/hadoop-2.7.5.tar.gz
tar zxvf hadoop-2.7.5.tar.gz 
cd hadoop-2.7.5/

Configure Hadoop

Config Env

vi .bashrc
export JAVA_HOME=/opt/jdk1.8.0_202
export HADOOP_PID_DIR=/data/hadooptemp

Configure slaves

vi slaves
test01
test02
test03

Configure core-site.xml

mkdir /data/hadoop
mkdir /data/hadooptemp

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop</value>
        <description>
            A base for other temporary directories.
        </description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://cluster01</value>
        <descr

Install MongoDB Community Edition

MongoDB only supports the 64-bit versions of these platforms.

Follow these steps to install MongoDB Community Edition using the yum package manager.

Create a /etc/yum.repos.d/mongodb-org-4.4.repo file so that you can install MongoDB directly using yum:

[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc

To install the latest stable version of MongoDB, issue the following command:

sudo yum install -y mongodb-org

Alternatively, to install a specific release of MongoDB, specify each component package individually and append the version number to the package name, as in the following example:

sudo yum install -y mongodb-org-4.4.11 mongodb-org-server-4.4.11 mongodb-org-shell-4.4.11 mongodb-org-mongos-4.4.11 mongodb-org-tools-4.4.11

You can specify

Krew

Make sure that git is installed.

Run this command to download and install krew:

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

Add the $HOME/.krew/bin directory to your PATH environment variable. To do this, update your .bashrc or .zshrc file and append the following line:

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

and restart your shell.

Run kubectl krew to check the installation.

kubectx + kubens

export HTTP_PROXY=http://192.168.0.201:6666
export HTTPS_PROXY=http://192.168.0.201:6666
kubectl krew install ctx
kubectl krew install ns

Download the latest Kafka release and extract it:

wget https://dlcdn.apache.org/kafka/3.1.0/kafka_2.12-3.1.0.tgz --no-check-certificate

Config Kafka Broker server.properties

#broker.id=0
log.dirs=/data/kafka
zookeeper.connect=test01:2181,test02:2181,test03:2181/kafka

Copy To All Nodes

scp -r kafka_2.12-3.1.0/ test02:~
scp -r kafka_2.12-3.1.0/ test03:~

Start kafka service

ssh test01 'cd kafka_2.12-3.1.0 && bin/kafka-server-start.sh -daemon config/server.properties'
ssh test02 'cd kafka_2.12-3.1.0 && bin/kafka-server-start.sh -daemon config/server.properties'
ssh test03 'cd kafka_2.12-3.1.0 && bin/kafka-server-start.sh -daemon config/server.properties'

Run Kafka Ui

with kafka-ui

docker run -d --name=kafkaui -p 19092:8080 \
    --add-host test01:192.168.0.161 \
    --add-host test02:192.168.0.162 \
    --add-host test03:192.168.0.163 \
    -e KAFKA_CLUSTERS_0_NAME=kafka01 \
    -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=test01:90

Install

yum install bind bind-utils

backup named.conf file

cp /etc/named.conf /etc/named.conf.bak

configure named.conf

listen-on port 53 { 127.0.0.1; 192.168.0.201; };
allow-query     { any; };
zone "louyj.com" IN{                
 type master;                       
 file "louyj.com.zone";
};

type：

• master指的是主dns解析
• slave指的是从dns解析
• hint指的是根域名解析（根提示域）
• forward指的是转发，转发不使用file

add zone

vi /var/named/louyj.com.zone

$TTL 1D
@ IN SOA  ns1.louyj.com. ns2.louyj.com. (
0;serial
1D;refresh
1H;retry
1W;expire
3H;minimum
);
@ IN  NS   ns1
@ IN  NS   ns2
ns1                               IN  A    192.168.0.201
ns2                               IN  A    192.168.0.200
www                               IN  A    192.168.0.201
esxi                              IN  A    192.168.0.201
cloud                             IN  A    192.168.0.201
image                             IN  A    192.168.0.201
media                             IN  A    192.168.0.201
no