Codis集群搭建

# Codis组件说明

![title](/api/file/getImage?fileId=598dd8af0f93bd0545000390)

## Codis Server

基于 redis-3.2.8 分支开发。增加了额外的数据结构，以支持 slot 有关的操作以及数据迁移指令。具体的修改可以参考文档 redis 的修改。

## Codis Proxy

客户端连接的 Redis 代理服务, 实现了 Redis 协议。 除部分命令不支持以外(不支持的命令列表)，表现的和原生的 Redis 没有区别（就像 Twemproxy）

- 对于同一个业务集群而言，可以同时部署多个 codis-proxy 实例
- 不同 codis-proxy 之间由 codis-dashboard 保证状态同步。

## Codis Dashboard

集群管理工具，支持 codis-proxy、codis-server 的添加、删除，以及据迁移等操作。在集群状态发生改变时，codis-dashboard 维护集群下所有 codis-proxy 的状态的一致性。

- 对于同一个业务集群而言，同一个时刻 codis-dashboard 只能有 0个或者1个；
- 所有对集群的修改都必须通过 codis-dashboard 完成。

## Codis Admin

集群管理的命令行工具。

可用于控制 codis-proxy、codis-dashboard 状态以及访问外部存储。

## Codis FE

集群管理界面

- 多个集群实例共享可以共享同一个前端展示页面；
- 通过配置文件管理后端 codis-dashboard 列表，配置文件可自动更新。

## Storage

为集群状态提供外部存储。

- 提供 Namespace 概念，不同集群的会按照不同 product name 进行组织；
- 目前仅提供了 Zookeeper、Etcd、Fs 三种实现，但是提供了抽象的 interface 可自行扩展。

# Codis集群搭建

## 安装zookeeper

[zookeeper集群安装](https://note.louyj.com/blog/post/louyj/zookeeper-%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA)

## 下载codis

https://github.com/CodisLabs/codis/releases

## 修改Codis配置

### 修改dashboard配置

# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
    # Quick Start
    coordinator_name = "zookeeper"
    coordinator_addr = "192.168.1.202:2181"
    
    # Set Codis Product Name/Auth.
    product_name = "cluster01"
    product_auth = 'stream!23$'
    
    # Set bind address for admin(rpc), tcp only.
    admin_addr = "0.0.0.0:18080"
    
    # Set arguments for data migration (only accept 'sync' & 'semi-async').
    migration_method = "semi-async"
    migration_parallel_slots = 100
    migration_async_maxbulks = 200
    migration_async_maxbytes = "32mb"
    migration_async_numkeys = 500
    migration_timeout = "30s"
    
    # Set configs for redis sentinel.
    sentinel_client_timeout = "10s"
    sentinel_quorum = 2
    sentinel_parallel_syncs = 1
    sentinel_down_after = "30s"
    sentinel_failover_timeout = "5m"
    sentinel_notification_script = ""
    sentinel_client_reconfig_script = ""

### 修改proxy配置

# Set Codis Product Name/Auth.
    product_name = "cluster01"
    product_auth = 'stream!23$'
    
    # Set auth for client session
    #   1. product_auth is used for auth validation among codis-dashboard,
    #      codis-proxy and codis-server.
    #   2. session_auth is different from product_auth, it requires clients
    #      to issue AUTH <PASSWORD> before processing any other commands.
    session_auth = 'stream!23$'
    
    # Set bind address for admin(rpc), tcp only.
    admin_addr = "0.0.0.0:11080"
    
    # Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
    proto_type = "tcp4"
    proxy_addr = "#{proxy_addr}"
    
    # Set jodis address & session timeout
    #   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".
    #   2. jodis_addr is short for jodis_coordinator_addr
    #   3. proxy will be registered as node:
    #        if jodis_compatible = true (not suggested):
    #          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)
    #        or else
    #          /jodis/{PRODUCT_NAME}/proxy-{HASHID}
    jodis_name = "zookeeper"
    jodis_addr = "192.168.1.202:2181"
    jodis_timeout = "20s"
    jodis_compatible = false
    
    # Set datacenter of proxy.
    proxy_datacenter = "default"
    
    # Set max number of alive sessions.
    proxy_max_clients = 1000
    
    # Set max offheap memory size. (0 to disable)
    proxy_max_offheap_size = "1024mb"
    
    # Set heap placeholder to reduce GC frequency.
    proxy_heap_placeholder = "256mb"
    
    # Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
    backend_ping_period = "5s"
    
    # Set backend recv buffer size & timeout.
    backend_recv_bufsize = "128kb"
    backend_recv_timeout = "30s"
    
    # Set backend send buffer & timeout.
    backend_send_bufsize = "128kb"
    backend_send_timeout = "30s"
    
    # Set backend pipeline buffer size.
    backend_max_pipeline = 20480
    
    # Set backend never read replica groups, default is false
    backend_primary_only = false
    
    # Set backend parallel connections per server
    backend_primary_parallel = 1
    backend_replica_parallel = 1
    
    # Set backend tcp keepalive period. (0 to disable)
    backend_keepalive_period = "75s"
    
    # Set number of databases of backend.
    backend_number_databases = 16
    
    # If there is no request from client for a long time, the connection will be closed. (0 to disable)
    # Set session recv buffer size & timeout.
    session_recv_bufsize = "128kb"
    session_recv_timeout = "30m"
    
    # Set session send buffer size & timeout.
    session_send_bufsize = "64kb"
    session_send_timeout = "30s"
    
    # Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
    # Set session pipeline buffer size.
    session_max_pipeline = 10000
    
    # Set session tcp keepalive period. (0 to disable)
    session_keepalive_period = "75s"
    
    # Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
    session_break_on_failure = false
    
    # Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
    metrics_report_server = ""
    metrics_report_period = "1s"
    
    # Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
    metrics_report_influxdb_server = ""
    metrics_report_influxdb_period = "1s"
    metrics_report_influxdb_username = ""
    metrics_report_influxdb_password = ""
    metrics_report_influxdb_database = ""
    
    # Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
    metrics_report_statsd_server = ""
    metrics_report_statsd_period = "1s"
    metrics_report_statsd_prefix = ""

## 启动服务

### 启动codis-dashboard

./admin/codis-dashboard-admin.sh start
     tail -100 ./log/codis-dashboard.log.
     
### 启动codis-proxy

./admin/codis-proxy-admin.sh start　6401
    tail -100 ./log/codis-proxy.log.

###　启动codis-server

./admin/codis-server-admin.sh start 6301
    tail -100 /tmp/redis_6301.log 
    
### 启动codis-fe

./admin/codis-fe-admin.sh start
    tail -100 ./log/codis-fe.log.

## 主备切换

### 启动sentinel

./admin/codis-sentinel-admin.sh start 6501

## 基准测试(codis vs twemproxy)

### 测试环境

codis 4cpu twemproxy 1cpu
redis 2主+2备, 纯内存模式
单机伪集群

### memtier_benchmark(set/get)

./memtier_benchmark -P redis --ratio=1:1  --test-time 30 -d 256  -c 50 -t 10  --pipeline 500 -p 6401 -a 'stream!23$'

**codis**
![title](/api/file/getImage?fileId=598dd9230f93bd0545000391)

**Twemproxy**
![title](/api/file/getImage?fileId=598dd9890f93bd0545000392)

### memtier_benchmark(set)

./memtier_benchmark -P redis --ratio=1:0  --test-time 30 -d 256  -c 50 -t 10  --pipeline 500 -p 6401 -a 'stream!23$'  
    
**codis**
![title](/api/file/getImage?fileId=598dda120f93bd0545000393)

**Twemproxy**
![title](/api/file/getImage?fileId=598dda4b0f93bd0545000394)

### memtier_benchmark(get)

./memtier_benchmark -P redis --ratio=0:1  --test-time 30 -d 256  -c 50 -t 10  --pipeline 500 -p 6401 -a 'stream!23$'  
    
**codis**
![title](/api/file/getImage?fileId=598ddaac0f93bd0545000395)

**Twemproxy**
![title](/api/file/getImage?fileId=598ddae00f93bd0545000396)

### redis_benchmark

## 自动化部署

### 配置python

#install pip
    yum -y install epel-release
    yum -y install python-pip
    yum clean all
    
    pip install setuptools
    pip install pyyaml
    pip install jinja2

### 安装sshpass

https://nchc.dl.sourceforge.net/project/sshpass/sshpass/1.06/sshpass-1.06.tar.gz
    tar zxvf sshpass-1.06.tar.gz 
    export PATH=$PATH:/path/to/sshpass-1.06

###配置ansible

git clone git://github.com/ansible/ansible.git -b stable-2.3
    cd ./ansible
    source ./hacking/env-setup
    cd $codis_dir/ansible

###集群配置

vi group_vars

---
    group: groupxxx
    owner: userxxx
    
    codis_dir: /path/to/codis-ansible/codis
    codis_install_dir: /path/to/codis3
    
    
    codis_proxy_ports: [port01,port02]
    codis_server_ports: [port01,port02]
    codis_server_slave_ports: [port01,port02]
    codis_sentinel_ports: [port01,port02]
    redis_password: redis_pass_xxx
    
    
    codis_zookeeper_addr: "x.x.x.x:xxxx"
    codis_dashboard_addr: "x.x.x.x:xxxxx"
    codis_fe_addr: "x.x.x.x:xxxx"
    
    codis_proxy_ncpu: 2
    codis_proxy_max_cpu: 2
    codis_proxy_log_level: WARN
    codis_fe_log_level: WARN
    
    
    ssh_user: userxxx
    ssh_pass_001: passxxx
    ssh_pass_002: passxxx
    
    cluster_name: cluster01
    
    
    vi hosts
    
    [codis-dashboard-servers]
    x.x.x.x  ansible_ssh_user="{{ssh_user}}" ansible_ssh_pass="{{ssh_pass_001}}" ansible_ssh_port=22
    [codis-fe-servers]
    x.x.x.x  ansible_ssh_user="{{ssh_user}}" ansible_ssh_pass="{{ssh_pass_001}}" ansible_ssh_port=22
    [codis-proxy-servers]
    x.x.x.x  ansible_ssh_user="{{ssh_user}}" ansible_ssh_pass="{{ssh_pass_001}}" ansible_ssh_port=22
    [codis-servers]
    x.x.x.x  ansible_ssh_user="{{ssh_user}}" ansible_ssh_pass="{{ssh_pass_001}}" ansible_ssh_port=22
    [redis-sentinel-servers]
    x.x.x.x  ansible_ssh_user="{{ssh_user}}" ansible_ssh_pass="{{ssh_pass_001}}" ansible_ssh_port=22

###部署codis

./ansible-deploy.sh
    ./ansible-start.sh
    ./ansible-stop.sh
    ./ansible-clean.sh

----------
## 附件

[codis3.2.0-go1.7.5-linux.tar.gz](https://note.louyj.com/api/file/getAttach?fileId=598ddba30f93bd0545000397)

[codis-ansible.tar.gz](https://note.louyj.com/api/file/getAttach?fileId=5999a44b0f93bd7b12000051)

----------

Navigation

Recent Posts

Friend Links