Redis之codis3.0.1集群简单实现步骤

Codis 由四部分组成:
Codis Proxy (codis-proxy) 代理服务器做请求转发，codis-proxy 本身实现了 Redis 协议，codis-proxy 本身是无状态的.可以执行多个
Codis Dashboard (codis-config) codis工具，包括添加删除节点，并且自带一个dashboard，点一点即可
Codis Redis (codis-server) 基于2.8.13开发的分支，只能使用codis-server
ZooKeeper/Etcd Codis 依赖 ZooKeeper 来存放数据路由表和 codis-proxy 节点的元信息, codis-config 发起的命令都会通过 ZooKeeper 同步到各个存活的 codis-proxy.
architecture (1).png
Codis 支持按照 Namespace 区分不同的产品, 拥有不同的 product name 的产品, 各项配置都不会冲突

1，codis由go语言写的，所以需要安装go语言包

[root@yum-down local]# wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
[root@yum-down local]# tar -zxvf go1.6.linux-amd64.tar.gz
[root@yum-down local]# cd /usr/local/go/src/
[root@yum-down src]# go version
go version go1.6 linux/amd64
[root@yum-down src]# mkdir /gopath/src/golang.org/x
[root@yum-down src]# go get github.com/tools/godep
[root@yum-down src]# vim /etc/profile
export PATH=$PATH:/usr/local/go/bin
export GOROOT=/usr/local/go
export PATH=$PATH:/gopath/bin
[root@yum-down local]# source /etc/profile
也可以yum安装：

[root@yum-down local]# yum -y install gcc gcc-c++ make git wget go
[root@yum-down codis]# cat /etc/profile.d/go.sh
export GOPATH=/gopath
export PATH=$GOPATH/bin:$JAVA_HOME/bin:$PATH
2，安装java

[root@yum-down local]# wget http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.rpm
[root@yum-down local]# rpm -ivh jdk-8u73-linux-x64.rpm
Preparing...                ########################################### [100%]
   1:jdk1.8.0_73            ########################################### [100%]
Unpacking JAR files...
    tools.jar...
    plugin.jar...
    javaws.jar...
    deploy.jar...
    rt.jar...
    jsse.jar...
    charsets.jar...
    localedata.jar...
    jfxrt.jar...
[root@yum-down opt]# cat /etc/profile.d/java
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin/:$PATH
export PATH=$PATH:/gopath/bin
[root@yum-down opt]# . /etc/profile.d/java
[root@yum-down local]# java -version
java version "1.8.0_73"
Java(TM) SE Runtime Environment (build 1.8.0_73-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.73-b02, mixed mode)
[root@yum-down local]#
3，zookeeper下载地址

[root@yum-down local]# wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
[root@yum-down local]# tar xf zookeeper-3.4.6.tar.gz
[root@yum-down local]# ln -s  zookeeper-3.4.6 zookeeper
[root@yum-down zookeeper]# mkdir /opt/{zk1,zk2,zk3}
复制配置文件：

[root@yum-down local]# cp /usr/local/zookeeper/conf/zoo_sample.cfg /opt/zoo.cfg
编辑配置文件，修改datadir.并且添加server（由于是一台机器，端口则不能相同），如下：

dataDir=/opt/zk1
clientPort=2181
server.1=10.10.0.250:2887:3887
server.2=10.10.0.250:2888:3888
server.3=10.10.0.250:2889:3889
将配置文件复制到每个目录下：

[root@yum-down local]# cp /opt/zoo.cfg /opt/zk1/zk1.cfg
[root@yum-down local]# cp /opt/zoo.cfg /opt/zk2/zk2.cfg
[root@yum-down local]# cp /opt/zoo.cfg /opt/zk3/zk3.cfg
修改完后如下：

[root@yum-down local]# grep '^[a-z]' /opt/zk1/zk1.cfg
tickTime=2000     心跳时间
initLimit=10      心跳探测间隔数
syncLimit=5    
dataDir=/opt/zk1
clientPort=2181
server.1=10.10.0.250:2887:3887  集群1
server.2=10.10.0.250:2888:3888  集群2
server.3=10.10.0.250:2889:3889  集群3
修改后如下：

[root@yum-down local]# grep '^[a-z]' /opt/zk2/zk2.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zk2
clientPort=2182
server.1=10.10.0.250:2887:3887
server.2=10.10.0.250:2888:3888
server.3=10.10.0.250:2889:3889
[root@yum-down local]# grep '^[a-z]' /opt/zk3/zk3.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zk3
clientPort=2183
server.1=10.10.0.250:2887:3887
server.2=10.10.0.250:2888:3888
server.3=10.10.0.250:2889:3889
[root@yum-down local]#
4，在一台机器上跑的话，在三个目录，并且创建id

[root@yum-down local]# cd /opt/
[root@yum-down opt]# echo "1" > zk1/myid
[root@yum-down opt]# echo "2" > zk2/myid
[root@yum-down opt]# echo "3" > zk3/myid
＊＊＊＊＊＊＊＊＊＊＊＊zookeeper环境变量＊＊＊＊＊＊＊＊＊

[root@yum-down opt]#　echo -e "export ZOOKEEPER=/usr/local/zookeeper nexport  PATH=$ZOOKEEPER/bin:$PATH" >> /etc/profile &&　source /etc/profile
＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊
启动三个zookeeper

[root@yum-down local]# /usr/local/zookeeper/bin/zkServer.sh start /opt/zk1/zk1.cfg
ZooKeeper JMX enabled by default
Using config: /opt/zk1/zk1.cfg
Starting zookeeper ... STARTED
[root@yum-down local]# /usr/local/zookeeper/bin/zkServer.sh start /opt/zk2/zk2.cfg
ZooKeeper JMX enabled by default
Using config: /opt/zk2/zk2.cfg
Starting zookeeper ... STARTED
[root@yum-down local]# /usr/local/zookeeper/bin/zkServer.sh start /opt/zk3/zk3.cfg
ZooKeeper JMX enabled by default
Using config: /opt/zk3/zk3.cfg
Starting zookeeper ... STARTED
[root@yum-down local]#
查看是否已经自动选举：

[root@yum-down opt]# /usr/local/zookeeper/bin/zkServer.sh status /opt/zk3/zk3.cfg
JMX enabled by default
Using config: /opt/zk3/zk3.cfg
Mode: follower  跟随者
[root@yum-down opt]# /usr/local/zookeeper/bin/zkServer.sh status /opt/zk2/zk2.cfg
JMX enabled by default
Using config: /opt/zk2/zk2.cfg
Mode: leader  领导
[root@yum-down opt]# /usr/local/zookeeper/bin/zkServer.sh status /opt/zk1/zk1.cfg
JMX enabled by default
Using config: /opt/zk1/zk1.cfg
Mode: follower  跟随者
[root@yum-down opt]#
连接测试：

[root@yum-down opt]# /usr/local/zookeeper/bin/zkCli.sh -server 10.10.0.250:2181
[zk: 10.10.0.250:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: 10.10.0.250:2181(CONNECTED) 1] ls /zookeeper
[quota]
[zk: 10.10.0.250:2181(CONNECTED) 2]
下载codis

[root@yum-down ~]# go get -u -d github.com/CodisLabs/codis
package github.com/CodisLabs/codis: no buildable Go source files in /opt/gopath/src/github.com/CodisLabs/codis
[root@yum-down local]# cd /gopath/src/github.com/CodisLabs/codis/
[root@yum-down codis]# make
[root@yum-down codis]# make gotest
codis-config、codis-proxy、codis-server三个可执行文件已经生成
[root@yum-down bin]# ls
assets  codis-config  codis-proxy  codis-server
执行bin/codis-config dashboard，将会看到一个页面
codis.png

[root@yum-down codis]# bin/codis-config dashboard
2016/03/04 07:22:03 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2016/03/04 07:22:03 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "10.10.0.250:18087", "pid": 33434}
2016/03/04 07:22:03 dashboard.go:144: [WARN] ********** Attention **********
2016/03/04 07:22:03 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2016/03/04 07:22:03 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually
2016/03/04 07:22:03 dashboard.go:147: [WARN] *******************************
2016/03/04 07:23:07 dashboard_apis.go:88: [ERROR] get server groups failed
[error]: zk: node does not exist
    1   /gopath/src/github.com/CodisLabs/codis/pkg/models/server_group.go:110
            github.com/CodisLabs/codis/pkg/models.ServerGroups
    0   /gopath/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard_apis.go:86
            main.apiGetServerGroupList
        ... ...
[stack]:
    0   /gopath/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard_apis.go:88
            main.apiGetServerGroupList
        ... ...
启动两个redis（确保redis已经安装）
创建两个redis

[root@yum-down opt]# mkdir date1
[root@yum-down opt]# mkdir date2
[root@yum-down opt]# cp /etc/redis/6379.conf ./
[root@yum-down opt]# vim 6379.conf
将端口，pid修改，如：8001

[root@yum-down opt]# grep "8001" 6379.conf
pidfile /var/run/redis_8001.pid
# Accept connections on the specified port, default is 8001.
port 8001
dir /opt/date1
# cluster-config-file nodes-8001.conf
[root@yum-down opt]# cp 6379.conf date1/8001.conf
8002

[root@yum-down opt]# grep "8002" 6379.conf
pidfile /var/run/redis_8002.pid
# Accept connections on the specified port, default is 8002.
port 8002
dir /opt/date2
# cluster-config-file nodes-8002.conf
[root@yum-down opt]#
启动

[root@yum-down opt]# /usr/local/redis/bin/redis-server /opt/date1/8001.conf
[root@yum-down opt]# /usr/local/redis/bin/redis-server /opt/date2/8002.conf
[root@yum-down opt]# ps aux|grep redis
root      36449  0.1  0.7 137448  7484 ?        Ssl  07:38   0:00 /usr/local/redis/bin/redis-server *:6379             
root      36485  0.0  0.7 137448  7436 ?        Ssl  07:48   0:00 /usr/local/redis/bin/redis-server *:8001             
root      36491  0.1  0.7 137448  7436 ?        Ssl  07:48   0:00 /usr/local/redis/bin/redis-server *:8002
启动后手动添加即可
cordis-crete-1.png
连接后查看

[root@yum-down opt]# /usr/local/redis/bin/redis-cli -h 10.10.0.250 -p 8002
role:slave
master_host:10.10.0.250
master_port:8001
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:85
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
手动提交提升主后，将会被孤立，主从将消失offline：
codis-提升主从.png

10.10.0.250:8002>
# Replication
role:master
connected_slaves:0
master_repl_offset:239
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
和之前的版本不同的是codis3.0.1的脚本放在CodisLabs/codis/bin下：

[root@yum-down bin]# ls
assets  codis-config  codis-proxy  codis-server
[root@yum-down bin]# ll
total 37520
drwxr-xr-x 4 root root     4096 Mar  4 06:30 assets
-rwxr-xr-x 1 root root 16141960 Mar  4 06:30 codis-config
-rwxr-xr-x 1 root root 15961984 Mar  4 06:29 codis-proxy
-rwxr-xr-x 1 root root  6311107 Mar  4 06:30 codis-server
[root@yum-down bin]# pwd
/gopath/src/github.com/CodisLabs/codis/bin
配置文件仍然是config.ini

coordinator=zookeeper
zk=10.10.0.250:2181
product=test
dashboard_addr=10.10.0.250:18087
并且他还可以添加服务的slot范围
数据迁移：
用gitlab作者的原话是：数据迁移的最小单位是 key, 我们在 codis redis 中添加了一些指令, 实现基于key的迁移, 如 SLOTSMGRT等 (命令列表), 每次会将特定 slot 一个随机的 key 发送给另外一个 codis redis 实例, 这个命令会确认对方已经接收, 同时删除本地的这个 k-v 键值, 返回这个 slot 的剩余 key 的数量, 整个操作是原子的.
支持动态的根据实例内存, 自动对slot进行迁移, 以均衡数据分布.

他有很多命令不支持：https://github.com/CodisLabs/codis/blob/3.0.1/doc/unsupported_cmds.md
至于codis-proxy设计到zookeeper，如果对zookeeper不熟悉的话，有很多问题是无法解决的

初始化 slots

[root@yum-down codis]# bin/codis-config slot init
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]#
设置 server group 服务的 slot 范围
添加group2

[root@yum-down codis]# bin/codis-config server add 2 10.10.0.250:6379 master
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]# bin/codis-config server add 2 10.10.0.250:8001 slave
{
  "msg": "OK",
  "ret": 0
}
添加group3
[root@yum-down codis]# bin/codis-config server add 3 10.10.0.250:8001 master
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]# bin/codis-config server add 3 10.10.0.250:8002 slave
{
  "msg": "OK",
  "ret": 0
}
设置server group服务器的slot范围

[root@yum-down codis]# bin/codis-config slot range-set 0 511 1 online
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]# bin/codis-config slot range-set 512 1023 2 online
{
  "msg": "OK",
  "ret": 0
}
codis-三个.png
启动codis-proxy

[root@yum-down codis]# bin/codis-proxy -c config.ini -L ./log/proxy.log  --cpu=8 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000
  _____  ____    ____/ /  (_)  _____
 / ___/ / __   / __  /  / /  / ___/
/ /__  / /_/ / / /_/ /  / /  (__  )
___/  ____/  __,_/  /_/  /____/

[root@yum-down codis]#
数据迁移：

[root@yum-down codis]# bin/codis-config slot migrate 0 511 2 --delay=10
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]#
codis到此为止，了解即可！