CentOS下批量部署Nginx环境步骤详解
安装环境:
System: Centos 6.3
Salt master: salt-master.example.com
Salt minion: salt-client01.example.com
Salt minion: salt-client02.example.com
一. 主控端配置
1. 配置master基本参数
# vi /etc/salt/master
添加:
nodegroups:
webgroup1: 'salt-client01.example.com'
webgroup2: 'salt-client02.example.com'
file_roots:
base:
- /srv/salt
pillar_roots:
base:
- /srv/pillar
Tip: 这里的nodegroups里的分组会应用到随后的pillar脚本下, 用来区分不同的salt-minion使用相应的pillar参数, 从而定制不同的minion使用不同的配置方案.
重启服务
# /etc/init.d/salt-master restart
2. 动态配置客户端系统连接数
使用python脚本编写grains_module, 实现动态配置被控主机连接数(CLI可用"limit -n"查询该值), 以便随后的Nginx配置文件中的worker_rlimit_nofile, worker_connections可动态调用脚本中max_open_file的参数值.
Tip: "ulimit -n" 是用来查询当前linux系统的最大打开文件数, 默认为1024
也可用"ulimit -a"来查看其他相关参数
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7819
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
暂时修改当前session的参数值
# ulimit -n 2048
永久修改需修改该参数值
# vi /etc/security/limits.conf
添加:
root soft nofile 2048
root hard nofile 2048
# vi /etc/pam.d/common-session
添加:
session required pam_limits.so
重启生效.
脚本具体配置如下:
# mkdir -p /srv/salt/_grains
# vi /srv/salt/_grains/nginx_config.py
#!/usr/bin/env python
import os,sys,commands
# return Nginx config grains value
def NginxGrains():
grains = {}
max_open_file=65536
try:
getulimit=commands.getstatusoutput('source /etc/profile && ulimit -n')
except Exception,e:
pass
if getulimit[0]==0:
max_open_file=int(getulimit[1])
grains['max_open_file'] = max_open_file
return grains
Tip: 该脚本会同步到远程后执行, 脚本实际就是获取并返回当前主机的最大打开数值, 最终返回值会赋予字典 grains['max_open_file']
同步grains模块:
# salt '*' saltutil.sync_all
刷新模块(让minion编译模块)
# salt '*' sys.reload_modules
验证max_open_file key的value
# salt '*' grains.item max_open_file
Tip: 这里笔者测试更改客户端最大文件打开值时发现了一个问题, 无论客户端如何更改这个值, 在验证key value时终会显示最早的系统初始值1024, 翻阅了大量文档, 发现minion端会将所有服务端的推送保存在(/var/cache/salt/minion), 这里删除这个缓存目录并重启salt-minion, 让其生成新的缓存目录, 从新同步grains模块, 新的vaule就会生效.
# salt '*' cmd.run 'rm -rf /var/cache/salt/minion && /etc/init.d/salt-minion restart'
3. 配置pillar
1). 定义入口sls
# vi /srv/pillar/top.sls
base:
webgroup1:
- match: nodegroup
- nginx.webserver1
webgroup2:
- match: nodegroup
- nginx.webserver2
Tip: 这里定义webgroup1的所有minions会使用pillar/nginx/webserver1.sls脚本, webgroup2所有minions使用pillar/nginx/webserver2.sls脚本, match: nodegroup 定义webgroup1, webgroup2匹配nodegroup组.
2). 定义webserver1, webserver2 sls
# mkdir /srv/pillar/nginx
# vi /srv/pillar/nginx/webserver1.sls
nginx:
hostname: webserver1
name: nginx
root: /www
source: salt://nginx/nginx.conf
file: /etc/nginx/nginx.conf
user: root
group: root
mode: 644
template: jinja
status: installed
# vi /srv/pillar/nginx/webserver2.sls
nginx:
hostname: webserver2
name: nginx
root: /data
source: salt://nginx/nginx.conf
file: /etc/nginx/nginx.conf
user: root
group: root
mode: 666
template: jinja
status: installed
3). 查看配置结果
# salt '*' pillar.data nginx
salt-client02.example.com:
----------
nginx:
----------
file:
/etc/nginx/nginx.conf
group:
root
hostname:
webserver2
mode:
666
name:
nginx
root:
/data
source:
salt://nginx/nginx.conf
status:
installed
template:
jinja
user:
root
salt-client01.example.com:
----------
nginx:
----------
file:
/etc/nginx/nginx.conf
group:
root
hostname:
webserver1
mode:
644
name:
nginx
root:
/www
source:
salt://nginx/nginx.conf
status:
installed
template:
jinja
user:
root
Tip: 可以通过字典 pillar['nginx']['root'] 在配置文件中调用
4. 配置state
1). 定义入口sls
# vi /srv/salt/top.sls
base:
'*':
- env
- nginx.deploy
2). 定义env, nginx服务状态管理配置sls
通用系统环境配置
# vi /srv/salt/env.sls
policycoreutils:
pkg.installed
policycoreutils-python:
pkg.installed
{% if grains['selinux']['enabled'] %}
setselinux:
selinux.mode:
- name: 'permissive'
- require:
- pkg: policycoreutils
- pkg: policycoreutils-python
/etc/sysconfig/selinux:
file.replace:
- pattern: 'SELINUX=w+'
- repl: 'SELINUX=disabled'
{% endif %}
iptables:
pkg:
- installed
iptables.flush:
- table: 'filter'
- save: True
cmd.run:
- name: 'service iptables save'
- require:
- pkg: iptables
Tip: 使用setenforce需要安装2个python依赖包, 通过判断系统默认的SELINUX是否打开, 从而确定是否关闭SELINUX, 如果不作此判断, 系统SELINUX若关闭, 部署时会报错.
NGINX配置
# mkdir /srv/salt/nginx
# vi /srv/salt/nginx/deploy.sls
{% if 'nginx' in pillar %}
{% set item = pillar['nginx'] %}
{{ item['name'] }}:
pkg:
- {{ item['status'] }}
file.managed:
- source: {{ item['source'] }}
- name: {{ item['file'] }}
- user: {{ item['user'] }}
- group: {{ item['group'] }}
- mode: {{ item['mode'] }}
- template: {{ item['template'] }}
service.running:
- enable: True
- reload: True
- watch:
- file: {{ item['file'] }}
- pkg: {{ item['name'] }}
{{ item['root'] }}:
file.directory:
- user: {{ item['name'] }}
- group: {{ item['name'] }}
- mode: 755
- makedirs: True
- recurse:
- user
- group
- mode
{% endif %}
Tip: "source: {{ item['source'] }}"等价于"source: {{ pillar['nginx']['source'] }}"为配置模板文件位置
"-enable: True" 等价于 "chkconfig nginx on"
"-reload True" 等价于 "service nginx reload", 若不加则默认执行"service nginx restart"
"-wotch -file:" 检查 /etc/nginx/nginx.conf是否发生变化
"-watch -pkg" 确保nginx安装成功.
"{{ item['root'] }}: file.directory:" 等价于mkdir /www
3). 定义Nginx配置文件（引用jinja模板）
# vi /srv/salt/nginx/nginx.conf
# For more information on configuration, see:
{% set item = pillar['nginx'] %}
user nginx;
worker_processes {{ grains['num_cpus'] }};
{% if grains['num_cpus'] == 2 %}
worker_cpu_affinity 01 10;
{% elif grains['num_cpus'] == 4 %}
worker_cpu_affinity 1000 0100 0010 0001;
{% elif grains['num_cpus'] >= 8 %}
worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000;
{% else %}
worker_cpu_affinity 1000 0100 0010 0001;
{% endif %}
worker_rlimit_nofile {{ grains['max_open_file'] }};
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;
events {
worker_connections {{ grains['max_open_file'] }};
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
# Load config files from the /etc/nginx/conf.d directory
# The default server is in conf.d/default.conf
#include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
server_name _;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root {{ item['root'] }};
index index.html index.htm;
}
error_page 404 /404.html;
location = /404.html {
root /usr/share/nginx/html;
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
}
Tip:
worker_processes参数采用grains['num_cpus'] 上报值（与设备CPU核数一致）；
worker_cpu_affinity分配多核CPU根据当前设备核数进行匹配，分别为248其它核；
worker_rlimit_nofile参数与grains['max_open_file'] 获取的系统ulimit -n一致；
worker_connections 参数理论上为grains['max_open_file']；
root参数为定制的pillar['nginx']['root']值。
4). 执行最终state配置, 将master的所有nginx配置部署到客户端
# salt '*' state.highstate
Tip: 可使用salt '*' -l debug state.highstate 查看部署过程中是否有错误
salt-client02.example.com:
----------
ID: policycoreutils
Function: pkg.installed
Result: True
Comment: Package policycoreutils is already installed.
Started: 14:35:20.183439
Duration: 1165.132 ms
Changes:
----------
ID: policycoreutils-python
Function: pkg.installed
Result: True
Comment: Package policycoreutils-python is already installed.
Started: 14:35:21.348902
Duration: 0.875 ms
Changes:
----------
ID: iptables
Function: pkg.installed
Result: True
Comment: Package iptables is already installed.
Started: 14:35:21.349904
Duration: 0.47 ms
Changes:
----------
ID: iptables
Function: iptables.flush
Result: True
Comment: Flush iptables rules in filter table chain ipv4 family
Started: 14:35:21.351883
Duration: 16.006 ms
Changes:
----------
locale:
iptables
----------
ID: iptables
Function: cmd.run
Name: service iptables save
Result: True
Comment: Command "service iptables save" run
Started: 14:35:21.371349
Duration: 50.934 ms
Changes:
----------
pid:
3352
retcode:
0
stderr:
stdout:
iptables: Saving firewall rules to /etc/sysconfig/iptables: ?[60G[?[0;32m OK ?[0;39m]
----------
ID: nginx
Function: pkg.installed
Result: True
Comment: The following packages were installed/updated: nginx
Started: 14:35:21.422715
Duration: 19128.275 ms
Changes:
----------
nginx:
----------
new:
1.0.15-12.el6
old:
----------
ID: nginx
Function: file.managed
Name: /etc/nginx/nginx.conf
Result: True
Comment: File /etc/nginx/nginx.conf updated
Started: 14:35:40.560042
Duration: 40.855 ms
Changes:
----------
diff:
---
+++
@@ -1,42 +1,70 @@
-# For more information on configuration, see:
-# * Official English Documentation: http://nginx.org/en/docs/
-# * Official Russian Documentation: http://nginx.org/ru/docs/
-
-user nginx;
-worker_processes 1;
-
-error_log /var/log/nginx/error.log;
-#error_log /var/log/nginx/error.log notice;
-#error_log /var/log/nginx/error.log info;
-
-pid /var/run/nginx.pid;
+# For more information on configuration, see:
-events {
- worker_connections 1024;
-}
-
-
-http {
- include /etc/nginx/mime.types;
- default_type application/octet-stream;
-
- log_format main '$remote_addr - $remote_user [$time_local] "$request" '
- '$status $body_bytes_sent "$http_referer" '
- '"$http_user_agent" "$http_x_forwarded_for"';
-
- access_log /var/log/nginx/access.log main;
-
- sendfile on;
- #tcp_nopush on;
-
- #keepalive_timeout 0;
- keepalive_timeout 65;
-
- #gzip on;
-
- # Load config files from the /etc/nginx/conf.d directory
- # The default server is in conf.d/default.conf
- include /etc/nginx/conf.d/*.conf;
-
-}
+user nginx;
+worker_processes 1;
+
+worker_cpu_affinity 1000 0100 0010 0001;
+
+worker_rlimit_nofile 1024;
+
+error_log /var/log/nginx/error.log;
+#error_log /var/log/nginx/error.log notice;
+#error_log /var/log/nginx/error.log info;
+
+pid /var/run/nginx.pid;
+
+events {
+ worker_connections 1024;
+}
+
+
+http {
+ include /etc/nginx/mime.types;
+ default_type application/octet-stream;
+
+ log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+ '$status $body_bytes_sent "$http_referer" '
+ '"$http_user_agent" "$http_x_forwarded_for"';
+
+ access_log /var/log/nginx/access.log main;
+
+ sendfile on;
+ #tcp_nopush on;
+
+ #keepalive_timeout 0;
+ keepalive_timeout 65;
+
+ #gzip on;
+
+ # Load config files from the /etc/nginx/conf.d directory
+ # The default server is in conf.d/default.conf
+ #include /etc/nginx/conf.d/*.conf;
+ server {
+ listen 80 default_server;
+ server_name _;
+
+ #charset koi8-r;
+
+ #access_log logs/host.access.log main;
+
+ location / {
+ root /data;
+ index index.html index.htm;
+ }
+
+ error_page 404 /404.html;
+ location = /404.html {
+ root /usr/share/nginx/html;
+ }
+
+ # redirect server error pages to the static page /50x.html
+ #
+ error_page 500 502 503 504 /50x.html;
+ location = /50x.html {
+ root /usr/share/nginx/html;
+ }
+
+ }
+
+}
mode:
0666
----------
ID: nginx
Function: service.running
Result: True
Comment: Service nginx has been enabled, and is running
Started: 14:35:40.608381
Duration: 320.864 ms
Changes:
----------
nginx:
True
----------
ID: /data
Function: file.directory
Result: True
Comment: Directory /data is in the correct state
Started: 14:35:40.929607
Duration: 2.265 ms
Changes:
Summary
------------
Succeeded: 9 (changed=5)
Failed: 0
------------
Total states run: 9
salt-client01.example.com:
----------
ID: policycoreutils
Function: pkg.installed
Result: True
Comment: Package policycoreutils is already installed.
Started: 18:14:48.158376
Duration: 1211.452 ms
Changes:
----------
ID: policycoreutils-python
Function: pkg.installed
Result: True
Comment: Package policycoreutils-python is already installed.
Started: 18:14:49.370171
Duration: 0.787 ms
Changes:
----------
ID: setselinux
Function: selinux.mode
Name: permissive
Result: True
Comment: SELinux is already in Permissive mode
Started: 18:14:49.372833
Duration: 0.652 ms
Changes:
----------
ID: /etc/sysconfig/selinux
Function: file.replace
Result: True
Comment: No changes needed to be made
Started: 18:14:49.377284
Duration: 2.507 ms
Changes:
----------
ID: iptables
Function: pkg.installed
Result: True
Comment: Package iptables is already installed.
Started: 18:14:49.379941
Duration: 0.597 ms
Changes:
----------
ID: iptables
Function: iptables.flush
Result: True
Comment: Flush iptables rules in filter table chain ipv4 family
Started: 18:14:49.381900
Duration: 21.393 ms
Changes:
----------
locale:
iptables
----------
ID: iptables
Function: cmd.run
Name: service iptables save
Result: True
Comment: Command "service iptables save" run
Started: 18:14:49.406906
Duration: 342.078 ms
Changes:
----------
pid:
3124
retcode:
0
stderr:
stdout:
iptables: Saving firewall rules to /etc/sysconfig/iptables: ?[60G[?[0;32m OK ?[0;39m]
----------
ID: nginx
Function: pkg.installed
Result: True
Comment: The following packages were installed/updated: nginx
Started: 18:14:49.749415
Duration: 23737.719 ms
Changes:
----------
nginx:
----------
new:
1.0.15-12.el6
old:
----------
ID: nginx
Function: file.managed
Name: /etc/nginx/nginx.conf
Result: True
Comment: File /etc/nginx/nginx.conf updated
Started: 18:15:13.490651
Duration: 52.469 ms
Changes:
----------
diff:
---
+++
@@ -1,42 +1,70 @@
-# For more information on configuration, see:
-# * Official English Documentation: http://nginx.org/en/docs/
-# * Official Russian Documentation: http://nginx.org/ru/docs/
-
-user nginx;
-worker_processes 1;
-
-error_log /var/log/nginx/error.log;
-#error_log /var/log/nginx/error.log notice;
-#error_log /var/log/nginx/error.log info;
-
-pid /var/run/nginx.pid;
+# For more information on configuration, see:
-events {
- worker_connections 1024;
-}
-
-
-http {
- include /etc/nginx/mime.types;
- default_type application/octet-stream;
-
- log_format main '$remote_addr - $remote_user [$time_local] "$request" '
- '$status $body_bytes_sent "$http_referer" '
- '"$http_user_agent" "$http_x_forwarded_for"';
-
- access_log /var/log/nginx/access.log main;
-
- sendfile on;
- #tcp_nopush on;
-
- #keepalive_timeout 0;
- keepalive_timeout 65;
-
- #gzip on;
-
- # Load config files from the /etc/nginx/conf.d directory
- # The default server is in conf.d/default.conf
- include /etc/nginx/conf.d/*.conf;
-
-}
+user nginx;
+worker_processes 1;
+
+worker_cpu_affinity 1000 0100 0010 0001;
+
+worker_rlimit_nofile 1024;
+
+error_log /var/log/nginx/error.log;
+#error_log /var/log/nginx/error.log notice;
+#error_log /var/log/nginx/error.log info;
+
+pid /var/run/nginx.pid;
+
+events {
+ worker_connections 1024;
+}
+
+
+http {
+ include /etc/nginx/mime.types;
+ default_type application/octet-stream;
+
+ log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+ '$status $body_bytes_sent "$http_referer" '
+ '"$http_user_agent" "$http_x_forwarded_for"';
+
+ access_log /var/log/nginx/access.log main;
+
+ sendfile on;
+ #tcp_nopush on;
+
+ #keepalive_timeout 0;
+ keepalive_timeout 65;
+
+ #gzip on;
+
+ # Load config files from the /etc/nginx/conf.d directory
+ # The default server is in conf.d/default.conf
+ #include /etc/nginx/conf.d/*.conf;
+ server {
+ listen 80 default_server;
+ server_name _;
+
+ #charset koi8-r;
+
+
