本文主要是介绍nginx做调度(自身用dns轮询保证高可用),varnish做后端的cache,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!
Nginx + varnish 构建高可用CDN节点集群
作者:随风 发布于:2012-6-19 10:29 Tuesday 分类:linux运维
OSAPUB 出品的集群方案,旨在抛砖引玉,给广大朋友提供高可用CDN节点集群方案
转载请标明出去:http://bbs.osapub.com/thread-3370-1-1.html
一、 环境描述
Linux server A (CentOS release 5.8 Final) 实IP:192.168.4.97 虚IP:192.168.4.96
Linux server B (CentOS release 5.8 Final) 实IP:192.168.4.99 虚IP:192.168.4.98
域名环境(DNS轮询解析到虚IP):
server.osapub.com 192.168.4.96
server.osapub.com 192.168.4.98
二、 简单架构示意图
描述:前端两台NGINX做DNS轮询,通过虚IP漂移+HA监控脚本相结合,实现前端两台NGINX高可用,利用NGINX反向代理功能对后端varnish实现高可用集群。
三、软件环境搭建
3.1 编译安装nginx
006 | mkdir -p /dist/{dist,src} |
010 | wget http://bbs.osapub.com/down/google-perftools-1.8.3. tar .gz &> /dev/null |
011 | wget http://bbs.osapub.com/down/libunwind-0.99. tar .gz &> /dev/null |
012 | wget http://bbs.osapub.com/down/pcre-8.01. tar .gz &> /dev/null |
013 | wget http://bbs.osapub.com/down/nginx-1.0.5. tar .gz &> /dev/null |
019 | tar zxf ../dist/libunwind-0.99. tar .gz |
024 | CFLAGS=-fPIC ./configure |
027 | make CFLAGS=-fPIC install |
028 | if [ "$?" == "0" ]; then |
029 | echo "libunwind-0.99安装成功." >> ./install_log.txt |
031 | echo "libunwind-0.99安装失败." >> ./install_log.txt |
039 | tar zxf ../dist/google-perftools-1.8.3. tar .gz |
040 | cd google-perftools-1.8.3/ |
042 | CHOST= "x86_64-pc-linux-gnu" CFLAGS= "-march=nocona -O2 -pipe" CXXFLAGS= "-march=nocona -O2 -pipe" \ |
047 | if [ "$?" == "0" ]; then |
048 | echo "google-perftools-1.8.3安装成功." >> ./install_log.txt |
050 | echo "google-perftools-1.8.3安装失败." >> ./install_log.txt |
054 | echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.conf |
061 | tar zxvf ../dist/pcre-8.01. tar .gz |
064 | CHOST= "x86_64-pc-linux-gnu" CFLAGS= "-march=nocona -O2 -pipe" CXXFLAGS= "-march=nocona -O2 -pipe" \ |
067 | if [ "$?" == "0" ]; then |
068 | echo "pcre-8.01安装成功." >> ./install_log.txt |
070 | echo "pcre-8.01安装失败." >> ./install_log.txt |
086 | tar zxf ../dist/nginx-1.0.5. tar .gz |
089 | sed -i 's#CFLAGS="$CFLAGS -g"#CFLAGS="$CFLAGS "#' auto/cc/gcc |
093 | CHOST= "x86_64-pc-linux-gnu" CFLAGS= "-march=nocona -O2 -pipe" CXXFLAGS= "-march=nocona -O2 -pipe" \ |
094 | ./configure --user=www --group=www \ |
095 | --prefix=/usr/ local /nginx \ |
096 | --with-http_stub_status_module \ |
097 | --with-google_perftools_module |
100 | if [ "$?" == "0" ]; then |
101 | echo "nginx-1.0.5安装成功." >> ./install_log.txt |
103 | echo "nginx-1.0.5安装失败." >> ./install_log.txt |
111 | chown -R www:www /data/logs |
115 | ln -s /data/conf/nginx/ conf |
117 | echo 'export LD_PRELOAD=/usr/local/lib/libtcmalloc.so' > /root/nginx_start |
118 | echo 'ulimit -SHn 51200' >> /root/nginx_start |
119 | echo '/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf' >> /root/nginx_start |
121 | echo '/usr/local/nginx/sbin/nginx -t' > /root/nginx_reload |
122 | echo 'kill -HUP `cat /usr/local/nginx/logs/nginx.pid`' >> /root/nginx_reload |
124 | chmod 700 /root/nginx_* |
3.2 编译安装varnish
07 | wget <a href= "\"http://bbs.osapub.com/down/varnish-3.0.0.tar.gz\"" target= "\"_blank\"" >http://bbs.osapub.com/down/varnish-3.0.0. tar .gz</a> &> /dev/null |
10 | export PKG_CONFIG_PATH=/usr/ local /lib/pkgconfig |
11 | tar zxvf ../dist/varnish-3.0.0. tar .gz |
14 | ./configure -prefix=/usr/ local /varnish - enable -debugging-symbols - enable -developer-warnings - enable -dependency-tracking |
16 | if [ "$?" == "0" ]; then |
17 | echo "varnish-3.0.0安装成功." >> ./install_log.txt |
19 | echo "varnish-3.0.0安装失败." >> ./install_log.txt |
24 | cat > /root/varnish_restart.sh <<EOF |
29 | /usr/ local /varnish/sbin/varnishd -n /data/varnish -f /usr/ local /varnish/vcl.conf -a 0.0.0.0:81 -s malloc,12G -g www -u www -w 4000,12000,10 -T 127.0.0.1:3500 -p http_max_hdr=256 -p http_req_hdr_len=8192 |
32 | cat > /root/varnish_start.sh <<EOF |
35 | /usr/ local /varnish/sbin/varnishd -n /data/varnish -f /usr/ local /varnish/vcl.conf -a 0.0.0.0:81 -s malloc,12G -g www -u www -w 4000,12000,10 -T 127.0.0.1:3500 -p http_max_hdr=256 -p http_req_hdr_len=8192 |
38 | chmod 755 /root/varnish* |
三、 配置varnish一些参数说明:
Backend servers
Varnish有后端(或称为源)服务器的概念。后端服务器是指Varnish提供加速服务的那台,通常提供内容。
第一件要做的事情是告诉Varnish,哪里能找到要加速的内容。
vcl_recv
vcl_recv是在请求开始时调用的。完成该子程序后,请求就被接收并解析了。用于确定是否需要服务请求,怎么服务,如果可用,使用哪个后端。
在vcl_recv中,你也可以修改请求。通常你可以修改cookie,或添加/移除请求头信息。
注意在vcl_recv中,只可以使用请求对象req。
vcl_fetch
vcl_fetch是在文档从后端被成功接收后调用的。通常用于调整响应头信息,触发ESI处理,万一请求失败就换个后端服务器。
在vcl_fecth中,你还可以使用请求对象req。还有个后端响应对象beresp。Beresp包含了后端的HTTP头信息。
varnish 3.X 配置参考文档:http://anykoro.sinaapp.com/?p=261
编辑:/usr/local/varnish/vcl.conf ,文件不存在则创建。注意:Server A 与server B 配置一致!
配置详情如下:
004 | backend server_osapub_com{ |
005 | .host = "192.168.4.97" ; |
019 | if (req.request == "BAN" ) { |
021 | if (!client.ip ~ purge) { |
022 | error 405 "Not allowed." ; |
024 | ban( "req.http.host == " + req.http.host + |
025 | "&& req.url == " + req.url); |
026 | error 200 "Ban added" ; |
031 | if (req.http.host ~ "^server.osapub.com" ||req.http.host ~ ".osapub.com" ) { |
034 | set req.backend = server_osapub_com; |
040 | if (req.restarts == 0) { |
041 | if (req.http.x-forwarded- for ) { |
042 | set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; |
045 | set req.http.X-Forwarded-For = client.ip; |
049 | if (req.http.Accept-Encoding) { |
050 | if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$" ) { |
051 | remove req.http.Accept-Encoding; |
052 | } elsif (req.http.Accept-Encoding ~ "gzip" ) { |
053 | set req.http.Accept-Encoding = "gzip" ;} |
054 | elsif (req.http.Accept-Encoding ~ "deflate" ) { |
055 | set req.http.Accept-Encoding = "deflate" ;} |
057 | remove req.http.Accept-Encoding; |
060 | if (req.http.Cache-Control ~ "no-cache" ) { |
063 | if (req.request != "GET" && |
064 | req.request != "HEAD" && |
065 | req.request != "PUT" && |
066 | req.request != "POST" && |
067 | req.request != "TRACE" && |
068 | req.request != "OPTIONS" && |
069 | req.request != "DELETE" ) { |
073 | if (req.request != "GET" && req.request != "HEAD" ) { |
077 | if (req.http.Authorization || req.http.Cookie ||req.http.Authenticate) { |
081 | if (req.request == "GET" && req.url ~ "(?i)\.php($|\?)" ){ |
085 | if (req.url ~ "\.(css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$" ) { |
086 | unset req.http.Cookie; |
095 | set beresp.grace = 5m; |
097 | if (beresp.status == 404 || beresp.status == 503 || beresp.status == 500 || beresp.status == 502) { |
098 | set beresp.http.X-Cacheable = "NO: beresp.status" ; |
099 | set beresp.http.X-Cacheable-status = beresp.status; |
100 | return (hit_for_pass); |
105 | if (req.url ~ "\.(php|shtml|asp|aspx|jsp|js|ashx)$" ) { |
106 | return (hit_for_pass); |
109 | if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$" ) { |
110 | unset beresp.http. set -cookie; |
113 | if (beresp.ttl <= 0s) { |
114 | set beresp.http.X-Cacheable = "NO: !beresp.cacheable" ; |
115 | return (hit_for_pass); |
118 | unset beresp.http.expires; |
127 | if (resp.http.magicmarker) { |
129 | unset resp.http.magicmarker; |
131 | set resp.http.age = "0" ; |
137 | set resp.http.X-Cache = "HIT" ; |
138 | set resp.http.X-Cache-Hits = obj.hits; |
141 | set resp.http.X-Cache = "MISS" ; |
145 | remove resp.http.X-Varnish; |
148 | remove resp.http.X-Powered-By; |
149 | remove resp.http.X-Drupal-Cache; |
156 | if (obj.status == 503 && req.restarts < 5) { |
157 | set obj.http.X-Restarts = req.restarts; |
165 | if (req.http.Cache-Control ~ "no-cache" ) { |
167 | if (! (req.http.Via || req.http.User-Agent ~ "bot|MSIE" )) { |
配置完成后执行:/root/varnish_start.sh 如果启动成功则表示配置无误!
四、 配置nginx编辑: /usr/local/nginx/conf/nginx.conf ,nginx 的配置方法请参考网上文档资料。
Server A(192.168.4.97) nginx配置如下:
004 | error_log /logs/nginx/nginx_error.log crit; |
007 | worker_rlimit_nofile 65500; |
013 | worker_connections 65500; |
019 | default_type application/octet-stream; |
021 | server_names_hash_bucket_size 128; |
022 | client_header_buffer_size 64k; |
023 | large_client_header_buffers 4 64k; |
024 | client_max_body_size 10m; |
032 | keepalive_timeout 60; |
035 | fastcgi_connect_timeout 200; |
036 | fastcgi_send_timeout 300; |
037 | fastcgi_read_timeout 600; |
038 | fastcgi_buffer_size 128k; |
039 | fastcgi_buffers 4 64k; |
040 | fastcgi_busy_buffers_size 128k; |
041 | fastcgi_temp_file_write_size 128k; |
042 | fastcgi_temp_path /dev/shm; |
044 | gzip_min_length 2048; |
046 | gzip_http_version 1.1; |
047 | gzip_types text/plain text/css application/xml application/x-javascript ; |
049 | log_format access '$remote_addr - $remote_user [$time_local] "$request" ' |
050 | '$status $body_bytes_sent "$http_referer" ' |
051 | '"$http_user_agent" $http_x_forwarded_for' ; |
055 | upstream varnish_server { |
056 | server 127.0.0.1:81 weight=2 max_fails=3 fail_timeout=30s; |
057 | server 192.168.4.99:81 weight=2 max_fails=3 fail_timeout=30s; |
064 | server_name 192.168.4.96 192.168.4.98; |
065 | index index.html index.htm index.php index.aspx; |
068 | proxy_set_header Host $host; |
069 | proxy_set_header X-Real-IP $remote_addr; |
070 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
072 | proxy_connect_timeout 200; |
073 | proxy_send_timeout 300; |
074 | proxy_read_timeout 500; |
075 | proxy_buffer_size 256k; |
076 | proxy_buffers 4 128k; |
077 | proxy_busy_buffers_size 256k; |
078 | proxy_temp_file_write_size 256k; |
079 | proxy_temp_path /dev/shm; |
081 | proxy_pass http://varnish_server; |
085 | access_log /logs/nginx/192.168.4.96.log access; |
092 | server_name 192.168.4.97; |
093 | index index.html index.htm index.php; |
094 | root /data/web/awstats/www; |
099 | fastcgi_pass 127.0.0.1:10080; |
100 | fastcgi_index index.php; |
103 | access_log /logs/nginx/awstats.osapub.com.log access; |
server B (192.168.4.99)配置如下:
004 | error_log /logs/nginx/nginx_error.log crit; |
007 | worker_rlimit_nofile 65500; |
013 | worker_connections 65500; |
019 | default_type application/octet-stream; |
021 | server_names_hash_bucket_size 128; |
022 | client_header_buffer_size 64k; |
023 | large_client_header_buffers 4 64k; |
024 | client_max_body_size 10m; |
032 | keepalive_timeout 60; |
035 | fastcgi_connect_timeout 200; |
036 | fastcgi_send_timeout 300; |
037 | fastcgi_read_timeout 600; |
038 | fastcgi_buffer_size 128k; |
039 | fastcgi_buffers 4 64k; |
040 | fastcgi_busy_buffers_size 128k; |
041 | fastcgi_temp_file_write_size 128k; |
042 | fastcgi_temp_path /dev/shm; |
044 | gzip_min_length 2048; |
046 | gzip_http_version 1.1; |
047 | gzip_types text/plain text/css application/xml application/x-javascript ; |
049 | log_format access '$remote_addr - $remote_user [$time_local] "$request" ' |
050 | '$status $body_bytes_sent "$http_referer" ' |
051 | '"$http_user_agent" $http_x_forwarded_for' ; |
055 | upstream varnish_server { |
056 | server 127.0.0.1:81 weight=2 max_fails=3 fail_timeout=30s; |
057 | server 192.168.4.97:81 weight=2 max_fails=3 fail_timeout=30s; |
064 | server_name 192.168.4.96 192.168.4.98; |
065 | index index.html index.htm index.php index.aspx; |
068 | proxy_set_header Host $host; |
069 | proxy_set_header X-Real-IP $remote_addr; |
070 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; |
072 | proxy_connect_timeout 200; |
073 | proxy_send_timeout 300; |
074 | proxy_read_timeout 500; |
075 | proxy_buffer_size 256k; |
076 | proxy_buffers 4 128k; |
077 | proxy_busy_buffers_size 256k; |
078 | proxy_temp_file_write_size 256k; |
079 | proxy_temp_path /dev/shm; |
081 | proxy_pass http://varnish_server; |
085 | access_log /logs/nginx/192.168.4.98.log access; |
092 | server_name 192.168.4.99; |
093 | index index.html index.htm index.php; |
094 | root /data/web/awstats/www; |
099 | fastcgi_pass 127.0.0.1:10080; |
100 | fastcgi_index index.php; |
103 | access_log /logs/nginx/awstats.osapub.com.log access; |
启动nginx:
/root/nginx_start
启动正常会监听80,82端口!
五、 运行nginx ha脚本
server A 的脚本下载地址:http://bbs.osapub.com/down/server_a_ha.tar.gz
解压力后得到三个脚本:
nginx_watchdog.sh
nginxha.sh
nginx_ha1.sh
server B 的脚本下载地址:http://bbs.osapub.com/down/server_b_ha.tar.gz
解压力后得到三个脚本:
nginx_watchdog.sh
nginxha.sh
nginx_ha2.sh
注意:脚本建议放到:/data/sh/ha 目录下,否则需要修改nginxha.sh 里面的程序路径!
六、 测试1,测试之前先把nginx 负载去掉,可以在前面加#号暂时注释,只保留本机解析,确保测试结果准确。
2,修改本机host文件
添加:192.168.4.99 server.osapub.com 到末尾,保存后访问。
3,正常结果显示如下:
到这里整个架构基本能运行起来了,根据大家的实际需求,对配置文件进行调优,HA脚本也可以进一步调优,关于报警,请参考社区自动安装mutt报警的脚本!
这篇关于nginx做调度(自身用dns轮询保证高可用),varnish做后端的cache的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!