基于Linux系统的高可用中小型网站集群架构说明
2020-06-01
加入收藏
网站集群组成所需服务器
主机名 eth0网卡 eth1网卡 服务简介
lb01 10.0.0.5/24 172.16.1.5/24 负载服务主
lb02 10.0.0.6/24 172.16.1.6/24 负载服务备
web01 10.0.0.7/24 172.16.1.7/24 动态php服务
web02 10.0.0.8/24 172.16.1.8/24 https访问、vpn服务
web03 10.0.0.9/24 172.16.1.9/24 ELK日志分析系统
db01 10.0.0.51/24 172.16.1.51/24 主数据库服务
db02 10.0.0.52/24 172.16.1.52/24 从数据库服务
nfs01 10.0.0.31/24 172.16.1.31/24 存储服务
backup 10.0.0.41/24 172.16.1.41/24 备份服务
m01 10.0.0.61/24 172.16.1.61/24 管理、yum、监控
m02 10.0.0.62/24 172.16.1.62/24 跳板机服务
网站集群各服务器作用
A.web服务：负责处理用户的请求，提供网页服务，将上传的数据需要放到存储服务器上。涉及核心软件：Nginx、PHP、Tomcat，NFS、定时任务服务等 B.负载均衡服务：负载均衡器，负责调度网站访问请求，需要2台主备服务器，同时做 高可用。涉及核心软件：Nginx和Keepalved C.数据库服务：提供bbs、blog、www等文章文本内容的存储和访问。涉及核心软件：MySQL、MariaDB、PHP等 D.存储服务：提供blog、bbs、www等服务上的共享图片、视频、附件数据存储。涉及核心软件：NFS E.备份服务：提供全网数据的备份服务，以及异地灾备、涉及软件核心：Rsync服务 F.实时同步服务：提供nfs服务与rsync服务实时数据同步，实现数据无差别还原与备份。涉及核心软件：sersync G.管理服务：管理所有服务器，包括批量分发文件、批量执行命令、批量配置服务、跳板机服务，远程拨号服务，yum仓库服务等。涉及核心软件：SSH、Ansible、跳板机等
网站集群架构图及思路
架构图
思路： 该架构是以web集群作为始发点，通过nginx+PHP相关服务搭建出aaa.xxx.com的基础网站，其中https访问，用户在通过负载均衡的分配访问到不同的web服务器上，进行浏览；当有数据发生变化的同时通过数据库的迁移保证动态数据信息实时同步，而数据库服务器与存储服务器保持共享存储从而达到静态数据信息的实时同步，再通过sersync实时同步将nfs服务器与数据库服务器中的数据备份到rsync服务器中,最后使用定时任务将整个架构中的重要数据、配置文件、脚本信息等进行压缩备份。而内部人员可通过vpn、跳板机、管理服务器对整个集群架构进行统一管理，同时记录集群日志和实时监控。
网站集群实现目标
实现目标为，当用户上传文本、音频、附件等数据信息的同时，做到数据库、存储服务器皆有该数据信息，并在定时任务后备份服务器中也存在该数据。
网站集群各服务器基础优化（模板机）
1.配置解析文件
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.5 lb01
10.0.0.6 lb02
10.0.0.7 web01
10.0.0.8 web02
10.0.0.9 web03
10.0.0.31 nfs01
10.0.0.41 backup
10.0.0.51 db01
10.0.0.61 m01
10.0.0.71 zabbix
2.修改主机名称和网卡地址（需添加多块网卡）
修改主机名称: hostnamectl set-hostname 名称
修改主机网卡地址: sed -ri 's#200#ip#g;/UUID|HWADDR/d'
/etc/sysconfig/network-scripts/ifcfg-eth[01] 3.更新yum源 a.curl -o
/etc/yum.repos.d/centos-Base.repo
http://mirrors.aliyun.com/repo/Centos-7.repo b.wget -O
/etc/yum.repos.d/epel.repo
http://mirrors.aliyun.com/repo/epel-7.repo c.rpm -Uvh
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm d.rpm -Uvh
https://mirror.webtatic.com/yum/el7/webtatic-release.rpm e.yum clean all
4.关闭selinux
vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
5.关闭防火墙 systemctl stop firewalld systemctl disable firewalld 6.修改为英文字符集 echo 'LANG="en_US.UTF-8"' >/etc/locale.conf source /etc/locale.conf echo $LANG 7.时间同步 ntpdate ntp1.aliyun.com 8.优化内核
cat >>/etc/sysctl.conf<<EOF
net.ipv4.tcp_fin_timeout = 2
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_keepalive_time = 600
net.ipv4.ip_local_port_range = 4000 65000
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.route.gc_timeout = 100
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.core.somaxconn = 16384
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_orphans = 16384
EOF
9.加大文件描述符
echo '* - nofile 65535 ' >>/etc/security/limits.conf
10.设置/tmp权限为777 chmod 777 /tmp
网站集群架构开始
1）web01==>LNMP动态网站搭建
A.环境部署
安装官方源的nginx服务
vim /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
yum install nginx -y systemctl start nginx systemctl enable nginx 安装mariadb数据库服务 yum install mariadb-server mariadb -y systemctl start mariadb.service systemctl enable mariadb.service 安装php-fpm
yum remove php-mysql php php-fpm php-common
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
yum install -y php71w php71w-cli php71w-common php71w-devel php71w-embedded php71w-gd php71w-mcrypt php71w-mbstring php71w-pdo php71w-xml php71w-fpm php71w-mysqlnd php71w-opcache php71w-pecl-memcached php71w-pecl-redis php71w-pecl-mongodb
vim /etc/php-fpm.d/www.conf
user = nginx
group = nginx
systemctl start php-fpm
B.搭建aaa.xxx.com
下载指定软件包：
https://cn.wordPress/ target=_blank class=infotextkey>WordPress.org/download/
vim /etc/nginx/cond.f/blog.conf
server {
listen 80;
server_name aaa.xxx.com;
location / {
root /usr/share/nginx/html/blog;
index index.php index.html;
}
location ~ .php$ {
root /usr/share/nginx/html/blog;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
systemctl restart nginx
cd /usr/share/nginx/html/
rz -y 选择软件包位置，进行传输
tar xf wordpress-5.2.3-zh_CN.tar.gz
mv wordpress blog
rm -rf wordpress-5.2.3-zh_CN.tar.gz
chown -R nginx. blog/
创建数据库信息
mysql -uroot -p123456
create database wordpress;
show databases;
grant all on wordpress.* to 'wordpress'@'localhost' identified by '123456';
select user,host from mysql.user;
本地解析hosts 在浏览器输入ip地址按步安装即可，输入对应的数据库名称及密码
aaa.xxx.com 同理
2）web02==>动态网站实现https访问
vim www.conf
server {
listen 80;
server_name www.oldboy.com;
rewrite ^/(.*)$ https://$host/$1 redirect;
}
server {
listen 443 ssl;
server_name www.oldboy.com;
ssl_certificate server.crt;
ssl_certificate_key server.key;
location / {
root /usr/share/nginx/html/www/install_package;
index index.php index.html;
}
location ~ .php$ {
root /usr/share/nginx/html/www/install_package;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
3）lb01/lb02==>nginx+keppalived负载均衡高可用
负载均衡：
vim /etc/nginx/nginx.com
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type Application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
upstream oldboy {
server 172.16.1.7:80;
}
server{
listen 80;
server_name localhost;
location / {
proxy_pass http://oldboy;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_next_upstream error timeout invalid_header http_404;
}
}
}
本地解析10.0.0.5 aaa.xxx.com 高可用： 负载均衡环境准备
yum install -y keepalived
echo 'net.ipv4.ip_nonlocal_bind = 1' >>/etc/sysctl.conf
sysctl -p
vim /etc/nginx/nginx.conf
server {
listen 10.0.0.3:80;
server_name localhost;
systemctl restart nginx
lb01：
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id lb01
}
vrrp_instance oldboy {
state MASTER
interface eth0
virtual_router_id 63
priority 110
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.3
}
}
lb02
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id lb02
}
vrrp_instance oldboy {
state BACKUP
interface eth0
virtual_router_id 63
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.3
}
}
systemctl start keepalived.service
本地解析：10.0.0.3 脑裂及nginx服务停止保证主备转换：
mkdir -p /server/scripts
vim /server/scripts/check_keepalivde.sh
#!/bin/bash
ip a|grep 10.0.0.3 &>/dev/null
if [ $? -eq 0 ]
then
echo "master and backup change"|mail -s "check keepalived server" xxx@163.com
fi
crontab -e
* * * * * sh /server/scripts/check_keepalived.sh
vim /server/scripts/check_web.sh
#!/bin/bash
port_info=$(netstat -lntup|grep -w 80|wc -l)
if [ $port_info -eq 0 ]
then
systemctl stop keepalived
fi
chmod +x /server/srcipts/check_web.sh
vim /etc/keepalived/keepalived.conf
vrrp_script check_web {
script "/server/scripts/check_web.sh"
interval 2
weight 2
}
track_script {
check_web
}
4）mariadb==>实现数据库迁移
web01服务器端(bbs、blog)
mysqldump -uroot -p123456 -A > /tmp/mysql_bak.sql
rsync -avz /tmp/mysql_bak.sql 172.16.1.51:/tmp
vim /usr/share/nginx/html/blog/wp-config.php
define('DB_HOST', '172.16.1.51');
vim /usr/share/nginx/html/bbs/config/config_global.php
$_config['db']['1']['dbhost'] = '172.16.1.51';
vim /usr/share/nginx/html/bbs/config/config_ucenter.php
define('UC_DBHOST', '172.16.1.51');
web02服务器端（www）
mysqldump -uroot -p123456 -A > /tmp/phpcms_bak.sql
rsync -avz /tmp/phpcms_bak.sql 172.16.1.51:/tmp
vim /usr/share/nginx/html/www/install_package/caches/configs/database.php
'hostname' => '172.16.1.51',
数据库服务器端
mysql -uroot -p123456 < /tmp/mysql_bak.sql
mysql -uroot -p123456 < /tmp/phpcms_bak.sql
mysql -uroot -p123456
grant all on wordpress.* to 'wordpress'@'172.16.1.%' identified by '123456';
grant all on ultrax.* to ''root@'172.16.1.%' identified by '123456';
grant all on phpcmsv9.* to 'phpcmsv9'@'172.16.1.%' identified by '123456';
exit
systemctl restart mariadb
5）nfs==>实现网站上传数据实时存储
web01服务器
mkdir {/tmp/wordpress_bak，/tmp/discuz_bak}
mv /usr/share/nginx/html/blog/wp-content/uploads/* /tmp/wordpress_bak
mv /usr/share/nginx/html/bbs/data/attachment/forum/* /tmp/discuz_bak/
mount -t nfs 172.16.1.31:/data/bbs /usr/share/nginx/html/bbs/data/attachment/forum
mount -t nfs 172.16.1.31:/data/blog /usr/share/nginx/html/blog/wp-content/uploads
mv /tmp/wordpress_bak/* /usr/share/nginx/html/blog/wp-content/uploads
mv /tmp/discuz_bak/* /usr/share/nginx/html/bbs/data/attachment/forum
web02服务器
mkdir /tmp/phpcms_bak
mv /usr/share/nginx/html/www/install_package/uploadfile/poster/* /tmp/phpcms_bak/
mount -t nfs 172.16.1.31:/data/www /usr/share/nginx/html/www/install_package/uploadfile/poster
mv /tmp/phpcms_bak/* /usr/share/nginx/html/www/install_package/uploadfile/poster
nfs服务器
mkdir -p /data/{bbs,blog,www}
vim /etc/exports
/data/blog 172.16.1.0/24(rw,sync,no_root_squash)
/data/www 172.16.1.0/24(rw,sync,no_root_squash)
/data/bbs 172.16.1.0/24(rw,sync,no_root_squash)
useradd -M -s /sbin/nologin nginx -u 995（与相关服务的属主属组及uid保持一致）
chown -R nginx. /data
systemctl restart nfs
配置自动挂载文件
6）rsync==>定时任务实现与nfs服务器数据备份与还原
backup服务器端
yum install -y rsync
vim /etc/rsyncd.conf
uid = rsync
gid = rsync
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 300
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup_db]
comment = "backup dir by oldboy"
path = /backup/db/
[backup_nfs]
comment = "backup dir by oldboy"
path = /backup/nfs/
useradd -M -s /sbin/nologin rsync
echo rsync_backup:123456 > /etc/rsync.password
chmod 600 /etc/rsync.password
mkdir /dackup
chown rsync.rsync /dackup
systemctl start rsyncd
nfs服务器端
yum install -y rsync
echo 123456 >/etc/rsync.password
chmod 600 /etc/rsync.password
nfs服务器定时任务
crontab -e
30 2 * * * /usr/bin/rsync -avz /data/ rsync_backup@172.16.1.41::backup_nfs --password-file=/etc/rsync.password
7）sersync==>实时同步数据库服务与备份服务
数据库服务
yum install -y lrzsz
mkdir -p /server/tools
rz -y ==>选择路径
unzip sersync_installdir_64bit.zip
cd /server/tools/sersync_installdir_64bit
mv sersync/ /usr/local/
vim /usr/local/sersync/conf/confxml.xml
<filter start="false">
<exclude expression="(.*).svn"></exclude>
<exclude expression="(.*).gz"></exclude>
<exclude expression="^info/*"></exclude>
<exclude expression="^static/*"></exclude>
</filter>
<inotify>
<delete start="true"/>
<createFolder start="true"/>
<createFile start="false"/>
<closeWrite start="true"/>
<moveFrom start="true"/>
<moveTo start="true"/>
<attrib start="false"/>
<modify start="false"/>
</inotify>
<sersync>
<localpath watch="/var/lib/mysql/">
<remote ip="172.16.1.41" name="backup_db"/>
<!--<remote ip="192.168.8.39" name="tongbu"/>-->
<!--<remote ip="192.168.8.40" name="tongbu"/>-->
</localpath>
<rsync>
<commonParams params="-az"/>
<auth start="true" users="rsync_backup" passwordfile="/etc/rsync.password"/>
<userDefinedPort start="false" port="874"/><!-- port=874 -->
<timeout start="false" time="100"/><!-- timeout=100 -->
<ssh start="false"/>
</rsync>
<failLog path="/tmp/rsync_fail_log.sh" timeToExecute="60"/><!--default every 60mins execute once-->
<crontab start="false" schedule="600"><!--600mins-->
<crontabfilter start="false">
<exclude expression="*.php"></exclude>
<exclude expression="info/*"></exclude>
</crontabfilter>
</crontab>
<plugin start="false" name="command"/>
</sersync>
<plugin name="command">
<param prefix="/bin/sh" suffix="" ignoreError="true"/> <!--prefix /opt/tongbu/mmm.sh suffix-->
<filter start="false">
<include expression="(.*).php"/>
<include expression="(.*).sh"/>
</filter>
</plugin>
<plugin name="socket">
<localpath watch="/opt/tongbu">
<deshost ip="192.168.138.20" port="8009"/>
</localpath>
</plugin>
<plugin name="refreshCDN">
<localpath watch="/data0/htdocs/aaa.xxx.com/site/">
<cdninfo domainname="aaa.xxx.com" port="80" username="xxxx" passwd="xxxx"/>
<sendurl base="http://pic.xoyo.com/cms"/>
<regexurl regex="false" match="cms.xoyo.com/site([/a-zA-Z0-9]*).xxx.com/images"/>
</localpath>
</plugin>
</head>
cd /usr/local/sersync/bin/
chmod +x sersync
yum install -y rsync
echo 123456 >/etc/rsync.password
chmod 600 /etc/rsync.password
./sersync -dro /usr/local/sersync/conf/confxml.xml
8）ansible==>实现一键化ssh远程连接
vim /server/scripts/distribute_key.sh
#!/bin/bash
. /etc/init.d/functions
# 创建秘钥对
if [ ! -f /root/.ssh/id_dsa ]
then
ssh-keygen -t dsa -f /root/.ssh/id_dsa -P "" >/dev/null
action "key pair create" /bin/true
else
action "key pair already exists" /bin/false
fi
# 分发公钥信息
for ip in 7 31 41 5 6 51 8
do
sshpass -p123456 ssh-copy-id -i ~/.ssh/id_dsa.pub 172.16.1.$ip -o StrictHostKeyChecking=no &>/dev/null
if [ $? -eq 0 ]
then
action "host 172.16.1.$ip pub_key distribute" /bin/true
echo ""
else
action "host 172.16.1.$ip pub_key distribute" /bin/false
echo ""
fi
done
9）rsync全网备份
nfs端： mkdir -p /server/srcipts
vim backup.sh
#!/bin/bash
Backup_IP=$(hostname -I|awk '{print $NF}')
mkdir -p /backup/$Backup_IP
# 1.
tar zchPf /backup/$Backup_IP/system_backup_$(date +%F_%w).tar.gz /var/spool/cron/root /etc/rc.local /server/scripts /etc/
exports
# 2.
find /backup/ -type f -name "*.tar.gz"|xargs md5sum >/backup/$Backup_IP/finger.txt
# 3.
rsync -az /backup/ rsync_backup@172.16.1.41::backup_backup --password-file=/etc/rsync.password
# 4.
find /backup/ -type f -name "*.tar.gz" -mtime +7 -delete
crontab -e
0 2 * * * /usr/bin/sh /server/scripts/backup.sh
web01端： mkdir /server/srcipts
vim backup.sh
#!/bin/bash
Backup_IP=$(hostname -I|awk '{print $NF}')
Backuo_html="/usr/share/nginx/html"
Backup_mysql="/var/lib/mysql"
mkdir -p /backup/$Backup_IP
# 1.
tar zchPf /backup/$Backup_IP/web_info_backup_$(date +%F_%w).tar.gz $Backup_html/bbs $Backup_html/blog
tar zchPf /backup/$Backup_IP/web_log_backup_$(date +%F_%w).tar.gz $Backup_mysql/ultrax $Backup_mysql/wordpress
tar zchPf /backup/$Backup_IP/system_backup_$(date +%F_%w).tar.gz /etc/rc.local /server/srcipts /var/spool/cron/root /etc/
nginx/conf.d/bbs.conf /etc/nginx/nginx.conf /etc/nginx/conf.d/blog.conf
# 2.
find /backup/ -type f -name "*.tar.gz"|xargs md5sum >/backup/$Backup_IP/finger.txt
# 3.
rsync -az /backup/ rsync_backup@172.16.1.41::backup_backup --password-file=/etc/rsync.password
# 4.
find /backup/ -type f -name "*.tar.gz" -mtime +7 -delete
echo 123456 > /etc/rsync.password
crontab -e
0 2 * * * /usr/bin/sh /server/scripts/backup.sh
web02端：同理 backup端：
mkdir -p /server/srcipts
vim /server/scripts/backup.sh
#!/bin/bash
# 1.
find /backup/backup/ -type f -name "finger.txt"|xargs md5sum -c >/tmp/check_data.txt
# 2.
find /backup/backup/ -type f -name "*.tar.gz" -mtime +180 ! -iname "*_2.tar.gz" -delete
crontab -e
0 6 * * * /usr/bin/sh /server/scripts/backup.sh
