Graylog4.2集群部署完整教程

2022-01-04 walkingcloud

加入收藏

GrayLog多节点集群架构如下

(图片可点击放大查看)

下面介绍Graylog4.2集群部署过程

基础环境准备

三台服务器：centos7.9的虚拟机

/data分区（LVM) 用于Elasticsearch日志数据存储

内存大小为6GB

IP地址和主机名如下：

192.168.31.211 graylog01 graylog01.walkingcloud.cn
192.168.31.212 graylog02 graylog02.walkingcloud.cn
192.168.31.213 graylog03 graylog03.walkingcloud.cn

均已关闭SElinux

(图片可点击放大查看)

一、配置MongoDB集群

1、三台均安装mongodb

cat > /etc/yum.repos.d/mongodb-org.repo << EOF
[mongodb-org]
name=MongoDB Repository
baseurl=https://mirrors.aliyun.com/mongodb/yum/redhat/$releasever/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc
EOF

yum install -y mongodb-org

(图片可点击放大查看)

2、三台均先启动mongodb

systemctl daemon-reload
systemctl enable mongod.service
systemctl start mongod.service
systemctl --type=service --state=active | grep mongod
firewall-cmd --add-port=27017/tcp --permanent --zone=public 
firewall-cmd --reload

(图片可点击放大查看)

3、主节点生成授权认证keyfile文件并拷贝到其它节点

openssl rand -base64 756 > /var/lib/mongo/access.keyfile
chown mongod:mongod /var/lib/mongo/access.keyfile
chmod 600 /var/lib/mongo/access.keyfile 
scp -rp /var/lib/mongo/access.keyfile root@graylog02:/var/lib/mongo/
scp -rp /var/lib/mongo/access.keyfile root@graylog03:/var/lib/mongo/

(图片可点击放大查看)

4、vi修改/etc/mongod.conf

以主节点为例，修改配置文件

vi /etc/mongod.conf
net:
  port: 27017
  bindIp: 192.168.31.211 

security:
  keyFile: /var/lib/mongo/access.keyfile

replication:
  replSetName: graylog-rs

(图片可点击放大查看)

graylog02修改成如下

(图片可点击放大查看)

graylog03修改成如下

(图片可点击放大查看)

5、初始化集群

这里为了本地能登录mongo

先将bindIp: 192.168.31.211修改为bindIp: 0.0.0.0
并重启服务
systemctl restart mongod.service

输入mongo进数据库

use admin

rs.initiate( {
   _id : "graylog-rs",
   members: [
      { _id: 0, host: "192.168.31.211:27017" },
      { _id: 1, host: "192.168.31.212:27017" },
      { _id: 2, host: "192.168.31.213:27017" }
   ]
})

rs.status()查看集群状态

(图片可点击放大查看)

6、创建graylog数据库并设置密码

#修改admin用户密码
use admin
db.createUser({user: "admin", pwd: "Admin@2021", roles: ["root"]})
db.auth("admin","Admin@2021")
#创建graylog数据库并设置密码
use graylog
db.createUser({
   user: "graylog", 
   pwd: "Graylog2021",
  "roles" : [{
      "role" : "dbOwner",
      "db" : "graylog"
    }, {
      "role" : "readWrite",
      "db" : "graylog"
    }]
})

(图片可点击放大查看)

7、将主节点bindIp配置恢复并重启mongod服务

将graylog01的bindIp修改成192.168.31.211并重启服务

这时使用账号和密码登录mongo

(图片可点击放大查看)

在graylog02 重启mongod服务，使graylog01重新变为Primary角色

二、搭建Elasticsearch集群

1、三台节点均以rpm包方式安装Elasticsearch

cd  /opt
wget https://mirrors.cloud.tencent.com/elasticstack/yum/elastic-7.x/7.16.2/elasticsearch-7.16.2-x86_64.rpm

rpm -ivh elasticsearch-7.16.2-x86_64.rpm

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
firewall-cmd --add-port=9200/tcp --permanent --zone=public 
firewall-cmd --reload

(图片可点击放大查看)

2、创建存储和日志文件夹

mkdir -p /data/elasticsearch/data
mkdir -p /data/elasticsearch/logs
chown -R elasticsearch:elasticsearch /data/elasticsearch

cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml_default

(图片可点击放大查看)

3、修改elasticsearch.yml配置文件

vi/etc/elasticsearch/elasticsearch.yml

1)graylog01节点

#添加并修改成如下行

cluster.name: graylog-cluster
action.auto_create_index: false
node.name: graylog01
node.master: true
node.data: true
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.31.211
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.31.211:9300", "192.168.31.212:9300", "192.168.31.213:9300"]
cluster.initial_master_nodes: ["graylog01"]

(图片可点击放大查看)

2)graylog02节点

cat  /etc/elasticsearch/elasticsearch.yml | grep -v "^#" | grep -v "^$"

cluster.name: graylog-cluster
action.auto_create_index: false
node.name: graylog02
node.master: false
node.data: true
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.31.212
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.31.211:9300", "192.168.31.212:9300", "192.168.31.213:9300"]

3)graylog03节点

cat  /etc/elasticsearch/elasticsearch.yml | grep -v "^#" | grep -v "^$"
cluster.name: graylog-cluster
action.auto_create_index: false
node.name: graylog03
node.master: false
node.data: true
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
network.host: 192.168.31.213
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.31.211:9300", "192.168.31.212:9300", "192.168.31.213:9300"]

4、三个节点均修改jvm.options配置文件上内存大小

firewall-cmd --add-port=9300/tcp --permanent --zone=public 
firewall-cmd --reload
vim /etc/elasticsearch/jvm.options
设置jvm内存大小为物理内存的一半
并重启elasticsearch.service
systemctl restart elasticsearch.service

(图片可点击放大查看)

5、查看elasticsearch集群状态

curl -s -XGET 'http://192.168.31.211:9200/_cluster/health?pretty=true'
curl -s -XGET 'http://192.168.31.211:9200/_cat/nodes?v'

(图片可点击放大查看)

三、安装GraylogServer并配置GraylogServer集群

1、安装jdk，pwgen和graylog-server

yum install -y JAVA-1.8.0-openjdk-headless.x86_64
yum install -y pwgen

(图片可点击放大查看)

rpm -ivh https://packages.graylog2.org/repo/packages/graylog-4.2-repository_latest.rpm
yum install graylog-server -y

(图片可点击放大查看)

systemctl enable graylog-server
systemctl start graylog-server

cp /etc/graylog/server/server.conf /etc/graylog/server/server.conf_default

pwgen -N 1 -s 96
XC0Epiv5SnWFdm82nsUWAJN3t2MHaFEaSFHd6RLPf1nzxwnmubT0n7NQdrK8jCDEOS05DtrkGHDDE61490OUJKBOOXIAT4LI

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d 'n' | sha256sum | cut -d" " -f1
Enter Password: Graylog@2021
10dfabb9595634675701865aa1c6e774d89d59f4a104ab128fbffcdaa3cf8f7b

(图片可点击放大查看)

2、修改graylog的主配置文件server.conf

主节点graylog01的配置如下

/etc/graylog/server/server.conf 
cat /etc/graylog/server/server.conf | grep -v "^#" | grep -v "^$"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = XC0Epiv5SnWFdm82nsUWAJN3t2MHaFEaSFHd6RLPf1nzxwnmubT0n7NQdrK8jCDEOS05DtrkGHDDE61490OUJKBOOXIAT4LI
root_password_sha2 = 10dfabb9595634675701865aa1c6e774d89d59f4a104ab128fbffcdaa3cf8f7b
root_timezone = Asia/Shanghai
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.31.211:9000
http_publish_uri = http://192.168.31.211:9000/
elasticsearch_hosts = http://graylog01:9200,http://graylog02:9200,http://graylog03:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = true
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 8
outputbuffer_processors = 16
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://graylog:Graylog2021@graylog01:27017,graylog02:27017,graylog03:27017/graylog?replicaSet=graylog-rs
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

(图片可点击放大查看)

3、重启graylog-server服务，并开放9000 web端口

建议将配置文件scp到graylog02和graylog03上

(图片可点击放大查看)

firewall-cmd --add-port=9000/tcp --permanent --zone=public 
firewall-cmd --reload
systemctl restart graylog-server

(图片可点击放大查看)

graylog02,graylog03节点修改如下三处即可

is_master = true
http_bind_address = 192.168.31.213:9000
http_publish_uri = http://192.168.31.213:9000/

(图片可点击放大查看)

登录Graylog主界面查看GrayLog集群情况

1、查看Nodes信息

(图片可点击放大查看)

2、查看Cluster信息

(图片可点击放大查看)

3、添加Input，可以设置成全局的Input

记得三台几点防火墙开放Input设置的端口，例如syslog1514

firewall-cmd --add-port=1514/udp --permanent --zone=public 
firewall-cmd --reload

(图片可点击放大查看)

Tips：修改graylog的jvm内存大小

(图片可点击放大查看)

本文参考如下链接完成

https://docs.graylog.org/v1/docs/multinode-setup
https://docs.mongodb.com/manual/tutorial/deploy-replica-set/
https://docs.mongodb.com/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#std-label-deploy-repl-set-with-auth
https://cloud.tencent.com/developer/article/1615815
https://zhuanlan.zhihu.com/p/120698020
https://www.cnblogs.com/opsdemo/p/15035379.html