本站消息

站长简介/公众号


站长简介:逗比程序员,理工宅男,前每日优鲜python全栈开发工程师,利用周末时间开发出本站,欢迎关注我的微信公众号:程序员总部,程序员的家,探索程序员的人生之路!分享IT最新技术,关注行业最新动向,让你永不落伍。了解同行们的工资,生活工作中的酸甜苦辣,谋求程序员的最终出路!

  价值13000svip视频教程,java大神匠心打造,零基础java开发工程师视频教程全套,基础+进阶+项目实战,包含课件和源码

  出租广告位,需要合作请联系站长


+关注
已关注

分类  

暂无分类

标签  

暂无标签

日期归档  

2021-05(18)

2021-06(40)

2021-07(5)

2021-08(46)

2021-09(44)

大数据平台实时数仓从0到1搭建之 - 08 集群脚本

发布于2021-10-05 12:57     阅读(105)     评论(0)     点赞(12)     收藏(2)


概述

目前涉及到的zookeeper和kafka,都是需要在每台机器上做启动停止操作,
本文使用shell脚本,处理集群的操作
重点语法:ssh host 命令
在对应的host节点上执行某个命令

环境变量

需要修改对应的用户的环境变量,
使用root用户执行脚本,则需要在/root/.bashrc中添加JAVA_HOME和PATH
每个节点都需要配置

eg.server110

[root@server110 ~]# vim /root/.bashrc
JAVA_HOME=/opt/modules/jdk1.8.0_181
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH

jps & jpsall.sh

jps命令是java里用来查看java进程的工具,但是只能查看当前节点的进程情况
思路就是循环所有节点,将jps命令发送到对应的节点上执行,脚本如下

[root@server110 opt]# vim jpsall.sh
#!/bin/bash
for i in server110 server111 server112
do
        echo ----------------jps $i --------------------
        ssh $i "jps"
done

zkServer.sh & zk.sh

思路:
1、循环所有节点
2、根据传入的参数,执行对应的命令,start | stop | status

[root@server110 opt]# vim zk.sh 
#!/bin/bash
case $1 in
"start")
{
        for s in server110 server111 server112
        do 
                ssh $s "/opt/modules/zookeeper-3.5.8/bin/zkServer.sh start"
                echo ----------$s zookeeper start-----------------             
        done
}
;;
"stop")
{
        for s in server110 server111 server112
        do
                ssh $s  "/opt/modules/zookeeper-3.5.8/bin/zkServer.sh stop"
                echo ----------$s zookeeper stop-----------------
        done
}
;;
"status")
{
        for s in server120 server121 server122
        do
                echo ----------$s zookeeper status----------------
                ssh $s  "/opt/modules/zookeeper-3.5.8/bin/zkServer.sh status"
        done
}
;;
*)
{
        echo "------./zk.sh start|stop|status-----"
}
;;
esac

kafka-server-start & kafka.sh

思路:
1、循环所有节点
2、根据传入的参数,执行对应的命令,start | stop

[root@server110 opt]# vim kafka.sh
#!/bin/bash
case $1 in
"start")
{
        for i in server110 server111 server112
        do
                ssh $i "/opt/modules/kafka_2.11-2.4.1/bin/kafka-server-start.sh -daemon /opt/modules/kafka_2.11-2.4.1/config/server.properties"
                echo --------kafka $i start------------------
        done
}
;;
"stop")
{
        for i in server110 server111 server112
        do
                ssh $i "/opt/modules/kafka_2.11-2.4.1/bin/kafka-server-stop.sh  /opt/modules/kafka_2.11-2.4.1/config/server.properties"
                echo --------kafka $i stop------------------
        done
}
;;
*)
{
        echo "------./kafka.sh start|stop-----"
}
;;
esac

测试

jpsall.sh

[root@server110 opt]# ./jpsall.sh 
----------------jps server110 --------------------
30802 Jps
26823 QuorumPeerMain
17387 NameNode
17916 NodeManager
29500 Kafka
17549 DataNode
----------------jps server111 --------------------
17265 DataNode
17540 NodeManager
128820 QuorumPeerMain
130404 Jps
129287 Kafka
17383 ResourceManager
129638 ConsoleProducer
18029 JobHistoryServer
----------------jps server112 --------------------
24642 ConsoleConsumer
17207 DataNode
24264 Kafka
17289 SecondaryNameNode
25436 Jps
23773 QuorumPeerMain
17407 NodeManager

kafka.sh

stop

[root@server110 opt]# ./kafka.sh stop
--------kafka server110 stop------------------
--------kafka server111 stop------------------
--------kafka server112 stop------------------
[root@server110 opt]# ./jpsall.sh 
----------------jps server110 --------------------
26823 QuorumPeerMain
17387 NameNode
31067 Jps
17916 NodeManager
17549 DataNode
----------------jps server111 --------------------
17265 DataNode
17540 NodeManager
128820 QuorumPeerMain
130644 Jps
17383 ResourceManager
18029 JobHistoryServer
----------------jps server112 --------------------
17207 DataNode
17289 SecondaryNameNode
23773 QuorumPeerMain
25662 Jps
17407 NodeManager

start

[root@server110 opt]# ./kafka.sh start
--------kafka server110 start------------------
--------kafka server111 start------------------
--------kafka server112 start------------------
[root@server110 opt]# ./jpsall.sh 
----------------jps server110 --------------------
26823 QuorumPeerMain
31450 Kafka
17387 NameNode
17916 NodeManager
17549 DataNode
31534 Jps
----------------jps server111 --------------------
17265 DataNode
17540 NodeManager
128820 QuorumPeerMain
131028 Kafka
17383 ResourceManager
18029 JobHistoryServer
334 Jps
----------------jps server112 --------------------
17207 DataNode
17289 SecondaryNameNode
23773 QuorumPeerMain
26046 Kafka
26110 Jps
17407 NodeManager

zk.sh

stop

[root@server110 opt]# ./zk.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.5.8/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
----------server110 zookeeper stop-----------------
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.5.8/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
----------server111 zookeeper stop-----------------
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.5.8/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
----------server112 zookeeper stop-----------------
[root@server110 opt]# ./jpsall.sh 
----------------jps server110 --------------------
17387 NameNode
17916 NodeManager
17549 DataNode
31679 Jps
----------------jps server111 --------------------
17265 DataNode
17540 NodeManager
470 Jps
17383 ResourceManager
18029 JobHistoryServer
----------------jps server112 --------------------
26276 Jps
17207 DataNode
17289 SecondaryNameNode
17407 NodeManager

start

[root@server110 opt]# ./zk.sh start
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.5.8/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
----------server110 zookeeper start-----------------
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.5.8/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
----------server111 zookeeper start-----------------
ZooKeeper JMX enabled by default
Using config: /opt/modules/zookeeper-3.5.8/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
----------server112 zookeeper start-----------------
[root@server110 opt]# ./jpsall.sh 
----------------jps server110 --------------------
31744 QuorumPeerMain
17387 NameNode
17916 NodeManager
17549 DataNode
31805 Jps
----------------jps server111 --------------------
17265 DataNode
595 Jps
17540 NodeManager
532 QuorumPeerMain
17383 ResourceManager
18029 JobHistoryServer
----------------jps server112 --------------------
26338 QuorumPeerMain
17207 DataNode
17289 SecondaryNameNode
26394 Jps
17407 NodeManager

完成

原文链接:https://blog.csdn.net/sinat_25528181/article/details/120595568



所属网站分类: 技术文章 > 博客

作者:小泽圈儿郎

链接:http://www.javaheidong.com/blog/article/297036/0ad75e156a605f5cb634/

来源:java黑洞网

任何形式的转载都请注明出处,如有侵权 一经发现 必将追究其法律责任

12 0
收藏该文
已收藏

评论内容:(最多支持255个字符)