日志采集实战

1. 启动zk集群(node2、node3、node4)，并查看状态。

   xxxxxxxxxx
   [root@node2 ~]# zkServer.sh start
   ZooKeeper JMX enabled by default
   Using config: /opt/zookeeper-3.5.7/bin/../conf/zoo.cfg
   Starting zookeeper ... STARTED
   [root@node2 ~]# zkServer.sh status
   ZooKeeper JMX enabled by default
   Using config: /opt/zookeeper-3.5.7/bin/../conf/zoo.cfg
   Client port found: 2181. Client address: localhost.
   Mode: follower
   

2. node2上调用脚步启动Kafka集群(node2、node3、node4)。

   xxxxxxxxxx
   [root@node2 ~]# kafka.sh start
   --------start node2's kafka----------
   --------start node3's kafka----------
   --------start node4's kafka----------
   

3. 在Kafka中创建对应的日志接收topic

   xxxxxxxxxx
   [root@node2 ~]# kafka-topics.sh --bootstrap-server node3:9092 --create --partitions 3 --replication-factor 2 --topic userlog
   Created topic userlog.
   [root@node2 ~]# kafka-topics.sh --bootstrap-server node3:9092 --list
   ......
   userlog
   

4. node3节点上将日志接口程序启动

   xxxxxxxxxx
   [root@node3 ~]# cd /opt/www/
   [root@node3 www]# ls
   E:  LogCollector-0.0.1-SNAPSHOT.jar
   [root@node3 www]# java -jar LogCollector-0.0.1-SNAPSHOT.jar 
   
     .   ____          _            __ _ _
    /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
   ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
    \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
     '  |____| .__|_| |_|_| |_\__, | / / / /
    =========|_|==============|___/=/_/_/_/
    :: Spring Boot ::        (v2.2.5.RELEASE)
   
   Starting UserlogcollectorApplication v0.0.1-SNAPSHOT on node3 with PID 3443 
   ......
   Started UserlogcollectorApplication in 1.392 seconds (JVM running for 1.685)
   

5. node3上启动Flume日志采集脚本监控目录日志

   xxxxxxxxxx
   flume-ng agent --name a -f /opt/flume-1.9.0/jobs/music_user.properties -Dflume.root.logger=INFO,console
   

6. node4或node2上启动一个kafka消费者

   xxxxxxxxxx
   kafka-console-consumer.sh --bootstrap-server node4:9092 --topic userlog
   

7. 修改MakeData项目的ProdeceUserLoginLog.java类，然后运行该类

   xxxxxxxxxx
   String url="http://node3:8686/collector/common/"+logType;
   

8. 首先IDEA控制台如下，表示信息发送成功！

   {"code":"0000","message":"success","logType":"userLoginLog"} {"code":"0000","message":"success","logType":"userLoginLog"} {"code":"0000","message":"success","logType":"userLoginLog"} {"code":"0000","message":"success","logType":"userLoginLog"} ......

9. node3 spring控制台

   {mid:92238,uid:40612} {mid:82543,uid:5950} {mid:2549,uid:99406} {mid:80567,uid:5007}

   ......

10. 查看flume消费者端：

    [root@node4 ~]# kafka-console-consumer.sh --bootstrap-server node4:9092 --topic userlog

    {mid:92238,uid:40612} {mid:82543,uid:5950} {mid:2549,uid:99406} {mid:80567,uid:5007}

    ......

    说明数据采集接口、Flume日志采集、以及Kafka等日志数据流转过程均正常。