FlinkOnYarnPerJob实战

采用Job分离模式:每个Flink Job运行，都会申请资源，运行属于自己的Flink 集群。

xxxxxxxxxx
/opt/flink-yarn/bin/flink run \
-t yarn-per-job -m yarn-cluster \
-yjm 1024 -ytm 1024 -ys 1 \
/opt/flink-yarn/examples/batch/WordCount.jar \
--input hdfs://mycluster/wordcount/input

#bin/flink run参数说明
-t:指定运行模式"local", "yarn-per-job", "yarn-session"...
-m:指定需要连接的jobmanager(主节点)地址，指定为 yarn-cluster表示使用Yarn集群做资源管理。
-yjm:JobManager可用内存，单位兆
-ytm:每个TM所在的Container可申请多少内存，单位兆
-ys:每个TM会有多少个Slot
-yd:分离模式（后台运行，不指定-yd, 终端会卡在提交的页面上）
-c: 主类全路径名（一个jar包中有多个主类，可以通过该选项指定）

查看日行日志：

org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface node2:45593 of application 'application_1670549096584_0001'. Job has been submitted with JobID 7a613dcb86fc7cc019b7ad0370779a29 Program execution finished Job with JobID 7a613dcb86fc7cc019b7ad0370779a29 has finished. Job Runtime: 9892 ms Accumulator Results:

67d5ba5e171dadef6b4255cb9f2d9de6 (java.util.ArrayList) [6 elements]

(andy,3) (hello,5) (joy,3) (mark,1) (rose,2) (tom,2)

提交Flink Job在Hadoop YARN执行时，最后给出如下错误提示：

解决办法：

• 在 flink 配置文件 flink-conf.yaml里添加一行配置

  xxxxxxxxxx
  [root@node1 ~]# vim /opt/flink-yarn/conf/flink-conf.yaml
  classloader.check-leaked-classloader: false
  

• 同步到其它服务器

  xxxxxxxxxx
  [root@node1 ~]# scp /opt/flink-yarn/conf/flink-conf.yaml root@node2:/opt/flink-yarn/conf/
  [root@node1 ~]# scp /opt/flink-yarn/conf/flink-conf.yaml root@node3:/opt/flink-yarn/conf/
  [root@node1 ~]# scp /opt/flink-yarn/conf/flink-conf.yaml root@node4:/opt/flink-yarn/conf/
  

• 重新执行提交命令，提示消失。

注意：执行过程中在其中一个节点中出现YarnJobClusterEntrypoint进程，执行完后消失。

点击ID