Flink常见问题

1.java.lang.NoClassDefFoundError: org/apache/flink/api/java/utils/ParameterTool

启动本地Flink程序,出现如下错误:

问题

Connected to the target VM, address: '127.0.0.1:57580', transport: 'socket'
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/utils/ParameterTool
 at com.gtcom.governance.utils.ExecutionEnvUtils.createParameterTool(ExecutionEnvUtils.java:27)
 at com.gtcom.governance.DataGovernanceApplication.main(DataGovernanceApplication.java:16)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.utils.ParameterTool
 at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
 ... 2 more

原因

1、我们很多flink在生产部署的时候，在flink启动程序中会包含常见的一些flink包,所以咱们本地依赖的包只需要在编译时候起作用，在程序运行时候不会打进我们的最后代码中，所以用到了:provided的maven命名空间。
2、provided的作用是：当项目打成war包时，当前的jar包不会压缩在当前的war包里
只在当前代码的编写和编译过程中产生依赖的作用。
3、编译部署的时候没有将依赖全部导入，因为本人用的是快速原型导入的maven，里面scope好多是provided的，这样导致依赖在编译的时候无法全部加载，生产环境的话如果已经装好了环境，

解决方式

那么这个在生产上跑是没问题的，但是调试的时候需要将这个勾选，否则会报错！

2.yarn-session模式下提交任务在dashboard中不展示

操作

启动yarn-session模式:

$FLINK_HOME/bin/yarn-session.sh -jm 1024m -tm 4096m

然后提交任务

$FLINK_HOME/bin/flink run $FLINK_HOME/examples/batch/WordCount.jar

然后发现在Apache Flink Dashboard中并没有出现任务。

原因

之前通过

$HADOOP_HOME/bin/yarn application -kill  application_1599402747874_0001（此application_1599402747874_0001就是我们的上面的application_id）

杀掉了我们的yarn session进程，但是我们通过jps -l查看进程的时候，还是出现以下进程:

6598 org.apache.hadoop.hdfs.server.datanode.DataNode
9463 org.apache.hadoop.yarn.server.nodemanager.NodeManager
19768 sun.tools.jps.Jps
9257 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
23260 org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
23612 org.apache.flink.runtime.taskexecutor.TaskManagerRunner
16974 org.apache.flink.yarn.cli.FlinkYarnSessionCli

然后通过杀掉:flink相关进程 然后重新启动yarn 再提交任务恢复

3.flink on yarn——报错：Deployment took more than 60 seconds. Please check if the requested resources are

系统启动后一直报错:

022-12-29 16:47:39,204 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:39,454 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:39,705 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:39,955 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:40,206 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:40,456 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:40,707 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-12-29 16:47:40,958 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
^C2022-12-29 16:47:40,998 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cancelling deployment from Deployment Failure Hook
2022-12-29 16:47:41,000 INFO  org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider [] - Connecting to ResourceManager at hadoop214/172.21.81.214:8032

似乎是资源不够使用。
解决方式:
解决方案：配置yarn-site.xml

  <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>102400</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>32</value>
    </property>
   <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>51200</value>
    </property>