IDEA调试Flink任务FAQ

自己写着玩的时候遇到的问题在此整理一下，要能捎带手帮各位看官解了惑那真是倍感荣幸，Flink版本1.12.0

IDEA调试Flink任务FAQ

• • IDEA调试Flink任务时需要WebUI
  • • 1.问题描述
    • 2.原因定位
    • 3.解决方法
  • IDEA调试FLink任务时想看运行日志
  • • 1.问题描述
    • 2.原因定位
    • 3.解决方法
  • checkpoint失败
  • • 1.问题描述
    • 2.原因定位
    • 3.解决方法

IDEA调试Flink任务时需要WebUI 1.问题描述

在idea进行flink任务开发时，希望可以通过web ui查看一些信息

2.原因定位

在本地调试的时候时通过StreamExecutionEnvironment.createLocalEnvironment()创建执行时上下文，该方法未启动 web monitoring UI，需要通过createLocalEnvironmentWithWebUI(Configuration)方法创建上下文，该方法不仅会创建本地执行环境，同时会启动web monitoring UI

3.解决方法

添加依赖

<dependency>
	<groupId>org.apache.flinkgroupId>
	<artifactId>flink-runtime-web_${scala.version}artifactId>
	<version>${flink.version}version>
dependency>

通过createLocalEnvironmentWithWebUI(Configuration)方法创建执行时上下文

Configuration conf = new Configuration();
//set web monitoring ui port
conf.setInteger(RestOptions.PORT, 12345);
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

如果不指定端口号可不可以？答案是可以！If the configuration key 'rest.port' was set in the configuration, that particular port will be used for the web UI. Otherwise, the default port (8081) will be used.

IDEA调试FLink任务时想看运行日志 1.问题描述

在idea调试flink任务时候想看日志，但是控制台只有三行干巴巴的提示，这时候该怎么办？不要慌，人家都给你链接提示你了

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

2.原因定位

点进链接，我们找到对应的原因，翻译一下就是少包，那咱就加一下

3.解决方法

添加依赖

<dependency>
	<groupId>org.slf4j</groupId>
	<artifactId>slf4j-simple</artifactId>
	<version>1.7.25</version>
</dependency>

添加完依赖以后，控制台的日志就像便秘以后抹了开塞露，裤衩裤衩都显示出来了。

checkpoint失败 1.问题描述

在本地调试flink任务时，发现状态checkpoint失败

2.原因定位

查看日志发现状态大小超过默认最大值，由于代码中没有指定StateBackend，因此默认使用MemoryStateBackend，内存中默认为5MB，The default maximal size that the snapshotted memory state may have (5 MiBytes).

java.util.concurrent.ExecutionException: java.io.IOException: Size of the state is larger than the maximum permitted memory-backed state. Size=846910422 , maxSize=5242880 . Consider using a different state backend, like the File System State backend.
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:583)
	at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:53)
	at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:115)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Size of the state is larger than the maximum permitted memory-backed state. Size=846910422 , maxSize=5242880 . Consider using a different state backend, like the File System State backend.
	at org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory.checkSize(MemCheckpointStreamFactory.java:64)
	at org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory$MemoryCheckpointOutputStream.closeAndGetBytes(MemCheckpointStreamFactory.java:145)
	at org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory$MemoryCheckpointOutputStream.closeAndGetHandle(MemCheckpointStreamFactory.java:126)
	at org.apache.flink.runtime.state.CheckpointStreamWithResultProvider$PrimaryStreamOnly.closeAndFinalizeCheckpointStreamResult(CheckpointStreamWithResultProvider.java:77)
	at org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:199)
	at org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:158)
	at org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:75)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:580)
	... 5 more

3.解决方法

MemoryStateBackend类中可以指定maxStateSize大小，可通过MemoryStateBackend(int maxStateSize)方法进行指定，方法如下

StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
env.setStateBackend(new MemoryStateBackend(1048576000));

在进行大状态存储时还是推荐使用其他的StateBackend方式，不建议使用MemoryStateBackend，具体原因可参考官网，不在此处赘述。