使用FlinkSQL查询MySQL写入Greenplum, 出现Java heap space问题,可以通过配置避免吗?

新手上路,请多包涵

环境是本机用docker部署的1.11.2版flink集群,就一个task manager。
尝试在SQL Client里查一张50w数据的mysql表并写入GP,结果task manager报Java heap space异常。似乎是select的数据量太大,导致频繁GC。
试过配置scan.fetch-size和lookup.cache.max-rows,没有效果。请问有什么办法能避免呢?
Task manager报错信息如下:

java.lang.IllegalArgumentException: open() failed.Java heap space
at org.apache.flink.connector.jdbc.table.JdbcRowDataInputFormat.open(JdbcRowDataInputFormat.java:209) ~[flink-connector-jdbc_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:85) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) ~[flink-dist_2.12-1.11.2.jar:1.11.2]

at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:213) ~[flink-dist_2.12-1.11.2.jar:1.11.2]
Caused by: java.sql.SQLException: Java heap space
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1005) ~[flink-sql-connector-mysql-cdc-1.0.0.jar:1.0.0]
at org.apache.flink.connector.jdbc.table.JdbcRowDataInputFormat.open(JdbcRowDataInputFormat.java:206) ~[flink-connector-jdbc_2.12-1.11.2.jar:1.11.2]

阅读 3.5k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进