前端响应超时、API-server 服务内存不足...碰见这类 DolphinScheduler 资源中心相关问题怎么办？

作者 | 刘森卡特加特大数据工程师
最近，有些用户小伙伴反映在使用 Apache DolphinScheduler 资源中心时会遇到问题，社区小伙伴整理了一些常见问题，希望帮大家解决燃眉之急。
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">[WARN] 2023-04-25 03:02:58.463 +0000 org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver:[434] - Failure in @ExceptionHandler org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler#exceptionHandler(Exception, HandlerMethod) java.lang.IllegalStateException: Could not resolve parameter [1] in public org.apache.dolphinscheduler.api.utils.Result org.apache.dolphinscheduler.api.exceptions.ApiExceptionHandler.exceptionHandler(java.lang.Exception,org.springframework.web.method.HandlerMethod): No suitable resolver at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:176) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:146) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428) at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75) at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142) at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80) at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1330) at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1141) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1087) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:517) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)</pre>
问题原因：前端响应超时
解决办法：
需要修改以下4个文件。
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">/api-server/ui/assets/service.33119ae9.js /api-server/ui/assets/service.33119ae9.js.gz /ui/assets/service.33119ae9.js /ui/assets/service.33119ae9.js.gz</pre>
并找到以下配置，然后将 15e3 修改为 15e5 或更高版本。
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">baseURL:"/dolphinscheduler",timeout:15e3</pre>
1.这里 service.33119ae9.并不是固定的名称，只要找到 service..js*的文件修改就行
2.如果使用docker部署的dolphinscheduler。是没有/ui/assets/service.33119ae9.js和/ui/assets/service.33119ae9.js.gz文件，忽略就行
3..gz文件使用vim就可以打开
4.重启api服务
5.清除浏览器缓存
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">[WARN] 2023-04-25 03:05:50.855 +0000 org.eclipse.jetty.server.HttpChannel:[776] - /dolphinscheduler/resources org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.lang.OutOfMemoryError: Java heap space at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:517) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)     at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)     at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)     at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)     at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)     at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)     at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)</pre>
问题原因：
api-server服务内存不足
解决办法：
更改bin/dolphinscheduler-daemon.sh的api-server项内存大小参数
Docker部署解决办法：
1.进入Docker实例
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">docker exec -it bbd60fe717af /bin/bash</pre>
2.修改/opt/dolphinscheduler/bin/start.sh
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">vim /opt/dolphinscheduler/bin/start.sh</pre>
3.内存参数从‘-Xms1g -Xmx1g -Xmn512m’ 修改为‘-Xms2g -Xmx2g -Xmn1g’。具体数值根据需要资源修改
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # BIN_DIR=$(dirname $0) DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..; pwd)} source "$DOLPHINSCHEDULER_HOME/conf/dolphinscheduler_env.sh" JAVA_OPTS=${JAVA_OPTS:-"-server -Duser.timezone=${SPRING_JACKSON_TIME_ZONE} -Xms2g -Xmx2g -Xmn1g -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof"} if [[ "$DOCKER" == "true" ]]; then   JAVA_OPTS="${JAVA_OPTS} -XX:-UseContainerSupport" fi $JAVA_HOME/bin/java $JAVA_OPTS \ -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \ org.apache.dolphinscheduler.api.ApiApplicationServer </pre>
3.12 BUG
<ul data-tool="mdnice编辑器" style="margin-top: 8px; margin-bottom: 8px; padding-left: 25px; color: black; list-style-type: disc;">
<li><section style="margin-top: 5px; margin-bottom: 5px; line-height: 26px; text-align: left; color: rgb(1,1,1); font-weight: 500;">修复任务组可能被多个任务抢走的问题 #13115</section></li></ul>
如果设置任务组，多个任务会抢占任务组不释放，导致master一直重试释放，无法继续执行任务
解决办法：
1.升级到3.13
2.取消任务组
<pre class="custom" data-tool="mdnice编辑器" style="margin-top: 10px; margin-bottom: 10px; border-radius: 5px; box-shadow: rgba(0, 0, 0, 0.55) 0px 2px 10px;"><span style="display: block; background: url(https://files.mdnice.com/user/3441/876cad08-0422-409d-bb5a-08afec5da8ee.svg); height: 30px; width: 100%; background-size: 40px; background-repeat: no-repeat; background-color: #282c34; margin-bottom: -7px; border-radius: 5px; background-position: 10px 10px;">[INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-357 - Begin to handle state event, TaskStateEvent(processInstanceId=357, taskInstanceId=4553, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=357-4553, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-357 - Handle task instance wait task group event, taskInstanceId: 4553 [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-356 - Begin to handle state event, TaskStateEvent(processInstanceId=356, taskInstanceId=4554, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=356-4554, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-356 - Handle task instance wait task group event, taskInstanceId: 4554 [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-360 - Begin to handle state event, TaskStateEvent(processInstanceId=360, taskInstanceId=4613, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=360-4613, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-360 - Handle task instance wait task group event, taskInstanceId: 4613 [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[286] - WorkflowInstance-355 - Begin to handle state event, TaskStateEvent(processInstanceId=355, taskInstanceId=4549, taskCode=0, status=null, type=WAIT_TASK_GROUP, key=355-4549, channel=null, context=null) [INFO] 2023-04-25 08:51:06.044 +0000 org.apache.dolphinscheduler.server.master.event.TaskWaitTaskGroupStateHandler:[34] - WorkflowInstance-355 - Handle task instance wait task group event, taskInstanceId: 4549</pre>
</section>

本文由白鲸开源科技提供发布支持！

前端响应超时、API-server 服务内存不足...碰见这类 DolphinScheduler 资源中心相关问题怎么办？

海豚调度

引用和评论

基于DolphinScheduler抽取通用EventBus组件：支持延迟与事件驱动

分布式数据库解析

做到真正0丢失、0重复：Apache SeaTunnel 实现万亿级数据一致性全解密

在 Kubernetes 上用 KubeBlocks + Dify 快速构建生产级 AIGC 应用

数据库的下一场革命：S3 延迟已降至原先的 10%，云数据库架构该进化了

Ape-DTS：开源 DTS 工具，助力自建 MySQL、PostgreSQL 迁移上云

好用的开源埋点方案-ClkLog埋点用户分析系统