ST5

社区小伙伴们,Apache SeaTunnel 5 月份月报姗姗来迟啦!

据社区统计,2026 年 5 月 apache/seatunnel 仓库累计合入 87 个 PR。本月核心迭代方向为持续完善 Connector-V2,补齐各类细节,做到生产环境可用;深耕 Zeta 引擎,优化高可用、故障恢复、监控和测试;同时强化 CI 安全与回归测试,保障主干分支迭代效率。

本月报内容包含:

  • 详细梳理当月合入 PR 的功能特性、性能优化、Bug 修复及架构改进(含全量清单)。
  • 深入分析关键技术变更的实现原理与影响范围(含 patch 级代码片段)。
  • 提供性能/稳定性对比的可复现实测口径(不虚构跑分)。
  • 总结版本演进趋势与技术发展方向。
  • 文章末尾列出 2026 年 5 月所有 PR 提交者(GitHub 用户名、贡献分类统计、贡献度排名)。

1. 当月总体画像

1.1 四大维度统计(87 PR)

从数据上来看,本月更像“把能力做实”的月份:大量修复集中在 HA/恢复/边界条件/资源与内存风险/可观测等生产痛点。

1.2 模块分布

  • seatunnel-connectors-v2:32
  • seatunnel-engine:21
  • seatunnel-connectors-v2/connector-cdc:8
  • seatunnel-e2e:6
  • docs:3
  • other:17

2. 当月功能特性与工程演进梳理

2.1 Connector-V2:HTTP Source 支持二进制下载(#10956)

这是本月最具“用户可感知”的连接器能力升级之一:把 HTTP Source 从“拉 JSON/text”扩展到“拉文件/二进制”。

从 PR patch 的变更摘要可直接验证(15 files changed, +758/-17):

  • 新增 format = "binary"
  • 新增 binary_chunk_size(默认 10MB)对大文件拆分;
  • 输出 schema 固定为 (data: bytes, relativePath: string, partIndex: long)
  • 限定 BATCH 模式;
  • 同步补齐 UT/E2E/文档(中英文)。

官方示例:

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Http {
    url = "http://example.com/files/report.pdf"
    method = "GET"
    format = "binary"
    binary_chunk_size = 10485760
    schema = {
      fields {
        data = bytes
        relativePath = string
        partIndex = long
      }
    }
  }
}

sink {
  LocalFile {
    path = "/tmp/download"
    file_format = "binary"
  }
}

影响范围与升级建议:

  • 对存量任务:只要默认 format=text 不变,影响可控;但升级后建议用小流量验证下游 transform/sink 是否正确处理 bytes 类型字段。
  • 对大文件:binary_chunk_size 把单次处理的峰值内存从“文件大小级别”压缩到“chunk 级别”,显著降低 OOM 风险(可用 RSS/GC 指标做量化验证)。

2.2 Engine/Zeta:dry-run 渐进式校验(#10763)

[Feature][Zeta] Implement proper dry-run mode with progressive validation layer0

工程价值:

  • 把配置错误从“运行时失败”前移到“提交前/启动前失败”,降低生产回滚成本。
  • 为后续做更复杂的“渐进式校验层”(例如 connector 参数校验、schema 约束、权限/资源预检)提供框架基础。

2.3 生态能力试点与工具链

本月的特性提交还体现为“控制面/可观测/边缘采集/开发者生态”的补齐:

  • 10878:STIP-24 Phase 1 EdgeSocket ingress(边缘采集 MVP)

  • 10491:数据血缘与性能分析 trace

  • 10184:Python SDK Client(SeaTunnel REST API)

  • 多表与 schema 演进(RabbitMQ/Cassandra/SqlServer CDC/Postgres-CDC 等)

3. 性能优化与资源风险治理

3.1 Kudu:依赖升级解决 Flink 1.15+ 兼容性风险(#10974)

该 PR 的价值更偏“可运行性/兼容性”,但标题为 Improve+Upgrade,被归为“性能优化”类。

实现原理:

  • 升级 kudu-client 版本以减少与 Flink 1.15+ classpath 冲突风险;
  • 去除对 Kudu shaded Guava 的显式引用,避免依赖内部 shaded API 漂移导致的运行时问题。

可复现实测口径(建议):

  • 在 Flink 1.15+ 环境跑同一份 Kudu 作业,比较:

    • 作业启动成功率(是否卡住/是否出现 NoClassDefFoundError)
    • 启动耗时分位数(P50/P95)
    • 30 分钟稳定性(是否出现 slot allocation/release loop)

3.2 Kafka:默认缓存队列深度从 1024 降到 2 防 OOM(#10954)

标题明确写出:Reduce default reader_cache_queue_size from 1024 to 2 to prevent OOM

量化解读(可验证推导):

  • 队列深度上限减少 512 倍(1024 -> 2)。
  • 若单条缓存对象占用较大(例如包含反序列化后的 row),该默认值调整可显著降低峰值内存与 GC 压力。

建议对比指标:

  • RSS 峰值、GC 次数、Young/Old GC 耗时、OOM 发生率、吞吐变化(rows/s)。

4. Bug 修复与架构改进是 5 月主旋律

4.1 Zeta:master failover 导致 job 永久卡住(#10836)

标题信息已给出影响范围:affects BATCH / bounded source / job shutdown phase

此问题可能导致典型生产危害:

  • 作业进入“应结束”状态但无法完成 -> 资源泄漏、重复调度、数据落库不一致风险上升。

建议上线验证:

  • 模拟 master failover(或节点故障)后,观察:

    • job 是否能走完 shutdown phase 并终态一致
    • 状态清理是否存在竞态(可结合 #10687 等修复一起观察)

4.2 安全修复:日志 REST API 路径穿越(#10628)

[Fix][Zeta] Fix path traversal vulnerability in log file REST API endpoints

建议运维动作:

  • 升级后在网关/Ingress 层补充访问控制(仅授权用户可访问日志 API)。
  • 配合安全扫描验证路径穿越已被阻断(构造 ../ 请求)。

4.3 竞态/恢复链路治理(#10687 / #10842 / #10877 等)

这些修复共同指向 Zeta 在 5 月显著加强“故障后恢复链路”的健壮性,尤其是节点失败后的 state 清理竞态与 restore 过程信息复用。

5. 关键技术变更深度分析

5.1 HTTP 二进制下载(#10956):设计、语义与影响范围

核心点:

  • format=binary 将 HTTP 响应体作为 raw bytes;
  • binary_chunk_size 控制大文件拆分;
  • 输出 schema 固定化,便于下游 sink 实现明确的写入语义;
  • 限定 BATCH,避免与流式“持续下载”语义混淆。

对用户的直接价值:

  • HTTP 既能“拉接口数据”,也能“拉文件”,降低链路复杂度。

风险与建议:

  • chunk 拆分后,下游 sink 必须具备“按 relativePath+partIndex 拼接/写入”的语义;建议优先配合官方提供的 binary file sink 或具备 bytes 写入能力的 sink。

5.2 SQLite upsert 语法修正(#10880):为什么是 EXCLUDED

SQLite 的 upsert 语义要求在冲突更新时引用 EXCLUDED.<col> 而非 VALUES(<col>)。该 PR 把 SQL 拼接修正为:

- `col`=VALUES(`col`)
+ `col`=EXCLUDED.`col`

并新增单测做精确断言,降低回归风险。

5.3 Milvus 错误异常引用修复(#10975):排障成本的“非功能性”优化

单行 diff 却是高价值修复:

- loadStateResponse.getException()
+ queryResultsR.getException()

修复后错误信息更真实,避免排障路径走偏。

5.4 Zeta flaky test 修复(#10891):把时序依赖变为可控状态

patch 显示测试从“真实 submitJob + 等待队列”改为“mock server + mock JobHistoryService + 构造 pendingJobDAGInfo”,减少非确定性因素,提升 CI 稳定性。

6. 性能/稳定性对比:统一可复现模板

5 月 PR 大量属于兼容性修复、稳定性治理、默认参数安全性调整,并不一定自带 benchmark/JMH 报告。为了让结论可验证,建议按变更类型选择指标:

  • 默认参数变更类:RSS 峰值、GC、OOM、吞吐变化。
  • chunk/切分策略类:峰值内存、端到端耗时、输出一致性(hash)。
  • 依赖升级兼容类:启动成功率、启动耗时分位数、长跑稳定性。

通用复现模板:

1) 固定数据规模与并行度;
2) 固定环境(CPU/内存/网络);
3) 固定观测指标;
4) 仅切换变更前/后版本;
5) 保留原始日志与指标文件。

感谢所有贡献者

在五月的时光里,Apache SeaTunnel 项目能够蓬勃发展、功能日益强大,离不开每一位贡献代码的小伙伴。衷心感谢大家的辛勤付出与无私奉献,是你们用一行行代码,为项目注入了源源不断的活力!

以下为 5 月份贡献者名单:

RankGitHub 用户名合入 PR 数主要贡献内容分类(计数)
1nzw921rx18功能特性×2; 性能优化×1; Bug 修复×6; 架构改进×9
2zhangshenghang12功能特性×0; 性能优化×0; Bug 修复×8; 架构改进×4
3DanielLeens8功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×7
4CosmosNi8功能特性×1; 性能优化×0; Bug 修复×6; 架构改进×1
5davidzollo5功能特性×1; 性能优化×1; Bug 修复×1; 架构改进×2
6JeremyXin4功能特性×0; 性能优化×0; Bug 修复×0; 架构改进×4
7CloverDew3功能特性×1; 性能优化×0; Bug 修复×2; 架构改进×0
8zhiliang-wu2功能特性×1; 性能优化×0; Bug 修复×0; 架构改进×1
9yzeng16182功能特性×1; 性能优化×0; Bug 修复×1; 架构改进×0
10QuakeWang2功能特性×0; 性能优化×0; Bug 修复×2; 架构改进×0
11dybyte2功能特性×0; 性能优化×0; Bug 修复×2; 架构改进×0
12zhangqs02051功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
13yht08271功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
14suhyeon7291功能特性×0; 性能优化×0; Bug 修复×0; 架构改进×1
15siwen-yu1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
16rucciva1功能特性×1; 性能优化×0; Bug 修复×0; 架构改进×0
17ricky21291功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
18puneetdixit2001功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
19NixonWahome1功能特性×1; 性能优化×0; Bug 修复×0; 架构改进×0
20MyeoungDev1功能特性×0; 性能优化×0; Bug 修复×0; 架构改进×1
21luciobvjr1功能特性×1; 性能优化×0; Bug 修复×0; 架构改进×0
22lm-ylj1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
23junjunclub1功能特性×1; 性能优化×0; Bug 修复×0; 架构改进×0
24hyboll1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
25HexMox1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
26CriysHot1功能特性×0; 性能优化×0; Bug 修复×0; 架构改进×1
27corgy-w1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
28Chris79OG1功能特性×0; 性能优化×0; Bug 修复×0; 架构改进×1
29chocoboxxf1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0
30chl-wxp1功能特性×0; 性能优化×0; Bug 修复×0; 架构改进×1
31Anthippi1功能特性×1; 性能优化×0; Bug 修复×0; 架构改进×0
32Adamyuanyuan1功能特性×0; 性能优化×0; Bug 修复×1; 架构改进×0

(排名规则:按当月合入 PR 数降序;贡献分类为本文四象限分类的计数汇总。)

附录:2026 年 5 月全部合入 PR 清单(87 条)

下表按合入日期排序,逐条列出当月所有 merged PR(含标题、提交者、分类、模块),可帮助读者快速定位:

合入日期PR标题提交者分类模块
2026-05-0210827[Bug] [Zeta] Fix negative array size exceptionlm-yljBug 修复seatunnel-engine
2026-05-0310846[Chore] Reduce dev branch required approving reviews to 1DanielLeens架构改进other
2026-05-0410763Feature Implement proper dry-run mode with progressive validation layer0junjunclub功能特性seatunnel-engine
2026-05-0510733Chore Re-enable dead-link check jobzhangshenghang架构改进seatunnel-engine
2026-05-0510582Fix Support vector index in table schema config parsingchocoboxxfBug 修复seatunnel-connectors-v2
2026-05-0610824BugfixFix the problem of scientific notation when restApi obtains read and write ratechl-wxp架构改进seatunnel-engine
2026-05-0610557Feature[RabbitMQ] Support multi-table read for RabbitMQ source (#10425)Anthippi功能特性seatunnel-connectors-v2
2026-05-0610367Fix Rename uniqueKeyFields to pkNames in dialect upsert APIscorgy-wBug 修复other
2026-05-0610842Fix Reuse job info during restorezhangshenghangBug 修复seatunnel-engine
2026-05-0610712Improve Replace Transform to support array-based replace_fieldsMyeoungDev架构改进seatunnel-e2e
2026-05-0710682 Improve Improve Redshift sink connector documentationdavidzollo架构改进other
2026-05-0710848Fix Fix OpenAI API key documentation linkzhangshenghangBug 修复other
2026-05-0710845BUG[MaxCompute] Fix Maxcompute source split stategyzhiliang-wu架构改进seatunnel-connectors-v2
2026-05-0710836Fix Job stuck permanently after master failover, unable to complete (affects BATCH / bounded source / job shutdown phase)nzw921rxBug 修复seatunnel-engine
2026-05-0810566Improve Enhance diagnostics for missing binlog and GTID during restorezhangshenghang架构改进seatunnel-e2e
2026-05-0810817Fix[File] Respect custom filename for binary sinkzhangshenghangBug 修复seatunnel-connectors-v2
2026-05-0810854Improve Add minimum deployment config and clarify backup-count N definition in Zeta separated cluster guidenzw921rx架构改进other
2026-05-1010841Fix Avoid telemetry collector blocking before CoordinatorService is readynzw921rxBug 修复seatunnel-engine
2026-05-1110862Fix Fix FAQ connector list linkszhangshenghangBug 修复other
2026-05-1110687Fix Guard state cleanup races after node failurezhangshenghangBug 修复seatunnel-engine
2026-05-1210875[Docs] Fix contributor guide dead linksDanielLeens架构改进other
2026-05-1210864Fix Add missing auth options to Chinese Elasticsearch sink docs and remove duplicate sections in English docsCosmosNiBug 修复other
2026-05-1210857Fix Fix Elasticsearch connector connection pool leak …CosmosNiBug 修复seatunnel-connectors-v2
2026-05-1210829BugFix Fix table or view doesn't exist in multitables CDB+PDB modeCriysHot架构改进seatunnel-connectors-v2
2026-05-1210831Feature[MaxCompute] Support Schema Name and Passwordless Authzhiliang-wu功能特性seatunnel-connectors-v2
2026-05-1210866Improve Improved clickhouse source enumerator splits allocation algorithm for subtasksJeremyXin架构改进seatunnel-connectors-v2
2026-05-1310604Feature Add option to disable sampling-based shardingnzw921rx功能特性seatunnel-connectors-v2
2026-05-1310762Fix Require explicit primary keys for Iceberg upsert modericky2129Bug 修复seatunnel-connectors-v2
2026-05-1310877Fix Stabilize coordinator restore testzhangshenghangBug 修复seatunnel-engine
2026-05-1310756Fix Upgrade okio to 3.6.0 to fix NoSuchFieldError: CompanionhybollBug 修复seatunnel-connectors-v2
2026-05-1310844Feature Add optional Markdown RAG metadata for file sourceyzeng1618功能特性seatunnel-connectors-v2
2026-05-1310628Fix Fix path traversal vulnerability in log file REST API endpointsdavidzolloBug 修复seatunnel-engine
2026-05-1310861Improve[JDBC] Add string split strategyzhangshenghang架构改进seatunnel-connectors-v2
2026-05-1310830Fix Fix enumerator re-registration synchronizationzhangshenghangBug 修复seatunnel-engine
2026-05-1410738Fix[MySQL] Preserve table comments, indexes, ENGINE, CHARSET, and COLLATE in auto-created tableszhangshenghangBug 修复seatunnel-e2e
2026-05-1410865Improve Refactor coordinator executor creation and fix failover lifecycle issuesnzw921rx架构改进seatunnel-engine
2026-05-1410884Fix Fix IMap storage SPI registration lost during shade packagingnzw921rxBug 修复seatunnel-engine
2026-05-1610892Docs Fix incorrect config key name in slot-allocation-strateg…suhyeon729架构改进other
2026-05-1710390Fix[JsonPath] Fix date/time conversion error when mul…yht0827Bug 修复other
2026-05-1710894Fix connector-typesense Not imported in seatunnel-distnzw921rxBug 修复other
2026-05-1710893[Bug] [Connector-V2] Fix Milvus sink collection_name target handlingpuneetdixit200Bug 修复seatunnel-connectors-v2
2026-05-1710860[Improve] [Zeta] Add observability metrics for engine state storesJeremyXin架构改进seatunnel-engine
2026-05-1810898fix typesense add missing import for MULTI_TABLE_SINK_REPLICA to pass…nzw921rx架构改进seatunnel-connectors-v2
2026-05-1810901Update collaborator chl-wxp to DanielLeens in .asf.yamlDanielLeens架构改进other
2026-05-2010158Fix Cache writer states during commit preparation to fix streaming mode lost data.AdamyuanyuanBug 修复other
2026-05-2110868Improve Improved milvus source enumerator splits allocation algorithm for subtasksJeremyXin架构改进seatunnel-connectors-v2
2026-05-2110783Improve Optimize changes job engine module calculationdavidzollo性能优化other
2026-05-2110716Improve Update docker-compose cluster setup to use service name discoveryDanielLeens架构改进other
2026-05-2210673Improve Improve health metrics retrieval and parsing with enhanced loggingzhangshenghang架构改进seatunnel-engine
2026-05-2210885Fix Fix MessageContentPartitioner static state causing cross-job partition routing interferenceCosmosNiBug 修复seatunnel-connectors-v2
2026-05-2210895Improve Add connector dist registration completeness checkDanielLeens架构改进other
2026-05-2310902Fix Fix SeaTunnel CLI reasoning replay for tool callsyzeng1618Bug 修复other
2026-05-2310887Build Support JDK23 lombok annotation processingDanielLeens架构改进other
2026-05-2310936[Docs] Fix dead issue link in connector common changelogDanielLeens架构改进other
2026-05-2310648[Fix] [Connector-V2] Fix Flink schema evolution hang caused by XA transaction MDL deadlock with MySQL CDCCloverDewBug 修复seatunnel-e2e
2026-05-2410940Improve Disable HTTP server in engine e2e tests to fix flaky testnzw921rx架构改进seatunnel-engine
2026-05-2410941Fix Prevent input row mutation in ReplaceTransformdybyteBug 修复other
2026-05-2510937Improve Stabilize jdbc e2e assertions and waits nzw921rx架构改进seatunnel-e2e
2026-05-2510896Feature[Cassandra] Add multi-table source support via…NixonWahome功能特性seatunnel-connectors-v2
2026-05-2510947Improve Add CI changes step timeout of 20mnzw921rx架构改进other
2026-05-2510948Fix Fix flaky jdbc-connectors-it-ddl tests with timeout tuning and DM wait strategynzw921rxBug 修复seatunnel-e2e
2026-05-2610706Fix[JDBC] Detect Hive partition keys in metadatazhangqs0205Bug 修复seatunnel-connectors-v2
2026-05-2610952Fix Fix nested ROW type conversion crash in Paimon connectorCosmosNiBug 修复seatunnel-connectors-v2
2026-05-2610184Feature Python SDK Client for SeaTunnel REST APIluciobvjr功能特性other
2026-05-2610334Feature[Postgres-CDC] Allow replica identity other than fullrucciva功能特性seatunnel-connectors-v2
2026-05-2710954Fix Reduce default reader_cache_queue_size from 1024 to 2 to prevent OOMnzw921rxBug 修复seatunnel-connectors-v2
2026-05-2710878Feature STIP-24 Phase 1: Support lightweight EdgeSocket ingress for edge collector MVPnzw921rx功能特性seatunnel-connectors-v2
2026-05-2710934FixJdbc Fix Dameng auto-create-table fails when table has column commentssiwen-yuBug 修复seatunnel-connectors-v2
2026-05-2710922Split unrelated client summary and Pulsar E2E follow-ups from PR 10491davidzollo架构改进seatunnel-e2e
2026-05-2710491[Feature] Add stain trace for data lineage and performance analysisdavidzollo功能特性seatunnel-engine
2026-05-2710890Feature[SqlServer CDC] support sqlserver schema evolutionCloverDew功能特性seatunnel-connectors-v2
2026-05-2710888Fix Fix Paimon nested row writeQuakeWangBug 修复seatunnel-connectors-v2
2026-05-2710859Fix Avoid replaying completed snapshot splits after restoreDanielLeensBug 修复seatunnel-connectors-v2
2026-05-2810867Improve Improved starrocks source enumerator splits allocation algorithm for subtasksJeremyXin架构改进seatunnel-connectors-v2
2026-05-2810966Fix Fix SMALLINT switch fall-through in Milvus source converterCosmosNiBug 修复seatunnel-connectors-v2
2026-05-2810932[hotfix] [docs] Add managed Kubernetes deployment notesChris79OG架构改进other
2026-05-2810946Improve Improve the stability of Kafka e2e testingnzw921rx架构改进seatunnel-e2e
2026-05-2910970Fix[SqlServer CDC] Fix SQL Server CDC DDL history query LSN boundary causing schema events to be lostCloverDewBug 修复seatunnel-connectors-v2
2026-05-2910454Improve Add slicing support and e2e coverage for Elasticsearch sourceCosmosNi架构改进seatunnel-connectors-v2
2026-05-3010974Improve Upgrade kudu-client from 1.11.1 to 1.15.0nzw921rx性能优化seatunnel-connectors-v2
2026-05-3010972Improve Improve the stability of sftp e2e testing e2enzw921rx架构改进seatunnel-e2e
2026-05-3010971Improve Improve the stability of Kudu e2e testing e2enzw921rx架构改进seatunnel-e2e
2026-05-3010975Fix Fix wrong exception reference in MilvusSourceReader query error handlingCosmosNiBug 修复seatunnel-connectors-v2
2026-05-3010956Feature Add binary format support for HTTP source connectorCosmosNi功能特性seatunnel-connectors-v2
2026-05-3010880Fix Update upsert SQL syntax and add unit tests for …HexMoxBug 修复seatunnel-connectors-v2
2026-05-3010891Fix Fix flaky pending job info testQuakeWangBug 修复seatunnel-engine
2026-05-3010853Fix Pin azure/setup-helm to approved SHAdybyteBug 修复other

SeaTunnel
110 声望28 粉丝

Apache SeaTunnel是下一代高性能、分布式、海量数据集成框架。通过我们的努力让数据同步更简单,更高效,大幅减少学习成本,加快分布式数据处理能力在生产环境落地。