flume hdfs sink 1个sink同时写500个分区如何配置效率高一些?

agent.sinks.k3.channel = c2
agent.sinks.k3.type = hdfs
agent.sinks.k3.hdfs.path = hdfs://10.31.13.198/flume/%{log_name}/log_date=%{log_date}/server_id=%{server}/
agent.sinks.k3.hdfs.fileType = SequenceFile
agent.sinks.k3.hdfs.codeC = gzip
agent.sinks.k3.hdfs.writeFormat = Text
agent.sinks.k3.hdfs.rollSize = 134217728
agent.sinks.k3.hdfs.rollCount = 0
agent.sinks.k3.hdfs.threadsPoolSize = 500
agent.sinks.k3.hdfs.rollTimerPoolSize = 50
agent.sinks.k3.hdfs.rollInterval = 1800
agent.sinks.k3.hdfs.batchSize = 2000
agent.sinks.k3.hdfs.minBlockReplicas = 1

这是我的配置,希望能提一些建议

阅读 1.8k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
logo
七牛云问答
子站问答
访问
宣传栏