kafka 集群测试与socket api

北风之神

本文不是讲怎么用kafka,是讲它运行时调用哪些核心标准c函数,属于底层基础知识。

图片.png
程序源码
java/py/node/go/rust/php/c++这些语言编写的ascii文本文件 我们称为源码

程序
大部分已经经过语言处理器【如编译器】处理好的文件,大部分在linux下一定的格式存储。此类程序大部分含有代码+数据的二进制格式。

进程
在linux终端下大部分经过bash控制进程启动它,并且启动的子进程会调用系统函数加载程序到内存中运行。

进程调用API
进程能调用的api大部分是linux提供的标准库api函数,但最终是调用系统syscall.

演示示例
1 kafka下载及quickstart
http://kafka.apache.org/docum...

2 启动各个服务进程
图片.png
图片.png
3 创建一个主题
图片.png
.sh文件是shell脚本文件,对shell编程不太了解的自行百度学习。
4 启动2个消费者进程和1个数据生产者进程
图片.png

5 消费者进程运行时调用的相关系统函数syscall
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(2)=[{"\200\0\0\0d\4\1\0\0\0\0\0\0\0\0\0", 16}, {"bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.79.140:9092\0", 112}], msg_controllen=0, msg_flags=0}, 0) = 128 <0.000018>

bash 控制进程得到终端输入的数据【控制进程的输入输出由伪终端驱动程序提供,数据来源于TCP,对控制进程不了解的记得百度,这是最简单的基础知识了】
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffaa55b4a10) = 17999 <0.000085>
克隆创建新任务【linux中将进程,线程称为任务】

setpgid(17999, 17999) = 0 <0.000011>
将自己设置为组长进程
execve("bin/kafka-console-consumer.sh", ["bin/kafka-console-consumer.sh", "--topic", "quickstart-events", "--from-beginning", "--bootstrap-server", "192.168.79.140:9092"], [/ 29 vars /]) = 0 <0.000795>
调用execve加载script shell脚本文件,后面的<0.000795> 是指该函数系统调用花费的时间.

open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 <0.000015>
加载dl库,它的功能是可以显示的调用其它的动态库

open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 <0.000152>
加载libc.so动态库,该库提供了大量的函数。

open("bin/kafka-console-consumer.sh", O_RDONLY) = 3 <0.000014>
打开脚本文件

dup2(3, 255) = 255 <0.000010>

read(255, "#!/bin/bash\n# Licensed to the Apache Software Foundation (ASF) under one or more\n# contributor license agreements. See the NOTICE file distributed with\n# this work for additional information regarding copyright ownership.\n# The ASF licenses this file to You under the Apache License, Version 2.0\n# (the \"License\"); you may not use this file except in compliance with\n# the License. You may obtain a copy of the License at\n#\n# http://www.apache.org/license...\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nif [ \"x$KAFKA_HEAP_OPTS\" = \"x\" ]; then\n export KAFKA_HEAP_OPTS=\"-Xmx512M\"\nfi\n\nexec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleConsumer \"$@\"\n", 945) = 945 <0.000011>
读取脚本内容

经过了重重调用,最终调用bin/java elf 文件【花哩胡哨的】
execve("/home/jdk1.8.0_261/bin/java", ["/home/jdk1.8.0_261/bin/java", "-Xmx512M", "-server", "-XX:+UseG1GC", "-XX:MaxGCPauseMillis=20", "-XX:InitiatingHeapOccupancyPercent=35", "-XX:+ExplicitGCInvokesConcur rent", "-XX:MaxInlineLevel=15", "-Djava.awt.headless=true", "-Dcom.sun.management.jmxremote", "-Dcom.sun.management.jmxremote.authenticate=false", "-Dcom.sun.management.jmxremote.ssl=false", "-Dkafka.logs.dir=/home/java/kafka_2.13- 2.7.0/bin/../logs", "-Dlog4j.configuration=file:/home/java/kafka_2.13-2.7.0/bin/../config/tools-log4j.properties", "-cp", ".:/home/jdk1.8.0_261/jre/lib/ext:/home/jdk1.8.0_261/lib/dt.jar:/home/jdk1.8.0_261/lib/tools.jar:/home/java/k afka_2.13-2.7.0/bin/../libs/activation-1.1.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/aopalliance-repackaged-2.6.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/argparse4j-0.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/audience -annotations-0.5.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/commons-cli-1.4.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/commons-lang3-3.8.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect-api-2.7.0.jar:/home/java/kafka_2.13- 2.7.0/bin/../libs/connect-basic-auth-extension-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect-file-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect-json-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect- mirror-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect-mirror-client-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect-runtime-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/connect-transforms-2.7.0.jar:/home/j ava/kafka_2.13-2.7.0/bin/../libs/hk2-api-2.6.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/hk2-locator-2.6.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/hk2-utils-2.6.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-annotation s-2.10.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-core-2.10.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-databind-2.10.5.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/home/java /kafka_2.13-2.7.0/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/home/java/kafka_2. 13-2.7.0/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-module-paranamer-2.10.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jackson-module-scala_2.13-2.10.5.jar:/home/java/kaf ka_2.13-2.7.0/bin/../libs/jakarta.activation-api-1.2.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jakarta.annotation-api-1.3.5.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jakarta.inject-2.6.1.jar:/home/java/kafka_2.13-2.7.0/bin/.. /libs/jakarta.validation-api-2.0.2.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/javassist-3.2 5.0-GA.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/javassist-3.26.0-GA.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/home/java/kafka_2.13- 2.7.0/bin/../libs/jaxb-api-2.3.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jersey-client-2.31.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jersey-common-2.31.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jersey-container-servlet-2.3 1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jersey-container-servlet-core-2.31.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jersey-hk2-2.31.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jersey-media-jaxb-2.31.jar:/home/java/kafka_2. 13-2.7.0/bin/../libs/jersey-server-2.31.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-client-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-continuation-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/. ./libs/jetty-http-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-io-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-security-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty- server-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-servlet-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-servlets-9.4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jetty-util-9 .4.33.v20201020.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/jopt-simple-5.0.4.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka_2.13-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka_2.13-2.7.0-sources.jar:/home/java/kafka_2. 13-2.7.0/bin/../libs/kafka-clients-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-log4j-appender-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-raft-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-streams- 2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-streams-examples-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-streams-scala_2.13-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-streams-test-utils-2.7.0.ja r:/home/java/kafka_2.13-2.7.0/bin/../libs/kafka-tools-2.7.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/log4j-1.2.17.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/lz4-java-1.7.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/maven-artif act-3.6.3.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/metrics-core-2.2.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/netty-buffer-4.1.51.Final.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/netty-codec-4.1.51.Final.jar:/home/java/kafk a_2.13-2.7.0/bin/../libs/netty-common-4.1.51.Final.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/netty-handler-4.1.51.Final.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/netty-resolver-4.1.51.Final.jar:/home/java/kafka_2.13-2.7.0/bin/. ./libs/netty-transport-4.1.51.Final.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/netty-transport-native-epoll-4.1.51.Final.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/netty-transport-native-unix-common-4.1.51.Final.jar:/home/java/ka fka_2.13-2.7.0/bin/../libs/osgi-resource-locator-1.0.3.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/paranamer-2.8.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/plexus-utils-3.2.1.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/reflections -0.9.12.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/rocksdbjni-5.18.4.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/scala-collection-compat_2.13-2.2.0.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/scala-java8-compat_2.13-0.9.1.jar:/hom e/java/kafka_2.13-2.7.0/bin/../libs/scala-library-2.13.3.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/scala-logging_2.13-3.9.2.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/scala-reflect-2.13.3.jar:/home/java/kafka_2.13-2.7.0/bin/../l ibs/slf4j-api-1.7.30.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/slf4j-log4j12-1.7.30.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/snappy-java-1.1.7.7.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/zookeeper-3.5.8.jar:/home/java/kafka_ 2.13-2.7.0/bin/../libs/zookeeper-jute-3.5.8.jar:/home/java/kafka_2.13-2.7.0/bin/../libs/zstd-jni-1.4.5-6.jar", "kafka.tools.ConsoleConsumer", "--topic", "quickstart-events", "--from-beginning", "--bootstrap-server", "192.168.79.140 :9092"], [/ 28 vars /]) = 0 <0.000362>

epoll_ctl(113, EPOLL_CTL_MOD, 116, {EPOLLIN, {u32=116, u64=116}}) = 0 <0.000013>
epoll_wait(113, [{EPOLLIN, {u32=116, u64=116}}], 4096, 2884) = 1 <0.000013>

116 是 socket文件描述符,read是libc.so标准库里提供的函数。
后面的数据是kafka自行封装的数据格式,可以看一些字符串
read(116, "\0\0\0\n\0\0\0\0\0\0\0]^e\v\2\22quickstart-events\2\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0\0\0\377\377\377\377\351\4\0\0\0\0\0\0\0\0\0\0\0D\0\0\0\0\2\241jK\357\0\0\0\0\0\0\0\ 0\1x\271\350\f)\0\0\1x\271\350\f)\377\377\377\377\377\377\377\377\377\377\377\377\377\377\0\0\0\1$\0\0\0\1\30i like money\0\0\0\0\0\0\0\0\1\0\0\0D\0\0\0\0\2\263\272f<\0\0\0\0\0\0\0\0\1x\271\350&4\0\0\1x\271\350&4\377\377\377\377\37 7\377\377\377\377\377\377\377\377\377\0\0\0\1$\0\0\0\1\30i love money\0\0\0\0\0\0\0\0\2\0\0\0>\0\0\0\0\2\342hn\266\0\0\0\0\0\0\0\0\1x\271\360\213f\0\0\1x\271\360\213f\377\377\377\377\377\377\377\377\377\377\377\377\377\377\0\0\0\1\ 30\0\0\0\1\ftest b\0\0\0\0\0\0\0\0\3\0\0\0>\0\0\0\0\2\265\362\347\377\0\0\0\0\0\0\0\0\1x\271\361\310\3\0\0\1x\271\361\310\3\377\377\377\377\377\377\377\377\377\377\377\377\377\377\0\0\0\1\30\0\0\0\1\ftest c\0\0\0\0\0\0\0\0\4\0\0\0? \0\0\0\0\2*\301{K\0\0\0\0\0\0\0\0\1x\271\362\6;\0\0\1x\271\362\6;\377\377\377\377\377\377\377\377\377\377\377\377\377\377\0\0\0\1\32\0\0\0\1\16test bb\0\0\0\0\0\0\0\0\5\0\0\0A\0\0\0\0\2<\257z\333\0\0\0\0\0\0\0\0\1x\271\362o\313\0\0 \1x\271\362o\313\377\377\377\377\377\377\377\377\377\377\377\377\377\377\0\0\0\1\36\0\0\0\1\22test nice\0\0\0\0\0\0\0\0\6\0\0\0B\0\0\0\0\2}0\223\342\0\0\0\0\0\0\0\0\1x\271\362\376\343\0\0\1x\271\362\376\343\377\377\377\377\377\377\ 377\377\377\377\377\377\377\377\0\0\0\1 \0\0\0\1\24test china\0\0\0\0\0\0\0\0\7\0\0\0B\0\0\0\0\2\312 u\254\0\0\0\0\0\0\0\0\1x\271\363f[\0\0\1x\271\363f[\377\377\377\377\377\377\377\377\377\377\377\377\377\377\0\0\0\1 \0\0\0\1\24tes t china\0\0\0\0", 691) = 691 <0.000015>

向终端输出接收的数据
write(1, "test china", 10) = 10 <0.000015>

图片.png

总结:
kafka整个程序运行时,调用的系统函数过多,每次调用都会耗费一定的调用时间,整个程序的性能并没有什么提升,如果用在项目中,硬件配置建议整大点,毕竟这玩意吃资源。跟个航母一样。耗油。

整个进程最终调用的系统函数跟其它编程语言是一样的,没有什么区别,用到的知识依然是TCP/IP,而TCP/IP的实现就是SOCKET API ,刚才我们已经看到epoll,write,read,sendmsg等这些函数。
大家经过调试,使用后,可以自己写一个简单的示例都能做出一个发布订阅功能的程序了。毕竟基础知识就这些。【当然了kafka实现时所使用的相关数据结构,算法不谈,你自己实现简单的程序时使用队列数据结构也就能实现】

最后,如果你对其它rust/python/go/c/php调用时是否使用了相同的标准函数,可以点击https://www.bilibili.com/vide...

欢迎加群讨论
图片.png

阅读 203

北风之神核潜战略专家

219 声望
26 粉丝
0 条评论
你知道吗?

北风之神核潜战略专家

219 声望
26 粉丝
宣传栏