Note, the original text is from https://blog.mygraphql.com/zh/posts/low-tec/trace/trace-istio/trace-istio-part4/ . If the reproduced picture you see is unclear, please return to the original text.
- Why to understand the HTTP reverse proxy process under upstream/downstream event-driven collaboration
- Reverse Engineering and Cloud Native Field Analysis Series Introduction
- BPF script output
- BPF script
- end
Why to understand the HTTP reverse proxy process under upstream/downstream event-driven collaboration
Unexpectedly, I really insisted on writing Part4. Believe it or not, the "why" section of each part is the hardest to write in my opinion :). If you're reading this series for the first time, don't worry, each section is relatively self-contained.
Unsurprisingly, the Envoy introduction you saw earlier described the features of Envoy like this:
- Written in C++, native bottom layer, no GC stop the world, so excellent performance
- Asynchronous event-driven, multiplexing, perfect solution to C10k problem
- Since a single thread is responsible for multiple connections, when the number of connections is large, the memory overhead of a large number of threads and the CPU context switching overhead are reduced.
These descriptions, of course, have their rationale. But many things that look beautiful from a distance may have many interesting and valuable things after macro zooming in. I believe that as long as the analysis is detailed enough, it is always possible to make some meaningful optimizations for our actual operating environment and traffic characteristics. It may be just to modify an Envoy / Kernel configuration, or it may be to modify a line of Envoy code. Or the process-related behavior of your application, such as the size of the buffer packet per write to the socket.
And these need to be built on the basis of understanding the implementation details. Unless you feel lucky or experience value is particularly good, you can guess.
Reverse Engineering and Cloud Native Field Analysis Series Introduction
A preview before we start, the reverse engineering and cloud native on-site analysis series (will) include:
- Part 1: eBPF Tracking Istio/Envoy Steps
- Part 2: eBPF tracking Istio/Envoy startup, monitoring and thread load balancing
- Part 3: eBPF tracks Istio/Envoy's downstream TCP connection accept, TLS handshake and filter_chain selection
- Part 4: eBPF tracks the HTTP reverse proxy process under the upstream/downstream event-driven collaboration of Istio/Envoy (this article)
- Part 5: L3/4 Network Fitler Interaction Mechanism of eBPF Tracking Istio/Envoy
- Part 6: eBPF tracking http filter of Istio/Envoy
- Part 7: eBPF tracks Istio/Envoy's http router
- Part 8: eBPF tracks Istio/Envoy cluster, connection pool and outbound load balance
In the series, I'll demonstrate how to get bpftrace to "read" object data in an envoy process written in C++11 at runtime.
In order not to scare people away, it is still an old routine, with more pictures and less code, but some pictures are a bit complicated. The programmer uncle started telling stories. 🚜
The overall process of HTTP reverse proxy
Overall, the general process of Socket event-driven HTTP reverse proxy is as follows:
As can be seen in the figure, there are 4 kinds of events that drive the entire process. The following sections will analyze them one by one.
In order to avoid getting lost in the details of each step at once, let's take a look at the general flow of all steps:
Figure: General diagram of Istio/Envoy module collaboration
Downstream Read Request module collaboration
Figure: Downstream Read-Ready Module Collaboration
Briefly describe the process:
- upstream socket readable callback
- Http::ConnectionManagerImpl reads the socket and increments it into Http1::ConnectionImpl
- Http1::ConnectionImpl calls nghttp2 to interpret HTTP requests incrementally
- If nghttp2 thinks that the HTTP Request request has been completely read, it calls
Http::ServerConnection::onMessageCompleteBase()
-
Http::ServerConnection::onMessageCompleteBase()
First stop downstream ReadReady listening -
Http::ServerConnection::onMessageCompleteBase()
callHttp::FilterManager
, initiatehttp filter chain
's decodeHeaders iterative process - In general, the last http filter of
http filter chain
f07acdd9cd64cbf60fe6e23d76704446--- isRouter::Filter
,Router::Filter::decodeHeaders()
is called - The logic of
Router::Filter::decodeHeaders()
is shown in the figure below.
Figure: Downstream Request Router module collaboration
Briefly describe the process:
-
Router::Filter
,Router::Filter::decodeHeaders()
are called - According to the configured Router rules, it is matched to the Cluster
- If the Cluster connection pool object does not exist, create a new one
- New
Envoy::Router::UpstreamRequest
object. - Call
Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream)
, encode HTTP header - After a series of load balancing algorithms, it matches the upstream host (endpoint)
Insufficient connections to the selected upstream host are found, then:
- Open a new socket fd (not connected)
- Register for WriteReady / Connected events of upstream socket FD . Prepare to write upstream request during event callback
- Use socket fd to initiate an asynchronous connection request to the upstream host
- Associate downstream and upstream fd
Upstream Write Request module collaboration
Briefly describe the process:
- upstream socket write ready callback
- It is found that the connection is successful, and the upstream socket is associated to
ConnectionPool::ActiveClient
- upstream socket write ready callback
- It is found that the connection is writable, and the upstream HTTP request is written
Upstream Read Response module collaboration
Downstream Write Response module collaboration
BPF script output
The above pictures are not derived from just reading the source code. It is necessary to observe the buried point through bpf:
- Record downstream FD (file descriptor of socket), which can be considered as the socket id in the process
- Bury the entry and exit of related functions at the process level and kernel level, and observe their entry and exit to participate in the stack
- Associate downstream FD and upstream FD
Of course, there are many details, but I don't plan to talk about them one by one. It is really necessary. Students who want to know more can contact me to discuss. The bpf code in the next section is the best detail.
#### Downstream Read Request: accept downstream socket, 可见 downstream socket 的 FD=41
OS handshaked TCP:
01:22:11 4215 wrk:worker_0 172.30.207.129 58708 172.21.206.232 15006 0/4096
sys_exit_accept4 fd=41
accept4+96
Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_
***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_
***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE
***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE
***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE
***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv
***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89
recv+108
***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm
#### Downstream Read Request: 选择 network filter chain
***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Ev
comm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12
Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
#### Downstream Read Request: 监听 upstream FD 事件
***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+107
Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&)+70
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+307
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0 : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4.
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+0
std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+62
std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
std::__1::__function::__func<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7, std::__1::allocator<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353
Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100
***************************
#### Downstream Read Request: 读 HTTP Request
***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET
***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
#### Downstream Read Request: downstream 的 HTTP Reuest 已经完全读取(onMessageComplete())。停止监听 downstream 的 ReadReady 事件
***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Network::ConnectionImpl::readDisable(bool)+1077
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
#### Downstream Read Request: 监听新建立的 upstream socket 的 FD=44 , 这里 socket 还处于未连接状态。现在监听 upstream socket 的 所有事件。以备后面异步连接成功时收到回调
***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+283
Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+75
Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>)+561
Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+126
non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+21
Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112
Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41
Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&)+357
Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0 : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4.
#### Downstream Read Request: 发现异步 upstream 连接
connecting to 8080:
01:23:36 4215 wrk:worker_0 127.0.0.6 36383 172.21.206.232 8080
connect+75
Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+104
Envoy::Network::SocketImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+64
Envoy::Network::ClientConnectionImpl::connect()+650
Envoy::Http::CodecClient::connect()+543
Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。