beyla源码：golang与非golang程序的探测

beyla中golang程序与非golang程序的ebpf采用了不同的探测方式。

• golang：使用uprobe监听用户库函数；
• 非golang：使用kprobe监听内核函数；

程序类型的定义：

// beyla/pkg/internal/svc/svc.go

type InstrumentableType int

const (
   InstrumentableGolang = InstrumentableType(iota)
   InstrumentableJava
   InstrumentableDotnet
   InstrumentablePython
   InstrumentableRuby
   InstrumentableNodejs
   InstrumentableRust
   InstrumentableGeneric
)

一.程序的区分方法

对golang与非golang程序，其区分方法是读elf可执行文件，然后查找symbols是否包含golang的function，以go nethttp为例:

• 读elf文件的symbols，然后查找其中是否有go http的function；

• http的function：

  • "net/http.serverHandler.ServeHTTP"
  • "net/http.(*conn).readRequest"
  • "net/http.(*response).WriteHeader"
  • "net/http.(*Transport).roundTrip"

源码，重点是inspectOffset(execElf)函数：

• 读取elf文件，解析其中的symbols；
• 如果探测到golang的用户函数，则认为是Golang程序；

// beyla/pkg/internal/discover/typer.go

func (t *typer) asInstrumentable(execElf *exec.FileInfo) Instrumentable {
    ...
    // look for suitable Go application first
    offsets, ok := t.inspectOffsets(execElf)
    if ok {
        // we found go offsets, let's see if this application is not a proxy
        if !isGoProxy(offsets) {
            return Instrumentable{Type: svc.InstrumentableGolang, FileInfo: execElf, Offsets: offsets}
        }
    } 
    ...
    detectedType := exec.FindProcLanguage(execElf.Pid, execElf.ELF)
    return Instrumentable{Type: detectedType, FileInfo: execElf, ChildPids: child}
}

按照注册监听时的function列表，在elf中查找symbols：

• findGoSymbolTable(elfF)负责从elf中提取symbols；

// beyla/pkg/internal/goexec/instructions.go

func instrumentationPoints(elfF *elf.File, funcNames []string) (map[string]FuncOffsets, error) {
    ...
    symTab, err := findGoSymbolTable(elfF)
    for _, f := range symTab.Funcs {
        ...
        if _, ok := functions[fName]; ok {
            offs, ok, err := findFuncOffset(&f, elfF)
            if ok {
                allOffsets[fName] = offs
            }
        }
    }
    return allOffsets, nil
}

重点看一下findGoSymbolTable()函数的实现：

• 首先，读取elf文件中section=.gopclntab的内容；
• 然后，读取elf文件中section=.text的内容；
• 最后，使用上面读取的内容，构造symTab；

• golang的这种elf结构，保证了：

  • 即使elf被stripped，也能通过 debug/gosym 库，将其中的symbols读取出来；
  • debug/gosym 是Go标准库中的一个包，用于解析Go程序的符号表信息；

// beyla/pkg/internal/goexec/instructions.go

func findGoSymbolTable(elfF *elf.File) (*gosym.Table, error) {
    var err error
    var pclndat []byte
    // program counter line table
    if sec := elfF.Section(".gopclntab"); sec != nil {
        if pclndat, err = sec.Data(); err != nil {
            return nil, fmt.Errorf("acquiring .gopclntab data: %w", err)
        }
    }
    txtSection := elfF.Section(".text")
    pcln := gosym.NewLineTable(pclndat, txtSection.Addr)
    symTab, err := gosym.NewTable(nil, pcln)
    ...
    return symTab, nil
}

看下elf中包含的sections:

• 可以看出，其中包gopclntab和text这两个section;

# objdump -h example-http

example-http     文件格式 elf64-x86-64

节：
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         0020e3e6  0000000000401000  0000000000401000  00001000  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .plt          00000260  000000000060f400  000000000060f400  0020f400  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  2 .rodata       000e2060  0000000000610000  0000000000610000  00210000  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .dynsym       000003f0  00000000006f2940  00000000006f2940  002f2940  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
...
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 12 .gosymtab     00000000  00000000006f4a90  00000000006f4a90  002f4a90  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 13 .gopclntab    001440b0  00000000006f4aa0  00000000006f4aa0  002f4aa0  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 14 .go.buildinfo 00000140  0000000000839000  0000000000839000  00439000  2**4
                  CONTENTS, ALLOC, LOAD, DATA
...
 24 .note.go.buildid 00000064  0000000000400f80  0000000000400f80  00000f80  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA

二.golang与非golang程序的监听探针

golang与非golang程序使用了不同的ebpf监听方法。

• golang程序：使用newGoTracerGroup()注册tracer的方法；
• 非golang程序：使用newNonGoTracersGroup()注册tracer的方法；

// beyla/pkg/internal/discover/attacher.go

func (ta *TraceAttacher) getTracer(ie *Instrumentable) (*ebpf.ProcessTracer, bool) {
    ...
    var programs []ebpf.Tracer
    switch ie.Type {
    case svc.InstrumentableGolang:
        ...
        tracerType = ebpf.Go
        programs = filterNotFoundPrograms(newGoTracersGroup(ta.Cfg, ta.Metrics), ie.Offsets)
    case svc.InstrumentableJava, svc.InstrumentableNodejs, svc.InstrumentableRuby, svc.InstrumentablePython, svc.InstrumentableDotnet, svc.InstrumentableGeneric, svc.InstrumentableRust:
        ...
        programs = newNonGoTracersGroup(ta.Cfg, ta.Metrics)
    }
    ...
}

1.golang程序的tracer:

// beyla/pkg/internal/discover/finder.go

func newGoTracersGroup(cfg *pipe.Config, metrics imetrics.Reporter) []ebpf.Tracer {
    // Each program is an eBPF source: net/http, grpc...
    return []ebpf.Tracer{
        nethttp.New(&cfg.EBPF, metrics),
        &nethttp.GinTracer{Tracer: *nethttp.New(&cfg.EBPF, metrics)},
        grpc.New(&cfg.EBPF, metrics),
        goruntime.New(&cfg.EBPF, metrics),
        gosql.New(&cfg.EBPF, metrics),
    }
}

以nethttp为例，ebpf监听的用户库函数：

// beyla/pkg/internal/ebpf/nethttp/nethttp.go

func (p *Tracer) GoProbes() map[string]ebpfcommon.FunctionPrograms {
   return map[string]ebpfcommon.FunctionPrograms{
      "net/http.serverHandler.ServeHTTP": {
         Start: p.bpfObjects.UprobeServeHTTP,
      },
      "net/http.(*conn).readRequest": {
         End: p.bpfObjects.UprobeReadRequestReturns,
      },
      "net/http.(*response).WriteHeader": {
         Start: p.bpfObjects.UprobeWriteHeader,
      },
      "net/http.(*Transport).roundTrip": { // HTTP client, works with Client.Do as well as using the RoundTripper directly
         Start: p.bpfObjects.UprobeRoundTrip,
         End:   p.bpfObjects.UprobeRoundTripReturn,
      },
   }
}

2.非golang程序的tracer:

// beyla/pkg/internal/discover/finder.go

func newNonGoTracersGroup(cfg *pipe.Config, metrics imetrics.Reporter) []ebpf.Tracer {
    return []ebpf.Tracer{httpfltr.New(cfg, metrics), httpssl.New(cfg, metrics)}
}

进入到httpfltr查看监听的kprobe函数：

// beyla/pkg/internal/ebpf/httpfltr/httpfltr.go

func (p *Tracer) KProbes() map[string]ebpfcommon.FunctionPrograms {
   return map[string]ebpfcommon.FunctionPrograms{
      // Both sys accept probes use the same kretprobe.
      // We could tap into __sys_accept4, but we might be more prone to
      // issues with the internal kernel code changing.
      "sys_accept": {
         Required: true,
         End:      p.bpfObjects.KretprobeSysAccept4,
      },
      "sys_accept4": {
         Required: true,
         End:      p.bpfObjects.KretprobeSysAccept4,
      },
      "sock_alloc": {
         Required: true,
         End:      p.bpfObjects.KretprobeSockAlloc,
      },
      "tcp_rcv_established": {
         Required: true,
         Start:    p.bpfObjects.KprobeTcpRcvEstablished,
      },
      // Tracking of HTTP client calls, by tapping into connect
      "sys_connect": {
         Required: true,
         End:      p.bpfObjects.KretprobeSysConnect,
      },
      "tcp_connect": {
         Required: true,
         Start:    p.bpfObjects.KprobeTcpConnect,
      },
      "tcp_sendmsg": {
         Required: true,
         Start:    p.bpfObjects.KprobeTcpSendmsg,
      },
      // Reading more than 160 bytes
      "tcp_recvmsg": {
         Required: true,
         Start:    p.bpfObjects.KprobeTcpRecvmsg,
         End:      p.bpfObjects.KretprobeTcpRecvmsg,
      },
   }
}

参考：

1.https://www.cnxct.com/why-golang-elf-binary-file-is-large-tha...