开源signoz实现可观测性的分析

头像
a朋
     上海
    阅读 16 分钟
    1

    Signoz是一个开源的APM(Application Performance Management)，它是应用可观测性的一个实践，使用OpenTelemetry协议，将traces/metrics/log融合在一起。

    SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

    本文重点关注traces和metrics方面的实现。

    一. 整体架构

    • app使用opentelmetry-sdk编写代码(支持java/golang/python等)；
    • app配置将metrics和tracing发送至otel-collector；

    • otel-collector定制实现了:

      • clickhouse-metrics-exporter: 将metrics发送至clickhouse；
      • clickhouse-trace-exporter：将trace发送至clickhouse:
    • query-server负责从clickhouse中查询metrics和trace信息，并提供前端API；

    image.png

    二. otel-collector

    1. app将metrics和trace发送至otel-collector

    app中引入opentelemetry:

    go.opentelemetry.io/otel

    若使用gin作为http框架：

    import "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

r := gin.Default()
r.Use(otelgin.Middleware(serviceName))

    app在运行时，指定ServiceName和collector地址：

    # SERVICE_NAME=goApp INSECURE_MODE=true OTEL_EXPORTER_OTLP_ENDPOINT=192.168.0.1:4317 go run main.go

    2. otel-collector的配置

    otel-collector的配置，分为receivers、processors、exporters，
    然后由service通过pipeline组织成完整的功能。

    receivers:
  opencensus:
    endpoint: 0.0.0.0:55678
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268

processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s

exporters:
  clickhousetraces:
    datasource: tcp://clickhouse:9000/?database=signoz_traces
  clickhousemetricswrite:
    endpoint: tcp://clickhouse:9000/?database=signoz_metrics
    resource_to_telemetry_conversion:
      enabled: true

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  extensions:
    - health_check
    - zpages
    - pprof
  pipelines:
    traces:
      receivers: [jaeger, otlp]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhousetraces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousemetricswrite]

    3. otel-collector实现clickhouse的exporter

    otel-collector将metrics和traces保存至clickhouse，为此，signoz在otel-collector中，实现了：

    • clickhousemetricsexporter;
    • clickhousetracesexporter;

    存储在clickhouse中的metrics信息：

    ## time_series信息
73048f54a32c :) select * from time_series_v2 limit 3;
SELECT *
FROM time_series_v2
LIMIT 3
Query id: 2540c93a-a396-42a7-b4e5-bed772ac00c5
┌─metric_name─────────────────────────────────┬──────────fingerprint─┬──timestamp_ms─┬─labels────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─labels_object──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ otelcol_exporter_enqueue_failed_log_records │   136087390202145591 │ 1659598169051 │ {"__name__":"otelcol_exporter_enqueue_failed_log_records","exporter":"clickhousemetricswrite","instance":"localhost:8888","job":"otel-collector-metrics","service_instance_id":"b3d22c64-5c09-44d7-9367-9716747842db","service_version":"latest"} │ ('otelcol_exporter_enqueue_failed_log_records','','','','','','','clickhousemetricswrite','','','localhost:8888','otel-collector-metrics','','','','','','','','','','b3d22c64-5c09-44d7-9367-9716747842db','','','latest','','','','','') │
│ otelcol_exporter_enqueue_failed_log_records │  7551020479515194203 │ 1659598139047 │ {"__name__":"otelcol_exporter_enqueue_failed_log_records","exporter":"prometheus","instance":"otel-collector:8888","job":"otel-collector","service_instance_id":"ed7abeec-d18f-4c4d-8bcb-64a9dea37976","service_version":"latest"}                │ ('otelcol_exporter_enqueue_failed_log_records','','','','','','','prometheus','','','otel-collector:8888','otel-collector','','','','','','','','','','ed7abeec-d18f-4c4d-8bcb-64a9dea37976','','','latest','','','','','')                │
│ otelcol_exporter_enqueue_failed_log_records │ 11388370004215594241 │ 1659598139047 │ {"__name__":"otelcol_exporter_enqueue_failed_log_records","exporter":"clickhousetraces","instance":"otel-collector:8888","job":"otel-collector","service_instance_id":"ed7abeec-d18f-4c4d-8bcb-64a9dea37976","service_version":"latest"}          │ ('otelcol_exporter_enqueue_failed_log_records','','','','','','','clickhousetraces','','','otel-collector:8888','otel-collector','','','','','','','','','','ed7abeec-d18f-4c4d-8bcb-64a9dea37976','','','latest','','','','','')          │
└─────────────────────────────────────────────┴──────────────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
    ## samples信息
73048f54a32c :) select * from samples_v2 limit 3
SELECT *
FROM samples_v2
LIMIT 3
Query id: 40dfc9f5-e76e-4ecc-8b50-c791fa9e041c
┌─metric_name─────────────────────────────────┬────────fingerprint─┬──timestamp_ms─┬─value─┐
│ otelcol_exporter_enqueue_failed_log_records │ 136087390202145591 │ 1659598160222 │     0 │
│ otelcol_exporter_enqueue_failed_log_records │ 136087390202145591 │ 1659598220222 │     0 │
│ otelcol_exporter_enqueue_failed_log_records │ 136087390202145591 │ 1659598280222 │     0 │
└─────────────────────────────────────────────┴────────────────────┴───────────────┴───────┘
3 rows in set. Elapsed: 0.016 sec.

    存储在clickhouse中的traces信息：

    73048f54a32c :) select  * from signoz_spans limit 3;
SELECT *
FROM signoz_spans
LIMIT 3
Query id: d9d84316-c6f0-4089-87a1-187132a295b1
Connecting to database signoz_traces at localhost:9000 as user default.
Connected to ClickHouse server version 22.4.5 revision 54455.
┌─────────────────────timestamp─┬─traceID──────────────────────────┬─model──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 2022-08-04 14:42:21.920273000 │ 00000000000000000000976d1a8457bf │ {"traceId":"00000000000000000000976d1a8457bf","spanId":"0c8946ee620b3b72","name":"HTTP GET /route","durationNano":35578000,"startTimeUnixNano":1659624141920273000,"serviceName":"route","kind":2,"references":[{"traceId":"00000000000000000000976d1a8457bf","spanId":"75509681018b9e65","refType":"CHILD_OF"}],"tagMap":{"client-uuid":"1f460e28dcd19136","component":"net/http","host.name":"93a9af66f59d","http.method":"GET","http.status_code":"200","http.url":"/route?dropoff=728%2C326\u0026pickup=232%2C495","ip":"172.18.0.3","opencensus.exporterversion":"Jaeger-Go-2.30.0","service.name":"route"},"event":["{\"timeUnixNano\":1659624141920308000,\"attributeMap\":{\"event\":\"HTTP request received\",\"level\":\"info\",\"method\":\"GET\",\"url\":\"/route?dropoff=728%2C326\\u0026pickup=232%2C495\"}}"]} │
│ 2022-08-04 14:42:21.920759000 │ 00000000000000000000976d1a8457bf │ {"traceId":"00000000000000000000976d1a8457bf","spanId":"3a48059caa1e422a","name":"HTTP GET /route","durationNano":50933000,"startTimeUnixNano":1659624141920759000,"serviceName":"route","kind":2,"references":[{"traceId":"00000000000000000000976d1a8457bf","spanId":"37835dd357b1d10c","refType":"CHILD_OF"}],"tagMap":{"client-uuid":"1f460e28dcd19136","component":"net/http","host.name":"93a9af66f59d","http.method":"GET","http.status_code":"200","http.url":"/route?dropoff=728%2C326\u0026pickup=745%2C522","ip":"172.18.0.3","opencensus.exporterversion":"Jaeger-Go-2.30.0","service.name":"route"},"event":["{\"timeUnixNano\":1659624141920796000,\"attributeMap\":{\"event\":\"HTTP request received\",\"level\":\"info\",\"method\":\"GET\",\"url\":\"/route?dropoff=728%2C326\\u0026pickup=745%2C522\"}}"]} │
│ 2022-08-04 14:42:21.920505000 │ 00000000000000000000976d1a8457bf │ {"traceId":"00000000000000000000976d1a8457bf","spanId":"2754f5b55262e27b","name":"HTTP GET /route","durationNano":71846000,"startTimeUnixNano":1659624141920505000,"serviceName":"route","kind":2,"references":[{"traceId":"00000000000000000000976d1a8457bf","spanId":"44b47488d0e49c98","refType":"CHILD_OF"}],"tagMap":{"client-uuid":"1f460e28dcd19136","component":"net/http","host.name":"93a9af66f59d","http.method":"GET","http.status_code":"200","http.url":"/route?dropoff=728%2C326\u0026pickup=462%2C723","ip":"172.18.0.3","opencensus.exporterversion":"Jaeger-Go-2.30.0","service.name":"route"},"event":["{\"timeUnixNano\":1659624141920525000,\"attributeMap\":{\"event\":\"HTTP request received\",\"level\":\"info\",\"method\":\"GET\",\"url\":\"/route?dropoff=728%2C326\\u0026pickup=462%2C723\"}}"]} │
└───────────────────────────────┴──────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

    三. Query Service

    query-service服务中读取clickhouse的metrics和traces，给前端提供了HTTP接口。

    另外，query-service还将prometheus引入作为sdk，实现了告警规则的创建及告警功能。

    1. 查询metrics

    API入口：/api/v1/query_range

    // app/http_handler.go
func (aH *APIHandler) RegisterRoutes(router *mux.Router) {
    router.HandleFunc("/api/v1/query_range", ViewAccess(aH.queryRangeMetrics)).Methods(http.MethodGet)
    ...
}

func (aH *APIHandler) queryRangeMetrics(w http.ResponseWriter, r *http.Request) {
    query, apiErrorObj := parseQueryRangeRequest(r)
    ...
    ctx := r.Context()
    ...
    res, qs, apiError := (*aH.reader).GetQueryRangeResult(ctx, query)
    ...
    response_data := &model.QueryData{
        ResultType: res.Value.Type(),
        Result:     res.Value,
        Stats:      qs,
    }
    aH.respond(w, response_data)
}

    为clickhouse实现了remote-read，故查询跟prometheus的查询代码差不多：

    • 构造查询对象；
    • 调用engine执行查询；
    // app/clickhouseReader/reader.go
func (r *ClickHouseReader) GetQueryRangeResult(ctx context.Context, query *model.QueryRangeParams) (*promql.Result, *stats.QueryStats, *model.ApiError) {
    qry, err := r.queryEngine.NewRangeQuery(r.remoteStorage, query.Query, query.Start, query.End, query.Step)
    ...
    res := qry.Exec(ctx)
    // Optional stats field in response if parameter "stats" is not empty.
    var qs *stats.QueryStats
    if query.Stats != "" {
        qs = stats.NewQueryStats(qry.Stats())
    }
    qry.Close()
    return res, qs, nil
}

    2. 查询traces

    查询的url: /api/v1/traces/00000000000000001348070b30cd6aec

    入口代码：/api/v1/traces

    // app/http_handler.go
func (aH *APIHandler) RegisterRoutes(router *mux.Router) {
    ...
    router.HandleFunc("/api/v1/traces/{traceId}", ViewAccess(aH.searchTraces)).Methods(http.MethodGet)
    ...
}
func (aH *APIHandler) searchTraces(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    traceId := vars["traceId"]
    result, err := (*aH.reader).SearchTraces(r.Context(), traceId)
    if aH.handleError(w, err, http.StatusBadRequest) {
        return
    }
    aH.writeJSON(w, r, result)
}

    执行查询，通过clickhouse的查询语法实现：

    // app/clickhouseReader/reader.go
func (r *ClickHouseReader) SearchTraces(ctx context.Context, traceId string) (*[]model.SearchSpansResult, error) {
    var searchScanReponses []model.SearchSpanDBReponseItem
    query := fmt.Sprintf("SELECT timestamp, traceID, model FROM %s.%s WHERE traceID=$1", r.traceDB, r.spansTable)
    err := r.db.Select(ctx, &searchScanReponses, query, traceId)
    ...
    searchSpansResult := []model.SearchSpansResult{
        {
            Columns: []string{"__time", "SpanId", "TraceId", "ServiceName", "Name", "Kind", "DurationNano", "TagsKeys", "TagsValues", "References", "Events", "HasError"},
            Events:  make([][]interface{}, len(searchScanReponses)),
        },
    }
    for i, item := range searchScanReponses {
        var jsonItem model.SearchSpanReponseItem
        json.Unmarshal([]byte(item.Model), &jsonItem)
        jsonItem.TimeUnixNano = uint64(item.Timestamp.UnixNano() / 1000000)
        spanEvents := jsonItem.GetValues()
        searchSpansResult[0].Events[i] = spanEvents
    }
    return &searchSpansResult, nil
}

    3. 创建告警规则

    页面上创建告警规则后，保存在sqlite中：

    sqlite> select * from rules;
1|2022-08-05 07:18:31.989405383+00:00|0|{"condition":{"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","name":"A","formulaOnly":false,"metricName":"up","tagFilters":{"op":"AND","items":[]},"groupBy":["job"],"aggregateOperator":1,"expression":"A","disabled":false,"toggleDisable":false,"toggleDelete":false}},"promQueries":{"A":{"query":"up ","stats":"","name":"A","legend":"","disabled":false}},"queryType":3},"op":"3","matchType":"1"},"labels":{"severity":"warning"},"annotations":{"description":"A new alert"},"evalWindow":"5m0s","alert":"service_down","source":"http://192.168.0.1:3301/alerts/new","ruleType":"promql_rule"}
sqlite>

    创建告警规则的入口：

    // app/http_handler.go
func (aH *APIHandler) RegisterRoutes(router *mux.Router) {
    ...
    router.HandleFunc("/api/v1/rules", EditAccess(aH.createRule)).Methods(http.MethodPost)
    ...
}
func (aH *APIHandler) createRule(w http.ResponseWriter, r *http.Request) {
    decoder := json.NewDecoder(r.Body)
    var postData map[string]string
    err := decoder.Decode(&postData)
    ...
    apiErrorObj := (*aH.reader).CreateRule(postData["data"])
    ...
    aH.respond(w, "rule successfully added")
}

    创建规则的具体实现：

    • 首先，向db插入一条rules记录；
    • 然后，给prometheus增加一条alertRule；
    • 最后，事务性的提交；
    // app/clickhouseReader/reader.go
func (r *ClickHouseReader) CreateRule(rule string) *model.ApiError {
    tx, err := r.localDB.Begin()
    ...
    var lastInsertId int64
    {
        // 向db插入一条记录
        stmt, err := tx.Prepare(`INSERT into rules (updated_at, data) VALUES($1,$2);`)
        ...
        defer stmt.Close()
        result, err := stmt.Exec(time.Now(), rule)
        ...
        lastInsertId, _ = result.LastInsertId()
        groupName := fmt.Sprintf("%d-groupname", lastInsertId)
        // 让prometheus新增一条AlertRule
        err = r.ruleManager.AddGroup(time.Duration(r.promConfig.GlobalConfig.EvaluationInterval), rule, groupName)
        if err != nil {
            tx.Rollback()
            return &model.ApiError{Typ: model.ErrorInternal, Err: err}
        }
    }
    err = tx.Commit()
    if err != nil {
        zap.S().Errorf("Error in committing transaction for INSERT to rules\n", err)
        return &model.ApiError{Typ: model.ErrorInternal, Err: err}
    }
    return nil
}

    signoz还部署了alertmanager实例，当query-service中的prometheus有告警触发时，将告警发送到alertmanager。

    signoz前端查询告警时，通过查询alertmanager实现：

    HTTP GET /api/alertmanager/alerts?active=true&inhibited=true&silenced=false

    四. 特点

    • 统一存储和读取metrics和traces，均存储在clickhouse；

      • 解决传统上metrics存prometheus，traces存Jager的痛点；

    • 集成了prometheus的告警规则功能:

      • 通过prometheus sdk引入，并未引入prometheus进程；
    • prometheus告警的去重、聚合、发送，通过alertmanager实例实现；

    参考：

    1.https://www.cnblogs.com/rongf...
    2.signoz docs: https://signoz.io/docs/archit...
    3.opentelmetry docs: https://opentelemetry.io/docs...

    prometheus监控apm
    a朋
    63 声望38 粉丝
    « 上一篇
    prometheus与exemplar
    下一篇 »
    prometheus-operator的Probe使用和原理分析

    引用和评论

    推荐阅读
    头像
    alertmanager源码：整体架构和流程分析

    a朋阅读 1.6k

    头像
    日志与追踪的完美融合：OpenTelemetry MDC 实践指南

    crossoverJie赞 3阅读 1.7k

    头像
    浅谈时序数据库内核：如何用单机扛住亿级数据写入

    泊浮目阅读 5k

    头像
    OpenTelemetry 实战：gRPC 监控的实现原理

    crossoverJie赞 1阅读 1.4k

    头像
    当ABAP遇见普罗米修斯

    注销阅读 1.9k

    头像
    OpenTelemetry 实战：从零实现应用指标监控

    crossoverJie阅读 1.1k

    头像
    免费网站监控工具推荐

    外滩运维专家阅读 366

    0 条评论
    得票最新