Cloud Native Enthusiasts Weekly: Uncovering Timestamp Jitter in Prometheus

Open source project recommendation

O11y toolkit

The O11y toolkit is a toolset to maintain, debug, and enhance your observability systems, improving our daily experience with metrics, logs, and link tracing. For example, oy-scrape-jitter is used to solve the problem of inconsistent timestamp intervals for metrics captured by Prometheus. Prometheus uses delta-of-delta encoding and timestamp XOR compression technology, which means that when the time interval of index capture is exactly the same, a timestamp only needs one bit to be stored in TSDB, which can greatly Save disk space. Although theoretically, the interval between each time Promtheus fetches indicators should be the same, in practice, there will be jitter, and the fetch interval is not completely consistent, so disk space will be wasted. oy-scrape-jitter can find metrics that jitter between scrapes, for example:

 $ ./oy-scrape-jitter --prometheus.url=https://prometheus.demo.do.prometheus.io/ --plot.file=jitter.png
level=info metric="up{env=\"demo\", instance=\"demo.do.prometheus.io:9093\", job=\"alertmanager\"}" aligned=239 unaligned=0 max_ms=0 pc=100.00%
level=info metric="up{env=\"demo\", instance=\"demo.do.prometheus.io:9100\", job=\"node\"}" aligned=233 unaligned=6 max_ms=23 pc=97.49%
level=info metric="up{instance=\"demo.do.prometheus.io:3000\", job=\"grafana\"}" aligned=237 unaligned=2 max_ms=14 pc=99.16%
level=info metric="up{instance=\"demo.do.prometheus.io:8996\", job=\"random\"}" aligned=179 unaligned=60 max_ms=2 pc=74.90%
level=info metric="up{instance=\"demo.do.prometheus.io:8997\", job=\"random\"}" aligned=239 unaligned=0 max_ms=0 pc=100.00%
level=info metric="up{instance=\"demo.do.prometheus.io:8998\", job=\"random\"}" aligned=237 unaligned=2 max_ms=3 pc=99.16%
level=info metric="up{instance=\"demo.do.prometheus.io:8999\", job=\"random\"}" aligned=237 unaligned=2 max_ms=4 pc=99.16%
level=info metric="up{instance=\"demo.do.prometheus.io:9090\", job=\"prometheus\"}" aligned=237 unaligned=2 max_ms=8 pc=99.16%
level=info metric="up{instance=\"http://localhost:9100\", job=\"blackbox\"}" aligned=233 unaligned=6 max_ms=2 pc=97.49%
level=info metric="up{instance=\"localhost:2019\", job=\"caddy\"}" aligned=239 unaligned=0 max_ms=0 pc=100.00%
level=info aligned_targets=3 unaligned_targets=7 max_ms=23

One of the output results is this:

 level=info metric="up{instance=\"demo.do.prometheus.io:8996\", job=\"random\"}" aligned=179 unaligned=60 max_ms=2 pc=74.90%

This means that the metric {instance="demo.do.prometheus.io:8996", job="random"} has more than 25% of samples with inconsistent fetch intervals.

Korb

Korb can be used to migrate PVCs to different StorageClasses in a Kubernetes cluster. E.g:

 $ ./korb --new-pvc-storage-class ontap-ssd redis-data-redis-master-0
DEBU[0000] Created client from kubeconfig                component=migrator kubeconfig=/home/jens/.kube/config
DEBU[0000] Got current namespace                         component=migrator namespace=prod-beryju-org
DEBU[0000] Got Source PVC                                component=migrator name=redis-data-redis-master-0 uid=e4b5476f-b965-4e81-bfee-d7cbbf4f6317
DEBU[0000] No new Name given, using old name             component=migrator
DEBU[0000] Compatible Strategies:                        component=migrator
DEBU[0000] Copy the PVC to the new Storage class and with new size and a new name, delete the old PVC, and copy it back to the old name.  component=migrator
DEBU[0000] Only one compatible strategy, running         component=migrator
DEBU[0000] Set timeout from PVC size                     component=strategy strategy=copy-twice-name timeout=8m0s
WARN[0000] This strategy assumes you've stopped all pods accessing this data.  component=strategy strategy=copy-twice-name
DEBU[0000] creating temporary PVC                        component=strategy stage=1 strategy=copy-twice-name
DEBU[0002] starting mover job                            component=strategy stage=2 strategy=copy-twice-name
DEBU[0004] Pod not in correct state yet                  component=mover-job phase=Pending
DEBU[0006] Pod not in correct state yet                  component=mover-job phase=Pending
[...]
[mover logs]: sending incremental file list
[mover logs]: ./
[mover logs]: appendonly.aof
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/3)
[mover logs]: dump.rdb
            175 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=0/3)
DEBU[0022] Cleaning up successful job                    component=mover-job
DEBU[0022] deleting original PVC                         component=strategy stage=3 strategy=copy-twice-name
DEBU[0024] creating final destination PVC                component=strategy stage=4 strategy=copy-twice-name
DEBU[0024] starting mover job to final PVC               component=strategy stage=5 strategy=copy-twice-name
DEBU[0026] Pod not in correct state yet                  component=mover-job phase=Pending
DEBU[0028] Pod not in correct state yet                  component=mover-job phase=Pending
[...]
[mover logs]: sending incremental file list
[mover logs]: ./
[mover logs]: appendonly.aof
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/3)
[mover logs]: dump.rdb
            175 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=0/3)
DEBU[0048] Cleaning up successful job                    component=mover-job
DEBU[0048] deleting temporary PVC                        component=strategy stage=6 strategy=copy-twice-name
INFO[0050] And we're done                                component=strategy strategy=copy-twice-name
INFO[0050] Cleaning up...                                component=strategy strategy=copy-twice-name

websocat

websocat is similar to curl, but designed specifically for Websockets, making it easier to test Websockets. E.g:

 A$ websocat -s 1234
Listening on ws://127.0.0.1:1234/
ABC
123

B$ websocat ws://127.0.0.1:1234/
ABC
123

"Site Reliability Workbook" Chinese Version

This book is a sequel to the Google SRE workbook, which is practical. This is the Chinese translation of this book.

Article recommendation

Quickly get started with OpenFunction Node.js asynchronous function service development

Recently, "OpenFunction successfully passed the vote of the CNCF Technical Oversight Committee (TOC) of the Cloud Native Computing Foundation and officially entered the CNCF Sandbox hosting". As a part of the OpenFunction community, I am looking forward to more developers and partners participating in the project, building and developing the community together, and "making serverless functions and applications easier to run"! At the same time, as one of the current maintainers of the Node.js function framework (Function Framework), I would like to take this opportunity to share with you the recent development progress of the Node.js function framework, especially the asynchronous implementation in version 0.4.1 . function support.

This article will introduce the current research and development progress and future work prospects of the Node.js functional framework from the following aspects.

National Star Aerospace SaaS System Containerization Best Practices

Guoxing Aerospace is an Al satellite Internet technology company. It hopes to build a PaaS cloud computing basic platform based on K8s, manage and use resources by multi-tenant, integrate CI/CD, support flexible expansion and upgrade of clusters, build an enterprise-level one-stop DevOps architecture, and improve the cluster. Resource monitorability, traceable operation audit. This article describes how National Star Aerospace uses KubeSphere to achieve this goal.

A new generation of mutual authentication with Cilium Service Mesh

Mutual Authentication has always been the cornerstone of security. Protocols and technologies such as SSH, mTLS or IPsec that we use every day rely on mutual authentication, and the cloud-native world is no exception. We hope to use strong mutual authentication to ensure Kubernetes and Intercommunication between services throughout the cloud-native infrastructure.

This article examines how Cilium and Cilium Service Mesh leverage eBPF to provide a new identity-based two-way authentication for services, with a high-performance data plane that can support any network protocol without requiring application changes, No need to inject Sidecar either.

Cloud Native Dynamics

Storage Capacity Tracking Meets Common Criteria in Kubernetes 1.24

Kubernetes version 1.24 includes storage capacity tracking as a generally available feature.

Storage capacity tracking allows CSI drivers to publish information about remaining capacity. Then, when a Pod has a volume that still needs to be configured, the kube-scheduler uses that information to choose an appropriate node for the Pod.

Without this information, Pods can get stuck and cannot be scheduled to the appropriate node, as the kube-scheduler has to blindly choose and always ends up choosing a node that cannot configure volumes due to the underlying storage managed by the CSI driver The system does not have enough capacity.

Because the storage capacity information published by the CSI driver might not be up-to-date in the future, it can still happen that a node gets picked and ends up not functioning properly. The volume configuration recovers from it by notifying the scheduler that it needs to retry with a different node.

KubeKey 2.1.0 released to enhance K8s offline delivery experience

On May 6, 2022, KubeKey 2.1.0 was officially released, which is the 8th official version of KubeKey. This version enhances the offline deployment capability and delivery experience, and supports "one cloud with multiple cores", that is, the same K8s cluster can contain both ARM64 nodes and AMD64 nodes.

KubeKey v2.1.0 is not compatible with products made by KubeKey v2.0.0, please use KubeKey v2.1.0 to re-export KubeKey products. This is because KubeKey v2.1.0 optimizes the storage structure of artifacts. At present, the product adopts the OCI standard to pull and archive the image, realizing the reuse of the basic image layer, and its storage space occupation can be reduced by nearly half.

Argo CD releases v2.4 RC version

Argo CD has released its first v2.4 release candidate. This version has the following changes:

• Argo CD provides a web user interface that acts as a super powerful Kubernetes dashboard to help developers better understand their applications;
• Introduced additional access control settings to control access to Kubernetes Pod logs and new web terminal functionality;
• Version v2.4 introduced OpenTelemery tracking integration, providing more visibility to the Argo CD operator;
• The list of supported architectures has been expanded to now include IBM Z (s390x) and PowerPC (ppc64le). Starting from v2.4, the official quay.io repository will provide images for the amd64, arm64, ppc64le and s390x architectures.

For more details and installation instructions, please review the release notes and upgrade notes .

Docker provides Docker Extensions, Docker Desktop for Linux

Docker has added new products to its portfolio to help developers find and add tools to Docker Desktop and assist developers working on Linux-based systems.

At the DockerCon 2022 virtual conference, Docker announced Docker Extensions and Docker Desktop for Linux. Docker extensions built by multiple partners relieve developers of the tedious work of installing, configuring, and managing new tools.

To improve developer productivity, Docker Desktop for Linux comes pre-installed with tools such as Docker Compose, Docker CLI, and Kubernetes, simplifying the installation and update process of the tool.

European KubeCon + CloudNativeCon 2022 is coming soon

From May 16th to May 20th, 2022, KubeCon + CloudNativeCon Europe 2022 will be held in Valencia, Spain, offline + online simultaneously!

This event has received the full support of many IT giants at home and abroad, bringing together the world's most active open source cloud native community, the most advanced technical representatives and the best industry practice!

Huo Bingjie, the founder of OpenFunction, will share the theme : Empowering autonomous driving with cloud-native serverless technology at 21:25-22:00 on May 18th, Beijing time. Welcome everyone to pay attention!

This article is published by OpenWrite , a multi-post blog platform!