使用 grep 搜索日志仍然很糟糕 - 关于新事物的编年史

  • Main Point: Grepping logs remains terrible even with modern hardware; a late-2014 Mac Mini with 4GB RAM and an HDD outperforms a desktop with 32GB RAM and an M.2 SSD when it comes to grepping logs.
  • Key Information:

    • The author last wrote about logging a month and a bit ago, almost a decade since stating that grepping logs is terrible.
    • The experiment compares a Mac Mini (Quickbeam) and a desktop (Telchar) using a simple script bin/lg to query VictoriaLogs.
    • The data consists of 4.4 million lines of JSON (about 4.4GB on disk) representing Caddy logs.
    • Simple grepping queries show that the Mac Mini is faster for some tasks like counting requests for a specific host.
    • When using a regexp, the database is slower on the desktop but still faster than grepping on the Mac Mini.
    • For aggregating data and showing hits per unique host, the Mac Mini with VictoriaLogs is significantly faster.
    • VictoriaLogs is a high-performance log database specifically built for logs, with structured data indexed and tokenized, making queries cheaper and data smaller.
  • Important Details:

    • The time taken for various operations on both the Mac Mini and the desktop is provided, showing the significant performance difference.
    • Examples of different queries and their results are given to illustrate the performance disparity.
    • The author emphasizes that purpose-built software like VictoriaLogs is crucial for efficient log querying and that grepping plain text logs is inefficient and slow.
阅读 7
0 条评论