• Background tasks in web apps: Client requests lead to time-expensive tasks. App enqueues background jobs to handle these, allowing faster response. Background processing does the heavy lifting. Leveraging background jobs is crucial for scaling web apps.
• Rails libraries for background jobs: As Rails developers, there are several libraries to choose from with different advantages. These make offloading heavy tasks easy.
• Sidekiq at Honeybadger: Used for most background processing jobs. Helped maintain a fast user experience but had limitations. Redis had limitations like memory exhaustion in the ElastiCache cluster, especially during high error traffic. Redis uses the volatile-lru eviction policy by default, which can lead to data eviction. Needed a system resilient to data loss.
• Using Kafka for ingesting data: Kafka is a distributed event pipeline with scalability and resiliency. Used it to process event data and aimed to use it for error ingestion. Already had AWS MSK cluster for Insights, so setting up for error ingestion was simple.
• Migrating from Sidekiq to Karafka: Honeybadger is a monolith and Karafka helps maintain the architecture. Jobs are batched and processed in Karafka. Error handling is different with Karafka having a Dead Letter Queue.
• Monitoring and scaling Karafka: Monitoring and scaling Karafka is complex. AWS CloudWatch metrics like SumOffsetLag and EstimatedMaxTimeLag are monitored. Karafka also provides its own instrumentation like processing_lag, consumption_lag, and duration. Scaling Sidekiq and Karafka is different. Scaling up/down Kafka consumers can be a lengthy operation. Currently monitoring SumOffsetLag as a scaling metric.
• Next for Karafka at Honeybadger: Running Kafka/Karafka for over a month with satisfaction. Can fallback to Sidekiq if needed. Learned more about working with Kafka and Karafka. Updated Honeybadger gem with new karafka plugin features. Have an Insights Karafka Dashboard to visualize data. Encourages customers to use the data to improve Kafka systems.