Cloudflare boosts security scanning capacity tenfold

Cloudflare has significantly enhanced its Security Insights service, achieving a tenfold increase in global scanning capacity to meet growing security demands. Previously, scans were infrequent and opt-in for many free accounts, leaving users vulnerable for extended periods. To address this, the company upgraded its system to process 100 scans per second, up from 10. This involved optimizing Kafka message processing through batching and parallel goroutines, and implementing a 'fast lane' and 'slow lane' system to prevent slow-processing messages from delaying others. Database query efficiency was also improved to handle large data volumes.