ClickHouse bottleneck impacts Cloudflare billing pipeline

Cloudflare encountered significant slowdowns in its billing pipeline, tracing the issue to a previously undetected bottleneck within ClickHouse, an open-source analytical database. The problem emerged after a table redesign aimed at enabling per-tenant data retention, which unexpectedly led to lock contention during query planning. Despite initial checks on standard performance metrics like I/O and memory showing normal levels, the critical billing jobs began failing to meet deadlines. This situation necessitated the development of patches to address the internal ClickHouse issue, highlighting the complexities of managing large-scale data platforms.