Every Obvious Approach Failed: Building an Async Database Client...(Sponsor: Clickhouse)

From the creator

Full presentation title: "Every Obvious Approach Failed: Building an Async Database Client That Actually Scales" by Sponsor: Clickhouse I maintain clickhouse-connect, the official ClickHouse Python client. Our first async implementation followed the standard playbook of wrapping the synchronous client in run_in_executor. That works until large result sets and real concurrency show up and tail latency starts spiking in ways ordinary profiling doesn’t explain. The problem isn’t specific to ClickHouse. When a database can stream data efficiently but decoding that data is CPU-heavy, network I/O and parsing stop being independent. You can make the socket non-blocking and still end up with an async system that behaves badly under load. This talk walks through the obvious designs I tried, why each one failed, and the architecture that finally held up which is async I/O on one side, synchronous CPU-bound parsing on the other, and a bounded queue between them to provide backpressure. I’ll show the pattern in the context of a database client, then generalize it to other Python systems that mix async I/O with heavy CPU work.

Published 2026-09-18 · Watch on YouTube ↗
Availability and playback restrictions are controlled by the original channel.