Talks - Alex Monahan, Gabor Szarnyas: Python and SQL: Better Together, Powered by DuckDB

From the creator

Data management systems and data scientists have a troubled relationship: Common systems such as Postgres or Spark are hard to set up and maintain, hard to transfer data into and out of, and hard to integrate into Python workflows. In response, data scientists have developed their own data wrangling tools including Pandas and Polars. These tools are more natural to use, but are limited in the amount of data they can process and the amount of automatic optimization. DuckDB is a novel database management system purpose-built to combine cutting edge data processing and dataframe-like ease of use. DuckDB integrates deeply with the Python data analytics and engineering ecosystems - its Python client has over 1.5 million downloads each month. DuckDB can read and write popular Python dataframe libraries: Pandas, Polars, and Apache Arrow. DuckDB can even query Pandas dataframes faster than Pandas itself. Beyond dataframes, reading and writing is supported for Postgres, MySQL, and SQLite databases, and across many file formats (even on cloud object storage). In addition to the friendliest SQL dialect in the world, DuckDB provides options for using Pythonic dataframe syntax directly on top of the database engine. DuckDB includes a relational API, an experimental PySpark-compatible API, and is the default engine for the Ibis portable dataframe library. DuckDB supports complex queries, is MIT licensed, and has no external dependencies - it is a single pip install away! It is fast, eas

Published 2024-10-01 · Watch on YouTube ↗
Availability and playback restrictions are controlled by the original channel.