Season's Savings | ₹5/mo for 3 months

₹199/month after first 3 months. Renews automatically. Terms apply.
Ep. 24 - Navigating the Open Source Data Landscape cover art

Ep. 24 - Navigating the Open Source Data Landscape

Ep. 24 - Navigating the Open Source Data Landscape

Listen for free

View show details
About this listen
Some guests need a long introduction. Marco Gorelli is one of them, so I’ll keep it short: mathematician, software engineer at Quansight Labs, former pandas maintainer, Polars contributor, and the creator of Narwhals, a library that gives you one API on top of the major dataframe libraries.We first met at PyCon Italy, when Narwhals was only a few months old, and we’ve run into each other at conferences since. This time we sat down properly and talked about the stuff that usually only comes up in a conference hallway: why some APIs win, how you get anyone to notice a new open source project, and what maintainers are dealing with now that coding agents can open pull requests.Here are the highlights.From pure maths to pandas maintainerMarco studied mathematics, followed by a master’s in maths and the foundations of computer science. Despite the name, it involved no programming at all, “much to some recruiters’ disappointment” when they tried to place him in very applied jobs.He went into data science around 2017–2018, when the field was booming. He pointed out that the way people talked about data science back then sounds a lot like the way people talk about AI today, which in turn sounds like how they talked about big data a few years earlier. (When was the last time you heard someone say “big data”?)The way into open source was very ordinary. He hit a bug in pandas at work and thought it might be fun to fix it. Then COVID hit, he had extra time, and he kept going. Eventually he became a pandas maintainer, and that led to a job at Quansight Labs working on open source full time.Why Polars won by not copying pandasAt Quansight, Marco got into Rust and Polars, a dataframe library written in Rust that avoids many of pandas’ historical design decisions.It’s easy to forget how recently people “would look at me as if I had three heads” if he mentioned Polars alongside pandas. Early on, the critics said nobody would use a dataframe library without the pandas index or the pandas API. Polars was adopted widely anyway. Meanwhile, several projects that copied pandas exactly, only faster, have mostly disappeared.Marco’s explanation: Ritchie Vink, the creator of Polars, treated not following the pandas API as a feature and not a limitation.> “Yes, there is a learning curve, but once you’re over that learning curve, everything is very consistent. Once you get used to the Polars way of thinking, it is a pleasure to work with.”His practical advice: the best time to try Polars is when you’re starting a new project. Don’t try to recreate your pandas habits. Learn the Polars way of thinking from the start and you get the benefits.Why pandas struggles to keep upI asked whether many of the complaints about pandas come from the fact that it has been around so long, through data mining, big data, machine learning, deep learning and now generative AI.Marco agreed and pointed to hardware. pandas was designed around 2008, when 16-core CPUs weren’t something everyone had. Today they’re common and cheap to rent. The problem is that the pandas API doesn’t make it obvious what can run in parallel. People say “pandas should just do things in parallel”, but the API doesn’t make clear how that would work.Polars makes it explicit. Each expression inside with_columns runs independently, so they can all run multi-threaded.> “The world has certainly changed around pandas, but it’s not easy for pandas to change itself at the same pace.”What you gain by going from user to contributorMarco named two things.1. You appreciate the invisible work. pandas is run entirely by volunteers, with the occasional grant. Polars has a company behind it, but much of what goes into the free tool isn’t driven by money. It’s driven by people who want to make data work better.2. You learn API design the hard way. Users often think there should be a built-in function for their use case. But if a library added a function for every use case, the API would become impossible to navigate, and every function would have 20 parameters. As the old pandas maintainer joke goes:> “There is no problem that cannot be solved by adding an extra keyword argument to read_csv.”(It’s probably past 50 by now.) A good tool has to be fairly minimal. It can’t solve everyone’s problem, but it can give you the building blocks to solve your own.Narwhals: copying Polars, not pandasI’ve always found it fascinating that something as simple as mean behaves differently across libraries. Marco gave an example right away: missing values. Most tools skip nulls by default, but the way you configure that differs between PyArrow, pandas, Polars and SQL.That’s the kind of problem Narwhals solves. Marco started it about two and a half years ago as a unified dataframe API, aimed mainly at library maintainers who want to accept different kinds of dataframe inputs.He had three options: copy pandas, copy another library, or invent a ...
adbl_web_anon_alc_button_suppression_t1
No reviews yet