Rendered at 10:58:37 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
benrutter 3 hours ago [-]
> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
nicce 2 hours ago [-]
> Version bumps should really be about removing deprecated cruft rather than shiny new features.
Can there be deprecated cruft without new features? :-D
kvdveer 17 minutes ago [-]
Ideally: no.
All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.
In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
dist-epoch 2 hours ago [-]
That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.
Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL.
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
bobson_dugnutt5 2 hours ago [-]
I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.
anotherpaul 2 hours ago [-]
I gave up pandas in favor of polars after someone at work did the same and I am very happy with it. Pandas API is just so much worse and much slower.
mgaunard 2 hours ago [-]
Both have terrible syntax that make SQL look like the most readable thing ever.
condwanaland 2 hours ago [-]
Could not agree less. Ive always found SQL an unreadable mess but tools like polars and dplyr are such elegant ways to manipulate data.
Pandas is a mess though.
world2vec 30 minutes ago [-]
There's no way SQL is more unreadable than polars. IMO it's the other way around.
benrutter 11 minutes ago [-]
> There's no way SQL is more unreadable than polars. IMO it's the other way around.
I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
aquafox 2 hours ago [-]
Coming from an R/dplyr background, I agree. Compare
df.select(
pl.col("x"),
(pl.col("w")/pl.col("z")).alias("y")
)
with
df |>
select(x, y = w/z)
orlp 1 hours ago [-]
from polars import col as C
df.select(C.x, y = C.w / C.z)
bobson_dugnutt5 2 hours ago [-]
Fair point, but you can do something like
`df.select("x", y=pl.col.w/pl.col.z)`
jcattle 56 minutes ago [-]
R really is/was the superior traditional data science language. Python ecosystem is slowly catching up though.
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
vovavili 11 minutes ago [-]
Altair and Positron should be just as good for your Polars @ Python needs. With software like Marimo notebooks and VegaFusion, Polars/Python experience starts beating R by quite a substantial margin.
bobson_dugnutt5 2 hours ago [-]
What is it about polars syntax you don't like? The fact that is very verbose? At first I wasn't a fan, but over time I've grown to really like it. That never happened to me with pandas, always felt the syntax was messy
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
fzumstein 2 hours ago [-]
I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.
refactor_master 1 hours ago [-]
The cool thing about polars is that you can conditionally collect expressions over many layers of business logic, and then compute the result at the end. Doing this in SQL ends up in a hodgepodge of strings and trimmed ends to please the syntax. You can also pretty effortlessly write quite complex conditionals directly in polars, and bridge it easily to the surrounding python.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
fzumstein 1 hours ago [-]
This sounds interesting! Do you have a specific example by any chance or blog post/doc references?
refactor_master 19 minutes ago [-]
[dead]
arn3n 53 minutes ago [-]
The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?
orlp 40 minutes ago [-]
Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
arn3n 21 minutes ago [-]
Cool, thanks for the explanation!
bluebarbet 52 minutes ago [-]
What does this project have to do with Serbia? Are the developers in Belgrade?
rfgplk 2 hours ago [-]
Seeing "release will land in the following weeks" kind of immediately turns me off.
thibaut_barrere 2 hours ago [-]
I like when large projects do that. This gives leeway for sister projects (eg wrappers) to anticipate, room for apps that use it intensively to test things out (release candidate etc), something which has really helped me in the past.
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.
NSPG911 18 minutes ago [-]
release will be [released] in the following weeks
marliechiller 2 hours ago [-]
What is your understanding of a Pre-Release then?
dbdr 2 hours ago [-]
Why?
irpap 2 hours ago [-]
I assume because “land” is a word Claude would choose.
mgaunard 2 hours ago [-]
Claude's wording (and knowledge) is based on what competent senior engineers would say.
tecleandor 2 hours ago [-]
None of the competent senior engineers I know say "not this, but that" every two sentences.
tancop 1 hours ago [-]
It's the only good word here. "Drop" can also mean the opposite and anything else sounds too formal. Don't get me started on "release will release".
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
Can there be deprecated cruft without new features? :-D
All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.
In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
Pandas is a mess though.
I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
df.select(
)with
df |> select(x, y = w/z)
`df.select("x", y=pl.col.w/pl.col.z)`
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.