Rendered at 10:58:43 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nithril 5 hours ago [-]
One thing that bothered me enough to comment on: "transcoding" doesn't seem like the right term in this context. "Encode/decode" is technically correct, but "compress/decompress" would have made the intent much clearer.
walrus01 2 hours ago [-]
Transcoding is generally used within the context of audio or video codecs, to convert a file from one format to another. And usually from one lossy format to another (eg: not raw uncompressed YUV420, YUV422P video or whatever that is stored in a lossless compression format). It's not clear to me why they're using it in this web page.
TacticalCoder 32 minutes ago [-]
Yeah, the following sentence in TFA is weird:
> Transcoding does not mean compressing everything. Images, video, and fonts are usually compressed already.
It's because they're already losslessly compressed that, precisely, they're the typical targets for transcoding.
r3trohack3r 13 hours ago [-]
Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files.
For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.
There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!
oefrha 7 hours ago [-]
NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.
edflsafoiewq 5 hours ago [-]
It was the second most reported problem on the 2025 Rust survey, after compilation times.
hinkley 3 hours ago [-]
I was informed many eons ago that one of the hot features of Rational was that when one user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them.
It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone.
You may be cleaving the problem along the wrong plane if you're worried about sharing between multiple projects on the same machine versus across all of the members of a single team.
Artifactory is older than dirt. And its main feature isn't even saving bandwidth, it's ensuring that versioned libraries can never be overwritten in place with a different version potentially containing a trojan. And it can be set up not to download new versions until they've survived long enough for someone to hopefully notice a supply chain attack.
my-huge-pony 2 hours ago [-]
>user made a change that required recompiling the headers, those headers were uploaded to the server so that other users didn't have to recompile them.
This sounds like a security nightmare - there is a reason we have build servers (well there are many reasons, but build safety is one of them).
But I have to admit the idea is clever.
tancop 3 hours ago [-]
Rust has the deadly combo of static linking, dependency unification and a feature flag system. That makes it hard to share cached dependencies between projects, and when your compiler is as complex as rustc the individual files will end up big.
It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help.
The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
eviks 2 hours ago [-]
Don't they have nightly for imperfect experiments like this?
jmtulloss 6 hours ago [-]
I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.
veyh 3 hours ago [-]
Use a cargo workspace. Helps with both compilation speed and disk usage.
vbezhenar 2 hours ago [-]
I did simpler trick with jars (basically zip archives). Java ecosystem loves huge directories full of jars. If one would just use good compressor like 7z over that directory, it won't compress that good. So I extracted every jar into a separate directory and then compressed them all with 7z. The results were very good.
I just repeated the process. So directory of jars is 368M. If I just 7z it, it'll be 282M. But if I unpack them (1.9G), and then 7z them, it'll be 96M. Pretty substantial win.
The drawback is that you probably can't easily restore previous jar file byte-for-byte which might matter for some use-cases. I guess it's possible to achieve byte-for-byte copy with more effort.
a_t48 12 hours ago [-]
I've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.
Twirrim 10 hours ago [-]
I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.
hinkley 3 hours ago [-]
Some compression libraries have an 'rsync compatibility mode', that plays some games with the block sizes to make it easier to rsync to not have to completely re-transmit a large compressed file because not the whole file changes every time.
I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.
colechristensen 9 hours ago [-]
You may or may not be familiar with the content defined chunking family of algorithms like FastCDC.
On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.
chickenbig 4 hours ago [-]
Zstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck.
It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.
I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
gopalv 14 hours ago [-]
> I don't see how that's possible if the cache no longer stores the uncompressed data.
Zstd has a seekable format for frames, similar to pigz --independent works.
Speaking of pigz, I've run into pigzpp[1], or rather its paper: "pigzpp: Fast, Parallel, Portable Compression for the Whole Stack"[2].
Turns out we can squeeze quite a bit more compression performance out of DEFLATE - ~10x in certain instances, 2x as a base minimum (read the paper for details).
I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.
kccqzy 14 hours ago [-]
Actually zstd internally splits data into frames, and frames can indicate the decompressed data size. So if we control the compressor we can make it so that all frames have the size information; it isn’t exactly seekable but at least it will not need to decompress the resource. https://python-zstandard.readthedocs.io/en/latest/concepts.h...
Given how fast zstd can decompress, this may or may not actually be a win: the time spent waiting for I/O might be so large that the decompression can fit within the wait time.
CodesInChaos 15 hours ago [-]
Which would have terrible performance for range requests starting late in a large file. For files that are frequently accessed that way, this could be prohibitive.
You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.
Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.
In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
mgerdts 10 hours ago [-]
For the cost of a small amount of metadata the offsets of every MiB or so could be stored. I did something like this with pigz as I was implementing multithreaded compressed and encrypted kernel zone suspend and resume for Solaris.
genxy 15 hours ago [-]
Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.
Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces
mgerdts 10 hours ago [-]
ZFS compresses recordsize or volblocksize chunks down to some whole number of disk blocks, as determined by 1 >> ashift. In practice, this typically means that each 128k chunk gets compressed to some number of sequential 512 or 4096 byte blocks. These compressed blocks are referenced by block pointers that contain flags indicating compression and what type.
butvacuum 12 hours ago [-]
afaik, ZFS will read an entire Record at a time- and that's the same granularity as its compression.
a_t48 12 hours ago [-]
It uses some form of keyframing. Not entirely free.
butvacuum 12 hours ago [-]
they said they didn't change the behavior for range requests. So, it'll still be the basic no compression. (eg, server side decompression)
thinkindie 15 hours ago [-]
Why not serving files compressed if the client supports it even though the origin served an uncompressed file?
theandrewbailey 15 hours ago [-]
I was thinking this. Zstd is widely supported in browsers, over 80% right now and will increase over time:
They already do this, at least for paid accounts. You can even decide what compression model you want them to serve on your behalf.
londons_explore 4 hours ago [-]
Zstd allows a 'dictionary'.
With a dictionary, small objects - even just a couple of bytes - compress well.
Compress the headers too, and suddenly it's worth doing
MayeulC 15 hours ago [-]
> We initially considered limiting transcoding to popular content
Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
donavanm 12 hours ago [-]
When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime.
Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again.
And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.
genxy 15 hours ago [-]
I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients.
Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.
One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
mort96 14 hours ago [-]
It has mostly become my go-to as well. Just wish it wasn't a Facebook product.
nijave 13 hours ago [-]
That part is a little bit confusing.
I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source).
Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size)
"Ignore cold assets" makes more sense with that framing
Although if that's the case, the CPU statement still is a bit confusing.
articulatepang 14 hours ago [-]
I agree! I came to the comment section to say exactly this. In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.
nijave 13 hours ago [-]
Curious how this application scheme compares to filesystem + transport compression. You'd end up potentially compressing and decompressing more often but the higher software doesn't need to know what's happening and the compression happens in kernel space.
ie btrfs
You could also layer on out of band dedupe and probably push out cache updates with btrfs snapshots although maybe that ends too convoluted
zahlman 13 hours ago [-]
It could save PyPI petabytes per month of bandwidth, too. (But it seems like this is also caused by broken CI systems failing to cache things locally.)
repsilat 12 hours ago [-]
Yeah. It's worth looking at your own code and infrastructure as well. In my previous job there were dozens of opportunities to realise big savings by swapping out gzip for other compression schemes -- usually zstd, occasionally lz4 or bz2 or brotli. Huge assets took less hard drive space, took less time to download, and took less time to decompress. The differences were not small, and resulted in appreciable improvements both in infra cost and dev productivity.
> Transcoding does not mean compressing everything. Images, video, and fonts are usually compressed already.
It's because they're already losslessly compressed that, precisely, they're the typical targets for transcoding.
For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.
There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!
It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone.
You may be cleaving the problem along the wrong plane if you're worried about sharing between multiple projects on the same machine versus across all of the members of a single team.
Artifactory is older than dirt. And its main feature isn't even saving bandwidth, it's ensuring that versioned libraries can never be overwritten in place with a different version potentially containing a trojan. And it can be set up not to download new versions until they've survived long enough for someone to hopefully notice a supply chain attack.
This sounds like a security nightmare - there is a reason we have build servers (well there are many reasons, but build safety is one of them).
But I have to admit the idea is clever.
It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help.
The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
I just repeated the process. So directory of jars is 368M. If I just 7z it, it'll be 282M. But if I unpack them (1.9G), and then 7z them, it'll be 96M. Pretty substantial win.
The drawback is that you probably can't easily restore previous jar file byte-for-byte which might matter for some use-cases. I guess it's possible to achieve byte-for-byte copy with more effort.
I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.
https://joshleeb.com/posts/chunking.html
https://www.usenix.org/conference/atc16/technical-sessions/p...
On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.
It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.
Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
Zstd has a seekable format for frames, similar to pigz --independent works.
[1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
Turns out we can squeeze quite a bit more compression performance out of DEFLATE - ~10x in certain instances, 2x as a base minimum (read the paper for details).
[1]: https://github.com/thammegowda/pigzpp
[2]: https://arxiv.org/abs/2608.24153
Given how fast zstd can decompress, this may or may not actually be a win: the time spent waiting for I/O might be so large that the decompression can fit within the wait time.
You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.
Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.
In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
There's this, but it doesn't seem to be getting much traction: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f...
Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces
https://caniuse.com/?search=zstd
With a dictionary, small objects - even just a couple of bytes - compress well.
Compress the headers too, and suddenly it's worth doing
Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again.
And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.
Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.
One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source).
Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size)
"Ignore cold assets" makes more sense with that framing
Although if that's the case, the CPU statement still is a bit confusing.
ie btrfs
You could also layer on out of band dedupe and probably push out cache updates with btrfs snapshots although maybe that ends too convoluted