Building better compression together with DivANS

Compressing your files is a good way to save space on your hard drive. At Dropbox’s scale, it’s not just a good idea; it is essential. Even a 1% improvement in compression efficiency can make a huge difference. That’s why we conduct research into lossless compression algorithms that are highly tuned for certain classes of files and storage, like Lepton for jpeg images, and Pied-Piper-esque lossless video encoding. For other file types, Dropbox currently uses the zlib compression format, which saves almost 8% of disk storage.

We introduce DivANS,

Read more

Balancing open source and proprietary IP—they can co-exist

Open source software can provide significant benefits to an organization—it can decrease product development time, distribute development across a community, and attract developers to your organization. It’s because of these benefits that we at Dropbox love open source. However, some organizations shy away from it due to perceived risks and fears around lost intellectual property (IP) rights. You’re not alone if you’re worried that once you’ve incorporated open source into your products or open sourced your own code that you’ve surrendered control over your most valuable assets, or worse, left your organization vulnerable to litigation with no defensive weapons to counter the threat.

Read more

Meet Securitybot: Open Sourcing Automated Security at Scale

Security incidents happen. And when they do, they need to be dealt with—quickly. That’s where detection comes into play. The faster incidents are detected, the faster they can be handed off to the security team and resolved. To make detection as fast as possible, teams are usually aided by monitoring infrastructure that fires off an alert any time something even slightly questionable occurs. These alerts can lead to a deluge of information, making it difficult for engineers to sift through. Even worse, a large number of these alerts are false positives, caused by engineers arbitrarily running sudo -i or nmap.

Read more

Lepton image compression: saving 22% losslessly from images at 15MB/s

We are pleased to announce the open source release of Lepton, our new streaming image compression format, under the Apache license.

Lepton achieves a 22% savings reduction for existing JPEG images, by predicting coefficients in JPEG blocks and feeding those predictions as context into an arithmetic coder. Lepton preserves the original file bit-for-bit perfectly. It compresses JPEG files at a rate of 5 megabytes per second and decodes them back to the original bits at 15 megabytes per second, securely, deterministically, and in under 24 megabytes of memory.

We have used Lepton to encode 16 billion images saved to Dropbox,

Read more

Lossless compression with Brotli in Rust for a bit of Pied Piper on the backend

Written by Daniel Reiter Horn and Mehant Baid, Serving Infrastructure team at Dropbox.

In HBO’s Silicon Valley, lossless video compression plays a pivotal role for Pied Piper as they struggle to stream HD content at high speed.

John P. Johnson/HBO

Inspired by Pied Piper, we created our own version of their algorithm Pied Piper at Hack Week. In fact, we’ve extended that work and have a bit-exact, lossless media compression algorithm that achieves extremely good results on a wide array of images. (Stay tuned for more on that!)

However,

Read more

Open Sourcing Pytest Tools

At Dropbox, we made the switch from testing with unittest to pytest. We love the features, fixtures, plugins, and customizability of pytest. To further improve our experience, we built a couple of tools (pytest-flakefinder, unittest2pytest) for working with pytest and released them as open source.

We developed the pytest-flakefinder plugin to help with a common problem, flaky tests. Tests that involve multiple threads, or that depend on certain ordering can often fail at a fairly low rate. A few flaky tests aren’t a big deal,

Read more