Lepton image compression: saving 22% losslessly from images at 15MB/s

We are pleased to announce the open source release of Lepton, our new streaming image compression format, under the Apache license.

Lepton achieves a 22% savings reduction for existing JPEG images, by predicting coefficients in JPEG blocks and feeding those predictions as context into an arithmetic coder. Lepton preserves the original file bit-for-bit perfectly. It compresses JPEG files at a rate of 5 megabytes per second and decodes them back to the original bits at 15 megabytes per second, securely, deterministically, and in under 24 megabytes of memory.

We have used Lepton to encode 16 billion images saved to Dropbox,

Read more

Lossless compression with Brotli in Rust for a bit of Pied Piper on the backend

Written by Daniel Reiter Horn and Mehant Baid, Serving Infrastructure team at Dropbox.

In HBO’s Silicon Valley, lossless video compression plays a pivotal role for Pied Piper as they struggle to stream HD content at high speed.

John P. Johnson/HBO

Inspired by Pied Piper, we created our own version of their algorithm Pied Piper at Hack Week. In fact, we’ve extended that work and have a bit-exact, lossless media compression algorithm that achieves extremely good results on a wide array of images. (Stay tuned for more on that!)

However,

Read more

Open Sourcing Pytest Tools

At Dropbox, we made the switch from testing with unittest to pytest. We love the features, fixtures, plugins, and customizability of pytest. To further improve our experience, we built a couple of tools (pytest-flakefinder, unittest2pytest) for working with pytest and released them as open source.

We developed the pytest-flakefinder plugin to help with a common problem, flaky tests. Tests that involve multiple threads, or that depend on certain ordering can often fail at a fairly low rate. A few flaky tests aren’t a big deal,

Read more

Open Sourcing Zulip – a Dropbox Hack Week Project

This year’s Dropbox Hack Week saw some incredible projects take shape – from the talented team that visited Baltimore to research food deserts, to a project to recreate the fictional Pied Piper algorithm from HBO’s Silicon Valley. One of the most special elements of Hack Week, though, is that often times we’re able to share these exciting projects openly with our users and our community.

At Dropbox, we love and depend on numerous excellent open source projects, and we consider contributing back to the open source community to be vitally important. Popular open source projects that Dropbox has released include the zxcvbn password strength estimator,

Read more

Open Sourcing Our Go Libraries

Dropbox owes a large share of its success to Python, a language that enabled us to iterate and develop quickly. However, as our infrastructure matures to support our ever growing user base, we started exploring ways to scale our systems in a more efficient manner. About a year ago, we decided to migrate our performance-critical backends from Python to Go to leverage better concurrency support and faster execution speed. This was a massive effort–around 200,000 lines of Go code–undertaken by a small team of engineers. At this point, we have successfully moved major parts of our infrastructure to Go.

One recurring theme that hindered our development progress was the lack of robust libraries needed for building large systems.

Read more

Introducing Pyston: an upcoming, JIT-based Python implementation

Hello everyone, I’m very excited to announce Pyston, a new open-source implementation of Python, currently under development at Dropbox.  The goal of the project is to produce a high-performance Python implementation that can push Python into domains dominated by traditional systems languages like C++.

Here at Dropbox, we love Python and try to use it for as much as we can.  As we scale and the problems we tackle grow, though, we’re starting to find that hitting our performance targets can sometimes become prohibitively difficult when staying on Python.  Sometimes, it can be less work to do a rewrite in another language.  

Read more