Lepton image compression: saving 22% losslessly from images at 15MB/s

We are pleased to announce the open source release of Lepton, our new streaming image compression format, under the Apache license.

Lepton achieves a 22% savings reduction for existing JPEG images, by predicting coefficients in JPEG blocks and feeding those predictions as context into an arithmetic coder. Lepton preserves the original file bit-for-bit perfectly. It compresses JPEG files at a rate of 5 megabytes per second and decodes them back to the original bits at 15 megabytes per second, securely, deterministically, and in under 24 megabytes of memory.

We have used Lepton to encode 16 billion images saved to Dropbox,

Read more

Pocket watch: Verifying exabytes of data


There is nothing more important to Dropbox than the safety of our user data. When we set out to build Magic Pocket, our in-house multi-exabyte storage system, durability was the requirement that underscored all aspects of the design and implementation. In this post we’ll discuss the mechanisms we use to ensure that Magic Pocket constantly maintains its extremely high level of durability.

This post is the second in a multi-part series on the design and implementation of Magic Pocket. If you haven’t already read the Magic Pocket design overview go do so now;

Read more

Lossless compression with Brotli in Rust for a bit of Pied Piper on the backend

Written by Daniel Reiter Horn and Mehant Baid, Serving Infrastructure team at Dropbox.

In HBO’s Silicon Valley, lossless video compression plays a pivotal role for Pied Piper as they struggle to stream HD content at high speed.

John P. Johnson/HBO

Inspired by Pied Piper, we created our own version of their algorithm Pied Piper at Hack Week. In fact, we’ve extended that work and have a bit-exact, lossless media compression algorithm that achieves extremely good results on a wide array of images. (Stay tuned for more on that!)


Read more

Going deeper with Project Infinite

Last month at Dropbox Open London, we unveiled a new technology preview: Project Infinite. Project Infinite is designed to enable you to access all of the content in your Dropbox—no matter how small the hard disk on your machine or how much stuff you have in your Dropbox. Today, we’d like to tell you more—from a technical perspective—about what this evolution means for the Dropbox desktop client.

Traditionally, Dropbox operated entirely in user space as a program just like any other on your machine. With Dropbox Infinite, we’re going deeper: into the kernel—the core of the operating system.

Read more

Enabling HTTP/2 for Dropbox web services: experiences and observations

At Dropbox, our traffic team recently upgraded the front-end Nginx servers to enable HTTP/2 for our web services. In this article, we would like to share our experiences and findings during the HTTP/2 transition. The overall upgrade was smooth for us, although there are also a couple of caveats that might be helpful to others.

Background: HTTP/2 and Dropbox web service infrastructure

HTTP/2 (RFC 7540) is the new major version of the HTTP protocol. It is based on SPDY and provides several performance optimizations compared to HTTP/1.1. These optimizations include more efficient header compression,

Read more

Inside the Magic Pocket

We’ve received a lot of positive feedback since announcing Magic Pocket, our in-house multi-exabyte storage system. We’re going to follow that announcement with a series of technical blog posts that offer a look behind the scenes at interesting aspects of the system, including our protection mechanisms, operational tooling, and innovations on the boundary between hardware and software. But first, we’ll need some context: in this post, we’ll give a high level architectural overview of Magic Pocket and the criteria it was designed to meet.

Read more