Optimizing web servers for high throughput and low latency


This is an expanded version of my talk at NginxConf 2017 on September 6, 2017. As an SRE on the Dropbox Traffic Team, I’m responsible for our Edge network: its reliability, performance, and efficiency. The Dropbox edge network is an nginx-based proxy tier designed to handle both latency-sensitive metadata transactions and high-throughput data transfers. In a system that is handling tens of gigabits per second while simultaneously processing tens of thousands latency-sensitive transactions, there are efficiency/performance optimizations throughout the proxy stack, from drivers and interrupts, through TCP/IP and kernel, to library, and application level tunings.

Read more

Evolution of Dropbox’s Edge Network

Update (November 14, 2017): Miami, Sydney, Paris, Milan and Madrid have been added to the Dropbox Edge Network.

Since launching Magic Pocket last year, we’ve been storing and serving more than 90 percent of our users’ data on our own custom-built infrastructure, which has helped us to be more efficient and improved performance for our users globally.

But with about 75 percent of our users located outside of the United States, moving onto our own custom-built data center was just the first step in realizing these benefits. As our data centers grew,

Read more

Introducing Cape

More than a billion files are saved to Dropbox every day, and we need to run many asynchronous jobs in response to these events to power various Dropbox features. Examples of these asynchronous jobs include indexing a file to enable search over its contents, generating previews of files to be displayed when the files are viewed on the Dropbox website, and delivering notifications of file changes to third-party apps using the Dropbox developer API. This is where Cape comes in — it’s a framework that enables real-time asynchronous processing of billions of events a day,

Read more

Augmented camera previews for the Dropbox Android Document Scanner

With Dropbox’s document scanner, a user can take a photo of a document with their phone and convert it into a clean, rectangular PDF. In our previous blog posts (Part 1, Part 2), we presented an overview of document scanner’s machine learning backend, along with its iOS implementation. This post will describe some of technical challenges associated with implementing the document scanner on Android.

We will specifically focus on all steps required to generate an augmented camera preview in order to achieve the following effect:

Read more

Creating a culture of accessibility

At Dropbox, we strive to make products that are easy for everyone to use. As part of that mission, we’ve been improving product accessibility for users with disabilities, and building a collaborative culture in which our engineers understand and value accessibility best practices as part of their process.

To create accessible products, you need to find opportunities to spread accessibility knowledge and enthusiasm in a sustainable way throughout your company. But awareness is one of the largest barriers to implementing these best practices into a product. Most computer science curriculums at colleges and universities don’t include in-depth coverage of accessibility (though organizations like Teach Access are working on changing that!).

Read more

Adding IPv6 connectivity support to the Dropbox desktop client

Computers on the internet are uniquely identified by an IP address. For decades the world has used Internet Protocol version 4 (IPv4), which allows for about 4 billion unique addresses. As more of the world has come online, and we carry internet-capable devices in our pockets, we have run out of IPv4 addresses. Layers and layers of workarounds have been built to mitigate the problem. The current protocol—Internet Protocol version 6 (IPv6)—fixes various problems with IPv4; it has a significantly expanded address space that allows for the creation of many more unique IP addresses. Unfortunately, IPv6 has suffered from lack of adoption.

Read more