Extending Magic Pocket Innovation with the first petabyte scale SMR drive deployment

Magic Pocket, the exabyte scale custom infrastructure we built to drive efficiency and performance for all Dropbox products, is an ongoing platform for innovation. We continually look for opportunities to increase storage density, reduce latency, improve reliability, and lower costs. The next step in this evolution is our new deployment of specially configured servers filled to capacity with high-density SMR (Shingled Magnetic Recording) drives.

Dropbox is the first major tech company to adopt SMR technology, and we’re currently adding hundreds of petabytes of new capacity with these high-density servers at a significant cost savings over conventional PMR (Perpendicular Magnetic Recording) drives.

Read more

Security culture, the Dropbox way

The Dropbox Security Team is responsible for securing around 1 exabyte of data, belonging to over half a billion registered users across the world. The responsibility for securing data at this scale extends far beyond the Dropbox Security Team—it takes a commitment from everyone at Dropbox to safeguard our users’ data every day. In other words, it takes a strong security culture.

The first core company value at Dropbox is “Be Worthy of Trust.” From a security perspective, this means keeping our users’ stuff safe. Our culture of security is built on this foundation of trust and is a fundamental part of our identity.

Read more

How we’re winning the battle against flaky tests

Testing is a crucial part of maintaining a code base, but not all tests validate what they’re testing for. Flaky tests—tests that fail sometimes but not always—are a universal problem, particularly in UI testing. In this blog post, we will discuss a new and simple approach we have taken to solve this problem. In particular, we found that a large fraction of most test code is setting up the conditions to test the actual business-logic we are interested in, and consequently a lot of the flakiness is due to errors in this setup phase. However, these errors don’t tell us anything about whether the primary test condition succeeded or failed,

Read more

Optimizing web servers for high throughput and low latency

This is an expanded version of my talk at NginxConf 2017 on September 6, 2017. As an SRE on the Dropbox Traffic Team, I’m responsible for our Edge network: its reliability, performance, and efficiency. The Dropbox edge network is an nginx-based proxy tier designed to handle both latency-sensitive metadata transactions and high-throughput data transfers. In a system that is handling tens of gigabits per second while simultaneously processing tens of thousands latency-sensitive transactions, there are efficiency/performance optimizations throughout the proxy stack, from drivers and interrupts, through TCP/IP and kernel, to library, and application level tunings.

Read more

Creating a Modern OCR Pipeline Using Computer Vision and Deep Learning

In this post we will take you behind the scenes on how we built a state-of-the-art Optical Character Recognition (OCR) pipeline for our mobile document scanner. We used computer vision and deep learning advances such as bi-directional Long Short Term Memory (LSTMs), Connectionist Temporal Classification (CTC), convolutional neural nets (CNNs), and more. In addition, we will also dive deep into what it took to actually make our OCR pipeline production-ready at Dropbox scale.

In previous posts we have described how Dropbox’s mobile document scanner works. The document scanner makes it possible to use your mobile phone to take photos and

Read more