Building better compression together with DivANS

Compressing your files is a good way to save space on your hard drive. At Dropbox’s scale, it’s not just a good idea; it is essential. Even a 1% improvement in compression efficiency can make a huge difference. That’s why we conduct research into lossless compression algorithms that are highly tuned for certain classes of files and storage, like Lepton for jpeg images, and Pied-Piper-esque lossless video encoding. For other file types, Dropbox currently uses the zlib compression format, which saves almost 8% of disk storage.

We introduce DivANS,

Read more

Extending Magic Pocket Innovation with the first petabyte scale SMR drive deployment

Magic Pocket, the exabyte scale custom infrastructure we built to drive efficiency and performance for all Dropbox products, is an ongoing platform for innovation. We continually look for opportunities to increase storage density, reduce latency, improve reliability, and lower costs. The next step in this evolution is our new deployment of specially configured servers filled to capacity with high-density SMR (Shingled Magnetic Recording) drives.

Dropbox is the first major tech company to adopt SMR technology, and we’re currently adding hundreds of petabytes of new capacity with these high-density servers at a significant cost savings over conventional PMR (Perpendicular Magnetic Recording) drives.

Read more

How we’re winning the battle against flaky tests

Testing is a crucial part of maintaining a code base, but not all tests validate what they’re testing for. Flaky tests—tests that fail sometimes but not always—are a universal problem, particularly in UI testing. In this blog post, we will discuss a new and simple approach we have taken to solve this problem. In particular, we found that a large fraction of most test code is setting up the conditions to test the actual business-logic we are interested in, and consequently a lot of the flakiness is due to errors in this setup phase. However, these errors don’t tell us anything about whether the primary test condition succeeded or failed,

Read more

Meet Bandaid, the Dropbox service proxy

With this post we begin a series of articles about our Service Oriented Architecture components at Dropbox, and the approaches we took in designing them. Bandaid, our service proxy, is one of these components. Follow along as we discuss Bandaid’s internal design and the approaches we chose for the implementation.

Bandaid started as a reverse proxy that compensated for inefficiencies in our server-side services. Later we developed it into a service proxy that accelerated adoption of Service Oriented Architecture at Dropbox.

A reverse proxy is a device or service that forwards requests from multiple clients to servers (i.e.

Read more

Deploying IPv6 in Dropbox Edge Network

In the past few months, we have gradually enabled IPv6 for all user-facing services in Dropbox edge network. We are serving about 15% of daily user requests in IPv6 globally. In this article, we share our experiences and lessons from enabling IPv6 in the edge network. We will cover the IPv6 design in the edge, the changes we made to support IPv6, how IPv6 was tested and rolled out to users, and issues we encountered. Note that this article is not about enabling IPv6 for internal services in our data centers, but rather focuses on making IPv6 available to users.

Read more

Handling System Failures during Payment Communication

Handling system failures during payment processing requires real-time identification of the issues in addition to offline detection, with the goal of eventual consistency. No matter what goes wrong, our top priority is to make sure that customers receive service for which they’ve been charged, and aren’t charged for service they haven’t received. Accurate payment processing is a crucial element in being worthy of trust, a core Dropbox company value.

In a standard system of this kind, failures might result in page load errors or a failed database transaction. System failures during a charge request can result in uncertainty about where the money for that request ended up: is it in our company’s account or still in the customer’s account?

Read more