Dropbox is an active customer of Amazon Web Services, currently operating one of the largest global deployments into S3, tens of thousands of EC2 instances, and heavily utilizing other services like SQS and Route 53. Pushing hundreds of gigabits per second through EC2/S3 is an everyday occurrence for us, and conducting massively parallel operations across our over one trillion objects in S3 happens on an ongoing basis.
However, that’s just half the story. We also have large physical datacenters split between two geographical regions, running tens of thousands of servers responsible for storing and serving the metadata for every file in Dropbox.
Making Carousel highly responsive was a critical part of providing an awesome user experience. Carousel wouldn’t be as usable or effective if the app stuttered or frequently caused users to wait while content loaded. In our last post, Drew discussed how we optimized our metadata loading pipeline to respond to data model changes quickly, while still providing fast lookups at UI bind time. With photo metadata in memory, our next challenge was drawing images to the screen. Dealing with tens of thousands of images while rendering at 60 frames per second was a challenge, especially in the mobile environments of iOS and Android.
In last week’s post, Kat described how we redesigned our new user experience from the ground up to make it a delight for users to get started on Dropbox from our mobile apps. In this post, I’ll go into more detail about everything we did to make the mobile-to-desktop transition simple for users. To recap the previous post, here’s a summary of the flow:
- The desktop connect flow allows users to log in to the website with only their phone, without typing in credentials, and initiates a download.
- The “meta-installer” downloads quickly because of its small file size.
At Dropbox, we treat growth as an integral part of the product experience. We look at major holes in user experience that slow growth, and we try to be creative in addressing the big picture, rather than trying to “growth hack.” We look for solutions that enable users to experience the full value of Dropbox.
Dropbox has always been about accessing your stuff anywhere. Back when Dropbox launched six years ago, that meant installing Dropbox on your desktop and then accessing photos and docs on the web or on a smartphone, for some. Our smartphone apps were a way of helping users who already had Dropbox on their desktop view docs on the go,
In the last post, Stephen explored the asynchronous client-server communication model we use in Carousel to provide a fast user experience, where interactions aren’t blocked on network calls. But network latency was not our only hurdle. In order to make the app feel fast and responsive, we also needed to minimize user-visible disk latency. One of Carousel’s primary goals was to make all of the user’s photos and videos accessible in one continuous view. We have users with over 100,000 photos in Dropbox, and we needed to build a metadata-loading pipeline that could accommodate them. This meant building a metadata-loading system that can show photos on-screen within a second of opening the app and provides smooth scrolling even as the user navigates back in time through their entire history of photos.
Our users love Dropbox for many reasons, sync performance being chief among them. We’re going to look at a recent performance improvement called Streaming Sync which can improve sync latency by up to 2x.
Prior to Streaming Sync, file synchronization was partitioned into two phases: upload and download. The entire file must be uploaded to our servers and committed to our databases before any other clients could learn of its existence. Streaming sync allows file contents to “stream” through our servers between your clients.
The Dropbox File System
First we’ll discuss the way Dropbox stores and syncs files.