Security incidents happen. And when they do, they need to be dealt with—quickly. That’s where detection comes into play. The faster incidents are detected, the faster they can be handed off to the security team and resolved. To make detection as fast as possible, teams are usually aided by monitoring infrastructure that fires off an alert any time something even slightly questionable occurs. These alerts can lead to a deluge of information, making it difficult for engineers to sift through. Even worse, a large number of these alerts are false positives, caused by engineers arbitrarily running
We are pleased to announce the open source release of Lepton, our new streaming image compression format, under the Apache license.
Lepton achieves a 22% savings reduction for existing JPEG images, by predicting coefficients in JPEG blocks and feeding those predictions as context into an arithmetic coder. Lepton preserves the original file bit-for-bit perfectly. It compresses JPEG files at a rate of 5 megabytes per second and decodes them back to the original bits at 15 megabytes per second, securely, deterministically, and in under 24 megabytes of memory.
We have used Lepton to encode 16 billion images saved to Dropbox,
Written by Daniel Reiter Horn and Mehant Baid, Serving Infrastructure team at Dropbox.
In HBO’s Silicon Valley, lossless video compression plays a pivotal role for Pied Piper as they struggle to stream HD content at high speed.
John P. Johnson/HBO
Inspired by Pied Piper, we created our own version of their algorithm Pied Piper at Hack Week. In fact, we’ve extended that work and have a bit-exact, lossless media compression algorithm that achieves extremely good results on a wide array of images. (Stay tuned for more on that!)
At Dropbox, we made the switch from testing with unittest to pytest. We love the features, fixtures, plugins, and customizability of pytest. To further improve our experience, we built a couple of tools (pytest-flakefinder, unittest2pytest) for working with pytest and released them as open source.
We developed the pytest-flakefinder plugin to help with a common problem, flaky tests. Tests that involve multiple threads, or that depend on certain ordering can often fail at a fairly low rate. A few flaky tests aren’t a big deal,
This year’s Dropbox Hack Week saw some incredible projects take shape – from the talented team that visited Baltimore to research food deserts, to a project to recreate the fictional Pied Piper algorithm from HBO’s Silicon Valley. One of the most special elements of Hack Week, though, is that often times we’re able to share these exciting projects openly with our users and our community.
At Dropbox, we love and depend on numerous excellent open source projects, and we consider contributing back to the open source community to be vitally important. Popular open source projects that Dropbox has released include the zxcvbn password strength estimator,
Dropbox owes a large share of its success to Python, a language that enabled us to iterate and develop quickly. However, as our infrastructure matures to support our ever growing user base, we started exploring ways to scale our systems in a more efficient manner. About a year ago, we decided to migrate our performance-critical backends from Python to Go to leverage better concurrency support and faster execution speed. This was a massive effort–around 200,000 lines of Go code–undertaken by a small team of engineers. At this point, we have successfully moved major parts of our infrastructure to Go.
One recurring theme that hindered our development progress was the lack of robust libraries needed for building large systems.