Nolan Lawson

By: Nolan Lawson
Published: 13 January 2016

"Rollup! Rollup for the mystery tour!" - Lennon and McCartney

Besides a slew of bugfixes and the new revs_limit feature, the big news about 5.2.0 is that PouchDB is now authored with ES6 modules rather than CommonJS. Using Rollup, it's transpiled into one big index.js for Node and an index-browser.js for Webpack/Browserify.

The most immediate benefit is that PouchDB's minified/gzipped size has dropped by 3KB (6.7% smaller!) with no loss of functionality. This is thanks to Rollup's ability to remove the typical Browserify/Webpack cruft by hoisting all submodules into a single scope.

Another benefit is that PouchDB is now insulated from subtle differences between build systems like Browserify, Webpack, Babel, and Rollup itself. This is an esoteric problem for JavaScript library authors, but I think it's worth going into detail, because it affects more libraries than just PouchDB.

I'll explore that topic later on in the post, but first, here's the full changelog for 5.2.0:

Changelog

Features and improvements

  • Implement revs_limit (#2839)
  • Migrate to ES6/Rollup, build one index.js (#4652)
  • Add ability to disable timeout in changes() (#4583)

Bugfixes

  • Return conflict error when inserting an unknown rev (#4712)
  • Catch XHR errors and propagate (#4595)
  • Fix and test Webpack (#4700)
  • Improve error when user sends invalid base64 (#4208)
  • Consider changes erroring a valid result (#4677)
  • Allow bulkGet requests without an explicit rev (#4530)
  • Fix for xhr.upload detection failing (#4560)
  • Coerce options which should be numbers to number (#4578)
  • Add timeout option to replication (#4540)
  • Remove binary string conversion (#4529)
  • Remove CORS explanation (#4677)
  • Add browser sniffing and remove nonce by default (#4543)
  • Remove unneeded host.headers from ajax (#4567)
  • Fix double-encoding name in http adapter (#4514)
  • Fix and document heartbeat for replication (#4538)
  • Fallback from _bulk_get on 50x requests (#4542)
  • Remove Cordova init checks (#4756)
  • Reset changedDocs in write_docs (#4627)

PouchDB and JavaScript build systems

To explain why we're now using Rollup to bundle one big index.js for Node, we'll need to take a look at the open-source JavaScript landscape, as it stands in 2016.

I've been working on PouchDB for about two years. In that short amount of time, I've seen the frontend ecosystem largely transition from prebuilt <script>-ready JavaScript files, usually distributed via Bower, CDN, or direct download, to Node modules distributed by npm and directly consumed by build tools like Browserify, Webpack, Babel, Grunt, Gulp, etc.

A big impact of this is that library authors no longer have control over the "last mile." Whereas our code used to be plopped directly into an HTML page (or merely concatenated), nowadays it's often parsed, chopped up, and transformed before it reaches that HTML page. This process inevitably introduces bugs and inconsistencies.

For instance, PouchDB has been struggling with Webpack support for much of the past year. The number of times we've had Webpack-related bugs filed on us, and the number of times we've later regressed because we only test with Browserify, eventually compelled us to test Webpack separately. This means we're essentially testing both Browserify and Webpack as build targets.

Think this sounds extreme? Consider this example.

Example 1: requiring JSON

Let's say you added this seemingly-innocuous line of code to your JavaScript library:

MyLibrary.version = require('./package.json').version;

This will work in Browserify and Node, but it will break in Webpack unless the end-user adds a special json-loader to their Webpack configuration.

Does your library do this? Do you have a dependency that does this? Then congratulations, you cannot fix this for your users. All you can do is document it in your README and/or endure the inevitable bug reports.

This is not something Webpack intends to fix, since it seems to constitute a philosophical difference between Browserify and Webpack. Their argument is that end-users should have control over how libraries are built. (You can read this discussion for details.)

Example 2: Browserify transforms

Another good example of Browserify/Webpack differences is Browserify transforms. These are modules that can transform your library at build time, when it's consumed by Browserify.

Transforms are very popular in the Browserify ecosystem, but unfortunately they don't work in Webpack unless you use the transform-loader. Worse, this is another thing that end-users are expected to configure for every dependency (and every dependency of a dependency…).

So if your project is built with Browserify, it's easy to accidentally add a Browserify transform, while forgetting that it will break for Webpack users. (We nearly made that mistake ourselves, before we started testing Webpack.) So the only reasonable choice for library authors is to avoid Browserify transforms entirely.

Example 3: strict mode

To stop picking on Webpack for a bit, I also recently blogged about a bug in the "buffer" project that was caused by Babel re-interpreting the code as strict, even though it was not distributed in strict mode. This is another great example of a "last mile" bug that ended up being something the library author needed to deal with.

Of course, in this case, the library author could have just said "not my bug." But then the Babel authors might have (quite reasonably) responded, "your library should be strict; we don't support non-strict code," at which point it would become a game of hot potato. And all the while, end-users would be adding messy hacks to their build scripts to work around the issue.

Lessons from the browser world

This story should sound familiar. We've been here before. JavaScript developers have always had to struggle with a hostile and unpredictable environment, which required us to write robust code that could anticipate bugs and implementation differences. Of course, I'm talking about the browser.

Now, I won't argue that writing code for Browserify, Webpack, Babel, and friends is nearly as difficult as writing code for a half-dozen browsers, but it's definitely getting there. So in my opinion, the only reasonable solution is to start bundling code before distribution on npm, which is desirable anyway since we have so many nice tools for transpiling future JavaScript to browser-safe ES5 (such as Babel itself).

The benefits are clear: we can use Browserify transforms and require('./package.json') and whatever else we want, as long as they're only compile steps that build a bundle to be consumed by Node (and therefore Browserify/Webpack). As a bonus, this allows us to build a smaller and sleeker bundle, which is also faster for the end-user to build, since most of the build steps have already been run in advance.

I believe most JavaScript libraries are already moving in this direction, so this puts PouchDB firmly in that bandwagon. But to be clear: we don't intend to start using all the various experimental ES6/ES7 features in PouchDB. In fact, we are only transforming ES6 modules into CommonJS ; in all other ways, PouchDB is still standard ES5. The reason we're doing this is that transpiled ES5 is often cruftier than hand-written ES5. So for now, we're avoiding it.

The goal of implementing CouchDB in JavaScript has always been an ambitious one. The fact that we've done so in ~45KB is an even more impressive accomplishment. But despite the challenges, we're going to keep the bar high; we even have a CI test that fails if our minified bundle ever exceeds 50KB.

Along those same lines, we also have an open issue to add the ability to create custom builds (e.g. no WebSQL, no map/reduce, no replication, etc.). Presumably this can be made even easier with Rollup's tree-shaking capabilities, but we're still investigating this feature.

Get in touch

As always, please file issues or tell us what you think. And as always, a big thanks to all of our new and existing contributors!