Nolan Lawson

By: Nolan Lawson
Published: 06 June 2016

"Hold the pickles, hold the lettuce; special orders don't upset us."

– Burger King jingle

When you're building a large open-source project, the challenge often becomes less about "how do we add new features?" and more about "how do we let people opt-out of the features they don't want?".

This is especially important in JavaScript projects, where the code is usually shipped to users' browsers, meaning every byte counts. Furthermore, with modern build systems based on npm, it's painful to sit and wait for a large dependency to be npm installed, especially if you're only using a small part of it.

Large dependencies can mean a loading spinner in your user's browser, as well as an npm spinner in your own console.

PouchDB – not so pocket-sized anymore

PouchDB is an ambitious project, and as such it's grown a lot over the years. It started out as a simple way to sync CouchDB to IndexedDB (originally only supporting Firefox Nightly!), whereas now it's used in a variety of contexts:

  • In Node.js using LevelDB
  • In hybrid apps using native SQLite
  • As its own server
  • As an AJAX interface to CouchDB
  • As a layer over WebSQL, LocalStorage, or even in-memory

With each new addition, PouchDB's codebase has steadily swollen to support the new use case. And while we've been pretty good at keeping the core size down (45kB min+gz is hefty for a JavaScript library, but not enormous), users still often ask us these sorts of questions:

  • What if I don't want map/reduce?
  • What if I'm only using the IndexedDB adapter?
  • What if I'm only using PouchDB to talk to CouchDB?

If you're not using any of the dependencies, it's a shame to include them in the bundle that gets shipped to your users. We feel ya.

Also, because PouchDB has some large native dependencies (such as LevelDB), the first npm install can take a very long time, and it can even fail on certain operating systems (Windows, looking at you). This is especially bothersome when you consider that those dependencies are only required if you're using PouchDB in Node.js. (And even then, you could just use the in-memory adapter instead.)

So unfortunately, folks who are only using PouchDB in the browser still have to pay the extra cost of installing LevelDB. And ditto for SQLite, even though it's only used for the optional node-websql adapter! This is a less-than-ideal scenario.

Breaking up is hard to do

With PouchDB 5.4.0, though, we've made a huge architectural change to support users who want a slimmer or more customized PouchDB experience. Starting with this release, many of the core modules that were previously internal to PouchDB can now be npm installed and configured separately.

By adopting this pattern, PouchDB is joining the likes of Babel, Lodash, Ember, and other projects that have been unbundled for quite some time. However, even though many JavaScripters now take this kind of unbundling for granted, it's still very difficult for library authors to implement. npm itself does not come with built-in tools to manage multi-package projects, and even npm link can quickly break down if you're trying to run a large test suite with multiple projects across multiple repos.

Luckily, the Babel authors have risen to the occasion and created Lerna, which is the tool they use to manage Babel's numerous sub-packages. Lerna allows JavaScript library authors to define multiple packages for a single repo, and then link and test them all as one. In other words, it's a tool to manage monorepos.

As it turns out, this monorepo strategy is perfect for PouchDB. Logically, we have many small interdependent components (such as the replication module, the map/reduce module, or the IndexedDB adapter), but we prefer to test them all in one large test suite, because a change in one module can have cascading effects in another module. Breaking everything up into many tiny repos, which would be tested, versioned, and developed independently, is just not practical for us.

Using Lerna, though, we can keep all of PouchDB's codebase in one repo, and still release it as separate packages. There are some tricky bits – such as how to manage source code versus bundled code, and how to measure code coverage – but in those cases Rollup remains a great tool to allow us to de-bundle and re-bundle at will. In particular, Rollup allows us to build a "debug" version of PouchDB with many more endpoints exposed, which we then test as a single index.js for measuring code coverage. Then, we are still free to ship a "production" version of PouchDB with those endpoints disabled.

With this release, we are also committing to the jsnext:main standard as promoted by Rollup, meaning that all of our packages (including pouchdb itself) now ship with both ES modules and CommonJS, and you are free to use whichever one you want. In general, the ES modules will tend to give you smaller bundle sizes (assuming you are using Rollup, SystemJS, or Webpack 2 as your bundler), whereas the CommonJS bundle is more appropriate for backwards compatibility. (Neither release makes use of ES6 features aside from ES modules, so you do not need a transpiler like Babel or Bublé).

Cool, how does it work?

PouchDB now ships with a few presets, which are a good demonstration of how the new plugin feature works. Here's pouchdb-browser (as CommonJS), which is the "browser" version of PouchDB:

var PouchDB = require('pouchdb-core');

PouchDB.plugin(require('pouchdb-adapter-idb'))
  .plugin(require('pouchdb-adapter-websql'))
  .plugin(require('pouchdb-adapter-http'))
  .plugin(require('pouchdb-mapreduce'))
  .plugin(require('pouchdb-replication'));

module.exports = PouchDB;

As you can see, pouchdb-browser is just an extension of pouchdb-core, with some key modules added in: the IndexedDB adapter, the WebSQL adapter, the HTTP adapter, map/reduce, and replication. If you're not using any of these features, you can just delete that plugin and enjoy the file savings.

Additionally, you'll notice that pouchdb-browser does not depend on LevelDB at all. Instead, that lives in pouchdb-node, a.k.a. the "Node" version of PouchDB:

var PouchDB = require('pouchdb-core');

PouchDB.plugin(require('pouchdb-adapter-leveldb'))
  .plugin(require('pouchdb-adapter-http'))
  .plugin(require('pouchdb-mapreduce'))
  .plugin(require('pouchdb-replication'));

module.exports = PouchDB;

The only difference here is that pouchdb-browser uses IndexedDB and WebSQL, whereas pouchdb-node uses LevelDB. Hence, LevelDB is not installed if you do npm install pouchdb-browser.

Also, if you'd like to use a custom adapter, those are now shipped separately. So if you want to use PouchDB in a purely in-memory mode, you can do:

var PouchDB = require('pouchdb-core')
  .plugin(require('pouchdb-adapter-memory'));

Or if you're only using PouchDB to talk to CouchDB:

var PouchDB = require('pouchdb-core')
  .plugin(require('pouchdb-adapter-http'));

This applies to all of the plugins we previously shipped in the "extras" API. Any of these can now be mixed and matched as desired:

  • pouchdb-adapter-fruitdown
  • pouchdb-adapter-localstorage
  • pouchdb-adapter-memory
  • pouchdb-adapter-node-websql

Note that the plugin order of the non-HTTP adapters matters. So for instance, if you wanted to fall back from IndexedDB to WebSQL to LocalStorage to in-memory, you would do:

var PouchDB = require('pouchdb-core')
  .plugin(require('pouchdb-adapter-idb'))
  .plugin(require('pouchdb-adapter-websql'))
  .plugin(require('pouchdb-adapter-localstorage'))
  .plugin(require('pouchdb-adapter-memory'));

This also opens up the possibility of more easily including custom third-party adapters, such as worker-pouch and socket-pouch.

How many bytes can it save?

Here is a breakdown of the bundle size of various possible PouchDB configurations, using Browserify, bundle-collapser, and Uglify.

Preset Browserify + Uglify Browserify + Uglify + Gzip
pouchdb 140.06 kB 45.15 kB
pouchdb-browser 148.3 kB 47.54 kB
pouchdb-http 64.29 kB 21.54 kB
PouchDB, IndexedDB only 83.47 kB 27.79 kB
PouchDB, WebSQL only 85.15 kB 28.16 kB
PouchDB, replication only 76.28 kB 25.26 kB
PouchDB, in-memory only 350.74 kB 100.81 kB

Note that pouchdb-browser is slightly bigger than pouchdb, because pouchdb is heavily optimized with Rollup before publishing, whereas pouchdb-browser is not. Also, some optional plugins (like pouchdb-adapter-memory) are still quite large. In the future, we may consider using Rollup more aggressively, or trying to reduce the size of pouchdb-core. Or we might just wait for the ecosystem to switch to tools that do automatic tree-shaking for ES modules, such as Rollup or Webpack 2. (If you use such a tool, and it supports jsnext:main, then you should already be able to get smaller bundle sizes.)

In the meantime, though, a major benefit of PouchDB unbundling is still in reduced npm install times, as well as avoiding compatibility issues with native dependencies like leveldown and sqlite3.

Backwards compatibility?

Yes indeed, as suggested by the 5.4.0 release version, this is a fully backwards-compatible change. The pouchdb module will still act exactly as it did before, including the dist/ and extras/ APIs. Essentially, a npm install pouchdb will install the kitchen sink. We believe this is the best approach, because it provides a batteries-included experience for beginners, without having to burden them with understanding a vast plugin API.

However, the extras/ API is deprecated, and will be removed in a later version. Furthermore, the PouchDB.utils, PouchDB.ajax, and PouchDB.Errors APIs, which have been undocumented and unrecommended for years, are now removed. Plugin authors should update their code to use the new sub-packages.

That said, there is an important note about the new sub-packages: all of them are pegged to PouchDB's version number, meaning that when PouchDB cuts a new release, each of the sub-packages will also be released with the same version number. Furthermore, not all of these sub-packages follow semantic versioning, because conceptually they are internal modules, which are only exposed on npm as an implementation detail. The APIs for these packages may change at any time, so you should use exact versioning if you decide to use them directly. These packages will be clearly marked as non-semver in their READMEs.

In the new system, the packages that serve as "plugins" (as well as pouchdb itself) are considered user-facing and semver-compliant. In particular, these packages follow semver:

  • pouchdb
  • pouchdb-adapter-fruitdown
  • pouchdb-adapter-http
  • pouchdb-adapter-idb
  • pouchdb-adapter-leveldb
  • pouchdb-adapter-localstorage
  • pouchdb-adapter-memory
  • pouchdb-adapter-node-websql
  • pouchdb-adapter-websql
  • pouchdb-browser
  • pouchdb-http
  • pouchdb-mapreduce
  • pouchdb-node
  • pouchdb-replication

Other modules, such as pouchdb-ajax, pouchdb-checkpointer, and pouchdb-promise, may be freely used, but they make no guarantees as to semver. As we did with the extras/ API, all of these are used at your own risk.

Scoped packages?

We considered using scoped packages (e.g. @pouchdb/foo) as opposed to prefixed packages (e.g. pouchdb-foo) for the new monorepo architecture. In fact, we even asked the community to vote on the issue, and as of today the vote stands at 27 for scopes and 24 for prefixes.

However, the core team ultimately decided that it's too early to hop onto the scoping bandwagon. There are still plenty of thorny issues with scopes, such as the fact that npm organizations are not free for open-source yet, meaning we would have to register a single pouchdb account and share it among all the contributors. (See the voting issue for discussion about other issues with scopes.) We look forward to these issues being resolved as scoping matures, at which point we may consider moving over to scoped packages.

For the foreseeable future, though, PouchDB's sub-packages will sport names like pouchdb-browser, pouchdb-ajax, and pouchdb-replication. Even if it's a bit old-fashioned, we hope that PouchDB's users will experience fewer surprises and confusion with this more traditional naming convention. And considering the popularity of ember-*, babel-*, and react-* packages, it's safe to say PouchDB is in good company.

Build-a-bear, build-a-pouch

Since there are dozens of combinations for building your own PouchDB, we encourage the community to mix-and-match, and to publish any presets that they find useful. As of 5.4.0, there are three presets that PouchDB will officially support as first-party packages:

  • pouchdb-browser – the "browser" version of PouchDB
  • pouchdb-node – the "Node" version of PouchDB
  • pouchdb-http – PouchDB as an HTTP interface to CouchDB (sans map/reduce)

In the future, we also plan to move some packages that have previously existed in third-party repos (including pouchdb-server, pouchdb-express-router, and pouchdb-find) into the main PouchDB repository, so that they can be tested, versioned, and released at the same time as PouchDB.

So please, try out the new customizable PouchDB, and let us know what you think! And if there are any bugs, there's only one repo you need to worry about, so please direct your bugs to the PouchDB issues page. Happy unbundling!