Architecture

CloudWall is javascript-only noBackend app framework, built on top of several jQuery plugins and JS libraries glued together. 

Like any complex system, CloudWall is a layered cake.
Click image to zoom.

FaaS. When you start cloudwall.me, you load framework, libs, UI plugins and several system apps. Site cloudwall.me does not store your data or your custom apps in any way – so it is somehow ‘framework as a service’. Abbrevs with ‘aaS’ are trendy – so why not?

Storage. All your docs and user apps are stored inside your browser, in IndexedDB or WebSQL storage, pumped up with PouchDB lib to be doc-oriented. 

Internal browser storage can have several virtual DBs, each with unique sync settings and set of apps. One local DB has reserved name cw and holds system docs, your profile, replication settings and passwords, cryptokeys and so on.

Clouds. Your and other users’ local DBs can be continuously synced with external clouds using CouchDB replication protocol. Cloud, shared and owned by a small group of users is denoted as private cloud.

Profile sync. To sync several devices with single user account you need all devices’ system DBs to be synced with a single CouchDB bucket – so in this case you need an external CouchDB instance running. Browsers by themselves can not communicate with each other.

Offline. CloudWall once opened in a browser tab can work offline. All libs are loaded during system start, all docs are local. Cloud replication is async by it’s nature and restarts when browser goes online.

Apps. CloudWall apps are jQuery.my manifests. They are structured JSON objects so fit perfectly with storage. There are several system tools for authoring, testing and deploying apps – right in browser tab. jQuery.my learning curve is very short and there are several example apps installed on first system launch.

Every DB can have its own set of apps, moreover app can have different set of components on different machines – it allows quick distributed app development. 

Security. To ensure sensitive data can’t be read even on occasional replication, CloudWall has built-in crypto lib. It forces encryption of all sensitive user profile docs on save, and can be optionally applied to encrypt some user docs.

To ensure external apps can not run without notice, system asks user trust confirmation if app was updated since last run.

Runtime

Like any complex system CloudWall is a layered cake. Top layer is what you see on your screen – user interface. Bottom layers – persistent in-browser DB and network clouds.

To avoid excessive documentation, most of CloudWall components are not in any way augmented or decorated – original APIs kept where it‘s safe and possible. 

Framework and debug

All CloudWall-specific code occupies global variable cw. Full list of methods visible inside depends on caller – user apps see a safer subset. Also any app executed has two objects, this.db and this.app. They accumulate all app-specific CloudWall runtime methods and properties.

Debug mode allow log to be printed to console and tighten restrictions. For example you can see global object PouchDB in a debug mode. In standard mode this object is deleted after system start, same for several other unsafe objects.

To run CloudWall in debug mode press key Tab, not Sign in button after you entered PIN on start.

DOM, controls and plugins

CloudWall utilizes HTML5 DOM features heavily. Most UI-related calls are made through jQuery and many HTML DOM nodes has several data properties mounted on them. 

CloudWall is UI-plugins friendly, unlike many other systems it requires no proxy shims to use plugins extending DOM features. CloudWall has 0.5Mb+ (180 kb gzipped) bunch of plugins mounted at start, they all are well documented and accessible using their native syntax.

Full list of plugins

jQuery.my lib

All controls you see on your screen are managed and linked with underlying data by jQuery.my library. Each CloudWall app is $.my manifest or hierarchy of manifests.

$.my manifest structure is pure JSON with javascript code incorporated in style of CouchDB puts functions in docs. Manifest is self-contained in general and need no external resources (code, images or CSS files) to run.

How to author an app

App runtime API

CloudWall slightly extends $.my out-of-box features. Every jQuery.my app is augmented with this.db and this.app objects visible at runtime. Their methods allow app to access storage and runtime CloudWall API functions. Also they allow app to stay in touch with DB and system state updates – both has method watch that returns promise with progress updater mounted.

Runtime extends built-in $.my code snippets mechanics and allow apps to use each other’s named components. System components are accessed same way – by string reference. 

Since each component is single javascript object (manifest) it can be mutated before use. It’s normal to obtain system confirm dialog manifest with this.db.form("cw.Sys.Confirm"), add some controls inside and then popup modal with result.

App runtime API reference

Slot manager

jQuery.my apps are started, get active or hidden, and then die being under control of Slot manager. It drives all that processes and supervises apps during runtime.

On app start request – which is always URL of #dbname/appname/params structure – slot manager parses switch request, check all params, starts app and renders it offscreen. If manifest started up properly, active slot is hidden and newborn app get visible.

Each app has start timeframe limit defined in app params by author. If start is not completed in a timeframe, slot is killed.

If document(s) with app components updated since last run, slot manager ask user to confirm trust before app start. 

Slot manager tracks app close requests and if app is and editor and has unsaved doc – close request get paused until user intervention.

Slot manager tracks and reflects app state to browser URL bar and tab title. 

Slot manager API reference

Components cache

Manifests must be cached and kept up to date on updates received. To speed up component access manifests are cached unfolded – it mean all JSON representation of functions and regexps are pre-evaluated into native functions and regexps. It also ensures context of these functions is properly restricted.

Each DB has independent component cache – so you can use different versions of apps in sibling DBs. There is one important exception: system components and those from system DB are visible in all other DBs and have priority over locals unless you are in debug mode.

Form cache API reference

Crypto component

The only component that see your settings doc decrypted. The only component that asks your PIN during start and guarantees that PIN never persist unencrypted in javascript-addressable space.

Crypto component is built on top of modified Gibberish AES – Open-SSL compatible cryptolib. It’s FIPS-certified and reliable, and extensions make it robust. Try twice in browser tab console cw.lib.enc({a:1,b:function(){}},"pwd") . Results must differ.

Note that cryptolib can encode and decode more or less full-featured javascript objects. Scope is lost, but type is preserved: cw.lib.dec(cw.lib.enc({a:/./},"pwd"),"pwd") 

Crypto API reference

Storage

PouchDB is used as primary low-level lib to create/read/write/sync DBs and docs. It is heavily augmented to ensure all docs and apps are cached during operations. 

Augmentation also ensures that system docs stay intact, and that docs are decrypted/encrypted if necessary during read/write.

Storage supports key range and per-document read, map-reduce queries, versioning and filtered master-master replication.

CloudWall extensions of PouchDB provide queries to memcache, excellent attachment manipulation mechanics and typed update notifiers, which sneeze while app-subscriber is inactive.

Storage API reference

Queries and ‘cache all’ strategy

PouchDB has db.query(mapFunctionName) method, that behaves very much the same as specimen in CouchDB – with some slight differences regarding UTF8 sorting and collation.

CloudWall has two proxies for it, but does not use it internally right now. Strategy ‘No complex DB queries’ is used where possible. 

Most docs in DBs has simple and understandable keys, which can be read in big chunks by key range, without map-reduce queries. Docs than cached in RAM (without attaches certainly) and all ops are made over cached versions.

Average JSON doc is few kilobytes long, so memcache of 5–10 thousands of docs is ok. Normal capacity of bulk DB read is 20+Mbytes per second, reading docs in chunks of 100-200 docs impose no distinguishable UI freeze. 

It’s suitable strategy – map-reduce iterators over in-memory collections are extremely, rocket fast. Even complex filter functions give milliseconds delays on hundreds and thousands of docs.

Doc _id subspaces

To make chunk reads by key range more reliable doc _id’s are generated in continuous subspaces. Most noticeable are three of them.

General doc id is doc creation timestamp of 10ms granularity plus random 8-char string. Doc ids are 20 chars long and can be obtained calling cw.lib.uuid()

So fetching general docs by key range returns time sorted doc list.

Typed doc id is system format for some classes of documents, that require typed fetching by keyrange. DB users for example have ids like user-somename-rg8jky67.

Component doc id is like cw-Post-Editor-34ef. All these ids occupy cw- to cw.  id range and always point to docs with a manifest inside.

Cloud sync 

Local DBs can be synced with external CouchDB instances. Replication is bi- or unidirectional. Sync succeeds if remote CouchDB has CORS settings properly configured. CORS settings are supported by CouchDB 1.5+, previous versions can not be in sync with CloudWall.

There are some CouchDB access rights configurations, which tend to produce errors and excessive traffic. They are described with fixes.

CloudWall syncs both in continuous and interval mode. You can set sync intervals for each DB.

As in any versioned storage, replication conflicts are pain. There are several conflict resolution tools, but each demands more or less end user attention. By default, system stores all conflicting revisions of a doc to ensure no data loss in any way.

When CloudWall creates local DB, it writes user signature doc – and this signature replicated if DB gets synced. This feature ensures all users, that can write to DB, know about each other existence.

Security considerations

Open and extendable CloudWall architecture certainly imposes many security questions. 

DOM is the problem

If one app runtime can access other app runtime through DOM – how can we make system safe? Iframes are not an answer, several dozens of them will stall any system. 

If app can delete any doc in DB, ain’t it potentially harmful? If app can even modify other apps – ain’t it a security hell?

Run it if you trust

In fact the only answer is to ask you if you trust this or that app each time you received an update. It’s like with people – you can not be sure if stranger ain’t gonna rob you, but you trust circumstances and somehow social strata you are in and one’s inhabitants.

Securing passwords and keys

The only entities that are first-class protected are cryptokeys, PIN and external DB passwords if any. To ensure no app can gain settings decrypted there is no built-in interface to change PIN or modify keys, you need to open reg.html. This page have different than main system set of methods and can modify protected settings.