Storing Image Data for offline web application (client-side storage database)

javascript html web-applications indexeddb leaflet
Dr.YSG picture Dr.YSG · Jan 1, 2013 · Viewed 35k times · Source

I have an offline web application using appcaching. I need to provide it about 10MB - 20MB of data that it will save (client-side) consisting mainly of PNG image files. The operation is as follows:

  1. Web application downloads and installs in appcache (uses manifest)
  2. Web app requests from server PNG data files (how? - see alternatives below)
  3. Occasionally web app resyncs with server, and does small partial updates/deletes/additions to PNG database
  4. FYI: Server is a JSON REST server, that can place files in wwwroot for pickup

Here is my current analysis of client-based "databases" that handle binary blob storage

SEE UPDATE at Bottom

  • AppCache (via manifest add all the PNG and then update on demand)

    • CON: any change of a PNG database item will mean complete download of all items in manifest (Really bad news!)

  • WebStorage

  • PhoneGap & SQLLite

    • CON: Sponsor will reject it as a native app requiring certification

  • ZIP file

    • Server creates a zip file, places it in wwwroot, and notifies client
    • user has to manually unzip (At least that is how I see it) and save to client file system
    • Web app uses FileSystem API to reference files
    • CON: ZIP might be too large (zip64?), long time to create
    • CON: Not sure if FileSystem API can always read out of the sandbox (I think so)

  • USB or SD card (back to the stone age....)

    • The user will be local to the server before going offline
    • So we could have him insert a SD card, let the server fill it with PNG files
    • Then the user will plug it into the laptop, tablet
    • Web app will use FileSystem API to read the files
    • CON: Not sure if FileSystem API can always read out of the sandbox (I think so)

  • WebSQL

    • CON: w3c has abandoned it (pretty bad)
    • I might consider a Javascript wrapper that uses IndexedDB and WebSQL as a fall-back

  • FileSystem API

  • IndexedDB

  • LawnChair JavaScript wrapper http://brian.io/lawnchair/

    • PRO: very clean wrapper for IndexedDB, WebSQL or whatever database you have (think polyfill)
    • CON: cannot store binary blobs, only data:uri (base64 encoding) (probably fatal flaw due to cost of de-encoding)

  • IndexedDB JQUERY polyFill https://github.com/axemclion/jquery-indexeddb

    • Parashuram has writtent a nice JQUERY wrapper for the raw IndexedDB interface
    • PRO: greatly simplifies using IndexedDB, I was hoping to add a shim/polyfill for Chrome FileSystemAPI
    • CON: It should handle blobs, but I was unable to get it to work

  • idb.filesystem.js http://ericbidelman.tumblr.com/post/21649963613/idb-filesystem-js-bringing-the-html5-filesystem-api

    • Eric Bidelman @ Google has written a well tested PolyFill the FileSystem API that uses Indexed DB as a fall back
    • PRO: FileSystem API is well suited for storing blobs
    • PRO: works great on FireFox and Chrome
      • PRO: great for synchronizing with cloud based CouchDB
    • CON: no clear why, but it is not working on IE10

  • PouchDB JavaScript Library http://pouchdb.com/

    • great for syncing a CouchDB with a local DB (uses either WebSQL or IndexedDB (not my problem though)
    • CON: NO CONS, PouchDB now supports binary blobs for all recent browsers (IE, Chrome, Firefox, Chrome on mobile, etc.) as well as many older browsers. That was not the case when I first did this post.

NOTE: to see a data:uri encoding of PNG I created an example at: http://jsbin.com/ivefak/1/edit

Desired/Usefull/Uneeded Features

  • No native (EXE, PhoneGap, ObjectiveC, etc) app on client (pure web application)
  • Only needs to run on latest Chrome, FireFox, IE10 for laptops
  • Strongly want same solution for Android Tablet (IOS would be nice too) but only need one browser to work (FF, Chrome, etc.)
  • Fast initial DB population
  • REQUIREMENT: Very fast retrieval of images by web application from storage (DB, file)
  • Not meant for consumers. We can restrict browsers, and ask user to do special setup & tasks, but let's minimize that

IndexedDB Implementations

  • There is an excellent article on how IE,FF,and Chrome internally implement this at: http://www.aaron-powell.com/web/indexeddb-storage
  • In short:
    • IE uses the same database format as Exchange and Active Directory for IndexedDB
    • Firefox is using SQLite so are kind of implementing a NoSQL database in to SQL database
    • Chrome (and WebKit) are using a Key/ Value store which has heritage in BigTable

My Current Results

  • I chose to use an IndexedDB approach (and polyfill with FileSystemAPI for Chrome until they ship blob support)
  • For fetching the tiles, I had a dilemna since the JQUERY folks are kvetching about adding this to AJAX
  • I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax() https://github.com/p-m-p/xhr2-lib
  • Performance for 100MB downloads (IE10 4s, Chrome 6s, FireFox 7s).
  • I could not get any of the IndexedDB wrappers to work for blobs (lawnchair, PouchDB, jquery-indexeddb, etc.)
  • I rolled my own wrapper, and performance is (IE10 2s, Chrome 3s, FireFox 10s)
  • With FF, I assume we are seeing the performance issue of using a relational DB (sqllite) for a non-sql storage
  • NOTE, Chrome has outstanding debug tools (developer tab, resources) for inspecting the state of the IndexedDB.

FINAL Results posted below as answer

Update

PouchDB now supports binary blobs for all recent browsers (IE, Chrome, Firefox, Chrome on mobile, etc.) as well as many older browsers. That was not the case when I first did this post.

Answer

Dr.YSG picture Dr.YSG · Jan 23, 2013

Results Offline blob cache for PNG slippy maps

Testing

  • 171 PNG files (total of 3.2MB)
  • Platforms tested: Chrome v24, FireFox 18, IE 10
  • Should also work with Chrome & FF for Android

Fetch from web server

  • using XHR2 (supported on almost all browsers) for blob download from web server
  • I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax()

Storage

Display

  • I am using Leaflet http://leafletjs.com/ to show the map tiles
  • I used the functional tile layer plugin by Ishmael Smyrnow for fetching the tile layer from the DB
  • I compared the DB-based tiles layer with a purely local (localhost://) storage
  • There is no noticeable difference in performance! between using IndexedDB and local files!

Results

  • Chrome: Fetch (6.551s), Store (8.247s), Total Elapsed Time: (13.714s)
  • FireFox: Fetch (0.422s), Store (31.519s), Total Elapsed Time: (32.836s)
  • IE 10: Fetch (0.668s), Store: (0.896s), Total Elapsed Time: (3.758s)

Related questions

What browsers support HTML5 WebSocket API?

I am going to develop an instant messaging application that runs in the browser. What browsers support the WebSocket API?

Read More
Using form input to access camera and immediately upload photos using web app

I came across this answer which is brilliant: In iPhone iOS6 and from Android ICS onwards, HTML5 has the following tag which allows you to take pictures from your device: <input type="file" accept="image/*" capture="camera"> Capture …

Read More
Javascript; communication between tabs/windows with same origin

I have two windows: window A and window B. window A and window B have same domain window A and window B doesn't have any parent window. Questions: Is it possible for window A to get a reference of window …

Read More