Clean master commit

2026-06-10 12:28:34 +03:00 · 2018-10-31 19:35:15 +00:00
commit 2cbfbc33ef
24 changed files with 5400 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,10 @@
+/dist
+/logs
+/npm-debug.log
+/node_modules
+/temp
+.DS_Store
+*.swp
+
+*service-account-key\.json
+src/config.js
--- a/20
+++ b/20
@@ -0,0 +1,20 @@
+FROM mhart/alpine-node:10.11
+
+LABEL authors="Lachlan Kermode <lk@forensic-architecture.org>"
+
+# Install app dependencies
+COPY package.json /www/package.json
+RUN cd /www; yarn
+
+# Copy app source
+COPY . /www
+WORKDIR /www
+RUN yarn build
+RUN mkdir -p temp
+
+# set your port
+ENV PORT 8080
+EXPOSE  8080
+
+# start command as per package.json
+CMD ["yarn", "start"]
--- a/20
+++ b/20
@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Jason Miller
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
--- a/README.md
+++ b/README.md
@@ -0,0 +1,148 @@
+<h1 align="center">
+  Datasheet Server
+</h1>
+
+<p align="center">
+  <strong>Turn spreadsheet data into a structured, dynamic API. </strong><br>
+</p>
+<!-- <p align="center">
+  <a href="https://github.com/gatsbyjs/gatsby/blob/master/LICENSE">
+    <img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="Gatsby is released under the MIT license." />
+  </a>
+  <a href="#configru">
+    <img src="https://circleci.com/gh/gatsbyjs/gatsby.svg?style=shield" alt="Current CircleCI build status." />
+  </a>
+  <a href="https://www.npmjs.org/package/gatsby">
+    <img src="https://img.shields.io/npm/v/gatsby.svg" alt="Current npm package version." />
+  </a>
+  <a href="https://npmcharts.com/compare/gatsby?minimal=true">
+    <img src="https://img.shields.io/npm/dm/gatsby.svg" alt="Downloads per month on npm." />
+  </a>
+  <a href="https://gatsbyjs.org/docs/how-to-submit-a-pr/">
+    <img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" alt="PRs welcome!" />
+  </a>
+</p> -->
+
+<h3 align="center">
+  <a href="#overview">Overview</a>
+  <span> · </span>
+  <a href="#configuration">Configuration</a>
+  <span> · </span>
+  <a href="#quickstart">Quickstart</a>
+</h3>
+
+Datasheet server makes resources from a spreadsheet available as a structured API.
+
+- **Manage structured data without developers**. Allows anyone to dynamically manage data, while simultaneously making this data available in a reliably structured format for frontend interfaces and other programmatic applications.
+- **Designed for a dynamic workflow**. References data in a spreadsheet source as a ground truth, but adds a layer of indirection that keep API routes from breaking when changes are made.
+- **Customisable data transformation**. Easily create new blueprints to specify the API structure that should be presented from source data.
+- **Extensible architecture**. Currently supports Google Sheet as a source and a REST-like query language, but structured modularly with an intention to support other sources and query languages.
+
+## [Overview](#overview)
+Datasheet server is a Node server developed at [Forensic Architecture](https://forensic-architecture.org) to make data that is being dynamically modified by researchers concurrently consumable for programmatic applications as an API. We use spreadsheets extensively to keep track of information during [our investigations](http://forensic-architecture.org/cases), and we often want to make this data available via structured queries to make it available in a frontend web application, a game engine, or another use case.
+
+Querying data directly from spreadsheets is brittle, as it relies on the maintenance of a rigid structure in the sheets at all times. By putting Datasheet Server as a proxy that sits in between source sheets and their consumers, it is possible to dynamically modify sheets without breaking applications. A data admin can then use Datasheet Server to ensure that applications always receive eligible data, without foregoing the spreadsheets as sources of truth.  
+
+### Design Concepts
+The codebase currently only supports Google Sheets as a source, and a REST-like format as a query language. It is designed, however, with extensibility in mind.
+
+**Sources**     
+A source represents a sheet-like collection of data, such as a Google Sheet. A source has one or more **tabs**, each of which contains a 2-dimensional grids of cells. Each cell contains a body of text (a string).
+
+**Resources**     
+The data from sources are made available as resources, which are structured blocks of data that are granularly accessible through a query language. Resources are the outfacing aspect of Datasheet Server, and represent the only kind of data that can be queried by applications. Each resource is configured with one or more **query languages**. (Currently only a REST-like query language supported.)
+
+**Blueprints**       
+Blueprints are a data structure that represent the way that infromation from **sources** are to be turned into **resources**. For each tab in a source, there is a corresponding Blueprint. Blueprints are created through a [blueprinter function](/src/blueprinters) invoked on the raw data from a source tab.
+
+Blueprints are JSON objects. There have two forms:
+
+1. _desaturated_ -- describes the resources and query languages available on data from a source tab.
+2. _saturated_ --  both describes resources available on data from a source etab, and contains that data.
+
+A desaturated Blueprint can be saturated by retrieving its data from the server's **model layer**, which stores tab data from sources.
+
+A JSON catalogue of the available blueprints (desaturated) in a server is available at `/api/blueprints`.
+
+## [Configuration](#configuration)
+Copy the [example.config.js](/src/example.config.js) in the [src](/src) directory into a file named 'config.js'. Modify the options in this file accordingly:
+
+| Option  | Description | Type |
+| ------- | ----------- | ---- |
+| port | The port at which the server will make data available.  | integer |
+| googleSheets  | The configuration object for [Google Sheet](https://www.google.co.uk/sheets/about/) data sources. See the [Sources](#source-google-sheets) section below. | object |
+
+#### [Sources](#sources)
+###### [Google Sheets](#source-google-sheets)
+In order to make the data from a Sheet accessible to the server, you need to [create a service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts). Once created, give the service account email access to each Sheet from which you want to serve data. ('View Only' access is sufficient, as the server never modifies data.)
+
+| Option | Description | Type |
+| ------ | ----------- | ---- |
+| email | The email address of the service account. This is available in the downloadable service account JSON in the `client_email` field. | string |
+| privateKey | The private key associated with the service account. This is available in the downloadable service account JSON in the `private_key` field. | string |
+| sheets | A list of objects, one for each sheet that is being used as a source. Each sheet object has a `name` (String), an `id` (String), and a `tabs` (object) field, which are explained below. | object |
+
+Each Google Sheet being used as a as source requires a corresponding object in `sheets`. The object should be structured as follows:
+
+| Option | Description | Type |
+| ------ | ----------- | ---- |
+| name | Used to refer to data served from this source | string |
+| id | The ID of the sheet in Google. (You can find it in the address bar when the Sheet is open in a browser. It is the string that follows 'spreadsheets/d/'). | string |
+| tabs | An object that maps each tab in the source to a Blueprinter. All of the Blueprinters in the [blueprinters folder](/lib/blueprinters) are available through a single import as at the top of [example.config.js](/src/example.config.js). <br> To correctly associate a Blueprinter, the object key needs to be _the tab name with all lowercase letters, and spaces replaced by a '-'_. For example, if the tab name in Google Sheets is 'Info About SHEEP', the object key should be 'info-about-sheep'. <br> The value should be the Blueprinter function that you want to use for the data in that tab. See the example of a configuration object below. <br>TODO: no Blueprinter is used by default. | object |
+
+###### Example Configuration Object
+```js
+import BP from './lib/blueprinters'
+
+export default {
+  port: 4040,
+  googleSheets: {
+    email: 'project-name@reliable-baptist-23338.iam.gserviceaccount.com',
+    privateKey: 'SOME_PRIVATE_KEY',
+    sheets: [
+      {
+        name: 'example',
+        id: '1s-vfBR8Uy-B-TLO_C5Ozw4z-L0E3hdP8ohMV761ouRI',
+        tabs: {
+          'objects': BP.byRow,
+        }
+      },
+    ]
+  }
+}
+
+```
+
+
+## [Quickstart](#quickstart)
+Clone the repository to your local:
+```
+git clone https://www.github.com/forensic-architecture/datasheet-server
+```
+### Run with Docker
+To create a new instance of the server with [Docker](https://www.docker.com/) installed, clone the repository, create a `config.js`, and build the image:
+```sh
+docker build -t datasheet-server .
+```
+You can then run the container and make available the relevant port (`4040` by default):
+```sh
+docker run -d -p 4040:4040 datasheet-server
+```
+If running on a cloud server, you'll probably need to zero out the host IP:
+```sh
+docker run -d -p 0.0.0.0:4040:4040 datasheet-server
+```
+
+### Run locally
+Install dependencies:
+```sh
+yarn # npm install
+```
+And run development server:
+```sh
+yarn dev # npm run dev
+```
+License
+-------
+
+MIT
--- a/package.json
+++ b/package.json
@@ -0,0 +1,75 @@
+{
+  "name": "grenfell-server",
+  "version": "0.3.0",
+  "description": "Starter project for an ES6 RESTful Express API",
+  "main": "dist",
+  "scripts": {
+    "dev": "nodemon -w src --exec \"babel-node src\"",
+    "build": "npx babel src -d dist",
+    "start": "node dist",
+    "lint": "eslint src",
+    "test": "ava --watch"
+  },
+  "eslintConfig": {
+    "extends": "eslint:recommended",
+    "parserOptions": {
+      "ecmaVersion": 7,
+      "sourceType": "module"
+    },
+    "env": {
+      "node": true
+    },
+    "rules": {
+      "no-console": 0,
+      "no-unused-vars": 1
+    }
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/developit/express-es6-rest-api.git"
+  },
+  "author": "Jason Miller <jason@developit.ca>",
+  "license": "MIT",
+  "dependencies": {
+    "body-parser": "^1.13.3",
+    "compression": "^1.5.2",
+    "cors": "^2.7.1",
+    "express": "^4.13.3",
+    "express-graphql": "^0.6.12",
+    "googleapis": "^32.0.0",
+    "graphql": "^0.13.2",
+    "morgan": "^1.8.0",
+    "mz": "^2.7.0",
+    "node-fetch": "^2.2.0",
+    "object-hash": "^1.3.0",
+    "ramda": "^0.25.0",
+    "resource-router-middleware": "^0.6.0"
+  },
+  "devDependencies": {
+    "@babel/cli": "^7.1.2",
+    "@babel/core": "^7.1.2",
+    "@babel/node": "^7.0.0",
+    "@babel/preset-env": "^7.1.0",
+    "@babel/register": "^7.0.0",
+    "ava": "1.0.0-beta.8",
+    "eslint": "^3.1.1",
+    "nodemon": "^1.9.2"
+  },
+  "babel": {
+    "presets": [
+      "@babel/preset-env"
+    ]
+  },
+  "ava": {
+    "files": [
+      "test/**/*.js"
+    ],
+    "require": [
+      "@babel/register"
+    ]
+  },
+  "bugs": {
+    "url": "https://github.com/developit/express-es6-rest-api/issues"
+  },
+  "homepage": "https://github.com/developit/express-es6-rest-api#readme"
+}
--- a/src/api/index.js
+++ b/src/api/index.js
@@ -0,0 +1,57 @@
+import {version} from "../../package.json";
+import {Router} from "express";
+import {idxSearcher} from "../lib/util";
+
+export default ({config, controller}) => {
+  let api = Router();
+
+  api.get("/", (req, res) => {
+    res.json({
+      version
+    });
+  });
+
+  api.get("/blueprints", (req, res) => {
+    res.json(controller.blueprints());
+  });
+
+  api.get("/:source/:tab/:resource/:frag", (req, res) => {
+    const {source, tab, resource, frag} = req.params;
+    controller
+      .retrieveFrag(source, tab, resource, frag)
+      .then(data => res.json(data))
+      .catch(err =>
+        res.json({
+          error: err.message
+        })
+      );
+  });
+
+  api.get("/:source/:tab/:resource", (req, res) => {
+    controller
+      .retrieve(req.params.source, req.params.tab, req.params.resource)
+      .then(data => res.json(data))
+      .catch(err =>
+        res.json({
+          error: err.message
+        })
+      );
+  });
+
+  api.get("/update", (req, res) => {
+    controller
+      .update()
+      .then(msg =>
+        res.json({
+          success: msg
+        })
+      )
+      .catch(err =>
+        res.json({
+          error: err.message
+        })
+      );
+  });
+
+  return api;
+};
--- a/src/blueprinters/byColumn.js
+++ b/src/blueprinters/byColumn.js
@@ -0,0 +1,34 @@
+import {defaultBlueprint, defaultRoute} from "../lib/blueprinters";
+
+/**
+ * byColumn - generate a Blueprint from a data source by column. Each column
+ * name is a resource, and all values in that column are the resource items.
+ *
+ * @param  {type} data - list of lists representing sheet data.
+ * @return {type} Blueprint
+ * generated.
+ */
+export default function byColumn(tabName, sourceName, sourceId, data) {
+  // Define Blueprint props
+  const bp = R.clone(defaultBlueprint);
+  bp.source = {
+    name: sourceName,
+    id: sourceId
+  };
+  bp.name = tabName;
+
+  // column names define routes
+  const labels = data[0];
+  labels.forEach(label => {
+    bp.routes[label] = R.clone(defaultRoute);
+  });
+
+  // remaining rows as data
+  data.forEach((row, idx) => {
+    if (idx == 0) return;
+    labels.forEach((label, idx) => {
+      bp.routes[label].data.push(row[idx]);
+    });
+  });
+  return bp;
+}
--- a/src/blueprinters/byGroup.js
+++ b/src/blueprinters/byGroup.js
@@ -0,0 +1,56 @@
+import R from "ramda";
+import {fmtObj, idxSearcher} from "../lib/util";
+import {defaultBlueprint, defaultRoute} from "../lib/blueprinters";
+
+/**
+ * byGroup - generate a Blueprint from a data source grouped by a column called 'group'
+ * The resource name defaults to 'groups', or a custom resource name can be passed.
+ * Each resource item is an object with values labelled according to column
+ * names. Items are inserted in the data list at idx = id.
+ *
+ * @param  {type} data         list of lists representing sheet data.
+ * @param  {type} label="groups"  name of resource in blueprint.
+ * @param  {type} name=""      name of blueprint.
+ * @return {type} Blueprint
+ */
+export default function byGroup(
+  tabName,
+  sourceName,
+  sourceId,
+  data,
+  label = "groups"
+) {
+  // Define Blueprint
+  const bp = R.clone(defaultBlueprint);
+  bp.source = {
+    name: sourceName,
+    id: sourceId
+  };
+  bp.name = tabName;
+
+  // Column names define routes
+  const itemLabels = data[0];
+  const fmt = fmtObj(itemLabels);
+  bp.routes[label] = R.clone(defaultRoute);
+  bp.routes[label].data = [];
+
+  const dataGroups = {};
+
+  data.forEach((row, idx) => {
+    if (idx == 0) return;
+    const group = fmt(row).group;
+    if (!dataGroups[group]) {
+      dataGroups[group] = [fmt(row)];
+    } else {
+      dataGroups[group].push(fmt(row));
+    }
+  });
+  Object.keys(dataGroups).forEach(groupKey => {
+    bp.routes[label].data.push({
+      group: groupKey,
+      group_label: dataGroups[groupKey][0].group_label,
+      data: dataGroups[groupKey]
+    });
+  });
+  return bp;
+}
--- a/src/blueprinters/byId.js
+++ b/src/blueprinters/byId.js
@@ -0,0 +1,42 @@
+import R from "ramda";
+import {fmtObj, idxSearcher} from "../lib/util";
+import {defaultBlueprint, defaultRoute} from "../lib/blueprinters";
+
+/**
+ * byId - generate a Blueprint from a data source by id, which is an integer.
+ * The resource name defaults to 'ids', or a custom resource name can be passed.
+ * Each resource item is an object with values labelled according to column
+ * names. Items are inserted in the data list at idx = id.
+ *
+ * @param  {type} data         list of lists representing sheet data.
+ * @param  {type} label="ids"  name of resource in blueprint.
+ * @param  {type} name=""      name of blueprint.
+ * @return {type} Blueprint
+ */
+export default function byId(
+  tabName,
+  sourceName,
+  sourceId,
+  data,
+  label = "ids"
+) {
+  // Define Blueprint
+  const bp = R.clone(defaultBlueprint);
+  bp.source = {
+    name: sourceName,
+    id: sourceId
+  };
+  bp.name = tabName;
+
+  // Column names define routes
+  const itemLabels = data[0];
+  const fmt = fmtObj(itemLabels);
+  bp.routes[label] = R.clone(defaultRoute);
+  bp.routes[label].data = [];
+
+  data.forEach((row, idx) => {
+    if (idx == 0) return;
+    bp.routes[label].data[fmt(row).id] = fmt(row);
+  });
+  return bp;
+}
--- a/src/blueprinters/byRow.js
+++ b/src/blueprinters/byRow.js
@@ -0,0 +1,41 @@
+import R from "ramda";
+import {fmtObj, idxSearcher} from "../lib/util";
+import {defaultBlueprint, defaultRoute} from "../lib/blueprinters";
+
+/**
+ * byRow - generate a Blueprint from a data source by row. The resource name
+ * defaults to 'rows', or a custom resource name can be passed. Each resource
+ * item is an object with values labelled according to column names.
+ *
+ * @param  {type} data         list of lists representing sheet data.
+ * @param  {type} label="rows" name of resource in blueprint.
+ * @param  {type} name=""      name of blueprint.
+ * @return {type} Blueprint
+ */
+export default function byRow(
+  tabName,
+  sourceName,
+  sourceId,
+  data,
+  label = "rows"
+) {
+  // Define Blueprint
+  const bp = R.clone(defaultBlueprint);
+  bp.source = {
+    name: sourceName,
+    id: sourceId
+  };
+  bp.name = tabName;
+
+  // Column names define routes
+  const itemLabels = data[0];
+  const fmt = fmtObj(itemLabels);
+  bp.routes[label] = R.clone(defaultRoute);
+  bp.routes[label].data = [];
+
+  data.forEach((row, idx) => {
+    if (idx == 0) return;
+    bp.routes[label].data.push(fmt(row));
+  });
+  return bp;
+}
--- a/src/blueprinters/byTree.js
+++ b/src/blueprinters/byTree.js
@@ -0,0 +1,67 @@
+import R from "ramda";
+import {fmtObj, idxSearcher} from "../lib/util";
+import {defaultBlueprint, defaultRoute} from "../lib/blueprinters";
+
+/**
+ * byTree - generate a Blueprint from a data source grouped by a column called 'group'
+ * The resource name defaults to 'groups', or a custom resource name can be passed.
+ * Each resource item is an object with values labelled according to column
+ * names. Items are inserted in the data list at idx = id.
+ *
+ * @param  {type} data         list of lists representing sheet data.
+ * @param  {type} label="groups"  name of resource in blueprint.
+ * @param  {type} name=""      name of blueprint.
+ * @return {type} Blueprint
+ */
+export default function byTree(
+  tabName,
+  sourceName,
+  sourceId,
+  data,
+  label = "tree"
+) {
+  // Define Blueprint
+  const bp = R.clone(defaultBlueprint);
+  bp.source = {
+    name: sourceName,
+    id: sourceId
+  };
+  bp.name = tabName;
+
+  // Column names define routes
+  bp.routes[label] = R.clone(defaultRoute);
+  bp.routes[label].data = {};
+
+  const tree = {
+    key: "tags",
+    children: {}
+  };
+
+  data.forEach(path => {
+    const root = path[0];
+    if (!tree.children[root])
+      tree.children[root] = {
+        key: root,
+        children: {}
+      };
+
+    let depth = 1;
+    let parentNode = tree.children[root];
+
+    while (depth < path.length) {
+      const node = path[depth];
+      if (!parentNode.children[node]) {
+        parentNode.children[node] = {
+          key: node,
+          children: {}
+        };
+      }
+      parentNode = parentNode.children[node];
+
+      depth++;
+    }
+  });
+
+  bp.routes[label].data = tree;
+  return bp;
+}
--- a/src/example.config.js
+++ b/src/example.config.js
@@ -0,0 +1,18 @@
+import BP from "./lib/blueprinters";
+
+export default {
+  port: 4040,
+  googleSheets: {
+    email: "project-name@reliable-baptist-23338.iam.gserviceaccount.com",
+    privateKey: "SOME_PRIVATE_KEY",
+    sheets: [
+      {
+        name: "example",
+        id: "1s-vfBR8Uy-B-TLO_C5Ozw4z-L0E3hdP8ohMV761ouRI",
+        tabs: {
+          objects: BP.byRow
+        }
+      }
+    ]
+  }
+};
--- a/src/index.js
+++ b/src/index.js
@@ -0,0 +1,31 @@
+import http from "http";
+import express from "express";
+import initialize from "./initialize";
+import middleware from "./middleware";
+import api from "./api";
+import config from "./config";
+
+let app = express();
+app.server = http.createServer(app);
+
+initialize(controller => {
+  app.use(
+    middleware({
+      config,
+      controller
+    })
+  );
+  app.use(
+    "/api",
+    api({
+      config,
+      controller
+    })
+  );
+
+  app.server.listen(process.env.PORT || config.port, () => {
+    console.log(`Started on port ${app.server.address().port}`);
+  });
+});
+
+export default app;
--- a/src/initialize.js
+++ b/src/initialize.js
@@ -0,0 +1,46 @@
+import StoreJson from "./models/StoreJson";
+import Fetcher from "./lib/Fetcher";
+import Controller from "./lib/Controller";
+import config from "./config";
+
+const {googleSheets} = config;
+const {sheets, privateKey, email} = googleSheets;
+
+function authenticate(_fetcher) {
+  return _fetcher.fetcher.authenticate(email, privateKey).then(msg => {
+    console.log(msg);
+    return true;
+  });
+}
+
+export default callback => {
+  const fetchers = sheets.map(sheet => {
+    return {
+      name: sheet.name,
+      fetcher: new Fetcher(new StoreJson(), sheet.name, sheet.id, sheet.tabs)
+    };
+  });
+
+  Promise.all(fetchers.map(authenticate))
+    .then(() => {
+      console.log(`===================`);
+      console.log(`grant access to: ${email}`);
+      console.log(`===================`);
+
+      // NB: reformat fetchers as config for controller
+      const config = {};
+      fetchers.forEach(fetcher => {
+        config[fetcher.name] = fetcher.fetcher;
+      });
+      const controller = new Controller(config);
+      callback(controller);
+    })
+    .catch(err => {
+      console.log(err);
+      console.log(
+        `ERROR: the server couldn't connect to all of the sheets you provided. Ensure you have granted access to ${
+          serviceAccount.email
+        } on ALL listed sheets.`
+      );
+    });
+};
--- a/src/lib/Controller.js
+++ b/src/lib/Controller.js
@@ -0,0 +1,54 @@
+/**
+ * Controller
+ *
+ */
+class Controller {
+  constructor(fetchers) {
+    this.fetchers = fetchers;
+  }
+
+  sourceExists(source) {
+    return true;
+    if (Object.keys(this.fetchers).indexOf(source) == -1) return false;
+  }
+
+  blueprints() {
+    return Object.keys(this.fetchers).map(
+      source => this.fetchers[source].blueprints
+    );
+  }
+
+  update() {
+    return Promise.all(
+      Object.keys(this.fetchers).map(source => {
+        return this.fetchers[source].update();
+      })
+    ).then(results => {
+      return "All sources updated";
+    });
+  }
+
+  retrieve(source, tab, resource) {
+    if (this.sourceExists(source)) {
+      const fetcher = this.fetchers[source];
+      return fetcher.retrieve(tab, resource);
+    } else {
+      return Promise.resolve().then(() => {
+        throw new Error(`Source ${source} not available.`);
+      });
+    }
+  }
+
+  retrieveFrag(source, tab, resource, frag) {
+    if (this.sourceExists(source)) {
+      const fetcher = this.fetchers[source];
+      return fetcher.retrieveFrag(tab, resource, frag);
+    } else {
+      return Promise.resolve().then(() => {
+        throw new Error(`Source ${source} not available.`);
+      });
+    }
+  }
+}
+
+export default Controller;
--- a/src/lib/Fetcher.js
+++ b/src/lib/Fetcher.js
@@ -0,0 +1,132 @@
+// FetcherTwo class interfaces with Google Sheet, and saves to a specified db
+import {google} from "googleapis";
+import {fmtSourceTitle, fmtBlueprinterTitles, deriveFilename, bp} from "./util";
+import {byRow, byId} from "./blueprinters";
+import R from "ramda";
+
+class Fetcher {
+  constructor(db, sourceName, sourceId, blueprinters) {
+    /*
+     * The database that the fetcher should use. This should be an instance of a model-compliant class.
+     * See models/Interface.js for the specifications for a model-compliant class.
+     */
+    this.db = db;
+
+    /*
+     * ID of the Google Sheet where the data is sourced. Note that the privateKey.client_email
+     * loaded here must be added to the sheet as an editor.
+     */
+    this.sourceId = sourceId;
+
+    /*
+     * The name of the source. This will prefix tabs saved in the database.
+     */
+    this.sourceName = sourceName;
+
+    /*
+     * These are the available tabs for storing and retrieving data.
+     * Each blueprinter is a function that returns a Blueprint from a
+     * list of lists (which will be retrieved from gsheets).
+     */
+    this.blueprinters = fmtBlueprinterTitles(blueprinters);
+    this.blueprints = {};
+    Object.keys(this.blueprinters).forEach(key => {
+      this.blueprints[key] = null;
+    });
+
+    /*
+     * Google API setup
+     */
+    this.sheets = google.sheets("v4");
+    this.auth = null;
+  }
+
+  /** returns a Promise that resolves if access is granted to the account, and rejects otherwise. */
+  authenticate(client_email, private_key) {
+    const googleAuth = new google.auth.JWT(client_email, null, private_key, [
+      "https://www.googleapis.com/auth/spreadsheets"
+    ]);
+    this.auth = googleAuth;
+    const {sourceId} = this;
+
+    return new Promise((resolve, reject) => {
+      googleAuth.authorize(function(err, tokens) {
+        if (err) {
+          reject(err);
+          return;
+        } else {
+          resolve(`Connected to ${sourceId}.`);
+        }
+      });
+    });
+  }
+
+  update() {
+    let tabTitles;
+    /* Retrieve all available routes on a given sheet, and store formatted copies of it where a formatter is available */
+    return this.sheets.spreadsheets
+      .get({
+        auth: this.auth,
+        spreadsheetId: this.sourceId
+      })
+      .then(response => {
+        tabTitles = response.data.sheets.map(sheet => sheet.properties.title);
+        return this.sheets.spreadsheets.values.batchGet({
+          auth: this.auth,
+          spreadsheetId: this.sourceId,
+          ranges: tabTitles
+        });
+      })
+      .then(results => {
+        const tabData = results.data.valueRanges;
+        return Promise.all(
+          tabData.map((tab, idx) => {
+            const {values} = tab;
+            if (values == undefined) {
+              return Promise.resolve({});
+            }
+            const name = tabTitles[idx];
+            return this.save(name, values);
+          })
+        );
+      })
+      .then(() => "All tabs updated");
+  }
+
+  save(tab, data) {
+    const title = fmtSourceTitle(tab);
+    if (Object.keys(this.blueprinters).indexOf(title) > -1) {
+      const blueprinters = this.blueprinters[title];
+
+      return blueprinters.map(blueprinter => {
+        const saturatedBp = blueprinter(
+          tab,
+          this.sourceName,
+          this.sourceId,
+          data
+        );
+        const blueprint = bp(saturatedBp); // TODO: come up with better semantics.
+        this.blueprints[title] = blueprint;
+        return this.db.save(saturatedBp);
+      });
+    } else {
+      // If it can't find a blueprinter for the tab title, default to byRow
+      return this.db.save(byRow(tab, this.sourceName, this.sourceId, data));
+    }
+  }
+
+  // NB: could combine these functions by checking kwargs length
+  retrieve(tab, resource) {
+    const title = fmtSourceTitle(tab);
+    const url = `${this.sourceName}/${tab}/${resource}`;
+    return this.db.load(url, this.blueprints[title]);
+  }
+
+  retrieveFrag(tab, resource, frag) {
+    const title = fmtSourceTitle(tab);
+    const url = `${this.sourceName}/${tab}/${resource}/${frag}`;
+    return this.db.load(url, this.blueprints[title]);
+  }
+}
+
+export default Fetcher;
--- a/src/lib/blueprinters.js
+++ b/src/lib/blueprinters.js
@@ -0,0 +1,28 @@
+import path from "path";
+import fs from "fs";
+
+const REL_PATH_TO_BPS = "../blueprinters";
+const allBps = {};
+
+export const defaultBlueprint = {
+  name: null,
+  id: null,
+  dialects: ["rest"], // supported dialects, can (eventually) be multiple
+  routes: {}
+};
+
+export const defaultRoute = {
+  options: {
+    fragment: true
+  },
+  data: []
+};
+
+// import all default exports from 'blueprinters' folder
+const normalizedPath = path.join(__dirname, REL_PATH_TO_BPS);
+fs.readdirSync(normalizedPath).forEach(file => {
+  const bpName = file.replace(".js", "");
+  allBps[bpName] = require(`${REL_PATH_TO_BPS}/${file}`).default;
+});
+
+module.exports = allBps;
--- a/src/lib/util.js
+++ b/src/lib/util.js
@@ -0,0 +1,87 @@
+import R from "ramda";
+
+String.prototype.replaceAll = function(search, replacement) {
+  const target = this;
+  return target.replace(new RegExp(search, "g"), replacement);
+};
+
+function camelize(str) {
+  return str.replace(/(?:^\w|[A-Z]|\b\w|\s+)/g, function(match, index) {
+    if (+match === 0) return ""; // or if (/\s+/.test(match)) for white spaces
+    // return index == 0 ? match.toLowerCase() : match.toUpperCase();
+    return match.toUpperCase();
+  });
+}
+
+export const fmtObj = R.curry(
+  (
+    columnNames,
+    row,
+    options = {
+      noSpacesInKeys: false,
+      hyphenatedKeys: false,
+      camelCaseKeys: false
+    }
+  ) => {
+    const obj = {};
+    const fmtColName = colName => {
+      if (options.camelCaseKeys) {
+        return camelize(colName);
+      } else if (options.hyphenatedKeys) {
+        return colName.toLowerCase().replaceAll(" ", "-");
+      } else if (options.noSpacesInKeys) {
+        return colName.replaceAll(" ", "");
+      } else {
+        return colName;
+      }
+    };
+    columnNames.forEach((columnName, idx) => {
+      obj[fmtColName(columnName)] = row[idx];
+    });
+    return obj;
+  }
+);
+
+/* search for object with key in array. Return index if exists, or -1 if not */
+export const idxSearcher = R.curry((attrName, searchValue, myArray) => {
+  for (var i = 0; i < myArray.length; i++) {
+    if (myArray[i][attrName] == searchValue) {
+      return i;
+    }
+  }
+  return -1;
+});
+
+/* more site specific functions. TODO: maybe move to another folder? */
+
+export function fmtSourceTitle(name) {
+  return name.replaceAll(" ", "-").toLowerCase();
+}
+
+export function fmtBlueprinterTitles(tabs) {
+  const obj = {};
+  Object.keys(tabs).forEach(tab => {
+    const name = fmtSourceTitle(tab);
+    obj[name] = tabs[tab];
+  });
+  return obj;
+}
+
+export function deriveFilename(source, tab) {
+  return `${fmtSourceTitle(source)}-${fmtSourceTitle(tab)}.json`;
+}
+
+export function bp(full) {
+  const blueprint = {
+    name: R.clone(full.name),
+    source: R.clone(full.source),
+    dialects: R.clone(full.dialects),
+    routes: {}
+  };
+  Object.keys(full.routes).forEach(route => {
+    blueprint.routes[route] = {
+      options: R.clone(full.routes[route].options)
+    };
+  });
+  return blueprint;
+}
--- a/src/middleware/index.js
+++ b/src/middleware/index.js
@@ -0,0 +1,17 @@
+import {Router, next} from "express";
+import {mapboxAccessToken} from "../config";
+import morgan from "morgan";
+import mapbox from "./mapbox";
+
+export default ({config, db}) => {
+  let routes = Router();
+
+  /* logging middleware */
+  routes.use(morgan("dev"));
+
+  if (mapboxAccessToken) {
+    routes.get("/mapbox/:z/:y/:x", mapbox(mapboxAccessToken));
+  }
+
+  return routes;
+};
--- a/src/middleware/mapbox/index.js
+++ b/src/middleware/mapbox/index.js
@@ -0,0 +1,16 @@
+import fetch from "node-fetch";
+import fs from "fs";
+// TODO: load images from mapbox API and store.
+
+const baseUrl = "http://a.tiles.mapbox.com/v4/mapbox.satellite";
+export default accessToken => (req, res) => {
+  const {x, y, z} = req.params;
+  // const filename = `${z}-${y}-${x}.png`
+  // const fileStream = fs.createWriteStream(`${z}-${y}-${x}.png`)
+  fetch(
+    `http://a.tiles.mapbox.com/v4/mapbox.satellite/${z}/${y}/${x}@2x.png?access_token=${accessToken}`
+  ).then(result => {
+    res.set("Content-Type", "image/png");
+    result.body.pipe(res);
+  });
+};
--- a/src/models/Interface.js
+++ b/src/models/Interface.js
@@ -0,0 +1,23 @@
+/**
+ * Model is a class whose sole responsibility is to save and load blueprints.
+ * It allows for different storage mechanisms for different kinds of blueprints.
+ */
+class Model {
+  /**
+   * save - save a Blueprint, using the information it contains.
+   *
+   * @param  {type} blueprint the Blueprint to be saved.
+   * @return {type} Promise   which returns True.
+   */
+  save(blueprint) {}
+
+  /**
+   * load - load a resource from a data model, using a Blueprint object as
+   * well as a REST-like  URL of the format /:source/:tab/:resource.
+   *
+   * @param  {type} url       String that represents the path to resource.
+   * @param  {type} blueprint Blueprint object (desaturated?).
+   * @return {type}           Object containing the resource data.
+   */
+  load(url, blueprint) {}
+}
--- a/src/models/StoreJson.js
+++ b/src/models/StoreJson.js
@@ -0,0 +1,65 @@
+import fs from "mz/fs";
+import hash from "object-hash";
+import {fmtSourceTitle} from "../lib/util";
+import path from "path";
+
+const STORAGE_DIRNAME = "temp";
+
+class StoreJson {
+  save(bp) {
+    return Promise.all(
+      Object.keys(bp.routes).map(route =>
+        fs.writeFile(
+          `${STORAGE_DIRNAME}/${fmtSourceTitle(
+            bp.source.name
+          )}__${fmtSourceTitle(bp.name)}__${route}.json`,
+          JSON.stringify(bp.routes[route].data)
+        )
+      )
+    );
+  }
+
+  load(url, bp) {
+    const parts = url.split("/");
+    const fname = `${STORAGE_DIRNAME}/${parts[0]}__${parts[1]}__${
+      parts[2]
+    }.json`;
+    return fs
+      .exists(fname)
+      .then(isAvailable => {
+        if (isAvailable) return fs.readFile(fname, "utf8");
+        else {
+          throw new Error("No resource exists");
+        }
+      })
+      .then(data => JSON.parse(data))
+      .then(data => {
+        if (parts.length === 3) {
+          // No lookup if the requested url doesn't have a fragment
+          return data;
+        } else if (parts[2] === "ids") {
+          // Do a lookup if fragment is included to filter a relevant item
+          // When the resource requested is 'ids'
+          const id = parseInt(parts[3]);
+          if (id !== NaN && id >= 0 && id < data.length) {
+            return data[id];
+          } else {
+            throw new Error(`Fragment index does not exist`);
+          }
+        } else {
+          // Do a lookup if fragment is included to filter a relevant item
+          const index = parseInt(parts[3]);
+          if (index !== NaN && index >= 0 && index < data.length) {
+            console.log(data, index);
+            return data.filter((vl, idx) => idx === index)[0];
+          } else {
+            throw new Error(`Fragment index does not exist`);
+          }
+        }
+      });
+  }
+
+  // TODO: add method to build blueprint from data source
+}
+
+export default StoreJson;
--- a/test/internals.js
+++ b/test/internals.js
@@ -0,0 +1,64 @@
+import test from 'ava'
+import R from 'ramda'
+import {
+  defaultBlueprint,
+  defaultRoute,
+  byColumn,
+  byRow
+} from '../src/lib/blueprinters'
+
+const egInput1 = [
+  ['h1', 'h2', 'h3'],
+  [1, 2, 3],
+  [4, 5, 6]
+]
+
+test('defaultBlueprint exports', t => {
+  const expected = {
+    name: null,
+    id: null,
+    dialects: ["rest"],
+    routes: {}
+  }
+  t.deepEqual(expected, defaultBlueprint)
+})
+
+test('byColumn blueprinter generates expected output', t => {
+  const actual = byColumn("eg ColumnBlueprint", "egSourceName", "egSourceId", egInput1)
+  const expected = R.clone(defaultBlueprint)
+  expected.name = "eg ColumnBlueprint"
+  expected.source = {
+    id: "egSourceId",
+    name: "egSourceName"
+  }
+  expected.routes['h1'] = R.clone(defaultRoute)
+  expected.routes['h1'].data = [1, 4]
+  expected.routes['h2'] = R.clone(defaultRoute)
+  expected.routes['h2'].data = [2, 5]
+  expected.routes['h3'] = R.clone(defaultRoute)
+  expected.routes['h3'].data = [3, 6]
+  t.deepEqual(expected, actual)
+})
+
+test('byRow blueprinter generates expected output', t => {
+  const actual = byRow("egRowBlueprint", "egSourceName", "egSourceId", egInput1, "items", )
+  const expected = R.clone(defaultBlueprint)
+  expected.name = "egRowBlueprint"
+  expected.source = {
+    id: "egSourceId",
+    name: "egSourceName"
+  }
+  expected.routes['items'] = R.clone(defaultRoute)
+  expected.routes['items'].data = [{
+      h1: 1,
+      h2: 2,
+      h3: 3
+    },
+    {
+      h1: 4,
+      h2: 5,
+      h3: 6
+    },
+  ]
+  t.deepEqual(expected, actual)
+})
--- a/yarn.lock
+++ b/yarn.lock