Updated view with validation button

Bump lodash from 4.17.15 to 4.17.19

README.md

@@ -36,45 +36,6 @@ Querying data directly from spreadsheets is brittle, as it relies on the mainten
 
 To see how to get a local instance of datasheet server running in practice, see [this wiki](https://github.com/forensic-architecture/timemap/wiki/running-timemap-and-datasheet-server-locally) explaining how to use it to feed data from a Google Sheet to a local instance of [Timemap](https://github.com/forensic-architecture/timemap).
 
-### Design Concepts
-The codebase currently only supports Google Sheets as a source, and a REST-like format as a query language. It is designed, however, with extensibility in mind.
-
-**Sources**     
-A source represents a sheet-like collection of data, such as a Google Sheet. A source has one or more **tabs**, each of which contains a 2-dimensional grids of cells. Each cell contains a body of text (a string).
-
-**Resources**     
-The data from sources are made available as resources, which are structured blocks of data that are granularly accessible through a query language. Resources are the outfacing aspect of Datasheet Server, and represent the only kind of data that can be queried by applications. Each resource is configured with one or more **query languages**. (Currently only a REST-like query language supported.)
-
-**Blueprints**       
-Blueprints are a data structure that represent the way that infromation from **sources** are to be turned into **resources**. For each tab in a source, there is a corresponding Blueprint. Blueprints are created through a [blueprinter function](/src/blueprinters) invoked on the raw data from a source tab.
-
-Blueprints are JSON objects. There have two forms:
-
-1. _desaturated_ -- describes the resources and query languages available on data from a source tab.
-2. _saturated_ --  both describes resources available on data from a source etab, and contains that data.
-
-A desaturated Blueprint can be saturated by retrieving its data from the server's **model layer**, which stores tab data from sources.
-
-A JSON catalogue of the available blueprints (desaturated) in a server is available at `/api/blueprints`.
-
-## [Configuration](#configuration)
-Copy the example environment file:
-```
-cp .env.example .env
-```
-Inside this file, you will need to modify at least the `SERVICE_ACCOUNT_EMAIL` and `SERVICE_ACCOUNT_PRIVATE_KEY` fields. These fields refer to the credentials of a [Google service account](https://cloud.google.com/iam/docs/understanding-service-accounts). Google requires that developers create these when attempting to access their services programmatically, so that they can attribute requests to users. Service accounts also contain identity information, which means that asset owners can allow certain service accounts access to certain sheets, just as one might differentially grant certain users access to certain cloud assets. 
-
-Once you have [created a service account](https://support.google.com/a/answer/7378726?hl=en), create and download an [API key for that account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). The JSON file for the API key that you download when you create it contains both a service account private key, and an email associated with the service account: add these respectively in the strings in .env for `SERVICE_ACCOUNT_PRIVATE_KEY` and `SERVICE_ACCOUNT_EMAIL`.
-
-The last thing to do is to grant the service account access to the sheet that
-datasheet-server is pulling from. You can add a service account to a sheet as
-you would any other Google user: just enter the email address associated. (Note
-that this step is not necessary if you are accessing a publicly available
-sheet.)
-
-Other configuration options, such as the port at which the server will expose
-resources, are also modifiable from the .env file.
-
 #### [Sources](#sources)
 Sources are specified in [src/config.js](https://github.com/forensic-architecture/datasheet-server/blob/develop/src/config.js). Datasheet server currently only supports Google Sheets as a source.
 
@@ -86,34 +47,13 @@ Each Google Sheet being used as a as source requires a corresponding object in `
 | Option | Description | Type |
 | ------ | ----------- | ---- |
 | name | Used to refer to data served from this source | string |
-| id | The ID of the sheet in Google. (You can find it in the address bar when the Sheet is open in a browser. It is the string that follows 'spreadsheets/d/'). | string |
+| path/id | The path to the XLSX sheet on your local, or the ID of the sheet in Google. (You can find it in the address bar when the Sheet is open in a browser. It is the string that follows 'spreadsheets/d/'). | string |
 | tabs | An object that maps each tab in the source to one or more Blueprinters. All of the Blueprinters in the [blueprinters folder](/lib/blueprinters) are available through a single import as at the top of [example.config.js](/src/example.config.js). <br> To correctly associate a Blueprinter, the object key needs to be _the tab name with all lowercase letters, and spaces replaced by a '-'_. For example, if the tab name in Google Sheets is 'Info About SHEEP', the object key should be 'info-about-sheep'. <br> The value should be the Blueprinter function that you want to use for the data in that tab. If you require more than one endpoint for a single tab, you can support multiple blueprinters by making the item an array. See the example of a configuration object below. <br>TODO: no Blueprinter is used by default. | object |
 
-###### Example Configuration Object
-```js
-import BP from './lib/blueprinters'
-
-export default {
-  googleSheets: {
-    email: 'project-name@reliable-baptist-23338.iam.gserviceaccount.com',
-    privateKey: 'SOME_PRIVATE_KEY',
-    sheets: [
-      {
-        name: 'example',
-        id: '1s-vfBR8Uy-B-TLO_C5Ozw4z-L0E3hdP8ohMV761ouRI',
-        tabs: {
-          'objects': BP.rows,
-          'fruit': [BP.columns, BP.ids],
-        }
-      },
-    ]
-  }
-}
-
-```
+See src/config.js for an example configuration sheet. 
 
 
-## [Quickstart](#quickstart)
+## [Quickstart](#quickstart)
 Clone the repository to your local:
 ```
 git clone https://www.github.com/forensic-architecture/datasheet-server

docs/design-concepts.md

@@ -0,0 +1,22 @@
+### Design Concepts
+The codebase currently only supports Google Sheets as a source, and a REST-like format as a query language. It is designed, however, with extensibility in mind.
+
+**Sources**     
+A source represents a sheet-like collection of data, such as a Google Sheet. A source has one or more **tabs**, each of which contains a 2-dimensional grids of cells. Each cell contains a body of text (a string).
+
+**Resources**     
+The data from sources are made available as resources, which are structured blocks of data that are granularly accessible through a query language. Resources are the outfacing aspect of Datasheet Server, and represent the only kind of data that can be queried by applications. Each resource is configured with one or more **query languages**. (Currently only a REST-like query language supported.)
+
+**Blueprints**       
+Blueprints are a data structure that represent the way that infromation from **sources** are to be turned into **resources**. For each tab in a source, there is a corresponding Blueprint. Blueprints are created through a [blueprinter function](/src/blueprinters) invoked on the raw data from a source tab.
+
+Blueprints are JSON objects. There have two forms:
+
+1. _desaturated_ -- describes the resources and query languages available on data from a source tab.
+2. _saturated_ --  both describes resources available on data from a source etab, and contains that data.
+
+A desaturated Blueprint can be saturated by retrieving its data from the server's **model layer**, which stores tab data from sources.
+
+A JSON catalogue of the available blueprints (desaturated) in a server is available at `/api/blueprints`.
+
+

docs/gsheet-config.md

@@ -0,0 +1,19 @@
+## [Configuration](#configuration)
+Copy the example environment file:
+```
+cp .env.example .env
+```
+Inside this file, you will need to modify at least the `SERVICE_ACCOUNT_EMAIL` and `SERVICE_ACCOUNT_PRIVATE_KEY` fields. These fields refer to the credentials of a [Google service account](https://cloud.google.com/iam/docs/understanding-service-accounts). Google requires that developers create these when attempting to access their services programmatically, so that they can attribute requests to users. Service accounts also contain identity information, which means that asset owners can allow certain service accounts access to certain sheets, just as one might differentially grant certain users access to certain cloud assets. 
+
+Once you have [created a service account](https://support.google.com/a/answer/7378726?hl=en), create and download an [API key for that account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). The JSON file for the API key that you download when you create it contains both a service account private key, and an email associated with the service account: add these respectively in the strings in .env for `SERVICE_ACCOUNT_PRIVATE_KEY` and `SERVICE_ACCOUNT_EMAIL`.
+
+The last thing to do is to grant the service account access to the sheet that
+datasheet-server is pulling from. You can add a service account to a sheet as
+you would any other Google user: just enter the email address associated. (Note
+that this step is not necessary if you are accessing a publicly available
+sheet.)
+
+Other configuration options, such as the port at which the server will expose
+resources, are also modifiable from the .env file.
+
+

package-lock.json

@@ -4600,9 +4600,9 @@
       }
     },
     "lodash": {
-      "version": "4.17.15",
-      "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.15.tgz",
-      "integrity": "sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A==",
+      "version": "4.17.19",
+      "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.19.tgz",
+      "integrity": "sha512-JNvd8XER9GQX0v2qJgsaN/mzFCNA5BRe/j8JN9d+tWyGLSodKQHKFicdwNYzWwI3wjRnaKPsGj1XkBjx/F96DQ==",
       "dev": true
     },
     "lodash.clone": {

package.json

@@ -28,6 +28,7 @@
     "express-handlebars": "^4.0.4",
     "googleapis": "^32.0.0",
     "graphql": "^0.13.2",
+    "moment": "^2.27.0",
     "morgan": "^1.8.0",
     "mz": "^2.7.0",
     "node-fetch": "^2.3.0",

src/api/index.js

@@ -58,6 +58,17 @@ export default ({ config, controller }) => {
       )
   })
 
+  // api.get('/:sheet/:tab/:resource/validate', (req, res) => {
+  //   const { sheet, tab, resource } = req.params
+  //   controller
+  //     .retrieve(sheet, tab, resource)
+  //     .then(data => res.json(data))
+  //     .catch(err =>
+  //       res.status(err.status || 404)
+  //         .send({ error: err.message })
+  //     )
+  // })
+
   // ERROR routes. Note that it is important that these come AFTER routes
   // like /update, so that the regex does not greedily match these routes.
 

src/blueprinters/deeprows.js

@@ -1,4 +1,5 @@
 import { fmtObj } from '../lib/util'
+import { getColumnValidation } from '../lib/validation'
 
 /**
  * Each resource item is an object with values labelled according
@@ -35,7 +36,6 @@ export default (data) => {
       }
     }
   })
-
   // generate the value for deep labels using the structure created
   data.forEach((row, idx) => {
     if (idx === 0) return
@@ -61,9 +61,11 @@ export default (data) => {
 
     // move values for flat labels over from base
     structure.__flat.forEach(label => {
+      const validatedLabel = getColumnValidation(label, baseRow[label])
+      console.log(validatedLabel)
       deepRow[label] = baseRow[label]
     })
-    if (deepRow['id'] && deepRow['id'] !== '') {
+    if (!Object.keys(deepRow).every(k => deepRow[k] === '')) {
       output.push(deepRow)
     }
   })

src/config.js

@@ -1,7 +1,13 @@
 import { timemap } from './lib'
 
 export default {
-  gsheets: [],
+  gsheets: [
+    {
+      name: 'us2020',
+      id: '1I_pgyTQJlIorTIEHBxw1mM1STn-SrIi66FKYxut61iM',
+      tabs: timemap.default
+    }
+  ],
   xlsx: [
     {
       name: 'timemap_data',

src/lib/util.js

@@ -1,4 +1,4 @@
-import R from 'ramda'
+import R from 'ramda';
 
 /* eslint-disable */
 String.prototype.replaceAll = function (search, replacement) {

src/lib/validation.js

@@ -0,0 +1,20 @@
+import moment from 'moment';
+
+const DATE_FORMAT = "MM/DD/YYYY";
+const TIME_REGEX = new RegExp("^([01]\d|2[0-3]):?([0-5]\d)$")
+
+
+export const getColumnValidation = (colName, value) => {
+  switch(colName) {
+    case 'longitude':
+      return isFinite(value) && Math.abs(value) <= 180;
+    case 'latitude':
+      return isFinite(value) && Math.abs(value) <= 90;
+    case 'date':
+      return moment(value, DATE_FORMAT, true).isValid();
+    case 'time':
+      return TIME_REGEX.test(value);
+    default:
+      return true
+  }
+}

views/blueprints.hbs

@@ -9,6 +9,7 @@
     </div>
     {{#each urls}}
     <div><a target="_blank" href="http://localhost:4040{{ this }}">{{ this }}</a></div>
+    <div class="bp-validate-button" target="_blank" href="http://localhost:4040{{ this }}">Validate</div>
     {{/each}}
   </div>
   {{ else }}
@@ -25,6 +26,7 @@
 :root {
   --grey: #8a8a8a;
   --btn-width: 200px;
+  --validate-btn-width: 150px;
 }
 .main-container {
   display: flex;
@@ -79,6 +81,21 @@
   color: black;
 }
 
+.bp-validate-button {
+  transition: all 0.3s ease;
+  text-transform: uppercase;
+  font-size: 12pt;
+  font-weight: bold;
+  width: var(--validate-btn-width);
+  display: flex;
+  justify-content: center;
+  align-items: center;
+  border: 3px solid red;
+  padding: .5em;
+  text-decoration: none;
+  color: red;
+}
+
 .bp-update-container:hover {
   background-color: black;
   transition: all 0.3s ease;