Jsonotron

How It Works

Each JSON document in your NoSQL database is designated a docType that defines it's schema.

This docType is a plain Javascript object that defines the data stored in the document as well as the logic for querying, updating and validating it.

Suppose you want to store JSON documents representing customers...

{
  "customerName": "Bob Brown",
  "dateOfBirth": "2000-01-04",
  "ordersPlaced": 0
}

This could be managed by a docType defined as follows...

const customerDocType = {
  /* The name/pluralName is used for programmatic access
      while title/pluraltitle is used for documentation. */
  name: 'customer',
  pluralName: 'customers',
  title: 'Customer',
  pluralTitle: 'Customers',

  /* A list of the fields that the JSON document can contain.
      Each field type is a reference to a JSON schema snippet. */
  fields: {
    customerName: { type: 'mediumString', canUpdate: true, description: 'The name of a customer.', example: 'A Non' },
    dateOfBirth: { type: 'date', canUpdate: true, description: 'A customers date of birth.', example: '2000-01-01' },
    ordersPlaced: { type: 'positiveIntegerOrZero', description: 'The number of orders placed by this customer.', example: 4  }
  },

  /* A constructor allows you to initialise fields. 
      The lookup instruction means the definition should come from the fields section. */
  ctor: {
    parameters: {
      customerName: { lookup: 'field' },
      dateOfBirth: { lookup: 'field' }
    },
    implementation: data => ({
      customerName: data.customerName,
      dateOfBirth: data.dateOfBirth,
      ordersPlaced: 0
    })
  }

  /* This policy allows us to fetch the whole collection in one go.
      Typically this will be turned off for large collections. */
  policy: {
    canFetchWholeCollection: true
  }
}

By passing the above docType to an instance of Jsonotron you can then perform CRUD operations on it straight away. In the examples that follow the RESTful interface provided by jsonotron-express is used.

The schema will be enforced automatically, so if you send invalid data then you'll get a sensible error message explaining what went wrong.

Getting Started

Initialise a new NodeJS project. NPM has a nice wizard for guiding you through the creation of a package.json file.

mkdir my-data-service
cd my-data-service
npm init

Choose one backend package and one interface component and install them using npm.

npm install jsonotron-memdocstore jsonotron-express --save

Some of the packages have peer dependencies that you need to install. NPM will warn about these in the console.

For example, if you're using the jsonotron-express package, you'll need to install ExpressJS.

npm install express --save

You will also need some additional packages to make setup easier.

If you're using the jsonotron-memdocstore package, you'll also need a UUID generator for the createMemDocStore constructor. The easiest one to grab is called uuid.

If you're using the jsonotron-express package you'll need a body-parser for handling JSON payloads. The easiest one to grab is called body-parser.

npm install uuid body-parser --save

Your package.json should now look something like the below.

{
  "name": "@karlhulme/my-data-service",
  "version": "1.0.0",

  "description": "A microservice for querying and updating curated JSON documents.",
  "main": "index.js",
  "scripts": {
    "test": "npm test"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/karlhulme/my-data-service.git"
  },
  "author": "Karl Hulme ",
  "license": "UNLICENSED",
  "dependencies": {
    "body-parser": "^1.19.0",
    "express": "^4.17.1",
    "jsonotron-express": "^0.1.3",
    "jsonotron-memdocstore": "^0.3.0"
    "uuid": "^3.3.3"
  }
}        

You can now create an index.js file that launches a Jsonotron-based microservice.

const bodyParser = require('body-parser')
const express = require('express')
const { createJsonotronExpress } = require('jsonotron-express')
const { createMemDocStore } = require('jsonotron-memdocstore')
const uuid = require('uuid/v4')

const PORT = 3000

const customerDocType = {
  name: 'customer',
  pluralName: 'customers',
  title: 'Customer',
  pluralTitle: 'Customers',
  policy: {
    canFetchWholeCollection: true
  },
  fields: {
    customerName: { type: 'mediumString', description: 'The name of a customer.', example: 'A Non' },
    dateOfBirth: { type: 'date', description: 'A customers date of birth.', example: '2000-01-01' }
  }
}

const adminRoleType = {
  name: 'admin',
  docPermissions: true
}

const runApp = () => {
  const memDocStore = createMemDocStore([], uuid)

  const docTypes = [customerDocType]
  const roleTypes = [adminRoleType]

  const jsonotronExpress = createJsonotronExpress({
    docStore: memDocStore,
    docTypes,
    roleTypes
  })

  const app = express()
  app.use(bodyParser.json())
  app.use(jsonotronExpress)

  app.listen(PORT, () => console.log('Listening...'))
}

runApp()

You can now query for all the customer documents. (The custom header tells Jsonotron which role to assume for the query.)

curl -H 'x-jsonotron-rolenames: admin' -H 'x-user-id: testUser' http://localhost:3000/data/customers

You can now improve the microservice by customising the docType definitions with more fields (including custom types), a constructor, filters, patching and update operations.

DocType Definitions

The docType below shows all the possible properties and what they do.

{
  /* The name fields are used for programmatic access
      while the title fields are used for documentation. */
  name: 'person',
  pluralName: 'persons',
  title: 'Person',
  pluralTitle: 'Persons',

  /* The optional policy object. */
  policy: {
    /* True if the entire collection can be fetched in a single request, defaults to false. */
    canFetchWholeCollection: true,

    /* True if documents can be replaced without using the constructor, defaults to false. */
    canReplaceDocuments: true,

    /* True if documents can be deleted, defaults to false. */
    canDeleteDocuments: true,

    /* The number of operations (default 10) to kept to prevent the lost/repeated update problem. */
    maxOpsSize: 5
  },

  /* A list of the fields that are stored in the JSON document.  Each field must
      have either a type and description field, or a ref & description field. */
  fields: {
    /* 'type' is a fieldType that identifies a JSON schema. (Custom field types can be provided to Jsonotron.)
       'isRequired' should be true if the property must be set.
       'canUpdate' should be true if the property can be patched.
       'description' describes the field, used for documentation.
       'isArray' should be true if the property is an array of the named type.
       'default' should provide a default value that is used during selection if it's not populated.
       'cacheDurationInSeconds' can be used by upstream services, such as GraphQL */       
    tenantId: { type: 'shortString', isRequired: true, description: 'The organisation that owns the record.', example: '1234' },
    shortName: { type: 'shortString', isRequired: true, canUpdate: true, description: 'A short informal name, typically the person\'s first name.', example: 'Bob' },
    fullName: { type: 'mediumString', isRequired: true, canUpdate: true, description: 'The person\'s full name.', example: 'Bob Brown' },
    dateOfBirth: { type: 'date', canUpdate: true, description: 'The date of birth.', example: '2000-01-01' },
    addressLines: { type: 'longString', canUpdate: true, description: 'The current residential address with each line separated by a newline (\\n) character.', example: '1 Buckingham Palace\nWindsor' },
    postCode: { type: 'shortString', canUpdate: true, description: 'A postal code.', example: 'W10 9KL' },
    pinCode: { type: 'positiveInteger', description: 'The code used for clocking in.', example: 1234 },
    favouriteColors: { type: 'shortString', isArray: true, description: 'An array of color names.', example: 'red' },
    allowMarketing: { type: 'yesNo', default: 'no', description: 'A value of \'yes\' indicates that the person is prepared to receive marketing or \'no\' if they are not.', example: 'yes' },
    heightInCms: { type: 'integer', default: 0, description: 'The height of the person in centimetres.', example: 150 },
    ownedCarId: { ref: 'car', cacheDurationInSeconds: 60, description: 'The car owned by this person.' }
  },

  /* This function is run every time a document changes to check that it's valid. */
  validate: doc => {
    if ((doc.addressLines || '').includes('castle')) {
      throw new Error('No castle dwellers allowed')
    }
  },

  /* The calculatedFields are available for querying but cannot be set or updated. */
  calculatedFields: {
    
    /* The name of a calculated field. */
    fullAddress: {

      /* The description used for documentation. */
      description: 'The full residential postal address.',

      /* The input fields on which this calculated field relies. */
      inputFields: ['addressLines', 'postCode'],

      /* A function that takes the input fields and computes a calculated value. */
      value: data => `${data.addressLines ? data.addressLines + '\n' : ''}${data.postCode || ''}`
    }
  },

  /* The filters provide access to a subset of documents of specific type. */ 
  filters: {
    
    /* The name of the filter. */
    byPostCode: {

      /* The description is used for documentation. */
      description: 'Fetch people that live at the appropriate post code.',

      /* A set of parameters that should be passed to the filter.
        These can follow the same rules as the main document fields or they can be
        specified as { lookup: 'field' }. */
      parameters: {
        postCode: { type: 'string', isRequired: true, description: 'The post code to match.', example: 'BH87 2KL' }
      },

      /* Returns any valid Javascript primitive which is then interpreted by the backing store.
          For jsonotron-memdocstore, this should be a comparision function.
          For the jsontron-cosmosdb, this should be a Azure Cosmos SQL WHERE clause. */
      implementation: input => d => d.postCode === input.postCode
    }
  },

  /* This section defines a constructor that is used to create new JSON documents. */
  ctor: {

    /* A set of fields using the same rules as the main document fields or they can be
        specified as { lookup: 'field' }. */
    parameters: {
      shortName: { lookup: 'field', isRequired: true },
      fullName: { lookup: 'field', isRequired: true },
      dateOfBirth: { lookup: 'field', isRequired: true },
      askedAboutMarketing: { type: 'boolean', isRequired: true, description: 'This is an additional field.', example: true }
    },

    /* A function that utilises the input fields to construct a new JSON document. */
    implementation: input => {
      return {
        tenantId: 'companyA',
        shortName: input.shortName,
        fullName: input.fullName,
        dateOfBirth: input.dateOfBirth,
        allowMarketing: input.askedAboutMarketing ? 'yes' : 'no'
      }
    }
  },

  /* This section defines the operations that mutate a JSON document. */
  operations: {

    /* The name of the operation. */
    replaceFavouriteColors: {

      /* The title of the operation, used for documentation. */
      title: 'Replace Favourite Colors',

      /* The description of the operation, used for documentation. */
      description: 'Replace the favourite colors of this person.',

      /* The input parameters of the operation that uses the same rules as the main
          document fields, or can use the { lookup: 'field' } syntax. */
      parameters: {
        favouriteColors: { lookup: 'field', isRequired: true }
      },

      /* A function that uses the existing document, and the input fields, to produce
          a new mutated document. */
      implementation: (doc, input) => ({
        favouriteColors: ['silver'].concat(input.favouriteColors)
      })
    }
  }
}

RoleType Definitions

The roleType below shows all the possible properties and what they do.

{
  /* The name of the role, used in each request to determine the permissions available. */
  name: 'exampleRole',

  /* The docPermissions property can be set to true, granting all permissions on all
      docTypes.  Otherwise use an object for finer-grain control. */
  docPermissions: {

    /* The name of a docType. Can be set to true to grant all permissions on that docType
        or an object for finer-grain control. */
    person: {

      /* True if all fields of a document can be queried, otherwise specific an object
          for finer-grain control. */
      query: {

        /* Either 'whitelist' or 'blacklist'  This determines if the fields property determines
            the fields which can be queried or the fields which cannot be queried. */
        fieldsTreatment: 'whitelist',

        /* A list of field names. */
        fields: [
          'id', 'shortName', 'fullName', 'dateOfBirth', 'addressLines',
          'postCode', 'favouriteColors', 'allowMarketing', 'fullAddress'
        ]
      },

      /* True if documents can be patched and all operations are available, otherwise
          use an object for fine-grain control. */
      update: {

        /* True if the fields marked with canUpdate can be patched by users of this role. */ 
        patch: true,

        /* A list of the operations that can be invoked by this role. */
        operations: ['replaceFavouriteColors', 'attemptToChangeId']
      },

      /* True if documents can be created using the docType constructor. */
      create: true,

      /* True if documents can be deleted. */
      delete: true,

      /* True if documents can be replaced.  Meaning the external system provides all fields
          as the constructor will be bypassed. */
      replace: true
    }
  }
}

An admin role that allows global access can be defined as shown below.

{
  name: 'admin',
  docPermissions: true
}