Author: John Ormerod, W3Partnership Limited

Date: December 2016

Copyright

Copyright subsists in all W3Partnership publications. Except as permitted by Copyright Law, no extract may be reproduced, stored in a retrieval system or transmitted in any form or by any means – electronic, photocopying, recording or otherwise – without the prior permission in writing from W3Partnership.

© W3Partnership Limited (2016)

Contact Address:

W3Partnership Limited

18a Kendale

Hemel Hempstead

Hertfordshire, HP3 8NN

Tel:      +44 (0) 845 8685041

 

All terms that are mentioned in this document that are known to be trademarks or service marks have been appropriately capitalised. W3Partnership Limited cannot attest to the accuracy of this information. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark.

 

Introduction

This article is part of a 4 paper series covering  “Open Source” tools available to help developers when creating REST/JSON APIs. The tools covered are those that I have used on projects and then returned to for subsequent development activities.

The tools  will be covered in categories, as follows:

Part 1

• API Definition
• Unit Testing

Part 2

• Calling APIs: Test Clients

Part 3

• Function and Load Tests
  • SoapUI
  • JMeter

Part 4

• Mocking back-end services

In part one we will look in detail at the following;

• JSON Editor Online (creating JSON)
• JSON Formatter and Validator and JSON Lint (formatting and validation)
• JSON Lint (syntax Validation)
• JSON Schema and Schema Guru Web (schema generation)
• JSON Schema Validator (validating JSON against its schema)
• Using JUnit for running unit tests during a build for:
  • validation of JSON and XML against a schema
  • transformations to/from XML and JSON

Some of the tools were recommended by colleagues on projects, some I discovered and the rest were mandated for use on a project.  I hope you will find the way I have shared my experiences of these tools, to be useful and interesting, and will investigate and use some of them yourself.

I recently worked on a simple demo application. This app, for an integration platform, provides the examples in this paper. The app integrates calls from two API providers and works as follows:

• Enter a postal code
• Receive a 5-day rain forecast, which will show the predicted rainfall for each day.

There are essentially two types of API development:

• Ones that provide access to back-end services on a web server
• Ones that integrate back-end APIs to create new functionality

My experience has been with the latter (WSO2 and MuleSoft), though most, if not all, of the tools covered here are applicable to both types of development.

API Definition

There are two primary initiatives for defining an API that have arisen:

• Swagger (renamed to the OpenAPI Specification in January 2016) see: Open API Initiative’s website
• RAML see: RAML Tools website

At the time of writing, these are competing specifications, for which Swagger has support from the largest number of organisations. As the author of a comparison between the two  states: “However, RAML has very good features, and I expect the Open API Initiative to pick some of the best parts from many different specifications, as it tries to standardize.” See: REST API Documentation Part 2 — Swagger, RAML, and Open API

It is not the intention to cover tools for defining Swagger and RAML here. What is important is the creation of a comprehensive definition of an API and its requests and responses.

JSON Creation

When integrating APIs, the JSON response to GET requests from existing APIs will be pre-defined. However, the response to your API will be created by manipulating responses from the called APIs, and as such, must be designed.

The JSON Editor Online

This  is an easy-to-use tool, where you define the target data structure and it generates the JSON.

If we pick up our example demo application;

5-day rain forecast for a postal code

What we want is an array of 5 elements, each of which has a date and rainfall value.

The first step is to add the result object

 

Then an array of forecasts

Enter forecast

 

Now add a JSON Object to hold the date and rain values

 

At any point, you can create the JSON code to check that it is how you intended it to be

 

Press the < button

 

Add the two fields for the first forecast array and check the JSON

 

There is a duplicate facility, so you can create the next 4 objects from the first one

Adjust the values, press the < button, and there we are:

>>>

JSON Validation

The JSON created by the above tool should be syntactically correct. On the other hand, JSON that you have hand-crafted needs to be validated, as does any JSON definitions coming from elsewhere. There are two stages in such validation:

• Is the syntax correct?
• Is it valid per its schema?

Format JSON & Check ITS Syntax

Single-line JSON, with spaces removed, is useful when compactness is required, such as in a Swagger or RAML definition. Formatting your JSON makes it easier to read. The tool I use for this is;

JSON Formatter and Validator

This  allows you to choose the tab value and the standard to be used for validating the definition.

Chose the standard and tab value, then press Process.

 

The result is:

 

The buttons provide facilities such as: copy to clipboard; download, collapse/expand

 

If you remove the closing ‘]’ for the array and validate, the result is:

 

 

I have also used

JSON Lint for syntax checking

 

Press the Validate JSON button – this also formats the JSON using a default value for the tabs

 

Validating Against a Schema

When defining an API in Swagger/RAML, you have the option of providing the schema in addition to a sample. The provision of a schema is essential for consumers of an API, so that they are aware of all constraints on the data. When you are creating an API you will need to create a schema for your JSON. The easiest way to get started is to use a tool that generates a first-cut schema.

Schema Generation

I have found that using the two following tools and combining their outputs to be effective.

1. JSON Schema

Paste your JSON in to the left-hand panel and press Generate Schema.

 

The left-hand panel has a very wide range of options for creating a richer definition.

What this tool doesn’t cater for is examples where their generated schemas would differ slightly due to alternative values or optional fields. The next tool allows you to provide several sample JSON definitions, and it attempts to merge them into a schema that handles them all. It has another useful feature, which is to provide some value ranges.

 

 

 

2. Schema Guru Web

 

Paste your JSON into the left-hand panel and press Submit. Notice that the JSON has been allocated a file name. If you paste in more JSON this will be held as another file. If both file are selected, the right-hand panel can show you a composite schema or the differences.

I used the schema generated by the first tool and pasted in validation criteria (suitably adjusted) from the second tool. A fully-defined schema will require manual additions for requirements that can’t be inferred from examples.

 

 

 

Validate JSON Against its Schema

Having created your schema, and added it to the Swagger/RAML definition, it’s time to check that any JSON intended for use is valid. My favoured tool is;

JSON Schema Validator

 

This is yet another two-panel tool. The left-hand panel holds the schema, the right-hand, the JSON.

Unit Testing

Here, we are looking at testing parts of an API’s implementation. My experience comes from the world of integration platforms, though the principles should be relevant to a back-end API.

The open-source tool here is JUnit (in Eclipse). The aim of this section is not in the use of JUnit itself, but the value that can be added by its judicious use.

All the following scenarios should be integrated into the build, so that they are always run.  In most cases this will be the only place that tests them.

Schema Validation

Use JUnit to create tests that verify XML and JSON against their schemas. As well as valid tests, include ones to check that the schema is picking up errors in the JSON and XML.

Transformations

The processing in Integration Platforms almost always involves transforming all or part of the payload. The most commonly used transformation is XML’s XSLT. In a REST/JSON world, this is commonly used to transform XML (from a SOAP back-end response) into JSON. However, XSLT doesn’t really help when transforming from JSON (to JSON or to XML).

 

From XML

JUnit

This should be used for running tests of XSLT transformations. Ideally, all valid combinations ‘of’ input should be transformed and the result compared with the expected result. Specific invalid combinations should also be included.

From JSON

I am not aware of any open-source technologies like XSLT, that transform from JSON. There are proprietary tools from MuleSoft (‘Dataweave’) and IBM’s API-Connect (‘Data Mapping’), which both feature a very similar-looking drag-and-drop facility, that generates mapping statements for items having a visual connection between input and output fields.

WSO2’s ESB has ‘hidden’ support, in that it converts JSON into an XML equivalent under the covers. This allows XSLT to be used to transform JSON input to XML (or JSON).

Note: Given the lack of an XSLT for JSON, I have very recently dusted off my knowledge of Groovy (see Introduction to Groovy), and successfully used that to transform JSON (for the ‘rain’ app), that was beyond my skill-level in DataWeave (and possible beyond DataWeave, anyway).

The use of Groovy for this purpose will be documented in a related paper, as its presence here was veering somewhat ‘off message’.

 

Conclusion

In conclusion, we have looked at the following products covering API Definition:

• JSON Editor Online (creating JSON)
• JSON Formatter and Validator and JSON Lint (formatting and validation)
• JSON Lint (syntax Validation)
• JSON Schema and Schema Guru Web (schema generation)
• JSON Schema Validator (validating JSON against its schema)
• Using JUnit for running unit tests during a build for:
  • validation of JSON and XML against a schema
  • transformations to/from XML and JSON

Some questions for the reader:

• Have I missed out some useful aspect of the tools covered?
• Do you have any positive or negative comments on the tools I have covered here?
• Are there any other open-source tools, in this area, that you use and would recommend, and why?

In Part 2 of this series, I will be covering the use of Test Clients for investigating and testing API calls:

• the Chrome app Postman
• the Firefox add-in RESTClient