Introduction

In part 1 of this series, I covered my experience with:

  • API Definition
  • Unit Testing.

In part 2, I covered:

  • Calling APIs: Test Clients

This will be followed by two more instalments:

In this (part 3), I cover:

  • Function and Load Tests
    • SoapUI
    • JMeter

This will be followed by the final instalment:

  • Part 4
    • Mocking back-end services

Some of the tools were recommended by colleagues on projects, some I discovered and the rest were mandated for use on a particular project.

Function and Load Tests

These tools allow you to define a set of calls (REST and SOAP), each with its own URL, header values and payload body. For each call, you can specify a variety of expected values in the response, typically: the http status, response headers and the payload body. A set of calls can also be set up to send requests at defined message rates, so as to check response times. The two tools that I have used in this way are:

  • Repeatable and volume tests
    • SoapUI
    • JMeter

I noticed that Postman now has the ability to perform at least some of the above functions, which makes it worth investigating.

SoapUI

From SmartBear. This is probably the most well-known of this kind of testing aid. It has a ‘Pro’ version, though I have only used the free version. It has a very old-fashioned MDI user interface with lots of pop-up windows used when defining tests – it takes a time to become familiar with it. Some things are easily forgotten if you don’t use it for some time. You also have to make lots of double-clicks – so beware RSI. On the plus side, it does do the job well and is robust and reliable.

In view of the number of intermediate pop-up windows, some of these will be omitted to reduce the number of images in this description.

Create TestCalls for an API

Create a new REST Project and provide the URL including any parameters.

Rename the request

Run the request by pressing the red > arrow on the left of the top line

You can see the JSON response, the headers and response time.

Create Test Criteria

The call is working. What is lacking is the ability to rerun the test and ensure that it works and returns the same values. This is done by adding the call to a Test Case.

This creates a new window for running and verifying this API call.

Add Assertions.

These are the expected results of a call.

Check that the http status code is 200

You can check for expected values in the response using JsonPath (if you want help using this see: JSONPath Online Evaluator. This example is checking the value returned for longitude.

The resulting check

It is a very good practice to give each assertion a descriptive name – this makes it much easier for other people to see what is being tested. (I worked on a project, where is a quiet time, we renamed hundreds of assertions)

Performance Checks

With SoapUI you can add Load Tests. The example uses the default values for number of threads and the test duration. Note: be careful when you are calling a public API that your load tests do not use reach the limit on call frequency and / or call count.

Different calls can be grouped in a Load Test, this example only shows the one we have been using

Note that you can define assertions for the load-test results, which can be one of the following’:

At the end of the run, you will see the following statistics

JMeter

Apache JMeter – yet another product from the Apache Foundation.

I have only used this for a short time. A project on which I was working recently, decided to switch all new testing from SoapUI to JMeter, as well as some of the existing largest tests. Reasons given were:

  • Tests run significantly faster, with a lower processing overhead
  • Required to make a significant reduction in the testing stage of builds
  • Whilst, idiosyncratic, its user interface is easier to use than SoapUI
  • Test suites can be defined as csv files – enables bulk definition, bypassing the UI.

Whilst JMeter is fully documented, most of the documentation is function-based rather than usage-based (few how-to descriptions). This makes it harder to get started, but once you have started, all facilities are defined, but some lack examples.

Create TestCalls for an API

Here we are going to re-create the same functionality as in SoapUI to call and verify the same REST API

Create a Test Plan

The starting point in JMeter is a Test Plan. Whenever you start JMeter you will see an empty Test Plan

Provide a meaningful name and comments

Create a thread Group

A Test Plan comprises one of more Thread Groups. For a functional test, we need a Thread Group with only a single thread and a single invocation

Provide a meaningful name and comments

Create an HTTP Request

Add a Sampler of type HTTP Request.

Note: do not include ‘http://’ in the Server Name or IP entry field. Port Number and Protocol can be empty and default to the values shown. Any Parameters and Body Data can be defined using the tabs shown

Add a View Results Tree

To see the response details in an equivalent way to SoapUI, add a View Results Tree

Call the API

Press the > Start button.

In the View Results Tree, select the name of the HTTP Request in the left-hand panel to see the results

Select the Response data tab to see the JSON payload

Create Test Criteria

The call is working. What is lacking is the ability to rerun the test and ensure that it works repeatedly and returns the same values. This is done by adding assertions.

Add a Response Code Assertion

This assertion will check that the HTTP response code is 200

To specify the 200 code, press the Add button as shown

Verify that it works- press the > Start button again

Force a temporary error, by changing postcode to one that does not exist.

The response code is 404, not 200

The HTTP repose assertion now appears as a failure

Add an Assertion for the JSON Response

Checking the value returned for say, ‘latitude’, is a two-stage process.

Firstly, the value must be extracted into a JMeter variable.

This is done using a JSON Extractor. Add one – it must be a child of a ‘Sampler’ (HTTP Request)

  • Variable names: make sure that they are all unique
  • JSON Path expressions: as required – this finds ‘latitude’
  • Match Numbers: if there are more than one value in the JSON Path result, (an array, for example), use the ‘nth’ value.
  • Default Values: returned if the path expression fails to find the value

Secondly, create a new assertion that will check the value extracted above against the expected value.

Add a new Response Assertion, that compares the value returned in the latitude_Response variable with the expected result.

Test it.

Force a failure by changing the expected value:

Restore the correct expected result.

Resequence the Test Components

You can use drag-and-drop to create a preferred sequence under the HTTP Request. Let’s place the Assertions together – drag the Latitude one above the JSON Extractor.

>>

Whilst, JMeter appears to be more ‘bitty’ than SoapUI, this in fact offers several advantages, two of which can be of immediate benefit:

  • The latitude_Response variablecan be reused anywhere in the thread group.
  • The HTTP 200 response assertion could be dragged from under the HTTP Request and to the Thread group, as a common (reusable) assertion

Performance Checks

In reality, you would create separate Thread Groups for running load tests. For the purpose of this demonstration, we will modify the Thread Properties.

Add a Response-Time Graph

Configure some Settings

Modify the Thread Group Properties

Disable the Results Tree and JSON payload assertion checks, to remove their overhead, but keep the response code check.

Use same values as in SoapUI – for comparison.

Add A View results in Table Listener

To see the results of the load test, we need to add one or more listeners that will process them. This one presents a summary of the results – it is similar to that from SoapUI

Run the Thread Group

If you compare these values with SoapUI, it looks as though JMeter can get through more work in the elapsed time, with lower overheads.

Conclusion

I hope you have found something of use and interest in how I have shared my experiences of using these tools, and will investigate and use them yourself.

A Bit More

I recently came across this article Top 6 API Testing Tools In 2017: REST & SOAP. I have included the link here as it’s relevant to the topic. The first product in the list is SoapUI, Postman is also included. As for the other four, I haven’t used them, so I can’t comment.