Introduction
In part 1 of this series, I covered my experience with:
- API Definition
- Unit Testing.
In part 2, I covered:
- Calling APIs: Test Clients
This will be followed by two more instalments:
In this (part 3), I cover:
- Function and Load Tests
- SoapUI
- JMeter
This will be followed by the final instalment:
- Part 4
- Mocking back-end services
Some of the tools were recommended by colleagues on projects, some I discovered and the rest were mandated for use on a particular project.
Function and Load Tests
These tools allow you to define a set of calls (REST and SOAP), each with its own URL, header values and payload body. For each call, you can specify a variety of expected values in the response, typically: the http status, response headers and the payload body. A set of calls can also be set up to send requests at defined message rates, so as to check response times. The two tools that I have used in this way are:
- Repeatable and volume tests
- SoapUI
- JMeter
I noticed that Postman now has the ability to perform at least some of the above functions, which makes it worth investigating.
SoapUI
From SmartBear. This is probably the most well-known of this kind of testing aid. It has a ‘Pro’ version, though I have only used the free version. It has a very old-fashioned MDI user interface with lots of pop-up windows used when defining tests – it takes a time to become familiar with it. Some things are easily forgotten if you don’t use it for some time. You also have to make lots of double-clicks – so beware RSI. On the plus side, it does do the job well and is robust and reliable.
In view of the number of intermediate pop-up windows, some of these will be omitted to reduce the number of images in this description.
Create TestCalls for an API
Create a new REST Project and provide the URL including any parameters.
Rename the request
Run the request by pressing the red > arrow on the left of the top line
You can see the JSON response, the headers and response time.
Create Test Criteria
The call is working. What is lacking is the ability to rerun the test and ensure that it works and returns the same values. This is done by adding the call to a Test Case.
This creates a new window for running and verifying this API call.
Add Assertions.
These are the expected results of a call.
Check that the http status code is 200
You can check for expected values in the response using JsonPath (if you want help using this see: JSONPath Online Evaluator. This example is checking the value returned for longitude.
The resulting check
It is a very good practice to give each assertion a descriptive name – this makes it much easier for other people to see what is being tested. (I worked on a project, where is a quiet time, we renamed hundreds of assertions)
Performance Checks
With SoapUI you can add Load Tests. The example uses the default values for number of threads and the test duration. Note: be careful when you are calling a public API that your load tests do not use reach the limit on call frequency and / or call count.
Different calls can be grouped in a Load Test, this example only shows the one we have been using
Note that you can define assertions for the load-test results, which can be one of the following’:
At the end of the run, you will see the following statistics
JMeter
Apache JMeter – yet another product from the Apache Foundation.
I have only used this for a short time. A project on which I was working recently, decided to switch all new testing from SoapUI to JMeter, as well as some of the existing largest tests. Reasons given were:
- Tests run significantly faster, with a lower processing overhead
- Required to make a significant reduction in the testing stage of builds
- Whilst, idiosyncratic, its user interface is easier to use than SoapUI
- Test suites can be defined as csv files – enables bulk definition, bypassing the UI.
Whilst JMeter is fully documented, most of the documentation is function-based rather than usage-based (few how-to descriptions). This makes it harder to get started, but once you have started, all facilities are defined, but some lack examples.
Create TestCalls for an API
Here we are going to re-create the same functionality as in SoapUI to call and verify the same REST API
Create a Test Plan
The starting point in JMeter is a Test Plan. Whenever you start JMeter you will see an empty Test Plan
Provide a meaningful name and comments
Create a thread Group
A Test Plan comprises one of more Thread Groups. For a functional test, we need a Thread Group with only a single thread and a single invocation
Provide a meaningful name and comments
Create an HTTP Request
Add a Sampler of type HTTP Request.
Note: do not include ‘http://’ in the Server Name or IP entry field. Port Number and Protocol can be empty and default to the values shown. Any Parameters and Body Data can be defined using the tabs shown
Add a View Results Tree
To see the response details in an equivalent way to SoapUI, add a View Results Tree
Call the API
Press the > Start button.
In the View Results Tree, select the name of the HTTP Request in the left-hand panel to see the results
Select the Response data tab to see the JSON payload
Create Test Criteria
The call is working. What is lacking is the ability to rerun the test and ensure that it works repeatedly and returns the same values. This is done by adding assertions.
Add a Response Code Assertion
This assertion will check that the HTTP response code is 200
To specify the 200 code, press the Add button as shown
Verify that it works- press the > Start button again
Force a temporary error, by changing postcode to one that does not exist.
The response code is 404, not 200
The HTTP repose assertion now appears as a failure
Add an Assertion for the JSON Response
Checking the value returned for say, ‘latitude’, is a two-stage process.
Firstly, the value must be extracted into a JMeter variable.
This is done using a JSON Extractor. Add one – it must be a child of a ‘Sampler’ (HTTP Request)
- Variable names: make sure that they are all unique
- JSON Path expressions: as required – this finds ‘latitude’
- Match Numbers: if there are more than one value in the JSON Path result, (an array, for example), use the ‘nth’ value.
- Default Values: returned if the path expression fails to find the value
Secondly, create a new assertion that will check the value extracted above against the expected value.
Add a new Response Assertion, that compares the value returned in the latitude_Response variable with the expected result.
Test it.
Force a failure by changing the expected value:
Restore the correct expected result.
Resequence the Test Components
You can use drag-and-drop to create a preferred sequence under the HTTP Request. Let’s place the Assertions together – drag the Latitude one above the JSON Extractor.
>>
Whilst, JMeter appears to be more ‘bitty’ than SoapUI, this in fact offers several advantages, two of which can be of immediate benefit:
- The latitude_Response variablecan be reused anywhere in the thread group.
- The HTTP 200 response assertion could be dragged from under the HTTP Request and to the Thread group, as a common (reusable) assertion
Performance Checks
In reality, you would create separate Thread Groups for running load tests. For the purpose of this demonstration, we will modify the Thread Properties.
Add a Response-Time Graph
Configure some Settings
Modify the Thread Group Properties
Disable the Results Tree and JSON payload assertion checks, to remove their overhead, but keep the response code check.
Use same values as in SoapUI – for comparison.
Add A View results in Table Listener
To see the results of the load test, we need to add one or more listeners that will process them. This one presents a summary of the results – it is similar to that from SoapUI
Run the Thread Group
If you compare these values with SoapUI, it looks as though JMeter can get through more work in the elapsed time, with lower overheads.
Conclusion
I hope you have found something of use and interest in how I have shared my experiences of using these tools, and will investigate and use them yourself.
A Bit More
I recently came across this article Top 6 API Testing Tools In 2017: REST & SOAP. I have included the link here as it’s relevant to the topic. The first product in the list is SoapUI, Postman is also included. As for the other four, I haven’t used them, so I can’t comment.