API testing is a fundamental component of software quality assurance, ensuring the reliability and functionality of an application’s API (Application Programming Interface). APIs typically serve as the communication bridge between the user facing front-end of a mobile app or website and a data source like a database, facilitating the seamless exchange and processing of data. APIs can also be used to exchange data with other APIs, systems, and applications without a user interface. The primary goal of API testing is to validate that the API functions as intended, adheres to specifications, enforces business rules, and interacts seamlessly with other software components. The testing approach encompasses various types of testing, including contract testing, integration testing, functional testing, performance and load testing, and security testing. API testing poses challenges stemming from the unique characteristics of APIs and the complexity of modern software systems. The absence of a user interface requires testers to rely on documentation and specialized tools, complicating the visual inspection of system behavior. Handling diverse data formats, such as JSON and XML, adds complexity, as does the reliance on external services and third-party integrations.

There are many guides available for API testing on the internet but many of them only cover theory and high level concepts with precious little context or practical application. This series on API Testing will explore each aspect of API testing in detail, provide real world examples, and discuss special points of interest that only comes with decades of API testing practice. Further, this guide stands as a specification for how Onward Source approaches API testing for the clients we work with.

A word about API applications

APIs can be intimidating. They can seem like they are a nebulous entity that receives requests and magically returns data. In actuality they are like any other computer program just like a desktop application or mobile app. With an API, the interface is data instead of a UI. And just like any other application an API takes in input, process it, and provides output. In this case the input and output is data. API applications typically run on a server and can be written in any one of a myriad of programming languages like Java, Python, Ruby, PHP, or increasingly more common, Javascript. APIs are packaged into an executable file like any other computer program and require system resources like CPU, RAM memory (APIs written in Java are notorious memory hogs), and storage memory, the same as any other application. These resources are exhaustible as the API application runs and should be monitored during testing and investigated whenever an especially perplexing bug is isolated. As we discuss the different aspects of API testing in this series we will make a point to discuss the touch-points each has with the API application itself.

OK, but what are microservices and how do they change the approach to API testing?

Microservice architecture is a design pattern whereby the different functions of the API are broken up into separate applications. Consider the two API design diagrams below.

The traditional API application design pattern combines all the API’s methods into a single application, making one single monolithic API application. This monolithic API architecture would then be installed on as many servers as needed to serve the amount of traffic needed. Any change to any one method in the monolithic API would necessitate a regression test of all APIs in the application, possibly requiring considerable time and slowing the release cadence. The right-hand diagram shows an example of the microservice API architecture. With the microservice API architecture an individual API application serves only a handful of methods. This API design allows for much more flexible server deployments where only the most utilized API application needs to be scaled to multiple servers and the least used can use a separate scaling schema. This architecture also lends itself to faster releases as only the API application and its respective methods that were changed need to be tested.

API Testing Tools

API testing tools can be as simple or as complex as needed for the context of the API that is being tested. Some tools offer a simple UI interface while others are code based. In some cases and specialized applications may be required to test an API that is coded from the ground up in a traditional programming language like Java or Python. Here are some of the tools typically used for API testing:

• Postman
  • Postman is a browser-based UI platform for testing APIs. Users can enter the endpoints, define the headers, provide parameters, and payloads to execute an API request and see the response. User can organize their API requests into collections and share them with other users.
• Hoppscotch
  • Hoppscotch is similar to Postman but can be installed locally or on your own server for browser-based access. This gives you all the benefits of Postman without all the disadvantages of cloud based products you have no control over.
• Swagger
  • Swagger is another browser-based API testing solution. The difference with Swagger is that the API requests are mostly set up automatically using a JSON definition of the API. Development teams may provide Swagger endpoints which list all the available API methods that can easily be accessed.
• Insomnia
  • Insomnia is another browser-based API testing solution that offers a more simplistic and easy to use design.
• SoapUI
  • SoapUI is a free desktop API testing tool that has a long history in the software QA community. SOAP based APIs can be tested with this tool.
• ReadyAPI
  • Ready API is a commercial tool for API testing offering functional testing, performance testing, and other tools.
• RestAssured
  • RestAssured is a Java library for testing APIs making this a code-based solution to API testing without a UI. Code-based API solutions may appeal more to those looking for a build runtime or CI/CD testing solution.
• Karate DSL
  • Karate DSL is another code-based API testing solution built on the Cucumber library with test cases defined in Javascript. Code-based API solutions may appeal more to those looking for a build runtime or CI/CD testing solution.
• JMeter
  • JMeter is a desktop/server application for API functional and load testing. JMeter provides a UI for creating tests which can then be run from either the UI or command line. JMeter tests can be set up to access external data sources, implement flow controls and logic, and run custom coded assertions.
• K6
  • K6 is a code-based API load testing application. Scripts are written in Javascript.
• Selenium, Cypress, Playwright
  • Selenium, Cypress, Playwright are popular UI test automation tools that can also be used to code-based API testing. If you have an existing UI test framework using the same solution for API testing may provide an advantage.

API Testing Types

Let’s get a brief description of the different types of testing that are executed during API testing. The rest of this API testing series of blog posts will dive deeper into each of these. It is important to understand that each of these types do not live in isolation and there can be some overlap in the testing and the kind of tests you actually use for each. For instance you may run a functional test to verify an integration is working. These types of tests define the objective, not necessarily the process.

Integration Testing

Before beginning more in-depth types of testing it is best to verify that all the different sub-systems and components involved in the API are communicating with each other. This can include verifying that the API is communicating with a database and other subsytems. Some of these subsystems can include:

• Databases
• Authorization systems
• Third party data providers
• Email Handlers
• Payment Providers
• Logging and Error Handling Systems
• Caching Schemes
• Scheduled Job Execution
• Queuing Services

During integration testing each integrated support service that works on behalf of the API will be tested in isolation to verify it is operating as the API needs it to. Without first testing these sub-systems we cannot assume that any other part of the API will function correctly.

Contract Testing

Contract testing includes testing how a client will communicate with the API and how the API will send data back to the client. Within the context of an API think of a “contract” as just a definition of how requests will be sent to the API and how responses will sent to the client.

Consider the contract for a webpage. The contract for a browser sending a URL to a server in a request for a webpage defines that the request will be sent with the ‘http’ protocol, include a domain name, and a path/filename. The contract further defines that the response will be a HTML document with tags for ‘html’ encapsulating ‘head’, ‘title’, and ‘body’ tags.

API Request contracts will define the following properties

• HTTP methods (POST, GET, PATCH, DELETE)
  • A definition of which HTTP methods are supported by the API.
• Endpoint URLs and paths
  • The domain the API can be accessed at as well as the path to access the API
• Headers
  • The HTTP headers that are required by the API for the request to be served. This can include authentication headers, content-type, and user agent data.
• Parameters
  • The listing of parameters and data types that the API requires for requests to be processed.
• Post body format
  • The specification for the data that is being sent to the API such as if it requires XML or JSON and the schema of the data with respect to parent and child objects.

API Response contracts will define the following properties

• HTTP status codes
  • Numeric codes that will be included in the HTTP Status header that define the response contents. Some examples include 200 for a successful response or 500 for a server error.
• Headers
  • A listing of response headers such as content-type and cache status information
• Response Body
  • The specification for the data that is being sent by the API such as if it is XML or JSON and the schema of the data with respect to parent and child objects.

The objective of contract testing is to verify that each method of the API conforms to the contract so that clients can reliably communicate with the API. This testing lays the foundation for all testing that is to come and is the most basic form of functional testing. Many QA organizations may stop their functional testing here and may not continue into more exhaustive testing of data testing, business rule processing, or permutation testing.

Functional Testing

Functional testing will constitute the bulk of the testing performed during API testing. Functional testing includes testing each major function of the API including the create, read, update, and delete functions, commonly referred to as CRUD functions. These functions will also be tested against the data caching policies that are employed to be certain the cache hit, cache miss, and data change cases are correctly handled.

During functional testing it can be useful to approach the API under test as if it were a UI application. With a UI application, each interface control will be tested along with each control value. Think of a UI dropdown control having 5 different options that the user can select from, and that selecting each option results in a different operation being initiated on the UI application. This can be synonymous with a API parameter having an enumerated list of 5 options, each of which initiates the processing of different business rules that are reflected in the API response. Just as with the UI application, we must test each of the 5 parameter options during our API test.

Some of the functional API testing activities include:

• Parameter testing
  • Testing each request parameter and parameter value. For instance if a parameter supports an enumerated list of values then each values will be tested.
• Response testing
  • Similar to the request parameter testing, each response field will be tested including verifying that each request value is included in the response.
• Data Validation
  • During this testing we check to verify that the data that is returned in a response is correct. This may include comparing the API response data with the source data such as the database that the data is sourced from. The objective of this verification is to confirm that the data in the database and the application of any business rules produces the same data that is in the API response.
• Error Handling
  • Error handling testing involves sending invalid requests to the API to verify that it properly handles the requests. The types of invalid requests may include unauthorized, improperly formatted, improperly encoded, or requests that violate business rules.
• Boundary Testing
  • Boundary testing is testing to verify that parameter values near or over a limit are properly handled by the API. For instance if an API parameter takes a value of 0 – 100 we will test that the correct response is provided when value ‘0’ and ‘100’ are sent. If an API accepts a CSV file as input with a limit of 50,000 records then we test with a CSV file with 50,000 records in it. We also test response fields with regard to boundary testing; verifying that given response field will return values at the minimum and maximum scopes.
• Cache Policy Testing
  • Cache testing may or may not be required depending on if the API is caching any API responses on the server. An API may cache responses to increase performance and reduce the load on the database. API responses that retrieve data that does not frequently change are good candidates for caching. Testing involves making a request to an API (HTTP GET method) that caches the response and then repeating the request to verify that the data is cached and retrieved from the cache correctly. Further testing would involve updating or deleting the data that is served in the response and then making the GET request again to verify that the API serves the updated data or does not respond with the record if it was deleted.
• Performance Testing
  • During functional testing we will get our first look at performance and load testing. Together with the product owners and development team a standard response time will need to be defined. Then during functional testing we will evaluate the API calls with different parameter values, responses, and business rules to verify that the API meets the response time requirements. The APIs must meet their performance requirements as a prerequisite for load testing to be performed.

These are just a few of the functional testing activities than can be performed. We will explore each of these in more depth, as well as some others, in a future blog post.

Load Testing

Load testing involves executing multiple simultaneous API requests to simulate usage by multiple users of a client application that is using the API. The objective of this testing is to discover if the API can serve multiple simultaneous requests and still remain performant and not suffer any functional degradation. This testing is also used to test any automated scaling policies put in place on the server to increase the number of API application instances when traffic increases during peak usage.

To execute this testing, a test script is created in an application called JMeter that simulates multiple simultaneous users. JMeter is an open source application maintained by the Apache Software Foundation and can be run on Windows, Mac, and Linux. It includes a front-end UI application for the creating of the load test script which is then typically run by command line from a host system for the load test.

The product owners and development team will need to be engaged to determine what load will be generated during the load test. The process for setting a target load is as follows:

• The user or business flows to be included in the load test are determine and set. These can be flows such as first time user experience (FTUE), user registration, general search, navigation, and usage, checkout, and payment flows.
• Discover which APIs are called during each of these flows.
• Determine how many API requests per second are generated by a typical user executing these flows.
• Determine the the expected amount of simultaneous users that are expected to execute these flows at peak usage periods.
• Once the JMeter script has been created that simulates the API usage from the flows that we defined in the first set, use JMeter to simulate the number of users (called ‘threads’ in JMeter) that generate the expected number of API requests per second by the number of simultaneous users making API requests at peak usage

Another way of determining the target load is to investigate server logs from past API usage in a production environment to determine the number of requests per second that were generated and then set up JMeter to reproduce that load.

The load test will run continuously for some length of time during which the server logs and resources will be monitored. API response times, any errors, change in API application health, or server resources are noted and reported in the load test report. The client application may also be intermittently, manually, tested while the API load test is in progress to assess the user experience under load.

Security Testing

API security testing is a crucial aspect of ensuring the integrity and protection of data and services exposed through APIs. API security testing involves evaluating the security measures implemented in an API to identify vulnerabilities and weaknesses. Some of the security testing methods include:

• Authentication and Authorization Testing
  • Verification of the authentication process for API access. This can include the testing of authentication tokens, session tokens, and expiration of tokens.
• Input Validation
  • Input validation testing may focus on verifying that the API is resistant to SQL injection and other injection attacks.
• Rate Limiting
  • Verifying that the API detects excessive requests coming from a single source and appropriately limits the sender from making excessive requests to the API.
• Logging and monitoring
  • Investigating any logging and monitoring policies that are put in place to ensure that they can detect and report attacks on the API.

What’s Next?

In our next blog post in this series we will discuss Integration Testing in more detail including the methods for testing and some of the tools used.