Ad hoc using and testing HTTP APIs

If you are a web developer like me, you always work with and test HTTP APIs. During the development of new endpoints, it is often fastest to just write an automated test, e.g., with pytest.

But there are other situations when this is not desirable or needed. Sometimes we want to verify some behavior quickly, experiment with API design, test endpoints without having an automated test suite, or even interact with third-party APIs. While today HTTP clients have many advanced features, in this post, I want to focus on three main use cases:

• Constructing HTTP API requests quickly
• Combining multiple requests (e.g., when we need first to authenticate and obtain a token using another endpoint)
• Saving requests for later reuse

To do that, I will explore four main categories of HTTP clients:

Command line tools

While there are others, curl and HTTPie are probably the most used tools in the terminal for sending HTTP requests. While curl is ubiquitous, HTTPie has a nicer output and shorter, more straightforward commands. The basic usage is on the command line in interactive way:

> http https://jsonplaceholder.typicode.com/todos/1

With just using the HTTPie command name (http) in front of an URL, we will get a colorized output and response headers:

Sending multiple requests is trivial in shell. All we have to do is to issue more commands. But how to take data from one request to another? The secret to request chaining is to parse the response and store it in a shell variable.

Let's look at another example, where I am passing a username and password to an authentication endpoint to get a JSON Web Token and use that token in the subsequent request. Notice that in HTTPie it is enough to define just the port :8000 if the target is localhost, set a different HTTP method (POST) as a second argument, and describe the top-level JSON body by separate key=value arguments:

> USER=petr@example.com
> PASSWD=test
> JWT=$(http POST :8000/api/user/token-auth/ username=$USER password=$PASSWD --print b | jq '.token' --raw-output)
> http http://localhost:8000/api/groups/ authorization:"JWT $JWT"

The above example assumes that the token is returned in the response JSON as a top-level property. First, I told HTTPie to output only the response body without the headers (--print b), pass the output to the JSON command-line parsing tool jq using the pipe (|), and extract the top-level token property in the JSON file. --raw-output will discard the quotation marks. Finally, the special syntax $( ) will ensure that the whole command inside it is run first, and its result is passed to the top-level command that assigns it to a shell variable. We can use the variable in the next endpoint call ($JWT).

If we take all the commands together and put them in a file, we can even create a simple shell script that could be run repeatedly:

#!/usr/bin/bash

USER=petr@example.com
PASSWD=abrakadabra
JWT=$(http POST :8000/api/user/token-auth/ username=$USER password=$PASSWD --print b | jq '.token' --raw-output)

http http://localhost:8000/api/groups/ authorization:"JWT $JWT"

Combining results like we just saw is a bit more complex. However, it can be helpful to end up with a script for future reuse.

Standalone GUI clients

Traditional GUI HTTP clients like Postman or Insomnia are usually a good start for using and testing APIs. The significant advantage over other tools is that we can typically use an existing API specification, like OpenAPI, to load a collection of API endpoints and use them immediately with only minor changes. When the API spec is unavailable yet or cannot be imported (e.g., due to an error in the spec), requests can still be crafted by hand in a visual interface.

Compared to other approaches, these clients also offer built-in organization of endpoint calls into collections and request chaining, as we will see in the following examples demonstrating the use of Insomnia.

Variables everywhere

One cool thing about Insomnia is that we can manage multiple environment configurations and define custom environments that can be reused everywhere, e.g., in the URL definitions, in request bodies, or headers. Clicking on "Manage environments" from the top-left context menu for environments or pressing CTRL + E will open a configuration like this:

Once variables are defined, and the correct environment selected, many text fields will allow us to use the variables by typing _.<variable name>. Start by typing _., then hitting Ctrl + Space, and Insomnia will offer the variables available. You will then notice that the variable is recognized by a purple box, like here in the endpoint URL:

Request chaining

Insomnia has built-in request chaining that can be used similarly to a variable. When we start typing response inside one of the text fields, the program will offer several options, with probably the most important one called Response -> Body attribute that can take a piece of data from another request's response. If we, for instance, need first to grab a token from another request's JSON body and use it as a JWT token header, we can do that this way.

In the first go, Insomnia will create a placeholder that can be clicked on to open a configuration popup:

JSONPath can be used to grab a property of the response JSON. For example, $.token will grab a top-level JSON property called token.

If we configure JWT token property like that, we can make authenticated request together with the authentication request in one go:

curl commands

As we discussed, we can utilize curl commands in our API testing on the command line. Insomnia has two neat features that can transform the testing in and out of the command line:

• Copy as curl: The endpoint context menu offers an option to copy a request as curl, which can be useful when sending the request to another person quickly or continuing the invocation from the command line. Learning how to make curl requests is also helpful, as we can create them in Insomnia and convert them later.
• Accept curl commands: Going back from a curl request to Insomnia is also possible! To do so, copy and paste your curl command into the URL text field, and Insomnia will parse it.

Other features

It is fair to say that besides ad-hoc API usage and testing, traditional GUI clients often contain other features like request history, sharing predefined requests across a team, running automated test suites, or tools for designing new APIs. But even for simple API invocations, the abilities are pretty good!

Scratchpad-like tools

Scratchpads are typically single-file environments to experiment and play. In software development, scratchpads typically involve writing and executing code, but there are some for sending HTTP requests as well. The main ones are REST Client for Visual Studio Code and the native IntelliJ HTTP Client.

Both are excellent alternatives to previous approaches. I am mainly a VS Code user, so I will show examples using the former. Similarly to the command line tools, it is possible to define requests using a concise text syntax, storing requests in text files for easy modification and copying and pasting requests and responses, but this time all inside a code editor.

Let's look at how a sample POST request would look in the REST Client:

POST http://localhost:8000/api/user/token-auth/
content-type: application/json

{
  "username": "petr@example.com",
  "password": "test"
}

The request is defined using the desired HTTP method at the beginning, the URL, headers below, and the body at the end. All we need to do is place it in a file ending with .http extension, and the client will show "Send Request" button above it to send it:

The response will open in a side panel (with image support). Instead of REST Client request format, we can alternatively use a curl command call that would otherwise be needed to be executed on the command line. The request body could also be loaded from a file instead (refer to the documentation).

Multiple requests can be combined in one file by separating them with ###. It is also possible to define and reuse variables. For instance, we can define a value for the JWT token and use it later in multiple requests. If the token needs to be changed, there is no need to modify all the requests (notice the additional Authorization header):

# comments can be used in the file when the line begins with #

# define a variable for JWT token:

@jwt = eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6InBldHJAZXhhbXBsZS5jb20iLCJpYXQiOjE2NjE0MjgwMjcsImV4cCI6MTY2MTQzMTYyNywianRpIjoiN2Q5M2Q5ZGItYTEyNy00YzNlLWE0NWItZmI4ZWMzMzRiNGU2IiwidXNlcl9pZCI6MSwidXNlcl9wcm9maWxlX2lkIjpbMV0sIm9yaWdfaWF0IjoxNjYxNDI4MDI3fQ.YjV-vUquH9YOu9-FQ8MDJzOZQyjXHwCzF4ZVUAKwc98

POST http://localhost:8000/api/user/token-auth/
content-type: application/json

{
  "username": "petr@example.com",
  "password": "test"
}

###

# second request can be placed in the file after ###

POST http://localhost:8000/api/database/rows/table/315/batch/
Content-Type: application/json
Authorization: JWT {{jwt}}

{
    "items": [
      {
        "field_2621": [
          {
            "id": 1
          }
        ]
      }
    ]
}

Besides custom variables, environment variables or prompt variables can be used too.

REST Client also supports various authentication protocols, GraphQL, has shortcuts, remembers cookies for subsequent requests, can display request history, and even generate code snippets for making requests in languages like Python or JavaScript.

Web browser tools

There are two main features from Firefox (or Chrome) development tools that can help us to send HTTP requests, which both built upon requests that we already made in the web browser previously:

• Edit and resend
• Copy

Edit and resend

Edit and resend feature found in Firefox development tools is useful when we need to issue a slightly different request while preserving all the headers and cookies. To do so, open the Network tab in the development tools, find a request you want to modify, and use the context menu to pick "Edit and resend". Once selected, we will be able to modify the request body, headers, or the HTTP method and send it again:

Copy

Copy feature that is available in both Firefox and Chrome can transform an issued request to another format, for example to create a curl command that can be invoked on the command line:

Fin

Sending HTTP requests is an essential skill for web developers, and knowing how to do it efficiently pays off. Different situations require different approaches, but for me lately, REST Client for VSCode or the request chaining in Insomnia have been game changers! Let me know on Twitter which one is your favorite!

Last updated on 27.12.2022.