Author: Scott Chamberlain
Date: 2024-07-18

# 6. Choosing a HTTP request class

#### Scott Chamberlain

#### 2024-07-18

There are a number of different crul classes that do HTTP requests. The following table compares features across the classes.

Class | HTTP verbs | Asynchronous? | Packges using
--- | --- | --- | ---
HttpClient | all | no | bcdata,chirps,duckduckr,gfonts,mindicador,nsapi,tradestatistics,viafr
Paginator | all except retry | no | —
ok | head,get | no | dams
Async | all except retry | yes | rdryad,rnoaa
AsyncVaried | all except retry | yes | rcrossref,taxize,mstrio
AsyncQueue | all except retry | yes | —

HttpClient is the main class for doing synchronous HTTP requests. It supports all HTTP verbs including retry. It was the first class in this package. See also `crul::ok()`, which builds on this class.

Paginator is a class for automating pagination. It requires an instance of `HttpClient` as it’s first parameter. It does not handle asynchronous requests at this time, but may in the future. `Paginator` may be the right class to use when you don’t know the total number of results. Beware however, that if there are A LOT of results (and a lot depends on your internet speed and the server response time) the requests may take a long time to finish - just plan wisely to fit your needs.

ok is a convienence function light wrapper around `HttpClient`. It’s use case is for determining if a URL is “up”, or “okay”. You don’t have to instantiate an R6 class as you do with the other classes discussed here, but you can pass an `HttpClient` class to it if you like.

With Async you can make HTTP requests in parallel. `Async` does not at this time support retry. It’s targeted at the use case where you don’t mind having request settings the same for all requests - you just pass in any number of URLs and all requests get the same headers, auth, curl options applied, if any.

AsyncVaried allows you to customize each request using `HttpRequest` (See below); that is, every HTTP request run asynchronously can have its own curl options, headers, etc.

AsyncQueue is the newest class, inheriting from `AsyncVaried`, introduced in curl v1. The motivation behind this class is: sometimes you may want to do HTTP requests in parallel, but there’s rate limiting or some other reason to want to not simply send off all requests immediately. `AsyncQueue` sets up a queue, splitting up requests into buckets, and executes requests based on a `sleep` or `req_per_min` (requests per minute) setting.

## HttpRequest

`HttpRequest` is related here, but not in the table above because it doesn’t do actual HTTP requests, but is used to build HTTP requests to pass in to `AsyncVaried`. The simplified class `Async` relative to `AsyncVaried` uses `HttpRequest` internally to build requests.

## More Async

See the async with crul vignette for more details on asynchronous requests.