Unpacking cURL Commands

cURL is the ultimate Swiss Army Knife for interacting with network protocols. But to be honest, I really only scratch the surface of what’s possible. Usually my workflow is something like this:

1. Copy a cURL command from my browser’s Developer Tools.
2. Test out the cURL command in a terminal.
3. Convert the cURL command into a programming language (normally Python or R).
4. Prosper.

I’m going to take a look at my favourite online tool for converting a cURL command to code and then see what other tools there, focusing on Python and R as target languages.

Web Tools

The quickest way to convert a cURL command into code is using a tool like curlconverter, which is a transpiler that translates a cURL command in a variety of other languages (Ansible, C, C#, ColdFusion, Clojure, Dart, Elixir, Go, HAR, HTTP, HTTPie, Java, JavaScript, Julia, JSON, Kotlin, Lua, MATLAB, Node.js, Objective-C, OCaml, Perl, PHP, PowerShell, Python, R, Ruby, Rust, Swift and Wget).

It’s an Open Source project and you can dig around in the repository.

There are other similar services (like this) but IMHO curlconverter is the best.

R Tools

The {httr2} package has a curl_translate() function that will convert cURL to R. First let’s load the package.

library(httr2)

Now try it out with the simplest cURL command.

curl_translate("curl https://example.com")

request("https://example.com") |> 
  req_perform()

A little underwhelming, right? But it should do precisely what we expect: get the content from that URL. Let’s check.

response <- request("https://example.com") |> 
  req_perform()

response$status_code

[1] 200

Nice. Now a somewhat more complicated cURL command. Stash the command in a variable.

CURL <- "curl 'https://example.com/' -H 'User-Agent: Mozilla/5.0' -H 'Accept: text/html,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'Priority: u=0, i'"

Now translate.

curl_translate(CURL)

request("https://example.com/") |> 
  req_headers(
    `User-Agent` = "Mozilla/5.0",
    Accept = "text/html,application/xml;q=0.9,*/*;q=0.8",
    `Accept-Language` = "en-US,en;q=0.5",
    `Accept-Encoding` = "gzip, deflate, br, zstd",
    `Upgrade-Insecure-Requests` = "1",
    Priority = "u=0, i",
  ) |> 
  req_perform()

All of the headers in the original command have been translated to work with {httr2}.

Python Tools

We’re going to look at two Python packages: uncurl and curlify. Install them first.

pip3 install uncurl curlify

The uncurl package can be used translate a cURL command to Python. The package comes with a command line executable.

uncurl "curl 'https://example.com/' -H 'User-Agent: Mozilla/5.0'"

requests.get("https://example.com/",
    headers={
        "User-Agent": "Mozilla/5.0"
    },
    cookies={},
    auth=(),
)

It can also be used in code. First import the package.

import uncurl

Now use the parse() function.

uncurl.parse("curl 'https://example.com/' -H 'User-Agent: Mozilla/5.0'")

requests.get("https://example.com/",
    headers={
        "User-Agent": "Mozilla/5.0"
    },
    cookies={},
    auth=(),
)

Magical!

The curlify package allows you to go in the opposite direction, translating a requests object into a cURL command.

First create a requests object.

import requests

response = requests.get("https://example.com")

Now translate.

import curlify

curlify.to_curl(response.request)

That returns a string with the following content (manually split across lines for readability):

"curl -X GET -H 'Accept: */*' -H 'Accept-Encoding: gzip, deflate' \
  -H 'Connection: keep-alive' -H 'User-Agent: python-requests/2.31.0' \
  https://example.com/"

There’s also a curlify2 package, which handles httpx requests.