Python Packages from GitHub

I’ve hit my head against this issue from time to time, so it seems like I need to document the solution somwewhere for each reference.

The Problem

I have a requirements.txt file which lists one or more packages that should be installed from GitHub repositories. The entries in requirements.txt all start with git+https://github.com/.

When I do pip3 install -r requirements.txt I get prompted for username and password. If I need to just provide these once then it’s not a big deal to give them manually. But if this needs to be done routinely then we need to automate.

The Solution

Run this:

git config --global \
    url."https://${GITHUB_TOKEN}@github.com/".insteadOf https://github.com/

🚨 This seems to require --global. I tried with --local and it didn’t work. Presumably --system would also work but doesn’t seem to be practical.

This tells Git that whever it encounters a repository specified by an HTTPS URL it should insert a GitHub token into the URL to provide credentials.

For this to work you’ll need to:

1. create a GitHub token with permission to read the repositories; and
2. export the token in the GITHUB_TOKEN environment variable.

export GITHUB_TOKEN=ghp_jF76DdIvyXeYoz06gk7Z3WdpqnsWPe99Kvxk

With that in place you should be able to pip3 install -r requirements.txt from GitHub unattended.

With Docker

This also works nicely in a Docker image.

FROM python:3.10.6

COPY ./requirements.txt ./

ARG GITHUB_TOKEN
RUN git config --global \
    url."https://${GITHUB_TOKEN}@github.com/".insteadOf https://github.com/

RUN pip install -r requirements.txt

Now build the image.

docker build --build-arg GITHUB_TOKEN=${GITHUB_TOKEN} .