The end-goal is to access GitHub API, and download Gems, in a respectful fashion by caching responses for a reasonable amount of time. All while also reducing developer friction by reducing the resulting command required a simple docker-compose up to serve a test site locally!

Overview

When Jekyll GitHub Metadata plugin makes requests of GitHub API, we redirect to NGINX reverse proxy. The NGINX reverse proxy then either serves a cached response, or makes the request to GitHub’s API caches successful response(s) then replies to Jekyll with the data.

When Jekyll (via bundle) attempts to install Ruby Gems, we have it check the mounted cached docker volume first. If dependencies are not cached, or out of date, then it’ll download and cache those packages too.

To help prevent cached artifacts, and authentication details, from being tracked and pushed by Git we also must update the .gitignore file. Though readers may wish to add extra protections via a pre-commit hook to add additional restrictions for tracking .env files.

Git configurations

.gitignore

.bundle

## Vim
*.swp
*.swo

## Development
.env

## Docker
docker-volumes

### Jekyll
_site
.sass-cache/
.jekyll-metadata
Notes for -- `.gitignore`
  • Tip; check the manual for githooks, specifically the pre-commit event, for hints on how to reject tracking .env files, eg. man --pager='less --pattern="^\s+pre-commit"' githooks

Jekyll configurations

Gemfile

source "https://rubygems.org"

gem "jekyll", "~> 3.9.5"

gem "minima", "~> 2.5.1"

gem "jekyll-github-metadata"

# If you have any plugins, put them here!
group :jekyll_plugins do
  gem "jekyll-feed", "~> 0.6"
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
# and associated library.
install_if -> { RUBY_PLATFORM =~ %r!mingw|mswin|java! } do
  gem "tzinfo", "~> 1.2"
  gem "tzinfo-data"
end

# Performance-booster for watching directories on Windows
gem "wdm", "~> 0.1.0", :install_if => Gem.win_platform?

##
# Required to make Docker happy
group :docker_compose do
  gem "webrick"
  gem "kramdown-parser-gfm"
end

group :custom_plugins do
  gem 'nokogiri'
end
Notes for -- `Gemfile`
  • gem "jekyll", "~> 3.9.5" is set explicitly to avoid SASS build errors introduced by Jekyll version 4 and, as to last update to this post, is the exact version used by GitHub Pages so setting should help avoid “it works on CI/CD” sorts of issues
  • gem "jekyll-github-metadata" check the official repository for additional details and configuration considerations
  • Other than last few entries, as noted, all others are defaults taken from newly initialized Jekyll project

_config.yml

title: S0AndS0
description: >- # this means to ignore newlines until "baseurl:"
  Miscellaneous tips, tricks, and thoughts from the author known as S0AndS0

baseurl: "/" # the subpath of your site, e.g. /blog
url: 'https://s0ands0.github.io'

##
# Make docker happy with metadata plugin
repository: S0AndS0/S0AndS0.github.io

# Build settings
markdown: kramdown
theme: minima
plugins:
  - jekyll-feed
  - jekyll-github-metadata

# Exclude from processing.
# The following items will not be processed, by default. Create a custom list
# to override the default setting.
exclude:
  - .env
  - Gemfile
  - Gemfile.lock
  - docker-compose
  - docker-compose.yaml
  - docker-volumes
  - node_modules
  - vendor/bundle/
  - vendor/cache/
  - vendor/gems/
  - vendor/ruby/
Notes for -- `_config.yml`
  • repository must be defined to make Jekyll GitHub Metadata plugin happy, check GitHub – Jekyll – Issue 4705 for further details.
  • exclude should list docker-compose and docker-volumes related directory/file(-s) so as to avoid re-building when those locations have file changes.
  • markdown: kramdown is co-dependent with [Gemfile][heading__gemfile]’s gem "kramdown-parser-gfm" configuration, so if utilizing a different MarkDown transpiler things may need additional adjustments.

NGINX configurations

docker-compose/nginx/templates/reverse-proxy-cache_api.github.com.conf.template

# vim: filetype=nginx

# Use cache path we have permissions for writing to within Docker
proxy_cache_path /tmp/nginx-cache/api.github.com levels=1:2 keys_zone=api.github.com:10m;

server {
	server_name api.github.com;
	listen ${NGINX_LISTEN_PORT};

	## Un-comment to possibly enable persistent logs
	# access_log /var/log/nginx/api.github.com-access.log;
	# error_log /var/log/nginx/api.github.com-error.log;

	##
	# Fix errors involving
	#
	#     *1 upstream sent too big header while reading response header from upstream
	proxy_busy_buffers_size 512k;
	proxy_buffers 4 512k;
	proxy_buffer_size 256k;

	location / {
		resolver 8.8.8.8;
		proxy_pass https://api.github.com;
		proxy_cache github;
		proxy_cache_valid 200 302 1d;
		proxy_ignore_headers Expires Cache-Control Set-Cookie X-Accel-Redirect X-Accel-Expires;
		proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;

		proxy_set_header User-Agent ${GITHUB_USER_NAME};
		proxy_set_header Authorization "Token ${GITHUB_AUTH_TOKEN}";
	}
}
Notes for -- `reverse-proxy-cache_api.github.com.conf.template`
  • All paths above are from the perspective of running within a Docker container
  • proxy_cache_path and proxy_cache_valid have long lifetimes, 1d (one day), which may need reduced if lots of updates are consistently being pushed to GitHub
  • listen and proxy_set_header environment variable values are expanded via the services.nginx.image internal Docker image script at /docker-entrypoint.d/20-envsubst-on-templates.sh, injected via services.nginx.env_file definition within the [docker-compose.yaml][heading__dockercomposeyaml] file, and values of environment variables are set within the following [.env/nginx.env][heading__envnginxenv] file.

Docker Compose configurations

.env/nginx.env

# shellcheck disable=all
NGINX_LISTEN_PORT=80
GITHUB_USER_NAME=Your-User-Name
GITHUB_AUTH_TOKEN='sOm3sEc4eTe'
Notes for -- `.env/nginx.env`

⚠ Replace values for GITHUB_USER_NAME, and GITHUB_AUTH_TOKEN, before proceeding!

Tip, the value for GITHUB_AUTH_TOKEN may be obtained via; https://github.com/settings/tokens/new

docker-compose.yaml

version: '2'

##
services:
  ##
  jekyll:
    image: jekyll/jekyll:latest
    container_name: service-jekyll
    command: jekyll serve --watch --force_polling --verbose --trace --incremental

    depends_on:
      - nginx

    volumes:
      - .:/srv/jekyll
      - ./docker-volumes/jekyll/bundle:/usr/local/bundle

    environment:
      ## Downgrade to non-SSL if using `networks.net-services.ipam`
      - PAGES_API_URL=http://api.github.com
      ## Or swap above with below if NGINX be playing with only one config
      # - PAGES_API_URL=http://host-nginx

    ## Point URL(s) at internally static IP address(es)
    extra_hosts:
      - api.github.com:172.21.0.2

    hostname: host-jekyll
    networks:
      - net-jekyll
      - net-services

    ports:
      - 4000:4000

  ##
  nginx:
    image: nginx:stable-alpine
    container_name: service-nginx
    ## Comment following to reduce output
    command: [nginx-debug, '-g', 'daemon off;']

    volumes:
      - ./docker-compose/nginx/templates:/etc/nginx/templates:ro
      ## Mostly to allow NGINX write access to paths protected by permissions
      - ./docker-volumes/nginx/cache:/tmp/nginx-cache
      - ./docker-volumes/nginx/run:/var/run
      - ./docker-volumes/nginx/logs:/var/logs/nginx

    env_file:
      - path: ./.env/nginx.env
        required: true

    hostname: host-nginx
    networks:
      - net-nginx
      - net-services

##
networks:
  net-jekyll:
    name: net-jekyll
    driver: bridge

  net-nginx:
    name: net-nginx
    driver: bridge

  ## Network over which services may chatter to each other over
  net-services:
    name: net-services
    driver: bridge
    ipam:
      config:
        - subnet: 172.21.0.0/16
          ip_range: 172.21.0.0/24
          gateway: 172.21.0.1
          aux_addresses:
            host-nginx: 172.21.0.2
            host-jekyll: 172.21.0.3
Notes for -- `docker-compose.yaml`
  • Most paths above are from the perspective of the host’s repository root, and volumes map to paths within a given Docker container.
  • networks.net-services.ipam, and services.jekyll.extra_hosts, may need adjusted on systems where the defined IP addresses are already claimed by another set of Docker containers.

Notes and tips

  • Reload nginx service within Docker

     docker compose nginx nginx -t &&
   docker compose nginx nginx -s reload

  • Restart jekyll container/service

     docker compose restart jekyll

  • Recreate network(s) after changing docker-compose.yaml file

     docker rm network net-services

  • Occasionally the following error(s) will pop, no idea why, and may be a bug in the plugin not respecting the PAGES_API_URL environment variable’s defined protocol and/or value

     service-jekyll  |    GitHub Metadata: Failed to open TCP connection to api.github.com:443 (Connection refused - connect(2) for "api.github.com" port 443)

  • The above, so far, has taken a little over sixteen hours to research, develop, and document for you dear reader(s)… So if it has helped ya, then feel free to show your appreciation in a way that’ll encourage similar publications.

Attribution