Sign in Subscribe
Tech

Docker Registry Build via Harbor

Build your own Docker registry

Henry

• 9 min read
Docker Registry Build via Harbor

1. Background

In the previous article, we have GitLab and GitLab Runner deployed on our server. In the next step, let’s build our own cloud native registry, using Harbor.
Harbor is an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management. Having a registry closer to the build and run environment can improve the image transfer efficiency. Harbor supports replication of images between registries, and also offers advanced security features such as user management, access control and activity auditing.
For more details, please refer to Harbor GitHub

2. Premise and Structure

  • Assume you have already finished deployment and configuration in GitLab and GitLab Runner
  • Harbor will be deployed on a different server to reduce server load (Where the GitLab and GitLab Runner are located)
  • To make full use of our server, an extra service will also be deployed on the same server, using Nginx SNI, just like the way we proceed with GitLab and GitLab Runner.
  • Although Harbor has its own official docker image and scripts, it DOES NOT supports ARM64 architecture. In this case, we will use Harbor packaged by Bitnami
  • The goal is to reuse the 443 port and make minimum modification of original conf file to save our effort when it comes to major version upgrades.
  • Overall, the structure will be: (Visitor) - (Harbor server Nginx, https) - (SNI) - (Bitnami Harbor build-in Nginx, local port A) - (Harbor docker, local port B)

3. Harbor Setup

  • Create Harbor folder
cd /your/path
mkdir harbor && cd harbor
  • Get docker-compose and preset config
curl -LO https://raw.githubusercontent.com/bitnami/containers/main/bitnami/harbor-portal/docker-compose.yml
curl -L https://github.com/bitnami/containers/archive/main.tar.gz | tar xz --strip=2 containers-main/bitnami/harbor-portal && cp -RL harbor-portal/config . && rm -rf harbor-portal
  • Modify docker-compose.yml
nano docker-compose.yml

To make it easy to maintain, I set all local volumes under the harbor folder (eg. ./folder:/docker/folder) And I’ll put comments on where you have to modify. Also, please change and match the passwords in the yml file.

version: '2.1'

services:
  registry:
    image: docker.io/bitnami/harbor-registry:2
    environment:
      - REGISTRY_HTTP_SECRET=CHANGEME
    volumes:
      - ./registry_data:/storage
      - ./config/registry/:/etc/registry/:ro
  registryctl:
    image: docker.io/bitnami/harbor-registryctl:2
    environment:
      - CORE_SECRET=CHANGEME
      - JOBSERVICE_SECRET=CHANGEME
      - REGISTRY_HTTP_SECRET=CHANGEME
    volumes:
      - ./registry_data:/storage
      - ./config/registry/:/etc/registry/:ro
      - ./config/registryctl/config.yml:/etc/registryctl/config.yml:ro
  postgresql:
    image: docker.io/bitnami/postgresql:13
    container_name: harbor-db
    environment:
      - POSTGRESQL_PASSWORD=bitnami
      - POSTGRESQL_DATABASE=registry
    volumes:
      - ./postgresql_data:/bitnami/postgresql
  core:
    image: docker.io/bitnami/harbor-core:2
    container_name: harbor-core
    depends_on:
      - registry
      - chartmuseum
    environment:
      - CORE_KEY=change-this-key
      - _REDIS_URL_CORE=redis://redis:6379/0
      - SYNC_REGISTRY=false
      - CHART_CACHE_DRIVER=redis
      - _REDIS_URL_REG=redis://redis:6379/1
      - PORT=8080
      - LOG_LEVEL=info
      - EXT_ENDPOINT=https://harbor.example.com # Put your site URL with https here
      - DATABASE_TYPE=postgresql
      - REGISTRY_CONTROLLER_URL=http://registryctl:8080
      - POSTGRESQL_HOST=postgresql
      - POSTGRESQL_PORT=5432
      - POSTGRESQL_DATABASE=registry
      - POSTGRESQL_USERNAME=postgres
      - POSTGRESQL_PASSWORD=bitnami
      - POSTGRESQL_SSLMODE=disable
      - REGISTRY_URL=http://registry:5000
      - TOKEN_SERVICE_URL=http://core:8080/service/token
      - HARBOR_ADMIN_PASSWORD=bitnami
      - CORE_SECRET=CHANGEME
      - JOBSERVICE_SECRET=CHANGEME
      - ADMIRAL_URL=
      - WITH_NOTARY=False
      - WITH_CHARTMUSEUM=True
      - CHART_REPOSITORY_URL=http://chartmuseum:8080
      - CORE_URL=http://core:8080
      - JOBSERVICE_URL=http://jobservice:8080
      - REGISTRY_STORAGE_PROVIDER_NAME=filesystem
      - REGISTRY_CREDENTIAL_USERNAME=harbor_registry_user
      - REGISTRY_CREDENTIAL_PASSWORD=harbor_registry_password
      - READ_ONLY=false
      - RELOAD_KEY=
    volumes:
      - ./core_data:/data
      - ./config/core/app.conf:/etc/core/app.conf:ro
      - ./config/core/private_key.pem:/etc/core/private_key.pem:ro
  portal:
    image: docker.io/bitnami/harbor-portal:2
    container_name: harbor-portal
    depends_on:
      - core
  jobservice:
    image: docker.io/bitnami/harbor-jobservice:2
    container_name: harbor-jobservice
    depends_on:
      - redis
      - core
    environment:
      - CORE_SECRET=CHANGEME
      - JOBSERVICE_SECRET=CHANGEME
      - CORE_URL=http://core:8080
      - REGISTRY_CONTROLLER_URL=http://registryctl:8080
      - REGISTRY_CREDENTIAL_USERNAME=harbor_registry_user
      - REGISTRY_CREDENTIAL_PASSWORD=harbor_registry_password
    volumes:
      - ./jobservice_data:/var/log/jobs
      - ./config/jobservice/config.yml:/etc/jobservice/config.yml:ro
  redis:
    image: docker.io/bitnami/redis:7.0
    environment:
      # ALLOW_EMPTY_PASSWORD is recommended only for development.
      - ALLOW_EMPTY_PASSWORD=yes
  harbor-nginx:
    image: docker.io/bitnami/nginx:1.23
    container_name: nginx
    volumes:
      - ./config/proxy/nginx.conf:/opt/bitnami/nginx/conf/nginx.conf:ro
    ports:
      - '127.0.0.1:22222:8080' # To reuse local port 443, let’s point it to a local port here, 22222
    depends_on:
      - postgresql
      - registry
      - core
      - portal
  chartmuseum:
    container_name: chartmuseum
    image: docker.io/bitnami/chartmuseum:0
    environment:
      - CACHE=redis
      - CACHE_REDIS_ADDR=redis:6379
      - CACHE_REDIS_DB=1
      - DEPTH=1
      - PORT=8080
      - STORAGE=local
      - STORAGE_LOCAL_ROOTDIR=/bitnami/data
      - ALLOW_OVERWRITE=true
      - INDEX_LIMIT=0
    volumes:
      - ./chartmuseum_data:/bitnami/data
volumes:
  registry_data:
    driver: local
  core_data:
    driver: local
  jobservice_data:
    driver: local
  postgresql_data:
    driver: local
  chartmuseum_data:
    driver: local

IMPORTANT: For the value of EXT_ENDPOINT, you have to set it to https here (https://harbor.example.com), because another Nginx will proxy pass through the build-in Nginx. If you leave the original http value here, you will not be able to docker login harbor.example.com, with error message with “unauthorized: authentication required.”

  • Modify Bitnami Harbor build in Nginx conf
cd /your/path/harbor/config/proxy
nano nginx.conf

Since we will use Nginx on our server to take over port 443 with SNI, in this case we will have to comment out the X-Forwarded-Proto in the conf.

worker_processes auto;
error_log         "/opt/bitnami/nginx/logs/error.log";
pid               "/opt/bitnami/nginx/tmp/nginx.pid";

events {
  worker_connections 1024;
  use epoll;
  multi_accept on;
}

http {
  tcp_nodelay on;

  # this is necessary for us to be able to disable request buffering in all cases
  proxy_http_version 1.1;

  upstream core {
    server core:8080;
  }

  upstream portal {
    server portal:8080;
  }

  log_format timed_combined '$remote_addr - '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent" '
    '$request_time $upstream_response_time $pipe';

  client_body_temp_path  "/opt/bitnami/nginx/tmp/client_body" 1 2;
  proxy_temp_path        "/opt/bitnami/nginx/tmp/proxy" 1 2;
  fastcgi_temp_path      "/opt/bitnami/nginx/tmp/fastcgi" 1 2;
  scgi_temp_path         "/opt/bitnami/nginx/tmp/scgi" 1 2;
  uwsgi_temp_path        "/opt/bitnami/nginx/tmp/uwsgi" 1 2;

  server {
    listen 8080;
    server_tokens off;
    # disable any limits to avoid HTTP 413 for large image uploads
    client_max_body_size 0;

    # costumized location config file can place to /opt/bitnami/nginx/conf with prefix harbor.http. and suffix .conf
    include /opt/bitnami/conf/nginx/conf.d/harbor.http.*.conf;

    location / {
      proxy_pass http://portal/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
#      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /c/ {
      proxy_pass http://core/c/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
#      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /api/ {
      proxy_pass http://core/api/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
 #     proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /chartrepo/ {
      proxy_pass http://core/chartrepo/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
#      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /v1/ {
      return 404;
    }

    location /v2/ {
      proxy_pass http://core/v2/;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
#      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /service/ {
      proxy_pass http://core/service/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # When setting up Harbor behind other proxy, such as an Nginx instance, remove the below line if the proxy already has similar settings.
#      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_buffering off;
      proxy_request_buffering off;
    }

    location /service/notifications {
      return 404;
    }
  }
}

Note: There are six places to be commented out, including location /, location /c/, location /api/, location /chartrepo/, location /v2/, location /service/.

  • Bring up the container
cd /your/path/harbor
docker-compose up -d

4. BookStack Deployment

This is just an example of a second application to be setup.

cd /your/path
mkdir bookstack && cd bookstack
nano docker-compose.yml

version: "3.8"
services:
  bookstack:
    image: ghcr.io/linuxserver/bookstack
    container_name: bookstack
    environment:
      - PUID=1000
      - PGID=1000
      - APP_URL=https://bookstack.example.com
      - DB_HOST=bookstack_db
      - DB_USER=bookstack
      - DB_PASS=password
      - DB_DATABASE=bookstackapp
    volumes:
      - ./data:/config
    ports:
      - 127.0.0.1:22223:80 # We have Harbor application pointed docker internal port from 8080 to local port 22222. So for this second app, we point BookStack from docker internal port from 80 to local port 22223.
    restart: unless-stopped
    depends_on:
      - bookstack_db
  bookstack_db:
    image: ghcr.io/linuxserver/mariadb
    container_name: bookstack_db
    environment:
      - PUID=1000
      - PGID=1000
      - MYSQL_ROOT_PASSWORD=password
      - TZ=Europe/London
      - MYSQL_DATABASE=bookstackapp
      - MYSQL_USER=bookstack
      - MYSQL_PASSWORD=password
    volumes:
      - ./db:/config
    restart: unless-stopped
  • Bring up the container
docker-compose up -d

5. Nginx Configuration

This is an “outer” Nginx to cover both Bitnami Harbor built-in Nginx and BookStack

nano /etc/nginx/nginx.conf
  • Add a stream block at the end
stream {
        map $ssl_preread_server_name $example_multi {
	    harbor.henrywithu.com harbor;
                bookstack.henrywithu.com bookstack;
        }
        upstream harbor {
                server 127.0.0.1:30001; # This port will be linked to local port 22222, as Harbor Registry service
        }
        upstream bookstack {
                server 127.0.0.1:30002; # This port will be linked to local port 22223 as BookStack service
        }
        server {
                listen 443      reuseport;
                listen [::]:443 reuseport;
                proxy_pass      $example_multi;
                ssl_preread     on;
	    #proxy_protocol  on;
        }
}
  • harbor.conf
nano /etc/nginx/conf.d/harbor.conf

server {
  listen 127.0.0.1:30001 ssl http2;

  ssl_certificate       /your/path/fullchain.pem;
  ssl_certificate_key   /your/path/privkey.pem;
#ssl_dhparam          /your/path/dhparams.pem; #optional
#Authenticated Origin Pull is optional. Please refer to https://developers.cloudflare.com/ssl/origin/authenticated-origin-pull/
#ssl_client_certificate  /etc/ssl/origin-pull-ca.pem;
#ssl_verify_client on;
  ssl_session_timeout 1d;
  ssl_session_cache shared:MozSSL:10m;
  ssl_session_tickets off;

  proxy_read_timeout 300;
  proxy_connect_timeout 300;
  proxy_send_timeout 300;

  client_max_body_size 10G; # define your max_upload_file_size

  ssl_protocols         TLSv1.2 TLSv1.3;
  ssl_ciphers           ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
  ssl_prefer_server_ciphers off;

  server_name           harbor.example.com;
    add_header Content-Security-Policy upgrade-insecure-requests;
  location / {
    proxy_redirect off;
    proxy_pass http://127.0.0.1:22222; # port 22222 matches the built-in Nginx service port
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}
  • bookstack.conf
nano /etc/nginx/conf.d/bookstack.conf

server {
  listen 127.0.0.1:30002 ssl http2;

  ssl_certificate       /your/path/fullchain.pem;
  ssl_certificate_key   /your/path/privkey.pem;
#ssl_dhparam          /your/path/dhparams.pem; #optional
#Authenticated Origin Pull is optional. Please refer to https://developers.cloudflare.com/ssl/origin/authenticated-origin-pull/
#ssl_client_certificate  /etc/ssl/origin-pull-ca.pem;
#ssl_verify_client on;
  ssl_session_timeout 1d;
  ssl_session_cache shared:MozSSL:10m;
  ssl_session_tickets off;

  proxy_read_timeout 300;
  proxy_connect_timeout 300;
  proxy_send_timeout 300;

  client_max_body_size 10G; # define your max_upload_file_size

  ssl_protocols         TLSv1.2 TLSv1.3;
  ssl_ciphers           ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
  ssl_prefer_server_ciphers off;

  server_name           bookstack.example.com;
    add_header Content-Security-Policy upgrade-insecure-requests;
  location / {
    proxy_redirect off;
    proxy_pass http://127.0.0.1:22223; # port 22223 matches the built-in Nginx service port
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}
  • httpsredirect.conf
nano /etc/nginx/conf.d/httpsredirect.conf

server {
        listen 80;
        server_name bookstack.example.com;
        if ($host = bookstack.example.com) {
                return 301 https://$host$request_uri;
        }
        server_name harbor.example.com;
        if ($host = harbor.example.com) {
                return 301 https://$host$request_uri;
        }
       return 404;
}
  • Restart Nginx
nginx -t
systemctl restart nginx

6. Harbor Post-Configuration

Now you can login to Harbor in your browser. The initial credentials can be found in your Harbor docker-compose.yml, which is admin/bitnami. Once you successfully logged in, please change the password immediately. Or you can create a new user with an admin role and then remove the default admin user.

  • Assume you created a user and created a new project named test
    Now let’s try to push an docker image to the Harbor Registry.
  • Login another server
# On a non-Harbor server
docker login harbor.example.com
  • You will be prompted to input username and password
USERNAME=<username_you_createed_in_Harbor>
PASSWORD=<password_you_set_in_Harbor>
  • The successful output will be:
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
  • Now check your local docker images
docker images
  • For example, let’s choose NODE:19-alpine
REPOSITORY TAG IMAGE ID CREATED SIZE
NODE 19-alpine 31724419e3ea 10 days ago 177MB
  • Put a tag on your local docker image with the remote Harbor Registry (Project name test has been created earlier)
docker tag NODE:19-alpine harbor.example.com/test/NODE:19-alpine
  • Push the docker image to Harbor Registry
docker push harbor.example.com/test/NODE:19-alpine

Now you can check the image in Harbor Registry from web.
For general docker commands, please refer to: The base command for the Docker

Enjoy


Copyright statement: Unless otherwise stated, all articles on this blog adopt the CC BY-NC-SA 4.0 license agreement. For non-commercial reprints and citations, please indicate the author: Henry, and original article URL. For commercial reprints, please contact the author for authorization.