dpb - distributed ports builder

Table of Contents

Introduction
#

In the world of OpenBSD, the tool dpb (Distributed Ports Builder) offers an efficient way to compile ports across multiple machines and perform bulk builds. This blog post describes my simple dpb setup, which was made possible by a handful of sponsors. The entire setup runs on a NetCup RS 2000 G11.

The goal is to provide an overview of how to configure a single instance for port building with minimal effort. Whether you’re trying dpb for the first time or looking for a straightforward guide, I hope this documentation will be useful both for beginners and for myself, as a reference for future setups since I don’t have an Ansible playbook for it ;).

No matter what you want to do. You want to read dpb(1) and bulk(8)!

How to setup
#

First and foremost: the partition table setup is crucial. A good reference for the required storage capacity for bulk, distfiles and packages is the bulk(8) manual. This will guide you through what’s needed:

Filesystem                                Size    Used   Avail Capacity  Mounted on
/dev/sd0a                                 986M    117M    820M    13%    /
/dev/sd0k                                 148G   22.3G    118G    16%    /bulk
/dev/sd0m                                 121G   83.2G   31.9G    73%    /bulk/data/distfiles
/dev/sd0n                                77.5G   44.2G   29.4G    61%    /bulk/data/packages
/dev/sd0p                                36.6G    141M   34.6G     1%    /home
/dev/sd0d                                 3.9G   26.0K    3.7G     1%    /tmp
/dev/sd0f                                29.1G    4.0G   23.6G    15%    /usr
/dev/sd0g                                 986M    322M    615M    35%    /usr/X11R6
/dev/sd0h                                19.4G    2.2G   16.2G    12%    /usr/local
/dev/sd0j                                 5.8G    2.0K    5.5G     1%    /usr/obj
/dev/sd0i                                 4.8G    1.5G    3.1G    34%    /usr/src
/dev/sd0e                                34.8G   56.3M   33.0G     1%    /var
127.0.0.1:/usr/ports/mystuff/wip-ports   29.1G    4.0G   23.6G    15%    /bulk/usr/ports
127.0.0.1:/bulk/data/packages            77.5G   44.2G   29.4G    61%    /var/www/htdocs/pub/packages

NFS
#

As you can see at the end there are two NFS entries. For me, these two, at least one, are very important. The main point is that I mount my ports tree /usr/ports/mystuff/wip-ports (Normal here would be /usr/ports but I always work on the my wip git repo) to the chroot’ed bulk build ports.

into the httpd htdocs directory to install them on my workstation if I also export the packages built in the chroot /build/data/packages as packages necessary. This is of course optional, but helps from time to time when I want and need to test big updates like Qt or KDE as a whole.

The config for this is in /etc/exports and /etc/fstab:

4ae05057e5183090.k /bulk ffs rw,wxallowed,nosuid 1 2
4ae05057e5183090.m /bulk/data/distfiles ffs rw,,nosuid 1 2
4ae05057e5183090.n /bulk/data/packages ffs rw,nodev,nosuid 1 2

127.0.0.1:/usr/ports/mystuff/wip-ports /bulk/usr/ports nfs rw,nodev,intr,nosuid,-a4 0 0
127.0.0.1:/bulk/data/packages /var/www/htdocs/pub/packages nfs rw,nodev,intr,nosuid,-a4 0

/usr/ports/mystuff/wip-ports 127.0.0.1
/bulk/data/packages 127.0.0.1

Of course you have to enable NFS locally rcctl enable mountd nfsd portmap

proot(1)
#

Let’s take a look at proot(1), which is really helpful. proot can fill up a chroot directory for ports building usage.

proot.conf:

chroot=/bulk

PORT_USER=rsadowski
WRKOBJDIR=/tmp/pobj
LOCKDIR=/tmp/locks
PLIST_REPOSITORY=/data/plist
DISTDIR=/data/distfiles
PACKAGE_REPOSITORY=/data/packages
actions=unpopulate_light
        chown_all
        ports_subdirs
        write_mk
        check_symlinks
        devs
snapshot=https://cdn.openbsd.org/pub/OpenBSD/snapshots/amd64

You can simply run doas proot -c proot.conf to setup you chroot.

dpb(1)
#

The last thing you need to do is start dpp(1). I start it like this

doas /usr/ports/infrastructure/bin/dpb -p8 -j8 -B /bulk -L /bulk/data/log/

My way of work
#

Here is my work flow to start my bulk build.

New host system. I always upgrade to the latest system with sysupgrade -s
Cleanup my last bulk and create a completely new one. I do this with the following helper script. I do this because my resources are limited and I always want to test my wip work-tree anyway.

#!/bin/sh

CHROOT=/bulk

WRKOBJDIR=/tmp/pobj
LOCKDIR=/tmp/locks
PACKAGE_REPOSITORY=/data/packages

echo "Remove WRKOBJDIR: $WRKOBJDIR"
doas rm -rf $CHROOT/$WRKOBJDIR/*

echo "Remove PACKAGE_REPOSITORY: $PACKAGE_REPOSITORY"
doas rm -rf $CHROOT/$PACKAGE_REPOSITORY/amd64/all/
doas rm -rf $CHROOT/$PACKAGE_REPOSITORY/amd64/ftp/
doas rm -rf $CHROOT/$PACKAGE_REPOSITORY/amd64/no-arch/
doas rm -rf $CHROOT/$PACKAGE_REPOSITORY/amd64/tmp/

doas mkdir $CHROOT/$PACKAGE_REPOSITORY/amd64/all/
doas chown _pbuild:_pbuild $CHROOT/$PACKAGE_REPOSITORY/amd64/all/

doas mkdir $CHROOT/$PACKAGE_REPOSITORY/amd64/ftp/
doas chown _pbuild:_pbuild $CHROOT/$PACKAGE_REPOSITORY/amd64/ftp/

doas mkdir $CHROOT/$PACKAGE_REPOSITORY/amd64/no-arch/
doas chown _pbuild:_pbuild $CHROOT/$PACKAGE_REPOSITORY/amd64/no-arch/

doas mkdir $CHROOT/$PACKAGE_REPOSITORY/amd64/tmp/
doas chown _pbuild:_pbuild $CHROOT/$PACKAGE_REPOSITORY/amd64/tmp/

echo "Remove LOCKDIR: $LOCKDIR"
doas rm -rf $CHROOT/$LOCKDIR/*

echo "Run proot..."
doas proot -c proot.conf

As above, start dpb in a tmux session and check it whenever I have time.

dpb-replay(1)
#

If an port has a build error or something and you can fix it in the ports tree, you can restart the build by wipe the port.

Connect to the running dpb session:

doas nc -U /bulk/data/log/control-p*

2.) wipe the port

doas nc -U /bulk/data/log/control-p*
dpb@pb[87078]$ wipe sysutils/salt
cleaning up sysutils/salt
dpb@pb[87078]$ bye

Exclude dpb hack
#

There is a patch by espie@ that allows you to exclude ports. The patch is only a hack and therefore not committed and will not. However, this patch lets you skip large or unwanted ports, reducing build times.

For example, my exclude-ports file contains:

editors/libreoffice
emulators/mame
emulators/qemu
games/warzone2100
lang/deno
lang/gambit
lang/mono
lang/sbcl
www/chromium
www/iridium
www/ungoogled-chromium

As you can see. All chromium engined browsers and some other huge end-user applications are excluded. It is used as follows and you can find the diff at the end:

doas /usr/ports/infrastructure/bin/dpb -p8 -j8 -B /bulk -L /bulk/data/log/ -E exclude-ports

I hope this helps you set up your dpb environment and streamline your workflow. Thanks for reading!

Index: infrastructure/bin/dpb
===================================================================
RCS file: /cvs/ports/infrastructure/bin/dpb,v
diff -u -p -r1.149 dpb
--- infrastructure/bin/dpb	14 Aug 2023 14:01:41 -0000	1.149
+++ infrastructure/bin/dpb	14 Aug 2023 18:54:24 -0000
@@ -152,6 +152,11 @@ $state->interpret_paths(@{$state->{xpath
 	$p->{dontjunk} = 1;
     });
 
+$state->interpret_paths(@{$state->{epaths}},
+    sub($p, $weight = undef) {
+	$state->{ignore}{$p->pkgpath} = 1;
+    });
+
 if ($state->{bad_paths}) {
 	$state->usage("Bad package path #1",
 	    join(" ", @{$state->{bad_paths}}));
Index: infrastructure/lib/DPB/Config.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/DPB/Config.pm,v
diff -u -p -r1.101 Config.pm
--- infrastructure/lib/DPB/Config.pm	2 Jan 2024 15:39:30 -0000	1.101
+++ infrastructure/lib/DPB/Config.pm	3 Jan 2024 00:32:50 -0000
@@ -83,6 +83,9 @@ sub parse_command_line($class, $state)
 		X => sub($opt) {
 			push(@{$state->{xpaths}}, $opt);
 		},
+		E => sub($opt) {
+			push(@{$state->{epaths}}, $opt);
+		},
 		b => sub($opt) {
 			push(@{$state->{build_files}}, $opt);
 		},
@@ -91,9 +94,10 @@ sub parse_command_line($class, $state)
 		},
 	};
 
-	$state->SUPER_handle_options('acemNqrRstuUvh:S:xX:A:B:C:f:F:I:j:J:M:p:P:b:l:L:',
-    "[-acemNqrRsuUvx] [-A arch] [-B chroot] [-C plist] [-f m] [-F m]",
-    "[-I pathlist] [-J p] [-j n] [-p parallel] [-P pathlist] [-h hosts]",
+	$state->SUPER_handle_options('acemNqrRstuUvh:S:xX:A:B:C:E:f:F:I:j:J:M:p:P:b:l:L:',
+    "[-acemNqrRsuUvx] [-A arch] [-B chroot] [-C plist]",
+    "[-E pathlist] [-f m] [-F m] [-I pathlist]",
+    "[-J p] [-j n] [-p parallel] [-P pathlist] [-h hosts]",
     "[-L logdir] [-l lockdir] [-b log] [-M threshold] [-X pathlist]",
     "[pathlist ...]");
 	for my $l (qw(j f F)) {
Index: infrastructure/lib/DPB/Engine.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/DPB/Engine.pm,v
diff -u -p -r1.151 Engine.pm
--- infrastructure/lib/DPB/Engine.pm	7 Jul 2023 14:43:56 -0000	1.151
+++ infrastructure/lib/DPB/Engine.pm	10 Jul 2023 15:41:37 -0000
@@ -482,6 +482,10 @@ sub new_path($self, $v)
 		$self->stub_out($v);
 		return;
 	}
+	if ($self->{state}{ignore}{$v->pkgpath}) {
+		$self->log('!', $v, "because of dpb options");
+		$self->stub_out($v);
+	}
 	if (defined $v->{info}{MISSING_FILES}) {
 		$self->add_fatal($v, ["fetch manually"], 
 		    "Missing distfiles: ".
´´´

Introduction #

How to setup #

NFS #

proot(1) #

dpb(1) #

My way of work #

dpb-replay(1) #

Exclude dpb hack #