Updating packages in NetBSD

Assuming you're using pkgsrc for your packages, updating regularly can be a bit of a nightmare because make update tends to remove most of your stuff and then regularly fails building the new ones, rendering your computer an expensive doorstop.

To alleviate this, I have used chroot environments to build my packages for a long time. Until recently I used mk/bulk/mksandbox which comes with pkgsrc, but is a little rough around the edges. For example, the unmodified version stopped working when X Window moved to /usr/X11R7 because it only null-mounts /usr/X11R6 (this is fixed). Also while the null mounts are very frugal, they are a bit messy and don't allow building in a different environment from the one your main system is using (such as i386 on an amd64, for example).

Reluctantly, I tried pkgtools/pkg_comp because it felt like overkill using a package for such a simple task. It turns out pkg_comp is comfortably light-weight, so switching from my old jail was a snap.

The following upgrade method makes sure that only packages are rebuilt that actually changed. Unfortunately it can lead to problems when dynamic libraries in the base system are upgraded. Recently, for example, the version number of /usr/lib/libintl changed from 0.0 to 1.0 on NetBSD. My upgrade method will produce new packages liked against version 1.0, while unchanged old packages still use version 0.0, which can lead to problems when the programs are being run. In this case, the only proper way to upgrade is to rebuild all packages that one wants to install by moving the existing /usr/pkgsrc/packages out of the way. For more discussion and other ways to upgrade packages, also see the NetBSD wiki.

To update, first I get a new pkgsrc tree:

$ cd /usr/pkgsrc
$ mv cvs.up cvs.up.0
$ cvs up -A 2>&1 | tee cvs.up
$ grep ^[^cUP] cvs.up

Then I create a pkg_chk configuration file that lists all my packages:

$ pkg_chk -g

I then save all my binary packages and clean any obsolete package and distfiles using lintpkgsrc. Removing outdated binary packages is essential because the bin-install target I use to build the new packages will silently pick up stale dependencies instead of rebuilding them, and unfortunately pkg_chk isn't clever enough to build all required packages first to prevent this from happening.

$ rm -r packages.old
$ mkdir -p packages.old/All.old
$ ln packages/All/* packages.old/All.old
$ lintpkgsrc -mopr

Then I build a fresh chroot environment and jump in:

$ sudo su
# pkg_comp removeroot
# pkg_comp makeroot
# pkg_comp build pkgtools/pkg_chk
# pkg_comp chroot

After going around mindlessly deleting packages using lintpkgsrc above, our binary packages are in a somewhat precarious state: the dependencies of some binary packages may be missing, meaning that a pkg_add of those packages will fail. Unfortunately, when this happens pkg_chk gives up instead of resorting to building the package and all its dependencies from source.

One possible workaround is to pass the -s switch to pkg_chk to prevent it from ever trying to install binary packages. Because I set UPDATE_TARGET=bin-install in /etc/mk.conf (see the pkg_comp configuration file), the update target invoked by pkg_chk will use binary packages if they exist and can be installed successfully. If pkg_add fails, the package and its dependencies are automatically rebuilt.

Although this does the trick, it is undesirable because it means we get many spurious runs of clean CLEANDEPENDS=yes actions which would not occur if pkg_chk had used pkg_add instead of make update. At least three solutions come to mind:

• Fix pkg_chk to first update all dependencies required by a package so pkg_add doesn't fail
• Fix pkg_chk so it behaves like bin-install and tries to build the package from source if pkg_add fails
• Fix the update target so it doesn't clean CLEANDEPENDS=yes if pkg_add succeeded

The final steps are building all packages in the chroot and then installing them in the real world:

# pkg_chk -a -r -s 2>&1 | tee /p/pkg_chk-ars.log
# ^D
# cd /usr/pkgsrc
# pkg_chk -a -r -b 2>&1 | tee pkg_chk-arb.log
# ^D
$

Redirecting the pkg_chk log file in the chroot environment is necessary because pkgsrc is mounted appropriately read-only. The log file will appear in /var/chroot/pkg_comp/default/p/.

It can't hurt to read the install messages for the installed packages using

$ pkg_info -Da

and do as they suggest (create users and groups, copy files from /usr/pkg/share/examples/rc.d to /etc/rc.d etc.)

Lastly, and mostly because I just spent a couple of hours looking for it (and reading some other interesting material in the process), here's a command to show you beforehand what make update will update:

$ make show-needs-update

defined in mk/flavor/pkg/utility.mk (obviously). And there are more treats in this file:

$ make show-installed-depends # alias sid
$ make show-depends-options