Building new packages from scratch

Today I tried yet another way to upgrade my packages. Since it was recently fixed and after thinking more about Hubert's upgrade procedure I decided to give mk/bulk/mksandbox another chance. For more discussion about upgrading packages, also see the NetBSD wiki. I have also described a method to upgrade only packages that changed.

This time, I wanted to skip the step of building packages and installing them later. Instead, I decided to build the packages in a sandbox on my /usr partition and simply move the resulting /usr/pkg, /var/db/pkg and /var/db/pkg.refcount directories into the right place when finished. This way, I get the benefit of having a working system while building the new one and being able to switch over to the new packages in one swell foop when finished. This requires the build environment in the sandbox to be fairly similar to the real system: in particular, the user and group ids in the sandbox must match the ones in the real system. Luckily, this is exactly the way that mk/bulk/mksandbox sets up the sandbox, so nothing special needs to be done.

Because all packages are rebuilt during the following upgrade procedure, there are no problems when dynamic libraries in the base system are upgraded. This happens fairly frequently in my case because the base system is NetBSD-current.

To update, I first get a new pkgsrc tree:

$ cd /usr/pkgsrc
$ mv cvs.up cvs.up.0
$ cvs up -A 2>&1 | tee cvs.up
$ grep ^[^cUP] cvs.up

Then I generate a pkg_chk configuration file of my required packages:

$ pkg_leaves -a | \
>       while read pkg; do
>               pkg_info -Q PKGPATH "$pkg";
>       done > /usr/pkgsrc/pkgchk.conf

or without pkg_leaves:

$ pkg_info -e "*" | \
>       while read pkg; do
>               if [ -z "`pkg_info -q -R "$pkg"`" ]; then
>                       pkg_info -Q PKGPATH "$pkg";
>               fi;
>       done > /usr/pkgsrc/pkgchk.conf

Don't fall into the trap of trying to use pkg_info -u for this purpose: it uses the package variable "automatic" which gets set to a completely random value in my experience. Instead, the above script prints all packages that are not depended on like in pkg/32827. This speeds up the following build procedure by reducing the number of lines in pkgchk.conf since each line of pkgchk.conf gives rise to an execution of make clean CLEANDEPENDS=yes.

I then save all my binary packages and clean any obsolete distfiles using lintpkgsrc.

$ rm -r packages-old
$ mv packages packages-old
$ lintpkgsrc -mopr

I also clean any stale working directories that might still be around:

$ rm -r /usr/pkgsrc/*/*/work

The only package that needs to be installed in the »host« environment is pkgtools/mksandbox. Note that it has to be called with an absolute path or bad things will happen! This is how I build a new sandbox in my home directory and jump inside:

$ sudo mksandbox $HOME/sb
$ cd $HOME/sb
$ sudo sh sandbox chroot
#

I then set up the /etc/mk.conf file with my desired settings for the build:

.ifdef BSD_PKG_MK
# Setting WRKOBJDIR is inconvenient for development:
#WRKOBJDIR=     /usr/obj/usr/pkgsrc
PKG_DEVELOPER=  yes

UPDATE_TARGET=  bin-install
BINPKG_SITES=   # prevent bin-install from accessing ftp
USE_DESTDIR=    yes
MAKE_JOBS=      8
SKIP_LICENSE_CHECK=yes

# Make Firefox look like Firefox
PKG_OPTIONS.firefox+= official-mozilla-branding
PKG_OPTIONS.gecko+=   official-mozilla-branding
PKG_OPTIONS.thunderbird+=official-mozilla-branding

PKG_DEFAULT_OPTIONS+= gnome

ALLOW_VULNERABLE_PACKAGES=yes

.if exists(/usr/pkg/bin/sudo)
SU_CMD=        /usr/pkg/bin/sudo /bin/sh -c
.endif
.endif

The first package that needs to be installed is pkg_chk because it is in charge of the following procedure.

# cd /usr/pkgsrc/pkgtools/pkg_chk
# make update

Don't do this, it breaks at least textproc/icu: I then tend to install pkgtools/autoswc which provides a cache for configure. This is useful because these tests are typically performed by every package just to produce the same answers to identical questions.

Now, all that is left to do is to start the build and get some coffee while it's working away:

# pkg_chk -akr 2>&1 | tee /usr/obj/pkgchk-akr.0.log

After the build is finished, the sandbox can be unmounted:

# ^D
$ sudo sh sandbox umount

It's probably worth checking for any special configuration in /usr/pkg/etc and merge differences you care to keep into the new tree:

$ diff -ur /usr/pkg/etc $HOME/sb/usr/pkg/etc

Also, you should check for any changes to files in /etc and apply them to your system's configuration. In particular, many packages set up users and groups that you have to copy with identical id's as in the chroot in order to match the owners of the installed files.

$ diff -ur /etc $HOME/sb/etc

Then I switch into single-user mode and exchange my /usr/pkg, /var/db/pkg and /var/db/pkg.refcount directories with the newly built ones:

$ sudo shutdown now
# mv /usr/pkg /usr/pkg-old
# mv /var/db/pkg /var/db/pkg-old
# mv /var/db/pkg.refcount /var/db/pkg.refcount-old
# mv $HOME/sb/usr/pkg /usr/pkg
# mv $HOME/sb/var/db/pkg /var/db/pkg
# mv $HOME/sb/var/db/pkg.refcount /var/db/pkg.refcount
# ^D

At this point, the system should boot back into multi-user mode using the new packages. If anything goes wrong, it's fairly easy to revert back to the old packages using the backups. For this to work properly it's essential that the sandbox directories are on the same partition as their final locations: if they're not, moving the created files while preserving hard links and the like is best done using pax -rw.

It can't hurt to read the install messages for the installed packages using

$ pkg_info -Da

and do as they suggest (create users and groups, copy files from /usr/pkg/share/examples/rc.d to /etc/rc.d etc.)

After everything works to satisfaction,

# rm -fr $HOME/sb
# rm -fr /usr/pkg-old
# rm -fr /var/db/pkg-old
# rm -fr /var/db/pkg.refcount-old