.\" $NetBSD: pkg_comp.8,v 1.2 2020/04/23 11:28:09 sborrill Exp $
.\"
.\" pkg_comp - Build packages inside a clean chroot environment
.\" Copyright (c) 2002, 2003, 2004, 2005 Julio M. Merino Vidal <jmmv@NetBSD.org>
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Neither the name of The NetBSD Foundation nor the names of its
.\"    contributors may be used to endorse or promote products derived
.\"    from this software without specific prior written permission.
.\" 3. Neither the name of author nor the names of its contributors may
.\"    be used to endorse or promote products derived from this software
.\"    without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
.\" PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd August 17, 2015
.Dt PKG_COMP 8
.Os
.Sh NAME
.Nm pkg_comp
.Nd build packages inside a sandbox
.Sh SYNOPSIS
.Nm
.Oo Fl Po
.Cm c Ns \&| Ns Cm C
.Pc
.Ar conf_file
.Oc
.Op Fl Nn
.Ar target
.Op Ar pkg_name ...
.Sh DESCRIPTION
.Nm ,
or
.Em Package Compiler
in its full name,
is a tool that makes easy the compilation of packages inside a clean
sandbox.
This allows an easy tracking of exact dependencies
and the correct behavior of a package in a fresh system installation.
.Pp
The behavior of
.Nm
is controlled through a small configuration file and a target (keep
reading to learn more).
The configuration file tells
.Nm
how to configure the new chroot environment, and the target specifies
which action to take.
.Pp
The following options are recognized:
.Bl -tag -width XcXconf_file
.It Fl C Ar conf_file
Use
.Ar conf_file
as configuration file (full path expected).
.It Fl c Ar conf_file
Use
.Ar conf_file
as configuration file (only base name expected).
.It Fl N
With the exception of
.Pa pkgtools/libkver
(see
.Va NETBSD_RELEASE )
avoid installation of default packages as well as
.Va INSTALL_PACKAGES
and
.Va BUILD_PACKAGES
during the creation of the chroot.
.It Fl n
Avoid installation of
.Va INSTALL_PACKAGES
and
.Va BUILD_PACKAGES
during the creation of the chroot.
.El
.Ss What to use it for?
You can use
.Nm
to achieve many goals when building packages.
Here are some ideas:
.Bl -bullet
.It
Build packages for other system versions.
For example, build packages for
.Nx 4.0
while you are running
.Nx 5.0 .
.It
Build packages using different
.Pa mk.conf
options than your current system, like changing the threading library,
.Sy COPTS ,
placement of configuration files, etc.
.It
Debug the build process of a package, checking if buildlinks work
properly.
.It
Avoid autoconf's side effects by keeping a separate chroot for each
project, like one for GNOME2 and another one for KDE3.
.It
Schedule builds of package sets for several different machines.
.El
.Sh CONTROL DIRECTORY
.Nm
needs to store several pieces of information when it is running.
Instead of using normal system trees, it uses a special directory inside the
chroot to avoid polluting the system.
It stores there scripts, object files, built packages, etc.
This directory is
.Pa $DESTDIR/pkg_comp ;
the symbolic link
.Pa $DESTDIR/p
is automatically created to ease pathnames when working inside the chroot.
.Sh CONFIGURATION
With
.Nm
you can maintain several configuration files so you can work with
different chroot jails easily.
To make this easy, configuration files are stored inside
.Pa $HOME/pkg_comp ,
followed by the configuration file name and the .conf suffix.
The default configuration file is
.Pa $HOME/pkg_comp/default.conf ,
and is always used if you do not specify another one.
The configuration file name is specified by the argument of the
.Fl c
option.
Alternatively you can specify any pathname as a configuration file
with the argument of the
.Fl C
option.
.Pp
Configuration files are simple shell scripts that define
variables.
The default values shown here are those written in the template when
issuing a maketemplate.
.Bl -tag -width indent
.It AUTO_PACKAGES
A list of packages to automatically build during the
.Sy auto
target.
A package is in the form
.Sq section/name ,
like
.Sq misc/colorls ,
or a plain name like
.Sq colorls .
Defaults to nothing.
.It AUTO_TARGET
The pkgsrc target to use when building packages in an automated fashion
(using the
.Ql auto
target).
Should be set to
.Ql package
or
.Ql bin-install ,
as other values are useless.
Defaults to
.Ql package .
.It BUILD_PACKAGES
A list of packages to automatically build after the
.Sy makeroot
target.
A package is in the form
.Sq section/name ,
like
.Sq misc/colorls ,
or a plain name like
.Sq colorls .
Defaults to nothing.
.It BUILD_PKG_COMP_TARGET
The pkgsrc target to use when building packages.
It can contain any target supported by the pkgsrc system, but
reasonable values are:
.Ql install ,
.Ql package
and
.Ql bin-install .
Defaults to
.Ql package .
.It COPYROOTCFG
If set to
.Ql yes ,
all configuration files (not directories) that reside inside
.Pa /root
are copied to
.Sy $DESTDIR/root .
Defaults to
.Ql no .
.It DESTDIR
The chroot jail directory.
Defaults to
.Pa /var/chroot/pkg_comp/default .
.It DISTRIBDIR
This is the directory which holds
.Nb
binary sets and X sets.
Its structure is the same as official release
distributions, that is, tarball files must reside inside
.Pa $DISTRIBDIR/binary/sets .
Defaults to
.Pa /var/pub/NetBSD .
.It DISTRIB_EXT
This is the file extension for the tarballs in
.Pa $DISTRIBDIR .
Recent NetBSD releases use tar.xz instead of tgz.
Defaults to 
.Pa tgz
.It EXTRAMK
Specifies a whitespace-separated list of files that must be appended to
.Pa $DESTDIR/etc/mk.conf .
This is useful to add special items to this configuration file.
Defaults to nothing.
.It INSTALL_PACKAGES
A list of packages to automatically install after the
.Sy makeroot
and before installing
.Sy BUILD_PACKAGES .
These are also installed within the sandbox created by the
.Sy auto
target, but before anything is built.
Each name must be the full package name, including the tgz suffix.
Packages are searched inside
.Pa $REAL_PACKAGES/All .
Defaults to nothing.
.It GENERATE_PKG_SUMMARY
If set to
.Sq yes ,
generate a new
.Pa $REAL_PACKAGES/pkg_summary.gz
file at the end of every package build by both the
.Sq auto
and
.Sq build
targets.
.It LOCALBASE
Where binary packages get installed.
Defaults to
.Pa /usr/pkg .
.It MKCONF_VARS
A list of variable names that will be appended to the generated
.Pa /etc/mk.conf
file, together with their values set in the configuration file.
Its default value contains all variables listed here.
.It NETBSD_RELEASE
Specifies which version number of
.Nx
is installed inside the chroot.
If set to
.Ql no ,
no special action is taken (this is useful if the system version inside
the chroot matches the outside one).
Otherwise, the package
.Pa pkgtools/libkver
will be installed inside the chroot, in a special purpose
prefix whose value can be set in
.Pa $DESTDIR/etc/mk.conf
via the configuration file
with the
.Va LIBKVER_STANDALONE_PREFIX
variable.
The libkver library will be configured inside the chroot, with the symbolic link
.Pa $DESTDIR/libkver_osrelease
and
.Va LD_PRELOAD
in default shells environments,
so that the
.Nx
version specified in
.Va NETBSD_RELEASE
overrides the host system version.
See
.Xr kver 3
for more information.
Defaults to
.Ql no .
.It PKG_DBDIR
Location of the packages database.
Defaults to
.Pa /var/db/pkg .
.It PKG_SYSCONFBASE
Base directory of configuration files.
Defaults to
.Pa /usr/pkg/etc .
.It PKGSRC_COMPILER
List of values specifying the chain of compilers to invoke when building
packages.
Defaults to
.Ql gcc .
If you are defining
.Va REAL_CCACHE ,
remember to prepend
.Ql ccache
to this variable's value.
.It PKGVULNDIR
Directory where the
.Pa vulnerabilities
file will be installed (inside the chroot).
Defaults to
.Pa /usr/pkg/share .
.It REAL_PKGVULNDIR
Directory where the system-wide
.Pa vulnerabilities
file resides (outside the chroot).
Defaults to
.Pa /usr/pkgsrc/distfiles .
.It ROOTSHELL
The shell of the root user.
Defaults to
.Pa /bin/ksh .
.It SETS
A list of binary sets to be extracted inside
.Sy DESTDIR .
Defaults to
.Ql base.tgz comp.tgz etc.tgz kern-GENERIC.tgz text.tgz .
If no kernel is extracted by these sets, an empty
.Pa /netbsd
file is created inside the chroot.
.It SETS_X11
A list of binary sets of the X Window system.
This has the same behavior
as
.Sy SETS .
If this variable is set to
.Ql no ,
no X Window is configured inside the chroot
jail and no other X variables take effect.
Defaults to
.Ql xbase.tgz xcomp.tgz xetc.tgz xfont.tgz xserver.tgz .
.It SYNC_UMOUNT
If set to
.Ql yes ,
run
.Xr sync 8
three times after all file systems have been unmounted.
Defaults to
.Ql no .
.It USE_AUDIT_PACKAGES
If set to
.Ql yes ,
let
.Nm
handle the
.Pa vulnerabilities
file automatically.
This means that it will install the system-wide
.Pa vulnerabilities
file inside the chroot when needed, keeping both in sync.
Defaults to
.Ql yes .
.It USE_GCC3
If set to
.Ql yes ,
the GNU C Compiler version 3 will be installed inside the chroot
environment and used to build all packages, using the
.Pa lang/gcc3
package.
Defaults to
.Ql no .
.El
.Ss Mounted file systems
In order to avoid duplicating huge system trees,
.Nm
takes advantage of file system layers.
By default, it uses
.Xr mount_null 8 ,
which duplicates a file system tree into another directory; although
you may want to use
.Xr mount_union 8 ,
or even
.Xr mount_overlay 8 .
If the
content of these variables is empty, that file system is not mounted.
.Pp
You can control which layer to use and which options you want with
special configuration options, as explained below.
.Pp
These file systems are mounted before entering the chroot and unmounted
after exiting.
In order to know if file systems are mounted or not, the
program uses a temporary file, called
.Pa $DESTDIR/pkg_comp/tmp/mount.stat ,
which controls the number of
.Nm
processes using the chroot environment.
If some of them crashes unexpectedly and you notice it does not try
to unmount the file systems, this status file may get out of sync.
Be sure to check that NO file systems are mounted when issuing a
.Sy removeroot .
.Bl -tag -width indent
.It REAL_CCACHE
Specifies where a global ccache directory resides in the real system.
Defaults to nothing, which disables the global cache.
Keep in mind that this is specially useful to keep the cache across
rebuilds of the sandbox, but be very careful if you plan to share a
cache directory between different sandboxes, as this can lead to problems.
.It REAL_DISTFILES
Specifies where distfiles reside in the real system.
Defaults to
.Pa /usr/pkgsrc/distfiles .
.It REAL_DISTFILES_OPTS
Mount options.
Defaults to
.Sy -t null -o rw .
.It REAL_PACKAGES
Specifies where to build binary packages.
This variable is specially useful.
Defaults to
.Pa /usr/pkgsrc/packages .
.It REAL_PACKAGES_OPTS
Mount options.
Defaults to
.Sy -t null -o rw .
.It REAL_PKGSRC
The pkgsrc tree.
This can be useful if you want to use several pkgsrc trees independently.
Defaults to
.Pa /usr/pkgsrc .
.It REAL_PKGSRC_OPTS
Mount options.
Defaults to
.Sy -t null -o ro .
.It REAL_SRC
The src system tree.
Usually useless, but may be needed by some packages.
Defaults to
.Pa /usr/src .
.It REAL_SRC_OPTS
Mount options.
Defaults to
.Sy -t null -o ro .
.It MAKEROOT_HOOKS
A whitespace separated list of functions or external scripts to be executed
after the sandbox is created.
Two arguments are given to each of them:
.Ar $DESTDIR ,
and the word
.Ar makeroot .
Defaults to nothing.
.It MOUNT_HOOKS
A whitespace separated list of functions or external scripts to be executed
after file systems are mounted.
Two arguments are given to each of them:
.Ar $DESTDIR ,
and the word
.Ar mount .
Defaults to nothing.
.It UMOUNT_HOOKS
A whitespace separated list of functions or external scripts to be executed
before file systems are unmounted.
Two arguments are given to each of them:
.Ar $DESTDIR ,
and the word
.Ar umount .
Defaults to nothing.
.El
.Sh TARGETS
A target specifies what
.Nm
should do (as in make).
The following list describes all supported targets,
in the logical order you should call them.
.Bl -tag -width indent
.It maketemplate
Create a sample
.Ar conf_file .
You should edit it after the creation as you will probably want to change
the default configuration, specially paths.
.It makeroot
Create the chroot environment, based on the specs of the configuration file.
This step is required before trying any other, except maketemplate.
.It build
Builds the specified packages inside the chroot.
You can pass the package names as a relative path within pkgsrc or as the
basename of the package directory (i.e. omitting the directory name).
.It install
Install the specified binary packages into the chroot.
Package names can contain globs.
The package files will be taken from within
.Sy REAL_PACKAGES .
.It chroot
Enters the chroot environment.
If no arguments are given,
.Va ROOTSHELL
is executed, otherwise whatever you typed.
If the first argument begins with a word prefixed by
.Li pkg_ ,
then the
.Ql chroot
argument can be omitted (it is implied).
.It removeroot
Remove the entire chroot tree.
You should do it with this target because it
will take care to umount needed mount points.
.It auto
This executes several targets automatically, using
.Sy AUTO_TARGET
as
.Sy BUILD_PKG_COMP_TARGET
during the build.
The order is: makeroot, build and removeroot.
This is useful to create binary packages of several pkgsrc and their
dependencies automatically.
For this to be useful, you need to set
.Sy REAL_PACKAGES
and use
.Sy AUTO_PACKAGES
or pass package names through the command line.
.Pp
If the magic word
.Ql resume
is passed as the unique argument to this target,
.Nm
will attempt to resume a previous automatic build for the given configuration.
.El
.Sh NOTES
This program uses nullfs to create virtual copies of real trees inside the
chroot environment.
.Pp
You need to install the
.Pa security/audit-packages
package in the host system (and have an up to date vulnerabilities database)
if you want security checks to work inside the
chroot environment.
.Sh SEE ALSO
.Xr pkg_delete 1 ,
.Xr pkgsrc 7 ,
.Xr mount_null 8 ,
.Xr sync 8 ,
.Xr sysctl 8
.Sh AUTHORS
.An Julio M. Merino Vidal Aq Mt jmmv@NetBSD.org