Merge #20619: guix: Quality of life improvements

570e43fe72 guix: Print build params inside/outside of container (Carl Dong)
2f9d1fdde6 guix: Move DISTSRC determination to guix-build.sh (Carl Dong)
0b7cd07bb5 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong)
d27ff8b86a guix: Add more sanity checks to guix-build.sh (Carl Dong)
57f9533146 guix: Add section headings to guix-build.sh (Carl Dong)
38b7b2ed72 genbuild: Specify rev-parse length (Carl Dong)
036dc740da docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong)
34f0fda2d3 guix: Small updates to README wording (Carl Dong)
402e3a5b1e guix: Update HOSTS README entry for new architectures (Carl Dong)
cfa7ceb21b guix: Remove README development environment section (Carl Dong)
93b6a8544a guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong)
0f31e24703 guix: Add SUBSTITUTE_URLS option (Carl Dong)
444fcfca90 guix: Make guix honor MAX_JOBS setting (Carl Dong)

Pull request description:

  After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them.

  Notable changes:
  1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment
  2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules
  3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process.
  4. README cleanup

ACKs for top commit:
  fanquake:
    ACK 570e43fe72 - lets move this forward.

Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328

contrib/guix/README.md

@@ -40,25 +40,27 @@ Otherwise, follow the [Guix installation guide][guix/bin-install].
 
 Guix allows us to achieve better binary security by using our CPU time to build
 everything from scratch. However, it doesn't sacrifice user choice in pursuit of
-this: users can decide whether or not to bootstrap and to use substitutes.
+this: users can decide whether or not to bootstrap and to use substitutes
+(pre-built packages).
 
 After installation, you may want to consider [adding substitute
-servers](#speeding-up-builds-with-substitute-servers) to speed up your build if
+servers](#speeding-up-builds-with-substitute-servers) from which to download
-that fits your security model (say, if you're just testing that this works).
+pre-built packages to speed up your build if that fits your security model (say,
-This is skippable if you're using the [Dockerfile][fanquake/guix-docker].
+if you're just testing that this works). Substitute servers are set up by
+default if you're using the [Dockerfile][fanquake/guix-docker].
 
-If you prefer not to use any substitutes, make sure to set
+If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
-`ADDITIONAL_GUIX_ENVIRONMENT_FLAGS` like the following snippet. The first build
+like in the following snippet. The first build will take a while, but the
-will take a while, but the resulting packages will be cached for future builds.
+resulting packages will be cached for future builds.
 
 ```sh
-export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--no-substitutes'
+export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
 ```
 
 Likewise, to perform a bootstrapped build (takes even longer):
 
 ```sh
-export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap --no-substitutes'
+export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap'
 ```
 
 ### Using a version of Guix with `guix time-machine` capabilities
@@ -82,17 +84,6 @@ export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
 
 ## Usage
 
-### As a Development Environment
-
-For a Bitcoin Core depends development environment, simply invoke
-
-```sh
-guix environment --manifest=contrib/guix/manifest.scm
-```
-
-And you'll land back in your shell with all the build dependencies required for
-a `depends` build injected into your environment.
-
 ### As a Tool for Deterministic Builds
 
 From the top of a clean Bitcoin Core repository:
@@ -113,10 +104,8 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
 * _**HOSTS**_
 
   Override the space-separated list of platform triples for which to perform a
-  bootstrappable build. _(defaults to "x86\_64-linux-gnu
+  bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
-  arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_
+  aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32")_
-
-  > Windows and OS X platform triplet support are WIP.
 
 * _**SOURCES_PATH**_
 
@@ -147,13 +136,29 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
   string) is interpreted the same way as not setting `V` at all, and that `V=0`
   has the same effect as `V=1`.
 
-* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
+* _**SUBSTITUTE_URLS**_
 
-  Additional flags to be passed to `guix environment`. For a fully-bootstrapped
+  A whitespace-delimited list of URLs from which to download pre-built packages.
+  A URL is only used if its signing key is authorized (refer to the [substitute
+  servers section](#speeding-up-builds-with-substitute-servers) for more
+  details).
+
+* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
+
+  Additional flags to be passed to all `guix` commands. For a fully-bootstrapped
   build, set this to `--bootstrap --no-substitutes` (refer to the [security
   model section](#choosing-your-security-model) for more details). Note that a
   fully-bootstrapped build will take quite a long time on the first run.
 
+* _**ADDITIONAL_GUIX_TIMEMACHINE_FLAGS**_
+
+  Additional flags to be passed to `guix time-machine`.
+
+* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
+
+  Additional flags to be passed to the invocation of `guix environment` inside
+  `guix time-machine`.
+
 ## Tips and Tricks
 
 ### Speeding up builds with substitute servers
@@ -161,14 +166,15 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
 _This whole section is automatically done in the convenience
 [Dockerfiles][fanquake/guix-docker]_
 
-For those who are used to life in the fast _(and trustful)_ lane, you can use
+For those who are used to life in the fast _(and trustful)_ lane, you can
-[substitute servers][guix/substitutes] to enable binary downloads of packages.
+specify [substitute servers][guix/substitutes] from which to download pre-built
+packages.
 
 > For those who only want to use substitutes from the official Guix build farm
 > and have authorized the build farm's signing key during Guix's installation,
 > you don't need to do anything.
 
-#### Authorize the signing keys
+#### Step 1: Authorize the signing keys
 
 For the official Guix build farm at https://ci.guix.gnu.org, run as root:
 
@@ -182,7 +188,7 @@ For dongcarl's substitute server at https://guix.carldong.io, run as root:
 wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
 ```
 
-#### Use the substitute servers
+#### Step 2: Specify the substitute servers
 
 The official Guix build farm at https://ci.guix.gnu.org is automatically used
 unless the `--no-substitutes` flag is supplied.
@@ -196,7 +202,7 @@ To use dongcarl's substitute server for Bitcoin Core builds after having
 [authorized his signing key](#authorize-the-signing-keys):
 
 ```
-export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"'
+export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
 ```
 
 ## FAQ
@@ -212,9 +218,9 @@ As mentioned at the bottom of [this manual page][guix/bin-install]:
 
 ### When will Guix be packaged in debian?
 
-Vagrant Cascadian has been making good progress on this
+Thanks to Vagrant Cascadian's diligent work, Guix is now [in debian
-[here][debian/guix-package]. We have all the pieces needed to put up an APT
+experimental][debian/guix-experimental]! Hopefully it will make its way into a
-repository and will likely put one up soon.
+release soon.
 
 [b17e]: http://bootstrappable.org/
 [r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
@@ -226,5 +232,5 @@ repository and will likely put one up soon.
 [guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
 [guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html
 
-[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
+[debian/guix-experimental]: https://packages.debian.org/experimental/guix
 [fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix

contrib/guix/guix-build.sh

@@ -2,6 +2,116 @@
 export LC_ALL=C
 set -e -o pipefail
 
+###################
+## Sanity Checks ##
+###################
+
+################
+# Check 1: Make sure that we can invoke required tools
+################
+for cmd in git make guix cat mkdir; do
+    if ! command -v "$cmd" > /dev/null 2>&1; then
+        echo "ERR: This script requires that '$cmd' is installed and available in your \$PATH"
+        exit 1
+    fi
+done
+
+################
+# Check 2: Make sure GUIX_BUILD_OPTIONS is empty
+################
+#
+# GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that
+# can perform builds. This seems like what we want instead of
+# ADDITIONAL_GUIX_COMMON_FLAGS, but the value of GUIX_BUILD_OPTIONS is actually
+# _appended_ to normal command-line options. Meaning that they will take
+# precedence over the command-specific ADDITIONAL_GUIX_<CMD>_FLAGS.
+#
+# This seems like a poor user experience. Thus we check for GUIX_BUILD_OPTIONS's
+# existence here and direct users of this script to use our (more flexible)
+# custom environment variables.
+if [ -n "$GUIX_BUILD_OPTIONS" ]; then
+cat << EOF
+Error: Environment variable GUIX_BUILD_OPTIONS is not empty:
+  '$GUIX_BUILD_OPTIONS'
+
+Unfortunately this script is incompatible with GUIX_BUILD_OPTIONS, please unset
+GUIX_BUILD_OPTIONS and use ADDITIONAL_GUIX_COMMON_FLAGS to set build options
+across guix commands or ADDITIONAL_GUIX_<CMD>_FLAGS to set build options for a
+specific guix command.
+
+See contrib/guix/README.md for more details.
+EOF
+exit 1
+fi
+
+################
+# Check 3: Make sure that we're not in a dirty worktree
+################
+if ! git diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then
+cat << EOF
+ERR: The current git worktree is dirty, which may lead to broken builds.
+
+     Aborting...
+
+Hint: To make your git worktree clean, You may want to:
+      1. Commit your changes,
+      2. Stash your changes, or
+      3. Set the 'FORCE_DIRTY_WORKTREE' environment variable if you insist on
+         using a dirty worktree
+EOF
+exit 1
+else
+    GIT_COMMIT=$(git rev-parse --short=12 HEAD)
+fi
+
+################
+# Check 4: Make sure that build directories do no exist
+################
+
+# Default to building for all supported HOSTs (overridable by environment)
+export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu
+                       x86_64-w64-mingw32}"
+
+DISTSRC_BASE="${DISTSRC_BASE:-${PWD}}"
+
+# Usage: distsrc_for_host HOST
+#
+#   HOST: The current platform triple we're building for
+#
+distsrc_for_host() {
+    echo "${DISTSRC_BASE}/distsrc-${GIT_COMMIT}-${1}"
+}
+
+# Accumulate a list of build directories that already exist...
+hosts_distsrc_exists=""
+for host in $HOSTS; do
+    if [ -e "$(distsrc_for_host "$host")" ]; then
+        hosts_distsrc_exists+=" ${host}"
+    fi
+done
+
+if [ -n "$hosts_distsrc_exists" ]; then
+# ...so that we can print them out nicely in an error message
+cat << EOF
+ERR: Build directories for this commit already exist for the following platform
+     triples you're attempting to build, probably because of previous builds.
+     Please remove, or otherwise deal with them prior to starting another build.
+
+     Aborting...
+
+EOF
+for host in $hosts_distsrc_exists; do
+    echo "     ${host} '$(distsrc_for_host "$host")'"
+done
+exit 1
+else
+    mkdir -p "$DISTSRC_BASE"
+fi
+
+#########
+# Setup #
+#########
+
 # Determine the maximum number of jobs to run simultaneously (overridable by
 # environment)
 MAX_JOBS="${MAX_JOBS:-$(nproc)}"
@@ -16,11 +126,23 @@ SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
 # Execute "$@" in a pinned, possibly older version of Guix, for reproducibility
 # across time.
 time-machine() {
+    # shellcheck disable=SC2086
     guix time-machine --url=https://github.com/dongcarl/guix.git \
                       --commit=b066c25026f21fb57677aa34692a5034338e7ee3 \
+                      --max-jobs="$MAX_JOBS" \
+                      ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
+                      ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
                       -- "$@"
 }
 
+# Make sure an output directory exists for our builds
+OUTDIR="${OUTDIR:-${PWD}/output}"
+[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
+
+#########
+# Build #
+#########
+
 # Function to be called when building for host ${1} and the user interrupts the
 # build
 int_trap() {
@@ -38,9 +160,9 @@ and untracked files and directories will be wiped, allowing you to start anew.
 EOF
 }
 
-# Deterministically build Bitcoin Core for HOSTs (overridable by environment)
+# Deterministically build Bitcoin Core
 # shellcheck disable=SC2153
-for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32}; do
+for host in $HOSTS; do
 
     # Display proper warning when the user interrupts the build
     trap 'int_trap ${host}' INT
@@ -50,6 +172,19 @@ for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv
         # for the particular $HOST we're building for
         export HOST="$host"
 
+        # shellcheck disable=SC2030
+cat << EOF
+INFO: Building commit ${GIT_COMMIT:?not set} for platform triple ${HOST:?not set}:
+      ...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
+      ...running at most ${MAX_JOBS:?not set} jobs
+      ...from worktree directory: '${PWD}'
+          ...bind-mounted in container to: '/bitcoin'
+      ...in build directory: '$(distsrc_for_host "$HOST")'
+          ...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")'
+      ...outputting in: '${OUTDIR:?not set}'
+          ...bind-mounted in container to: '/outdir'
+EOF
+
         # Run the build script 'contrib/guix/libexec/build.sh' in the build
         # container specified by 'contrib/guix/manifest.scm'.
         #
@@ -99,20 +234,36 @@ for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv
         #     make the downloaded depends sources available to it. The sources
         #     should have been downloaded prior to this invocation.
         #
-        # shellcheck disable=SC2086
+        #  ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"}
+        #
+        #                     fetch substitute from SUBSTITUTE_URLS if they are
+        #                     authorized
+        #
+        #    Depending on the user's security model, it may be desirable to use
+        #    substitutes (pre-built packages) from servers that the user trusts.
+        #    Please read the README.md in the same directory as this file for
+        #    more information.
+        #
+        # shellcheck disable=SC2086,SC2031
         time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \
                                  --container \
                                  --pure \
                                  --no-cwd \
                                  --share="$PWD"=/bitcoin \
+                                 --share="$DISTSRC_BASE"=/distsrc-base \
+                                 --share="$OUTDIR"=/outdir \
                                  --expose="$(git rev-parse --git-common-dir)" \
                                  ${SOURCES_PATH:+--share="$SOURCES_PATH"} \
-                                 ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
+                                 --max-jobs="$MAX_JOBS" \
+                                 ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
+                                 ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
                                  -- env HOST="$host" \
                                         MAX_JOBS="$MAX_JOBS" \
                                         SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
                                         ${V:+V=1} \
                                         ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
+                                        DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \
+                                        OUTDIR=/outdir \
                                       bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
     )
 

contrib/guix/libexec/build.sh

@@ -11,9 +11,15 @@ if [ -n "$V" ]; then
     export VERBOSE="$V"
 fi
 
-# Check that environment variables assumed to be set by the environment are set
+# Check that required environment variables are set
-echo "Building for platform triple ${HOST:?not set} with reference timestamp ${SOURCE_DATE_EPOCH:?not set}..."
+cat << EOF
-echo "At most ${MAX_JOBS:?not set} jobs will run at once..."
+Required environment variables as seen inside the container:
+    HOST: ${HOST:?not set}
+    SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set}
+    MAX_JOBS: ${MAX_JOBS:?not set}
+    DISTSRC: ${DISTSRC:?not set}
+    OUTDIR: ${OUTDIR:?not set}
+EOF
 
 #####################
 # Environment Setup #
@@ -23,19 +29,6 @@ echo "At most ${MAX_JOBS:?not set} jobs will run at once..."
 # $HOSTs after successfully building.
 BASEPREFIX="${PWD}/depends"
 
-# Setup an output directory for our build
-OUTDIR="${OUTDIR:-${PWD}/output}"
-[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
-
-# Setup the directory where our Bitcoin Core build for HOST will occur
-DISTSRC="${DISTSRC:-${PWD}/distsrc-${HOST}}"
-if [ -e "$DISTSRC" ]; then
-    echo "DISTSRC directory '${DISTSRC}' exists, probably because of previous builds... Aborting..."
-    exit 1
-else
-    mkdir -p "$DISTSRC"
-fi
-
 # Given a package name and an output name, return the path of that output in our
 # current guix environment
 store_path() {
@@ -189,6 +182,7 @@ esac
 
 # Make $HOST-specific native binaries from depends available in $PATH
 export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}"
+mkdir -p "$DISTSRC"
 (
     cd "$DISTSRC"
 

doc/guix.md

@@ -0,0 +1,3 @@
+# Bootstrappable Bitcoin Core Builds
+
+See [contrib/guix/README.md](../contrib/guix/README.md)

share/genbuild.sh

@@ -31,7 +31,7 @@ if [ "${BITCOIN_GENBUILD_NO_GIT}" != "1" ] && [ -e "$(command -v git)" ] && [ "$
     fi
 
     # otherwise generate suffix from git, i.e. string like "59887e8-dirty"
-    GIT_COMMIT=$(git rev-parse --short HEAD)
+    GIT_COMMIT=$(git rev-parse --short=12 HEAD)
     git diff-index --quiet HEAD -- || GIT_COMMIT="$GIT_COMMIT-dirty"
 fi