Previously, when updating multiple packages, we just updated them in arbitrary order. However, when some of those packages depended on each other, it could happen that some of the intermediary commits would not build because of version constraints on dependencies.
If we want each commit in the history to build when feasible, we need to consider four different scenarios:
1. Updated dependant is compatible with both the old and the new version of the dependency. Order of commits does not matter. But updating dependents first (i.e. reverse topological order) is useful since it allows building each package on the commit that updates it with minimal rebuilds.
2. Updated dependant raises the minimal dependency version. Dependency needs to be updated first (i.e. topological order).
3. Old dependant sets the maximal dependency version. Dependant needs to be updated first (i.e. reverse topological order).
4. Updated dependant depends on exact version of dependency and they are expected to be updated in lockstep. The earlier commit will be broken no matter the order.
This change allows selecting the order of updates to facilitate the first three scenarios. Since most package sets only have loose version constraints, the reverse topological order will generally be the most convenient. In major package set updates like bumping GNOME release, there will be exceptions (e.g. libadwaita typically requires GTK 4 from the same release) but those were probably in broken order before as well.
The downside of this feature is that it is quite slow – it requires instantiating each package and then querying Nix store for requisites.
It may also fail to detect dependency if there are multiple variants of the package and dependant uses a different one than the canonical one.
Testing with:
env GNOME_UPDATE_STABILITY=unstable NIX_PATH=nixpkgs=$HOME/Projects/nixpkgs nix-shell maintainers/scripts/update.nix --arg predicate '(path: pkg: path == ["gnome-shell"] || path == ["mutter"] || path == ["glib"] || path == ["gtk3"] || path == ["pango"] || path == ["gnome-text-editor"])' --argstr order reverse-topological --argstr commit true --argstr max-workers 4
Use `Process.communicate()` instead of `Process.wait()` to ensure the
`stdin` and `stdout` OS pipe buffers don't get full and cause a deadlock
waiting for the buffers to get emptied.
The context manager would previously just terminate early on exception.
As a result, the worktree and branch would not get pruned when update script failed.
Let’s wrap the cleanup code in `finally` block as suggested by Python docs:
https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager
When updating a package using nix-update-script with `--argstr commit true`,
update.nix would not detect the changes because nix-update would stage them
and `git diff` would be empty.
We now detect both staged and unstaged changes to handle this use case.
When an update script fails, it might still modify the source tree.
These changes would then be committed in the next update attempt.
Let’s make sure the worktree is clean before updating to avoid that.
`update.nix` extracts `passthru.updateScript` attributes in the main repo
and when they are relative paths (e.g. `./update.sh`), Nix will resolve them
to absolute paths in the main repo.
Update scripts can use $(dirname $0) to get the location of files they
should update but that would point to the main repo.
We want them to modify the appropriate git worktree instead
so we replace the prefix accordingly.
`git rev-parse --show-toplevel` will resolve symlinks but, fortunately,
Nix will do that as well, so the path will match:
https://github.com/NixOS/nixpkgs/pull/98304#issuecomment-695761754
Instead of having the updateScript support returning JSON object,
it should be sufficient to specify attrPath in passthru.updateScript.
It is much easier to use.
The former is now considered experimental.
Update scripts can now declare features using
passthru.updateScript = {
command = [ ../../update.sh pname ];
supportedFeatures = [ "commit" ];
};
A `commit` feature means that when the update script finishes successfully,
it will print a JSON list like the following:
[
{
"attrPath": "volume_key",
"oldVersion": "0.3.11",
"newVersion": "0.3.12",
"files": [
"/path/to/nixpkgs/pkgs/development/libraries/volume-key/default.nix"
]
}
]
and data from that will be used when update.nix is run with --argstr commit true
to create commits.
We will create a new git worktree for each thread in the pool and run the update
script there. Then we will commit the change and cherry pick it in the main repo,
releasing the worktree for a next change.
To make updating large attribute sets faster, the update scripts
are now run in parallel.
Please note the following changes in semantics:
- The string passed to updateScript needs to be a path to an executable file.
- The updateScript can also be a list: the tail elements will then be passed
to the head as command line arguments.
