2022917223 Add secp256k1_selftest call (Pieter Wuille)
3bfca788b0 Remove explicit enabling of default modules (Pieter Wuille)
4462cb0498 Adapt to libsecp256k1 API changes (Pieter Wuille)
9d47e7b71b Squashed 'src/secp256k1/' changes from 44c2452fd3..21ffe4b22a (Pieter Wuille)
Pull request description:
Now that libsecp256k1 has a release (https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-December/021271.html), update the subtree to match it.
The changes themselves are not very impactful for Bitcoin Core, but include:
* It's no longer needed to specify whether contexts are for signing or verification or both (all contexts support everything), so make use of that in this PR.
* Verification operations can use the static context now, removing the need for some infrastructure in pubkey.cpp to make sure a context exists.
* Most modules are now enabled by default, so we can drop explicit enabling for them.
* CI improvements (in particular, MSVC and more recent MacOS)
* Introduction of an internal int128 type, which has no effect for GCC/Clang builds, but enables 128-bit multiplication in MSVC, giving a ~20% speedup there (but still slower than GCC/Clang).
* Release process changes (process documentation, changelog, ...).
ACKs for top commit:
Sjors:
ACK 2022917223, but 4462cb0498 could use more eyes on it.
achow101:
ACK 2022917223
jonasnick:
utACK 2022917223
Tree-SHA512: 8a9fe28852abe74abd6f96fef16a94d5a427b1d99bff4caab1699014d24698aab9b966a5364a46ed1001c07a7c1d825154ed4e6557c7decce952b77330a8616b
Fixes the issue presented in #25037 in a single (easily backportable)
diff, with no additional refactoring/changes.
Can be tested with:
```bash
./configure \
--disable-tests \
--disable-bench \
--without-libs \
--without-daemon \
--without-gui \
--disable-fuzz-binary \
--without-utils \
--enable-util-util
```
These headers are already included in a default set which are checked
early during configure.
We already use at least sys/types.h and unistd.h unconditionally in
configure.
98868633d1 Bugfix: configure: bitcoin-{cli,tx,util} don't need UPnP, NAT-PMP, or ZMQ (Luke Dashjr)
Pull request description:
As with #23345, these other tools likewise don't use various deps.
ACKs for top commit:
achow101:
ACK 98868633d1
Tree-SHA512: 4be056b8e0c9f69834229aa257187457de1bc34214d320b770834e21ecc1f0ca7aa7b9689fba525928947bfabbb461528795f709014fb9618b82f088fe64f271
d216d714aa Revert "build: Use Homebrew's sqlite package if it is available" (fanquake)
Pull request description:
This reverts ee7b84e63c from #20527.
That change was made without any rationale, maybe other than, a brew
installed version might be newer, and that's "better". However when
building from source on macOS, it just results in drastically worse
performance, and issues / confusion like #25724.
The difference in performance can be observed using the example from https://github.com/bitcoin/bitcoin/issues/25724#issuecomment-1213554922,
but minified i.e:
```bash
time src/bitcoin-cli createwallet speedy true
time src/bitcoin-cli importdescriptors '[
{"desc":"raw(00145846369f3d6ba366d6f5a903fb5cf4dca3763c0e)#k9wh6v62","timestamp":"now"},
{"desc":"raw(001420800aabf13f3a4c4ce3ce4c66cecf1d17f21a6e)#6m0hlfh4","timestamp":"now"},
{"desc":"raw(0014c6bf9715e06d73ebf9b3b02d5cc48d24d8bbabc1)#wyavh36r","timestamp":"now"},
{"desc":"raw(00141ba7807b3f46af113beaea5c698428ce7138cd8a)#jctdsups","timestamp":"now"},
{"desc":"raw(00140c1bd27f10fff01b36ddf3c1febaa1acff19b080)#9s6nc3pk","timestamp":"now"},
{"desc":"raw(00141226e31987e4bc2e63c0ee12908f675e40464b20)#9pp7qm39","timestamp":"now"},
{"desc":"raw(0014f73f149f7503960a5e849c6ee7a8a8c336f631cb)#qtkxv9fc","timestamp":"now"},
{"desc":"raw(0014c8ccb4d81ffc769fc5fdd8d7eed69b0e0cae5749)#hn39qayv","timestamp":"now"},
{"desc":"raw(001498565aead2d67a22a6021d55210f2a917fc22169)#6ar3vwsx","timestamp":"now"},
{"desc":"raw(001403013248ac0cd9eabe176cad162cda2a19f771e1)#4m47mukd","timestamp":"now"}
]'
```
Running master, when building from souce and using brew installed
sqlite, this takes ~3.4s. With this PR, the same operation takes ~0.3s.
Resolves the "build from source" portion of #25724. Building from
depends is still not ideal, however I have some other changes that might
help improve things in that case.
Related performance issue reports:
* https://github.com/bitcoindevkit/bdk/issues/749
* https://bitcoin.stackexchange.com/questions/113898/bitcoin-v23-is-10-times-slower-than-v22-on-macos-for-basic-regtest-tests
* https://github.com/bitcoin/bitcoin/issues/25724
* https://github.com/bitcoin/bitcoin/pull/25985#issuecomment-1245942400
ACKs for top commit:
achow101:
ACK d216d714aa
jarolrod:
ACK d216d714aa
hebasto:
ACK d216d714aa, I have reviewed the code and it looks OK, I agree it can be merged. No conflicts with our build [docs](d216d714aa/doc/build-osx.md (descriptor-wallet-support)).
Tree-SHA512: 1bb4b44385b11fa9fe66edd7449278f9e47a6cc679b7111f9adf17db94c34e29c9cceafc917454e134420db40b24b56da29226af6f43e6dbeff822b79b77ed60
We currently perform the same check twice, to put the same set of flags
in two different variables. Split the checks so we test for crc and crypto
extensions independently.
If we don't want to split, we should just delete the second AX_CHECK_COMPILE_FLAG
check, and set ARM_CRC_CXXFLAGS & ARM_CRC_CXXFLAGS at the same time.
We already use a mix of <cstdlib> and stdlib.h unconditionally throughout
the codebase.
Us checking this header also duplicates work already done by autotools.
Currently stdlib.h is checked for 3 times during a ./configure run, after
this change, at least it's only twice.
We already use a mix of <cstdio> and stdio.h unconditionally throughout
the codebase.
Us checking this header also duplicates work already done by autotools.
Currently stdio.h is checked for 3 times during a ./configure run, after
this change, at least it's only twice.
We don't include strings.h anywhere.
This is also already checked for by autoconf, so us checking for it just
means a 3rd existence check during ./configure.
This reverts ee7b84e63c from #20527.
This change was made without any rationale, maybe other than a brew
installed version might be newer, and that's "better". However when
building from source on macOS, it just results in drastically worse
perofrmance, and results in issues / confusions like #25724.
Resolves the "build from source" portion of #25724. Building from
depends is still not ideal, however I have some other changes that might
help improve things in that case.
The difference in performance can be observed using the example from
https://github.com/bitcoin/bitcoin/issues/25724#issuecomment-1213554922,
but minified to only 10 descriptors. i.e:
```bash
time src/bitcoin-cli createwallet speedy true
time src/bitcoin-cli importdescriptors '[
{"desc":"raw(00145846369f3d6ba366d6f5a903fb5cf4dca3763c0e)#k9wh6v62","timestamp":"now"},
{"desc":"raw(001420800aabf13f3a4c4ce3ce4c66cecf1d17f21a6e)#6m0hlfh4","timestamp":"now"},
{"desc":"raw(0014c6bf9715e06d73ebf9b3b02d5cc48d24d8bbabc1)#wyavh36r","timestamp":"now"},
{"desc":"raw(00141ba7807b3f46af113beaea5c698428ce7138cd8a)#jctdsups","timestamp":"now"},
{"desc":"raw(00140c1bd27f10fff01b36ddf3c1febaa1acff19b080)#9s6nc3pk","timestamp":"now"},
{"desc":"raw(00141226e31987e4bc2e63c0ee12908f675e40464b20)#9pp7qm39","timestamp":"now"},
{"desc":"raw(0014f73f149f7503960a5e849c6ee7a8a8c336f631cb)#qtkxv9fc","timestamp":"now"},
{"desc":"raw(0014c8ccb4d81ffc769fc5fdd8d7eed69b0e0cae5749)#hn39qayv","timestamp":"now"},
{"desc":"raw(001498565aead2d67a22a6021d55210f2a917fc22169)#6ar3vwsx","timestamp":"now"},
{"desc":"raw(001403013248ac0cd9eabe176cad162cda2a19f771e1)#4m47mukd","timestamp":"now"}
]'
```
Running master, when building from souce and using brew installed
sqlite, this takes ~3.4s. With this PR, the same operation takes ~0.3s.
8df063e537 build: Fix help string for `--enable-external-signer` configure option (Hennadii Stepanov)
Pull request description:
This PR is a follow up of bitcoin/bitcoin#24065 and fixes the help string according to the actual default value 816ca01650/configure.ac (L324-L327)
ACKs for top commit:
kristapsk:
cr utACK 8df063e537
jarolrod:
ACK 8df063e537
Tree-SHA512: ad3f457a53c9238ddd8ded9efd1224e564e6cb9da8b7ff7733a11e32a7daad5c0f6c6223509218f44944a874470cb0d2447897662eaf4e78c763b30785717c50
9aeeb75cf9 Add symlinks for hardcoded Makefiles in out of tree builds (Pablo Greco)
Pull request description:
When doing out of tree builds, some hardwired Makefiles are not symlinked, which makes it a bit more uncomfortable to run some instances of make.
There's no "real" functionality loss without this patch because the symlinked files are just for quick access to thinks in the main Makefile
ACKs for top commit:
hebasto:
ACK 9aeeb75cf9, tested on Ubuntu 22.04.
Tree-SHA512: 656f73c387584cee34f66b3f95993267a40b915762949c7a84b73ba2ea8d37b7b5850733377110e0110ed2f7da64e6a5f9b303812080fe7815154dbb40c8a44c
Our usage of std::atomic is with it's own exchange function, not
std::atomic_exchange. So we should be looking specifically for that
function.
Additionally, -pthread and -lpthread have an effect on whether -latomic
will be needed, so the atomics check needs to use these flags as well.
This will make the flags in use better match what is actually used when
linking.
This removes the need for -latomic for riscv builds, which resolves a
guix cross architecture reproducibility issue.
Boost conatiner_hash (included via functional -> multi_index) uses
std::unary_function, which was deprecated in C++11, and "removed" in
C++17. It's use causes wanrings with newer compilers, i.e GCC 12.1.
```bash
/bitcoin/depends/aarch64-unknown-linux-gnu/include/boost/container_hash/hash.hpp:131:33:
warning: 'template<class _Arg, class _Result> struct std::unary_function' is deprecated [-Wdeprecated-declarations]
131 | struct hash_base : std::unary_function<T, std::size_t> {};
| ^~~~~~~~~~~~~~
In file included from /usr/include/c++/12/bits/unique_ptr.h:37,
from /usr/include/c++/12/memory:76,
from ./init.h:10,
from init.cpp:10:
/usr/include/c++/12/bits/stl_function.h:117:12: note: declared here
117 | struct unary_function
```
Use the MACRO outlined in
https://github.com/boostorg/container_hash/issues/22, to prevent it's
use.
BOOST_NO_CXX98_FUNCTION_BASE:
> The standard library no longer supports std::unary_function and std::binary_function.
> They were deprecated in C++11 and is removed from C++14.
See:
https://github.com/boostorg/config/pull/430https://en.cppreference.com/w/cpp/utility/functional/unary_functionhttps://www.boost.org/doc/libs/master/libs/config/doc/html/boost_config/boost_macro_reference.html
58a9601dff build: globally define NOMINMAX (fanquake)
Pull request description:
Define (and document) `NOMINMAX` once, rather than across multiple
source files.
Defining this prevents the definition of min/max macros when using
mingw-w64, which may conflict with unprefixed std::min/max usage. While
that might not be the case for us, we'd always prefer to use the standard
library in any case.
For example:
73cadc06c6/mingw-w64-headers/include/ntdef.h (L289-L300)
Note that we already define NOMINMAX globally when building with MSVC.
Guix Build (arm64):
```bash
d3a3b7045dc1677f6a0a2a73a484f156c81ae764058003d9e870b346912b744a guix-build-58a9601dffa6/output/arm-linux-gnueabihf/SHA256SUMS.part
3e66540a3f8c8a10864ab2fed69581241fa41af86bbb028e5f7c3dd4ba859c64 guix-build-58a9601dffa6/output/arm-linux-gnueabihf/bitcoin-58a9601dffa6-arm-linux-gnueabihf-debug.tar.gz
78756e20d45e327cfd7f9e65858bf6d3814bcbe08f9f825fd6dfc9dff999ea6d guix-build-58a9601dffa6/output/arm-linux-gnueabihf/bitcoin-58a9601dffa6-arm-linux-gnueabihf.tar.gz
11073e88d4fd0411c5119a3dca3a90788693fa9aa5134339c84be98ae893cd77 guix-build-58a9601dffa6/output/arm64-apple-darwin/SHA256SUMS.part
deffd5f8c6286be34bc35e71ec70300bacb37e1b1a83e67c0833cb57d7a45529 guix-build-58a9601dffa6/output/arm64-apple-darwin/bitcoin-58a9601dffa6-arm64-apple-darwin-unsigned.dmg
acee7e98c5ec41f67e86c78dc5b45fa8bc82de86a04b8c43dbf9c59e7aff36a9 guix-build-58a9601dffa6/output/arm64-apple-darwin/bitcoin-58a9601dffa6-arm64-apple-darwin-unsigned.tar.gz
83f7cbaf6680fe8981db9260b97ca87d609a76c0857a744c7d406645d2484e1b guix-build-58a9601dffa6/output/arm64-apple-darwin/bitcoin-58a9601dffa6-arm64-apple-darwin.tar.gz
b8c73b40a5e307e9e7e482ce92164990d442f3f105a5240ec6eb96a775cb35d5 guix-build-58a9601dffa6/output/dist-archive/bitcoin-58a9601dffa6.tar.gz
cc435cd925771af7e261d0121047339ea8fddb0d1548b699c12108a62988cd32 guix-build-58a9601dffa6/output/powerpc64-linux-gnu/SHA256SUMS.part
7a68bd3181a054056b0a5eb6e830b90ac4ba8435114127d5f1720643011aa78f guix-build-58a9601dffa6/output/powerpc64-linux-gnu/bitcoin-58a9601dffa6-powerpc64-linux-gnu-debug.tar.gz
bc55b95e263c455a964d9463a3ee60dabee1d10cefc6641ed29a3b1b317d61e0 guix-build-58a9601dffa6/output/powerpc64-linux-gnu/bitcoin-58a9601dffa6-powerpc64-linux-gnu.tar.gz
49df78009d80af02262806c6c395e2c884a979b1ea13d01aa27d8188403e29d1 guix-build-58a9601dffa6/output/powerpc64le-linux-gnu/SHA256SUMS.part
29dc7a0e10707b3511fa2afb6977df7ebbb67f796d8be5a042abc14eba764aef guix-build-58a9601dffa6/output/powerpc64le-linux-gnu/bitcoin-58a9601dffa6-powerpc64le-linux-gnu-debug.tar.gz
51b7f8e1bccff1e2ce1860bbc382eefe648b90cc3374cdfa3a95a7454386e77d guix-build-58a9601dffa6/output/powerpc64le-linux-gnu/bitcoin-58a9601dffa6-powerpc64le-linux-gnu.tar.gz
e62e46d8cebbbfc0f587e930acb648fcae99cfe8b2f63aeba98e46e3338fe1e3 guix-build-58a9601dffa6/output/riscv64-linux-gnu/SHA256SUMS.part
fa5d0a074ca586583bf08dbf748909b3ff5e0a54a2e5aaa88abec666e17b4e72 guix-build-58a9601dffa6/output/riscv64-linux-gnu/bitcoin-58a9601dffa6-riscv64-linux-gnu-debug.tar.gz
684b2917fd27a41f884bb6870f7fac847d52b6f8b40df5779d1c674409f7cd14 guix-build-58a9601dffa6/output/riscv64-linux-gnu/bitcoin-58a9601dffa6-riscv64-linux-gnu.tar.gz
7d7cfd0212b49eec48c7f8dc0d97add53096685dfd646feac466c27a45d20c97 guix-build-58a9601dffa6/output/x86_64-apple-darwin/SHA256SUMS.part
d70ae6d060b7832f8741dc5d1958cc0d32702605c863254303107246deec0aa6 guix-build-58a9601dffa6/output/x86_64-apple-darwin/bitcoin-58a9601dffa6-x86_64-apple-darwin-unsigned.dmg
930f3ec43896404208ebdb582c9175e3a5a2470d778722e0001addde84dad99a guix-build-58a9601dffa6/output/x86_64-apple-darwin/bitcoin-58a9601dffa6-x86_64-apple-darwin-unsigned.tar.gz
2d8a9d12aadcf60634db953fcb8bd496a002608e9a64eb7d60bb7ffe1f94489f guix-build-58a9601dffa6/output/x86_64-apple-darwin/bitcoin-58a9601dffa6-x86_64-apple-darwin.tar.gz
10363729ece6e1c2cbdf435483006191bf17d1def2d318ff8357197d91c06ded guix-build-58a9601dffa6/output/x86_64-linux-gnu/SHA256SUMS.part
d50ec8e4f72e8b064b196eb0ece212f7b0b126f4b8b644c4451084cbf0416072 guix-build-58a9601dffa6/output/x86_64-linux-gnu/bitcoin-58a9601dffa6-x86_64-linux-gnu-debug.tar.gz
471e12b8715ecff4d99121c4bb3288ef4b005ca468810a714c67ea3e7c6669e9 guix-build-58a9601dffa6/output/x86_64-linux-gnu/bitcoin-58a9601dffa6-x86_64-linux-gnu.tar.gz
d63946401952d131fdf5df9442c52151d86e53f019234b5ad16fdef0d2976356 guix-build-58a9601dffa6/output/x86_64-w64-mingw32/SHA256SUMS.part
5359782e1eb6f449338f18e053ad82f25382d968690208ae5739d9338eb7bdc7 guix-build-58a9601dffa6/output/x86_64-w64-mingw32/bitcoin-58a9601dffa6-win64-debug.zip
0d387d5a4cb1d712556a3fe5b4bd1e928bb5fbbe57a85ee06c746f132a6b1ec5 guix-build-58a9601dffa6/output/x86_64-w64-mingw32/bitcoin-58a9601dffa6-win64-setup-unsigned.exe
dbfd7419d1d764e853a9dc041e276669b488aea4a80e21e4a175b6c3e512e70c guix-build-58a9601dffa6/output/x86_64-w64-mingw32/bitcoin-58a9601dffa6-win64-unsigned.tar.gz
0ba07504d9d5a12af9144e8b386b2640b48dba067d47c694a44ecffe56b0c0fc guix-build-58a9601dffa6/output/x86_64-w64-mingw32/bitcoin-58a9601dffa6-win64.zip
```
ACKs for top commit:
laanwj:
Code review ACK 58a9601dff
Tree-SHA512: d1c22b3d0d21ef8f9f605ef6ca06353e3f48536d84f3531f93d613a6ccbbe62f12fae0ed09e8b9a8940b0ef33f9d41d9991eb56fbe7c4ab48f0ce7fcf44e08b1
Define (and document) `NOMINMAX` once, rather than across multiple
source files.
Defining this prevents the definition of min/max macros when using
mingw-w64, which may conflict with unprefixed std::min/max usage. While
that might not be the case for us, we'd always prefer to use the standard
library in any case.
For example:
73cadc06c6/mingw-w64-headers/include/ntdef.h (L289-L300)
f0f5cd79b5 Bugfix: configure: Define default for use_libevent (Luke Dashjr)
Pull request description:
Another trivial fix like #25051 - I think this is the only other one missing.
ACKs for top commit:
laanwj:
Code review ACK f0f5cd79b5
Tree-SHA512: 888c2e6d032ef1de5af635e2a9b2b8ab560c86bd10a6cee54aa9aa62ae43f03c19889bb6a2b64cf8982d4cd514f97ca3ed743c71ed0651e9295a4b1726955b9b
Code introduced in #15649 added usage of `timingsafe_bcmp()`, if
available, otherwise falling back to our own implementation. However
the relevant build system check was never added, so currently, we'll
always just use our implementation, as HAVE_TIMINGSAFE_BCMP will never
be defined.
Add the check for timingsafe_bcmp. Note that as far as I'm aware, it's
only available on OpenBSD.
06e18e0b53 build: use BOOST_MULTI_INDEX_ENABLE_SAFE_MODE when debugging (fanquake)
Pull request description:
Use of this macro enables precondition checks for iterators and functions of the library. It's use is recommended in debug builds. See https://www.boost.org/doc/libs/1_78_0/libs/multi_index/doc/tutorial/debug.html for more info.
There is also a `BOOST_MULTI_INDEX_ENABLE_INVARIANT_CHECKING` macro:
> When this mode is in effect, all public functions of Boost.MultiIndex will perform post-execution tests aimed at ensuring that the basic internal invariants of the data structures managed are preserved.
ACKs for top commit:
laanwj:
Concept and code review ACK 06e18e0b53
Tree-SHA512: 7ee489eccda81c7dbca9210af6d3007d5b2c704b645139d2714c077af157789dd9478c29d0d212e210e96686ea83713aaf3d458e879122b3cde64f3e3e3789d2
7fd0860d12 Bugfix: configure: Define defaults for enable_arm_{crc,shani} (Luke Dashjr)
Pull request description:
Fix for #17398 and #24115
Trivial, mostly for consistency (you'd have to *try* to break this)
ACKs for top commit:
pk-b2:
ACK 7fd0860d12
seejee:
ACK 7fd0860d12
vincenzopalazzo:
ACK 7fd0860d12
Tree-SHA512: 51c389787c369f431ca57071f03392438bff9fd41f128c63ce74ca30d2257213f8be225efcb5c1329ad80b714f44427d721215d4f848cc8e63060fa5bc8f1f2e
165903406e build: Fix `AC_CHECK_HEADERS` and `AC_CHECK_LIB` for `libnatpmp` package (Hennadii Stepanov)
65cddf604c build: Fix `AC_CHECK_HEADERS` and `AC_CHECK_LIB` for `miniupnpc` package (Hennadii Stepanov)
bbbcb96638 build, refactor: Fix indentation (Hennadii Stepanov)
Pull request description:
Apparently, bitcoin/bitcoin#24391 broke the [ability](https://github.com/bitcoin/bitcoin/pull/22397) of the `configure` script to pick up Homebrew's `miniupnpc` and `libnatpmp` packages on macOS Apple M1.
This PR fixes it.
ACKs for top commit:
promag:
Tested ACK 165903406e
jarolrod:
tACK 165903406e
Tree-SHA512: 93988f59f425890d60582b93d4ac5b2ad03011a5c6dbb44678a3ca591da7518c1c741bc1045b2c763bbe887947f32293b38d55fd7a96f09d2092ad34baa1db21
035fa1f07a build: Remove LIBTOOL_APP_LDFLAGS for bitcoin-chainstate (Cory Fields)
3f0595095d docs: Add libbitcoinkernel_la_SOURCES explanation (Carl Dong)
94ad45deb2 ci: Build libbitcoinkernel (Carl Dong)
26b2e7ffb3 build: Extract the libbitcoinkernel library (Carl Dong)
1df44dd20c b-cs: Define G_TRANSLATION_FUN in bitcoinkernel.cpp (Carl Dong)
83a0bb7cc9 build: Separate lib_LTLIBRARIES initialization (Carl Dong)
c1e16cb31f build: Create .la library for bitcoincrypto (Carl Dong)
8bdfe057c7 build: Create .la library for leveldb (Carl Dong)
05d1525b6d build: Create .la library for crc32c (Carl Dong)
64caf94479 build: Remove vestigial LIBLEVELDB_SSE42 (Carl Dong)
1392e8e2d8 build: Don't add unrelated libs to LIBTEST_* (Carl Dong)
Pull request description:
Part of: #24303
This PR introduces a `libbitcoinkernel` static library linking in the minimal list of files necessary to use our consensus engine as-is. `bitcoin-chainstate` introduced in #24304 now will link against `libbitcoinkernel`.
Most of the changes are related to the build system.
Please read the commit messages for more details.
ACKs for top commit:
theuni:
This may be my favorite PR ever. It's a privilege to ACK 035fa1f07a.
Tree-SHA512: b755edc3471c7c1098847e9b16ab182a6abb7582563d9da516de376a770ac7543c6fdb24238ddd4d3d2d458f905a0c0614b8667aab182aa7e6b80c1cca7090bc
9b0a13a289 tidy: Add include-what-you-use (fanquake)
74cd038e30 refactor: fix includes in src/init (fanquake)
c79ad935f0 refactor: fix includes in src/compat (fanquake)
Pull request description:
We recently added a [`clang-tidy` job](https://github.com/bitcoin/bitcoin/blob/master/ci/test/00_setup_env_native_tidy.sh) to the CI, which generates a compilation database. We can leverage that now existing database to begin running [include-what-you-use](https://include-what-you-use.org/) over the codebase.
This PR demonstrates using a mapping_file to indicate fixups / includes that may differ from IWYU suggestions. In this case, I've added some fixups for glibc includes that I've [upstreamed changes for](https://github.com/include-what-you-use/include-what-you-use/pull/1026):
```bash
# Fixups / upstreamed changes
[
{ include: [ "<bits/termios-c_lflag.h>", private, "<termios.h>", public ] },
{ include: [ "<bits/termios-struct.h>", private, "<termios.h>", public ] },
{ include: [ "<bits/termios-tcflow.h>", private, "<termios.h>", public ] },
]
```
The include "fixing" commits of this PR:
* Adds missing includes.
* Swaps C headers for their C++ counterparts.
* Removes the pointless / unmaintainable `//for abc, xyz` comments. When using IWYU, if anyone wants to see / generate those comments, to see why something is included, it is trivial to do so (IWYU outputs them by default). i.e:
```cpp
// The full include-list for compat/stdin.cpp:
#include <compat/stdin.h>
#include <poll.h> // for poll, pollfd, POLLIN
#include <termios.h> // for tcgetattr, tcsetattr
#include <unistd.h> // for isatty, STDIN_FILENO
```
TODO:
- [ ] Qt mapping_file. There is one in the IWYU repo, but it's for Qt 5.11. Needs testing.
- [ ] Boost mapping_file. There is one in the IWYU repo, but it's for Boost 1.75. Needs testing.
I'm not suggesting we turn this on the for entire codebase, or immediately go-nuts refactoring all includes. However I think our dependency includes are now slim enough, and our CI infrastructure in place such that we can start doing this in some capacity, and just automate away include fixups / refactorings etc.
ACKs for top commit:
MarcoFalke:
review ACK 9b0a13a289
jonatack:
ACK 9b0a13a289 reviewed changes and run CI output in https://cirrus-ci.com/task/4750910332076032
Tree-SHA512: 00beab5a5f2a6fc179abf08321a15391ecccaa91ab56f3c50c511e7b29a0d7c95d8bb43eac2c31489711086f6f77319d43d803cf8ea458e7cd234a780d9ae69e
I strongly recommend reviewing with the following git-diff flags:
--patience --color-moved=dimmed-zebra
Extract out a libbitcoinkernel library linking in all files necessary
for using our consensus engine as-is. Link bitcoin-chainstate against
it.
See previous commit "build: Add example bitcoin-chainstate executable"
for more context.
We explicitly specify -fvisibility=default, which effectively overrides
the effects of --enable-reduced-exports since libbitcoinkernel requires
default symbol visibility
When compiling for mingw-w64, specify -static in both:
- ..._la_CXXFLAGS so that libtool will avoid building two versions of
each object (one PIC, one non-PIC). We just need the one that is
suitable for static linking.
- ..._la_LDFLAGS so that libtool will create a static library.
If we don't specify this, then libtool will prefer the non-static PIC
version of the object, which is built with -DDLL_EXPORT -DPIC for
mingw-w64 targets. This can cause symbol resolution problems when we
link this library against an executable that does specify -all-static,
since that will be built without the -DDLL_EXPORT flag.
Unfortunately, this means that for mingw-w64 we can only build a static
version of the library for now. This will be fixed.
However, on other targets, the shared library creation works fine.
-----
Note to users: You need to either specify:
--enable-experimental-util-chainstate
or,
--with-experimental-kernel-lib
To build the libbitcionkernel library. See the configure help for more
details.
build shared libbitcoinkernel where we can
This change makes naming of `install_name_tool` consistent across
the whole build system.
-BEGIN VERIFY SCRIPT-
sed --in-place --expression='s/INSTALLNAMETOOL/INSTALL_NAME_TOOL/g' $(git grep --files-with-matches 'INSTALLNAMETOOL')
-END VERIFY SCRIPT-
1dd8cbfbc6 build: don't compress macOS DMG (fanquake)
Pull request description:
Skip compressing the macOS DMG, and drop related build steps and dependencies. Uncompressed the DMG increases from ~16mb to ~30mb, which compared to other software a user may download, (Firefox 125mb, VLC 52mb, Open Office 176mb), is still relatively small. When contrasted against the 100's of GB of blockchain data a node will download, an additional 15mb to get the release binary, isn't much additional overhead. Note that if / when we build with LTO enabled for releases, this size will shrink back down significantly again.
`native_libdmg-hfsplus` is not maintained, and I doubt the DMG creation feature will ever be fixed. If at some point `xorrisofs` supports compressing dmgs, we could enable that.
Guix Build on x86_64:
```bash
25b7c8bb7bc8ea014d43cebb844a842d2ac8d5a343039a820d24b649c9e6bc8a guix-build-1dd8cbfbc631/output/arm64-apple-darwin/SHA256SUMS.part
16beb5c52c9bf51b5ce9ef5a0d17c0038238a833383586a1b14acbca78533e4b guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.dmg
d8f89a61a7448d6334dbb3639386a7b6340542393933f35421a9e6dfc724e455 guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.tar.gz
11617dc261ef602433f5bb29956a40a9085dbc783f519f75fbe06e80970148d0 guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin.tar.gz
aa8550d4a394d3161d14ec5e6012ed07354135afb022e905a1946785b4665664 guix-build-1dd8cbfbc631/output/dist-archive/bitcoin-1dd8cbfbc631.tar.gz
2b837f2f971a9738d0b7b8497f7ded740ef5e67c8baa7f30ca33e6b7d826eec8 guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/SHA256SUMS.part
db972b2c06dbde5525a3f9e6ceb9c20a8120bc9a6f15e1d852a4bfac09d88569 guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.dmg
50fe990c3f9923ee92195125faf6517396e7c1b017a8f4f7d52e991ebce52f0c guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.tar.gz
1d9022b0ae46ead41046c40f82291ce363760660a3cd6e6ef6a5b1128b90faef guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin.tar.gz
```
Guix Build on arm64:
```bash
```
ACKs for top commit:
Sjors:
re-tACK 1dd8cbfbc6 on Intel macOS
laanwj:
Build system changes code review ACK 1dd8cbfbc6, I don't know anything about MacOS application formats and their internals so do not have an opinion on the contents of this change.
jarolrod:
ACK 1dd8cbfbc6
Tree-SHA512: 04c5bf78f26a9877777093ec4c50c457107bef59d720839ea5e7d7e4f7961dfee9f86b40cf791524a9e60e9e77403a797e9fcdae3849b60b759f9f66cc31b6ab
c9c4e6cadd build: Do not define `PROVIDE_FUZZ_MAIN_FUNCTION` macro unconditionally (Hennadii Stepanov)
Pull request description:
No need to define the `PROVIDE_FUZZ_MAIN_FUNCTION` macro when the build system has been configured with the `--disable-fuzz-binary` option.
See https://github.com/bitcoin/bitcoin/pull/24336#pullrequestreview-881368272.
ACKs for top commit:
MarcoFalke:
Approach ACK c9c4e6cadd did not review or test 🐤
fanquake:
ACK c9c4e6cadd Checked that `PROVIDE_FUZZ_MAIN_FUNCTION` isn't defined when configuring with `--disable-fuzz-binary`.
Tree-SHA512: 54fbf02ba9f5ecc61b176b8ea7d05e308788d4de3f97ed40913e731300d9dc0edfdfcbf8e0a6e74cf1b2e2ae63f6208a34e03b9c8d203d070c457c4a7d9b5f2c
404c53062b key: use secp256k1_schnorrsig_sign32 over deprecated secp256k1_schnorrsig_sign (fanquake)
ee30bf7c01 build: remove some no-longer-needed var unexporting from configure (fanquake)
2656629767 build: remove --enable-experimental from libsecp256k1 configure (fanquake)
d960d4fd3a build: fix MSVC build after subtree update (dhruv)
afb7a6fe06 Squashed 'src/secp256k1/' changes from 0559fc6e41..8746600eec (fanquake)
Pull request description:
The motivation for this bump is some small build cleanups, including [dropping the `--enable-experimental`](80cf4eea5f) flag from the libsecp configure invocation, as well as some [now-redundant](https://github.com/bitcoin-core/secp256k1/pull/1090) `pkg-config` variable exporting from our own configure. We also get the benefit of a slightly more efficient libsecp configure due to https://github.com/bitcoin-core/secp256k1/pull/1088.
This also includes a change in our code to migrate from using the [now deprecated](99e6568fc6) `secp256k1_schnorrsig_sign` to `secp256k1_schnorrsig_sign32`.
Guix Build (on x86_64):
```bash
b9f6ad90c75f7edd7c4444c6c3401d8b6ab29a8da22ae22ddaedd94688227b5d guix-build-404c53062bb8/output/aarch64-linux-gnu/SHA256SUMS.part
250d47ae299d8385d5590518fa2adaabde76e2566fd27e12bf36b62663d13e13 guix-build-404c53062bb8/output/aarch64-linux-gnu/bitcoin-404c53062bb8-aarch64-linux-gnu-debug.tar.gz
48d610dc6f5169f925f782571dac2f082695f89008beadad4adef4c1b583a612 guix-build-404c53062bb8/output/aarch64-linux-gnu/bitcoin-404c53062bb8-aarch64-linux-gnu.tar.gz
8f04ee26e4079719e3935bd0e4287cc11a2a16875bf01e2a63d67492a1fa5367 guix-build-404c53062bb8/output/arm-linux-gnueabihf/SHA256SUMS.part
7d7d7fcfb032bda92e53abd8d608257f0ef17b1e3e52a1414260b896786fb2dc guix-build-404c53062bb8/output/arm-linux-gnueabihf/bitcoin-404c53062bb8-arm-linux-gnueabihf-debug.tar.gz
30bae2ff3d044f4e39f992a68f6b296b7be2aea350bca4a0415c739a32c20bd9 guix-build-404c53062bb8/output/arm-linux-gnueabihf/bitcoin-404c53062bb8-arm-linux-gnueabihf.tar.gz
5f550fb0b950250eeffce3480ec6403530b0880570a5860ef6c32a3e92eac92f guix-build-404c53062bb8/output/arm64-apple-darwin/SHA256SUMS.part
c10664d13aeec8c860bf72be833c738973ae18e4d28cdf08b2f9bee960ebff1d guix-build-404c53062bb8/output/arm64-apple-darwin/bitcoin-404c53062bb8-arm64-apple-darwin-unsigned.dmg
becab75b11cf4ca6f559f8eef835f3574629f6eb932ac716ed4f8c044a85831f guix-build-404c53062bb8/output/arm64-apple-darwin/bitcoin-404c53062bb8-arm64-apple-darwin-unsigned.tar.gz
bc86433652fe3552f6a13088191364ae7514c9fe3a244da86a6db096bb4922fc guix-build-404c53062bb8/output/arm64-apple-darwin/bitcoin-404c53062bb8-arm64-apple-darwin.tar.gz
1f585cb9a1356343df4b2726ecfe2598c9903304afb047c047c2cef318555dd3 guix-build-404c53062bb8/output/dist-archive/bitcoin-404c53062bb8.tar.gz
9ede534ba2c6cecb550473eead195627327e826ebb0118e23d60ab482d40e241 guix-build-404c53062bb8/output/powerpc64-linux-gnu/SHA256SUMS.part
77ddb7d7d639b1dd4508468a8ef27e45b35c8b2f8624584a70e6b64798a4ea7a guix-build-404c53062bb8/output/powerpc64-linux-gnu/bitcoin-404c53062bb8-powerpc64-linux-gnu-debug.tar.gz
36178c1f1c12942ff05275daa3570f8b45419ee8d9f391d750afb405219986f0 guix-build-404c53062bb8/output/powerpc64-linux-gnu/bitcoin-404c53062bb8-powerpc64-linux-gnu.tar.gz
8a15a4da7a9a5e00c49d9aeedf3c6fc666c0d230be1369eac7caf4571d5905e0 guix-build-404c53062bb8/output/powerpc64le-linux-gnu/SHA256SUMS.part
400c58113f2d07c87e03c8528b292c6aca808a2bccae4b041cad3a26a05b6aad guix-build-404c53062bb8/output/powerpc64le-linux-gnu/bitcoin-404c53062bb8-powerpc64le-linux-gnu-debug.tar.gz
3b9f9d8614ac3a27416e53354b2b0a64d364f91493e9d0f41583a6f492546824 guix-build-404c53062bb8/output/powerpc64le-linux-gnu/bitcoin-404c53062bb8-powerpc64le-linux-gnu.tar.gz
98506b23ee08ad8af958f816da2e4518d661e88d5c6308de1f5e3b2fc787b86c guix-build-404c53062bb8/output/riscv64-linux-gnu/SHA256SUMS.part
c701a7b77cea4fdc2588b511f1b2c71b89c83bfba19fdb2ac113a5a4b14ac392 guix-build-404c53062bb8/output/riscv64-linux-gnu/bitcoin-404c53062bb8-riscv64-linux-gnu-debug.tar.gz
34d58e6392cd58b3c76e30cd8600c0dbefba7e9c6d5df78c3ef23e81c4e4d26a guix-build-404c53062bb8/output/riscv64-linux-gnu/bitcoin-404c53062bb8-riscv64-linux-gnu.tar.gz
92fa30e9c6d81dd1e1514b65d3e1abe68ded897237cd99f66aa760d445109c04 guix-build-404c53062bb8/output/x86_64-apple-darwin/SHA256SUMS.part
bee180b02f178ae9980ef159f65913a71cbd037c4aff5f2906af5f174a677da3 guix-build-404c53062bb8/output/x86_64-apple-darwin/bitcoin-404c53062bb8-x86_64-apple-darwin-unsigned.dmg
ad7d18d779ab7a7944817d1f368d0a6bdd174bf1211b0f90180c8ccf04ec4062 guix-build-404c53062bb8/output/x86_64-apple-darwin/bitcoin-404c53062bb8-x86_64-apple-darwin-unsigned.tar.gz
7489d1d5d48ad95cf58bb11b5fdeccadac6fa758784fb498529fca2330abe069 guix-build-404c53062bb8/output/x86_64-apple-darwin/bitcoin-404c53062bb8-x86_64-apple-darwin.tar.gz
74660fb0ebce2a08b03980a57bffcad62e078dc967a74d2395660ff51c019640 guix-build-404c53062bb8/output/x86_64-linux-gnu/SHA256SUMS.part
cd377fa6b46276c2f8a32e199e6f9adf6aa67315688656709d6dc0744d54a837 guix-build-404c53062bb8/output/x86_64-linux-gnu/bitcoin-404c53062bb8-x86_64-linux-gnu-debug.tar.gz
919c521950369d8ad46db2d15b00abb488abfb080d157a41b2db429122a428ed guix-build-404c53062bb8/output/x86_64-linux-gnu/bitcoin-404c53062bb8-x86_64-linux-gnu.tar.gz
2debca995d432965a8786b6ff74aed42e9e2f1cb0fecbe2d9fc5b850c192fcff guix-build-404c53062bb8/output/x86_64-w64-mingw32/SHA256SUMS.part
e33169f684fb031ec18ed39812617d3eb263257f6c7564b8f4c974ad05fe672c guix-build-404c53062bb8/output/x86_64-w64-mingw32/bitcoin-404c53062bb8-win64-debug.zip
029d0a4180cb908d517fcf689dcf46d42fbf383e11dc609711617066ae039ab0 guix-build-404c53062bb8/output/x86_64-w64-mingw32/bitcoin-404c53062bb8-win64-setup-unsigned.exe
7e349c688cac66436562c4805f420b0536db5a3b3abf54d0e8c7752f59874a5c guix-build-404c53062bb8/output/x86_64-w64-mingw32/bitcoin-404c53062bb8-win64-unsigned.tar.gz
1bff98e82e95c93d6060227408502f5e2d8597d526b912cb6dc0a90ae3094a8f guix-build-404c53062bb8/output/x86_64-w64-mingw32/bitcoin-404c53062bb8-win64.zip
```
ACKs for top commit:
laanwj:
Code review ACK 404c53062b, I checked the changes to our tree thoroughly but didn't review all upstream secp256k1 changes in detail.
gruve-p:
ACK 404c53062b
real-or-random:
utACK 404c53062b I reviewed the diff to Core, I'm with updating to libsecp256k1 master, but I haven't verified that the libsecp256k1 tree here has been updated correctly
Tree-SHA512: e6a6db93ea60ed500df5065178784a915da94adfa7bd45fdbd7b19d701154987ff38c1df7f318119e6c2cb98e28e1ea2eb725bef93d4088403e14537ebffb032
76c60d7b31 test: validation:block_connected tracepoint test (0xb10c)
260e28ece8 test: utxocache:* tracepoint tests (0xb10c)
34b27bac68 test: net:in/out_message tracepoint tests (0xb10c)
c934087b62 test: checks for tracepoint tests (0xb10c)
Pull request description:
This adds functional tests for the USDT tracepoints added in https://github.com/bitcoin/bitcoin/pull/22006 and https://github.com/bitcoin/bitcoin/pull/22902. This partially fixes#23296. The tests **are probably skipped** on most systems as these tests require:
- a Linux system with a kernel that supports BPF (and available kernel headers)
- that Bitcoin Core is compiled with tracepoints for USDT support (default when compiled with depends)
- [bcc](https://github.com/iovisor/bcc) installed
- the tests are run with a privileged user that is able to e.g. do BPF syscalls and load BPF maps
The tests are not yet run in our CI as the CirrusCI containers lack the required permissions (see https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845). Running the tests in a VM in the CI could work, but I haven't experimented with this yet. The priority was to get the actual tests done first to ensure the tracepoints work as intended for the v23.0 release. Running the tracepoint tests in the CI is planned as the next step to finish #23296.
The tests can, however, be run against e.g. release candidates by hand. Additionally, they provide a starting point for tests for future tracepoints. PRs adding new tracepoint should include tests. This makes reviewing these PRs easier.
The tests require privileges to execute BPF sycalls (`CAP_SYS_ADMIN` before Linux kernel 5.8 and `CAP_BPF` and `CAP_PERFMON` on 5.8+) and permissions to `/sys/kernel/debug/tracing/`. It's currently recommended to run the tests in a virtual machine (or on a VPS) where it's sensible to use the `root` user to gain these privileges. Never run python scripts you haven't carefully reviewed with `root` permissions! It's unclear if a non-root user can even gain the required privileges. This needs more experimenting.
The goal here is to test the tracepoint interface to make sure the [documented interface](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#tracepoint-documentation) does not break by accident. The tracepoints expose implementation details. This means we also need to rely on implementation details of Bitcoin Core in these functional tests to trigger the tracepoints. An example is the test of the `utxocache:flush` tracepoint: On Bitcoin Core shutdown, the UTXO cache is flushed twice. The corresponding tracepoint test expects two flushes, too - if not, the test fails. Changing implementation details could cause these tests to fail and the tracepoint API to break. However, we purposefully treat the tracepoints only as [**semi-stable**](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#semi-stable-api). The tracepoints should not block refactors or changes to other internals.
ACKs for top commit:
jb55:
tACK 76c60d7b31
laanwj:
Tested ACK 76c60d7b31
Tree-SHA512: 9a63d945c68102e59d751bd8d2805ddd7b37185408fa831d28a9cb6641b701961389b55f216c475df7d4771154e735625067ee957fc74f454ad7a7921255364c
7b00595d33 build: stop overriding user CXXFLAGS (fanquake)
3e2ef23c3e build: stop overriding user LDFLAGS (fanquake)
35c3fd43c3 build: stop overriding user CPPFLAGS (fanquake)
bc7cc57607 doc: explain why we clear CXXFLAGS with enable-debug (fanquake)
Pull request description:
Historically our build system has hijacked `CXXFLAGS` and friends, and this has always been a source of complaints from users and developers. With this PR, we move away from using `CXXFLAGS`, `CPPFLAGS` and `LDFLAGS`, and instead use `CORE_*FLAGS` variables for our flags / options, leaving autoconfs `FLAG` vars to the user.
Note that there are currently two cases where we will at least clear `CXXFLAGS` (if not alreaddy overridden by the user), when doing debugging or when coverage is enabled, to avoid Autoconfs `-g -O2` CXXFLAG default.
ACKs for top commit:
hebasto:
ACK 7b00595d33
Tree-SHA512: bda936a7aa8f98a1bf1552306845cb4bbab54e19a7a0b9ce3210e10fef70db146e9fe42a0cc8c50b2908506771b5b96f39c334e41323b70ec878e4010373096c
0d01272cd8 build: don't use Boost multi_index serialization (fanquake)
Pull request description:
We don't use the serialization or archiving facilities of multi_index.
So globally disable support, which gives a minor improvement in build
time, i.e less preprocessing work, given we don't link any Boost libs.
See: https://www.boost.org/doc/libs/1_78_0/libs/multi_index/doc/tutorial/creation.html
> Serialization capabilities are automatically provided by just linking with the appropriate Boost.Serialization library module: it is not necessary to explicitly include any header from Boost.Serialization, apart from those declaring the type of archive used in the process. If not used, however, serialization support can be disabled by globally defining the macro BOOST_MULTI_INDEX_DISABLE_SERIALIZATION. Disabling serialization for Boost.MultiIndex can yield a small improvement in build times, and may be necessary in those defective compilers that fail to correctly process Boost.Serialization headers.
ACKs for top commit:
MarcoFalke:
cr ACK 0d01272cd8
Tree-SHA512: 87c664a2f142dc6b8f8598341f9829be3fda8cf614d73cc9a894c8033ee40c6daa9b50f4049ecb1f1e3aaf342568d9a5f5c65af1e04c36ee3a9cb46eca95767b
d4ba2b2cbc compat: remove strnlen back-compat code (fanquake)
Pull request description:
This was needed for mingw (not mingw-w64), and some older versions of
macOS, which we no-longer support.
ACKs for top commit:
hebasto:
ACK d4ba2b2cbc
Tree-SHA512: d1beb9df58464feea3076091361d7d46e4a8901e347644a5fa6f24e052ca24ee0c7c0dd3f2a3d682b0204bf50430fa89eac62121691ea08af6dcf6b907bdec87
532c64a726 build: Fix Boost.Process test for Boost 1.78 (Hennadii Stepanov)
Pull request description:
Rebased #24415 with Luke's suggestion.
Fixes#24413.
ACKs for top commit:
hebasto:
ACK 532c64a726, tested on Mac mini (M1, 2020) + macOS Monterey 12.3 (21E230).
Tree-SHA512: 74f779695f6bbc45a2b7341a1402f747cc0d433d74825c7196cb9f156db0c0299895365f01665bd0bff12a8ebb5ea33a29b9a52f5eac0007ec35d1dca6544705
5a157eb370 Bugfix: configure: Only avoid -isystem for exact /usr/include path (Luke Dashjr)
556ee6f2fa Bugfix: configure: Quote SUPPRESS_WARNINGS sufficiently to preserve brackets (Luke Dashjr)
Pull request description:
The regex includes `[/ ]` which is supposed to match either a forward slash or a space, but m4 treats the brackets as special characters and effectively strips them out, leading to -isystem /usr/include paths except for in the typical scenario where it is the final parameter in the flag string.
ACKs for top commit:
hebasto:
ACK 5a157eb370, tested on Ubuntu 22.04 with clang 14.0.
vasild:
ACK 5a157eb370
Tree-SHA512: 5c8c282b647b7853b8fad1b5b473703c4a0635073d2685a8ac984151046e2c6a859e6972465419d27356dd29a47f21a2a3a6ad402ec434fe1f9882e5a35f0749
999982b06c build: Add --enable-c++20 option (MarcoFalke)
fae679065e Add CSerializedNetMsg::Copy() helper (MarcoFalke)
fabb7c4ba6 Make fs.h C++20 compliant (MarcoFalke)
fae2220f4e scheduler: Capture ‘this’ explicitly in lambda (MarcoFalke)
Pull request description:
This is for CI and devs only and doesn't change that C++17 is the standard we are currently using. The option `--enable-c++20` allows CI to check that the C++17 code in the repo is also valid C++20. (There are some cases where valid C++17 doesn't compile under C++20).
Also, it allows developers to easily play with C++20 in the codebase.
ACKs for top commit:
ryanofsky:
Code review ACK 999982b06c. Since last review was rebased, and enum-conversion change was dropped, and CSerializedNetMsg copy workaround was added
fanquake:
utACK 999982b06c
Tree-SHA512: afc95ba03ea2b937017fc8e2b1449379cd2b6f7093c430d2e344c665a00c51e402d6651cbcbd0be8118ea1e54c3a86e67d2021d19ba1d4da67168e9fcb6b6f83
The regex includes [/ ] which is supposed to match either a forward slash or a
space, but m4 treats the brackets as special characters and effectively strips
them out, leading to -isystem /usr/include paths except for in the typical
scenario where it is the final parameter in the flag string.
2c03cec2ff ci: Build bitcoin-chainstate (Carl Dong)
095aa6ca37 build: Add example bitcoin-chainstate executable (Carl Dong)
Pull request description:
Part of: #24303
This PR introduces an example/demo `bitcoin-chainstate` executable using said library which can print out information about a datadir and take in new blocks on stdin.
Please read the commit messages for more details.
-----
#### You may ask: WTF?! Why is `index/*.cpp`, etc. being linked in?
This PR is meant only to capture the state of dependencies in our consensus engine as of right now. There are many things to decouple from consensus, which will be done in subsequent PRs. Listing the files out right now in `bitcoin_chainstate_SOURCES` is purely to give us a clear picture of the task at hand, it is **not** to say that these dependencies _belongs_ there in any way.
### TODO
1. Clean up `bitcoin-chainstate.cpp`
It is quite ugly, with a lot of comments I've left for myself, I should clean it up to the best of my abilities (the ugliness of our init/shutdown might be the upper bound on cleanliness here...)
ACKs for top commit:
ajtowns:
ACK 2c03cec2ff
ryanofsky:
Code review ACK 2c03cec2ff. Just rebase, comments, formatting change since last review
MarcoFalke:
re-ACK 2c03cec2ff 🏔
Tree-SHA512: 86e7fb5718caa577df8abc8288c754f4a590650d974df9d2f6476c87ed25c70f923c4db651c6963f33498fc7a3a31f6692b9a75cbc996bf4888c5dac2f34a13b
On the master branch, bump to 23.99 (pre-24.0).
Tree-SHA512: 1e3b0cee8a2b5080170b59a4c445a3c1b69b99152e8eec7eba7080ab447cc6f9c6bd8f69df2b18ee9416de44a6ed88009a200ad26e89275f6230339330d12314
774323e378 ci: Force `--enable-external-signer` to prevent future regressions (Hennadii Stepanov)
69978858a4 build: Fix Boost.Process check for Boost 1.73 and older (Hennadii Stepanov)
2199ef79cb build: Fix a non-portable use of `test` (Hennadii Stepanov)
d436c488d4 build, refactor: Replace tabs with spaces (Hennadii Stepanov)
Pull request description:
On master (5f44c5c428) Boost.Process check false fails without the `-lpthread` flag.
```
$ grep -C 2 pthread_detach config.log
/usr/bin/ld: /tmp/cczCQfQv.o: in function `boost::asio::detail::posix_global_impl<boost::asio::system_context>::~posix_global_impl()':
conftest.cpp:(.text._ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED2Ev[_ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED5Ev]+0xa3): undefined reference to `pthread_join'
/usr/bin/ld: conftest.cpp:(.text._ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED2Ev[_ZN5boost4asio6detail17posix_global_implINS0_14system_contextEED5Ev]+0xc4): undefined reference to `pthread_detach'
collect2: error: ld returned 1 exit status
configure:26674: $? = 1
```
Not required for Boost 1.74+.
ACKs for top commit:
laanwj:
Code review ACK 774323e378, is a bugfix/workaround, seems fine to merge last minute for 23.0.
Tree-SHA512: 2a9d4b67fd8910e107af972d8c223286b7c933bc310616f86c8b6d8c903438916980fc76bd7e37f2698f6ce5361dc706cbf2933d1ac2c081bcabe1b83ca7d6b6
For testing the USDT tracepoint API in the functional tests we
require:
- that we are on a Linux system*
- that Bitcoin Core is compiled with tracepoints
- that bcc and the the Python bcc module [0] is installed
- that we run the tests with the required permissions**
otherwise we skip the tests.
*: We currently only support tracepoints on Linux. Tracepoints are
not compiled on other platforms.
**: Currently, we check for root permissions via getuid == 0. It's
unclear if it's even possible to run the tests a non-root user
with e.g. CAP_BPF, CAP_PERFMON, and access to /sys/kernel/debug/
tracing/. Anyone running these tests as root should carefully
review them first and then run them in a disposable VM.
[0]: https://github.com/iovisor/bcc/blob/master/INSTALL.md
faef344f84 Print enable_fuzz_binary in configure (MarcoFalke)
Pull request description:
A *disabled* `enable_fuzz` on current master does *not* mean the the fuzz binary is not compiled. This is confusion, so fix it.
* `enable_fuzz` toggles compilation flags for fuzzing and disables all other target. There is no need to print this in the configure result, because the compilation flags are already printed. Also, all other targets are already printed as `no`.
* `enable_fuzz_binary` does what it says it does and is currently not printed. So print it.
ACKs for top commit:
hebasto:
ACK faef344f84, tested on Linux Mint 20.2 (x86_64):
Tree-SHA512: 9b02b05c4b9c5fc92cf3487497392690303c36eace5e217f18b4349f059b5a23a7c0e0d030fb6fa7bbad83e927576a5e81c00099164f9ed8e185c0969dc17689
abc057c603 build: Add Boost.Process usage check (Hennadii Stepanov)
Pull request description:
This PR adds a check that Boost.Process can be used without linking any libraries (header-only).
Disable the functionality if that is not the case.
Fixesbitcoin/bitcoin#24314.
ACKs for top commit:
fanquake:
ACK abc057c603
Tree-SHA512: ed2a32b1f751ec6f88cc7220766edd4cdab93c1d0515c892aa3094ee8d5b13ef569830d6e7a7a00c0197b117585dc526d00d943cc99a1f8c8a66ac4e20fe2061
The bitcoin-chainstate executable serves to surface the dependencies
required by a program wishing to use Bitcoin Core's consensus engine as
it is right now.
More broadly, the _SOURCES list serves as a guiding "North Star" for the
libbitcoinkernel project: as we decouple more and more modules of the
codebase from our consensus engine, this _SOURCES list will grow shorter
and shorter. One day, only what is critical to our consensus engine will
remain. Right now, it's "the minimal list of files to link in to even
use our consensus engine".
[META] In a future commit the libbitcoinkernel library will be extracted
from bitcoin-chainstate, and the libbitcoinkernel library's
_SOURCES list will be the list that we aim to shrink.
0c49e52b22 build: remove unneeded getentropy detection (HAVE_GETENTROPY) (Sebastian Falbesoner)
5cd15ffdce random: use arc4random on OpenBSD (Sebastian Falbesoner)
Pull request description:
Inspired by a discussion on obtaining randomness on various OSes in a secp256k1 PR (https://github.com/bitcoin-core/secp256k1/pull/748#discussion_r524605472, see also https://bitcoincore.reviews/libsecp256k1-748), I think it makes sense to follow best practices and use `arc4random_buf` rather than `getentropy` on OpenBSD in our random module.
The [getentropy(2) man page](https://man.openbsd.org/getentropy.2) states:
```
getentropy() is not intended for regular code; please use the
arc4random(3) family of functions instead.
```
The [arc4random(3) man page](https://man.openbsd.org/arc4random.3) states:
```
Use of these functions is encouraged for almost all random number
consumption because the other interfaces are deficient in either quality,
portability, standardization, or availability.
```
On the linked PR discussion worries about using RC4 internally has been expressed (see https://security.stackexchange.com/questions/85601/is-arc4random-secure-enough/172905#172905), but this would only affect users of OpenBSD <5.5, using a version that was released more than 8 years ago.
ACKs for top commit:
laanwj:
Tested ACK 0c49e52b22
Tree-SHA512: b5ed3d0718962c5a3839db9a28f93d08a0ac93094cc664f83bc4cf1cfad25049e6240b7b81fe06b71e6a3a0ca24a2c337eab088abec5470ad014e10c04fdb216
c62d763fc3 Necessary improvements to make configure work without libevent installed (Perlover)
091ccc38c2 The evhttp_connection_get_peer function from libevent changes the type of the second parameter. Fixing the problem. (Perlover)
Pull request description:
The second parameter of evhttp_connection_get_peer in libevent already has type as `const char **`
The compilation of bitcoind with the fresh libevent occurs errors
Details: https://github.com/bitcoin/bitcoin/issues/23606
ACKs for top commit:
laanwj:
Code review ACK c62d763fc3
luke-jr:
tACK c62d763fc3
Tree-SHA512: d1c8062d90bd0d55c582dae2c3a7e5ee1b6c7ca872bf4aa7fe6f45a52ac4a8f59464215759d961f8efde0efbeeade31b08daf9387d7d50d7622baa1c06992d83
6200fbf54f build: rename --enable-ebpf to --enable-usdt (0xb10c)
e158a2a7aa build: add systemtap's sys/sdt.h as depends (0xb10c)
Pull request description:
There has been light conceptual agreement on including the Userspace, Statically Defined Tracing tracepoints in Bitcoin Core release builds. This, for example, enables user to hook into production deployments, if they need to. Binaries don't have to be switched out. This is possible because we don't do [expensive computations](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#no-expensive-computations-for-tracepoints) only needed for the tracepoints. The tracepoints are NOPs when not used.
Systemtap's `sys/sdt.h` header is required to build Bitcoin Core with USDT support. The header file defines the `DTRACE_PROBE` macros used in [`src/util/trace.h`](https://github.com/bitcoin/bitcoin/blob/master/src/util/trace.h). This PR adds Systemtap 4.5 (May 2021) as dependency. GUIX builds for Linux hosts now include the tracepoints.
Closes https://github.com/bitcoin/bitcoin/issues/23297.
ACKs for top commit:
fanquake:
ACK 6200fbf54f - tested enabling / disabling and with/without SDT from depends. We can follow up with #23819, #23907 and #23296, and if any serious issues arise before feature freeze, it is easy for us to flip depends such that USDT becomes opt-in, rather than opt-out, and thus, releases would be tracepoint free.
Tree-SHA512: 0263f44892bf8450e8a593e4de7a498243687f8d81269e1c3283fa8354922c7cf93fddef4b92cf5192d33798424aa5812e03e68ef8de31af078a32dd34021382
eBPF is a Linux kernel technology used to "extend the capabilities
of the kernel without requiring to change kernel source code or
load kernel modules". While Userspace, Statically Defined Tracing
(USDT) uses eBPF under the hood, --enable-usdt better resembles that
support for USDT is enabled, and tracepoints will be included in the
binary.
e09773d20a build: use a static .tiff for macOS .dmg over generating (fanquake)
Pull request description:
For demonstration, after [discussion in #23778](https://github.com/bitcoin/bitcoin/pull/23778#issuecomment-1003005503), and the question as to why we can't just have a `background.tiff` that we copy into the macOS DMG, and do away with the somewhat convoluted image generation steps.
From my understanding, the only reason we have this image generation as part of our build system is so that forks of Core can adapt the imagery for their own branding via `PACKAGE_NAME`. It don't think it provides much value to us, and could just have a static .tiff that we copy into the dmg (replacing the .svg that currently lives in macdeploy/).
Doing this would eliminate the following build dependencies:
For native macOS:
* `sed` (usage in Makefile.am)
* `librsvg` (rsvg-convert)
* `tiffutil`
Linux macOS cross-compile:
* `sed` (usage in Makefille.am)
* `librsvg`
* `tiffcp`
* `convert` (imagemagick)
* `font-tuffy`
Guix Build:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
c98d67796863f4b1bab0ad600d46bd74e744d94072cbd4bc856a6aeaba3bb329 guix-build-e09773d20a92/output/dist-archive/bitcoin-e09773d20a92.tar.gz
3336f90bab312798cb7665e2b4ae24d1a270fb240647d5fed8dbfcd83e3ed37e guix-build-e09773d20a92/output/x86_64-apple-darwin/SHA256SUMS.part
8fd680c7ee158c64bad212385df7b0b302c6c2143d4e672b4b0eb5da41f9256d guix-build-e09773d20a92/output/x86_64-apple-darwin/bitcoin-e09773d20a92-osx-unsigned.dmg
34f54177c2f0700e8cfaf5d85d91e404807cd9d411e22006cdff82653e5f4af2 guix-build-e09773d20a92/output/x86_64-apple-darwin/bitcoin-e09773d20a92-osx-unsigned.tar.gz
da6b8f54ef755d40330c8eac4f5bd0329637e827be9ee61318600d5d0bdcc3dc guix-build-e09773d20a92/output/x86_64-apple-darwin/bitcoin-e09773d20a92-osx64.tar.gz
```
ACKs for top commit:
hebasto:
ACK e09773d20a
jarolrod:
ACK e09773d20a
Zero-1729:
ACK e09773d20a
Tree-SHA512: 0ad06699a5451daa8cfaaa46759eb7bd85254a72e23f857f70d433a2ffb1a4bf6dd464d9c4ac9f8c20aab045f4e2b61c6dcdcbcceef96ce515b1a0c501665b1f
1bf3809dd1 build: use host_os instead of TARGET_OS in configure output (fanquake)
Pull request description:
`TARGET_OS` was convenient, as a readable host name for most of our
targeted platforms, however unless we add more code to configure to
detect more hosts, it's easier just use `host_os` (it's also more
informative).
i.e FreeBSD master
```bash
target os =
build os = freebsd13.0
```
this PR:
```bash
target os = freebsd13.0
build os = freebsd13.0
```
ACKs for top commit:
hebasto:
ACK 1bf3809dd1
Tree-SHA512: 606d92f60ce3f2f6ab1f54e29b5c179048c62ba51336b272c081b1e009128dd83705b181cfe30991c7a51d9c63e8ba2076bfed9e6112e7d1a74a7f947c5754f5
TARGET_OS was conveninent, as a readable host name for most of our
targetted platforms, however unless we add more code to configure to
detect more hosts, it's easier just use host_os (it's also more
informative).
i.e FreeBSD master
```bash
target os =
build os = freebsd13.0
```
this PR:
```bash
target os = freebsd13.0
build os = freebsd13.0
```
Variables that are declared with AC_ARG_VAR macro are substituted via
AC_SUBST macro.
PKG_CHECK_MODULES macro already has AC_ARG_VAR(${PACKAGE}_CFLAGS) and
AC_ARG_VAR(${PACKAGE}_LIBS).
314195c8be Remove unnecessary cast in CKey::SignSchnorr (Pieter Wuille)
a1f76cdb22 Remove --disable-openssl-tests for libsecp256k1 configure (Pieter Wuille)
86dbc4d075 Squashed 'src/secp256k1/' changes from be8d9c262f..0559fc6e41 (Pieter Wuille)
Pull request description:
The motivation for this bump is getting rid of a cast in `CKey::SignSchnorr`; the `aux_rand` argument isn't modified by the `secp256k1_schnorrsig_sign` function, but was marked as non-`const` anyway. This is fixed now (bitcoin-core/secp256k1#966), and the cast is removed in this PR.
There are a few other relevant changes:
* (bitcoin-core/secp256k1#956): replaces a runtime-computed table with a precomputed one; this adds arouns 1 MiB to the binary size, but is a step towards significantly simplifying the API. If 1 MiB is too much, it can be reduced by 2 or 4 (or more) for a slight verification performance reduction.
* (bitcoin-core/secp256k1#983): removes (test/bench only) OpenSSL support entirely, removing the need to pass `--disable-openssl-tests` (see #23314).
* (bitcoin-core/secp256k1#810): mild performance increase for 64-bit non-x86 platforms.
* (bitcoin-core/secp256k1#1002): Make aux_rnd32==NULL behave identical to 0x0000..00 (which impacts BIP341/BIP342 signing in Bitcoin Core, making it more strictly BIP340 compliant, though not in a manner that affects security).
ACKs for top commit:
fanquake:
ACK 314195c8be - this includes a nice simplification to the lilbsecp build system (and thus our build system), and fixes issues like #22854. Did a Guix build on x86 (above), as well as a build on arm64 (except for the arm64 host):
Tree-SHA512: 0e048390fc148fbbdf5b98d9cce8c71067564e7d69d97b68347808a9bc45a04f4fc653c392c880d79d5d8b9cf282195520955581ac4f1595f6a948080cf5949d
66a20a54a2 build, qt: Drop support for `i686-linux-android` host (Hennadii Stepanov)
Pull request description:
There are no reasons to keep support for `i686-linux-android` host, which is actually broken in master (50c502f54a), and this fact has been unnoticed for months :)
https://github.com/bitcoin/bitcoin/pull/23675#issuecomment-986206434:
> I'm surprised `i686-linux-android` ABI is still supported. I would love to drop it...
https://github.com/bitcoin/bitcoin/pull/23675#issuecomment-991340132
> What is `i686-linux-android`? 32-bit x86 android? is that really a thing?
ACKs for top commit:
prusnak:
utACK 66a20a54a2
Tree-SHA512: 211f794de2fc569f0ade2a4da805b8bfd4ce2ab0930c5d427acc4f5d015fcdc4911f02fc64f6401197f7641aed79944a9594be80c817547be3269cdd721cf79b
We already use this in the blockfilter code, so not sure we need to maintain two
different ways of testing for the same functionality. Consolidate on testing
for __SIZEOF_INT128__, which we already use, is supported by the compilers we
care about, and is also used by libsecp256k1.
Very old shells suffered from bugs which meant that prefixing variables
with an "x" to ensure that the lefthand side of a comparison always
started with an alphanumeric character was needed. Modern shells don't
suffer from this issue (i.e Bash was fixed in 1996).
In any case, we've already got unprefixed checks used in our codebase,
i.e https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L292,
and have dependencies (in depends) that also use unprefixed comparisons.
I think it's time that we can consolidate on not using the x-prefix
workaround. At best it's mostly just confusing.
More info:
https://github.com/koalaman/shellcheck/wiki/SC2268https://www.vidarholen.net/contents/blog/?p=1035
68e5aafde3 build: add `--enable-lto` configuration option (fanquake)
Pull request description:
It's been 5 years since using LTO was first suggested for use when building Bitcoin Core, and it's time to revisit it again. Compilers, and their LTO implementations, have matured, and Bitcoin Core has come a long way in terms of pruning dependencies which may have proved troublesome (i.e Boost previously had issues when using LTO). We'll have even less Boost code after moving to `std::filesystem` (#20744).
Experimenting with LTO came up on IRC last night:
> sipa: jamesob: i'm interested in knowing whether "-flto" and/or "-fdata-sections -ffunction-sections -Wl,--gc-sections" are possible/beneficial with our current compiler suite; what would be a good way to have your test infrastructure benchmark things?
So this PR just adds the bare minimum to make it easier to configure, compile and perform some bench-marking using `-flto`. This PR doesn't do anything depends wise, however if we decide this is what we want to do, I'll expand the changes here.
I had previously had a PR open (#18605) to perform link time garbage collection (`-ffunction-sections -fdata-sections` & `-Wl,--gc-sections`), however moving straight to using LTO would be preferable.
Note that our minimum required set of compilers, GCC 8.1 and Clang 7, all support the `-flto` option.
Related #18579.
Previous discussion: #10616, #14277.
Previous related PRs: #10800 (`-flto`), #16791 (ThinLTO).
Guix build:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
1f3a7c5be4169aaa444b481d3e65a7bb72da9007fee6e6c416ded2e70f97374b guix-build-68e5aafde3e8/output/aarch64-linux-gnu/SHA256SUMS.part
fa8f4cf223d9aaf0b2c1ef55ce61256a19cd1ad7f42b99d0b98c9a52fe6ad8ba guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu-debug.tar.gz
9a9967078cd1849b4e85db619e1f55d305c6d44e9e013067c0e8d62c1ba54087 guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu.tar.gz
18c71f30722102baaf3dfda67f7c7aac38723510b142e8df8ee7063c5d499368 guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/SHA256SUMS.part
0854cc0d17c045a118df2a24e4cf36d727e7e7e2dea37c2492ee21b71cb79b4b guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf-debug.tar.gz
215256897dde4e8412ed60473376c694a80c5479fb08039107fb62435f2816ef guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf.tar.gz
5fad0d9d12bc514ec46ed5d66fd29b7da1376a4a69c3b692936f1ab2356e2f85 guix-build-68e5aafde3e8/output/dist-archive/bitcoin-68e5aafde3e8.tar.gz
4f32989d4ab1946048ca7caee9a983fa875be262282562f5a3e040f4bf92158e guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/SHA256SUMS.part
ae45df309ae8ada52891efac0a369a69fed4ab93847a7bc4150a62230df4c8d7 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu-debug.tar.gz
0ced227de15cb578567131271e2effe80681b4d7a436c92bf1caec735a576fa4 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu.tar.gz
26fc5d2ccc1bc17ee0a146cacada6f4909d90c136ae640c8337332adce414ee0 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/SHA256SUMS.part
9956b544d90a62a8ba9fc9dc6b6b7f0efe193357332ec19e88053a89d4aab37e guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu-debug.tar.gz
be8e39ceea1d36086ce5fa93bfb138c68d3bdf0dd6950b192dfa27a65cce3836 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu.tar.gz
a7755edc394972885c4c77a7798007e5ba4126b177c4ff6224275c4fb8f3b1c4 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/SHA256SUMS.part
b6d252993d8aae7582ad6385fe53c61c54c284c68ece6cb2b2d1ac9554e06139 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu-debug.tar.gz
bb4860f3bbd815f800333124ff901d880741792ab47097f49bda3a6931144da0 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu.tar.gz
3dd17deed5c5935fb28b62dfc7afca5caab0d67862cdcbf3337edae73e1d0c4c guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/SHA256SUMS.part
fa2d68c54fda0816188c81ce2201a77340b82645da2ffe412526f92c297a82df guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.dmg
f6e5accdcd201f522b6426e4d8cc9b3643d4d43a57d268fa0e79ea9a34cfac01 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.tar.gz
4e5a127df957d1c73b65925d685f6620e7bc5667efcb6dcd98be76effc22fc12 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx64.tar.gz
56ccd216a69acafacbdc6bae0bdcc1faa50b6a51be1aebfa7068206c88b3241a guix-build-68e5aafde3e8/output/x86_64-linux-gnu/SHA256SUMS.part
77b93dd5fad322636853e5b0244ffafd97cc97f3b4b4ee755d5f830b75d77d13 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu-debug.tar.gz
1feda932fc127b900316a232432b91e46e57ee12a81e12a7d888fdc3296219c1 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu.tar.gz
aa7c53ab4164b3736049065c3c24391fc5bd7f26b4bda4aa877c378f0636a125 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/SHA256SUMS.part
5e76148e67aef7e91e70074bfadc08e94373449ac3b966f4343b04d230c778fd guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win-unsigned.tar.gz
34123e3d818beeb70113caeda66945bc7cb9d9e987515d5b149bd17b4b38da90 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-debug.zip
2bba7f40a2b23c6ea3d47c4f564ab54201bf27f7f57103a98cc9bceea4e70c4d guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-setup-unsigned.exe
0e7e124144af4a92a4344cf70a3b7c06fbd2b8782aee7ede7263893afa3a5ef0 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64.zip
```
ACKs for top commit:
laanwj:
Code review ACK 68e5aafde3
Tree-SHA512: 5c25249cc178b9d54159e268390c974b739df9458d773e23c14b14d808f87f7afe314058b3c068601a9132042321973b0c9b6f81becb925665eca2738ae9a613
cf7292597e configure.ac: remove Bashism (Matt Whitlock)
Pull request description:
Configure scripts are supposed to adhere to the POSIX shell language. The POSIX `test` builtin does not implement an `==` operator. Bash does, but not all systems have Bash installed as `/bin/sh`. In particular, many systems use the lighter-weight Dash as the default POSIX shell. Dash emits the following error when running `configure`:
```
./configure: 39065: test: xno: unexpected operator
```
This PR removes the Bashism and restores correct operation with POSIX-compliant shells like Dash.
ACKs for top commit:
katesalazar:
ACK cf7292597e.
laanwj:
Code review ACK cf7292597e
Tree-SHA512: 578c873fba52e0472baed9e024bddcf58a0e088600bd5854f3011f1f8d135773ad923bb16baefc960d17ecedee9cc980b36aaa70fb32eb9bc7de93f7fe60541d
From what I can see the only platform this drops support for is CentOS
7. CentOS 7 reached the end of it's "full update" support at the end of
2020. It does receive maintenance updates until 2024, however I don't
think supporting glibc 2.17 until 2024 is realistic. Note that anyone
wanting to self-compile and target a glibc 2.17 runtime could build with
--disable-threadlocal.
glibc 2.18 was released in August 2013.
https://sourceware.org/legacy-ml/libc-alpha/2013-08/msg00160.html
29173d6c6c ubsan: add minisketch exceptions (Cory Fields)
54b5e1aeab Add thin Minisketch wrapper to pick best implementation (Pieter Wuille)
ee9dc71c1b Add basic minisketch tests (Pieter Wuille)
0659f12b13 Add minisketch dependency (Gleb Naumenko)
0eb7928ab8 Add MSVC build configuration for libminisketch (Pieter Wuille)
8bc166d5b1 build: add minisketch build file and include it (Cory Fields)
b2904ceb85 build: add configure checks for minisketch (Cory Fields)
b6487dc4ef Squashed 'src/minisketch/' content from commit 89629eb2c7 (fanquake)
Pull request description:
This takes over #21859, which has [recently switched](https://github.com/bitcoin/bitcoin/pull/21859#issuecomment-921899200) to my integration branch. A few more build issues came up (and have been fixed) since, and after discussing with sipa it was decided I would open a PR to shepherd any final changes through.
> This adds a `src/minisketch` subtree, taken from the master branch of https://github.com/sipa/minisketch, to prepare for Erlay implementation (see #21515). It gets configured for just supporting 32-bit fields (the only ones we're interested in in the context of Erlay), and some code on top is added:
> * A very basic unit test (just to make sure compilation & running works; actual correctness checking is done through minisketch's own tests).
> * A wrapper in `minisketchwrapper.{cpp,h}` that runs a benchmark to determine which field implementation to use.
Only changes since my last update to the branch in the previous PR have been rebasing on master and fixing an issue with a header in an introduced file.
ACKs for top commit:
naumenkogs:
ACK 29173d6c6c
Tree-SHA512: 1217d3228db1dd0de12c2919314e1c3626c18a416cf6291fec99d37e34fb6eec8e28d9e9fb935f8590273b8836cbadac313a15f05b4fd9f9d3024c8ce2c80d02
AC_DEFINE'd values won't be passed down to minisketch because it does not
use bitcoin-config.h. Thus we need a way to know if we should manually add
defines for minisketch files.
17ae2601c7 build: remove build stubs for external leveldb (Cory Fields)
Pull request description:
Presumably these stubs indicate to packagers that external leveldb is meant to be supported in some way. It is not. Remove the stubs to avoid sending any mixed messages.
For context, this was reported on IRC:
> \<Talkless> bitcoind fails to start with undefined symbol: _ZTIN7leveldb6LoggerE in Debian Sid after leveldb upgraded from 1.22 to 1.23: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=996486
ACKs for top commit:
fanquake:
ACK 17ae2601c7
hebasto:
ACK 17ae2601c7. I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 2f1ac2cb30dac64791933a245a2b66ce237bde3955e6f4a6b7ec181248f77a9b1b10597d865d3e2c2b6def696af70de40e905ec274e4ae7cccd1daf461473957
These tests are failing when run against OpenSSL 3, and have been
removed upstream, https://github.com/bitcoin-core/secp256k1/pull/983, so
disabled them for now to avoid `make check` failures.
Note that this will also remove warning output from our build, due to
the use of deprecated OpenSSL API functions. See #23048.
0f95247246 Integrate univalue into our buildsystem (Cory Fields)
9b49ed656f Squashed 'src/univalue/' changes from 98fadc0909..a44caf65fe (fanquake)
Pull request description:
This PR more tightly integrates building Univalue into our build system. This follows the same approach we use for [LevelDB](https://github.com/bitcoin-core/leveldb/), ([`Makefile.leveldb.include`](https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.leveldb.include)), and [CRC32C](https://github.com/bitcoin-core/crc32c) ([`Makefile.crc32c.include`](https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.crc32c.include)), and will be the same approach we use for [minisketch](https://github.com/sipa/minisketch); see #23114.
This approach yields a number of benefits, including:
* Faster configuration due to one less subconfigure being run during `./configure` i.e 22s with this PR vs 26s
* Faster autoconf i.e 13s with this PR vs 17s
* Improved caching
* No more issues with compiler flags i.e https://github.com/bitcoin/bitcoin/pull/12467
* More direct control means we can build exactly the objects we want
There might be one argument against making this change, which is that builders should have the option to use "proper shared/system libraries". However, I think that falls down for a few reasons. The first being that we already don't support building with a number of system libraries (secp256k1, leveldb, crc32c); some for good reason. Univalue is really the odd one out at the moment.
Note that the only fork of Core I'm aware of, that actively patches in support for using system libs, also explicitly marks them as ["DANGEROUS"](a886811721/configure.ac (L1430)) and ["NOT SUPPORTED"](a886811721/configure.ac (L1312)). So it would seem they exist more to satisfy a distro requirement, as opposed to something that anyone should, or would actually use in practice.
PRs like #22412 highlight the "issue" with us operating with our own Univalue fork, where we actively fix bugs, and make improvements, when upstream (https://github.com/jgarzik/univalue) may not be taking those improvements, and by all accounts, is not currently actively maintained. Bitcoin Core should not be hamstrung into not being able to fix bugs in a library, and/or have to litter our source with "workarounds", i.e #22412, for bugs we've already fixed, based on the fact that an upstream project is not actively being maintained. Allowing builders to use system libs is really only exacerbating this problem, with little benefit to our project. Bitcoin Core is not quite like your average piece of distro packaged software.
There is the potential for us to give the same treatment to libsecp256k1, however it seems doing that is currently less straightforward.
ACKs for top commit:
dongcarl:
ACK 0f95247246 less my comment above, always nice to have an include-able `sources.mk` which makes integration easier.
theuni:
ACK 0f95247246. Thanks fanquake for keeping this going.
Tree-SHA512: a7f2e41ee7cba06ae72388638e86b264eca1b9a8b81c15d1d7b45df960c88c3b91578b4ade020f8cc61d75cf8d16914575f9a78fa4cef9c12be63504ed804b99
Presumably these stubs indicate to packagers that external leveldb is meant to
be supported in some way. It is not. Remove the stubs to avoid sending any
mixed messages.
aa69fd6caf build: Drop -Wno-unused-local-typedef (Hennadii Stepanov)
672e8c5d07 build: remove -Wunused-variable (fanquake)
5239af0574 build: remove -Wswitch (fanquake)
0375906e0a build: use loop-analysis over range-loop-analysis (fanquake)
12712fa2c4 build: remove -Wsign-compare (fanquake)
Pull request description:
This remove the addition of flags that are already part of other options, such as `-Wall` or `-Wextra`; see each commit message for details. All of the flags being removed here already exist as part of `-Wall` as of GCC 8, or, for Clang, all exist in `-Wmost` (included in `-Wall)`, or as part of `-Wextra` as of Clang 7. Both of which are our minimum required compilers.
Also cherry-picks one change from #21458.
To give an example of how GCCs `-Wall` has changed over the last few releases:
### 11.x to trunk (12.x)
Added:
```bash
-Wzero-length-bounds
-Wmismatched-dealloc
-Wmismatched-new-delete (only for C/C++)
```
### 10.x to 11.x
Added:
```bash
-Warray-parameter=2 (C and Objective-C only)
-Wrange-loop-construct (only for C++)
-Wsizeof-array-div
-Wvla-parameter (C and Objective-C only)
```
Removed:
```bash
-Wenum-conversion in C/ObjC;
```
### 9.x to 10.x
Added:
```bash
-Wenum-conversion in C/ObjC;
-Wformat-overflow
-Wformat-truncation
-Wzero-length-bounds
```
### 8.x to 9.x
Added:
```bash
-Wpessimizing-move
```
Removed:
```bash
-Wstringop-truncation
```
### 7.x to 8.x
Added:
```bash
-Wcatch-value (C++ and Objective-C++ only)
-Wmissing-attributes
-Wmultistatement-macros
-Wrestrict
-Wsizeof-pointer-div
-Wstringop-truncation
```
[Clang Warning Options](https://clang.llvm.org/docs/DiagnosticsReference.html)
[GCC Warning Options](https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html)
ACKs for top commit:
meshcollider:
utACK aa69fd6caf
Tree-SHA512: 34dde6bd773c864202c151eaa35f902d03fb531c27fe5e1ef659225da03acade2efe5df56df3efb4df5bbded3d395348ce03c25b837fce83be53af3352f0f2bc
ce69e18947 scripts: remove pixie.py (fanquake)
00b85d0b13 scripts: only parse the binary once in security-check.py (fanquake)
cad40a5b16 scripts: use LIEF for ELF checks in security-check.py (fanquake)
8242ae230e scripts: only parse the binary once in symbol-check.py (fanquake)
309eac9019 scripts: use LIEF for ELF checks in symbol-check.py (fanquake)
610a8a8e39 test-*-check: Pass in *FLAGS and compile with them (Carl Dong)
Pull request description:
This finishes the transition to using LIEF for the ELF symbol and security checks.
Note that there's currently a work around used for identifying RISCV binaries (just checking the interpreter). I've sent a PR upstream, https://github.com/lief-project/LIEF/pull/562, and we should be able to drop that when using LIEF 0.12.0 and onwards.
ACKs for top commit:
dongcarl:
Code Review ACK ce69e18947
laanwj:
Code review ACK ce69e18947
Tree-SHA512: 911ba693cd9777ad1fc1f66dff6c4d3630a907351215380cbde5b14a4bbf5cf7eebf52eafa7e86b27deabd2d93d1b403f34aabd356b5ceaab3cc6e9941a01dd4
38fd709fa5 build: make --enable-werror just -Werror (fanquake)
Pull request description:
No longer special case a set of warnings, to make up our own -Werror,
just use -Werror outright. This shouldn't really have any effect on
existing builders, who were already using `--enable-werror`, and is more
inline with what they would expect `--enable-werror` to be, which is
erroring on any/all warnings.
We keep `-Wno-error=return-type` because we know that is broken when using
mingw-w64. It should only be applied when cross-compiling for Windows.
Similar to the change in #20544, but with (hopefully) less work-arounds,
and other bundled changes. A step towards some configure "cleanups".
ACKs for top commit:
hebasto:
ACK 38fd709fa5 (also see https://github.com/bitcoin/bitcoin/pull/23149#issuecomment-940420776), tested:
MarcoFalke:
Concept ACK 38fd709fa5
Tree-SHA512: 37f1857d9408442cab63e40f9280427b73e09cdf03146b19c1339d7e44abd78e93df7f270ca1da0e83b79343cd3ea915f7b9e4e347488b5bc5ceaaa7540e5926
This addresses issues like the one in #12467, where some of our compiler flags
end up being dropped during the subconfigure of Univalue. Specifically, we're
still using the compiler-default c++ version rather than forcing c++17.
We can drop the need subconfigure completely in favor of a tighter build
integration, where the sources are listed separately from the build recipes,
so that they may be included directly by upstream projects. This is
similar to the way leveldb build integration works in Core.
Core benefits of this approach include:
- Better caching (for ex. ccache and autoconf)
- No need for a slow subconfigure
- Faster autoconf
- No more missing compile flags
- Compile only the objects needed
There are no benefits to Univalue itself that I can think of. These changes
should be a no-op there, and to downstreams as well until they take advantage
of the new sources.mk.
This also removes the option to use an external univalue to avoid similar ABI
issues with mystery binaries.
Co-authored-by: fanquake <fanquake@gmail.com>
0bc666b053 doc: add info for debugging with relative paths (S3RK)
a8b515c317 configure: keep relative paths in debug info (S3RK)
Pull request description:
This is a follow-up for #20353 that fixes#21885
It also adds a small section to assist debugging without absolute paths in debug info.
ACKs for top commit:
kallewoof:
Tested ACK 0bc666b053
Zero-1729:
Light crACK 0bc666b053
Tree-SHA512: d4b75183c3d3a0f59fe786841fb230581de87f6fe04cf7224e4b89c520d45513ba729d4ad8c0e62dd1dbaaa7a25741f04d036bc047f92842e76c9cc31ea47fb2
4747da3a5b Add syscall sandboxing (seccomp-bpf) (practicalswift)
Pull request description:
Add experimental syscall sandboxing using seccomp-bpf (Linux secure computing mode).
Enable filtering of system calls using seccomp-bpf: allow only explicitly allowlisted (expected) syscalls to be called.
The syscall sandboxing implemented in this PR is an experimental feature currently available only under Linux x86-64.
To enable the experimental syscall sandbox the `-sandbox=<mode>` option must be passed to `bitcoind`:
```
-sandbox=<mode>
Use the experimental syscall sandbox in the specified mode
(-sandbox=log-and-abort or -sandbox=abort). Allow only expected
syscalls to be used by bitcoind. Note that this is an
experimental new feature that may cause bitcoind to exit or crash
unexpectedly: use with caution. In the "log-and-abort" mode the
invocation of an unexpected syscall results in a debug handler
being invoked which will log the incident and terminate the
program (without executing the unexpected syscall). In the
"abort" mode the invocation of an unexpected syscall results in
the entire process being killed immediately by the kernel without
executing the unexpected syscall.
```
The allowed syscalls are defined on a per thread basis.
I've used this feature since summer 2020 and I find it to be a helpful testing/debugging addition which makes it much easier to reason about the actual capabilities required of each type of thread in Bitcoin Core.
---
Quick start guide:
```
$ ./configure
$ src/bitcoind -regtest -debug=util -sandbox=log-and-abort
…
2021-06-09T12:34:56Z Experimental syscall sandbox enabled (-sandbox=log-and-abort): bitcoind will terminate if an unexpected (not allowlisted) syscall is invoked.
…
2021-06-09T12:34:56Z Syscall filter installed for thread "addcon"
2021-06-09T12:34:56Z Syscall filter installed for thread "dnsseed"
2021-06-09T12:34:56Z Syscall filter installed for thread "net"
2021-06-09T12:34:56Z Syscall filter installed for thread "msghand"
2021-06-09T12:34:56Z Syscall filter installed for thread "opencon"
2021-06-09T12:34:56Z Syscall filter installed for thread "init"
…
# A simulated execve call to show the sandbox in action:
2021-06-09T12:34:56Z ERROR: The syscall "execve" (syscall number 59) is not allowed by the syscall sandbox in thread "msghand". Please report.
…
Aborted (core dumped)
$
```
---
[About seccomp and seccomp-bpf](https://en.wikipedia.org/wiki/Seccomp):
> In computer security, seccomp (short for secure computing mode) is a facility in the Linux kernel. seccomp allows a process to make a one-way transition into a "secure" state where it cannot make any system calls except exit(), sigreturn(), and read() and write() to already-open file descriptors. Should it attempt any other system calls, the kernel will terminate the process with SIGKILL or SIGSYS. In this sense, it does not virtualize the system's resources but isolates the process from them entirely.
>
> […]
>
> seccomp-bpf is an extension to seccomp that allows filtering of system calls using a configurable policy implemented using Berkeley Packet Filter rules. It is used by OpenSSH and vsftpd as well as the Google Chrome/Chromium web browsers on Chrome OS and Linux. (In this regard seccomp-bpf achieves similar functionality, but with more flexibility and higher performance, to the older systrace—which seems to be no longer supported for Linux.)
ACKs for top commit:
laanwj:
Code review and lightly tested ACK 4747da3a5b
Tree-SHA512: e1c28e323eb4409a46157b7cc0fc29a057ba58d1ee2de268962e2ade28ebd4421b5c2536c64a3af6e9bd3f54016600fec88d016adb49864b63edea51ad838e17
No longer special case a set of warnings, to make up our own -Werror,
just use -Werror outright. This shouldn't really have any effect on
existing builders, who were already using --enable-werror, and is more
inline with what they would expect --enable-werror to be, which is
erroring on any/all warnings.
We keep -Wno-error=return-type because we know that is broken when using
mingw-w64. It should only be applied when cross-compiling for Windows.
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
It was [pointed out in #23030](https://github.com/bitcoin/bitcoin/pull/23030#issuecomment-922893367) that we might be able to get rid of our weak linking of [`getauxval()`](https://man7.org/linux/man-pages/man3/getauxval.3.html) (`HAVE_WEAK_GETAUXVAL`) entirely, with only Android being a potential holdout:
> I wonder if it's time to get rid of HAVE_WEAK_GETAUXVAL. I think it's confusing. Either we build against a C library that has this functionality, or not. We don't do this weak linking thing for any other symbols and recently got rid of the other glibc backwards compatibility stuff.
> Unless there is still a current platform that really needs it (Android?), I'd prefer to remove it from the build system, it has caused enough issues.
After looking at Android further, it would seem that given we are moving to using `std::filesystem`, which [requires NDK version 22 and later](https://github.com/android/ndk/wiki/Changelog-r22), and `getauxval` has been available in the since [API version 18](https://developer.android.com/ndk/guides/cpu-features#features_using_libcs_getauxval3), that shouldn't really be an issue. Support for API levels < 19 will be dropped with the NDK 24 release, and according to [one website](https://apilevels.com/), supporting API level 18+ will cover ~99% of devices. Note that in the CI we currently build with NDK version 22 and API level 28.
The other change in this PR is removing the include of headers for ARM intrinsics, from the check for strong `getauxval()` support in configure, as they shouldn't be needed. Including these headers also meant that the check would basically only succeed when building for ARM. This would be an issue if we remove weak linking, as we wouldn't detect `getauxval()` as supported on other platforms. Note that we also use `getauxval()` in our RNG when it's available.
I've checked that with these changes we detect support for strong `getauxval()` on Alpine (muslibc). On Linux, previously we'd be detecting support for weak getauxval(), now we detect strong support. Note that we already require glibc 2.17, and `getauxval()` was introduced in `2.16`.
This is an alternative / supersedes #23030.
`crc32c`'s hardware accelerated code doesn't handle ARM 32-bit at all.
Make the check in `configure.ac` check for this architecture explicitly.
For the release binaries, the current `configure.ac` check happens
to work: it enables it on aarch64 but disables it for armhf. However
some combination of compiler version and settings might ostensibly cause
this check to succeed on armhf (as reported on IRC). So make the 64-bit
platform requirement explicit.
3ec633ef1a build: improve check for ::(w)system (fanquake)
Pull request description:
`AC_DEFINE()` takes `HAVE_STD__SYSTEM || HAVE_WSYSTEM` literally, meaning you
end up with the following in bitcoin-config.h:
```cpp
/* std::system or ::wsystem */
#define HAVE_SYSTEM HAVE_STD__SYSTEM || HAVE_WSYSTEM
```
This works for the preprocessor, because `HAVE_SYSTEM`, is defined, just unusually. Remove this in favor of setting `have_any_system` in either case, given we don't actually use `HAVE_STD__SYSTEM` or `HAVE_WSYSTEM`, and defining `HAVE_SYSTEM` to 1 thereafter.
ACKs for top commit:
laanwj:
Code review ACK 3ec633ef1a
Tree-SHA512: 02c39ba3179136ec1dc28df026b7fa5d732914c85622298ba7ec880f1ae9324208d322a47be451a5c2ff2e165ad1d446bae92e7018db8e517e7ac38fca25a0a3
At this point, or minimum required glibc is implicitly 2.18, due to
thread_local support being enabled by default. However, users can
disable thread_local support to maintain 2.17 ccompat for now, which is
currently done in the Guix build.
`AC_DEFINE()` takes `HAVE_STD__SYSTEM || HAVE_WSYSTEM` literally, meaning you
end up with the following in bitcoin-config.h:
```cpp
/* std::system or ::wsystem */
#define HAVE_SYSTEM HAVE_STD__SYSTEM || HAVE_WSYSTEM
```
This works for the preprocessor, because `HAVE_SYSTEM`, is defined, just unusually.
Remove this in favor of defining `HAVE_SYSTEM` to 1 in either case, given we
don't actually use `HAVE_STD__SYSTEM` or `HAVE_WSYSTEM`. We just use ::system if
we aren't building for Windows.
2445df4eb3 build: Fix macOS Apple Silicon build with miniupnpc and libnatpmp (Hennadii Stepanov)
Pull request description:
On master (7a49fdc581) the `configure` script does not pick up Homebrew's `miniupnpc` and `libnatpmp` packages on macOS Apple Silicon:
```
% ./configure --with-miniupnpc
...
checking for miniupnpc/miniupnpc.h... no
checking for miniupnpc/upnpcommands.h... no
checking for miniupnpc/upnperrors.h... no
...
checking whether to build with support for UPnP... configure: error: "UPnP requested but cannot be built. Use --without-miniupnpc."
```
```
% ./configure --with-natpmp
...
checking for natpmp.h... no
...
checking whether to build with support for NAT-PMP... configure: error: NAT-PMP requested but cannot be built. Use --without-natpmp
```
The preferred Homebrew [prefix for Apple Silicon](https://docs.brew.sh/Installation) is `/opt/homebrew`. Therefore, if we do not use `pkg-config` to detect packages, we should set the `CPPFLAGS` and `LDFLAGS` variables for them explicitly.
ACKs for top commit:
Zero-1729:
re-tACK 2445df4eb3 (re-tested on an M1 Machine running macOS 11.4).
jarolrod:
re-ACK 2445df4eb3
Tree-SHA512: d623d26492f463812bf66ca519847ff4b23d517466b6c51c3caf3642a582d02e5f03ce57915742b29f01bf9bceb731a3978ef9a5fdc82e568bcb62548eda758a
b367745cfe ci: Make Cirrus CI Windows build with --enable-werror (Hennadii Stepanov)
c713bb2b24 Fix Windows build with --enable-werror on Ubuntu Focal (Hennadii Stepanov)
Pull request description:
This PR makes possible to cross-compile Windows build with `--enable-werror --enable-suppress-external-warnings`.
Some problems are fixed, others are silenced.
Also `--enable-werror` is enabled for Cirrus CI Windows build (the last one on Cirrus CI without `--enable-werror`).
ACKs for top commit:
practicalswift:
cr ACK b367745cfe: patch looks correct
laanwj:
Code review ACK b367745cfe
vasild:
ACK b367745cfe
jarolrod:
ACK b367745cfe
Tree-SHA512: 64f5c99b7dad4c0efce80cd45d7074f275bd8411235dc9e0841287bdab64b812c6f8f9d632c35531d0b8210148531f53aaaac77be7699b29d2d6aaae304dbee0
The preferred Homebrew prefix for Apple Silicon is /opt/homebrew.
Therefore, if we do not use pkg-config to detect packages, we should set
the CPPFLAGS and LDFLAGS variables for them explicitly.
e4c8bb62e4 build: Fix undefined reference to __mulodi4 (Hennadii Stepanov)
Pull request description:
When compiling with clang on 32-bit systems the `__mulodi4` symbol is defined in compiler-rt only.
Fixes#21294.
See more:
- https://bugs.llvm.org/show_bug.cgi?id=16404
- https://bugs.llvm.org/show_bug.cgi?id=28629
ACKs for top commit:
MarcoFalke:
tested-only ACK e4c8bb62e4
luke-jr:
utACK e4c8bb62e4
fanquake:
ACK e4c8bb62e4 - it's a bit of an awkward workaround to carry, but at-least it's contained to the fuzzers.
Tree-SHA512: 93edb4ed568027702b1b9aba953ad50889b834ef97fde3cb99d1ce70076d9c00aa13f95c86b12d6f59b24fa90108d93742f920e15119901a2848fb337ab859a1
3c4c8e79ba build: Add -Werror=implicit-fallthrough compile flag (Hennadii Stepanov)
014110c47d Use C++17 [[fallthrough]] attribute, and drop -Wno-implicit-fallthrough (Hennadii Stepanov)
Pull request description:
ACKs for top commit:
fanquake:
ACK 3c4c8e79ba - looks ok to me now. Checked that warnings occur in our code & leveldb by removing a `[[fallthrough]]` or `FALLTHROUGH_INTENDED`.
jarolrod:
ACK 3c4c8e79ba
theStack:
ACK 3c4c8e79ba
Tree-SHA512: 4dce91f0f26b8a3de09bd92bb3d7e1995e078e3a8b3ff861c4fbf6c0b32b2327d063633b07b89c4aa94a1141d7f78d46d9d43ab8df865273e342693ad30645b6
8f7704d032 build: improve detection of eBPF support (fanquake)
Pull request description:
Just checking for the `sys/sdt.h` header isn't enough, as systems like macOS have the header, but it doesn't actually have the `DTRACE_PROBE*` probes, which leads to [compile failures](https://github.com/bitcoin/bitcoin/pull/22006#issuecomment-859559004). The contents of `sys/sdt.h` in the macOS SDK is:
```bash
#ifndef _SYS_SDT_H
#define _SYS_SDT_H
/*
* This is a wrapper header that wraps the mach visible sdt.h header so that
* the header file ends up visible where software expects it to be. We also
* do the C/C++ symbol wrapping here, since Mach headers are technically C
* interfaces.
*
* Note: The process of adding USDT probes to code is slightly different
* than documented in the "Solaris Dynamic Tracing Guide".
* The DTRACE_PROBE*() macros are not supported on Mac OS X -- instead see
* "BUILDING CODE CONTAINING USDT PROBES" in the dtrace(1) manpage
*
*/
#include <sys/cdefs.h>
__BEGIN_DECLS
#include <mach/sdt.h>
__END_DECLS
#endif /* _SYS_SDT_H */
```
The `BUILDING CODE CONTAINING USDT PROBES` section from the dtrace manpage is available [here](https://gist.github.com/fanquake/e56c9866d53b326646d04ab43a8df9e2), and outlines the more involved process of using USDT probes on macOS.
ACKs for top commit:
jb55:
utACK 8f7704d032
practicalswift:
cr ACK 8f7704d032
hebasto:
ACK 8f7704d032, tested on macOS Big Sur 11.4 (20F71) and on Linux Mint 20.1 (x86_64) with depends.
Tree-SHA512: 5f1351d0ac2e655fccb22a5454f415906404fdaa336fd89b54ef49ca50a442c44ab92d063cba3f161cb8ea0679c92ae3cd6cfbbcb19728cac21116247a017df5
2f5bdcbc31 gui: misc external signer fixes and translation hints (Sjors Provoost)
d672404466 refactor: make ExternalSigner NetworkArg() and m_chain private (Sjors Provoost)
4455145e26 refactor: reduce #ifdef ENABLE_EXTERNAL_SIGNER usage (Sjors Provoost)
5be90c907e build: enable external signer by default (Sjors Provoost)
7d9453041b refactor: clean up external_signer.h includes (Sjors Provoost)
fc0eca31b3 fuzz: fix fuzz binary linking order (Sjors Provoost)
Pull request description:
This follows the introduction of GUI support in https://github.com/bitcoin-core/gui/pull/4
I don't think we should expect GUI users to self compile. This also enables external signer support by default for RPC users.
In addition this PR reduces the number of `#ifdef ENABLE_EXTERNAL_SIGNER`, which also fixes#21919. When compiled with `--disable-external-signer` such wallets can't be created in RPC or GUI, but they can be loaded. Attempting any action that calls HWI will trigger an error.
Side-note: this PR may or may not (currently) break CI for the GUI repository, as explained here: https://github.com/bitcoin-core/gui/pull/4#issuecomment-769859001
ACKs for top commit:
achow101:
ACK 2f5bdcbc31
hebasto:
re-ACK 2f5bdcbc31
Tree-SHA512: 1b71c5a8bea2be077ee9fa33a01130c957a0cf90951d4b7b04d3d0ef826bb77e474c3963abddfef2e2c1ea99d9c72cd2302d1eb9b5fcb7ba0bd2a625f006aa05
Just checking for the `sys/sdt.h` header isn't enough, as systems like
macOS have the header, but it doesn't actually have the dtrace probes,
which leads to compile failures.
5c7ee1b2da libsecp256k1 no longer has --with-bignum= configure option (Pieter Wuille)
bdca9bcb6c Squashed 'src/secp256k1/' changes from 3967d96bf1..efad3506a8 (Pieter Wuille)
cabb566123 Disable certain false positive warnings for libsecp256k1 msvc build (Pieter Wuille)
Pull request description:
This updates our src/secp256k1 subtree to the latest upstream master. The changes include:
* The introduction of safegcd-based modular inverses, reducing ECDSA signing time by 25%-30% and ECDSA verification time by 15%-17%.
* [Original paper](https://gcd.cr.yp.to/papers.html) by Daniel J. Bernstein and Bo-Yin Yang
* [Implementation](https://github.com/bitcoin-core/secp256k1/pull/767) by Peter Dettman; [final](https://github.com/bitcoin-core/secp256k1/pull/831) version
* [Explanation](https://github.com/bitcoin-core/secp256k1/blob/master/doc/safegcd_implementation.md) of the algorithm using Python snippets
* [Analysis](https://github.com/sipa/safegcd-bounds) of the maximum number of iterations the algorithm needs
* [Formal proof in Coq](https://medium.com/blockstream/a-formal-proof-of-safegcd-bounds-695e1735a348) by Russell O'Connor, for a high-level equivalent algorithm
* Removal of libgmp as an (optional) dependency (which wasn't used in the Bitcoin Core build)
* CI changes (Travis -> Cirrus)
* Build system improvements
ACKs for top commit:
laanwj:
Tested ACK 5c7ee1b2da
Tree-SHA512: ad8ac3746264d279556a4aa7efdde3733e114fdba8856dd53218588521f04d83950366f5c1ea8fd56329b4c7fe08eedf8e206f8f26dbe3f0f81852e138655431
e9f948c727 build: Convert warnings into errors when testing for -fstack-clash-protection (Hennadii Stepanov)
Pull request description:
Apple clang version 12.0.5 (clang-1205.0.22.9) that is a part of Xcode 12.5, and is based on LLVM clang 11.1.0, fires spammy warnings:
```
clang: warning: argument unused during compilation: '-fstack-clash-protection' [-Wunused-command-line-argument]
```
From the https://github.com/apple/llvm-project:
```
$ git log --oneline | grep 'stack-clash-protection'
00065d5cbd02 Revert "-fstack-clash-protection: Return an actual error when used on unsupported OS"
4d59c8fdb955 -fstack-clash-protection: Return an actual error when used on unsupported OS
df3bfaa39071 [Driver] Change -fnostack-clash-protection to -fno-stack-clash-protection
68e07da3e5d5 [clang][PowerPC] Enable -fstack-clash-protection option for ppc64
515bfc66eace [SystemZ] Implement -fstack-clash-protection
e67cbac81211 Support -fstack-clash-protection for x86
454621160066 Revert "Support -fstack-clash-protection for x86"
0fd51a4554f5 Support -fstack-clash-protection for x86
658495e6ecd4 Revert "Support -fstack-clash-protection for x86"
e229017732bc Support -fstack-clash-protection for x86
b03c3d8c6209 Revert "Support -fstack-clash-protection for x86"
4a1a0690ad68 Support -fstack-clash-protection for x86
f6d98429fcdb Revert "Support -fstack-clash-protection for x86"
39f50da2a357 Support -fstack-clash-protection for x86
```
I suppose, that Apple clang-1205.0.22.9 ends with on of the "Revert..." commits.
This PR prevents using of the `-fstack-clash-protection` flag if it causes warnings.
---
System: macOS Big Sur 11.3 (20E232).
ACKs for top commit:
jarolrod:
re-ACK e9f948c727
Sjors:
tACK e9f948c727 on macOS 11.3.1
Tree-SHA512: 30186da67f9b0f34418014860c766c2e7f622405520f1cbbc1095d4aa4038b0a86014d76076f318a4b1b09170a96d8167c21d7f53a760e26017f486e1a7d39d4
When building with Clang, if `-fstack-clash-protection` is used with an
unsupported target, it may result in hundreds of
`-Wunused-command-line-argument` warnings at compile time. This is
currently the case when building for at least Darwin using Apple or LLVM
Clang.
Unsupported targets may also include *BSD, however that is changing; see
further discussion in https://reviews.llvm.org/D92245 and
https://reviews.freebsd.org/D27366.
Note that this option is already skipped for Windows.
a5491882a0 build: fix configuring when building depends with NO_BDB=1 (fanquake)
Pull request description:
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets), `./configure` will fail as it still tries to find bdb. i.e:
```bash
make -C depends/ NO_QT=1 NO_BDB=1 NO_UPNP=1 NO_ZMQ=1 NO_NATPMP=1 -j8
...
copying packages: native_b2 boost libevent sqlite
./autogen.sh
./configure --prefix=/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu
...
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of bdb, without opting out of wallets entirely, and still configure successfully. I think I've tested across most potential configurations. i.e:
```bash
./configure (bdb and sqlite on system)
bdb & sqlite are both are available
./configure --without-bdb (bdb and sqlite on system)
only sqlite
./configure --without-sqlite (bdb and sqlite on system)
only bdb
./configure --disable-wallet (bdb and sqlite on system)
neither bdb or sqlite
depends NO_WALLET=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
neither bdb or sqlite
depends NO_BDB=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only sqlite
depends NO_SQLITE=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only bdb
depends
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
bdb and sqlite
```
ACKs for top commit:
laanwj:
Code review ACK a5491882a0
jarolrod:
ACK a5491882a0
Tree-SHA512: baf7d2543a401db0d846095415ff449c04ecfb4a74c734dc51e79453702f9051210daeef686970f11fcffd32cdfadbc58acd54f0706aceecfb3edb0ff17310d7
7abac98d3e configure: Support -f{debug,macro}-prefix-map (Anthony Towns)
Pull request description:
When bitcoin is checked out in two directories (eg via git worktree) object files between the two will differ due to the full path being included in the debug section. `-fdebug-prefix-map` is used to replace this with "." to avoid this unnecessary difference and allow ccache to share objects between worktrees (provided the source and compile options are the same).
Also provide `-fmacro-prefix-map` if supported so that the working dir is not encoded in `__FILE__` macros.
ACKs for top commit:
practicalswift:
cr ACK 7abac98d3e: patch looks correct
fanquake:
ACK 7abac98d3e
Tree-SHA512: b6a37c1728ec3b2e552f244da0e66db113c1e7662c7ac502e12ff466f3dbfbfefae12695ca135137c50dbb1c4c5d84059116c0cd09b391a17466dc77b8726679
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets),
./configure will fail as it still tries to find bdb. i.e:
```bash
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of
bdb without opting out of wallets entirely, and still configure
successfully.
a4e970adb6 build: enable -Wdocumentation if suppressing external warnings (fanquake)
3b0078f958 doc: fixup -Wdocumentation issues (fanquake)
c6edcf1c71 build: suppress libevent warnings if supressing external warnings (fanquake)
Pull request description:
Enable `-Wdocumentation` by taking advantage of our `--enable-suppress-external-warnings` flag. Most of the CIs are using this flag now, so any regressions should be caught.
This also required modifying libevents flags when suppressing warnings, as depending on the version being built against, that could generate a large number of warnings. i.e:
```bash
In file included from httpserver.cpp:34:
In file included from ./support/events.h:12:
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:464:11: warning: parameter 'req' not found in the function declaration [-Wdocumentation]
@param req a request object
^~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:465:11: warning: parameter 'databuf' not found in the function declaration [-Wdocumentation]
@param databuf the data chunk to send as part of the reply.
^~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:467:11: warning: parameter 'call' not found in the function declaration [-Wdocumentation]
@param call back's argument.
^~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:939:4: warning: declaration is marked with '@deprecated' command but does not have a deprecation attribute [-Wdocumentation-deprecated-sync]
@deprecated This function is deprecated; you probably want to use
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:946:1: note: add a deprecation attribute to the declaration to silence this warning
char *evhttp_decode_uri(const char *uri);
^
__AVAILABILITY_INTERNAL_DEPRECATED
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:979:5: warning: declaration is marked with '@deprecated' command but does not have a deprecation attribute [-Wdocumentation-deprecated-sync]
@deprecated This function is deprecated as of Libevent 2.0.9. Use
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:987:1: note: add a deprecation attribute to the declaration to silence this warning
int evhttp_parse_query(const char *uri, struct evkeyvalq *headers);
^
__AVAILABILITY_INTERNAL_DEPRECATED
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:1002:11: warning: parameter 'query_parse' not found in the function declaration [-Wdocumentation]
@param query_parse the query portion of the URI
^~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:1002:11: note: did you mean 'uri'?
@param query_parse the query portion of the URI
^~~~~~~~~~~
uri
69 warnings generated.
```
Note that a lot of these have already been fixed upstream.
ACKs for top commit:
laanwj:
Concept and code review ACK a4e970adb6
practicalswift:
cr ACK a4e970adb6: automatic compiler feedback comes sooner and is more reliable than manual reviewer feedback
jonatack:
Light ACK a4e970adb6 skimmed the changes, clang 11 build is clean with the change, verified -Wdocumentation build warnings with this change when a doc fix was reverted
Tree-SHA512: 57a1e30cffcc8bcceee72d85f58ebe29eae525861c70acb237541bd480c51ede89875c033042c0af376fdbb49fb7f588ef9282a47c6e78f9d4501c41f1b21eb6
The register keyword was deprecated in C++11, and removed in C++17. Now
that we require C++17, we shouldn't have to supress warnings for a
non-existant feature.
0eabb2abed build: Remove unused header from the build system (Hennadii Stepanov)
Pull request description:
The only `#include <miniupnpc/miniwget.h>` was removed in #16659.
ACKs for top commit:
practicalswift:
cr ACK 0eabb2abed
fanquake:
ACK 0eabb2abed
Tree-SHA512: 630da03875c851e80286561eae0f966c89624cbb17b90f70e2bec9a69146e79d088fc176e07a4906915770ac1cdb11341a7a431ea7cf6a59d2816e927486f335
246774e264 depends: fix Qt precompiled headers bug (Igor Cota)
8e7ad4146d depends: disable Qt Vulkan support on Android (Igor Cota)
ba46adaa1a CI: add Android APK build to cirrus (Igor Cota)
7563720e30 CI: add Android APK build script (Igor Cota)
ebfb10cb75 Qt: add Android packaging support (Igor Cota)
Pull request description:
This PR is the third and final piece of the basic Android support puzzle - it depends on https://github.com/bitcoin/bitcoin/pull/16110 and is related to https://github.com/bitcoin/bitcoin/pull/16883. It introduces an `android` directory under `qt` and a simple way to build an Android package of `bitcoin-qt`:
1. Build depends for Android as described in the [README](https://github.com/bitcoin/bitcoin/blob/master/depends/README.md)
2. Configure with one of the resulting prefixes
3. Run `make && make apk` in `src/qt`
The resulting APK files will be in `android/build/outputs/apk`. You can install them manually or with [adb](https://developer.android.com/studio/command-line/adb). One can also open the `android` directory in Android Studio for that integrated development and debugging experience. `BitcoinQtActivity` is your starting point.
Under the hood makefile `apk` target:
1. Renames the `bitcoin-qt` binary to `libbitcoin-qt.so` and copies it over to a folder under `android/libs` depending on which prefix and corresponding [ABI](https://developer.android.com/ndk/guides/abis.html#sa) `bitcoin-qt` was built for
2. Takes `libc++_shared.so` from the Android NDK and puts in the same place. It [must be included](https://developer.android.com/ndk/guides/cpp-support) in the APK
3. Extracts Qt for Android Java support files from the `qtbase` archive in `depends/sources` to `android/src`
There is also just a tiny bit of `ifdef`'d code to make the Qt Widgets menus usable. It's not pretty but it works and is a stepping stone towards https://github.com/bitcoin/bitcoin/pull/16883.
ACKs for top commit:
MarcoFalke:
cr ACK 246774e264
laanwj:
Code review ACK 246774e264
Tree-SHA512: ba30a746576a167545223c35a51ae60bb0838818779fc152c210f5af1413961b2a6ab6af520ff92cbc8dcd5dcb663e81ca960f021218430c1f76397ed4cead6c
Introduce an android directory under qt and allow one to package bitcoin-qt for Android by running make apk.
Add bitcoin-qt Android build instructions.
This has never worked with any of the mingw-w64 compilers we use, and
the -O0 is causing issues for builders applying spectre mitigations.
Recent discussion on https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458
also indicates that this should just not be used on Windows.
e017a913d0 bitcoind: Add -daemonwait option to wait for initialization (Wladimir J. van der Laan)
c3e6fdee6d shutdown: Use RAII TokenPipe in shutdown (Wladimir J. van der Laan)
612f746a8f util: Add RAII TokenPipe (Wladimir J. van der Laan)
Pull request description:
This adds a `-daemonwait` flag that does the same as `-daemon` except that it, from a user perspective, backgrounds the process only after initialization is complete. This is similar to the behaviour of some other software such as c-lightning.
This can be useful when the process launching bitcoind wants to guarantee that either the RPC server is running, or that initialization failed, before continuing. The exit code indicates the initialization result.
The use of the libc function `daemon()` is replaced by a custom implementation which is inspired by the [glibc implementation](https://github.com/lattera/glibc/blob/master/misc/daemon.c#L44), but which also creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is that no MACOS-specific pragmas are needed anymore to silence a deprecation warning.
TODO:
- [x] Factor out `token_read` and `token_write` to an utility, and use them in `shutdown.cpp` as well—this is exactly the same kind of communication mechanism.
- [x] RAII-ify pipe endpoints.
- [x] Improve granularity of the `configure.ac` checks. This currently still checks for the function `daemon()` which makes no sense as it's not used. It should check for individual functions such as
`fork()` and `setsid()` etc—the former being required, the second optional.
- [-] ~~Signal propagation during initialization: if say, pressing Ctrl-C during `-daemonwait` it would be good to pass this SIGINT on to the child process instead of detaching the parent process and letting the child run free.~~ This is not necessary, see https://github.com/bitcoin/bitcoin/pull/21007#issuecomment-769007341.
Future:
- Consider if it makes sense to use this in the RPC tests (there would be no more need for "is RPC ready" polling loops). I think this is out of scope for this PR.
ACKs for top commit:
jonatack:
Tested ACK e017a913d0 checked change since previous review is move-only
Tree-SHA512: 53369b8ca2247e4cf3af8cb2cfd5b3399e8e0e3296423d64be987004758162a7ddc1287b01a92d7692328edcb2da4cf05d279b1b4ef61a665b71440ab6a6dbe2
This adds a `-daemonwait` flag that does the same as `-daemon` except
it, from a user perspective, backgrounds the process only after
initialization is complete.
This can be useful when the process launching bitcoind wants to
guarantee that either the RPC server is running, or that initialization
failed, before continuing. The exit code indicates the initialization
result.
This replaces the use of the libc function `daemon()` by a custom
implementation which is inspired by the glibc implementation, but also
creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is
that no MACOS-specific pragmas are needed anymore to silence a
deprecation warning.
faa06ecc9c build: Bump minimum Qt version to 5.9.5 (Hennadii Stepanov)
Pull request description:
Close#20104.
ACKs for top commit:
laanwj:
Code review ACK faa06ecc9c
jarolrod:
ACK faa06ecc9c
fanquake:
ACK faa06ecc9c - this should be ok to do now.
Tree-SHA512: 7295472b5fd37ffb30f044e88c39d375a5a5187d3f2d44d4e73d0eb0c7fd923cf9949c2ddab6cddd8c5da7e375fff38112b6ea9779da4fecce6f024d05ba9c08
fae216a73d scripted-diff: Rename MakeFuzzingContext to MakeNoLogFileContext (MarcoFalke)
fa4fbec03e scripted-diff: Rename PROVIDE_MAIN_FUNCTION -> PROVIDE_FUZZ_MAIN_FUNCTION (MarcoFalke)
Pull request description:
Split out two renames from #21003:
* `PROVIDE_FUZZ_MAIN_FUNCTION`. *Reason*: This in only used by fuzzing, so the name should indicate that.
* `MakeNoLogFileContext`. *Reason*: Better reflects what the helper does. Also, prepares it to be used in non-fuzz tests in the future.
ACKs for top commit:
practicalswift:
cr ACK fae216a73d: scripted-diff looks correct
Tree-SHA512: e5d347746f5da72b0c86fd4f07ac2e4b3016e88e8c97a830c73bd79d0af6d0245fe7712487fc20344d6cc25958941716c1678124a123930407e3a437265b71df
9bac71350d build: make HAVE_O_CLOEXEC available outside LevelDB (bugfix) (Sebastian Falbesoner)
584fd91d2d init: only use pipe2 if availabile, check in configure (Sebastian Falbesoner)
Pull request description:
The result of the O_CLOEXEC availability check is currently only set in the Makefile and passed to LevelDB (see `LEVELDB_CPPFLAGS_INT` in `src/Makefile.leveldb.include`), but not defined to be used in our codebase. This means that code within the preprocessor conditional `#if HAVE_O_CLOEXEC` was actually never compiled. On the master branch this is currently used for pipe creation in `src/shutdown.cpp`, PR #21007 moves this part to a new module (I found the issue while testing that PR).
The fix is similar to the one in #19803, which solved the same problem for HAVE_FDATASYNC.
In the course of working on the PR it turned out that pipe2 is not available an all platforms, hence a configure check and a corresponding define HAVE_PIPE2 is introduced and used.
The PR can be tested by anyone with a system that has pipe2 and O_CLOEXEC available by putting gibberish into the HAVE_O_CLOEXEC block: on master, everything should compile fine, on PR, the compiler should abort with an error. At least that's my naive way of testing preprocessor logic, happy to hear more sophisticated ways :-)
ACKs for top commit:
laanwj:
Code review ACK 9bac71350d
Tree-SHA512: aec89faf6ba52b6f014c610ebef7b725d9e967207d58b42a4a71afc9f1268fcb673ecc85b33a2a3debba8105a304dd7edaba4208c5373fcef2ab83e48a170051
This option replaces --with-boost-process
This prepares external signer support to be disabled by default.
It adds a configure option to enable this feature and to check
if Boost::Process is present.
This also exposes ENABLE_EXTERNAL_SIGNER to the test suite via test/config.ini
c5da2749e2 build: actually stop configure if Boost isn't available (fanquake)
cad8b527ea build: explicitly install libboost-dev package (fanquake)
Pull request description:
If Boost is not found via AX_BOOST_BASE, we don't actually stop
configuring, only a warning is emitted:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version MINIMUM_REQUIRED_BOOST or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
```
Instead we usually fail when one of the other AX_BOOST_* macros fails to find a library. These macros are slowly being
removed, and in any case, it makes more sense to fail earlier if Boost is missing.
If Boost is unavailable, the failure now looks like:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version 1.58.0 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
configure: error: Boost is not available!
```
Note that we now just pass the version into AX_BOOST_BASE, which fixes it's display in the output (rather than showing `MINIMUM_REQUIRED_BOOST`).
This PR also has a commit that adds `libboost-dev` to our install instructions and CI. This package is currently installed as a side-effect of installing our other libboost-*-dev packages. However as those continue to disappear, it makes sense to install boost-dev explicitly.
ACKs for top commit:
laanwj:
Code review ACK c5da2749e2
MarcoFalke:
Concept ACK c5da2749e2
Tree-SHA512: f866062f9d7d3a2316b6c887f17c664b9cfff41fdc0cb99ca79d641240fb01a5ae0d34140e515bc465219e1b43d5ca84f7c55f48b9c5b45a80ff2795dafd072b
If Boost is not found via AX_BOOST_BASE, we don't actually stop
configuring, only a warning is emitted:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version MINIMUM_REQUIRED_BOOST or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
```
Instead we would usually fail when one of the other
AX_BOOST_* macros fails to find a library. These macros are slowly being
removed, and in any case, it makes more sense to fail earlier if Boost
is missing.
If Boost is unavailable, the failure now looks like:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version 1.58.0 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
configure: error: Boost is not available!
```
Note that we now just pass the version into AX_BOOST_BASE, which fixes
it's display in the output (rather than MINIMUM_REQUIRED_BOOST).
Performing a series of link checks for a Boost component that is
header-only doesn't make much sense, and currently means we just have
another confusing Boost macro in our tree. I'm not sure why this was
originally done this way; maybe Sjors or luke-jr can elaborate
(#15382 (929cda5470))?
The macro also has the side-effect of producing confusing error
messages. i.e in #20744, the CI is currently failing with:
```bash
checking for boostlib >= 1.58.0 (105800) lib path in "/tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/lib"... yes
checking for boostlib >= 1.58.0 (105800)... yes
checking whether the Boost::Process library is available... yes
configure: error: Could not find a version of the Boost::Process library!
```
This isn't useful, given there is no such thing as a `Boost::Process`
library.
This PR just removes the macro entirely, but maintains a `--with-boost-process`
(defaulting to off), flag to configure. Hopefully this will also be
removed, in favour of `--enable-disable-external-signer` if/when #16546
is merged.
060a2a64d4 ci: remove boost thread installation (fanquake)
06e1d7d81d build: don't build or use Boost Thread (fanquake)
7097add83c refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef8 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d4
vasild:
ACK 060a2a64d4
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
We are already testing for this, and our test works correctly with a Darwin
target, where the macro does not. Darwin targets do not support "protected"
visibility.
