72c9a1fe94 test: Check that network stays suspended after dumptxoutset if it was off before (Fabian Jahr)
Pull request description:
Follow-up to #30817 which covered the robustness of `dumptxoutset`: network is deactivated during the run but re-activated even when an issue was encountered. But it did not cover the case if the user had deactivated the network themselves before. In that case the user may want the network to stay off so the network is not reactivated after `dumptxoutset` finishes. A test for this behavior is added here.
ACKs for top commit:
achow101:
ACK 72c9a1fe94
pablomartin4btc:
ACK 72c9a1fe94
theStack:
utACK 72c9a1fe94
tdb3:
tested ACK 72c9a1fe94
Tree-SHA512: 18a57c5782e99a018414db0597e88debeb32666712c2a75dddbb55135d8f1ddd1eeacba8bbbd35fc03b6c4ab0522fe074ec08edea729560b018f51efabf00e89
This avoids low-level log errors that are supposed to only occur when
there is an actual problem with the block on disk missing unexpectedly,
but not in the case where the block and/or undo data are expected not to be there.
It changes behavior such that in the first case (block index indicates
data is available but retrieving it fails) an error is thrown.
It also adjusts a functional tests that tried to simulate not
having undo data (but having block data) by deleting the undo file.
This situation should occur reality because block and undo data are pruned together.
Instead, test this situation with a block that hasn't been connected.
This improves the error message of the getblock and getblockstats rpc and prevents calls to
ReadRawBlockFromDisk(), which are unnecessary if we know
from the header nStatus field that the block is not available.
19f4a7c95a test: Wait for local services to update in feature_assumeutxo (Fabian Jahr)
Pull request description:
Closes#30878
It seems like there is a race where the block is stored locally and `getblock` does not error anymore, but `ActivateBestChain` has not finished yet, so the local services are not updated yet either. Fix this by waiting for the local services to update.
Can be reproduced locally by adding the sleep here:
```cpp
──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
src/validation.cpp:3567: bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr< │
──────────────────────────────────────────────────────────────────────────────────────────────────────────┘
}
if (WITH_LOCK(::cs_main, return m_disabled)) {
std::this_thread::sleep_for(std::chrono::seconds(10));
// Background chainstate has reached the snapshot base block, so exit.
// Restart indexes to resume indexing for all blocks unique to the snapshot
```
ACKs for top commit:
maflcko:
review-only ACK 19f4a7c95a
achow101:
ACK 19f4a7c95a
pablomartin4btc:
tACK 19f4a7c95a
furszy:
Code review ACK [19f4a7c](19f4a7c95a).
Tree-SHA512: 70dad3795988956c5e20f2d2d895fb56c5e3ce257c7547d3fd729c88949f0e24cb34594da1537bce8794ad02b2db44e7e46e995aa32539cd4dd84c4f1d4bb1c4
fa5bc450d5 util: Use compile-time check for LogConnectFailure (MarcoFalke)
fa7087b896 util: Use compile-time check for FatalErrorf (MarcoFalke)
faa62c0112 util: Add ConstevalFormatString (MarcoFalke)
fae7b83eb5 lint: Remove forbidden functions from lint-format-strings.py (MarcoFalke)
Pull request description:
The `test/lint/lint-format-strings.py` was designed to count the number of format specifiers and assert that they are equal to the number of parameters passed to the format function. The goal seems reasonable, but the implementation has many problems:
* It is written in Python, meaning that C++ code can not be parsed correctly. Currently it relies on brittle regex and string parsing.
* Apart from the parsing errors, there are also many logic errors. For example, `count_format_specifiers` allows a mix of positional specifiers and non-positional specifiers, which can lead to runtime format bugs. Also, `count_format_specifiers` silently skipped over "special" format specifiers, which are valid in tinyformat, which again can lead to runtime format bugs being undetected.
* The brittle logic has a history of breaking in pull requests that are otherwise fine. This causes the CI to fail and the pull request being blocked from progress until the bug in the linter is fixed, or the code is rewritten to work around the bug.
* It is only run in the CI, or when the developer invokes the script. It would be better if the developer got the error message at compile-time, directly when writing the code.
Fix all issues by using a `consteval` checked format string in `FatalErrorf` and `LogConnectFailure`.
This is the first step toward https://github.com/bitcoin/bitcoin/issues/30530 and a follow-up will apply the approach to the other places.
ACKs for top commit:
stickies-v:
re-ACK fa5bc450d5
l0rinc:
ACK fa5bc450d5
hodlinator:
ACK fa5bc450d5
ryanofsky:
Code review ACK fa5bc450d5
Tree-SHA512: d6189096b16083143687ed1b1559cf4f92f97dd87bc5d00673e44f4fb9fce7bb7b215cfdfc39b6e6a24f0b75a79a03ededce966639e554f7172e1fc22cf015ae
72b46f28bf test: fix exclude parsing for functional runner (Max Edwards)
Pull request description:
This restores previous behaviour of being able to exclude a test by name without having to specify .py extension.
It was noticed in https://github.com/bitcoin/bitcoin/issues/30851 that tests were no longer being excluded.
PR https://github.com/bitcoin/bitcoin/pull/30244 introduced being able to exclude a specific tests based on args (such as `--exclude "rpc_bind.py --ipv6`) but it made the wrong assumption that test names intended to be excluded would include the .py extension.
The following https://github.com/bitcoin/bitcoin/pull/30244#issuecomment-2344009687 shows that this is not how the `--exclude` flag was used in CI.
https://github.com/bitcoin/bitcoin/pull/30244#issuecomment-2344009687 gave three examples of `--exclude` being used in CI so I compared the number of tests that the runner would run for these three examples in three situations, before #30244 was introduced, in master today and with this PR applied.
Example:
`--previous-releases --coverage --extended --exclude feature_dbcrash`
Test count:
Before #30244 introduced: 314
Master: 315
With this PR: 314
Example:
`--exclude feature_init,rpc_bind,feature_bind_extra`
Test count:
Before #30244 introduced: 306
Master 311
With this PR: 306
Example:
`--exclude rpc_bind,feature_bind_extra`
Before #30244 introduced: 307
Master 311
With this PR: 307
I've also tested that the functionality introduced with #30244 remains and we can still exclude specific tests by argument.
ACKs for top commit:
maflcko:
review ACK 72b46f28bf
willcl-ark:
ACK 72b46f28bf
Tree-SHA512: 37c0e3115f4e3efdf9705f4ff8cd86a5cc906aacc1ab26b0f767f5fb6a953034332b29b0667073f8382a48a2fe9d649b7e60493daf04061260adaa421419d8c8
2ad560139b Remove unused src_dir param from run_tests (Lőrinc)
Pull request description:
The `src_dir` usage was removed in a8a2e364ac (diff-437d7f6e9f2229879b60aae574a8217f14c643bbf3cfa9225d8011d6d52df00cL598), making the parameter unused.
Top commit has no ACKs.
Tree-SHA512: 1fd8b93811b4ab467ba5a160a4fe204e9606e1bf237c7595ed6f8b7821cf59d2a776c0e1e154852a45b2a35e5bdbd8996314e4f63a9c750f21b9a17875cb636a
992f83bb6f test: add coverage for assumeUTXO honest peers disconnection (furszy)
6d5812e5c8 assumeUTXO: fix peers disconnection during sync (furszy)
Pull request description:
Because AssumeUTXO nodes prioritize tip synchronization, they relay their local
address through the network before completing the background chain sync.
This, combined with the advertising of full-node service (`NODE_NETWORK`), can
result in an honest peer in IBD connecting to the AssumeUTXO node (while syncing)
and requesting an historical block the node does not have. This behavior leads to
an abrupt disconnection due to perceived unresponsiveness from the AssumeUTXO
node.
This lack of response occurs because nodes ignore `getdata` requests when they do
not have the block data available (further discussion can be found in #30385).
Fix this by refraining from signaling full-node service support while the
background chain is being synced. During this period, the node will only
signal `NODE_NETWORK_LIMITED` support. Then, full-node (`NODE_NETWORK`)
support will be re-enabled once the background chain sync is completed.
Thanks mzumsande for a post-#30385 convo too.
Testing notes:
Just cherry-pick the second commit (bb08c22) on master.
It will fail there, due to the IBD node requesting historical blocks to the snapshot
node - which is bad because the snapshot node will ignore the requests and
stall + disconnect after some time.
ACKs for top commit:
achow101:
ACK 992f83bb6f
naumenkogs:
ACK 992f83bb6f
mzumsande:
ACK 992f83bb6f
Tree-SHA512: fef525d1cf3200c2dd89a346be9c82d77f2e28ddaaea1f490a435e180d1a47a371cadea508349777d740ab56e94be536ad8f7d61cc81f6550c58b609b3779ed3
082779d606 test: Add explicit onion bind to p2p_permissions (Ava Chow)
Pull request description:
When the bind option is replaced in the bitcoin.conf, bitcoind will attempd to bind to the default tor listening port. If another bitcoind is running that is already bound to that port, the bind will fail which, since #22729, causes the test to fail.
This failure can be avoided by explicitly binding the tor port when the bind is removed.
ACKs for top commit:
tdb3:
ACK 082779d606
theStack:
re-ACK 082779d606
glozow:
ACK 082779d606
Tree-SHA512: 4acb69ea2e00aeacf9e7c9ab9595ceaf0e0d2adbd795602034b2184197d9bad54c7bc9f3da43ef9c52a71869fe96ba8c87fc5b7c37880f258f5a2aaab2b4046c
Exercising and verifying the following points:
1. An IBD node can sync headers from an AssumeUTXO node at
any time.
2. IBD nodes do not request historical blocks from AssumeUTXO
nodes while they are syncing the background-chain.
3. The assumeUTXO node dynamically adjusts the network services
it offers according to its state.
4. IBD nodes can fully sync from AssumeUTXO nodes after they
finish the background-chain sync.
Because AssumeUTXO nodes prioritize tip synchronization, they relay their local
address through the network before completing the background chain sync.
This, combined with the advertising of full-node service (NODE_NETWORK), can
result in an honest peer in IBD connecting to the AssumeUTXO node (while syncing)
and requesting an historical block the node does not have. This behavior leads to
an abrupt disconnection due to perceived unresponsiveness (lack of response)
from the AssumeUTXO node.
This lack of response occurs because nodes ignore getdata requests when they do
not have the block data available (further discussion can be found in PR 30385).
Fix this by refraining from signaling full-node service support while the
background chain is being synced. During this period, the node will only
signal 'NODE_NETWORK_LIMITED' support. Then, full-node ('NODE_NETWORK')
support will be re-enabled once the background chain sync is completed.
When the bind option is replaced in the bitcoin.conf, bitcoind will
attempd to bind to the default tor listening port. If another bitcoind
is running that is already bound to that port, the bind will fail which,
since #22729, causes the test to fail.
This failure can be avoided by explicitly binding the tor port when the
bind is removed.
c2b779da4e refactor: Manage dumptxoutset RAII classes with std::optional (Fabian Jahr)
4b5bf335ad test: Add coverage for failing dumptxoutset behavior (Fabian Jahr)
Pull request description:
This adds a test that checks that network activity is not suspended if dumptxoutset fails in the middle of its process which is implemented with the `NetworkDisable` RAII class. I would have liked to add coverage for the `TemporaryRollback` RAII class but that seems a lot more tricky since the failure needs to happen at some point after the rollback and on the scale of our test chain here I couldn't find a way to do it yet. This was requested by pablomartin4btc here: https://github.com/bitcoin/bitcoin/pull/30808#pullrequestreview-2280450117. To test the test you can comment out the content of the destructor of `NetworkDisable`.
It also addresses the feedback by ryanofsky to use `std::optional` instead of `std::unique_ptr` for the management of the RAII object: https://github.com/bitcoin/bitcoin/pull/30808#discussion_r1744149228
ACKs for top commit:
achow101:
ACK c2b779da4e
pablomartin4btc:
cr & tACK c2b779da4e
tdb3:
ACK c2b779da4e
BrandonOdiwuor:
Code Review ACK c2b779da4e
theStack:
ACK c2b779da4e
Tree-SHA512: 9556e75014a2599bb870b70faf887608b332f2312626333f771d4ec11c04f863a2cf17e223ec473d4e8b0c9e8008394a4e0c321561f7ef3a2eec713dcfaea58a
ee47ca29d6 init: fix fatal error on '-wallet' negated option value (furszy)
Pull request description:
Currently, if users provide a double negated value such as '-nowallet=0' or a non-boolean
convertible value to a negated option such as '-nowallet=not_a_boolean', the initialization
process results in a fatal error, causing an unclean shutdown and displaying a poorly
descriptive error message:
"JSON value of type bool is not of expected type string." (On bitcoind. The GUI
does not display any error msg - upcoming PR -).
This PR fixes the issue by ensuring that only string values are returned in the
the "wallet" settings list, failing otherwise. It also improves the clarity of the
returned error message.
Note:
This bug was introduced in https://github.com/bitcoin/bitcoin/pull/22217. Where the `GetArgs("-wallet")` call was
replaced by `GetSettingsList("-wallet")`.
ACKs for top commit:
achow101:
ACK ee47ca29d6
ryanofsky:
Code review ACK ee47ca29d6, just adding the suggested test since last review
TheCharlatan:
ACK ee47ca29d6
ismaelsadeeq:
Tested ACK ee47ca29d6
Tree-SHA512: 5f01076f74a048019bb70791160f0accc2db7a457d969cb23687bed81ccbbdec1dda68311e7c6e2dd56250e23e8d926d4066e5014b2a99a2fc202e24ed264fbd
27c976d11a fix: increase consistency of rpcauth parsing (tdb3)
2ad3689512 test: add norpcauth test (tdb3)
67df0dec1a test: blank rpcauth CLI interaction (tdb3)
ecc98ccff2 test: add cases for blank rpcauth (tdb3)
Pull request description:
The current `rpcauth` parsing behavior is inconsistent and unintuitive (see https://github.com/bitcoin/bitcoin/pull/29141#issuecomment-1972085251 and additional details below).
The current behavior inconsistently treats empty `rpcauth` as an error (or not) depending on the location within CLI/bitcoin.conf and the location of adjacent valid `rpcauth` params.
Empty `rpcauth` is now consistently treated as an error and prevents bitcoind from starting.
Continuation of the upforgrabs PR #29141.
### Additional details:
Current `rpcauth` behavior is nonsensical:
- If an empty `rpcauth` argument was specified as the last command line argument, it would cause all other `rpcauth` arguments to be ignored.
- If an empty `rpcauth` argument was specified on the command line followed by any nonempty `rpcauth` argument, it would cause an error.
- If an empty `rpcauth=` line was specified after non-empty rpcauth line in the config file it would cause an error.
- If an empty `rpcauth=` line in a config file was first it would cause other rpcauth entries in the config file to be ignored, unless there were `-rpcauth` command line arguments and the last one was nonempty, in which case it would cause an error.
New behavior is simple:
- If an empty rpcauth config line or command line argument is used it will cause an error
ACKs for top commit:
naiyoma:
Tested ACK [27c976d11a)
achow101:
ACK 27c976d11a
ryanofsky:
Code review ACK 27c976d11a. Since last review commit message was just tweaked to clarify previous behavior.
Tree-SHA512: af2e9dd60d1ad030409ae2c3805ab139c7435327823d9f8bbeede815f376cb696a5929b08a6e8c8b5f7278ed49cfb231789f9041bd57f1f03ec96501b669da5b
Following changes were made:
1) Catch and signal error for duplicate string destinations.
2) Catch and signal error for invalid value type.
3) Catch and signal error for string destination not found in tx outputs.
4) Improved 'InterpretSubtractFeeFromOutputInstructions()' code organization.
5) Added test coverage for all possible error failures.
Also, fixed two PEP 8 warnings at the 'wallet_sendmany.py' file:
- PEP 8: E302 expected 2 blank lines, found 1 at the SendmanyTest class declaration.
- PEP 8: E303 too many blank lines (2) at skip_test_if_missing_module() and set_test_params()
fadbcd51fc bench: Remove redundant logging benchmarks (MarcoFalke)
fa8dd952e2 bench: Use LogInfo instead of the deprecated alias LogPrintf (MarcoFalke)
Pull request description:
`LogPrint*ThreadNames` is redundant with `LogWith(out)ThreadNames`,
because they all measure toggling the thread names (and check that it
has no effect on performance).
Fix it by removing the redundant ones. This also allows to drop a deprecated logging alias.
ACKs for top commit:
stickies-v:
ACK fadbcd51fc
Tree-SHA512: 4fe137f374aa4ee1aa0e1da4a1f9839c0e52c23dbb93198ecafee98de39d311cc47304bba4191f3807aa00c51b1eae543e3f270f03d341c84910e5e341a1d475
In case of a failure to create the dump, the node should not be left in an inconsistent state like deactivated network activity or an invalidated blockchain.
LogPrint*ThreadNames is redundant with LogWith(out)ThreadNames, because
they all measure toggling the thread names (and check that it has no
effect on performance).
This also allows to remove unused and deprecated macros.
8131bf7483 ci: parse TEST_RUNNER_EXTRA into an array (Max Edwards)
c4762b0aa0 test: allow excluding func test by name and arg (Max Edwards)
Pull request description:
While working on CI I wanted to disable some functional tests so I used the `TEST_RUNNER_EXTRA` var. The problem I had was tests that have flags such as `rpc_bind.py --ipv6` must be passed in quotes otherwise the `--ipv6` portion will be considered an argument to `test_runner.py` rather than a test name.
This change allows proper parsing of quotes and complex values such as:
```shell
TEST_RUNNER_EXTRA='--exclude "rpc_bind.py --ipv6,feature_proxy.py"'
```
Update:
While testing this it was noticed that `test_runner.py` when given `--exclude "rpc_bind.py --ipv6"` will exclude all `rpc_bind.py` tests so this PR has been updated to include a change to the test runner to only exclude the specific test if you pass an arg or exclude all tests of that name if you do not pass an arg. `--exclude rpc_bind.py` will exclude all three variants and `--exclude rpc_bind --ipv6` will only exclude the IPV6 variant.
ACKs for top commit:
maflcko:
ACK 8131bf7483
achow101:
ACK 8131bf7483
hebasto:
ACK 8131bf7483, tested on Ubuntu 23.10 and Windows 11.
Tree-SHA512: 82b73f12d627f533d8e5be4a518d455ef4427a755bbe03ccd11d0bb70c7ff3cee76220b0264fcfb236661c4cf5deba034cbfc2372b96d5861f3436c21eae8264
fafdb7df34 lint: Speed up flake8 checks (MarcoFalke)
faf17df7fb lint: Document missing py_lint dependency (MarcoFalke)
faebeb828f lint: Remove python whitespace and shadowing lint rules (MarcoFalke)
7777047835 lint: Remove python lint rules that are SyntaxError (MarcoFalke)
faaf3e53f0 test: [refactor] Fix F841 flake8 (MarcoFalke)
444421db69 test: [refactor] Fix E714 pycodestyle (MarcoFalke)
Pull request description:
The checks have many issues:
* Some checks that could in theory hide bugs are not applied -> Fix them and apply them going forward
* Some checks are redundant Python 2 checks, or of low value -> Remove them
* The checks are slow -> Speed them up from ~10 seconds to about ~20 milliseconds
ACKs for top commit:
davidgumberg:
review and tested reACK fafdb7df34
kevkevinpal:
ACK [fafdb7d](fafdb7df34)
achow101:
ACK fafdb7df34
Tree-SHA512: a0488b722cfaf7071bd6848cd3be002e0b6c38af80d8b5cbb08613c0b174ef63277289f960db8ac31adb09fe563a4973203b8fb10b83cbcfdc6f0ef39bd04410
c8e6771af0 test: restrict multiple CLI arguments (naiyoma)
8838c4f171 common/args.h: automate check for multiple cli commands (naiyoma)
Pull request description:
This PR is a continuation of the validation suggested [here](https://github.com/bitcoin/bitcoin/pull/27815) to ensure that only one Request Handler can be specified at a time.
ACKs for top commit:
stratospher:
reACK c8e6771.
achow101:
ACK c8e6771af0
tdb3:
cr re ACK c8e6771af0
Tree-SHA512: f4fe036fee342a54f1a7ac702ac35c40bf3d420fde6ab16313a75327292d5ee5c8ece1be9f852a13fcf73da1148b143b37b4894e294052abdde6eefb2e8c6f3f
ec317bc44b test: update satoshi_round function (naiyoma)
Pull request description:
This PR refactors `satoshi_round` to accept different rounding modes and make rounding a required argument.
Continuation of https://github.com/bitcoin/bitcoin/pull/23225
ACKs for top commit:
maflcko:
review ACK ec317bc44b
achow101:
ACK ec317bc44b
AngusP:
ACK ec317bc44b
Tree-SHA512: 070f0aa6f66e58bff7412cae6b71f5f4ab8c718c7b5eeba4bb604fe22c6416a1ced0474294f504e1c28943ddc073104466b5944b43bae27e42bee5ca85afa468
fb6d51eb25 signet/miner: Use argparse exclusive groups (Anthony Towns)
338a266a9a signet/miner: add support for a poolnum/poolid tag in mined blocks (Anthony Towns)
409ab7d35b signet/miner: add Generate.mine function (Anthony Towns)
7b31332370 signet/miner: add Generate.gbt function (Anthony Towns)
85c5c0bea9 signet/miner: add Generate.next_block_time function (Anthony Towns)
5540e6ca49 signet/miner: move next_block_* functions into new Generator class (Anthony Towns)
35f4631196 signet/miner: rename do_decode_psbt to decode_psbt (Anthony Towns)
aac040b439 signet/miner: drop create_coinbase function (Anthony Towns)
16951f549e signet/miner: drop do_createpsbt function (Anthony Towns)
3aed0a4284 signet/miner: drop get_reward_address function (Anthony Towns)
Pull request description:
Refactors the code a bunch, and adds `--poolnum` / `--poolid` options so that signers can tag their coinbases in a way that explorers can recognise (see also https://github.com/bitcoin-data/mining-pools/pull/82 and https://github.com/mempool/mempool/issues/2903).
The refactoring in particular helps enable the "try using inquisition's getblocktemplate, and if that doesn't work fall back to core's getblocktemplate" logic, as described/implemented in https://github.com/bitcoin-inquisition/bitcoin/pull/7
ACKs for top commit:
achow101:
ACK fb6d51eb25
danielabrozzoni:
Code review ACK fb6d51eb25
Tree-SHA512: d84095c4045ab196685b847e04ce2cdaedf387bc2527430ede918318dc5b70bf3d87b754264016f895f506fac70d4fdea5ef3cd8c3c375fd586afeae01e045e5
6eeb188d40 test: adds seednode functional tests (Sergi Delgado Segura)
3270f0adad net: Favor peers from addrman over fetching seednodes (Sergi Delgado Segura)
Pull request description:
This is a follow-up of #28016 motivated by https://github.com/bitcoin/bitcoin/pull/28016#pullrequestreview-1913140932 and https://github.com/bitcoin/bitcoin/pull/28016#issuecomment-1984448937.
The current behavior of seednode fetching is pretty eager: we do it as the first step under `ThreadOpenNetworkConnections` even if some peers may be queryable from our addrman. This poses two potential issues:
- First, if permanently set (e.g. running with seednode in a config file) we'd be signaling such seed every time we restart our node
- Second, we will be giving the seed node way too much influence over our addrman, populating the latter with data from the former even when unnecessary
This changes the behavior to only add seednodes to `m_addr_fetch` if our addrman is empty, or little by little after we've spent some time trying addresses from our addrman. Also, seednodes are added to `m_addr_fetch` in random order, to avoid signaling the same node in case more than one seed is added and we happen to try them over multiple restarts
ACKs for top commit:
achow101:
ACK 6eeb188d40
cbergqvist:
ACK 6eeb188d40
itornaza:
Tested ACK 6eeb188d40
tdb3:
ACK 6eeb188d40
Tree-SHA512: b04445412f22018852d6bef4d3f1e88425ee6ddb434f61dcffa9e0c41b8e31f8c56f83858d5c7686289c86dc4c9476c437df15ea61a47082e2bb2e073cc62f15
94b0adcc37 rpc, refactor: Prevent potential race conditions in dumptxoutset (Fabian Jahr)
e868a6e070 doc: Improve assumeutxo guide and add more docs/comments (Fabian Jahr)
b29c21fc92 assumeutxo: Remove devtools/utxo_snapshot.sh (Fabian Jahr)
20a1c77aa7 contrib: Remove test_utxo_snapshots.sh (Fabian Jahr)
8426850352 test: Test for dumptxoutset at specific height (Fabian Jahr)
993cafe7e4 RPC: Add type parameter to dumptxoutset (Fabian Jahr)
fccf4f91d2 RPC: Extract ReconsiderBlock helper (Fabian Jahr)
446ce51c21 RPC: Extract InvalidateBlock helper (Fabian Jahr)
Pull request description:
This adds a height parameter to the `dumptxoutset` RPC. This internalizes the workflow that was previously done by scripts: roll back the chain to the height we actually want the snapshot from, create the snapshot, roll forward to the real tip again.
The nice thing about internalizing this functionality is that we can write tests for the code and it gives us more options to make the functionality robust. The shell scripts we have so far will be more cumbersome to maintain in the long run, especially since we will only notice later when we have broken them. I think it's safe to remove these `test_utxo_snapshots.sh` as well when we have this option in `dumptxoutset` because we have also added some good additional functional test coverage for this functionality.
ACKs for top commit:
Sjors:
re-utACK 94b0adcc37
achow101:
ACK 94b0adcc37
mzumsande:
ACK 94b0adcc37
pablomartin4btc:
re-ACK 94b0adcc37
Tree-SHA512: a4c9af5f687d1ca7bfb579a36f363882823386b5fa80c05de531b05a2782b5da6ff5baf3ada4bca8f32f63975d86f1948175abed9affe51fc958472b5f838dab
When the tracepoint was introduced in 8f37f5c2a5,
the connect_block duration was passed in microseconds `µs`.
By starting to use steady clock in fabf1cdb20
this changed to nanoseconds `ns`. As the test only checked if the
duration value is `> 0` as a plausibility check, this went unnoticed.
I detected this when setting up monitoring for block validation time
as part of the Great Consensus Cleanup Revival discussion.
This change casts the duration explicitly to nanoseconds (as it has been
nanoseconds for the last three releases; switching back now would 'break'
the broken API again; there don't seem to be many users affected), updates
the documentation and adds a check for an upper bound to the tracepoint
interface tests. The upper bound is quite lax as mining the block takes
much longer than connecting the empty test block. It's however able to
detect incorrect duration units passed.
This is used in both cases - TCP server (accept) and TCP client (connect).
The message "Connected & Listening address:port" is confusing.
Print both ends of the TCP connection.
fa247e6e8c test: Avoid intermittent timeout in p2p_headers_sync_with_minchainwork.py (MarcoFalke)
Pull request description:
Similar to https://github.com/bitcoin/bitcoin/pull/30705:
The goal of this test case is to check that the sync works at all, not to check any timeout.
On extremely slow hardware (for example qemu virtual hardware), downloading the 4110 BLOCKS_TO_MINE may take longer than the block download timeout.
Fix it by pinning the time using mocktime temporarily, and advance it immediately after the sync.
ACKs for top commit:
stratospher:
ACK fa247e6. Checked the timeout downloading block logs before/after using `setmocktime`.
tdb3:
ACK fa247e6e8c
Tree-SHA512: f61632a8d9e484f1b888aafbf87f7adf71b8692387bd77f603cdbc0de49f30d42e654741d46ae1ff8b9706a5559ee0faabdb192ed0db7449010b68bfcdbaa42d
fa09cb41f5 refactor: Remove unused LogPrint (MarcoFalke)
3333415890 scripted-diff: LogPrint -> LogDebug (MarcoFalke)
Pull request description:
`LogPrint` has many issues:
* It seems to indicate that something is being "printed", however config options such as `-printtoconsole` actually control what and where something is logged.
* It does not mention the log severity (debug).
* It is a deprecated alias for `LogDebug`, according to the dev notes.
* It wastes review cycles, because reviewers sometimes point out that it is deprecated.
* It makes the code inconsistent, when both are used, possibly even in lines right next to each other (like in `InitHTTPServer`)
Fix all issues by removing the deprecated alias.
I checked all conflicting pull requests and at the time of writing there are no conflicts, except in pull requests that are marked as draft, are yet unreviewed, or are blocked on feedback for other reasons. So I think it is fine to do now.
ACKs for top commit:
stickies-v:
ACK fa09cb41f5
danielabrozzoni:
utACK fa09cb41f5
TheCharlatan:
ACK fa09cb41f5
Tree-SHA512: 14270f4cfa3906025a0b994cbb5b2e3c8c2427c0beb19c717a505a2ccbfb1fd1ecf2fd03f6c52d22cde69a8d057e50d2207119fab2c2bc8228db3f10d4288d0f
faa382ae76 ci, doc: Drop reference to `src/.bear-tidy-config` (Hennadii Stepanov)
d71ac76842 build: Remove Autotools-based build system (Hennadii Stepanov)
e268b48419 doc: Adjust `doc/design/libraries.md` (Hennadii Stepanov)
d209e4f156 doc: Drop mentions of `share/genbuild.sh` (Hennadii Stepanov)
Pull request description:
This PR deletes the Autotools-based build system.
The MSVC build system is deleted in https://github.com/bitcoin/bitcoin/pull/30731.
ACKs for top commit:
maflcko:
re-ACK faa382ae76🍦
TheCharlatan:
ACK faa382ae76
fanquake:
ACK faa382ae76
Tree-SHA512: 53df977b5b199a1c38f7f61a042a62b24831c559ba65a461b4ac1c96a1a56e2dfd676df79f1358fd1cc1749ff27e7b548086157f337d4f596c1054cb3d2d5739
bd7ce05f9d test: fix `TestShell` initialization (late follow-up for #30463) (Sebastian Falbesoner)
Pull request description:
Creating a `TestShell` instance as stated in the [docs](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test-shell.md) currently fails on master:
```
$ python3
Python 3.10.13 (main, Mar 15 2024, 07:36:23) [Clang 16.0.6 ] on openbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, "/home/thestack/bitcoin/test/functional")
>>> from test_framework.test_shell import TestShell
>>> test = TestShell().setup(num_nodes=2, setup_clean_chain=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/thestack/bitcoin/test/functional/test_framework/test_shell.py", line 70, in __new__
TestShell.instance = TestShell.__TestShell()
TypeError: BitcoinTestFramework.__init__() missing 1 required positional argument: 'test_file'
```
Since #30463, BitcoinTestFramework instances expect the path of the calling test at construction, in order to find shared data like the configuration (config.ini) and the cache. Note that in contrast to actual functional tests, we can't simply pass `__file__` here, as the test shell module sits within the `test_framework` subfolder, so we have to navigate up to the parent directory and append some dummy test file name.
On the long-term we should probably add some TestShell instantation smoke-test to detect issues like this early. As I'm not too familiar with the CI I'm not sure what is a good way to achieve this (a functional test obviously can't be used, as that's already a BitcoinTestFramework test in itself), but happy to take suggestions.
ACKs for top commit:
ismaelsadeeq:
Tested ACK bd7ce05f9d
danielabrozzoni:
tACK bd7ce05f9d
brunoerg:
ACK bd7ce05f9d
Tree-SHA512: c3a2365e2cda48a233ee724673c490787981354914f33e10eadbbad9c68e8403d84c5551229a611401e743886539de380ba4bfcb77032b6c85731e3bbe962dc1
a563f41232 Remove second node since only 1 is needed for the test (Martin Saposnic)
1f4cdb3d69 Replace custom funding tx creation with MiniWallet. (Martin Saposnic)
Pull request description:
In response to issue https://github.com/bitcoin/bitcoin/issues/30600, optimizations have been implemented to enhance test efficiency and readability:
This PR refactors the `rpc_signrawtransactionwithkey.py` functional test to use MiniWallet for creating funding transactions. This simplifies the test code and improves performance by eliminating the need to mine new blocks for each funding transaction.
Key changes:
- Replaced custom `send_to_address` method with MiniWallet's `send_to` method
- Removed unnecessary setup of a clean chain and second node
- Simplified transaction creation and signing process
ACKs for top commit:
glozow:
ACK a563f41232
ismaelsadeeq:
code review ACK a563f41232
theStack:
ACK a563f41232
Tree-SHA512: 318959f89702b169453d537dafb822f5ef1921db1088941d8bbdb3171dd7a6ecad590e57a3802bc37bcf8992267ed6ffa7f156b229d9817ebf812bd35df509b5
Previously they may have taken more than 10 seconds. Now they should
finish in less than one second.
This also allows to drop one dependency to be installed.
The rules have many issues:
* Most are redundant, because Python already has a built-in
IndentationError, a subclass of SyntaxError, to enforce whitespace.
* They are not enforced consistently anyway, see for examples [1][2]
below.
* They are stylistic rules where the author intentionally formatted the
code to be easier to read. Starting to enforce them now would make the
code harder to read and create frustration in the future.
Fix all issues by removing them.
[1]:
test/functional/feature_cltv.py:63:35: E272 [*] Multiple spaces before keyword
|
61 | # | Script to prepend to scriptSig | nSequence | nLockTime |
62 | # +-------------------------------------------------+------------+--------------+
63 | [[OP_CHECKLOCKTIMEVERIFY], None, None],
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E272
[2]:
contrib/asmap/asmap.py:395:13: E306 [*] Expected 1 blank line before a nested definition, found 0
|
393 | prefix.pop()
394 | hole = not fill and (lhole or rhole)
395 | def candidate(ctx: Optional[int], res0: Optional[list[ASNEntry]],
| ^^^ E306
Any kind of syntax error is already reported, so there is no need to
enumerate all possible types of syntax errors of ancient versions of
Python 2 or 3.
a0abcbd382 doc: Mention multipath specifier (Ava Chow)
0019f61fc5 tests: Test importing of multipath descriptors (Ava Chow)
f97d5c137d wallet, rpc: Allow importdescriptors to import multipath descriptors (Ava Chow)
32dcbca3fb rpc: Allow importmulti to import multipath descriptors correctly (Ava Chow)
64dfe3ce4b wallet: Move internal to be per key when importing (Ava Chow)
1692245525 tests: Multipath descriptors for scantxoutset and deriveaddresses (Ava Chow)
cddc0ba9a9 rpc: Have deriveaddresses derive receiving and change (Ava Chow)
360456cd22 tests: Multipath descriptors for getdescriptorinfo (Ava Chow)
a90eee444c tests: Add unit tests for multipath descriptors (Ava Chow)
1bbf46e2da descriptors: Change Parse to return vector of descriptors (Ava Chow)
0d640c6f02 descriptors: Have ParseKeypath handle multipath specifiers (Ava Chow)
a5f39b1034 descriptors: Change ParseScript to return vector of descriptors (Ava Chow)
0d55deae15 descriptors: Add DescriptorImpl::Clone (Ava Chow)
7e86541f72 descriptors: Add PubkeyProvider::Clone (Ava Chow)
Pull request description:
It is convenient to have a descriptor which specifies both receiving and change addresses in a single string. However, as discussed in https://github.com/bitcoin/bitcoin/issues/17190#issuecomment-895515768, it is not feasible to use a generic multipath specification like BIP 88 due to combinatorial blow up and that it would result in unexpected descriptors.
To resolve that problem, this PR proposes a targeted solution which allows only a single pair of 2 derivation indexes to be inserted in the place of a single derivation index. So instead of two descriptor `wpkh(xpub.../0/0/*)` and `wpkh(xpub.../0/1/*)` to represent receive and change addresses, this could be written as `wpkh(xpub.../0/<0;1>/*)`. The multipath specifier is of the form `<NUM;NUM>`. Each `NUM` can have its own hardened specifier, e.g. `<0;1h>` is valid. The multipath specifier can also only appear in one path index in the derivation path.
This results in the parser returning two descriptors. The first descriptor uses the first `NUM` in all pairs present, and the second uses the second `NUM`. In our implementation, if a multipath descriptor is not provided, a pair is still returned, but the second element is just `nullptr`.
The wallet will not output the multipath descriptors (yet). Furthermore, when a multipath descriptor is imported, it is expanded to the two descriptors and each imported on its own, with the second descriptor being implicitly for internal (change) addresses. There is no change to how the wallet stores or outputs descriptors (yet).
Note that the path specifier is different from what was proposed. It uses angle brackets and the semicolon because these are unused characters available in the character set and I wanted to avoid conflicts with characters already in use in descriptors.
Closes#17190
ACKs for top commit:
darosior:
re-ACK a0abcbd382
mjdietzx:
reACK a0abcbd382
pythcoiner:
reACK a0abcbd
furszy:
Code review ACK a0abcbd
glozow:
light code review ACK a0abcbd382
Tree-SHA512: 84ea40b3fd1b762194acd021cae018c2f09b98e595f5e87de5c832c265cfe8a6d0bc4dae25785392fa90db0f6301ddf9aea787980a29c74f81d04b711ac446c2
41051290ab cmake: Ignore build subdirectories within source directory (Hennadii Stepanov)
6ce50fd9d0 doc: Update for CMake-based build system (Hennadii Stepanov)
9730288a0c ci: Migrate CI scripts to CMake (Hennadii Stepanov)
c360837ca5 cmake, lint: Adjust `lint_includes_build_config` (Hennadii Stepanov)
3885441ee0 cmake: Add presets for native Windows builds (Hennadii Stepanov)
7681746b20 cmake: Add vcpkg manifest file (Hennadii Stepanov)
8b6f1c4353 cmake: Add `Coverage` and `CoverageFuzz` scripts (Hennadii Stepanov)
65bdbc1ff2 cmake: Add `docs` build target (Hennadii Stepanov)
fb75ebbc33 cmake: Add compiler diagnostic flags (Hennadii Stepanov)
e821f0a37a cmake: Migrate Guix build scripts to CMake (Hennadii Stepanov)
747adb6ffe cmake: Add `Maintenance` module (Hennadii Stepanov)
1f60b30df0 cmake: Add `APPEND_{CPP,C,CXX,LD}FLAGS` cache variables (Hennadii Stepanov)
2b43c45b13 cmake: Add `AddWindowsResources` module (Hennadii Stepanov)
973a3b0c5d cmake: Implement `install` build target (Hennadii Stepanov)
84ac35cfd4 cmake: Add cross-compiling support (Hennadii Stepanov)
0d01c228a7 build: Generate `toolchain.cmake` in depends (Hennadii Stepanov)
91a799247d depends: Add host-specific `cmake_system_version` variables (Hennadii Stepanov)
9b31209b4c depends: Rename `cmake_system` -> `cmake_system_name` (Hennadii Stepanov)
4a5208a81d Revert "build, qt: Do not install *.prl files" (Hennadii Stepanov)
6522af62af depends: Amend handling flags environment variables (Hennadii Stepanov)
90cec4d251 cmake: Add `MULTIPROCESS` option (Hennadii Stepanov)
bb1a450dcb cmake: Build `bitcoin-chainstate` executable (Hennadii Stepanov)
aed38ea58c cmake: Build `bitcoinkernel` library (Hennadii Stepanov)
975d67369b cmake: Build `test_bitcoin-qt` executable (Hennadii Stepanov)
10fcc668a3 cmake: Add `WITH_DBUS` option (Hennadii Stepanov)
5bb5a4bc75 cmake: Add `libqrencode` optional package support (Hennadii Stepanov)
57a6e2ef4a cmake: Build `bitcoin-qt` executable (Hennadii Stepanov)
30f642952c cmake: Add `WERROR` option (Hennadii Stepanov)
c98d4a4c34 cmake: Add `REDUCE_EXPORTS` option (Hennadii Stepanov)
a01cb6e63f cmake: Add `HARDENING` option (Hennadii Stepanov)
a8a2e364ac cmake: Add Python-based tests (Hennadii Stepanov)
3d85379570 cmake: Add fuzzing options (Hennadii Stepanov)
908530e312 cmake: Add `SANITIZERS` option (Hennadii Stepanov)
8bb0e85631 cmake: Build `bench_bitcoin` executable (Hennadii Stepanov)
801735163a cmake: Add external signer support (Hennadii Stepanov)
353e0c9e96 cmake: Add `systemtap-sdt` optional package support (Hennadii Stepanov)
d2fda82b49 cmake: Add `libzmq` optional package support (Hennadii Stepanov)
ae7b39a0e1 cmake: Add `libminiupnpc` optional package support (Hennadii Stepanov)
6480e1dcdb cmake: Add `libnatpmp` optional package support (Hennadii Stepanov)
e73e9304a1 cmake: Build `bitcoin-util` executable (Hennadii Stepanov)
027c6d7caa cmake: Build `bitcoin-tx` executable (Hennadii Stepanov)
d10c5c34c3 cmake: Add wallet functionality (Hennadii Stepanov)
ab2e99b0d9 cmake: Create test suite for `ctest` (Hennadii Stepanov)
959370bd76 cmake: Build `test_bitcoin` executable (Hennadii Stepanov)
b27bf9700d cmake: Build `bitcoin-cli` executable (Hennadii Stepanov)
a9813df826 cmake: Build `bitcoind` executable (Hennadii Stepanov)
97829ce2d5 cmake: Add `FindLibevent` module (Hennadii Stepanov)
3118e40c61 cmake: Build `bitcoin_consensus` library (Hennadii Stepanov)
809a2f1929 cmake: Build `bitcoin_util` static library (Hennadii Stepanov)
0a9a521a70 cmake: Build `bitcoin_crypto` library (Hennadii Stepanov)
958971f476 cmake: Build `univalue` static library (Hennadii Stepanov)
752747fda8 cmake: Generate `obj/build.h` header (Hennadii Stepanov)
1f0a78edf3 cmake: Build `minisketch` static library (Hennadii Stepanov)
12bfbc8154 cmake: Build `leveldb` static library (Hennadii Stepanov)
51985c5304 cmake: Build `crc32c` static library (Hennadii Stepanov)
db7a198f29 cmake: Build `secp256k1` subtree (Hennadii Stepanov)
dbb7ed14e8 cmake: Add `ccache` support (Hennadii Stepanov)
cedfdf6c72 cmake: Redefine/adjust per-configuration flags (Hennadii Stepanov)
b6b5e732c8 cmake: Add global compiler and linker flags (Hennadii Stepanov)
f98327931b cmake: Add `TryAppendLinkerFlag` module (Hennadii Stepanov)
4a0af29697 cmake: Add `TryAppendCXXFlags` module (Hennadii Stepanov)
35cffc497d cmake: Add POSIX threads support (Hennadii Stepanov)
fd72d00ffe cmake: Add position independent code support (Hennadii Stepanov)
07069e2bb0 cmake: Add introspection module (Hennadii Stepanov)
27d687fc1f cmake: Add `config/bitcoin-config.h` support (Hennadii Stepanov)
fe5cdace5f cmake: Print compiler and linker flags in summary (Hennadii Stepanov)
70683884c5 cmake: Introduce interface libraries to encapsulate common flags (Hennadii Stepanov)
a2317e27b7 cmake: Add root `CMakeLists.txt` file (Hennadii Stepanov)
Pull request description:
This PR introduces a new CMake-based build system, which is a drop-in replacement for the current Autotools-based build system.
ML announcement: https://groups.google.com/g/bitcoindev/c/hgKkfQWzrTo
As discussed during the recent CoreDev meetup in April, the switch from Autotools to CMake is intended to happen as soon as possible after branching 28.x off, which means that 29.0 will be built using CMake.
This PR branch is essentially the [staging branch](https://github.com/hebasto/bitcoin/tree/cmake-staging), with every change reviewed and tested by a group of contributors, including (in alphabetical order):
- [**achow101**](https://github.com/achow101)
- [**fanquake**](https://github.com/fanquake)
- [**maflcko**](https://github.com/maflcko)
- [**m3dwards**](https://github.com/m3dwards)
- [**pablomartin4btc**](https://github.com/pablomartin4btc)
- [**real-or-random**](https://github.com/real-or-random)
- [**ryanofsky**](https://github.com/ryanofsky)
- [**sipsorcery**](https://github.com/sipsorcery)
- [**TheCharlatan**](https://github.com/TheCharlatan)
- [**theStack**](https://github.com/theStack)
- [**theuni**](https://github.com/theuni)
- [**vasild**](https://github.com/vasild)
Reviewing in a separate staging repo was suggested in https://github.com/bitcoin/bitcoin/pull/27060#issuecomment-1431798320.
The accompanying changes to the OSS-Fuzz project are available in https://github.com/hebasto/oss-fuzz/pull/8.
Please refer to the [build options parity table](https://gist.github.com/hebasto/2ef97d3a726bfce08ded9df07f7dab5e). The "auto" value is no longer available; non-default values must be specified explicitly. Additionally, the new default values have been chosen to suit the everyday build experience for the majority of developers.
System requirements for using the CMake-based build system:
- CMake >= 3.22 (if not available in your system's repository, it can be downloaded from https://cmake.org/download/)
- a build tool of your choice:
- any Make (GNU Make is no longer a requirement); GNU Make is still required to build depends
- Ninja (https://ninja-build.org/)
- MSBuild
- Xcode
A note for Windows users: The default installation of the latest version of MSVC 17.10.4 includes both CMake 3.28.3 and the vcpkg package manager).
---
We, the build system developers, kindly ask reviewers to refrain from making suggestions that are not directly related to the migration process or can be implemented separately. Bugs in the scripts and errors in the updated documentation should be the focus of this PR. Please be advised that comments not aligned with this PR's goal may be ignored.
Thank you all for your understanding.
ACKs for top commit:
maflcko:
review ACK 41051290ab🐥
sipsorcery:
ACK 41051290ab.
vasild:
ACK 41051290ab
TheCharlatan:
ACK 41051290ab
pablomartin4btc:
tACK 41051290ab
i-am-yuvi:
tACK [`4105129`](41051290ab)
theuni:
ACK 41051290ab.
fanquake:
ACK 41051290ab
Tree-SHA512: 6c1445054436c6c00ad63bfa0f19d64091a2b25c9bd694f85bf2218ac358ffb774d6c000685b3ca1e9b50401babed989fa2a0694b774c211d226bfd1944c9b39
setup_clean_chain=True is deleted so it uses the default.
Also, vout is now returned from send_to_address,
so now there is no need to fetch it manually
Also remove not-needed code that was used with the old
transaction handling.
18d65d2772 test: use uint256::FromUserHex for RANDOM_CTX_SEED (stickies-v)
6819e5a329 node: use uint256::FromUserHex for -assumevalid parsing (stickies-v)
2e58fdb544 util: remove unused IsHexNumber (stickies-v)
8a44d7d3c1 node: use uint256::FromUserHex for -minimumchainwork parsing (stickies-v)
70e2c87737 refactor: add uint256::FromUserHex helper (stickies-v)
85b7cbfcbe test: unittest chainstatemanager_args (stickies-v)
Pull request description:
Since fad2991ba0, `uint256S` has been [deprecated](fad2991ba0 (diff-800776e2dda39116e889839f69409571a5d397de048a141da7e4003bc099e3e2R138)) because it is less robust than the `base_blob::FromHex()` introduced in [the same PR](https://github.com/bitcoin/bitcoin/pull/30482). Specifically, it tries to recover from length-mismatches, recover from untrimmed whitespace, 0x-prefix and garbage at the end, instead of simply requiring exactly 64 hex-only characters. _(see also #30532)_
This PR carves out the few `uint256S` callsites that may potentially prove a bit more controversial to change because they deal with user input and backwards incompatible behaviour change.
The main behaviour change introduced in this PR is:
- `-minimumchainwork` will raise an error when input is longer than 64 hex digits
- `-assumevalid` will raise an error when input contains invalid hex characters, or when it is longer than 64 hex digits
- test: the optional RANDOM_CTX_SEED env var will now cause tests to abort when it contains invalid hex characters, or when it is longer than 64 hex digits
After this PR, the remaining work to remove `uint256S` completely is almost entirely mechanical and/or test related. I will open that PR once #30560 is merged because it builds on that.
ACKs for top commit:
hodlinator:
re-ACK 18d65d2772
l0rinc:
ACK 18d65d2772
achow101:
ACK 18d65d2772
ryanofsky:
Code review ACK 18d65d2772. Very nice change that cleans up the API, adds checking for invalid values, makes parsing of values more consistent, and adds test coverage.
Tree-SHA512: ec118ea3d56e1dfbc4c79acdbfc797f65c4d2107b0ca9577c848b4ab9b7cb8d05db9f3c7fe8441a09291aca41bf671572431f4eddc59be8fb53abbea76bbbf86
fa5b58ea01 test: Avoid intermittent block download timeout in p2p_ibd_stalling (MarcoFalke)
Pull request description:
Fixes#30704
The goal of the test is to check the stalling timeout, not the block download timeout.
On extremely slow hardware (for example qemu virtual hardware), downloading the 1023 blocks may take longer than the block download timeout.
Fix it by pinning the time using mocktime, and only advance it when testing the stalling timeout.
ACKs for top commit:
tdb3:
CR ACK fa5b58ea01
brunoerg:
utACK fa5b58ea01
Tree-SHA512: 9a9221f264bea52be5e9fe81fd319f5a6970cd315cc5e9f5e2e049c5d84619b19b9f6f075cda8d34565c2d6c17a88fb57e195c66c271e40f73119a77caecb6d7
Because we don't have type checking for command-line/settings/config
args, strings are interpreted as 'false' for non-boolean args.
By convention, this "forces" us to interpret negated strings as 'true',
which conflicts with the negated option definition in all the settings
classes (they expect negated options to always be false and ignore any
other value preceding them). Consequently, when retrieving all "wallet"
values from the command-line/settings/config, we also fetch the negated
string boolean value, which is not of the expected 'string' type.
This mismatch leads to an internal fatal error, resulting in an unclean
shutdown during initialization. Furthermore, this error displays a poorly
descriptive error message:
"JSON value of type bool is not of expected type string"
This commit fixes the fatal error by ensuring that only string values are
returned in the "wallet" settings list, failing otherwise. It also improves
the clarity of the returned error message.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
e1d5dd732d test: check xor.dat recreated when missing (tdb3)
d1610962bf test: add null block xor key (tdb3)
1ad999b9da refactor: lift NUM_XOR_BYTES (tdb3)
d8399584dd refactor: move read_xor_key() to TestNode (tdb3)
d43948c3ef refactor: use unlink rather than os.remove (tdb3)
c8176f758b test: add blocks_key_path (tdb3)
Pull request description:
Builds on PR #30657.
Refactors `read_xor_key()` from `util.py` to `test_node.py` (comment https://github.com/bitcoin/bitcoin/pull/30657#discussion_r1723358327)
Adds a check that `xor.dat` is created when missing (comment https://github.com/bitcoin/bitcoin/pull/30657#discussion_r1717724161)
Help states:
```
-blocksxor
Whether an XOR-key applies to blocksdir *.dat files. The created XOR-key
will be zeros for an existing blocksdir or when `-blocksxor=0` is
set, and random for a freshly initialized blocksdir. (default: 1)
```
ACKs for top commit:
maflcko:
ACK e1d5dd732d
achow101:
ACK e1d5dd732d
theStack:
re-ACK e1d5dd732d
brunoerg:
reACK e1d5dd732d
Tree-SHA512: 325912ef646ec88e0a58e1ece263a2b04cbc06497e8fe5fcd603e509e80c6bcf84b09dd52dfac60e23013f07fc2b2f6db851ed0598649c3593f452c4a1424bd9
fa5aeab3cb test: Avoid duplicate curl call in get_previous_releases.py (MarcoFalke)
Pull request description:
Seems odd having to translate `404` to "Binary tag was not found". Also, it seems odd to write a for-loop over a list with one item.
Fix both issues by just using a single call to `curl --fail ...`.
Can be tested with: `test/get_previous_releases.py -b v99.99.99`
Before:
```
Releases directory: releases
Fetching: https://bitcoincore.org/bin/bitcoin-core-99.99.99/bitcoin-99.99.99-x86_64-linux-gnu.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 286k 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
Binary tag was not found
```
After:
```
Releases directory: releases
Fetching: https://bitcoincore.org/bin/bitcoin-core-99.99.99/bitcoin-99.99.99-x86_64-linux-gnu.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 286k 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
curl: (22) The requested URL returned error: 404
ACKs for top commit:
fanquake:
ACK fa5aeab3cb
brunoerg:
utACK fa5aeab3cb
tdb3:
tested ACK fa5aeab3cb
Tree-SHA512: d5d31e0bccdd9de9b4a8ecf2e69348f4e8cee773050c8259b61db1ce5de73f6fbfffbe8c4d2571f7bef2de29cb42fd244573deebfbec614e487e76ef41681b9c
Removes dependency on unsafe and deprecated uint256S.
This makes parsing more strict, by returning an error
when the input contains non-hex characters, or when it
contains more than 64 hex digits.
Also make feature_assumevalid.py more robust by using CBlock.hash
which is guaranteed to be 64 characters long, as opposed to the
variable-length hex(CBlock.sha256)
Removes dependency on unsafe and deprecated uint256S.
This makes parsing more strict, by returning an error
when the input contains more than 64 hex digits.
cccc5bfd35 test: Enable detect_leaks=1 in ASAN_OPTIONS explicitly (MarcoFalke)
Pull request description:
It should be enabled by default, but being explicit can't hurt.
ACKs for top commit:
fanquake:
ACK cccc5bfd35
Tree-SHA512: ed284abd05c7a99c30b509844aa75785a5ccb506d8296a71347b4c328750a6a4ed1f87e7a3ec36ab17f27b467c033cc8ca5eb5e2b951f2ae7473327c5eb1ddae
59ff17e5af miner: adjust clock to timewarp rule (Sjors Provoost)
e929054e12 Add timewarp attack mitigation test (Sjors Provoost)
e85f386c4b consensus: enable BIP94 on regtest (Sjors Provoost)
dd154b0568 consensus: lower regtest nPowTargetTimespan to 144 (Sjors Provoost)
Pull request description:
Because #30647 reduced the timewarp attack threshold from 7200s to 600s, our miner code will fail to propose a block template (on testnet4) if the last block of the previous period has a timestamp two hours in the future. This PR fixes that and also adds a test.
The non-test changes in the last commit should be in v28, otherwise miners have to patch it themselves. If necessary I can split that out into a separate PR, but I prefer to get the tests in as well.
In order to add the test, we activate BIP94 on regtest.
In order for the test to run faster, we reduce its difficulty retarget period to 144, the same number that's already used for softfork activation logic. Regtest does not actually adjust its difficulty, so this change has no effect (except for `getnetworkhashps`, see commit).
An alternative approach would be to run this test on testnet4, by hardcoding its first 2015 in the test suite. But since the timewarp mitigation is a serious candidate for a future mainnet softfork, it seems better to just deploy it on regtest.
The next commits add a test and fix the miner code.
The `MAX_TIMEWARP` constant is moved to `consensus.h` so both validation and miner code have access to it.
ACKs for top commit:
achow101:
ACK 59ff17e5af
fjahr:
ACK 59ff17e5af
glozow:
ACK 59ff17e5af
Tree-SHA512: 50af9fdcba9b0d5c57e1efd5feffd870bd11b5318f1f8b0aabf684657f2d33ab108d5f00b1475fe0d38e8e0badc97249ef8dda20c7f47fcc1698bc1008798830
917e70a620 test: assumeutxo: check that UTXO-querying RPCs operate on snapshot chainstate (Sebastian Falbesoner)
Pull request description:
Inspired by some manual testing I did for #28553, this PR checks that RPCs which explicitly query the UTXO set database (i.e. `gettxoutsetinfo`, `scantxoutset` and `gettxout`) operate on the snapshot chainstate as expected.
ACKs for top commit:
fjahr:
utACK 917e70a620
achow101:
ACK 917e70a620
tdb3:
ACK 917e70a620
Tree-SHA512: 40ecd1c5dd879234df1667fa5444a1fbbee9b7c456f597dc982d1a2bce46fe9107711b005ab829e570ef919a4914792f72f342d71d92bad2ae9434b5e68d5bd3
This currently has no effect due to fPowNoRetargeting,
except for the getnetworkhashps when called with -1.
It will when the next commit enforces the timewarp attack mitigation on regtest.
Previous rpcauth behavior was to sometimes
ignore empty -rpcauth= settings, and other times
treat them as errors.
Empty rpcauth is now consistently treated
as an error and prevents bitcoind from starting.
Updates associated test cases.
Also updates to non-deprecated logging macro.
Co-Authored-By: Luke Dashjr <luke-jr+git@utopios.org>
Co-Authored-By: Ryan Ofsky <ryan@ofsky.org>
77ff0ec1f1 contrib: support reading XORed blocks in linearize-data.py script (Sebastian Falbesoner)
Pull request description:
This PR is a small follow-up for #28052, adding support for the block linearization script to handle XORed blocksdir *.dat files. Note that if no xor.dat file exists, the XOR pattern is set to all-zeros, in order to still support blockdirs that have been created with versions earlier than 28.x.
Partly fixes issue #30599.
ACKs for top commit:
achow101:
ACK 77ff0ec1f1
tdb3:
ACK 77ff0ec1f1
hodlinator:
ACK 77ff0ec1f1
Tree-SHA512: 011eb02e2411de373cbbf4b26db4640fc693a20be8c2430529fba6e36a3a3abfdfdc3b005d330f9ec2846bfad9bfbf34231c574ba99289ef37dd51a68e6e7f3d
6b2dcba076 wallet: List sqlite wallets with empty string name (Ava Chow)
3ddbdd1815 wallet: Ignore .bak files when listing wallet files (Ava Chow)
Pull request description:
When the default wallet is migrated, we do not rename the wallet so we end up having a descriptor wallet with the empty string as its name and the wallet.dat file in the root of the walletdir. This is supposed to be an unsupported configuration and there is no other way to achieve this (other than file copying), but the wallet loading code does not disallow loading such wallets. However `listwalletdir` does not currently list the default wallet if it is sqlite. This is confusing to users, so change `listwalletdir` to include these wallets.
Additionally, the migration of the default wallet, and of any plain wallet files in the walletdir, produces a backup file in the walletdir itself. Since these backups are a BDB file, `listwalletdir` will detect them as being another wallet that we could open, but this is erroneous and could lead to confusion and potentially funds loss if both the backup and the migrated wallet are in use simultaneously. To reduce the likelihood of this issue, don't list these wallets in `listwalletdir`.
***
Possibly we could have more stringent checks on loading to resolve these issues, but I'm concerned that that will just confuse users and gratuitously break things that already worked.
Since the original intent was to disallow default wallets for sqlite/descriptors, a possible alternative would be to prevent people from loading such wallets and change migration to rename those wallets. However, given that this behavior with migrating default wallets has existed since default wallet migration was fixed, I think that making such a change would be confusing and break things for no good reason. Although perhaps we should still do the renaming.
For the backups, we could also change loading to refuse to load any wallet named with `.bak` (or `.legacy.bak`) as such wallets can still be loaded by giving the path to them directly, which some users may do to "restore" the backup. However restricting what can be loaded based on filename seems a little heavyhanded. It wouldn't be funds loss though since the correct way to restore the backup is with `restorewallet`.
ACKs for top commit:
fjahr:
Code review ACK 6b2dcba076
furszy:
Code ACK 6b2dcba076
glozow:
ACK 6b2dcba076
Tree-SHA512: 0b033f6ed55830f8a054afea3fb2cf1fa82a94040053ebfaf123bda36c99f45d3f01a2aec4ed02fed9c61bb3d320b047ed892d7f6644b5a356a7bc5974b10cff
00618e8745 assumeutxo: Drop block height from metadata (Fabian Jahr)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/30514 which has more context and shows how the issue can be reproduced. Since the value in question is removed, there is no test to add to reproduce anything.
This is an alternative approach to #30516 with much of the [code being suggested there](https://github.com/bitcoin/bitcoin/pull/30516#discussion_r1689146902).
ACKs for top commit:
maflcko:
re-ACK 00618e8745🎌
achow101:
ACK 00618e8745
theStack:
Code-review ACK 00618e8745
ismaelsadeeq:
Re-ACK 00618e8745
mzumsande:
ACK 00618e8745
Tree-SHA512: db9575247bae838ad7742a27a216faaf55bb11e022f9afdd05752bb09bbf9614717d0ad64304ff5722a16bf41d8dea888af544e4ae26dcaa528c1add0269a4a8
Although it is not explicitly possible to create a default wallet with
descriptors, it is possible to migrate a default wallet and have it end
up being a default wallet with descriptors. These wallets should be
listed by ListDatabases so that it appears in wallet directory listings
to avoid user confusion.
Migration creates backup files in the wallet directory with .bak as the
extension. This pollutes the output of listwalletdir with backup files
that most users should not need to care about.
6bfa26048d testnet: Add timewarp attack prevention for Testnet4 (Fabian Jahr)
0100907ca1 testnet: Add Testnet4 difficulty adjustment rules fix (Fabian Jahr)
74a04f9e7a testnet: Introduce Testnet4 (Fabian Jahr)
Pull request description:
To supplement the [ongoing conceptual discussion about a testnet reset](https://groups.google.com/g/bitcoindev/c/9bL00vRj7OU/m/9yCPo3uUBwAJ) I have drafted a move to v4 including a fix to the difficulty adjustment mechanism, which was part of the motivation that started the discussion.
Conceptual considerations:
- The conceptual discussion about doing a testnet4 or softforking the fix into testnet3 is outside of the scope of this PR and I would ask reviewers to contribute their opinions on this on the ML instead. However, I am happy to adapt this PR to a softfork change on testnet3 if there is consensus for that instead.
- The difficulty adjustment fix suggested here touches the `CalculateNextWorkRequired` function and uses the same logic used in `GetNextWorkRequired` to find the last previous block that was not mined with difficulty 1 under the exceptionf. An alternative fix briefly mentioned on the mailing list by Jameson Lopp would be to "restrict the special testnet minimum difficulty rule so that it can't be triggered on the block right before a difficulty retarget". That would also fix the issue but I find my suggestion here a bit more elegant.
ACKs for top commit:
jsarenik:
tACK 6bfa26048d
achow101:
ACK 6bfa26048d
murchandamus:
tACK 6bfa26048d
Tree-SHA512: 0b8b69a621406a944da5be551b863d065358ba94d85dd3b80d83c412660e230ee93b27316081fbee9b4851cc4ff8585db64c7dfa26cb5148ac835663f2712c3d
Refactor satoshi_round function to accept different rounding modes.
Updated call site to use the revised `satoshi_round` function.
Co-authored-by: Kate Salazar <52637275+katesalazar@users.noreply.github.com>
e9de0a76b9 doc: release note for 30212 (willcl-ark)
87b1880525 rpc: clarify ALREADY_IN_CHAIN rpc errors (willcl-ark)
Pull request description:
Closes: #19363
Renaming this error improves clarity around the returned error both internally and externally when a transactions' outputs are already found in the utxo set (`TransactionError::ALREADY_IN_CHAIN -> TransactionError::ALREADY_IN_UTXO_SET`)
ACKs for top commit:
tdb3:
ACK e9de0a76b9
ismaelsadeeq:
ACK e9de0a76b9
ryanofsky:
Code review ACK e9de0a76b9.
Tree-SHA512: 7d2617200909790340951fe56a241448f9ce511900777cb2a712e8b9c0778a27d1f912b460f82335844224f1abb4322bc898ca076440959edade55c082a09237
fa895c7283 mingw: Document mode wbx workaround (MarcoFalke)
fa359255fe Add -blocksxor boolean option (MarcoFalke)
fa7f7ac040 Return XOR AutoFile from BlockManager::Open*File() (MarcoFalke)
Pull request description:
Currently the *.dat files in the blocksdir store the data received from remote peers as-is. This may be problematic when a program other than Bitcoin Core tries to interpret them by accident. For example, an anti-virus program or other program may scan them and move them into quarantine, or delete them, or corrupt them. This may cause Bitcoin Core to fail a reorg, or fail to reply to block requests (via P2P, RPC, REST, ...).
Fix this, similar to https://github.com/bitcoin/bitcoin/pull/6650, by rolling a random XOR pattern over the dat files when writing or reading them.
Obviously this can only protect against programs that accidentally and unintentionally are trying to mess with the dat files. Any program that intentionally wants to mess with the dat files can still trivially do so.
The XOR pattern is only applied when the blocksdir is freshly created, and there is an option to disable it (on creation), so that people can disable it, if needed.
ACKs for top commit:
achow101:
ACK fa895c7283
TheCharlatan:
Re-ACK fa895c7283
hodlinator:
ACK fa895c7283
Tree-SHA512: c92a6a717da83bc33a9b8671a779eeefde2c63b192362ba1d71e6535ee31d08e2802b74acc908345197de9daac6930e4771595ee25b09acd5a67f7ea34854720
fa530ec543 rpc: Return precise loadtxoutset error messages (MarcoFalke)
faa5c86dbf refactor: Use untranslated error message in ActivateSnapshot (MarcoFalke)
Pull request description:
The error messages should never happen in normal operation. However, if
they do, they are helpful to return to the user to debug the issue. For
example, to notice a truncated file.
This fixes https://github.com/bitcoin/bitcoin/issues/28621
Also includes a minor refactor commit.
ACKs for top commit:
fjahr:
Code review ACK fa530ec543
ryanofsky:
Code review ACK fa530ec543, just adjusting error messages a little since last review. (Thanks!)
Tree-SHA512: 224968c9b13d082ca2ed1f6a8fcc5f51ff16d6c96bd38c3679699505b54337b99cccaf7a8474391f6b11f9ccb101977b4e626898c1217eae95802e290cf105f1
When using `sendrawtransaction` the ALREADY_IN_CHAIN error help string
may be confusing.
Rename TransactionError::ALREADY_IN_CHAIN to
TransactionError::ALREADY_IN_UTXO_SET and update the rpc help string.
Remove backwards compatibility alias as no longer required.
fa3ea3b83c test: Fix intermittent issue in p2p_v2_misbehaving.py (MarcoFalke)
55555574d1 net: Log accepted connection after m_nodes.push_back (MarcoFalke)
Pull request description:
Fix the two issues reported in https://github.com/bitcoin/bitcoin/pull/30468/files#r1688444784:
* Delay a debug log line for consistency.
* Fix an intermittent test issue.
They are completely separate fixes, but both `net` related.
ACKs for top commit:
0xB10C:
Code Review ACK fa3ea3b83c
stratospher:
tested ACK fa3ea3b.
Tree-SHA512: cd6b6e164b317058a305a5c3e38c56c9a814a7469039e1143f1d7addfbc91b0a28506873356b373d97448b46cb6fbe94a1309df82e34c855540b241a09489e8b
75648cea5a test: add P2A ProduceSignature coverage (Greg Sanders)
7998ce6b20 Add release note for P2A output feature (Greg Sanders)
71c9b02a04 test: add P2A coverage for decodescript (Greg Sanders)
1349e9ec15 test: Add anchor mempool acceptance test (Greg Sanders)
9d89209937 policy: stop 3rd party wtxid malleability of anchor spend (Greg Sanders)
b60aaf8b23 policy: make anchor spend standard (Greg Sanders)
455fca86cf policy: Add OP_1 <0x4e73> as a standard output type (Greg Sanders)
Pull request description:
This is a sub-feature taken out of the original proposal for ephemeral anchors #30239
This PR makes *spending* of `OP_1 <0x4e73>` (i.e. `bc1pfeessrawgf`) standard. Creation of this output type is already standard.
Any future witness output types are considered relay-standard to create, but not to spend. This preserves upgrade hooks, such as a completely new output type for a softfork such as BIP341. It also gives us a bit of room to use a new output type for policy uses.
This particular sized witness program has no other known use-cases (https://bitcoin.stackexchange.com/a/110664/17078), s it affords insufficient cryptographic security for a secure commitment to data, such as a script or a public key. This makes this type of output "keyless", or unauthenticated.
As a witness program, the `scriptSig` of the input MUST be blank, by BIP141. This helps ensure txid-stability of the spending transaction, which may be required for smart contracting wallets. If we do not use segwit, a miner can simply insert an `OP_NOP` in the `scriptSig` without effecting the result of program execution.
An additional relay restriction is to disallow non-empty witness data, which an adversary may use to penalize the "honest" transactor when RBF'ing the transaction due to the incremental fee requirement of RBF rules.
The intended use-case for this output type is to "anchor" the transaction with a spending child to bring exogenous CPFP fees into the transaction package, encouraging the inclusion of the package in a block. The minimal size of creation and spending of this output makes it an attractive contrast to outputs like `p2sh(OP_TRUE)` and `p2wsh(OP_TRUE)` which
are significantly larger in vbyte terms.
Combined with TRUC transactions which limits the size of child transactions significantly, this is an attractive option for presigned transactions that need to be fee-bumped after the fact.
ACKs for top commit:
sdaftuar:
utACK 75648cea5a
theStack:
re-ACK 75648cea5a
ismaelsadeeq:
re-ACK 75648cea5a via [diff](e7ce6dc070..75648cea5a)
glozow:
ACK 75648cea5a
tdb3:
ACK 75648cea5a
Tree-SHA512: d529de23d20857e6cdb40fa611d0446b49989eaafed06c28280e8fd1897f1ed8d89a4eabbec1bbf8df3d319910066c3dbbba5a70a87ff0b2967d5205db32ad1e
fa46a1b74b test: Avoid CScript() as default function argument (MarcoFalke)
fadf621825 test: Make leaf_script mandatory when scriptpath is set in TaprootSignatureMsg (MarcoFalke)
Pull request description:
Unlike other function calls in default arguments, CScript should not cause any issues in the tests, because they are const.
However, this change allows to enable the "function-call-in-default-argument (B008)" lint rule, which will help to catch severe test bugs, such as https://github.com/bitcoin/bitcoin/issues/30543#issuecomment-2259260024 .
The lint rule will be enabled in a follow-up, when all violations are fixed.
ACKs for top commit:
instagibbs:
utACK fa46a1b74b
theStack:
lgtm ACK fa46a1b74b
ismaelsadeeq:
Tested ACK fa46a1b74b
Tree-SHA512: bc68b15121d50ead0fc70ad772360a7829908aedeaff8426efcb8a67f33117f67d26b4f5da94fa735dd8de9c9ff65fc10a29323f1b12f238b75486fa7cc32a89
afd237bb5d [fuzz] Harness for version handshake (dergoegge)
a90ab4aec9 scripted-diff: Rename lazily initialized bloom filters (dergoegge)
82de1bc478 [net processing] Lazily initialize m_recent_confirmed_transactions (dergoegge)
fa0c87f19c [net processing] Lazily initialize m_recent_rejects_reconsiderable (dergoegge)
662e8db2d3 [net processing] Lazily initialize m_recent_rejects (dergoegge)
Pull request description:
This adds a fuzzing harness dedicated to the version handshake. To avoid determinism issues, the harness creates necessary components each iteration (addrman, peerman, etc). A harness like this would have easily caught https://bitcoincore.org/en/2024/07/03/disclose-timestamp-overflow/.
As a performance optimization, this PR includes a change to `PeerManager` to lazily initialize various filters (to avoid large unnecessary memory allocations each iteration).
ACKs for top commit:
brunoerg:
ACK afd237bb5d
marcofleon:
Tested ACK afd237bb5d. I compared the coverage of `net_processing` from this harness to the `process_message` and `process_messages` harnesses to see the differences. This target hits more specific parts of the version handshake. The stability looks good as well, at about 94%.
glozow:
utACK afd237bb5d lazy blooms look ok
mzumsande:
Code Review ACK afd237bb5d
Tree-SHA512: 62bba20aec0cd220e62368354891f9790b81ad75e8adf7b22a76a6d4663bd26aedc4cae8083658a75ea9043d60aad3f0e58ad36bd7bbbf93ff1d16e317bf15cc
This does not cause any issues, because CScript in the tests are const.
However, this change allows to enable the
"function-call-in-default-argument (B008)" lint rule.
a6efc7e16e test: fix intermittent failures in feature_proxy.py (Martin Zumsande)
Pull request description:
Fixes#29871
If addnode connections are made with v2transport and the peer immediately disconnects us, reconnections with v1 are scheduled. This could interfere with later checks depending on timing. Avoid this by using `v2transport=False` in the addnode rpc - this test isn't about the message layer anyway, so running it with v2 would add no value.
ACKs for top commit:
maflcko:
ACK a6efc7e16e
tdb3:
cr re ACK a6efc7e16e
Tree-SHA512: 39353a392e75e4c6257d971ceecb65fb76ec6d3b121a087869831c24b767a18f57e2ae2968da445c7fa731cb03053c90df37dd2cd6e86f786ad4121bc68ca235
This removes the default value, because there should not be a use-case
to fall back to a an empty leaf_script by default. (If there was, it
could trivially be added back)
In python, if the default value is a mutable object (here: a class)
its shared over all instances, so that one instance being changed
would affect others to be changed as well.
This was likely the source of various intermittent bugs in the
functional tests.
If addnode connections are made with v2transport and the peer immediately disconnects us, reconnections
with v1 are scheduled. This could interfere with later checks depending on timing. Avoid this by using
`v2transport=False` in the addnode rpc - this test isn't about the message layer anyway, so running it
with v2 would add no value.
17845e7f21 rpc: add utxo's blockhash and number of confirmations to scantxoutset output (Luis Schwab)
Pull request description:
This PR resolves#30478 by adding two fields to the `scantxoutset` RPC:
- blockhash: the blockhash that an UTXO was created
- confirmations: the number of confirmations an UTXO has relative to the chaintip.
The rationale for the first field is that a blockhash is a much more reliable identifier than the height:
> When using the scantxoutset RPC, the current behaviour is to show the block height of the UTXO. This is not optimal, as block height is ambiguous, especially in the case of a block reorganization happening at the same instant of the query. In this case, an UTXO that does not exist would be assumed to exist, unless the chain's tip hash is recorded before the scan, and make sure it still exists after, as per https://github.com/bitcoindevkit/bdk/issues/895#issuecomment-1475766797 comment by evanlinjin.
The second one was suggested by maflcko, and I agree it's useful for human users:
> While touching this, another thing to add could be the number of confirmations? I understand that this wouldn't help machine consumers of the interface, but human callers may find it useful?
This will yield an RPC output like so:
```diff
bitcoin-cli scantxoutset start "[\"addr(bc1q5q9344vdyjkcgv79ve3tldz4jmx4lf7knmnx6r)\"]"
{
"success": true,
"txouts": 185259116,
"height": 853622,
"bestblock": "00000000000000000002e97d9be8f0ddf31829cf873061b938c10b0f80f708b2",
"unspents": [
{
"txid": "fae435084345fe26e464994aebc6544875bca0b897bf4ce52a65901ae28ace92",
"vout": 0,
"scriptPubKey": "0014a00b1ad58d24ad8433c56662bfb45596cd5fa7d6",
"desc": "addr(bc1q5q9344vdyjkcgv79ve3tldz4jmx4lf7knmnx6r)#smk4xmt7",
"amount": 0.00091190,
"coinbase": false,
"height": 852741,
+ "blockhash": "00000000000000000002eefe7e7db44d5619c3dace4c65f3fdcd2913d4945c13",
+ "confirmations": 882
}
],
"total_amount": 0.00091190
}
```
ACKs for top commit:
sipa:
utACK 17845e7f21
Eunovo:
ACK 17845e7f21
tdb3:
ACK 17845e7f21
Tree-SHA512: 02366d0004e5d547522115ef0efe6794a35978db53dda12c675cfae38197bf43f0bf89ca99a3d79e3d2cff95186015fe1ab764abb8ab82bda440ae9302ad973b
The error messages should never happen in normal operation. However, if
they do, they are helpful to return to the user to debug the issue. For
example, to notice a truncated file.
e4b0dabb21 test: add functional test for tagged MiniWallet instances (Sebastian Falbesoner)
3162c917e9 test: fix MiniWallet internal key derivation for tagged instances (Sebastian Falbesoner)
c9f7364ab2 test: fix MiniWallet script-path spend (missing parity bit in leaf version) (Sebastian Falbesoner)
7774c314fb test: refactor: return TaprootInfo from P2TR address creation routine (Sebastian Falbesoner)
Pull request description:
This PR fixes a dormant bug in MiniWallet that exists since support for P2TR was initially added in #23371 (see commit 041abfebe4).
In the course of spending the output, the leaf version byte of the control block in the witness stack doesn't set the parity bit, i.e. we were so far just lucky that the used combinations of relevant data (internal pubkey, leaf script / version) didn't result in a tweaked pubkey with odd y-parity. If that was the case, we'd get the following validation error:
`mandatory-script-verify-flag-failed (Witness program hash mismatch) (-26)`
Since MiniWallets can now optionally be tagged (#29939), resulting in different internal pubkeys, the issue is more prevalent now. Fix it by passing the parity bit, as specified in BIP341.
Can be tested with the following patch (fails on master, succeeds on PR):
```diff
diff --git a/test/functional/test_framework/mempool_util.py b/test/functional/test_framework/mempool_util.py
index 148cc935ed..7ebe858681 100644
--- a/test/functional/test_framework/mempool_util.py
+++ b/test/functional/test_framework/mempool_util.py
@@ -42,7 +42,7 @@ def fill_mempool(test_framework, node):
# Generate UTXOs to flood the mempool
# 1 to create a tx initially that will be evicted from the mempool later
# 75 transactions each with a fee rate higher than the previous one
- ephemeral_miniwallet = MiniWallet(node, tag_name="fill_mempool_ephemeral_wallet")
+ ephemeral_miniwallet = MiniWallet(node, tag_name="fill_mempool_ephemeral_wallet3")
test_framework.generate(ephemeral_miniwallet, 1 + num_of_batches * tx_batch_size)
# Mine enough blocks so that the UTXOs are allowed to be spent
```
In addition to that, another bug is fixed where the internal key derivation failed, as not every pseudorandom hash results in a valid x-only pubkey. Fix this by treating the hash result as private key and calculate the x-only public key out of that, to be used then as internal key.
Fixes#30528.
ACKs for top commit:
glozow:
ACK e4b0dabb21
rkrux:
reACK [e4b0dab](e4b0dabb21)
hodlinator:
ACK e4b0dabb21
Tree-SHA512: a16f33f76bcb1012857cc3129438a9f6badf28aa2b1d25696da0d385ba5866b46de0f1f93ba777ed9263fe6952f98d7d9c44ea0c0170a2bcc86cbef90bf6ac58
fac0c3d4bf doc: Add release notes for two pull requests (MarcoFalke)
fa7b57e5f5 refactor: Replace ParseHashStr with FromHex (MarcoFalke)
fa90777245 rest: Reject truncated hex txid early in getutxos parsing (MarcoFalke)
fab6ddbee6 refactor: Expose FromHex in transaction_identifier (MarcoFalke)
fad2991ba0 refactor: Implement strict uint256::FromHex() (MarcoFalke)
fa103db2bb scripted-diff: Rename SetHex to SetHexDeprecated (MarcoFalke)
fafe4b8051 test: refactor: Replace SetHex with uint256 constructor directly (MarcoFalke)
Pull request description:
In `rest_getutxos` truncated txids such as `aa` or `ff` are accepted. This is brittle at best.
Fix it by rejecting any truncated (or overlarge) input.
----
Review note: This also starts a major refactor to rework hex parsing in Bitcoin Core, meaning that a few refactor commits are included as well. They are explained individually in the commit message and the work will be continued in the future.
ACKs for top commit:
stickies-v:
re-ACK fac0c3d4bf - only doc and test updates to address review comments, thanks!
hodlinator:
ACK fac0c3d4bf
Tree-SHA512: 473feb3fcf6118443435d1dd321006135b0b54689bfbbcb1697bb5811a449bef51f475c715de6911ff3c4ea3bdb75f601861ff93347bc4414d6b9e5298105dd7
25bf86a225 [test]: ensure `estimatesmartfee` default mode is `economical` (ismaelsadeeq)
41a2545046 [fees]: change `estimatesmartfee` default mode to `economical` (ismaelsadeeq)
Pull request description:
Fixes#30009
This PR changes the `estimatesmartfee` default mode to `economical`.
This was also suggested on IRC https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2024-04-26#1021609
- `conservative` mode: This is the `estimatesmartfee` RPC mode which considers a longer history of blocks. It potentially returns a higher fee rate and is more likely to be sufficient for the desired target, but it is not as responsive to short-term drops in the prevailing fee market.
- `economical` mode: This is the `estimatesmartfee` RPC mode where estimates are potentially lower and more responsive to short-term drops in the prevailing fee market.
Since users are likely to use the default mode, this change will reduce overestimation for many users. The conservative mode remains available for those who wish to opt-in.
For an in-depth analysis of how significantly the `conservative` mode overestimates, see
https://delvingbitcoin.org/t/bitcoind-policy-estimator-modes-analysis/964.
ACKs for top commit:
instagibbs:
reACK 25bf86a225
glozow:
ACK 25bf86a225
willcl-ark:
ACK 25bf86a225
Tree-SHA512: 78ebda667eb9c8f87dcc2f0e6c14968bd1de30358dc77a13611b186fb8427ad97d9f537bad6e32e0a1aa477ccd8c64fee4d41e19308ef3cb184ff1664e6ba8a6
Without the fix, the test could fail intermittently. For example:
node0 2024-07-22T16:31:54.104994Z [httpworker.0] [rpc/request.cpp:232] [parse] [rpc] ThreadRPCServer method=setmocktime user=__cookie__
test 2024-07-22T16:31:54.291000Z TestFramework (INFO): Sending first 4 bytes of ellswift which match network magic
test 2024-07-22T16:31:54.292000Z TestFramework (INFO): If a response is received, assertion failure would happen in our custom data_received() function
test 2024-07-22T16:31:54.292000Z TestFramework.p2p (DEBUG): Connecting to Bitcoin Node: 127.0.0.1:12644
test 2024-07-22T16:31:54.293000Z TestFramework.p2p (DEBUG): Connected & Listening: 127.0.0.1:12644
test 2024-07-22T16:31:54.588000Z TestFramework.p2p (DEBUG): sending 4050 bytes of garbage data
test 2024-07-22T16:31:54.588000Z TestFramework (INFO): Sending remaining ellswift and garbage which are different from V1_PREFIX. Since a response is
test 2024-07-22T16:31:54.588000Z TestFramework (INFO): expected now, our custom data_received() function wouldn't result in assertion failure
node0 2024-07-22T16:31:55.523868Z (mocktime: 2024-07-22T16:31:54Z) [net] [net.cpp:3764] [CNode] [net] Added connection peer=0
node0 2024-07-22T16:31:55.625145Z (mocktime: 2024-07-22T16:31:54Z) [net] [net.cpp:1814] [CreateNodeFromAcceptedSocket] [net] connection from 127.0.0.1:45154 accepted
node0 2024-07-22T16:31:55.625769Z (mocktime: 2024-07-22T16:31:54Z) [http] [httpserver.cpp:305] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:33320
node0 2024-07-22T16:31:55.626543Z (mocktime: 2024-07-22T16:31:54Z) [httpworker.1] [rpc/request.cpp:232] [parse] [rpc] ThreadRPCServer method=getpeerinfo user=__cookie__
test 2024-07-22T16:31:55.818000Z TestFramework (ERROR): Unexpected exception caught during testing
Traceback (most recent call last):
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
self.run_test()
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_v2_misbehaving.py", line 133, in run_test
self.test_earlykeyresponse()
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_v2_misbehaving.py", line 151, in test_earlykeyresponse
self.wait_until(lambda: node0.getpeerinfo()[-1]["bytesrecv"] > 4)
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 791, in wait_until
return wait_until_helper_internal(test_function, timeout=timeout, timeout_factor=self.options.timeout_factor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 289, in wait_until_helper_internal
if predicate():
^^^^^^^^^^^
File "/ci_container_base/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_v2_misbehaving.py", line 151, in <lambda>
self.wait_until(lambda: node0.getpeerinfo()[-1]["bytesrecv"] > 4)
~~~~~~~~~~~~~~~~~~~^^^^
IndexError: list index out of range
d63ef73800 test: Add loadtxoutset test with tip on snapshot block (Fabian Jahr)
c2f86d4bcb test: Remove already resolved assumeutxo todo comments (Fabian Jahr)
Pull request description:
The first commit removes three Todos that have been addressed previously (see commit message for details).
The second message resolves another todo by adding the missing test case. This is a special case of "the tip has more work than the snapshot" where the tip is the same block as the snapshot base block.
Related to #28648.
ACKs for top commit:
jrakibi:
ACK [d63ef73](d63ef73800)
achow101:
ACK d63ef73800
maflcko:
ACK d63ef73800
alfonsoromanz:
Re ACK d63ef73800
Tree-SHA512: 8d5a25fc0b26531db3a9740132694138f2103b7b42eeb1d4a64095bfc901c1372e23601c0855c7def84c8a4e185d10611e4e830c4e479f1b663ae6ed53abb130
a8e3af1a82 qa: Do not assume running `feature_asmap.py` from source directory (Hennadii Stepanov)
9bf7ca6cad qa: Consider `cache` and `config.ini` relative to invocation directory (Hennadii Stepanov)
a0473442d1 scripted-diff: Add `__file__` argument to `BitcoinTestFramework.init()` (Hennadii Stepanov)
Pull request description:
This PR includes changes split from https://github.com/bitcoin/bitcoin/pull/30454. They improve the functional test framework, allowing users to [run individual functional tests](https://github.com/hebasto/bitcoin/issues/146) from the build directory in the new CMake-based build system.
This functionality is not available for out-of-source builds using the current Autotools-based build system, which always requires write permissions for the source directory. Nevertheless, this PR can be tested as suggested in https://github.com/bitcoin/bitcoin/pull/30463#issuecomment-2232618421:
1. Make an out-of-source build:
```
$ ./autogen.sh
$ mkdir ../build && cd ../build
$ ../bitcoin/configure
$ make
```
2. Create a symlink in the build directory to a functional test:
```
$ ln --symbolic ../../../bitcoin/test/functional/wallet_disable.py ./test/functional/
```
3. Run this symlink:
```
$ ./test/functional/wallet_disable.py
```
The last command fails on the master branch:
```
Traceback (most recent call last):
File "/home/hebasto/git/build/./test/functional/wallet_disable.py", line 31, in <module>
DisableWalletTest().main()
^^^^^^^^^^^^^^^^^^^
File "/home/hebasto/git/bitcoin/test/functional/test_framework/test_framework.py", line 106, in __init__
self.parse_args()
File "/home/hebasto/git/bitcoin/test/functional/test_framework/test_framework.py", line 210, in parse_args
config.read_file(open(self.options.configfile))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/hebasto/git/bitcoin/test/config.ini'
```
and succeeds with this PR.
ACKs for top commit:
maflcko:
tested ACK a8e3af1a82🎨
glozow:
ACK a8e3af1a82, tested with the steps in op
stickies-v:
ACK a8e3af1a82
Tree-SHA512: 899e4efc09edec13ea3f5b47825d03173fb21d3569c360deda7fa6a56b99b4d24e09ad4f0883bad1ee926b1c706e47ba07c6a6160c63c07c82b3cf4ae5816e91
Also pulls out the guarding assert and calls it explicitly before the test function is called. This is already done before the existing call of the test function so it was not needed there.
- "Valid snapshot file, but referencing a snapshot block that turns out
to be invalid, or has an invalid parent" has been addressed in #30267
- "An ancestor of snapshot block" - If chain tip refers to blocks in this context then any successful load is addressing this because if we have synced past the snapshot base block we fail because we don't need assumeutxo anymore. And if this is about headers then this is the `test_headers_not_synced()` case.
- "A descendant of the snapshot block" - If this refers to blocks the
`test_snapshot_with_less_work()` addressed this and if it is just headers in this case again it would be represented in all of the successful loads in the test.
Co-authored-by: Alfonso Roman Zubeldia <alfonsoromanz24@gmail.com>
55b6d7be68 validation: Don't load a snapshot if it's not in the best header chain. (Martin Zumsande)
Pull request description:
This was suggested by me in the discussion of #30288, which has more context.
If the snapshot is not an ancestor of the most-work header (`m_best_header`), syncing from that alternative chain leading to `m_best_header` should be prioritised. Therefore it's not useful loading the snapshot in this situation.
If the other chain turns out to be invalid or the chain with the snapshot retrieves additional headers so that it's the most-work one again (see functional test), `m_best_header` will change and loading the snapshot will be possible again.
Because of the work required to generate a conflicting headers chain, a situation with two conflicting chains should only be possible under extreme circumstances, such as major forks.
ACKs for top commit:
fjahr:
re-ACK 55b6d7be68
achow101:
ACK 55b6d7be68
alfonsoromanz:
Re ACK 55b6d7be68
Tree-SHA512: 4fbea5ab1038ae353fc949a186041cf9b397e7ce4ac59ff36f881c9437b4f22ada922490ead5b2661389eb1ca0f3d1e7e7e6a4261057678643e71594a691ac36
fac932bf93 refactor: Use util::Split to avoid a harmless unsigned-integer-overflow (MarcoFalke)
fab54db9f1 rest: Reject negative outpoint index in getutxos parsing (MarcoFalke)
Pull request description:
In `rest_getutxos` outpoint indexes such as `+N` or `-N` are accepted. This should be harmless, because any index out of range should be treated as a non-existent utxo. However, a negative index can't exist ever, so it seems better to reject all signs, whether `+` or `-`.
ACKs for top commit:
achow101:
ACK fac932bf93
hodlinator:
ut-ACK fac932bf93
tdb3:
re ACK fac932bf93
danielabrozzoni:
ACK fac932bf93
brunoerg:
reACK fac932bf93
Tree-SHA512: 8f1a75248cb61e1c4beceded6ed170db83b07f30fbcf93a26acfffc00ec4546572366eff87907a7e1423d7d3a2a9e57a0a7a9bacb787c86463f842d7161c16bc
faed5d3870 test: Non-Shy version sender (MarcoFalke)
Pull request description:
After `add_outbound_p2p_connection`, the test framework normally sends a version message only in reply to a received version. This is fine, but the protocol does not require this and tolerates a version to be sent earlier.
However, this is untested, and the missing test coverage leads to bugs being missed. For example https://github.com/bitcoin/bitcoin/pull/30394#pullrequestreview-2166824948
Fix it by adding a test.
ACKs for top commit:
brunoerg:
ACK faed5d3870
tdb3:
ACK faed5d3870
theStack:
tACK faed5d3870
glozow:
ACK faed5d3870
Tree-SHA512: dbf527a39c932e994a1e8248ba78058000811a4bf69275278f1fd1e545716ac4d2d3be5dcf362976bbafa2a49f91d13e3601daf71d29e9c556179b01af62c03c
734076c6de [wallet, rpc]: add `max_tx_weight` to tx funding options (ismaelsadeeq)
b6fc5043c1 [wallet]: update the data type of `change_output_size`, `change_spend_size` and `tx_noinputs_size` to `int` (ismaelsadeeq)
baab0d2d43 [doc]: update reason for deducting change output weight (ismaelsadeeq)
7f61d31a5c [refactor]: update coin selection algorithms input parameter `max_weight` name (ismaelsadeeq)
Pull request description:
This PR taken over from #29264
The PR added an option `max_tx_weight` to transaction funding RPC's that ensures the resulting transaction weight does not exceed the specified `max_tx_weight` limit.
If `max_tx_weight` is not given `MAX_STANDARD_TX_WEIGHT` is used as the max threshold.
This PR addressed outstanding review comments in #29264
For more context and rationale behind this PR see https://delvingbitcoin.org/t/lightning-transactions-with-v3-and-ephemeral-anchors/418/11?u=instagibbs
ACKs for top commit:
achow101:
ACK 734076c6de
furszy:
utACK 734076c6de
rkrux:
reACK [734076c](734076c6de)
Tree-SHA512: 013501aa443d239ee2ac01bccfc5296490c27b4edebe5cfca6b96c842375e895e5cfeb5424e82e359be581460f8be92095855763a62779a18ccd5bdfdd7ddce7
c322bddd08 test: bump mocktime after node has received and sent bytes (stratospher)
Pull request description:
Fixes an intermittent failure for `p2p_v2_misbehaving.py` reported in https://github.com/bitcoin/bitcoin/pull/30420#discussion_r1680462164.
A [different error message](262260ce1e/src/net.cpp (L1970)) `"socket no message in first %i seconds"` will be displayed if `m_last_send=0` or if `m_last_recv is 0`. Fix this by:
1. mocktime bump is done after all the bytes are received. (`m_last_recv is not 0 now`)
2. wait until bytes are sent by `TestNode`/`bitcoind` (`m_last_send is not 0 now`)
See https://cirrus-ci.com/task/5359619151757312?logs=ci#L3935 for an example failure (I wasn't able to reproduce the intermittent failure locally but I think the fix is logical)
ACKs for top commit:
maflcko:
reACK c322bddd08
Tree-SHA512: 1c05524c2819041eb2001c2baf2c912d4f812a39347f784f212634e8c53131357a73116a46b4b7542bc7fc8c1370c4d36fc9898a2cbdb40bcee61105123c4a35
fa6390df20 doc: getaddressinfo[isscript] is optional (MarcoFalke)
Pull request description:
`isscript` is unknown for unknown witness versions, so it should be marked optional in the docs
Fixes https://github.com/bitcoin/bitcoin/issues/30456
ACKs for top commit:
stickies-v:
ACK fa6390df20
tdb3:
ACK fa6390df20
Tree-SHA512: f728f18e0871923225e0bf29594f8095997456cf55409f42087b5f70f95bef10f984323b48d2b484b6705f23b04e9e8a3fe42446830638fdd70453c18fd7f189
a different error message "socket no message in first %i seconds"
will be displayed if m_last_send=0 or if m_last_recv is 0. make
the test robust by ensuring that they will not be 0 before
bumping mocktime.
7e36dca657 test: add test for modififed walletprocesspsbt calls (willcl-ark)
39cea21ec5 wallet: fix FillPSBT errantly showing as complete (willcl-ark)
Pull request description:
Fixes: #30077
Fix cases of calls to `FillPSBT` returning `complete=true` when it's not
the case.
This can happen when some inputs have been signed but the transaction is
subsequently modified, e.g. in the context of PayJoins.
Also fixes a related bug where a finalized hex string is attempted to be
added during `walletprocesspsbt` but a CHECK_NONFATAL causes an abort.
ACKs for top commit:
achow101:
ACK 7e36dca657
ismaelsadeeq:
Tested ACK 7e36dca657
pinheadmz:
re-ACK 7e36dca657
Tree-SHA512: e35d19789899c543866d86d513506494d672e4bed9aa36a995dbec4e72f0a8ec5536b57c4a940a18002ae4a8efd0b007c77ba64e57cd52af98e4ac0e7bf650d6
In CMake-based build system (1) `config.ini` is created in the build
directory, and (2) `cache` must also be created in the same directory.
This change enables running individual functional tests from the build
directory.
bca346a970 net: require P2P binds to succeed (Vasil Dimov)
af552534ab net: report an error if unable to bind on the Tor listening addr:port (Vasil Dimov)
9a7e5f4d68 net: don't extra bind for Tor if binds are restricted (Vasil Dimov)
Pull request description:
Make it possible to disable the Tor binding on `127.0.0.1:8334` and stop startup if any P2P bind fails instead of "if all P2P binds fail".
Fixes https://github.com/bitcoin/bitcoin/issues/22726
Fixes https://github.com/bitcoin/bitcoin/issues/22727
ACKs for top commit:
achow101:
ACK bca346a970
cbergqvist:
ACK bca346a970
pinheadmz:
ACK bca346a970
Tree-SHA512: fabef89a957191eea4f3e3b6109d2b8389f27ecc74440a920b0c10f31fff00a85bcfd1eb3c91826c7169c618f4de8a8d0a260e2caf40fd854f07ea9a980d8603
fa6270737e rpc: Use CHECK_NONFATAL over Assert (MarcoFalke)
Pull request description:
Any RPC method should not abort the whole node when an internal logic error happens.
Fix it by just aborting this single RPC method call when an error happens.
Also, fix the linter to find the fixed cases.
ACKs for top commit:
achow101:
ACK fa6270737e
stickies-v:
ACK fa6270737e
tdb3:
ACK fa6270737e
hodlinator:
ACK fa6270737e
Tree-SHA512: dad2f31b01a66578949009499e4385fb4d72f0f897419f2a6e0ea02e799b9a31e6ecb5a67fa5d27fcbc7939fe8acd62dc04e877b35831493b7f2c604dec7dc64
16bd283b3a Reapply "test: p2p: check that connecting to ourself leads to disconnect" (Sebastian Falbesoner)
0dbcd4c148 net: prevent sending messages in `NetEventsInterface::InitializeNode` (Sebastian Falbesoner)
66673f1c13 net: fix race condition in self-connect detection (Sebastian Falbesoner)
Pull request description:
This PR fixes a recently discovered race condition in the self-connect detection (see #30362 and #30368).
Initiating an outbound network connection currently involves the following steps after the socket connection is established (see [`CConnman::OpenNetworkConnection`](bd5d1688b4/src/net.cpp (L2923-L2930)) method):
1. set up node state
2. queue VERSION message (both steps 1 and 2 happen in [`InitializeNode`](bd5d1688b4/src/net_processing.cpp (L1662-L1683)))
3. add new node to vector `m_nodes`
If we connect to ourself, it can happen that the sent VERSION message (step 2) is received and processed locally *before* the node object is added to the connection manager's `m_nodes` vector (step 3). In this case, the self-connect remains undiscovered, as the detection doesn't find the outbound peer in `m_nodes` yet (see `CConnman::CheckIncomingNonce`).
Fix this by swapping the order of 2. and 3., by taking the `PushNodeVersion` call out of `InitializeNode` and doing that in the `SendMessages` method instead, which is only called for `CNode` instances in `m_nodes`.
The temporarily reverted test introduced in #30362 is readded. Fixes#30368.
Thanks go to vasild, mzumsande and dergoegge for suggestions on how to fix this (see https://github.com/bitcoin/bitcoin/issues/30368#issuecomment-2200625017 ff. and https://github.com/bitcoin/bitcoin/pull/30394#discussion_r1668290789).
ACKs for top commit:
naiyoma:
tested ACK [16bd283b3a), built and tested locally, test passes successfully.
mzumsande:
ACK 16bd283b3a
tdb3:
ACK 16bd283b3a
glozow:
ACK 16bd283b3a
dergoegge:
ACK 16bd283b3a
Tree-SHA512: 5b8aced6cda8deb38d4cd3fe4980b8af505d37ffa0925afaa734c5d81efe9d490dc48a42e1d0d45dd2961c0e1172a3d5b6582ae9a2d642f2592a17fbdc184445
c6d43367a1 test: Fix intermittent failure in p2p_v2_misbehaving.py (stratospher)
Pull request description:
Fixes#30419.
Make sure that ellswift computation is complete in the `NetworkThread` in `test/functional/p2p_v2_misbehaving.py` before sending the ellswift in the `MainThread`.
One way to reproduce this failure on master would be:
```diff
diff --git a/test/functional/test_framework/v2_p2p.py b/test/functional/test_framework/v2_p2p.py
index 87600c36de..ea0615ef3b 100644
--- a/test/functional/test_framework/v2_p2p.py
+++ b/test/functional/test_framework/v2_p2p.py
@@ -111,6 +111,7 @@ class EncryptedP2PState:
def generate_keypair_and_garbage(self, garbage_len=None):
"""Generates ellswift keypair and 4095 bytes garbage at max"""
+ import time; time.sleep(3)
self.privkey_ours, self.ellswift_ours = ellswift_create()
if garbage_len is None:
garbage_len = random.randrange(MAX_GARBAGE_LEN + 1)
```
ACKs for top commit:
maflcko:
ACK c6d43367a1
mzumsande:
Code Review ACK c6d43367a1
tdb3:
cr and t ACK c6d43367a1
Tree-SHA512: dfc3a6afa09773b7e84d35aff0aa14e0b8a4475860e0b31ab5c1a8d54911c814f07138f624fea651fba90cc5c526c0d05c3fe33d2ce0ad833b2be3a3caa9f522
The ellswift bytes are computed in the NetworkThread and sent in
the MainThread. Add a `wait_until()` to make sure that ellswift
computation is completed in NetworkThread before sending it in
the MainThread. Also wait until bytes sent are actually received
and use mocktime for more robust disconnection checking.
00b8e26bd6 test: fix inconsistency in fundrawtransaction weight limits test (furszy)
Pull request description:
Fix https://github.com/bitcoin/bitcoin/pull/30309#discussion_r1657628378 inconsistency.
Currently, the test is passing due to a mistake in the test inputs
selection process. We are selecting the parent transaction change
output as one of the inputs of the transaction to fund, which
helps to surpass the target amount when it shouldn't due to the
fee reduction.
The failure arises when the test behaves as intended by its coder;
that is, when it does not select the change output. In this case,
the pre-selected inputs aren't enough to cover the target amount.
Fix this by excluding the parent transaction's change output from
the inputs selection and including an extra input to cover the tx
fee.
The CI failure can be replicated with the following patch in master:
```diff
diff --git a/test/functional/wallet_fundrawtransaction.py b/test/functional/wallet_fundrawtransaction.py
--- a/test/functional/wallet_fundrawtransaction.py(revision 9b480f7a25)
+++ b/test/functional/wallet_fundrawtransaction.py(date 1720652934739)
@@ -1322,7 +1322,7 @@
outputs = []
for _ in range(1472):
outputs.append({wallet.getnewaddress(address_type="legacy"): 0.1})
- txid = self.nodes[0].send(outputs=outputs)["txid"]
+ txid = self.nodes[0].send(outputs=outputs, change_position=0)["txid"]
self.generate(self.nodes[0], 1)
# 272 WU per input (273 when high-s); picking 1471 inputs will exceed the max standard tx weight.
@@ -1330,7 +1330,7 @@
# 1) Try to fund transaction only using the preset inputs
input_weights = []
- for i in range(1471):
+ for i in range(1, 1472): # skip first output as it is the parent tx change output
input_weights.append({"txid": txid, "vout": i, "weight": 273})
assert_raises_rpc_error(-4, "Transaction too large", wallet.fundrawtransaction, hexstring=rawtx, input_weights=input_weights)
```
ACKs for top commit:
achow101:
ACK 00b8e26bd6
ismaelsadeeq:
Code review and Tested ACK 00b8e26bd6
Tree-SHA512: 5ef792961b7fad4999fc30aa03366432103ddf672ca5cbb366c9eab4c2e46d5ae1ab0c073dfc4fbb2b4e63203653bc0e54463c731c5f8655140207ba5f8e542e
If the snapshot is not an ancestor of the most-work header (m_best_header),
syncing from that alternative chain should be prioritised.
Therefore don't accept loading a snapshot in this situation.
If that other chain turns out to be invalid, m_best_header
would be reset and loading the snapshot should be possible again.
Because of the work required to generate a conflicting headers chain,
this should only be possible under extreme circumstances, such as major forks.
8ce3739edb test: verify wallet is still active post-migration failure (furszy)
771bc60f13 wallet: Use LegacyDataSPKM when loading (Ava Chow)
61d872f1b3 wallet: Move MigrateToDescriptor and DeleteRecords to LegacyDataSPKM (Ava Chow)
b231f4d556 wallet: Move LegacyScriptPubKeyMan::IsMine to LegacyDataSPKM (Ava Chow)
7461d0c006 wallet: Move LegacySPKM data storage and handling to LegacyDataSPKM (Ava Chow)
517e204bac Change MigrateLegacyToDescriptor to reopen wallet as BERKELEY_RO (Ava Chow)
Pull request description:
#26606 introduced `BerkeleyRODatabase` which is an independent parser for BDB files. This PR uses this in legacy wallet migration so that migration will continue to work once the legacy wallet and BDB are removed. `LegacyDataSPKM` is introduced to have the minimum data and functions necessary for a legacy wallet to be loaded for migration.
ACKs for top commit:
cbergqvist:
ACK 8ce3739edb
theStack:
Code-review ACK 8ce3739edb
furszy:
Code review ACK 8ce3739edb
Tree-SHA512: dccea12d6c597de15e3e42f97ab483cfd069e103611200279a177e021e8e9c4e74387c4f45d2e58b3a1e7e2bdb32a1d2d2060b1f8086c03eeaa0c68579d9d54e
Currently, the test is passing due to a mistake in the test inputs
selection process. We are selecting the parent transaction change
output as one of the inputs of the transaction to fund, which
helps to surpass the target amount when it shouldn't due to the
fee reduction.
The failure arises when the test behaves as intended by its coder;
that is, when it does not select the change output. In this case,
the pre-selected inputs aren't enough to cover the target amount.
Fix this by excluding the parent transaction's change output from
the inputs selection and including an extra input to cover the tx
fee.
8789dc8f31 doc: Add note to getblockfrompeer on missing undo data (Fabian Jahr)
4a1975008b rpc: Make pruneheight also reflect undo data presence (Fabian Jahr)
96b4facc91 refactor, blockstorage: Generalize GetFirstStoredBlock (Fabian Jahr)
Pull request description:
The function `GetFirstStoredBlock()` helps us find the first block for which we have data. So far this function only looked for a block with `BLOCK_HAVE_DATA`. However, this doesn't mean that we also have the undo data of that block, and undo data might be required for what a user would like to do with those blocks. One example of how this might happen is if some blocks were fetched using the `getblockfrompeer` RPC. Blocks fetched from a peer will have data but no undo data.
The first commit here allows `GetFirstStoredBlock()` to check for undo data as well by passing a parameter. This alone is useful for #29553 and I would use it there.
In the second commit I am applying the undo check to the RPCs that report `pruneheight` to the user. I find this much more intuitive because I think the user expects to be able to do all operations on blocks up until the `pruneheight` but that is not the case if undo data is missing. I personally ran into this once before and now again when testing for assumeutxo when I had used `getblockfrompeer`. The following commit adds test coverage for this change of behavior.
The last commit adds a note in the docs of `getblockfrompeer` that undo data will not be available.
ACKs for top commit:
achow101:
ACK 8789dc8f31
furszy:
Code review ACK 8789dc8f31.
stickies-v:
ACK 8789dc8f31
Tree-SHA512: 90ae8bdd07a496ade579aa25240609c61c9ed173ad38d30533f6c631fe674e5a41727478ade69ca4b71a571ad94c9da4b33ebba6b5d8821109313c2de3bdfb3d
5b7f70ba26 test: loadtxoutset in divergent chain with less work (Alfonso Roman Zubeldia)
d35efe1efc p2p: Start downloading historical blocks from common ancestor (Martin Zumsande)
Pull request description:
This PR adds a test to cover the scenario of loading an assumeutxo snapshot when the current chain tip is not an ancestor of the snapshot block but has less work.
During the review process, a bug was discovered where blocks between the last common ancestor and the background tip were not being requested if the background tip was not an ancestor of the snapshot block. mzumsande suggested a fix (65343ec49a6b73c4197dfc38e1c2f433b0a3838a) to start downloading historical blocks from the last common ancestor to address this issue. This fix has been incorporated into the PR with a slight modification.
Related to https://github.com/bitcoin/bitcoin/issues/28648
ACKs for top commit:
fjahr:
tACK 5b7f70ba26
achow101:
ACK 5b7f70ba26
mzumsande:
Code Review ACK 5b7f70ba26
Tree-SHA512: f8957349686a6a1292165ea9e0fd8c912d21466072632a10f8ef9d852a5f430bc6b2a531e6884a4dbf2e3adb28b3d512b25919e78f5804a67320ef54c3b1aaf6
d93b794709 tests: improve wallet multisig descriptor test and docs (Michael Dietz)
Pull request description:
It is best to store all key origin information
(master key fingerprint and all derivation steps)
in the multisig descriptor. Being explicit with
this information should be beneficial if this approach is used with other wallets/signers (whether hardware or software). There is no harm including all of this with xpubs (if anything it simplifies the test code) and makes this example/docs more complete and safer incase it is referenced by others.
ACKs for top commit:
S3RK:
Code Review ACK d93b794709
achow101:
ACK d93b794709
Tree-SHA512: 0e5c4d13f060489405e6cf50c8a09911f5a0cee71023649235afd80a5e3aae38d52c6e12ad4660205b9357b09f45596941391bdcf6fceccbe07c4e5a1592a482
c9dacd958d test: Check that non empty version packet is ignored and no disconnection happens (stratospher)
997cc00b95 test: Check that disconnection happens when AAD isn't filled (stratospher)
b5e6238fdb test: Check that disconnection happens when garbage sent/received are different (stratospher)
ad1482d5a2 test: Check that disconnection happens when wrong garbage terminator is sent (stratospher)
e351576862 test: Check that disconnection happens when >4095 garbage bytes is sent (stratospher)
e075fd131d test: Introduce test types and modify v2 handshake function accordingly (stratospher)
7d07daa623 log: Add V2 handshake timeout (stratospher)
d4a1da8543 test: Make global TRANSPORT_VERSION variable an instance variable (stratospher)
c642b08c4e test: Log when the garbage is actually sent to transport layer (stratospher)
86cca2cba2 test: Support disconnect waiting for add_p2p_connection (stratospher)
bf9669af9c test: Rename early key response test and move random_bitflip to util (stratospher)
Pull request description:
Add tests for the following v2 handshake scenarios:
1. Disconnection happens when > `MAX_GARBAGE_LEN` bytes garbage is sent
2. Disconnection happens when incorrect garbage terminator is sent
3. Disconnection happens when garbage bytes are tampered with
4. Disconnection happens when AAD of first encrypted packet after the garbage terminator is not filled
5. bitcoind ignores non-empty version packet and no disconnection happens
All these tests require a modified v2 P2P class (different from `EncryptedP2PState` used in `v2_p2p.py`) to implement our custom handshake behaviour based on different scenarios and have been kept in a single test file (`test/functional/p2p_v2_misbehaving.py`). Shifted the test in `test/functional/p2p_v2_earlykeyresponse.py` which is of the same pattern to this file too.
ACKs for top commit:
achow101:
ACK c9dacd958d
mzumsande:
ACK c9dacd958d
theStack:
Code-review ACK c9dacd958d
Tree-SHA512: 90df81f0c7f4ecf0a47762d290a618ded92cde9f83d3ef3cc70e1b005ecb16125ec39a9d80ce95f99e695d29abd63443240cb5490aa57c5bc8fa2e52149a0672
de71d4dece fuzz: improve utxo_snapshot target (Martin Zumsande)
Pull request description:
Add the possibility of giving more guidance to the creation of the metadata and/or coins, so that the fuzzer gets the chance
to reach more error conditions in ActivateSnapshot and sometimes successfully creates a valid snapshot.
This also changes the asserts for the success case that were outdated (after #29370) and only didn't result in a crash because the fuzzer wasn't able to reach this code before.
ACKs for top commit:
maflcko:
re-ACK de71d4dece🎆
fjahr:
utACK de71d4dece
TheCharlatan:
ACK de71d4dece
Tree-SHA512: 346974d594164544d8cd3df7d8362c905fd93116215e9f5df308dfdac55bab04d727bfd7fd001cf11318682d11ee329b4b4a43308124c04d64b67840ab8a58a0
9e13ccc50e psbt: Check non witness utxo outpoint early (Ava Chow)
Pull request description:
A common issue that our fuzzers keep finding is that outpoints don't exist in the non witness utxos. Instead of trying to track this down and checking in various individual places, do the check early during deserialization. This also unifies the error message returned for this class of problems.
ACKs for top commit:
maflcko:
lgtm ACK 9e13ccc50e
S3RK:
tACK 9e13ccc50e
dergoegge:
utACK 9e13ccc50e
Tree-SHA512: 81b8055b146c6358052226578ddfec0ae5bd877968c7f4f62dc3d6a684545ea568f37c7f1bd619918441af9e453ba8b26531a2280d218da37fa15480f1b45d0e
Add the possibility of giving more guidance to the creation of the
metadata and/or coins, so that the fuzzer gets the chance
to reach more error conditions in ActivateSnapshot and sometimes
successfully creates a valid snapshot.
This also changes the asserts for the success case that were outdated,
and only didn't result in a crash because the fuzzer wasn't able
to reach this code before.
ce8094246e random: replace construct/assign with explicit Reseed() (Pieter Wuille)
2ae392d561 random: use LogError for init failure (Pieter Wuille)
97e16f5704 tests: make fuzz tests (mostly) deterministic with fixed seed (Pieter Wuille)
2c91330dd6 random: cleanup order, comments, static (Pieter Wuille)
8e31cf9c9b net, net_processing: use existing RNG objects more (Pieter Wuille)
d5fcbe966b random: improve precision of MakeExponentiallyDistributed (Pieter Wuille)
cfb0dfe2cf random: convert GetExponentialRand into rand_exp_duration (Pieter Wuille)
4eaa239dc3 random: convert GetRand{Micros,Millis} into randrange (Pieter Wuille)
82de1b80d9 net: use GetRandMicros for cache expiration (Pieter Wuille)
ddc184d999 random: get rid of GetRand by inlining (Pieter Wuille)
e2d1f84858 random: make GetRand() support entire range (incl. max) (Pieter Wuille)
810cdf6b4e tests: overhaul deterministic test randomness (Pieter Wuille)
6cfdc5b104 random: convert XoRoShiRo128PlusPlus into full RNG (Pieter Wuille)
8cc2f45065 random: move XoRoShiRo128PlusPlus into random module (Pieter Wuille)
8f5ac0d0b6 xoroshiro128plusplus: drop comment about nonexisting copy() (Pieter Wuille)
8924f5120f random: modernize XoRoShiRo128PlusPlus a bit (Pieter Wuille)
ddb7d26cfd random: add RandomMixin::randbits with compile-known bits (Pieter Wuille)
21ce9d8658 random: Improve RandomMixin::randbits (Pieter Wuille)
9b14d3d2da random: refactor: move rand* utilities to RandomMixin (Pieter Wuille)
40dd86fc3b random: use BasicByte concept in randbytes (Pieter Wuille)
27cefc7fd6 random: add a few noexcepts to FastRandomContext (Pieter Wuille)
b3b382dde2 random: move rand256() and randbytes() to .h file (Pieter Wuille)
493a2e024e random: write rand256() in function of fillrand() (Pieter Wuille)
Pull request description:
This PR contains a number of vaguely-related improvements to the random module.
The specific changes and more detailed rationale is in the commit messages, but the highlights are:
* `XoRoShiRo128PlusPlus` (previously a test-only RNG) moves to random.h and becomes `InsecureRandomContext`, which is even faster than `FastRandomContext` but non-cryptographic. It also gets all helper randomness functions (`randrange`, `fillrand`, ...), making it a lot more succinct to use.
* During tests, **all** randomness is made deterministic (except for `GetStrongRandBytes`) but non-repeating (like `GetRand()` used to be when `g_mock_deterministic_tests` was used), either fixed, or from a random seed (overridden by env var).
* Several infrequently used top-level functions (`GetRandMillis`, `GetRandMicros`, `GetExponentialRand`) are converted into member functions of `FastRandomContext` (and `InsecureRandomContext`).
* `GetRand<T>()` (without argument) can now return the maximum value of the type (previously e.g. `GetRand<uint32_t>()` would never return 0xffffffff).
ACKs for top commit:
achow101:
ACK ce8094246e
maflcko:
re-ACK ce8094246e🐈
hodlinator:
ACK ce8094246e
dergoegge:
utACK ce8094246e
Tree-SHA512: 79bc0cbafaf27e95012c1ce2947a8ca6f9a3c78af5f1f16e69354b6fc9b987a28858adf4cd356dc5baf21163e9af8dcc24e70f8d7173be870e8a3ddcdd47c02c
Updating to MLC v0.18.0 includes a new feature which will ignore all
files ignored by git: `--gitignore`.
This helps avoid false-positives flagged by this linter in non-project
files, such as a developer might expect to have in their directory (e.g.
guix-builds, python venvs, etc.)
This test checks that we can successfully process PSBTs and opt out of
finalization.
Previously trying to call `walletprocesspsbt` would attempt to
auto-finalize (as a convenience), and would not permit opt-out of
finalization, instead aborting via `CHECK_NONFATAL`.
2342b46c45 test: Add coverage for getchaintxstats in assumeutxo context (Fabian Jahr)
faf2a6750b rpc: Reorder getchaintxstats output (MarcoFalke)
fa2dada0c9 rpc: Avoid getchaintxstats invalid results (MarcoFalke)
Pull request description:
The `getchaintxstats` RPC reply during AU background download may return non-zero, but invalid, values for `window_tx_count` and `txrate`.
For example, `txcount` may be zero for a to-be-downloaded block, but may be non-zero for an ancestor block which is already downloaded. Thus, the values returned may be negative (and cause intermediate integer sanitizer violations).
Also, `txcount` may be accurate for the snapshot base block, or a descendant of it. However it may be zero for an ancestor block that still needs to be downloaded. Thus, the values returned may be positive, but wrong.
Fix all issues by skipping the returned value if either `txcount` is unset (equal to zero).
Also, skip `txcount` in the returned value, if it is unset (equal to zero).
Fixes https://github.com/bitcoin/bitcoin/issues/29328
ACKs for top commit:
fjahr:
re-ACK 2342b46c45
achow101:
ACK 2342b46c45
mzumsande:
ACK 2342b46c45
Tree-SHA512: 931cecc40ee5dc0f96be728db7eb297155f8343076cd29c8b8c050c99fd1d568b80f54c9459a34ca7a9489c2474c729796d00eeb1934d6a9f7b4d6a53e3ec430
926b8e39dc [doc] add release note for TRUC (glozow)
19a9b90617 use version=3 instead of v3 in debug strings (glozow)
881fac8e60 scripted-diff: change names from V3 to TRUC (glozow)
a573dd2617 [doc] replace mentions of v3 with TRUC (glozow)
089b5757df rename mempool_accept_v3.py to mempool_truc.py (glozow)
f543852a89 rename policy/v3_policy.* to policy/truc_policy.* (glozow)
Pull request description:
Adds a release note for TRUC policy which will be live in v28.0.
For clarity, replaces mentions of "v3" with "TRUC" in most places. Suggested in
- https://github.com/bitcoin/bitcoin/pull/29496#discussion_r1629749583
- https://github.com/bitcoin/bitcoin/pull/29496#discussion_r1624500904
I changed error strings from "v3-violation" to "TRUC-violation" but left v3 in the debug strings because I think it might be clearer for somebody who is debugging. Similarly, I left some variables unchanged because I think they're more descriptive this way, e.g. `tx_v3_from_v2_and_v3`. I'm happy to debate places that should or shouldn't be documented differently in this PR, whatever is clearest to everyone.
ACKs for top commit:
instagibbs:
reACK 926b8e39dc
achow101:
ACK 926b8e39dc
ismaelsadeeq:
Code review ACK 926b8e39dc
Tree-SHA512: 16c88add0a29dc6d1236c4d45f34a17b850f6727b231953cbd52eb9f7268d1d802563eadfc8b7928c94ed3d7a615275dd103e57e81439ebf3ba2b12efa1e42af
2f9bde69f4 test: Remove unnecessary restart in assumeutxo test (Fabian Jahr)
19ce3d407e assumeutxo: Check snapshot base block is not marked invalid (Fabian Jahr)
80315c0118 refactor: Move early loadtxoutset checks into ActiveSnapshot (Fabian Jahr)
Pull request description:
This was discovered in a discussion in #29996
If the base block of the snapshot is marked invalid or part of an invalid chain, we currently still load the snapshot and get stuck in a weird state where we have the snapshot chainstate but it will never connect to our valid chain.
While this scenario is highly unlikely to occur on mainnet, it still seems good to prevent this inconsistent state.
The behavior change described above is in the second commit.
The first commit refactors the early checks in the `loadtxoutset` RPC by moving them into `ActivateSnapshot()` in order to have the chance to cover them by unit tests in the future and have a more consistent interface. Previously checks were spread out between `rpc/blockchain.cpp` and `validation.cpp`. In order to be able to return the error message to users of the RPC, the return type of `ActivateSnapshot()` is changed from `bool` to `util::Result`.
The third commit removes an unnecessary restart introduced in #29428.
ACKs for top commit:
mzumsande:
re-ACK 2f9bde6
alfonsoromanz:
Re-ACK 2f9bde69f4. The RPC code looks much cleaner after the refactor. Also, it seems very useful to get the error message in the RPC response rather than having to rely on the logs in some scenarios if you are an RPC user.
achow101:
ACK 2f9bde69f4
Tree-SHA512: 5328dd88c3c7be3f1be97c9eef52ac3666c27188c30a798b3e949f3ffcb83be075127c107e4046f7f39f961a79911ea3d61b61f3c11e451b3e4c541c264eeed4
8ec24bdad8 test: Added coverage to Block not found error using gettxoutsetinfo (kevkevinpal)
Pull request description:
#### Description
There were no tests that checked for the `Block not found` error called in `ParseHashOrHeight` when using `gettxoutsetinfo`, this change adds coverage to it.
You can see there are no tests that do the following by doing the below
`grep -nri "Block not found.*gettxoutsetinfo" ./test/functional/`
which leads to no results
ACKs for top commit:
achow101:
ACK 8ec24bdad8
tdb3:
ACK 8ec24bdad8
kristapsk:
ACK 8ec24bdad8
brunoerg:
crACK 8ec24bdad8
alfonsoromanz:
Re ACK 8ec24bdad8
Tree-SHA512: 2c61c681e7304c679cc3d7dd13af1b795780e85716c25c7423d68104e253d01271e048e21bc21be35dbc7ec1a4fde94e439542f3cfd669fe5a16478c5fa982ab
e38eadb2c2 test: change comments to `self.log.info` for `test_addnode_getaddednodeinfo` (brunoerg)
c838e3b610 test: add coverage for `node` field of `getaddednodeinfo` RPC (brunoerg)
Pull request description:
We currently do not test a successful call to `getaddednodeinfo` filtering by `node`, we only test it with an unknown address and checks whether it fails. This PR adds coverage to it.
ACKs for top commit:
kevkevinpal:
ACK [e38eadb](e38eadb2c2)
achow101:
ACK e38eadb2c2
tdb3:
re ACK e38eadb2c2
BrandonOdiwuor:
Code Review ACK e38eadb2c2
rkrux:
tACK [e38eadb](e38eadb2c2)
Tree-SHA512: e9f768b7aa86e58b0b0ced089ead57040ff9a5204493da1ab99c8bc897b6dcdce7c856855f74c52010fceef19af1e12a39eee9f8f2e7294b42476b6f980fe754
In the Tor case, this prevents us from telling the Tor daemon to send
our incoming connections from the Tor network to an address where we
do not listen (we tried to listen but failed probably because another
application is already listening).
In the other cases (IPv4/IPv6 binds) this also prevents unpleasant
surprises caused by continuing operations even on bind failure. For
example, another application may be listening on portX, bitcoind tries
to bind on portX and portY, only succeeds with portY and continues
operation leaving the user thinking that his bitcoind is listening on
portX whereas another application is listening (the error message in
the log could easily be missed).
Avoid having the functional testing framework start multiple `bitcoind`s
that try to listen on the same `127.0.0.1:18445` (Tor listen for
regtest) if `bind_to_localhost_only` is set to `False`.
Also fix a typo in `test-shell.md` related to `bind_to_localhost_only`.
Fixes https://github.com/bitcoin/bitcoin/issues/22727
If only `-bind=addr:port` is given (without `-bind=...=onion`) then we
would bind to `addr:port` _and_ to `127.0.0.1:8334` in addition which
may be unexpected, assuming the semantic of `-bind=addr:port` is
"bind _only_ to `addr:port`".
Change the above to not do the additional bind: if only
`-bind=addr:port` is given (without `-bind=...=onion`) then bind to
`addr:port` (only). If we are creating a Tor hidden service then use
`addr:port` as target (same behavior as before
https://github.com/bitcoin/bitcoin/pull/19991).
This allows disabling binding on the onion port.
Fixes https://github.com/bitcoin/bitcoin/issues/22726
Keep mentions of v3 in debug strings to help people who might not know
that TRUC is applied when version=3.
Also keep variable names in tests, as it is less verbose to keep v3 and v2.
The migration process reloads the wallet after all failures.
This commit tests the behavior by trying to obtain a new address
after a decryption failure during migration.
Convert XoRoShiRo128PlusPlus into a full RandomMixin-based RNG class,
providing all utility functionality that FastRandomContext has. In doing so,
it is renamed to InsecureRandomContext, highlighting its non-cryptographic
nature.
To do this, a fillrand fallback is added to RandomMixin (where it is used by
InsecureRandomContext), but FastRandomContext still uses its own fillrand.
In many cases, it is known at compile time how many bits are requested from
randbits. Provide a variant of randbits that accepts this number as a template,
to make sure the compiler can make use of this knowledge. This is used immediately
in rand32() and randbool(), and a few further call sites.
The previous randbits code would, when requesting more randomness than available
in its random bits buffer, discard the remaining entropy and generate new.
Benchmarks show that it's usually better to first consume the existing randomness
and only then generate new ones. This adds some complexity to randbits, but it
doesn't weigh up against the reduced need to generate more randomness.
The "connect to ourself" detection logic has been first introduced
by Satoshi in October 2009, together with a couple of other changes
and a version bump to "v0.1.6 BETA" (see commit
cc0b4c3b62).
73f0a6cbd0 doc: detail -rpccookieperms option (willcl-ark)
d2afa2690c test: add rpccookieperms test (willcl-ark)
f467aede78 init: add option for rpccookie permissions (willcl-ark)
7df03f1a92 util: add perm string helper functions (willcl-ark)
Pull request description:
This PR picks up #26088 by aureleoules which adds a bitcoind launch option `-rpccookieperms` to set the file permissions of the cookie generated by bitcoin core.
Example usage to make the generated cookie group-readable: `./src/bitcoind -rpccookieperms=group`.
Accepted values for `-rpccookieperms` are `[owner|group|all]`. We let `fs::perms` handle platform-specific permissions changes.
ACKs for top commit:
achow101:
ACK 73f0a6cbd0
ryanofsky:
Code review ACK 73f0a6cbd0. Main change since last review is no longer throwing a skip exception in the rpc test on windows, so other checks can run after it, and overall test result is passing, not skipped. Also were clarifying renames and documentation improvements.
tdb3:
cr ACK 73f0a6cbd0
Tree-SHA512: e800d59a44aca10e1c58ca69bf3fdde9f6ccf5eab4b7b962645af6d6bc0cfa3a357701e409c8c60d8d7744fcd33a91e77ada11790aa88cd7811ef60fab86ab11
This allows a transaction's weight to be bound under a certain
weight if possible and desired. This can be beneficial for future
RBF attempts, or whenever a more restricted spend topology is
desired.
Co-authored-by: Greg Sanders <gsanders87@gmail.com>
72b226882f wallet: notify when preset + automatic inputs exceed max weight (furszy)
Pull request description:
Small change. Found it while finishing my review on #29523. This does not interfere with it.
Basically, we are erroring out early when the automatic coin selection process exceeds the maximum weight, but we are not doing so when the user-preselected inputs combined with the wallet-selected inputs exceed the maximum weight.
This change avoids signing all inputs before erroring out and introduces test coverage for `fundrawtransaction`.
ACKs for top commit:
achow101:
ACK 72b226882f
tdb3:
re ACK for 72b226882f
rkrux:
tACK [72b2268](72b226882f)
ismaelsadeeq:
utACK 72b226882f
Tree-SHA512: d77be19231023383a9c79a5d66b642dcbc6ebfc31a363e0b9f063c44898720a7859ec211cdbc0914ac7a3bfdf15e52fb8fc20d97f171431f70492c0f159dbc36
a9716c53f0 rpc: call IsInitialBlockDownload via miner interface (Sjors Provoost)
dda0b0834f rpc: minize getTipHash() calls in gbt (Sjors Provoost)
7b4d3249ce rpc: call processNewBlock via miner interface (Sjors Provoost)
9e228351e7 rpc: getTransactionsUpdated via miner interface (Sjors Provoost)
64ebb0f971 Always pass options to BlockAssembler constructor (Sjors Provoost)
4bf2e361da rpc: call CreateNewBlock via miner interface (Sjors Provoost)
404b01c436 rpc: getblocktemplate getTipHash() via Miner interface (Sjors Provoost)
d8a3496b5a rpc: call TestBlockValidity via miner interface (Sjors Provoost)
8ecb681678 Introduce Mining interface (Sjors Provoost)
Pull request description:
Introduce a `Mining` interface for the `getblocktemplate`, `generateblock` and other mining RPCs to use now, and for Stratum v2 to use later.
Suggested here: https://github.com/bitcoin/bitcoin/pull/29346#issuecomment-2108528652
The selection of methods added to the interface is mostly based on what the Template Provider in #29432 uses. It could be expanded further so that `rpc/mining.cpp` no longer needs `EnsureMemPool` and `EnsureChainman`.
This PR should be a pure refactor.
ACKs for top commit:
tdb3:
re ACK a9716c53f0
itornaza:
Code review and std-tests ACK a9716c53f0
ryanofsky:
Code review ACK a9716c53f0 with one minor suggestion in case you update. Only changes since last review were other small changes to the interface.
Tree-SHA512: cf97f87d6e9ed89da3835a0730da3b24a7b14c8605ea221149103a5915e79598cf082a95f2bc88e33f1c450e3d4aad88aed1163a29195acca88bcace055af724
This test type is represented using SEND_NO_AAD. If AAD of the first encrypted packet
sent after the garbage terminator (optional decoy packet/version packet) hasn't been
filled, disconnection happens.
This test type is represented using WRONG_GARBAGE.
Here, garbage bytes sent to TestNode are assumed to be tampered with and
do not correspond to the garbage bytes which P2PInterface calculated and
uses.
This test type is represented using WRONG_GARBAGE_TERMINATOR.
since the wrong garbage terminator is sent to TestNode, TestNode
will interpret all of the gabage bytes, wrong garbage terminator,
decoy messages and version packet it receives as garbage bytes.
If the length of all these is more than 4095 + 16, it will result
in a missing garbage terminator error. otherwise, it will result
in a V2 handshake timeout error.
Send only MAX_GARBAGE_LEN//2 bytes of garbage data to TestNode
so that the total length received by the TestNode is at max
= (MAX_GARBAGE_LEN//2) + 16 + 10*120 + 20 = 3283 bytes
(which is less than 4095 + 16 bytes) and we get a consistent
V2 handshake timeout error message.
If we do not limit the garbage length sent, we will intermittently
get both missing garbage terminator error and V2 handshake
timeout error based on the garbage length and decoy packets length
which are chosen at random.
6eecba475e net_processing: make MaybePunishNodeFor{Block,Tx} return void (Pieter Wuille)
ae60d485da net_processing: remove Misbehavior score and increments (Pieter Wuille)
6457c31197 net_processing: make all Misbehaving increments = 100 (Pieter Wuille)
5120ab1478 net_processing: drop 8 headers threshold for incoming BIP130 (Pieter Wuille)
944c54290d net_processing: drop Misbehavior for unconnecting headers (Pieter Wuille)
9f66ac7cf1 net_processing: do not treat non-connecting headers as response (Pieter Wuille)
Pull request description:
So far, discouragement of peers triggers when their misbehavior score exceeds 100 points. Most types of misbehavior increment the score by 100, triggering immediate discouragement, but some types do not. This PR makes all increments equal to either 100 (meaning any misbehavior will immediately cause disconnection and discouragement) or 0 (making the behavior effectively unconditionally allowed), and then removes the logic for score accumulation.
This simplifies the code a bit, but also makes protocol expectations clearer: if a peer misbehaves, they get disconnected. There is no good reason why certain types of protocol violations should be permitted 4 times (howmuch=20) or 9 times (howmuch=10), while many others are never allowed. Furthermore, the distinction between these looks arbitrary.
The specific types of misbehavior that are changed to 100 are:
* Sending us a `block` which does not connect to our header tree (which necessarily must have been unsollicited). [used to be score 10]
* Sending us a `headers` with a non-continuous headers sequence. [used to be score 20]
* Sending us more than 1000 addresses in a single `addr` or `addrv2` message [used to be score 20]
* Sending us more than 50000 invs in a single `inv` message [used to be score 20]
* Sending us more than 2000 headers in a single `headers` message [used to be score 20]
The specific types of misbehavior that are changed to 0 are:
* Sending us 10 (*) separate BIP130 headers announcements that do not connect to our block tree [used to be score 20]
* Sending us more than 8 headers in a single `headers` message (which thus does not get treated as a BIP130 announcement) that does not connect to our block tree. [used to be score 10]
I believe that none of these behaviors are unavoidable, except for the one marked (*) which can in theory happen still due to interaction between BIP130 and variations in system clocks (the max 2 hour in the future rule). This one has been removed entirely. In order to remove the impact of the bug it was designed to deal with, without relying on misbehavior, a separate improvement is included that makes `getheaders`-tracking more accurate.
In another unrelated improvement, this also gets rid of the 8 header limit heuristic to determine whether an incoming non-connecting `headers` is a potential BIP130 announcement, as this rule is no longer needed to prevent spurious Misbehavior. Instead, any non-connecting `headers` is now treated as a potential announcement.
ACKs for top commit:
sr-gi:
ACK [6eecba4](6eecba475e)
achow101:
ACK 6eecba475e
mzumsande:
Code Review ACK 6eecba475e
glozow:
light code review / concept ACK 6eecba475e
Tree-SHA512: e11e8a652c4ec048d8961086110a3594feefbb821e13f45c14ef81016377be0db44b5311751ef635d6e026def1960aff33f644e78ece11cfb54f2b7daa96f946
94ed4fbf8e Add release note for size 2 package rbf (Greg Sanders)
afd52d8e63 doc: update package RBF comment (Greg Sanders)
6e3c4394cf mempool: Improve logging of replaced transactions (Greg Sanders)
d3466e4cc5 CheckPackageMempoolAcceptResult: Check package rbf invariants (Greg Sanders)
316d7b63c9 Fuzz: pass mempool to CheckPackageMempoolAcceptResult (Greg Sanders)
4d15bcf448 [test] package rbf (glozow)
dc21f61c72 [policy] package rbf (Suhas Daftuar)
5da3967815 PackageV3Checks: Relax assumptions (Greg Sanders)
Pull request description:
Allows any 2 transaction package with no in-mempool ancestors to do package RBF when directly conflicting with other mempool clusters of size two or less.
Proposed validation steps:
1) If the transaction package is of size 1, legacy rbf rules apply.
2) Otherwise the transaction package consists of a (parent, child) pair with no other in-mempool ancestors (or descendants, obviously), so it is also going to create a cluster of size 2. If larger, fail.
3) The package rbf may not evict more than 100 transactions from the mempool(bip125 rule 5)
4) The package is a single chunk
5) Every directly conflicted mempool transaction is connected to at most 1 other in-mempool transaction (ie the cluster size of the conflict is at most 2).
6) Diagram check: We ensure that the replacement is strictly superior, improving the mempool
7) The total fee of the package, minus the total fee of what is being evicted, is at least the minrelayfee * size of the package (equivalent to bip125 rule 3 and 4)
Post-cluster mempool this will likely be expanded to general package rbf, but this is what we can safely support today.
ACKs for top commit:
achow101:
ACK 94ed4fbf8e
glozow:
reACK 94ed4fbf8e via range-diff
ismaelsadeeq:
re-ACK 94ed4fbf8e
theStack:
Code-review ACK 94ed4fbf8e
murchandamus:
utACK 94ed4fbf8e
Tree-SHA512: 9bd383e695964f362f147482bbf73b1e77c4d792bda2e91d7f30d74b3540a09146a5528baf86854a113005581e8c75f04737302517b7d5124296bd7a151e3992
881724d443 test: Added test coverage to listsinceblock rpc (kevkevinpal)
Pull request description:
This change is meant to add test coverage to this rpc error https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpc/transactions.cpp#L666C53-L666C79
This is done by renaming the first block in the blocks folder
---
Doing a quick grep for the error code in our functional tests leads to zero results
`grep -nri "Can't read block from disk" ./test/functional/`
ACKs for top commit:
achow101:
ACK 881724d443
tdb3:
re ACK for 881724d443
rkrux:
tACK [881724](881724d443)
Tree-SHA512: c5dff20cf014d0181f49d6b161f1364e1c6b79e8661047f77f07e21e59f4d1f2fd6f745538c8fc5bd6d4244650a840dd64d184634366f7c21fa67141a60af44a
5cf0a1f230 test: add `createmultisig` P2MS encoding test for all n (1..20) (Sebastian Falbesoner)
0570d2c204 test: add unit test for `keys_to_multisig_script` (Sebastian Falbesoner)
0c41fc3fa5 test: fix `keys_to_multisig_script` (P2MS) helper for n/k > 16 (Sebastian Falbesoner)
Pull request description:
While reviewing #28307, I noticed that the test framework's `key_to_multisig_script` helper (introduced in #23305) is broken for pubkey count (n) and threshold (k) values larger than 16. This is due to the implementation currently enforcing a direct single-byte data push (using `CScriptOp.encode_op_n`), which obviously fails for values 17+. Fix that by passing the numbers directly to the CScript list, where it's automatically converted to minimally-encoded pushes (see class method `CScript.__coerce_instance`, branch `isinstance(other, int)`).
The second commit adds a unit test to ensure that the encoding is correct.
ACKs for top commit:
achow101:
ACK 5cf0a1f230
tdb3:
ACK 5cf0a1f230
rkrux:
reACK [5cf0a1f](5cf0a1f230)
Tree-SHA512: 4168a165c3f483ec8e37a27dba1628a7ea0063545a2b7e74d9e20d753fddd7e33d37e1a190434fa6dca39adf9eef5d0211f7a0c1c7b44979f0a3bb350e267562
Prior to this commit, TestEncryptedP2PState would always
send initial_v2_handshake bytes in 2 parts (as required
by early key response test).
For generalising this test and having different v2 handshake
behaviour based on the test type, special behaviours like
sending initial_v2_handshake bytes in 2 parts are executed
only if test_type is set to EARLY_KEY_RESPONSE.
Adds argument --resultsfile to test_runner.py.
Writes comma-separated functional test name, status,
and duration to the file provided with the argument.
Also fixes minor typo in test_runner.py
f58beabe75 test: bumpfee with user specified fee_rate ignores walletIncrementalRelayFee (ismaelsadeeq)
436e88f433 bumpfee: ignore WALLET_INCREMENTAL_RELAY_FEE when user specifies fee rate (ismaelsadeeq)
Pull request description:
Fixes#26973
When using the `bumpfee` RPC and manually specifying `fee_rate`, there should be no requirement that the new fee must be at least the sum of the original fee and `incrementalFee` (maximum of `relayIncrementalFee` and `WALLET_INCREMENTAL_RELAY_FEE`).
This restriction should only apply when user did not specify `fee_rate`.
> because the GUI doesn't let the user specify the new fee rate yet (https://github.com/bitcoin-core/gui/issues/647), it would be very annoying to have to bump 20 times to increment by 20 sat/vbyte.
The restriction should instead be the new fee must be at least the sum of the original fee and `incrementalFee` (`relayIncrementalFee`)
ACKs for top commit:
achow101:
ACK f58beabe75
murchandamus:
ACK f58beabe75
Tree-SHA512: 193259f87173b7d5a8e68e0e29f2ca7e75c550e3cf0dee3d6d822b5b1e07c2e6dec0bfc8fb435855736ebced97a10dbdbfef72e8c5abde06fdefcba122f2e7f1
e2779ce98b test: cover more errors for `signrawtransactionwithkey` RPC (brunoerg)
Pull request description:
This PR adds test coverage for the following errors for the `signrawtransactionwithkey` RPC:
- Invalid private key
- TX decode failed
For reference: https://maflcko.github.io/b-c-cov/total.coverage/src/rpc/rawtransaction.cpp.gcov.html
ACKs for top commit:
maflcko:
ACK e2779ce98b
kevkevinpal:
ACK [e2779ce](e2779ce98b)
tdb3:
ACK e2779ce98b
BrandonOdiwuor:
Code Review ACK e2779ce98b
Tree-SHA512: 41c7e990684b60645cf4ccec8aad5ebbe61da221871eb3c1685b2bb1eebda58b29358502cb1525b7c7a2b612e2bebf449ed0bae14ab663b4641c528a9c013b5b
0fcbfdb7ad Support running individual lint checks (David Gumberg)
Pull request description:
This PR was split out from #29965:
Adds support for running individual tests in the rust lint suite by passing `--lint=LINT_TO_RUN` to the lint runner. This PR also adds a corresponding help message.
When running with `cargo run`, arguments after a double dash (`--`) are passed to the binary instead of the cargo command. For example, in order to run the linter check that tabs are not used as whitespace:
```console
cd test/lint/test_runner && cargo run -- --lint=tabs_whitespace
```
ACKs for top commit:
maflcko:
ACK 0fcbfdb7ad
achow101:
ACK 0fcbfdb7ad
marcofleon:
Tested ACK 0fcbfdb7ad. Ran `cargo run` with various of the individual tests and with bad input. Also ran it with no arguments. Everything works as expected and help message looks good.
Tree-SHA512: 48fe4aa9fbb2acef5f8e3c17382ae22e0e350ae6ad9aeeb1a3c0a9192de98809f98728e32b8db24a36906ace999e35626ebd6cb2ca05f74146d21e9b6fb14615
429ec1aaaa refactor: Rename CTransaction::nVersion to version (Ava Chow)
27e70f1f5b consensus: Store transaction nVersion as uint32_t (Ava Chow)
Pull request description:
Given that the use of a transaction's nVersion is always as an unsigned int, it doesn't make sense to store it as signed and then cast it to unsigned everywhere it is used and displayed.
Since a few alternative implementations have recently been revealed to have made an error with this signedness that would have resulted in consensus failure, I think it makes sense for us to just make this always unsigned to make it clear that the version is treated as unsigned. This would also help us avoid future potential issues with signedness of this value.
I believe that this is safe and does not actually change what transactions would or would not be considered both standard and consensus valid. Within consensus, the only use of the version in consensus is in BIP68 validation which was already casting it to uint32_t. Within policy, although it is used as a signed int for the transaction version number check, I do not think that this change would change standardness. Standard transactions are limited to the range [1, 2]. Negative numbers would have fallen under the < 1 condition, but by making it unsigned, they are still non-standard under the > 2 condition.
Unsigned and signed ints are serialized and unserialized the same way so there is no change in serialization.
ACKs for top commit:
maflcko:
ACK 429ec1aaaa 🐿
glozow:
ACK 429ec1aaaa
shaavan:
ACK 429ec1aaaa💯
Tree-SHA512: 0bcd92a245d7d16c3665d2d4e815a4ef28207ad4a1fb46c6f0203cdafeab1b82c4e95e4bdce7805d80a4f4a46074f6542abad708e970550d38a00d759e3dcef1
47f705b33f tests: add fuzz tests for BitSet (Pieter Wuille)
59a6df6bd5 util: add BitSet (Pieter Wuille)
Pull request description:
Extracted from #30126.
This introduces the `BitSet` data structure, inspired by `std::bitset`, but with a few features that cannot be implemented on top without efficiency loss:
* Finding the first set bit (`First`)
* Finding the last set bit (`Last`)
* Iterating over all set bits (`begin` and `end`).
And a few other operators/member functions that help readability for #30126:
* `operator-` for set subtraction
* `Overlaps()` for testing whether intersection is non-empty
* `IsSupersetOf()` for testing (non-strict) supersetness
* `IsSubsetOf()` for testing (non-strict) subsetness
* `Fill()` to construct a set with all numbers from 0 to n-1, inclusive
* `Singleton()` to construct a set with one specific element.
Everything is tested through a simulation-based fuzz test that compares the behavior with normal `std::bitset` equivalent operations.
ACKs for top commit:
instagibbs:
ACK 47f705b33f
achow101:
ACK 47f705b33f
cbergqvist:
re-ACK 47f705b33f
theStack:
Code-review ACK 47f705b33f
Tree-SHA512: e451bf4b801f193239ee434b6b614f5a2ac7bb49c70af5aba24c2ac0c54acbef4672556800e4ac799ae835632bdba716209c5ca8c37433a6883dab4eb7cd67c1
24bc46c83b cli: Add warning for duplicate port definition (tdb3)
e208fb5d3b cli: Sanitize ports in rpcconnect and rpcport (tdb3)
Pull request description:
Adds invalid port detection to bitcoin-cli for -rpcconnect and -rpcport.
In addition to detecting malformed/invalid ports (e.g. those outside of the 16-bit port range, not numbers, etc.), bitcoin-cli also now considers usage of port 0 to be invalid. bitcoin-cli previously considered port 0 to be valid and attempted to use it to reach bitcoind.
Functional tests were added for invalid port detection as well as port prioritization.
Additionally, a warning is provided when a port is specified in both -rpcconnect and -rpcport.
This PR is an alternate approach to PR #27820 (e.g. SplitHostPort is unmodified).
Considered an alternative to 127.0.0.1 being specified in functional tests, but at first glance, this might need an update to test_framework/util.py (e.g. rpc_url), which might be left to a future PR.
ACKs for top commit:
S3RK:
light code review ACK 24bc46c83b
achow101:
ACK 24bc46c83b
cbergqvist:
re ACK 24bc46c83b
Tree-SHA512: c83ab6a30a08dd1ac8b368a7dcc2b4f23170f0b61dd67ffcad7bcda05096d333bcb9821fba11018151f55b2929c0a333bfec15b8bb863d83f41fc1974c6efca5
Not every pseudorandom hash result is a valid x-only public key,
so the pubkey tweaking in the course of creating the output public
key would fail about every second time.
Fix this by treating the hash result as private key and calculate
the x-only public key out of that, to be used then as internal key.
ab98e6fd03 test: add coverage for errors for `combinerawtransaction` RPC (brunoerg)
Pull request description:
This PR adds test coverage for the following errors for the `combinerawtransaction` RPC:
* Tx decode failed
* Missing transactions
* Input not found or already spent
For reference: https://maflcko.github.io/b-c-cov/total.coverage/src/rpc/rawtransaction.cpp.gcov.html
ACKs for top commit:
maflcko:
lgtm ACK ab98e6fd03
tdb3:
ACK ab98e6fd03
Tree-SHA512: 8a133c25dad2e1b236e0278a88796f60f763e3fd6fbbc080f926bb23f9dcc55599aa242d6e0c4ec15a179d9ded10a1f17ee5b6063719107ea84e6099f10416b2
0000276b31 test: Remove redundant verack check (MarcoFalke)
Pull request description:
Currently the sync in `connect_nodes` mentions the `version` and `verack` message types, but only checks the `verack`. Neither check is required, as the `pong` check implies both. In case of failure, the debug log will have to be consulted anyway, so the redundant check doesn't add value.
Also clarify in the comments that the goal is to check the flag `fSuccessfullyConnected` indirectly.
ACKs for top commit:
furszy:
utACK 0000276b31
brunoerg:
ACK 0000276b31
tdb3:
ACK 0000276b31
Tree-SHA512: f9ddcb1436d2f70da462a8dd470ecfc90a534dd6507c23877ef7626e7c02326c077001a42ad0171a87fba5c5275d1970d8c5e5d82c56c8412de944856fdfd6db
This commit fixes a dormant bug in MiniWallet that exists since
support for P2TR was initially added in #23371 (see commit
041abfebe4).
In the course of spending the output, the leaf version byte of the
control block in the witness stack doesn't set the parity bit, i.e.
we were so far just lucky that the used combinations of relevant
data (internal pubkey, leaf script / version) didn't result in a
tweaked pubkey with odd y-parity. If that was the case, we'd get the
following validation error:
`mandatory-script-verify-flag-failed (Witness program hash mismatch) (-26)`
Since MiniWallets can now optionally be tagged (#29939), resulting
in different internal pubkeys, the issue is more prevalent now.
Fix it by passing the parity bit, as specified in BIP341.
Rather than only returning the internal key from the P2TR anyone-can-spend
address creation routine, provide the whole TaprootInfo object, which in turn
contains a dictionary of TaprootLeafInfo object for named leaves.
This data is used in MiniWallet for the default ADDRESS_OP_TRUE mode, in order
to deduplicate the witness script and leaf version of the control block.
39d135e79f test: MiniWallet: respect fee_rate for target_weight, use in mempool_limit.py (Sebastian Falbesoner)
b2f0a9f8b0 test: add framework functional test for MiniWallet's tx padding (Sebastian Falbesoner)
c17550bc3a test: MiniWallet: fix tx padding (`target_weight`) for large sizes, improve accuracy (Sebastian Falbesoner)
Pull request description:
MiniWallet allows to create padded transactions that are equal or slightly above a certain `target_weight` (first introduced in PR #25379, commit 1d6b438ef0), which can be useful especially for mempool-related tests, e.g. for policy limit checks or scenarios to trigger mempool eviction. Currently the `target_weight` parameter doesn't play together with `fee_rate` though, as the fee calculation is incorrectly based on the tx vsize before the padding output is added, so the fee-rate is consequently far off. This means users are forced to pass an absolute fee, which can be quite inconvenient and leads to lots of duplicated "calculate absolute fee from fee-rate and vsize" code with the pattern `fee = (feerate / 1000) * (weight // 4)` on the call-sites.
This PR first improves the tx padding itself to be more accurate, adds a functional test for it, and fixes the `fee_rate` treatment for the `{create,send}_self_transfer` methods. (Next step would be to enable this also for the `_self_transfer_multi` methods, but those currently don't even offer a `fee_rate` parameter). Finally, the ability to pass both `target_weight` and `fee_rate` is used in the `mempool_limit.py` functional test. There might be more use-cases in other tests, that could be done in a follow-up.
ACKs for top commit:
rkrux:
tACK [39d135e](39d135e79f)
ismaelsadeeq:
Code Review ACK 39d135e79f🚀
glozow:
light review ACK 39d135e79f
Tree-SHA512: 6bf8e853a921576d463291d619cdfd6a7e74cf92f61933a563800ac0b3c023a06569b581243166906f56b3c5e8858fec2d8a6910d55899e904221f847eb0953d
f68cba29b3 blockman: Replace m_reindexing with m_blockfiles_indexed (Ryan Ofsky)
1b1c6dcca0 test: Add functional test for continuing a reindex (TheCharlatan)
201c1a9282 indexes: Don't wipe indexes again when already reindexing (TheCharlatan)
804f09dfa1 kernel: Add less confusing reindex options (Ryan Ofsky)
e172553223 validation: Remove needs_init from LoadBlockIndex (TheCharlatan)
533eab7d67 bugfix: Streamline setting reindex option (TheCharlatan)
Pull request description:
When restarting `bitcoind` during an ongoing reindex without setting the `-reindex` flag again, the block and coins db is left intact, but any data from the optional indexes is discarded. While not a bug per se, wiping the data again is
wasteful, both in terms of having to write it again, as well as potentially leading to longer startup times. So keep the index data instead when continuing a prior reindex.
Also includes a bugfix and smaller code cleanups around the reindexing code. The bug was introduced in b47bd95920: "kernel: De-globalize fReindex".
ACKs for top commit:
stickies-v:
ACK f68cba29b3
fjahr:
Code review ACK f68cba29b3
furszy:
Code review ACK f68cba29b3
ryanofsky:
Code review ACK f68cba29b3. Only changes since last review were cherry-picking suggested commits that rename variables, improving comments, and making some tweaks to test code.
Tree-SHA512: b252228cc76e9f1eaac56d5bd9e4eac23408e0fc04aeffd97a85417f046229364673ee1ca7410b9b6e7b692b03f13ece17c42a10176da0d7e975a8915deb98ca
This adds a bitset module that implements a BitSet<N> class, a variant
of std::bitset with a few additional features that cannot be implemented
in a wrapper without performance loss (specifically, finding first and
last bit set, or iterating over all set bits).
1f6ab1215b minor: remove unnecessary semicolons from RPC content type examples (Matthew Zipkin)
b225295298 test: use json-rpc 2.0 in all functional tests by default (Matthew Zipkin)
391843b029 bitcoin-cli: use json-rpc 2.0 (Matthew Zipkin)
d39bdf3397 test: remove unused variable in interface_rpc.py (Matthew Zipkin)
0ead71df8c doc: update and link for JSON-RPC 2.0 (Matthew Zipkin)
Pull request description:
This is a follow-up to #27101.
- Addresses [post-merge comments ](https://github.com/bitcoin/bitcoin/pull/27101#discussion_r1606723428)
- bitcoin-cli uses JSON-RPC 2.0
- functional tests use JSON-RPC 2.0 by default (exceptions are in the regression tests added by #27101)
ACKs for top commit:
tdb3:
ACK 1f6ab1215b
cbergqvist:
ACK 1f6ab1215b
Tree-SHA512: 49bf14c70464081280216ece538a2f5ec810bac80a86a83ad3284f0f1b017edf755a1a74a45be279effe00218170cafde7c2de58aed07097a95c2c6b837a6b6c
In order to ensure that the change of nVersion to a uint32_t in the
previous commit has no effect, rename nVersion to version in this commit
so that reviewers can easily spot if a spot was missed or if there is a
check somewhere whose semantics have changed.
30a01134cd [doc] update bips.md for 431 (glozow)
9dbe6a03f0 [test] wallet uses CURRENT_VERSION which is 2 (glozow)
539404fe0f [policy] make v3 transactions standard (glozow)
052ede75af [refactor] use TRUC_VERSION in place of 3 (glozow)
Pull request description:
Make `nVersion=3` (which is currently nonstandard on mainnet) standard.
Note that we will treat these transactions as Topologically Restricted Until Confirmation (TRUC). Spec is in BIP 431 and implementation is in #28948, #29306, and #29873
See #27463 for overall project tracking, and #29319 for information about relevance to cluster mempool.
ACKs for top commit:
sdaftuar:
utACK 30a01134c
achow101:
ACK 30a01134cd
instagibbs:
utACK 30a01134cd
murchandamus:
ACK 30a01134cd
ismaelsadeeq:
ACK 30a01134cd🛰️
Tree-SHA512: 2a4aec0442c860e792a061d83e36483c1f1b426f946efbdf664c8db97a596e498b535707e1d3a900218429486ea69fd4552e3d476526a6883cbd5556c6534b48
fa52e13ee8 test: Remove struct.pack from almost all places (MarcoFalke)
fa826db477 scripted-diff: test: Use int.to_bytes over struct packing (MarcoFalke)
faf2a975ad test: Use int.to_bytes over struct packing (MarcoFalke)
faf3cd659a test: Normalize struct.pack format (MarcoFalke)
Pull request description:
`struct.pack` has many issues:
* The format string consists of characters that may be confusing and may need to be looked up in the documentation, as opposed to using easy to understand self-documenting code.
This lead to many test bugs, which weren't hit, which is fine, but still confusing. Ref: https://github.com/bitcoin/bitcoin/pull/29400, https://github.com/bitcoin/bitcoin/pull/29399, https://github.com/bitcoin/bitcoin/pull/29363, fa3886b7c6, ...
Fix all issues by using the built-in `int` helpers `to_bytes` via a scripted diff.
Review notes:
* For `struct.pack` and `int.to_bytes` the error behavior is the same, although the error messages are not identical.
* Two `struct.pack` remain. One for float serialization in a C++ code comment, and one for native serialization.
ACKs for top commit:
achow101:
ACK fa52e13ee8
rkrux:
tACK [fa52e13](fa52e13ee8)
theStack:
Code-review ACK fa52e13ee8
Tree-SHA512: ee80d935b68ae43d1654b047e84ceb80abbd20306df35cca848b3f7874634b518560ddcbc7e714e2e7a19241e153dee64556dc4701287ae811e26e4f8c57fe3e
The helper assumes that the n and k values have to be provided as a
single byte push operation, which is only possible for values up to 16.
Fix that by passing the numbers directly to the CScript list, where it's
automatically converted to minimally-encoded pushes (see class
method `CScript.__coerce_instance`, branch `isinstance(other, int)`).
In case of 17..20, this means that the data-pushes are done with two
bytes using OP_PUSH1 (0x01), e.g. for n=20: 0x01,0x14
4444de152f test: Set mocktime in p2p_disconnect_ban.py to avoid intermittent test failure (MarcoFalke)
fa6aa4027c test: Fix typos and use names args (MarcoFalke)
Pull request description:
Otherwise, the test may fail on slow hardware when running in valgrind.
Also, use named args for the absolute timepoint, while touching this file.
ACKs for top commit:
tdb3:
ACK for 4444de152f
AngusP:
re-ACK 4444de152f
Tree-SHA512: 660269c8dd18887d69b284f38656899caf028159ce83ddf921f3e9c080a5d0e663989f0e42b4baf4c4939f20f34da0e7e844dff9b7c91d0cab570c60958bd0e1
07aba8dd21 functional test: ensure confirmed utxo being sourced for 2nd chain (Greg Sanders)
Pull request description:
The test could fail/stop testing what we want if non-confirmed utxos become sourced through some internal change to `MiniWallet`; better to just fetch confirmed explicitly.
ACKs for top commit:
achow101:
ACK 07aba8dd21
ismaelsadeeq:
utACK 07aba8dd21
theStack:
ACK 07aba8dd21
Tree-SHA512: 66795fdf881139ed91bde0f8239a46bd9bc70bb311fa97c0e2b5537e1fd2a1fd36bf3a225fc77b9695deb835a9d6d29879aa1e05ea5054b9a33a400e199da014
2451a217dd test: addmultisigaddress, coverage for script size limits (furszy)
53302a0981 bugfix: addmultisigaddress, add unsupported operation for redeem scripts over 520 bytes (furszy)
9be6065cc0 test: coverage for 16-20 segwit multisig scripts (furszy)
9d9a91c4ea rpc: bugfix, incorrect segwit redeem script size used in signrawtransactionwithkey (furszy)
0c9fedfc45 fix incorrect multisig redeem script size limit for segwit (furszy)
f7a173b578 test: rpc_createmultisig, decouple 'test_sortedmulti_descriptors_bip67' (furszy)
4f33dbd8f8 test: rpc_createmultisig, decouple 'test_mixing_uncompressed_and_compressed_keys' (furszy)
25a81705d3 test: rpc_createmultisig, remove unnecessary checkbalances() (furszy)
b5a3289433 test: refactor, multiple cleanups in rpc_createmultisig.py (furszy)
3635d43268 test: rpc_createmultisig, remove manual wallet initialization (furszy)
Pull request description:
Fixing https://github.com/bitcoin/bitcoin/issues/28250#issuecomment-1674830104 and more.
Currently, redeem scripts longer than 520 bytes, which are technically valid under segwit rules, have flaws in the following processes:
1) The multisig creation process fails to deduce the output descriptor, resulting in the generation of an incorrect descriptor. Additionally, the accompanying user warning is also inaccurate.
2) The `signrawtransactionwithkey` RPC command fail to sign them.
3) The legacy wallet `addmultisigaddress` wrongly discards them.
The issue arises because most of these flows are utilizing the legacy spkm keystore, which imposes
the [p2sh max redeem script size rule](ded6873340/src/script/signingprovider.cpp (L160)) on all scripts. Which blocks segwit redeem scripts longer than
the max element size in all the previously mentioned processes (`createmultisig`, `addmultisigaddress`, and
`signrawtransactionwithkey`).
This PR fixes the problem, enabling the creation of multisig output descriptors involving more than 15 keys and
allowing the signing of these scripts, along with other post-segwit redeem scripts that surpass the 520-byte
p2sh limit.
Important note:
Instead of adding support for these longer redeem scripts in the legacy wallet, an "unsupported operation"
error has been added. The reasons behind this decision are:
1) The introduction of this feature brings about a compatibility-breaking change that requires downgrade
protection; older wallets would be unable to interact with these "new" legacy wallets.
2) Considering the ongoing deprecation of the legacy spkm, this issue provides another compelling
reason to transition towards descriptors.
Testing notes:
To easily verify each of the fixes, I decoupled the tests into standalone commits. So they can be
cherry-picked on top of master. Where `rpc_createmultisig.py` (with and without the `--legacy-wallet`
arg) will fail without the bugs fixes commits.
Extra note:
The initial commits improves the `rpc_createmultisig.py` test in many ways. I found this test very
antiquated, screaming for an update and cleanup.
ACKs for top commit:
pinheadmz:
ACK 2451a217dd
theStack:
Code-review ACK 2451a217dd
achow101:
ACK 2451a217dd
Tree-SHA512: 71794533cbd46b3a1079fb4e9d190d3ea3b615de0cbfa443466e14f05e4616ca90e12ce2bf07113515ea8113e64a560ad572bb9ea9d4835b6fb67b6ae596167f
df6dc2aaae test: Assumeutxo: snapshots with less work should not be loaded (Hernan Marino)
Pull request description:
This PR adds a test which checks that snapshots with less accumulated work than the node's active chain, should not be loaded and return with an error. Although in a different context of discussion the missing test was detect in a thread in https://github.com/bitcoin/bitcoin/pull/29394 (see https://github.com/bitcoin/bitcoin/pull/29394#discussion_r1484122214)
ACKs for top commit:
maflcko:
utACK df6dc2aaae
kevkevinpal:
utACK [df6dc2a](df6dc2aaae)
achow101:
ACK df6dc2aaae
alfonsoromanz:
Re ACK df6dc2aaae. Make is successful and the test passes.
Tree-SHA512: 07a394b4b288cc8ad3f66ed4e70dcda468db18113e9442eb7215cf491768432d55efaaa5b79d633094917e05475a30f0c5e4f64f8f2da293ba306891b4485560
71aae72e1f test: test sendall does ancestor aware funding (ishaanam)
36757941a0 wallet, rpc: implement ancestor aware funding for sendall (ishaanam)
544131f3fb rpc, test: test sendall spends unconfirmed change and unconfirmed inputs when specified (ishaanam)
Pull request description:
This PR:
- Adds a functional test that `sendall` spends unconfirmed change
- Adds a functional test that `sendall` spends regular unconfirmed inputs when specified by user
- Adds ancestor aware funding to `sendall` by using `calculateCombinedBumpFee` and adjusting the effective value accordingly
- Adds a functional test for ancestor aware funding in `sendall`
ACKs for top commit:
S3RK:
ACK 71aae72e1f
achow101:
ACK 71aae72e1f
furszy:
ACK 71aae72e1f
Tree-SHA512: acaeb7c65166ce53123a1d6cb5012197202246acc02ef9f37a28154cc93afdbd77c25e840ab79bdc7e0b88904014a43ab1ddea79d5337dc310ea210634ab61f0
This removes the need to actually track misbehavior score (see further commit), because any
Misbehaving node will immediately hit the discouragement threshold.
This misbehavior was originally intended to prevent bandwidth wastage due to
actually observed very broken (but likely non-malicious) nodes that respond
to GETHEADERS with a response unrelated to the request, triggering a request
cycle.
This has however largely been addressed by the previous commit, which causes
non-connecting HEADERS that are received while a GETHEADERS has not been
responded to, to be ignored, as long as they do not time out (2 minutes).
With that, the specific misbehavior is largely irrelevant (for inbound peers,
it is now harmless; for outbound peers, the eviction logic will eventually
kick them out if they're not keeping up with headers at all).
Since https://github.com/bitcoin/bitcoin/pull/25454 we keep track of the last
GETHEADERS request that was sent and wasn't responded to. So far, every incoming
HEADERS message is treated as a response to whatever GETHEADERS was last sent,
regardless of its contents.
This commit makes this tracking more accurate, by only treating HEADERS messages
which (1) are empty, (2) connect to our existing block header tree, or (3) are a
continuation of a low-work headers sync as responses that clear the "outstanding
GETHEADERS" state (m_last_getheaders_timestamp).
That means that HEADERS messages which do not satisfy any of the above criteria
will be ignored, not triggering a GETHEADERS, and potentially (for now, but see
later commit) increase misbehavior score.
4b7d984269 lint: add markdown hyperlink checker (willcl-ark)
Pull request description:
Potential followup to: #30025
This should prevent us reintroducing broken markdown links.
It does not test "online" (external) links, only those within this repo. Both relative and absolute links are parsed successfully if they resolve.
ACKs for top commit:
maflcko:
re-utACK 4b7d984269
davidgumberg:
reACK 4b7d984269
Tree-SHA512: 9bc40d700b73499c046bb76157bc139f32ec3850f64ef813bbf7f18f9c01a253abe6a857d6f559890165f2bd26e7742c05d86232cd9b8efb33ff85d735f4f095
Currently, transport version is a global variable declared as
TRANSPORT_VERSION in v2_p2p.py. Making it an instance variable
would help in sending non empty transport version packets for
testing purposes. It might also help EncryptedP2PState be more
extensible in far future protocol upgrades.
This adds a markdown hyperlink check task to the lint test_runner. It
relies on having the [`mlc`](https://crates.io/crates/mlc) binary found
on $PATH, but will fail with `success` if the binary is not found.
`mlc` is also added to the ci/04_install.sh script run by the
containerfile.
Note that broken markdown hyperlinks will be detected in untracked
markdown files found in a dirty working directory (including e.g.
.venv).
2fd34ba504 Add sanity checks for various ATMPArgs booleans (Greg Sanders)
20d8936d8b [refactor] make some members MemPoolAccept-wide (glozow)
cbbfe719b2 cpfp carveout is excluded in packages (glozow)
69f7ab05ba Add m_allow_sibling_eviction as separate ATMPArgs flag (Greg Sanders)
57ee3029dd Add description for m_test_accept (Greg Sanders)
Pull request description:
First few commits of https://github.com/bitcoin/bitcoin/pull/28984 to set the stage for the package RBF logic.
These refactors are preparation for evaluating an RBF in a multi-proposed-transaction context instead of only a single proposed transaction. Also, carveouts and sibling evictions only should work in single RBF cases so add logic to preclude multi-tx cases in the future.
No behavior changes aside from bailing earlier from failed carve-outs.
ACKs for top commit:
glozow:
reACK 2fd34ba504 via range-diff
sr-gi:
utACK [2fd34ba](2fd34ba504)
theStack:
re-ACK 2fd34ba504
Tree-SHA512: 5071c5b8d9b8d2c9faa278c8c4df31de288cb407a68e4d55544c588caff6c86160cce7825453549c6ed69e29d9ccb5ee2d4a518b18f563bfb12f2ced073fe42a
542e13b293 rpc: Enhance metadata of the dumptxoutset output (Fabian Jahr)
4d8e5edbaa assumeutxo: Add documentation on dumptxoutset serialization format (Fabian Jahr)
c14ed7f384 assumeutxo: Add test for changed coin size value (Fabian Jahr)
de95953d87 rpc: Optimize serialization disk space of dumptxoutset (Fabian Jahr)
Pull request description:
The second attempt at implementing the `dumptxoutset` space optimization as suggested in #25675. Closes#25675.
This builds on the work done in #26045, addresses open feedback, adds some further improvements (most importantly usage of compact size), documentation, and an additional test.
The [original snapshot at height 830,000](https://github.com/bitcoin/bitcoin/pull/29551) came in at 10.82 GB. With this change, the same snapshot is 8.94 GB, a reduction of 17.4%.
This also enhances the metadata of the output file and adds the following data to allow for better error handling and make future upgrades easier:
- A newly introduced utxo set magic
- A version number
- The network magic
- The block height
ACKs for top commit:
achow101:
ACK 542e13b293
TheCharlatan:
Re-ACK 542e13b293
theStack:
ACK 542e13b293
Tree-SHA512: 0825d30e5c3c364062db3c6cbca4e3c680e6e6d3e259fa70c0c2b2a7020f24a47406a623582040988d5c7745b08649c31110df4c10656aa25f3f27eb35843d99
The behavior is not new, but this rule exits earlier than before.
Previously, a carve out could have been granted in PreChecks() but then
nullified in PackageMempoolChecks() when CheckPackageLimits() is called
with the default limits.
154b2b2296 [fuzz] V3_MAX_VSIZE and effective ancestor/descendant size limits (glozow)
a29f1df289 [policy] restrict all v3 transactions to 10kvB (glozow)
d578e2e354 [policy] explicitly require non-v3 for CPFP carve out (glozow)
Pull request description:
Opening for discussion / conceptual review.
We like the idea of a smaller maximum transaction size because:
- It lowers potential replacement cost (i.e. harder to do Rule 3 pinning via gigantic transaction)
- They are easier to bin-pack in block template production
- They equate to a tighter memory limit in data structures that are bounded by a number of transactions (e.g. orphanage and vExtraTxnForCompact). For example, the current memory bounds for orphanage is 100KvB * 100 = 40MB, and guaranteeing 1 tx per peer would require reserving a pretty large space.
History for `MAX_STANDARD_TX_WEIGHT=100KvB` (copied from https://github.com/bitcoin/bitcoin/pull/29873#issuecomment-2115459510):
- 2010-09-13 In 3df62878c3 satoshi added a 100kB (MAX_BLOCK_SIZE_GEN/5 with MBS_GEN = MAX_BLOCK_SIZE/2) limit on new transactions in CreateTransaction()
- 2013-02-04 https://github.com/bitcoin/bitcoin/pull/2273 In gavin gave that constant a name, and made it apply to transaction relay as well
Lowering `MAX_STANDARD_TX_WEIGHT` for all txns is not being proposed, as there are existing apps/protocols that rely on large transactions. However, it's been brought up that we should consider this for TRUCs (which is especially designed to avoid Rule 3 pinning).
This reduction should be ok because using nVersion=3 isn't standard yet, so this wouldn't break somebody's existing use case. If we find that this is too small, we can always increase it later. Decreasing would be much more difficult.
~[Expected size of a commitment transaction](https://github.com/lightning/bolts/blob/master/03-transactions.md#expected-weight-of-the-commitment-transaction) is within (900 + 172 * 483 + 224) / 4 = 21050vB~ EDIT: this is incorrect, but perhaps not something that should affect how we choose this number.
ACKs for top commit:
sdaftuar:
ACK 154b2b2296
achow101:
ACK 154b2b2296
instagibbs:
ACK 154b2b2296
t-bast:
ACK 154b2b2296
murchandamus:
crACK 154b2b2296
Tree-SHA512: 89392a460908a8ea9f547d90e00f5181de0eaa9d2c4f2766140a91294ade3229b3d181833cad9afc93a0d0e8c4b96ee2f5aeda7c50ad7e6f3a8320b9e0c5ae97
6629d1d0f8 test: improve robustness of connect_nodes() (furszy)
Pull request description:
Decoupled from #27837 because this can help other too, found it investigating a CI failure https://cirrus-ci.com/task/5805115213348864?logs=ci#L3200.
The `connect_nodes` function in the test framework relies on a stable number of peer
connections to verify that the new connection between the nodes is successfully established.
This approach is fragile, as any of the peers involved in the process can drop, lose, or
create a connection at any step, causing subsequent `wait_until` checks to stall indefinitely
even when the peers in question were connected successfully.
This commit improves the situation by using the nodes' subversion and the connection
direction (inbound/outbound) to identify the exact peer connection and perform the
checks exclusively on it.
ACKs for top commit:
stratospher:
reACK 6629d1d.
achow101:
ACK 6629d1d0f8
maflcko:
utACK 6629d1d0f8
AngusP:
re-ACK 6629d1d0f8
Tree-SHA512: 5f345c0ce49ea81b643e97c5cffd133e182838752c27592fcdeac14ad10919fb4b7ff38e289e42a7c3c638a170bd0d0b7a9cd493898997a2082a7b7ceef4aeeb
1e54d61c46 test: add coverage for `mapped_as` from `getrawaddrman` (brunoerg)
8c2714907d net: rpc: return peer's mapped AS in getrawaddrman (brunoerg)
Pull request description:
This PR adds two new fields in `getrawaddrman` RPC: "mapped_as" and "source_mapped_as". These fields are used to return the ASN (Autonomous System Number) mapped to the peer and its source. With these informations we can have a better view of the bucketing logic with ASMap specially in projects like [addrman-observer](https://github.com/0xb10c/addrman-observer).
ACKs for top commit:
fjahr:
Code review ACK 1e54d61c46
virtu:
ACK [1e54d61](1e54d61c46)
0xB10C:
ACK 1e54d61c46
glozow:
ACK 1e54d61c46
Tree-SHA512: af86bcc7a2e69bebd3fa9eaa2e527e0758c44c0a958de7292514d5f99f8f01f5df3bae11400451268e0255f738ff3acccc77f48fe129937512f1e9d9963c4c5e
Test addresses are being mapped according to the ASMap
file provided properly. Compare the result of the `getrawaddrman`
RPC with the result from the ASMap Health Check.
b3efb48673 protocol: make message types constexpr (Vasil Dimov)
2fa9de06c2 net: make the list of known message types a compile time constant (Vasil Dimov)
Pull request description:
Turn the `std::vector` to `std::array` because it is cheaper and allows us to have the number of the messages as a compile time constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in future code to build other `std::array`s with that size.
---
This change is part of https://github.com/bitcoin/bitcoin/pull/29418 but it makes sense on its own and would be good to have it, regardless of the fate of https://github.com/bitcoin/bitcoin/pull/29418. Also, if this is merged, that would reduce the size of https://github.com/bitcoin/bitcoin/pull/29418, thus the current standalone PR.
ACKs for top commit:
achow101:
ACK b3efb48673
jonatack:
ACK b3efb48673
maflcko:
utACK b3efb48673🎊
willcl-ark:
ACK b3efb48673
Tree-SHA512: 6d3860c138c64514ebab13d97ea67893e2d346dfac30a48c3d9bc769a1970407375ea4170afdb522411ced306a14a9af4eede99e964d1fb1ea3efff5d5eb57af
The 'connect_nodes' function in the test framework relies
on a stable number of peer connections to verify the new
connection between the nodes is successfully established.
This approach is fragile, as any of the peers involved in
the process can drop, lose, or create a connection at any
step, causing subsequent 'wait_until' checks to stall
indefinitely even when the peers in question are connected
successfully.
This commit improves the situation by using the nodes' subversion
and the connection direction (inbound/outbound) to identify the
exact peer connection and perform the checks exclusively on it.
d51fbab4b3 wallet, test: Be able to always swap BDB endianness (Ava Chow)
0b753156ce test: Test bdb_ro dump of wallet without reset LSNs (Ava Chow)
c1984f1282 test: Test dumping dbs with overflow pages (Ava Chow)
fd7b16e391 test: Test dumps of other endian BDB files (Ava Chow)
6ace3e953f bdb: Be able to make byteswapped databases (Ava Chow)
d9878903fb Error if LSNs are not reset (Ava Chow)
4d7a3ae78e Berkeley RO Database fuzz test (TheCharlatan)
3568dce9e9 tests: Add BerkeleyRO to db prefix tests (Ava Chow)
70cfbfdadf wallettool: Optionally use BERKELEY_RO as format when dumping BDB wallets (Ava Chow)
dd57713f6e Add MakeBerkeleyRODatabase (Ava Chow)
6e50bee67d Implement handling of other endianness in BerkeleyRODatabase (Ava Chow)
cdd61c9cc1 wallet: implement independent BDB deserializer in BerkeleyRODatabase (Ava Chow)
ecba230979 wallet: implement BerkeleyRODatabase::Backup (Ava Chow)
0c8e728476 wallet: implement BerkeleyROBatch (Ava Chow)
756ff9b478 wallet: add dummy BerkeleyRODatabase and BerkeleyROBatch classes (Ava Chow)
ca18aea5c4 Add AutoFile::seek and tell (Ava Chow)
Pull request description:
Split from #26596
This PR adds `BerkeleyRODatabase` which is an independent implementation of a BDB file parser. It provides read only access to a BDB file, and can therefore be used as a read only database backend for wallets. This will be used for dumping legacy wallet records and migrating legacy wallets without the need for BDB itself.
Wallettool's `dump` command is changed to use `BerkeleyRODatabase` instead of `BerkeleyDatabase` (and `CWallet` itself) to demonstrate that this parser works and to test it against the existing wallettool functional tests.
ACKs for top commit:
josibake:
reACK d51fbab4b3
TheCharlatan:
Re-ACK d51fbab4b3
furszy:
reACK d51fbab4b3
laanwj:
re-ACK d51fbab4b3
theStack:
ACK d51fbab4b3
Tree-SHA512: 1e7b97edf223b2974eed2e9eac1179fc82bb6359e0a66b7d2a0c8b9fa515eae9ea036f1edf7c76cdab2e75ad994962b134b41056ccfbc33b8d54f0859e86657b
8950053636 test: remove unneeded `-maxorphantx=1000` settings (Sebastian Falbesoner)
Pull request description:
It's unclear what the motivation for increasing the orphan pool is here, and it seems that this not needed at all. None of these tests involve orphan transactions explicitly, and if they would occur occasionally, there is no good reason to prefer a value of 1000 over the default of 100 (see DEFAULT_MAX_ORPHAN_TRANSACTIONS).
ACKs for top commit:
maflcko:
utACK 8950053636
edilmedeiros:
Tested ACK 8950053636
AngusP:
tACK 8950053636
glozow:
ACK 8950053636 From skimming the tests, it appears that none of these need a larger `-maxorphantx`.
Tree-SHA512: 81d4a4fb2ea92b97119f21cbc6c4b1240d863269932e6adf4982aead9726f20652523a4707add3ad38eb332d4452de41de6735265f22e62298f3b4b45de75a57
9365baa489 test: add conflicting topology test case (Greg Sanders)
Pull request description:
We want to ensure that even if topologies
that are acceptable are relaxed, like
removing package-not-child-with-unconfirmed-parents, that we don't end up accepting packages we shouldn't.
ACKs for top commit:
glozow:
reACK 9365baa489
rkrux:
reACK [9365baa](9365baa489)
Tree-SHA512: d58661064ca099ac0447c331a5020c74c0cdfe24259aa875592805bbd63de1bf23aa7ced9ff485fef90dc0602fcb997e631aaf1aa2e9805d2cf5f0e5c9b2f0e2
It is best to store all key origin information
(master key fingerprint and all derivation steps)
in the multisig descriptor. Being explicit with
this information should be beneficial if this approach
is used with other wallets/signers (whether hardware
or software). There is no harm including all of this
with xpubs (if anything it simplifies the test code)
and makes this example/docs more complete and safer
incase it is referenced by others.
It's unclear what the motivation for increasing the orphan pool is, and
it seems that this not needed at all. None of these tests involve orphan
transactions explicitly, and if they would occur occasionally, there is
no good reason to prefer a value of 1000 over the default of 100 (see
DEFAULT_MAX_ORPHAN_TRANSACTIONS).
9408a04e42 tests, fuzz: use new NUMS_H const (josibake)
b946f8a4c5 crypto: add NUMS_H const (josibake)
Pull request description:
Broken out from #28122
---
[BIP341](https://github.com/bitcoin/bips/blob/master/bip-0341.mediawiki#constructing-and-spending-taproot-outputs) defines a NUMS point `H` as *H = lift_x(0x50929b74c1a04954b78b4b6035e97a5e078a5a0f28ec96d547bfee9ace803ac0)* which is [constructed](11af7015de/src/modules/rangeproof/main_impl.h (L16)) by taking the hash of the standard uncompressed encoding of the [secp256k1](https://www.secg.org/sec2-v2.pdf) base point G as X coordinate."
Add this as a constant so it can be used in our codebase. My primary motivation is BIP352 specifies a special case for when taproot spends use `H` as the internal key, but outside of BIP352 it seems generally useful to have `H` in the codebase, for testing or other use cases.
ACKs for top commit:
paplorinc:
re-ACK 9408a04e42
achow101:
ACK 9408a04e42
theStack:
Code-review ACK 9408a04e42
Tree-SHA512: ad84492f5d635c0cb05bd82546079ded7e5138e95361f20d8285a9ad6e69c10ee2cc3fe46e16b46ef03c4253c8bee1051911c6b91264c90c3b1ad33a824bff4b
Currently, we log the number of bytes of garbage when it is
generated. The log is a better fit for when the garbage
actually gets sent to the transport layer.
cbc6c440e3 doc: add comments and release-notes for JSON-RPC 2.0 (Matthew Zipkin)
e7ee80dcf2 rpc: JSON-RPC 2.0 should not respond to "notifications" (Matthew Zipkin)
bf1a1f1662 rpc: Avoid returning HTTP errors for JSON-RPC 2.0 requests (Matthew Zipkin)
466b90562f rpc: Add "jsonrpc" field and drop null "result"/"error" fields (Matthew Zipkin)
2ca1460ae3 rpc: identify JSON-RPC 2.0 requests (Matthew Zipkin)
a64a2b77e0 rpc: refactor single/batch requests (Matthew Zipkin)
df6e3756d6 rpc: Avoid copies in JSONRPCReplyObj() (Matthew Zipkin)
09416f9ec4 test: cover JSONRPC 2.0 requests, batches, and notifications (Matthew Zipkin)
4202c170da test: refactor interface_rpc.py (Matthew Zipkin)
Pull request description:
Closes https://github.com/bitcoin/bitcoin/issues/2960
Bitcoin Core's JSONRPC server behaves with a special blend of 1.0, 1.1 and 2.0 behaviors. This introduces compliance issues with more strict clients. There are the major misbehaviors that I found:
- returning non-200 HTTP codes for RPC errors like "Method not found" (this is not a server error or an HTTP error)
- returning both `"error"` and `"result"` fields together in a response object.
- different error-handling behavior for single and batched RPC requests (batches contain errors in the response but single requests will actually throw HTTP errors)
https://github.com/bitcoin/bitcoin/pull/15495 added regression tests after a discussion in https://github.com/bitcoin/bitcoin/pull/15381 to kinda lock in our RPC behavior to preserve backwards compatibility.
https://github.com/bitcoin/bitcoin/pull/12435 was an attempt to allow strict 2.0 compliance behind a flag, but was abandoned.
The approach in this PR is not strict and preserves backwards compatibility in a familiar bitcoin-y way: all old behavior is preserved, but new rules are applied to clients that opt in. One of the rules in the [JSON RPC 2.0 spec](https://www.jsonrpc.org/specification#request_object) is that the kv pair `"jsonrpc": "2.0"` must be present in the request. Well, let's just use that to trigger strict 2.0 behavior! When that kv pair is included in a request object, the [response will adhere to strict JSON-RPC 2.0 rules](https://www.jsonrpc.org/specification#response_object), essentially:
- always return HTTP 200 "OK" unless there really is a server error or malformed request
- either return `"error"` OR `"result"` but never both
- same behavior for single and batch requests
If this is merged next steps can be:
- Refactor bitcoin-cli to always use strict 2.0
- Refactor the python test framework to always use strict 2.0 for everything
- Begin deprecation process for 1.0/1.1 behavior (?)
If we can one day remove the old 1.0/1.1 behavior we can clean up the rpc code quite a bit.
ACKs for top commit:
cbergqvist:
re ACK cbc6c440e3
ryanofsky:
Code review ACK cbc6c440e3. Just suggested changes since the last review: changing uncaught exception error code from PARSE_ERROR to MISC_ERROR, renaming a few things, and adding comments.
tdb3:
re ACK for cbc6c440e3
Tree-SHA512: 0b702ed32368b34b29ad570d090951a7aeb56e3b0f2baf745bd32fdc58ef68fee6b0b8fad901f1ca42573ed714b150303829cddad4a34ca7ad847350feeedb36
0fb17bf61a [log] updates in TxOrphanage (glozow)
b16da7eda7 [functional test] attackers sending mutated orphans (glozow)
6675f6428d [unit test] TxOrphanage handling of same-txid-different-witness txns (glozow)
8923edfc1f [p2p] allow entries with the same txid in TxOrphanage (glozow)
c31f148166 [refactor] TxOrphanage::EraseTx by wtxid (glozow)
efcc593017 [refactor] TxOrphanage::HaveTx only by wtxid (glozow)
7e475b9648 [p2p] don't query orphanage by txid (glozow)
Pull request description:
Part of #27463 in the "make orphan handling more robust" section.
Currently the main map in `TxOrphanage` is indexed by txid; we do not allow 2 transactions with the same txid into TxOrphanage. This means that if we receive a transaction and want to store it in orphanage, we'll fail to do so if a same-txid-different-witness version of the tx already exists in the orphanage. The existing orphanage entry can stay until it expires 20 minutes later, or until we find that it is invalid.
This means an attacker can try to block/delay us accepting an orphan transaction by sending a mutated version of the child ahead of time. See included test.
Prior to #28970, we don't rely on the orphanage for anything and it would be relatively difficult to guess what transaction will go to a node's orphanage. After the parent(s) are accepted, if anybody sends us the correct transaction, we'll end up accepting it. However, this is a bit more painful for 1p1c: it's easier for an attacker to tell when a tx is going to hit a node's orphanage, and we need to store the correct orphan + receive the parent before we'll consider the package. If we start out with a bad orphan, we can't evict it until we receive the parent + try the 1p1c, and then we'll need to download the real child, put it in orphanage, download the parent again, and then retry 1p1c.
ACKs for top commit:
AngusP:
ACK 0fb17bf61a
itornaza:
trACK 0fb17bf61a
instagibbs:
ACK 0fb17bf61a
theStack:
ACK 0fb17bf61a
sr-gi:
crACK [0fb17bf](0fb17bf61a)
stickies-v:
ACK 0fb17bf61a
Tree-SHA512: edcbac7287c628bc27036920c2d4e4f63ec65087fbac1de9319c4f541515d669fc4e5fdc30c8b9a248b720da42b89153d388e91c7bf5caf4bc5b3b931ded1f59
For JSON-RPC 2.0 requests we need to distinguish between
a missing "id" field and "id":null. This is accomplished
by making the JSONRPCRequest id property a
std::optional<UniValue> with a default value of
UniValue::VNULL.
A side-effect of this change for non-2.0 requests is that request which do not
specify an "id" field will no longer return "id": null in the response.
Avoid returning HTTP status errors for non-batch JSON-RPC 2.0 requests if the
RPC method failed but the HTTP request was otherwise valid. Batch requests
already did not return HTTP errors previously.
We want to ensure that even if topologies
that are acceptable are relaxed, like
removing package-not-child-with-unconfirmed-parents,
that we don't end up accepting packages we shouldn't.
e912717ff6 test: add missing comparison of node1's mempool in MempoolPackagesTest (umiumi)
Pull request description:
#29941 Recreated a pull request because there was a conflict. Trying to resolve the conflict but the old one automatically closed.
Add missing comparison for TODO comments in `mempool_packages.py`
Also, notice that the ancestor size limits and descendant size limits actually implemented in #21800 , so I removed the todo for those two size limits.
ACKs for top commit:
kevkevinpal:
ACK [e912717](e912717ff6)
achow101:
ACK e912717ff6
alfonsoromanz:
Tested ACK e912717ff6. The code looks good to me and the test execution is successful.
rkrux:
tACK [e912717](e912717ff6)
Tree-SHA512: 8cb51746b0547369344c9ceef59599bfe9c91d424687af5e24dc6641f9e99fb433515d79c724e71fd3d5e02994f0cef623d3674367b8296b05c3c6fcdde282ef
fd6a7d3a13 test: use sleepy wait-for-log in reindex readonly (Matthew Zipkin)
Pull request description:
Also rename the busy wait-for-log method to prevent recurrence. See https://github.com/bitcoin/bitcoin/pull/27039#discussion_r1532578152
ACKs for top commit:
maflcko:
utACK fd6a7d3a13
achow101:
ACK fd6a7d3a13
tdb3:
ACK for fd6a7d3a13
rkrux:
ACK [fd6a7d3](fd6a7d3a13)
Tree-SHA512: 7ff0574833df1ec843159b35ee88b8bb345a513ac13ed0b72abd1bf330c454a3f9df4d927871b9e3d37bfcc07542b06ef63acef8e822cd18499adae8cbb0cda8
dd8fa86193 test: use tagged ephemeral MiniWallet instance in fill_mempool (Sebastian Falbesoner)
b2037ad4ae test: add MiniWallet tagging support to avoid UTXO mixing (Sebastian Falbesoner)
c8e6d08236 test: refactor: eliminate COINBASE_MATURITY magic number in fill_mempool (Sebastian Falbesoner)
4f347140b1 test: refactor: move fill_mempool to new module mempool_util (Sebastian Falbesoner)
Pull request description:
Different MiniWallet instances using the same mode (either ADDRESS_OP_TRUE, RAW_OP_TRUE or RAW_P2PK) currently always create and spend UTXOs with identical output scripts, which can cause unintentional tx dependencies (see e.g. the discussion in https://github.com/bitcoin/bitcoin/pull/29827#discussion_r1565443465). In order to avoid mixing of UTXOs between instances, this PR introduces the possibility to provide a MiniWallet tag name, that is used to derive a different internal key for the taproot construction, leading to a different P2TR output script. Note that since we use script-path spending and only the key-path is changed here, no changes in the MiniWallet spending logic are needed.
The new tagging option is then used in the `fill_mempool` helper to create an ephemeral wallet for the filling txs, as suggested in https://github.com/bitcoin/bitcoin/pull/29827#discussion_r1565964264. To avoid circular dependencies, `fill_mempool` is moved to a new module `mempool_util.py` first.
I'm still not sure if a generic word like "tag" is the right term for what this tries to achieve, happy to pick up better suggestions. Also, maybe passing a tag name is overkill and a boolean flag like "random_output_script" is sufficient?
ACKs for top commit:
glozow:
ACK dd8fa86193
achow101:
ACK dd8fa86193
rkrux:
tACK [dd8fa86](dd8fa86193)
brunoerg:
utACK dd8fa86193
Tree-SHA512: 5ef3558c3ef5ac32cfa79c8f751972ca6bceaa332cd7daac7e93412a88e30dec472cb041c0845b04abf8a317036d31ebddfc3234e609ed442417894c2bdeeac9
d53d848347 test: adds outbound eviction tests for non outbound-full-relay peers (Sergi Delgado Segura)
a8d9a0edc7 test: adds outbound eviction functional tests, updates comment in ConsiderEviction (Sergi Delgado Segura)
Pull request description:
## Motivation
While checking the outbound eviction code I realized a case was not considered within the comments, which in turn made me realize we had no functional tests for the outbound eviction case (when I went to check/add the test case).
This PR updates the aforementioned comment and adds functional tests to cover the outbound eviction logic, in addition to the existing unit tests found at `src/test/denialofservice_tests.cpp`.
ACKs for top commit:
davidgumberg:
reACK d53d848347
tdb3:
Re ACK for d53d848347
achow101:
ACK d53d848347
cbergqvist:
ACK d53d848347
Tree-SHA512: 633b84bb1229fe21e2f650c1beada33ca7f190b64eafd64df2266516d21175e5d652e019ff7114f00cb8bd19f5817dc19e65adf75767a88e24dc0842ce40c63e
b259b0e8d3 [Test] Assumeutxo: ensure failure when importing a snapshot twice (Alfonso Roman Zubeldia)
Pull request description:
I am getting familiar with the `assume_utxo` tests and I found that the scenario of trying to activate a snapshot twice is not covered. This test is to ensure failure when loading a snapshot if there is already a snapshot-based chainstate.
ACKs for top commit:
fjahr:
Code review ACK b259b0e8d3
kevkevinpal:
tACK [b259b0e](b259b0e8d3)
achow101:
ACK b259b0e8d3
rkrux:
tACK [b259b0e](b259b0e8d3)
Tree-SHA512: 3510861390d0e40cdad6861b728df04827a1b63e642f3d956aee66ed2770b1cb7e3aa3eb00c62eb9da0544703c943cc5296936c9ebfcac18c719741c354421bb
78e52f663f doc: rpc: fix submitpackage examples (stickies-v)
1a875d4049 rpc: update min package size error message in submitpackage (stickies-v)
f9ece258aa doc: rpc: submitpackage takes sorted array (stickies-v)
17f74512f0 test: add bounds checking for submitpackage RPC (stickies-v)
Pull request description:
`submitpackage` requires the package to be topologically sorted with the child being the last element in the array, but this is not documented in the RPC method or the error messages.
Also sneaking in some other minor improvements that I found while going through the code:
- Informing the user that `package` needs to be an array of length between `1` and `MAX_PACKAGE_COUNT` is confusing when `IsChildWithPackage()` requires that the package size >= 2. Remove this check to avoid code duplication and sending a confusing error message.
- fixups to the `submitpackage` examples
ACKs for top commit:
fjahr:
re-ACK 78e52f663f
instagibbs:
ACK 78e52f663f
achow101:
ACK 78e52f663f
glozow:
utACK 78e52f663f
Tree-SHA512: a8845621bb1cbf784167fc7c82cb8ceb105868b65b26d3465f072d1c04ef3699e85a21a524ade805d423bcecbc34f7d5bff12f2c21cbd902ae1fb154193ebdc9
98570fe29b test: add coverage for parsing cryptographically invalid pubkeys (Sebastian Falbesoner)
c740b154d1 rpc: use `HexToPubKey` helper for all legacy pubkey-parsing RPCs (Sebastian Falbesoner)
100e8a75bf rpc: check and throw specific pubkey parsing errors in `HexToPubKey` (Sebastian Falbesoner)
Pull request description:
Parsing legacy public keys can fail for three reasons (in this order):
- pubkey is not in hex
- pubkey has an invalid length (not 33 or 65 bytes for compressed/uncompressed, respectively)
- pubkey is crytographically invalid, i.e. is not on curve (`CPubKey.IsFullyValid()` check)
Many RPCs currently perform these checks manually with different error messages, even though we already have a `HexToPubKey` helper. This PR puts all three checks in this helper (the length check was done on the call-sites before), adds specific error messages for each case, and consequently uses it for all RPCs that parse legacy pubkeys. This leads to deduplicated code and also to more consistent and detailed error messages for the user.
Affected RPC calls are `createmultisig`, `addmultisigaddress`, `importpubkey`, `importmulti`, `fundrawtransaction`, `walletcreatefundedpsbt`, `send` and `sendall`.
Note that the error code (-5 a.k.a. `RPC_INVALID_ADDRESS_OR_KEY`) doesn't change in any of the causes, so the changes are not breaking RPC API compatibility. Only the messages are more specific.
The last commits adds test coverage for the cryptographically invalid (not-on-curve) pubkey case which wasn't exercised before.
ACKs for top commit:
stratospher:
tested ACK 98570fe.
davidgumberg:
ACK 98570fe29b
Eunovo:
Tested ACK 98570fe29b
achow101:
ACK 98570fe29b
Tree-SHA512: cfa474176e95b5b18f3a9da28fdd9e87195cd58994c1331198f2840925fff322fd323a6371feab74a1b32e4b9ea58a6dc732fa751b4cdd45402c1029af609ece
ee67bba76c test: added test coverage to loadtxoutset (kevkevin)
Pull request description:
The functional test coverage did not cover the rpc error of "Couldn't open file..." for loadtxoutset and this test adds coverage for it
This adds coverage to this line
https://github.com/bitcoin/bitcoin/blob/master/src/rpc/blockchain.cpp#L2777
ACKs for top commit:
maflcko:
ACK ee67bba76c
davidgumberg:
LGTM ACK ee67bba76c
rkrux:
ACK [ee67bba](ee67bba76c)
alfonsoromanz:
ACK ee67bba76c. Code looks good to me. I also ran `test/functional/feature_assumeutxo.py` to make sure all tests passes, including this one.
tdb3:
ACK for ee67bba76c
Tree-SHA512: 210a7eb928f625d2a8d9acb63ee83cb4aaec9c267e5a0c52ad219c2935466e2cdc68667e30ad29566e6060981587e5bec42805d296f6e60f9b3b13f3330575f2
fa09451f8e Add lint check for bitcoin-config.h include IWYU pragma (MarcoFalke)
dddd40ba82 scripted-diff: Add IWYU pragma keep to bitcoin-config.h includes (MarcoFalke)
Pull request description:
The `bitcoin-config.h` includes have issues:
* The header is incompatible with iwyu, because symbols may be defined or not defined. So the `IWYU pragma: keep` is needed to keep the include when a symbol is not defined on a platform. Compare the previous discussion in https://github.com/bitcoin/bitcoin/pull/29408#issuecomment-1948959711
* Guarding the includes by `HAVE_CONFIG_H` is verbose and brittle. Now that all build config dependencies have been removed from low level headers, the benefits are questionable, and the guard can be removed. The linter could also be tricked by guarding the include by `#if defined(HAVE_C0NFIG_H)` (`O` replaced by `0`). Compare the previous discussion in https://github.com/bitcoin/bitcoin/pull/29404#discussion_r1483189853 .
ACKs for top commit:
achow101:
ACK fa09451f8e
TheCharlatan:
ACK fa09451f8e
hebasto:
re-ACK fa09451f8e, only rebased since my recent [review](https://github.com/bitcoin/bitcoin/pull/29494#pullrequestreview-2028864535) (`timedata.cpp` removed in https://github.com/bitcoin/bitcoin/pull/29623).
Tree-SHA512: 47cb973f7f24bc625acc4e78683371863675d186780236d55d886cf4130e05a78bb04f1d731aae7088313b8e963a9677cc77cf518187dbd99d776f6421ca9b52
42fb5311b1 rpc: return warnings as an array instead of just a single one (stickies-v)
Pull request description:
The RPC documentation for `getblockchaininfo`, `getmininginfo` and `getnetworkinfo` states that "warnings" returns "any network and blockchain warnings". In practice, only a single warning (i.e. the latest one that is set) is returned, the other ones are ignored.
Fix that by returning all warnings as an array.
As a side benefit, clean up the GetWarnings() logic.
Since this PR changes the RPC result schema, I've added release notes. Users can temporarily revert to the old results by using `-deprecatedrpc=warnings`, until it's removed in a future version.
---
Some historical context from git log:
- when `GetWarnings` was introduced in 401926283a, it was used in the `getinfo` RPC, where only a [single error/warning was returned](401926283a (diff-7442c48d42cd5455a79915a0f00cce5e13359db46437a32b812876edb0a5ccddR250)) (similar to how it is now).
- later on, "warnings" RPC response fields were introduced, e.g. in ef2a3de25c, with the description [stating](ef2a3de25c (diff-1021bd3c74415ad9719bd764ad6ca35af5dfb33b1cd863c0be49bdf52518af54R411)) that it returned "any network warnings" but in practice still only a single warning was returned
ACKs for top commit:
achow101:
re-ACK 42fb5311b1
tdb3:
Re ACK for 42fb5311b1
TheCharlatan:
ACK 42fb5311b1
maflcko:
ACK 42fb5311b1🔺
Tree-SHA512: 4225ed8979cd5f030dec785a80e7452a041ad5703445da79d2906ada983ed0bbe7b15889d663d75aae4a77d92e302c93e93eca185c7bd47c9cce29e12f752bd3
This exercises the bug fixed by previous commits, where
we were unable to generate and sign for segwit redeem scripts
(in this case multisig redeem scripts) longer than 520 bytes.
and also, this adds coverage for legacy 15-15 multisig script
generation and signing.
And also, simplified the test a bit by re-using the already existing 'wallet_multi'
(instead of creating a new one). Plus, removed the 'is_bdb_compiled()' calls
which were there basically to check if the test has the wallet compiled or not.
The function exists merely to check that the node2's wallet
received the transactions created during all the 'do_multisig()'
calls.
It was created as a standalone function because 'getbalance()'
only returns something when transactions are confirmed. So,
the rationale on that time was to have a method mining blocks
to confirm the recently created transactions to be able to
check the incoming balance.
This is why we have the "moved" class field.
This change removes all the hardcoded amounts and verifies
node2 balance reception directly inside 'do_multisig()'.
Cleaning up the test in the following ways:
* Generate priv-pub key pairs used for testing only once (instead of doing it 4 times).
* Simplifies 'wmulti' wallet creation, load and unload process.
* Removes confusing class members initialized and updated inside a nested for-loop.
* Simplifies do_multisig() outpoint detection:
The outpoint index information is already contained in MiniWallet's
`send_to` return value dictionary as "sent_vout".
Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
ffc674595c Replace remaining "520" magic numbers with MAX_SCRIPT_ELEMENT_SIZE (Jon Atack)
Pull request description:
Noticed these while reviewing BIPs yesterday.
It would be clearer and more future-proof to refer to their constant name.
ACKs for top commit:
instagibbs:
ACK ffc674595c
sipa:
ACK ffc674595c
achow101:
ACK ffc674595c
glozow:
ACK ffc674595c, agree it's clearer for these comments to refer to the greppable name of the limit rather than the number
Tree-SHA512: 462afc1c64543877ac58cb3acdb01d42c6d08abfb362802f29f3482d75401a2a8adadbc2facd222a9a9fefcaab6854865ea400f50ad60bec17831d29f7798afe
ec1f1abfef test:Validate UTXO snapshot with coin_height > base_height & amount > money_supply (jrakibi)
Pull request description:
### Ensure snapshot loading fails for coins exceeding base height
**Objective**: This test verifies that snapshot loading is correctly rejected for coins with a height greater than the base height.
**Update**:
- Added `test_invalid_snapshot_wrong_coin_code` to `feature_assumeutxo.py`.
- The test artificially sets a coin's height above 299 in a snapshot and checks for load failure.
- Edit: Added a test case for outputs whose amounts surpass the MAX_MONEY supply limit.
This implementation addresses the request for enhancing `assumeutxo` testing as outlined in issue #28648
---
**Edit: This is an explanation on how I arrive at content values: b"\x84\x58" and b"\xCA\xD2\x8F\x5A"**
You can use this tool to decode the utxo snapshot https://github.com/jrakibi/utxo-live
Here’s an overview of how it’s done:
The serialization format for a UTXO in the snapshot is as follows:
1. Transaction ID (txid) - 32 bytes
2. Output Index (outnum)- 4 bytes
3. VARINT (code) - A varible-length integer encoding the height and whether the transaction is a coinbase. The format of this VARINT is (height << 1) | coinbase_flag.
4. VARINT (amount_v) - A variable-length integer that represents a compressed format of the output amount (in satoshis).
For the test cases mentioned:
* **`b"\x84\x58"`** - This value corresponds to a VARINT representing the height and coinbase flag. Once we decode this code, we can extract the height and coinbase using `height = code_decoded >> 1` and `coinbase = code_decoded & 0x01`. In our case, with code_decoded = 728, it results in `height = 364` and `coinbase = 0`.
* **`b"\xCA\xD2\x8F\x5A"`** - This byte sequence represents a compressed amount value. The decompression function takes this value and translates it into a full amount in satoshis. In our case, the decompression of this amount translates to a number larger than the maximum allowed value of coins (21 million BTC)
ACKs for top commit:
fjahr:
re-ACK ec1f1abfef
maflcko:
ACK ec1f1abfef👑
achow101:
ACK ec1f1abfef
Tree-SHA512: 42b36fd1d76e9bc45861028acbb776bd2710c5c8bff2f75c751ed505995fbc1d4bc698df3be24a99f20bcf6a534615d2d9678fb3394162b88133eaec88ca2120
fa9be2f795 lint: [doc] Clarify Windows line endings (CR LF) not to be used (MarcoFalke)
Pull request description:
It has been this case since the linter was introduced years ago. Given a misunderstanding (https://github.com/bitcoin/bitcoin/pull/28074#issuecomment-2088028856), clarify the docs.
ACKs for top commit:
brunoerg:
ACK fa9be2f795
Tree-SHA512: be714db9df533e0962ed84102ffdb72717902949b930d58cf5a79cba36297f6b2b1f75e65a2c1f46bcb8e2f4ad5d025f3d15210f468a5ec9631a580b74f923ea
e504b1fa1f test: Add test case for spending bare multisig (Brandon Odiwuor)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/29113
ACKs for top commit:
ajtowns:
ACK e504b1fa1f ; LGTM and just checking the 1-of-3 case seems fine
maflcko:
utACK e504b1fa1f
achow101:
ACK e504b1fa1f
willcl-ark:
reACK e504b1fa1f
Tree-SHA512: 641a12599efa34e1a3eb65b125318df326628fef3e6886410ea9e63a044664fad7bcad46d1d6f41ddc59630746b9963cedb569c2682b5940b32b9225883da8f2
The RPC documentation for `getblockchaininfo`, `getmininginfo` and
`getnetworkinfo` states that "warnings" returns "any network and
blockchain warnings". In practice, only a single warning is returned.
Fix that by returning all warnings as an array.
As a side benefit, cleans up the GetWarnings() logic.
c6be144c4b Remove timedata (stickies-v)
92e72b5d0d [net processing] Move IgnoresIncomingTxs to PeerManagerInfo (dergoegge)
7d9c3ec622 [net processing] Introduce PeerManagerInfo (dergoegge)
ee178dfcc1 Add TimeOffsets helper class (stickies-v)
55361a15d1 [net processing] Use std::chrono for type-safe time offsets (stickies-v)
038fd979ef [net processing] Move nTimeOffset to net_processing (dergoegge)
Pull request description:
[An earlier approach](1d226ae1f9/) in #28956 involved simplifying and refactoring the network-adjusted time calculation logic, but this was eventually [left out](https://github.com/bitcoin/bitcoin/pull/28956#issuecomment-1904214370) of the PR to make it easier for reviewers to focus on consensus logic changes.
Since network-adjusted time is now only used for warning/informational purposes, cleaning up the logic (building on @dergoegge's approach in #28956) should be quite straightforward and uncontroversial. The main changes are:
- Previously, we would only calculate the time offset from the first 199 outbound peers that we connected to. This limitation is now removed, and we have a proper rolling calculation. I've reduced the set to 50 outbound peers, which seems plenty.
- Previously, we would automatically use the network-adjusted time if the difference was < 70 mins, and warn the user if the difference was larger than that. Since there is no longer any automated time adjustment, I've changed the warning threshold to ~~20~~ 10 minutes (which is an arbitrary number).
- Previously, a warning would only be raised once, and then never again until node restart. This behaviour is now updated to 1) warn to log for every new outbound peer for as long as we appear out of sync, 2) have the RPC warning toggled on/off whenever we go in/out of sync, and 3) have the GUI warn whenever we are out of sync (again), but limited to 1 messagebox per 60 minutes
- no more globals
- remove the `-maxtimeadjustment` startup arg
Closes#4521
ACKs for top commit:
sr-gi:
Re-ACK [c6be144](c6be144c4b)
achow101:
reACK c6be144c4b
dergoegge:
utACK c6be144c4b
Tree-SHA512: 1063d639542e882186cdcea67d225ad1f97847f44253621a8c4b36c4d777e8f5cb0efe86bc279f01e819d33056ae4364c3300cc7400c087fb16c3f39b3e16b96
e518a8bf8a [functional test] opportunistic 1p1c package submission (glozow)
87c5c524d6 [p2p] opportunistically accept 1-parent-1-child packages (glozow)
6c51e1d7d0 [p2p] add separate rejections cache for reconsiderable txns (glozow)
410ebd6efa [fuzz] break out parent functions and add GetChildrenFrom* coverage (glozow)
d095316c1c [unit test] TxOrphanage::GetChildrenFrom* (glozow)
2f51cd680f [txorphanage] add method to get all orphans spending a tx (glozow)
092c978a42 [txpackages] add canonical way to get hash of package (glozow)
c3c1e15831 [doc] restore comment about why we check if ptx HasWitness before caching rejected txid (glozow)
6f4da19cc3 guard against MempoolAcceptResult::m_replaced_transactions (glozow)
Pull request description:
This enables 1p1c packages to propagate in the "happy case" (i.e. not reliable if there are adversaries) and contains a lot of package relay-related code. See https://github.com/bitcoin/bitcoin/issues/27463 for overall package relay tracking.
Rationale: This is "non-robust 1-parent-1-child package relay" which is immediately useful.
- Relaying 1-parent-1-child CPFP when mempool min feerate is high would be a subset of all package relay use cases, but a pretty significant improvement over what we have today, where such transactions don't propagate at all. [1]
- Today, a miner can run this with a normal/small maxmempool to get revenue from 1p1c CPFP'd transactions without losing out on the ones with parents below mempool minimum feerate.
- The majority of this code is useful for building more featureful/robust package relay e.g. see the code in #27742.
The first 2 commits are followups from #29619:
- https://github.com/bitcoin/bitcoin/pull/29619#discussion_r1523094034
- https://github.com/bitcoin/bitcoin/pull/29619#discussion_r1519819257
Q: What makes this short of a more full package relay feature?
(1) it only supports packages in which 1 of the parents needs to be CPFP'd by the child. That includes 1-parent-1-child packages and situations in which the other parents already pay for themselves (and are thus in mempool already when the package is submitted). More general package relay is a future improvement that requires more engineering in mempool and validation - see #27463.
(2) We rely on having kept the child in orphanage, and don't make any attempt to protect it while we wait to receive the parent. If we are experiencing a lot of orphanage churn (e.g. an adversary is purposefully sending us a lot of transactions with missing inputs), we will fail to submit packages. This limitation has been around for 12+ years, see #27742 which adds a token bucket scheme for protecting package-related orphans at a limited rate per peer.
(3) Our orphan-handling logic is somewhat opportunistic; we don't make much effort to resolve an orphan beyond asking the child's sender for the parents. This means we may miss packages if the first sender fails to give us the parent (intentionally or unintentionally). To make this more robust, we need receiver-side logic to retry orphan resolution with multiple peers. This is also an existing problem which has a proposed solution in #28031.
[1]: see this writeup and its links 02ec218c78/bip-0331.mediawiki (propagate-high-feerate-transactions)
ACKs for top commit:
sr-gi:
tACK e518a8bf8a
instagibbs:
reACK e518a8bf8a
theStack:
Code-review ACK e518a8bf8a📦
dergoegge:
light Code review ACK e518a8bf8a
achow101:
ACK e518a8bf8a
Tree-SHA512: 632579fbe7160cb763bbec6d82ca0dab484d5dbbc7aea90c187c0b9833b8d7c1e5d13b8587379edd3a3b4a02a5a1809020369e9cd09a4ebaf729921f65c15943
f8a141c2da test: Don't rely on incentive incompatible replacement in mempool_accept_v3.py (Suhas Daftuar)
Pull request description:
In the sibling eviction test, we're currently testing that a transaction with ancestor feerate (and mining score) of 179 s/b is able to replace a transaction with ancestor feerate (and mining score) of 300 s/b, due to a shortcoming in our current RBF rules.
In preparation for fixing our RBF rules to not allow such replacements, fix the test by bumping the fee of the replacement to be a bit higher.
ACKs for top commit:
glozow:
ACK f8a141c2da
instagibbs:
ACK f8a141c2da
Tree-SHA512: 0babe60be2f41634301e434fedb7abc765daaa37c2c280acb569eaf02a793369d81401ab02b8ae1689bda4872f475bd4c2f48cae4a54a61ece20db0a014e23ac
f19f0a2e5a test: Run framework unit tests in parallel (tdb3)
Pull request description:
Functional test framework unit tests are currently run prior to all other functional tests.
This PR enables execution of the test framework unit tests in parallel with the functional tests, rather than before the functional tests, saving runtime and more efficiently using available cores.
This is a follow up to https://github.com/bitcoin/bitcoin/pull/29470#issuecomment-1962313977
### New behavior:
1) When running all tests, the framework unit tests are run in parallel with the other tests (unless explicitly skipped with `--exclude`). This parallelization introduces marginal time savings when running all tests, depending on the machine used. As an example, a 2-3% time savings (9 seconds) was observed on a machine using `--jobs=18` (with 18 available cores).
2) When running specific functional tests, framework unit tests are now skipped by default. Framework unit tests can be added by including `feature_framework_unit_tests.py` in the list of specific tests being executed. The rationale for skipping by default is that if the tester is running specific functional tests, there is a conscious decision to focus testing, and choosing to run all tests (where unit tests are run by default) would be a next step.
3) The `--skipunit` option is now removed since unit tests are parallelized (they no longer delay other tests). Unit tests are treated equally as functional tests.
### Implementation notes:
Since `TextTestRunner` can be noisy (even with verbosity=0, and therefore trigger job failure through the presence of non-failure stderr output), the approach taken was to send output to stdout, and forward test result (as determined by `TestResult` returned). This aligns with the previous check for unit test failure (`if not result.wasSuccessful():`).
This approach was tested by inserting `self.assertEquals(True, False)` into test_framework/address.py and seeing specifics of the failure reported.
```
135/302 - feature_framework_unit_tests.py failed, Duration: 0 s
stdout:
.F
======================================================================
FAIL: test_bech32_decode (test_framework.address.TestFrameworkScript.test_bech32_decode)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/dev/myrepos/bitcoin/test/functional/test_framework/address.py", line 228, in test_bech32_decode
self.assertEqual(True, False)
AssertionError: True != False
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (failures=1)
stderr:
```
There was an initial thought to parallelize the execution of the unit tests themselves (i.e. run the 12 unit test files in parallel), however, this is not anticipated to further reduce runtime meaningfully and is anticipated to add unnecessary complexity.
ACKs for top commit:
maflcko:
ACK f19f0a2e5a🌽
achow101:
ACK f19f0a2e5a
kevkevinpal:
Approach ACK f19f0a2e5a
Tree-SHA512: ab9f82c30371b2242bc7a263ea0e25d35e68e2ddf223d2a55498ad940d1e5b73bba76cce8b264d71e2ed31b753430d8ef8d57efc1e4fd9ced7fb845e27f4f47e
Peer protection is only given to outbound-full-relay peers. Add a negative
test to check that other type of outbound peers are not given protection under
the circumstances that outbound-full-relay would
3e9c736a26 test: fix accurate multisig sigop count (BIP16), add unit test (Sebastian Falbesoner)
Pull request description:
In the course of reviewing #29589 I noticed the following buggy call-site of `CScriptOp.decode_op_n` in the CScript's `GetSigOpCount` method:
4cc99df44a/test/functional/test_framework/script.py (L591-L593)
This should be `lastOpcode` rather than `opcode`. The latter is either OP_CHECKMULTISIG or OP_CHECKMULTISIGVERIFY at this point, so `decode_op_n` would result in an error. Also, in `CScript.raw_iter`, we have to return the op as `CScriptOp` type instead of a bare integer, otherwise we can't call the decode method on it. To prevent this in the future, add some simple unit tests for `GetSigOpCount`.
Note that this was unnoticed, as the code part was never hit so far in the test framework.
ACKs for top commit:
achow101:
ACK 3e9c736a26
Christewart:
ACK 3e9c736a26
rkrux:
tACK [3e9c736](3e9c736a26)
hernanmarino:
tACK 3e9c736a26
Tree-SHA512: 51647bb6d462fbd101effd851afdbd6ad198c0567888cd4fdcac389a9fb4bd3d7e648095c6944fd8875d36272107ebaabdc62d0e2423289055588c12294d05a7
c4f857cc30 test: Extends wait_for_getheaders so a specific block hash can be checked (Sergi Delgado Segura)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/18614
Previously, `wait_for_getheaders` would check whether a node had received **any** getheaders message. This implied that, if a test needed to check for a specific block hash within a headers message, it had to make sure that it was checking the desired message. This normally involved having to manually clear `last_message`. This method, apart from being too verbose, was error-prone, given an undesired `getheaders` would make tests pass.
This adds the ability to check for a specific block_hash within the last `getheaders` message.
ACKs for top commit:
achow101:
ACK c4f857cc30
BrandonOdiwuor:
crACK c4f857cc30
cbergqvist:
ACK c4f857cc30
stratospher:
tested ACK c4f857c. went through all getheaders messages sent in the tests and checked that it's the one we want.
Tree-SHA512: afc9a31673344dfaaefcf692ec2ab65958c3d4c005f5f3af525e9960f0622d8246d5311e59aba06cfd5c9e0ef9eb90a7fc8e210f030bfbe67b897c061efdeed1
3bf4f8db66 lint: scripted-diff verification also requires GNU grep (Sjors Provoost)
Pull request description:
I noticed while trying to verify all historical `scripted-diff:` commits on macOS that some scripts require GNU sed.
For example 0d6d2b650d uses `git grep --perl-regexp`.
ACKs for top commit:
hernanmarino:
cr ACK 3bf4f8db66
maflcko:
utACK 3bf4f8db66
achow101:
ACK 3bf4f8db66
alfonsoromanz:
Tested ACK 3bf4f8db66
kristapsk:
cr utACK 3bf4f8db66
Tree-SHA512: 09a060ab1bafad03df60d0f20c3dd1451850868dbd66ea38b18178b6230c1f06cf48622db82d9c51422d5689962ee0cd7aae0a31f84bd6d878215e6d73c1d47e
fa6ab0d020 rpc: Reword SighashFromStr error message (MarcoFalke)
Pull request description:
Put quotes around the parameter. In theory, `std::quoted` should be used, but that seems overkill.
This should avoid error messages such as `A valid sighash parameter is not a valid sighash parameter. (code -8)`.
Also, it should fix fuzz false positives when searching for internal bugs in the `rpc` fuzz target. For example, `ZGVzY3JpcHRvcnByb2Nlc3Nwc2J0XP9ce1tdXOVJbnRlcm5hbCBidWcgZGV0ZWN0ZWQAXQ0AHfcAXQ1p7TJv`.
ACKs for top commit:
dergoegge:
ACK fa6ab0d020
brunoerg:
utACK fa6ab0d020
Tree-SHA512: e2c0cc0126de61873a863af38b7b0a23d2dadd596ca0418dae2ad091e8acfb6a9d657c376d59187bb008989dc78c6b44fe518590e5217e4049a867b220c9fb18
Reorganize functional test framework unit tests to run in parallel
with other functional tests.
The option `skipunit` is removed, since unit tests no longer delay
functional test execution.
Unit tests are run by default when running all tests, and can be
run explicitly with `feature_framework_unit_tests.py` when running
a subset of tests.
4357158c47 wallet: return and display signer error (Sjors Provoost)
dc55531087 wallet: compare address returned by displayaddress (Sjors Provoost)
6c1a2cc09a test: use h marker for external signer mock (Sjors Provoost)
Pull request description:
* HWI returns the requested address: as a sanity check, we now compare that to what we expected
* external signer documentation now reflects that HWI alternatives must implement this check
* both RPC and GUI will now return an error text, rather than just fail (the GUI even failed silently in some cases)
ACKs for top commit:
brunoerg:
ACK 4357158c47
achow101:
ACK 4357158c47
Tree-SHA512: 4f56edf3846745c8e7d08ef55cf29e8bb468256457149377c5f02da097931f9ca0c06bdbd856dc2385cde4fd11e4dc3b634c5a48814ff27f5562c8a25d43da93
e30e8625bb test: remove duplicated ban test (brunoerg)
Pull request description:
Test the ban list is preserved through restart has been done by both `rpc_setban` and `p2p_disconnect_ban`. Since `p2p_disconnect_ban` does it in a more elegant way, we can keep only it and remove the other one.
bf1b6383db/test/functional/p2p_disconnect_ban.py (L74-L110)
ACKs for top commit:
achow101:
ACK e30e8625bb
tdb3:
ACK for e30e8625bb.
hernanmarino:
tested ACK e30e8625bb
BrandonOdiwuor:
ACK e30e8625bb
alfonsoromanz:
ACK e30e8625bb
Tree-SHA512: e89624f23011e6ffd76c31b2933b8386711e1d2c03366d6b3ea850484a4fd571f69971cdbc75ce2f546d541cb3fc7f4d495a5a011217d879746414e3286ac111
6d91cb781c test: add unit tests for `calculate_input_weight` (Sebastian Falbesoner)
f81fad5e0f test: introduce and use `calculate_input_weight` helper (Sebastian Falbesoner)
Pull request description:
Rather than manually estimating an input's weight by adding up all the involved components (fixed-size skeleton, compact-serialized lengths, and the actual scriptSig / witness stack items) we can simply take use of the serialization classes `CTxIn` / `CTxInWitness` instead, to achieve the same with significantly less code.
The new helper is used in the functional tests rpc_psbt.py and wallet_send.py, where the previous manual estimation code was
duplicated. Unit tests are added in the second commit.
ACKs for top commit:
kevkevinpal:
tACK [6d91cb7](6d91cb781c)
QureshiFaisal:
tACK [6d91cb7](6d91cb781c)
achow101:
ACK 6d91cb781c
AngusP:
tACK 6d91cb781c
rkrux:
tACK [6d91cb7](6d91cb781c)
Tree-SHA512: 04424e4d94d0e13745a9c11df2dd3697c98552bbb0e792c4af67ecbb66060adc3cc0cefc202cdee2d9db0baf85b8bedf2eb339ac4b316d986b5f10f6b70c5a33
You can use this tool to decode the utxo snapshot https://github.com/jrakibi/utxo-live
Here’s an overview of how it’s done:
The serialization forma for a UTXO in the snapshot is as follows:
1. Transaction ID (txid) - 32 bytes
2. Output Index (outnum)- 4 bytes
3. VARINT (code) - A varible-length integer encoding the height and whether the transaction is a coinbase. The format of this VARINT is (height << 1) | coinbase_flag.
4. VARINT (amount_v) - A variable-length integer that represents a compressed format of the output amount (in satoshis).
For the test cases mentioned:
* b"\x84\x58" - This value corresponds to a VARINT representing the height and coinbase flag. Once we decode this code, we can extract the height and coinbase using height = code_decoded >> 1 and coinbase = code_decoded & 0x01. In our case, with code_decoded = 728, it results in height = 364 and coinbase = 0.
* b"\xCA\xD2\x8F\x5A" - This byte sequence represents a compressed amount value. The decompression function takes this value and translates it into a full amount in satoshis. In our case, the decompression of this amount translates to a number larger than the maximum allowed value of coins (21 million BTC)
test:Validate UTXO snapshot with coin_height > base_height & amount > money_supply
test:Validate UTXO snapshot with coin_height > base_height & amount > money_supply
fa6c300a99 test: Fix intermittent timeout in p2p_tx_download.py (MarcoFalke)
Pull request description:
Currently the test passes, but may fail during shutdown, because blocks and transactions are synced with `NUM_INBOUND` * `self.num_nodes` peers, which may take a long time.
There is no need for this test to have this amount of inbounds.
So avoid the extraneous inbounds to speed up the test and avoid the intermittent test failures.
ACKs for top commit:
instagibbs:
ACK fa6c300a99
fjahr:
Thanks, ACK fa6c300a99
achow101:
ACK fa6c300a99
theStack:
ACK fa6c300a99
Tree-SHA512: 0a480fd1db293ed8571ae629557cf81d5a79ec883e9e635f22c8a7cf48427161249ad2180b66c67661306f696c977b8e06ad520bd11911f119c9c95b3ffc9134
6b02c11d66 test: Fix intermittent issue in p2p_handshake.py (stratospher)
Pull request description:
When establishing outbound connections [`TestNode` --------> `P2PConnection`], `P2PConnection` listens for a single connection from `TestNode` on a [port which is fixed based on `p2p_idx`](312f54278f/test/functional/test_framework/p2p.py (L746)).
If we reuse the same port when disconnecting and establishing connections again, we might hit this scenario where:
- disconnection is done on python side for `P2PConnection`
- disconnection not complete on c++ side for `TestNode`
- we're trying to establish a new connection on same port again
Prevent this scenario from happening by ensuring disconnection on c++ side for TestNode as well.
One way to reproduce this on master would be adding a sleep statement before disconnection happens on c++ side.
```diff
diff --git a/src/net.cpp b/src/net.cpp
index e388f05b03..62507d1f39 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -2112,6 +2112,7 @@ void CConnman::SocketHandlerConnected(const std::vector<CNode*>& nodes,
if (!pnode->fDisconnect) {
LogPrint(BCLog::NET, "socket closed for peer=%d\n", pnode->GetId());
}
+ std::this_thread::sleep_for(std::chrono::milliseconds(1000));
pnode->CloseSocketDisconnect();
}
else if (nBytes < 0)
```
ACKs for top commit:
maflcko:
lgtm ACK 6b02c11d66
mzumsande:
Tested ACK 6b02c11d66
BrandonOdiwuor:
Tested ACK 6b02c11d66
theStack:
Tested ACK 6b02c11d66
glozow:
ACK 6b02c11d66
Tree-SHA512: 69509edb61ba45739fd585b6cc8a254f412975c124a5b5a52688288ecaaffd264dd76019b8290cc34c26c3ac2dfe477965ee5a11d7aabdd8e4d2a75229a4a068
21d0e6c7b7 doc: release notes for PR 27679 (Matthew Zipkin)
791dea204e test: cover unix sockets in zmq interface (Matthew Zipkin)
c87b0a0ff4 zmq: accept unix domain socket address for notifier (Matthew Zipkin)
Pull request description:
This is a follow-up to https://github.com/bitcoin/bitcoin/pull/27375, allowing ZMQ notifications to be published to a UNIX domain socket.
Fortunately, libzmq handles unix sockets already, all we really have to do to support it is allow the format in the actual option.
[libzmq](https://libzmq.readthedocs.io/en/latest/zmq_ipc.html) uses the prefix `ipc://` as opposed to `unix:` which is [used by Tor](https://gitlab.torproject.org/tpo/core/tor/-/blob/main/doc/man/tor.1.txt?ref_type=heads#L1475) and now also by [bitcoind](a85e5a7c9a/doc/release-notes-27375.md (L5)) so we need to switch that internally.
As far as I can tell, [LND](d20a764486/zmq.go (L38)) supports `ipc://` and `unix://` (notice the double slashes).
With this patch, LND can connect to bitcoind using unix sockets:
Example:
*bitcoin.conf*:
```
zmqpubrawblock=unix:/tmp/zmqsb
zmqpubrawtx=unix:/tmp/zmqst
```
*lnd.conf*:
```
bitcoind.zmqpubrawblock=ipc:///tmp/zmqsb
bitcoind.zmqpubrawtx=ipc:///tmp/zmqst
```
ACKs for top commit:
laanwj:
Code review ACK 21d0e6c7b7
tdb3:
crACK for 21d0e6c7b7. Changes lgtm. Will follow up with some testing within the next few days as time allows.
achow101:
ACK 21d0e6c7b7
guggero:
Tested and code review ACK 21d0e6c7b7
Tree-SHA512: ffd50222e80dd029d903e5ddde37b83f72dfec1856a3f7ce49da3b54a45de8daaf80eea1629a30f58559f4b8ded0b29809548c0638cd1c2811b2736ad8b73030
60ca5d5508 test: p2p: add test for rejected tx request logic (`m_recent_rejects` filter) (Sebastian Falbesoner)
e9dc511a7e fixup: get all utxos up front in fill_mempool, discourage wallet mixing (glozow)
Pull request description:
Motivated by the discussion in #28970 (https://github.com/bitcoin/bitcoin/pull/28970#discussion_r1553911167), this PR adds test coverage for the logic around the `m_recent_rejects` filter, in particular that the filter is cleared after a new block comes in:
f0794cbd40/src/net_processing.cpp (L2199-L2206)
As expected, the second part of the test fails if the following patch is applied:
```diff
diff --git a/src/net_processing.cpp b/src/net_processing.cpp
index 6996af38cb..5cb1090e70 100644
--- a/src/net_processing.cpp
+++ b/src/net_processing.cpp
@@ -2202,7 +2202,7 @@ bool PeerManagerImpl::AlreadyHaveTx(const GenTxid& gtxid)
// or a double-spend. Reset the rejects filter and give those
// txs a second chance.
hashRecentRejectsChainTip = m_chainman.ActiveChain().Tip()->GetBlockHash();
- m_recent_rejects.reset();
+ //m_recent_rejects.reset();
}
const uint256& hash = gtxid.GetHash();
```
I'm still not sure in which file this test fits best, and if there is already test coverage for the first part of the test somewhere. Happy for any suggestions.
ACKs for top commit:
maflcko:
ACK 60ca5d5508🍳
glozow:
code review ACK 60ca5d5508
instagibbs:
ACK 60ca5d5508
Tree-SHA512: 9cab43858e8f84db04a708151e6775c9cfc68c20ff53096220eac0b2c406f31aaf9223e8e04be345e95bf0a3f6dd15efac50b0ebeb1582a48a4560b3ab0bcba5
If we reuse the same port when disconnecting and establishing connections
again, we might hit this scenario:
- disconnection is done on python side for P2PConnection
- disconnection is not complete on c++ side for TestNode
- we're trying to establish a new connection on same port again
Prevent this scenario from happening by ensuring disconnection on c++
side for TestNode as well.
1ae5b208d3 test: fix intermittent failure in p2p_compactblocks_hb.py (Martin Zumsande)
Pull request description:
Fixes#29860
As a result of node1 receiving a block, it sends out SENDCMPCT messages to some of its peers to update the high-bandwidth status. We need to wait until those are received and processed by the peers to avoid intermittent failures. Before, we'd only wait until all peers have synced with the new block (within `generate`) which is not sufficient.
I could reproduce the failure by adding a `std::this_thread::sleep_for(std::chrono::milliseconds(1000));` sleep to the [net_processing code](c7567d9223/src/net_processing.cpp (L3763)) that processes `NetMsgType::SENDCMPCT`.
ACKs for top commit:
instagibbs:
ACK 1ae5b208d3
alfonsoromanz:
Tested ACK 1ae5b208d3
glozow:
ACK 1ae5b208d3
Tree-SHA512: 47c29616e73a5e0ff966fc231e4f672c1a6892511e5c10a3905b30ad6b2a3d1267fa0a88bd8f64b523fe580199d22a43545c84e361879e5096483152065c4b9a
b7ba60f81a test: add coverage for -reindex and assumeutxo (Martin Zumsande)
e57f951805 init, validation: Fix -reindex option with an existing snapshot (Martin Zumsande)
Pull request description:
In c711ca186f logic was introduced that `-reindex` and `-reindex-chainstate` will delete the snapshot chainstate.
This doesn't work currently, instead of deleting the snapshot chainstate the node crashes with an assert (this can be triggered by applying the added test commit on master).
Fix this, and another bug that would prevent the new active chainstate from having a mempool after `-reindex` has deleted the snapshot (also covered by the test).
ACKs for top commit:
fjahr:
re-ACK b7ba60f81a
hernanmarino:
crACK b7ba60f81a . Good fix
BrandonOdiwuor:
re-ACK b7ba60f81a
byaye:
Tested ACK b7ba60f81a
Tree-SHA512: c168f36997d7677d590af37b10427870f5d30123abf1c76032a16661e486735373bfa7e049e6aca439526fbcb6d619f970bf9d042196c851bf058a75a32fafdc
As a result of node1 receiving a block, it sends out
SENDCMPCT messages to its peers to update the status.
We need to wait until those are received and
processed by the peers to avoid intermittent failures.
Both RPC and GUI now render a useful error message instead of (silently) failing.
Replace bool with util::Result<void> to clarify that this either succeeds or returns an error message.
c2e0489b71 [rpc, bugfix] Enforce maximum value for setmocktime (dergoegge)
Pull request description:
The maximum value for our mocktime must be representable in nanoseconds, otherwise we end up with negative values returned from `NodeClock::now()`.
Found through fuzzing:
```
$ echo "c2V0bW9ja3RpbWVcZTptYf9w/3NldG3///////////////9p////ZP///ymL//////89////Nv9L////////LXkBAABpAA==" | base64 --decode > rpc-8cab9148ab4418ebd1923c213e9d3fe9c9b49b39.crash
$ FUZZ=rpc ./src/test/fuzz/fuzz rpc-8cab9148ab4418ebd1923c213e9d3fe9c9b49b39.crash
fuzz_libfuzzer: util/time.cpp:28: static NodeClock::time_point NodeClock::now(): Assertion `ret > 0s' failed.
```
ACKs for top commit:
maflcko:
re-ACK c2e0489b71
brunoerg:
crACK c2e0489b71
glozow:
ACK c2e0489b71
Tree-SHA512: d7e237ca37bedd74a6b085fb6e726a142705371044c77488f593f35afe70aeca756fdba86920294b1d322c7a9b2cde9ce4e1b7d410a6ccc1fd7c6f3a6e77200a
A common issue that our fuzzers keep finding is that outpoints don't
exist in the non witness utxos. Instead of trying to track this down and
checking in various individual places, do the check early during
deserialization.
4ba1d0b553 fuzz: Add coverage for client_maxfeerate (Greg Sanders)
91d7d8f22a AcceptMultipleTransactions: Fix workspace client_maxfeerate (Greg Sanders)
f3aa5bd5eb fill_mempool: assertions and docsctring update (Greg Sanders)
a3da63e8fe Move fill_mempool to util function (Greg Sanders)
73b68bd8b4 fill_mempool: remove subtest-specific comment (Greg Sanders)
Pull request description:
Bug causes an `Assume()` failure due to the expectation that the individual result should be invalid when done over `submitpackage` via rpc.
Bug introduced by https://github.com/bitcoin/bitcoin/pull/28950 , and I discovered it rebasing https://github.com/bitcoin/bitcoin/pull/28984 since it's easier to hit in that test scenario.
Tests in place were only checking `AcceptSingleTransaction`-level checks due to package evaluation only triggering when minfee is too high for the parent transaction.
Added test along with fix, moving the fill_mempool utility into a common area for re-use.
ACKs for top commit:
glozow:
reACK 4ba1d0b553
theStack:
ACK 4ba1d0b553
ismaelsadeeq:
re-ACK 4ba1d0b553 via [diff](4fe7d150eb..4ba1d0b553)
Tree-SHA512: 3729bdf7f25d04e232f173ccee04ddbb2afdaafa3d04292a01cecf58fb11b3b2bc133e8490277f1a67622b62d17929c242dc980f9bb647896beea4332ee35306
This helper class is an alternative to CMedianFilter, but without a
lot of the special logic and exceptions that we needed while it was
still used for consensus.
d5a715536e build: remove boost::process dependency for building external signer support (Sebastian Falbesoner)
70434b1c44 external_signer: replace boost::process with cpp-subprocess (Sebastian Falbesoner)
cc8b9875b1 Add `cpp-subprocess` header-only library (Hennadii Stepanov)
Pull request description:
Closes https://github.com/bitcoin/bitcoin/issues/24907.
This PR is based on **theStack**'s [work](https://github.com/bitcoin/bitcoin/issues/24907#issuecomment-1466087049).
The `subprocess.hpp` header has been sourced from the [upstream repo](https://github.com/arun11299/cpp-subprocess) with the only modification being the removal of convenience functions, which are not utilized in our codebase.
Windows-related changes will be addressed in subsequent follow-ups.
ACKs for top commit:
achow101:
reACK d5a715536e
Sjors:
re-tACK d5a715536e
theStack:
Light re-ACK d5a715536e
fanquake:
ACK d5a715536e - with the expectation that this code is going to be maintained as our own. Next PRs should:
Tree-SHA512: d7fb6fecc3f5792496204190afb7d85b3e207b858fb1a75efe483c05260843b81b27d14b299323bb667c990e87a07197059afea3796cf218ed8b614086bd3611
47cedee776 fuzz: Introduce `BITCOINFUZZ` environment variable (Hennadii Stepanov)
1573e9a11e fuzz, refactor: Deduplicate fuzz binary path creation (Hennadii Stepanov)
Pull request description:
These changes are split from https://github.com/bitcoin/bitcoin/pull/29774 and can be beneficial on their own.
The new `BITCOINFUZZ` environment variable complements the already existing set of variables used by tests: b5d21182e5/test/functional/test_framework/test_framework.py (L238-L243)
ACKs for top commit:
maflcko:
lgtm ACK 47cedee776
davidgumberg:
utACK 47cedee776
Tree-SHA512: 45809cfd13dc4a45c44cc433184352e84726cb95bea80fd8f581c59a0b8b0a5495260ff66922f9c57c38adbdbdd102439238f370fd49d6ea27a241a5e6249895
d4e36ae80d test: Update --tmpdir doc string to say directory must not exist (kevkevin)
Pull request description:
The error message given if passing an existing dir to --tmpdir is confusing so this makes it clear that the directory must not already exist
This change is motivated by this comment https://github.com/bitcoin/bitcoin/pull/29335#issuecomment-1960913020
ACKs for top commit:
maflcko:
lgtm ACK d4e36ae80d
davidgumberg:
ACK d4e36ae80d
Tree-SHA512: fb31fd079767abbf94076615817943f35f5c9262fc97e65c631a18d33b3a343fe6a2d151613256e632d2b372ab2de0435f4712309b4a77ed3c663fd93a7dcdd1
93fae5ae7c test: remove immediate tx relay workaround in wallet_groups.py (Sebastian Falbesoner)
Pull request description:
Reverts commit ab4efad51b (PR #26970). This workaround is not needed anymore, as since #27114 the test sets the noban permission for both in- and outbound connections via the `noban_tx_relay` setting, and we don't have to rely on this topology hack anymore. See commit c985eb854c (kudos to brunoerg!).
Can be tested by executing `$ time ./test/functional/wallet_groups.py` both on master and PR and verifying that the execution time is roughly equal.
ACKs for top commit:
maflcko:
lgtm ACK 93fae5ae7c
brunoerg:
utACK 93fae5ae7c
Tree-SHA512: b949fd05b4308815ba02d0ee4d1318f642b930288dd03223f46db7db745177af1c070bc7058743ac27963c5ad90564089867cc12f31fee94812a16919c353bab
49c0b8b228 test: Bump timeouts in feature_index_prune and wallet_importdescriptors (Christopher Bergqvist)
Pull request description:
Timeout issues where encountered when running functional tests with `--jobs=16 --extended`.
Note that running `--extended` without `--jobs=16` does not trigger the issues.
Tested under NixOS on a Xeon CPU with 16 logical cores.
(A few tests are skipped locally as I haven't enabled BPF and a few other things).
## Measurements
Line in `feature_index_prune.py` took 101.6s, 96.6s, 103.0s across 3 runs on my machine.
Default limit is 60, suggested to increase limit to 150 seconds.
Line in the `wallet_importdescriptors.py --descriptors` took 5.4s, 5.7s, 6.0s across 3 runs.
Suggested to increase from 5 to 10 seconds.
## Logs
Output slightly modified by separate change that lets code run past given timeouts and the provides more information - "Took 101.6 seconds to complete, 69.4% over the given limit.".
<details>
<summary>
Click to expand.
</summary>
### feature_index_prune.py
```
52/305 - feature_index_prune.py failed, Duration: 250 s
stdout:
2024-04-01T22:25:24.010000Z TestFramework (INFO): PRNG seed is: 990421162716295219
2024-04-01T22:25:24.014000Z TestFramework (INFO): Initializing test directory /mnt/tmp/test_runner_₿_🏃_20240402_002516/feature_index_prune_302
2024-04-01T22:25:24.913000Z TestFramework (INFO): check if we can access blockfilters and coinstats when pruning is enabled but no blocks are actually pruned
2024-04-01T22:26:48.417000Z TestFramework (INFO): prune some blocks
2024-04-01T22:26:48.460000Z TestFramework (INFO): check if we can access the tips blockfilter and coinstats when we have pruned some blocks
2024-04-01T22:26:48.483000Z TestFramework (INFO): check if we can access the blockfilter and coinstats of a pruned block
2024-04-01T22:26:59.175000Z TestFramework (INFO): make sure trying to access the indices throws errors
2024-04-01T22:27:50.422000Z TestFramework (INFO): prune exactly up to the indices best blocks while the indices are disabled
2024-04-01T22:27:52.596000Z TestFramework (INFO): make sure that we can continue with the partially synced indices after having pruned up to the index height
2024-04-01T22:29:34.242000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: '''
self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats)#, timeout=150)
'''
2024-04-01T22:29:34.244000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/home/chris/Documents/Code/bitcoin-core/test/functional/test_framework/test_framework.py", line 132, in main
self.run_test()
File "/home/chris/Documents/Code/bitcoin-core/test/functional/feature_index_prune.py", line 117, in run_test
self.sync_index(height=1500)
File "/home/chris/Documents/Code/bitcoin-core/test/functional/feature_index_prune.py", line 34, in sync_index
self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats)#, timeout=150)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/chris/Documents/Code/bitcoin-core/test/functional/test_framework/test_framework.py", line 780, in wait_until
return wait_until_helper_internal(test_function, timeout=timeout, timeout_factor=self.options.timeout_factor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/chris/Documents/Code/bitcoin-core/test/functional/test_framework/util.py", line 305, in wait_until_helper_internal
raise AssertionError(m)
AssertionError: Predicate '''
self.wait_until(lambda: self.nodes[1].getindexinfo() == expected_stats)#, timeout=150)
''' not true after 60 seconds. Took 101.6 seconds to complete, 69.4% over the given limit.
2024-04-01T22:29:34.298000Z TestFramework (INFO): Stopping nodes
2024-04-01T22:29:34.511000Z TestFramework (WARNING): Not cleaning up dir /mnt/tmp/test_runner_₿_🏃_20240402_002516/feature_index_prune_302
2024-04-01T22:29:34.511000Z TestFramework (ERROR): Test failed. Test logging available at /mnt/tmp/test_runner_₿_🏃_20240402_002516/feature_index_prune_302/test_framework.log
2024-04-01T22:29:34.511000Z TestFramework (ERROR):
2024-04-01T22:29:34.512000Z TestFramework (ERROR): Hint: Call /home/chris/Documents/Code/bitcoin-core/test/functional/combine_logs.py '/mnt/tmp/test_runner_₿_🏃_20240402_002516/feature_index_prune_302' to consolidate all logs
2024-04-01T22:29:34.512000Z TestFramework (ERROR):
2024-04-01T22:29:34.512000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-01T22:29:34.512000Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2024-04-01T22:29:34.512000Z TestFramework (ERROR):
stderr:
53/305 - p2p_blockfilters.py passed, Duration: 130 s
```
### wallet_importdescriptors.py --descriptors
```
297/305 - wallet_importdescriptors.py --descriptors failed, Duration: 76 s
stdout:
2024-04-01T22:48:27.663000Z TestFramework (INFO): PRNG seed is: 8528678505617325332
2024-04-01T22:48:27.664000Z TestFramework (INFO): Initializing test directory /mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98
2024-04-01T22:48:28.021000Z TestFramework (INFO): Setting up wallets
2024-04-01T22:48:28.100000Z TestFramework (INFO): Mining coins
2024-04-01T22:48:29.714000Z TestFramework (INFO): Import should fail if a descriptor is not provided
2024-04-01T22:48:29.725000Z TestFramework (INFO): Should import a p2pkh descriptor
2024-04-01T22:48:29.740000Z TestFramework (INFO): Test can import same descriptor with public key twice
2024-04-01T22:48:29.760000Z TestFramework (INFO): Test can update descriptor label
2024-04-01T22:48:29.785000Z TestFramework (INFO): Internal addresses cannot have labels
2024-04-01T22:48:29.788000Z TestFramework (INFO): Internal addresses should be detected as such
2024-04-01T22:48:29.854000Z TestFramework (INFO): Should not import a p2sh-p2wpkh descriptor without checksum
2024-04-01T22:48:29.855000Z TestFramework (INFO): Should not import a p2sh-p2wpkh descriptor that has range specified
2024-04-01T22:48:29.858000Z TestFramework (INFO): Should not import a p2sh-p2wpkh descriptor and have it set to active
2024-04-01T22:48:29.860000Z TestFramework (INFO): Should import a (non-active) p2sh-p2wpkh descriptor
2024-04-01T22:48:29.984000Z TestFramework (INFO): Should import a 1-of-2 bare multisig from descriptor
2024-04-01T22:48:30.002000Z TestFramework (INFO): Should not treat individual keys from the imported bare multisig as watchonly
2024-04-01T22:48:30.005000Z TestFramework (INFO): Ranged descriptors cannot have labels
2024-04-01T22:48:30.014000Z TestFramework (INFO): Private keys required for private keys enabled wallet
2024-04-01T22:48:30.027000Z TestFramework (INFO): Ranged descriptor import should warn without a specified range
2024-04-01T22:48:30.065000Z TestFramework (INFO): Should not import a ranged descriptor that includes xpriv into a watch-only wallet
2024-04-01T22:48:30.070000Z TestFramework (INFO): Should not import a descriptor with hardened derivations when private keys are disabled
2024-04-01T22:48:30.108000Z TestFramework (INFO): Verify we can only extend descriptor's range
2024-04-01T22:48:30.364000Z TestFramework (INFO): Check we can change descriptor internal flag
2024-04-01T22:48:30.536000Z TestFramework (INFO): Key ranges should be imported in order
2024-04-01T22:48:30.708000Z TestFramework (INFO): Check we can change next_index
2024-04-01T22:48:30.838000Z TestFramework (INFO): Check imported descriptors are not active by default
2024-04-01T22:48:30.870000Z TestFramework (INFO): Check can activate inactive descriptor
2024-04-01T22:48:30.903000Z TestFramework (INFO): Check can deactivate active descriptor
2024-04-01T22:48:30.924000Z TestFramework (INFO): Verify activation state is persistent
2024-04-01T22:48:30.973000Z TestFramework (INFO): Should import a descriptor with a WIF private key as spendable
2024-04-01T22:48:30.987000Z TestFramework (INFO): Test can import same descriptor with private key twice
2024-04-01T22:48:32.173000Z TestFramework (INFO): Test that multisigs can be imported, signed for, and getnewaddress'd
2024-04-01T22:48:43.803000Z TestFramework (INFO): Multisig with distributed keys
2024-04-01T22:48:48.895000Z TestFramework (INFO): We can create and use a huge multisig under P2WSH
2024-04-01T22:49:05.628000Z TestFramework (INFO): Under P2SH, multisig are standard with up to 15 compressed keys
2024-04-01T22:49:20.258000Z TestFramework (INFO): Amending multisig with new private keys
2024-04-01T22:49:23.306000Z TestFramework (INFO): Combo descriptors cannot be active
2024-04-01T22:49:23.313000Z TestFramework (INFO): Descriptors with no type cannot be active
2024-04-01T22:49:23.348000Z TestFramework (INFO): Test importing a descriptor to an encrypted wallet
2024-04-01T22:49:43.957000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/home/chris/Documents/Code/bitcoin-core/test/functional/test_framework/test_framework.py", line 132, in main
self.run_test()
File "/home/chris/Documents/Code/bitcoin-core/test/functional/wallet_importdescriptors.py", line 691, in run_test
with self.nodes[0].assert_debug_log(expected_msgs=["Rescan started from block 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206... (slow variant inspecting all blocks)"], timeout=5):#10):
File "/nix/store/rac8pxbi1vapwrlqzbrkycbyg521djzw-python3-3.11.6/lib/python3.11/contextlib.py", line 144, in __exit__
next(self.gen)
File "/home/chris/Documents/Code/bitcoin-core/test/functional/test_framework/test_node.py", line 493, in assert_debug_log
self._raise_assertion_error(f'Expected messages "{expected_msgs}" found too late, took {now - start:.1f} seconds, {((now - start) / (time_end - start)) - 1:.1%} over the given limit. Log:\n\n{print_log}\n\n')
File "/home/chris/Documents/Code/bitcoin-core/test/functional/test_framework/test_node.py", line 188, in _raise_assertion_error
raise AssertionError(self._node_msg(msg))
AssertionError: [node 0] Expected messages "['Rescan started from block 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206... (slow variant inspecting all blocks)']" found too late, took 5.4 seconds, 8.9% over the given limit. Log:
- 2024-04-01T22:49:33.066512Z [http] [httpserver.cpp:306] [http_request_cb] [http] Received a POST request for /wallet/encrypted_wallet from 127.0.0.1:47658
- 2024-04-01T22:49:33.066668Z [httpworker.0] [rpc/request.cpp:187] [parse] [rpc] ThreadRPCServer method=importdescriptors user=__cookie__
- 2024-04-01T22:49:33.070999Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT INTO main VALUES(?, ?)
- 2024-04-01T22:49:33.071061Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: DELETE FROM main WHERE key = ?
- 2024-04-01T22:49:33.071137Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: BEGIN TRANSACTION
- 2024-04-01T22:49:33.074190Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
- 2024-04-01T22:49:33.075564Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
...<thousands of almost identical lines>...
- 2024-04-01T22:49:38.416139Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
- 2024-04-01T22:49:38.416528Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
- 2024-04-01T22:49:38.427946Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: COMMIT TRANSACTION
- 2024-04-01T22:49:38.429778Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
- 2024-04-01T22:49:38.429916Z [httpworker.0] [wallet/sqlite.cpp:57] [TraceSqlCallback] [/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/node0/regtest/wallets/encrypted_wallet/wallet.dat] SQLite Statement: INSERT or REPLACE into main values(?, ?)
- 2024-04-01T22:49:38.430001Z [httpworker.0] [wallet/wallet.h:933] [WalletLogPrintf] [encrypted_wallet] Setting spkMan to active: id = c6149b35399517457b0b1d8ccdd7efda25a2f20fc7f8167adda8e79b10e260b7, type = legacy, internal = false
- 2024-04-01T22:49:38.430134Z [httpworker.0] [wallet/wallet.h:933] [WalletLogPrintf] [encrypted_wallet] RescanFromTime: Rescanning last 329 blocks
- 2024-04-01T22:49:38.430170Z [httpworker.0] [wallet/wallet.h:933] [WalletLogPrintf] [encrypted_wallet] Rescan started from block 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206... (slow variant inspecting all blocks)
- 2024-04-01T22:49:38.441914Z [httpworker.0] [wallet/scriptpubkeyman.h:258] [WalletLogPrintf] [encrypted_wallet] MarkUnusedAddresses: Detected a used keypool item at index 4000, mark all keypool items up to this item as used
2024-04-01T22:49:44.029000Z TestFramework (INFO): Stopping nodes
2024-04-01T22:49:44.132000Z TestFramework (WARNING): Not cleaning up dir /mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98
2024-04-01T22:49:44.132000Z TestFramework (ERROR): Test failed. Test logging available at /mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98/test_framework.log
2024-04-01T22:49:44.132000Z TestFramework (ERROR):
2024-04-01T22:49:44.133000Z TestFramework (ERROR): Hint: Call /home/chris/Documents/Code/bitcoin-core/test/functional/combine_logs.py '/mnt/tmp/test_runner_₿_🏃_20240402_004231/wallet_importdescriptors_98' to consolidate all logs
2024-04-01T22:49:44.133000Z TestFramework (ERROR):
2024-04-01T22:49:44.133000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-01T22:49:44.133000Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
2024-04-01T22:49:44.133000Z TestFramework (ERROR):
stderr:
Remaining jobs: [feature_pruning.py, feature_dbcrash.py, feature_assumeutxo.py, rpc_scantxoutset.py, feature_coinstatsindex.py, p2p_node_network_limited.py --v1transport, p2p_node_network_limited.py --v2transport, feature_config_args.py]
298/305 - p2p_node_network_limited.py --v1transport passed, Duration: 24 s
```
</details>
## Related
Almost identical timeout in `feature_index_prune.py` in #27091 on MacOS, and for `wallet_importdescriptors.py --descriptors` in #27282 on Alpine & CI.
ACKs for top commit:
maflcko:
lgtm ACK 49c0b8b228
tdb3:
ACK for 49c0b8b228
itornaza:
approach ACK 49c0b8b228
BrandonOdiwuor:
crACK 49c0b8b228
Tree-SHA512: f62ade74701588d76bfe838b7e7bbda1db38fd98688fd5d13c2c008064027add2ee9d053dee602d84919fab4c9bf53183c31819d94a6174066f237d0f6a62086
Reverts commit ab4efad51b (PR #26970).
This workaround is not needed anymore, as since #27114 the test sets
the noban permission for both in- and outbound connections via the
`noban_tx_relay` setting, and we don't have to rely on these topology
hacks anymore. See commit c985eb854c.
The `BITCOINFUZZ` environment variable allows to override the default
path to the fuzz binary.
It complements the already existing set of variables used by tests:
- BITCOIND
- BITCOINCLI
- BITCOINUTIL
- BITCOINWALLET
Rather than manually estimating an input's weight by adding up all the
involved components (fixed-size skeleton, compact-serialized lengths,
and the actual scriptSig / witness stack items) we can simply take use
of the serialization classes `CTxIn` / `CTxInWitness` instead, to
achieve the same with significantly less code.
The new helper is used in the functional tests rpc_psbt.py and
wallet_send.py, where the previous manual estimation code was
duplicated.
Previously, `wait_for_getheaders` would check whether a node had received **any**
getheaders message. This implied that, if a test needed to check for a specific block
hash within a headers message, it had to make sure that it was checking the desired message.
This normally involved having to manually clear `last_message`. This method, apart from being
too verbose, was error prone, given an undesired `getheaders` would make tests pass.
This adds the ability to check for a specific block_hash within the last `getheaders` message.
Timeout issues where encountered when running functional tests with `--jobs=16 --extended`.
Line in `feature_index_prune.py` took 101.6s, 96.6s, 103.0s across 3 runs on my machine, default limit is 60.
Line in the `wallet_importdescriptors.py --descriptors` took 5.4s, 5.7s, 6.0s across 3 runs.
61560d5e93 test: makes timeout a forced named argument in tests methods that use it (Sergi Delgado Segura)
Pull request description:
This makes calls to such methods more explicit and less error-prone.
Motivated by https://github.com/bitcoin/bitcoin/pull/29736#discussion_r1540654057
ACKs for top commit:
maflcko:
lgtm ACK 61560d5e93
brunoerg:
ACK 61560d5e93
BrandonOdiwuor:
crACK 61560d5e93
AngusP:
ACK 61560d5e93
stratospher:
tested ACK 61560d5.
Tree-SHA512: 8d6ec3fe1076c868ddbd3050f3c242dbd83cc123f560db3d3b0ed74968e6050dc9ebf4e7c716af9cc1b290c97d736c2fc2ac936b0b69ebdbceed934dae7d55d9
2eb5175de8 test: fix StopIteration exception in p2p_node_network_limited.py (furszy)
Pull request description:
Fixes#29731
The `next()` call throws an exception if the default parameter is omitted and the iterator is exhausted.
Fix it by providing a default value.
The failure can be tested by commenting out lines 90 and 91 in the test (the `self.connect_nodes(2, 0)`). Since there is no connection, the node in question retrieves a single element in the 'getchaintips()' call. This scenario without the fix, aborts the test right away, throwing an `StopIteration` exception, and with the fix, the test properly waits until the timeout (`wait_until()` call).
ACKs for top commit:
maflcko:
review ACK 2eb5175de8
brunoerg:
crACK 2eb5175de8
BrandonOdiwuor:
crACK 2eb5175de8
tdb3:
Tested ACK for 2eb5175de8.
Tree-SHA512: b0873eb4d3334146fd250cd2cd23add3e744877033c8bfa4eb8dff36633100604adf49dd7846856ddfa88c9768663f095db705c00eef3641618df8fc13f8c2c5
601edd8ee8 ci: use codespell 2.2.6 (fanquake)
52fa0d285f doc: fix some typos (crazeteam)
b5ed13a240 doc: Fix typos (RoboSchmied)
Pull request description:
Combines the recent PRs to fix typos so they can be merged.
ACKs for top commit:
brunoerg:
crACK 601edd8ee8
tdb3:
crACK 601edd8ee8
kristapsk:
cr utACK 601edd8ee8
Tree-SHA512: d054b1dad1336d6b9291cc5d5252d4debf6424a993d4edd6a97d7c15055a7fc48a333d30967f72e7dc9c6c1d9a9038ca8bb5e219c529f4c2365ea48404a508d0
746b6d8839 test: Add test for createwalletdescriptor (Ava Chow)
2402b63062 wallet: Test upgrade of pre-taproot wallet to have tr() descriptors (Ava Chow)
460ae1bf67 wallet, rpc: Add createwalletdescriptor RPC (Ava Chow)
8e1a475062 wallet: Be able to retrieve single key from descriptors (Ava Chow)
85b1fb19dd wallet: Add GetActiveHDPubKeys to retrieve xpubs from active descriptors (Ava Chow)
73926f2d31 wallet, descspkm: Refactor wallet descriptor generation to standalone func (Andrew Chow)
54e74f46ea wallet: Refactor function for single DescSPKM setup (Andrew Chow)
3b09d0eb7f tests: Test for gethdkeys (Ava Chow)
5febe28c9e wallet, rpc: Add gethdkeys RPC (Ava Chow)
66632e5c24 wallet: Add IsActiveScriptPubKeyMan (Ava Chow)
fa6a259985 desc spkm: Add functions to retrieve specific private keys (Ava Chow)
fe67841464 descriptor: Be able to get the pubkeys involved in a descriptor (Ava Chow)
ef6745879d key: Add constructor for CExtKey that takes CExtPubKey and CKey (Ava Chow)
Pull request description:
This PR adds a `createwalletdescriptor` RPC which allows users to add new automatically generated descriptors to their wallet, e.g. to upgrade a 0.21.x wallet to contain a taproot descriptor. This RPC takes 3 arguments: the output type to create a descriptor for, whether the descriptor will be internal or external, and the HD key to use if the user wishes to use a specific key. The HD key is an optional parameter. If it is not specified, the wallet will use the key shared by the active descriptors, if they are all single key. For most users in the expected upgrade scenario, this should be sufficient. In more advanced cases, the user must specify the HD key to use.
Currently, specified HD keys must already exist in the wallet. To make it easier for the user to know, `gethdkeys` is also added to list out the HD keys in use by all of the descriptors in the wallet. This will include all HD keys, whether we have the private key, for it, which descriptors use it and their activeness, and optionally the extended private key. In this way, users with more complex wallets will be still be able to get HD keys from their wallet for use in other scenarios, and if they want to use `createwalletdescriptor`, they can easily get the keys that they can specify to it.
See also https://github.com/bitcoin/bitcoin/pull/26728#issuecomment-1866961865
ACKs for top commit:
Sjors:
re-utACK 746b6d8839
furszy:
ACK 746b6d8
ryanofsky:
Code review ACK 746b6d8839, and this looks ready to merge. There were various suggested changes since last review where main change seems to be switching `gethdkeys` output to use normalized descriptors (removing hardened path components).
Tree-SHA512: f2849101e6fbf1f59cb031eaaaee97af5b1ae92aaab54c5716940d210f08ab4fc952df2725b636596cd5747b8f5beb1a7a533425bc10d09da02659473516fbda
The `next()` call throws an exception if the default parameter is omitted and the iterator is exhausted.
Fix it by providing a default value.
The failure can be tested by commenting out lines 90 and 91 in the test (the `self.connect_nodes(2, 0)``).
Since there is no connection, the node in question retrieves a single element in the 'getchaintips()' call.
This scenario without the fix, aborts the test right away, throwing an StopIteration exception, and with
the fix, the test properly waits until the timeout (wait_until() call).
5952292133 wallet, rpc: show mempool conflicts in `gettransaction` result (ishaanam)
54e07ee22f wallet: track mempool conflicts (ishaanam)
d64922b590 wallet refactor: use CWalletTx member functions to determine tx state (ishaanam)
ffe5ff1fb6 scripted-diff: wallet: s/TxStateConflicted/TxStateBlockConflicted (ishaanam)
180973a941 test: Add tests for wallet mempool conflicts (ishaanam)
Pull request description:
The `mempool_conflicts` variable is added to `CWalletTx`, it is a set of txids of txs in the mempool conflicting with the wallet tx or a wallet tx's parent. This PR only changes how mempool-conflicted txs are dealt with in memory.
`IsSpent` now returns false for an output being spent by a mempool conflicted transaction where it previously returned true.
A txid is added to `mempool_conflicts` during `transactionAddedToMempool`. A txid is removed from `mempool_conflicts` during `transactionRemovedFromMempool`.
This PR also adds a `mempoolconflicts` field to the `gettransaction` wallet RPC result.
Builds on #27145
Second attempt at #18600
ACKs for top commit:
achow101:
ACK 5952292133
ryanofsky:
Code review ACK 5952292133. Just small suggested changes since last review
furszy:
ACK 59522921
Tree-SHA512: 615779606723dbb6c2e302681d8e58ae2052ffee52d721ee0389746ddbbcf4b4c4afacf01ddf42b6405bc6f883520524186a955bf6b628fe9b3ae54cffc56a29
Upstream repo: https://github.com/arun11299/cpp-subprocess
Commit: 4025693decacaceb9420efedbf4967a04cb028e7
The "Convenience Functions" section is unused in our codebase, so it has
been removed.
Test the ban list is preserved through restart has been
done by both `rpc_setban` and `p2p_disconnect_ban`.
Since `p2p_disconnect_ban` does it in a more elegant
way, we can keep only it and remove the duplicated one.
a8bfc3dea1 test: add coverage for bech32m in `wallet_keypool_topup` (brunoerg)
Pull request description:
0dcac51049 added coverage for all keypool addresses types in `wallet_keypool_topup` (4y ago). Now we have bech23m, so this PR adds it.
ACKs for top commit:
achow101:
ACK a8bfc3dea1
marcofleon:
ACK a8bfc3dea1. Definitely a more straightfoward addition to the test. Reviewed the code, built the PR branch and ran all functional tests without issues.
furszy:
utACK a8bfc3dea
Tree-SHA512: aa830b723a7a54b23744f9fb3cf5214452c4ffc8e3bbe0e8bd980bdf902e61c3dd2fd57361b82c5c0c5224aa0774158daf34b6b2188edda0a971f82111976051
fa1146d01b lint: Fix COMMIT_RANGE issues (MarcoFalke)
Pull request description:
`COMMIT_RANGE` has problems on forks or local branches:
* When `LOCAL_BRANCH` is set, it assumes the presence of a `master` branch, and that the `master` branch is up-to-date. Both of which may be false. (See also discussion in https://github.com/bitcoin/bitcoin/pull/29274#discussion_r1504226422)
* When `COMMIT_RANGE` isn't set in `lint-git-commit-check.py`, and `--prev-commits` isn't set either, it has the same (broken) assumptions.
Fix all issues by simply assuming a merge commit exists. This allows to drop `LOCAL_BRANCH`. It also allows to drop `SKIP_EMPTY_NOT_A_PR`, because scripts will already skip an empty range. Finally, it allows to drop `--prev-commits n`, because one can simply say `COMMIT_RANGE='HEAD~n..HEAD'` to achieve the same.
ACKs for top commit:
Sjors:
tACK fa1146d01b
Tree-SHA512: f1477a38267fd4fdb8d396211a5d6bed5f418798c7edaba43487957aaf726cd45244ccf15187b3dd896d398fa1df3fe0a37323e49cf44d60a2018786ed41e5ba
The addpeeraddress calls can fail due to collisions. As we are using a
deteministic addrman, they won't fail with the current bucket/position
calculation. However, if the calculation is changed, they might collide
and fail silently causing tests using `seed_addrman()` to fail.
Assert that the addpeeraddress calls are successful.
824f47294a node: Use log levels in noui_ThreadSafeMessageBox (TheCharlatan)
ddc7872c08 node: Make translations of fatal errors consistent (TheCharlatan)
Pull request description:
The extra `bilingual_str` argument of the fatal error notifications and `node::AbortNode()` is often unused and when used usually contains the same string as the message argument. It also seems to be confusing, since it is not consistently used for errors requiring user action. For example some assumeutxo fatal errors require the user to do something, but are not translated.
So simplify the fatal error and abort node interfaces by only passing a translated string. This slightly changes the fatal errors displayed to the user.
ACKs for top commit:
stickies-v:
re-ACK 824f47294a
maflcko:
ACK 824f47294a🔎
achow101:
ACK 824f47294a
hebasto:
re-ACK 824f47294a.
Tree-SHA512: 2868ee7b045fe7f3ac582ce5039141b398480b7627734976201dafaaef7544b8461635a7292fee4a7f32ff1bfc26f9bd4d0c292dca424ba42fb7fc4483d7ce8d
99954f914f test: fix test to ensure hidden RPC is present in detailed help (stratospher)
0d01f6f0c6 test: remove unused mocktime in test_addpeeraddress (0xb10c)
6205466512 rpc: "addpeeraddress tried" return error on failure (0xb10c)
Pull request description:
When trying to add an address to the IP address manager tried table, it's first added to the new table and then moved to the tried table. Previously, adding a conflicting address to the address manager's tried table with test-only `addpeeraddress tried=true` RPC would return `{ "success": true }`. However, the address would not be added to the tried table, but would remain in the new table. This caused, e.g., issue #28964.
This is fixed by new returning `{ "success": false, "error": "..." }` for failed tried table additions. Since the address remaining in the new table can't be removed (the address manager interface does not support removing addresses at the moment and adding this seems to be a bigger effort), an error message is returned. This indicates to a user why the RPC failed and allows accounting for the extra address in the new table. This is done in the functional test for the `getrawaddrman` RPC.
Fixes#28964
ACKs for top commit:
achow101:
ACK 99954f914f
stratospher:
reACK 99954f9. 🚀
brunoerg:
utACK 99954f914f
Tree-SHA512: 2f1299410c0582ebc2071271ba789a8abed905f9a510821f77afbcf2a555ec31397578ea55cbcd162fb828be27afedd3246c7b13ad8883f2f745bb8e04364a76
The test requires that limited nodes are not peered with when
the node's system time exceeds ~ 24h of the node's chaintip
timestamp, as per PeerManagerImpl::GetDesirableServiceFlags.
By patching this test to modify the timestamp of the chaintip as
opposed to mocking the node's system time, we make it resilient
to future commits where the node raises a warning if it detects
its system time is too much out of sync with its outbound peers.
See https://github.com/bitcoin/bitcoin/pull/29623
The extra `bilingual_str` argument of the fatal error notifications and
`node::AbortNode()` is often unused and when used usually contains the
same string as the message argument. It also seems to be confusing,
since it is not consistently used for errors requiring user action. For
example some assumeutxo fatal errors require the user to do something,
but are not translated.
So simplify the fatal error and abort node interfaces by only passing a
translated string. This slightly changes the fatal errors displayed to
the user.
Also de-duplicate the abort error log since it is repeated in noui.cpp.
Behavior changes are:
- if a tx has a mempool conflict, the wallet will not attempt to
rebroadcast it
- if a txo is spent by a mempool-conflicted tx, that txo is no
longer considered spent
9d9a7458a2 assumeutxo: Remove BLOCK_ASSUMED_VALID flag (Ryan Ofsky)
ef174e9ed2 test: assumeutxo snapshot block CheckBlockIndex crash test (Ryan Ofsky)
0391458d76 test: assumeutxo stale block CheckBlockIndex crash test (Ryan Ofsky)
ef29c8b662 assumeutxo: Get rid of faked nTx and nChainTx values (Ryan Ofsky)
9b97d5bbf9 doc: Improve comments describing setBlockIndexCandidates checks (Ryan Ofsky)
0fd915ee6b validation: Check GuessVerificationProgress is not called with disconnected block (Ryan Ofsky)
63e8fc912c ci: add getchaintxstats ubsan suppressions (Ryan Ofsky)
f252e687ec assumeutxo test: Add RPC test for fake nTx and nChainTx values (Ryan Ofsky)
Pull request description:
The `PopulateAndValidateSnapshot` function introduced in f6e2da5fb7 from #19806 has been setting fake `nTx` and `nChainTx` values that can show up in RPC results (https://github.com/bitcoin/bitcoin/issues/29328) and make `CBlockIndex` state hard to reason about, because it is difficult to know whether the values are real or fake.
Revert to previous behavior of setting `nTx` and `nChainTx` to 0 when the values are unknown, instead of faking them. Also drop no-longer needed `BLOCK_ASSUMED_VALID` flag.
Dropping the faked values also fixes assert failures in the `CheckBlockIndex` `(pindex->nChainTx == pindex->nTx + prev_chain_tx)` check that could happen previously if forked or out-of-order blocks before the snapshot got submitted while the snapshot was being validated. The PR includes two commits adding tests for these failures and describing them in detail.
Compatibility note: This change could cause new `-checkblockindex` failures if a snapshot was loaded by a previous version of Bitcoin Core and not fully validated, because fake `nTx` values will have been saved to the block index. It would be pretty easy to avoid these failures by adding some compatibility code to `LoadBlockIndex` and changing `nTx` values from 1 to 0 when they are fake (when `(pindex->nStatus & BLOCK_VALID_MASK) < BLOCK_VALID_TRANSACTIONS`), but a little simpler not to worry about being compatible in this case.
ACKs for top commit:
Sjors:
re-ACK 9d9a7458a2
achow101:
ACK 9d9a7458a2
mzumsande:
Tested ACK 9d9a7458a2
maflcko:
ACK 9d9a7458a2🎯
Tree-SHA512: b1e1e2731ec36be30d5f5914042517219378fc31486674030c29d9c7488ed83fb60ba7095600f469dc32f0d8ba79c49ff7706303006507654e1762f26ee416e0
dfcef536d0 blockstorage: do not flush block to disk if it is already there (Matthew Zipkin)
Pull request description:
Closes https://github.com/bitcoin/bitcoin/issues/2039
When reindexing from flat-file block storage there is no need to write anything back to disk, since the block data is already there. This PR skips flushing to disk those blocks that already have a known position in the datastore. Skipping this means that users can write-protect the `blk` files on disk which may be useful for security or even safely sharing that data between multiple bitcoind instances.
`FindBlockPos()` may also flush the undo data file, but again this is skipped if the corresponding block position is known, like during the initial stage of a reindex when block data is being indexed. Once the block index is complete the validation mechanism will call `ConnectBlock()` which will save undo data at that time.
The call stack looks like this:
```
init()
ThreadImport() <-- process fReindex flag
LoadExternalBlockFile()
AcceptBlock()
SaveBlockToDisk()
FindBlockPos()
FlushBlockFile() <-- unnecessary if block is already on disk
```
A larger refactor of this part of the code was started by mzumsande here: https://github.com/mzumsande/bitcoin/tree/202207_refactor_findblockpos including this fix, reviewers can let me know if the changes should be combined.
ACKs for top commit:
sipa:
utACK dfcef536d0
mzumsande:
re-ACK dfcef536d0
achow101:
ACK dfcef536d0
furszy:
Rebase diff ACK dfcef53.
Tree-SHA512: 385c5ac1288b325135398d0ddd3ab788fa98cc0ca19bd2474c74039f2ce70d5088c1d1c9d4dd10aefcbd4c757767ec5805d07ba8cee9289a66f96e6f9eaa5279
2f23987849 test: p2p: check limited peers desirability (depending on best block depth) (Sebastian Falbesoner)
c4a67d396d test: p2p: check disconnect due to lack of desirable service flags (Sebastian Falbesoner)
405ac819af test: p2p: support disconnect waiting for `add_outbound_p2p_connection` (Sebastian Falbesoner)
Pull request description:
This PR adds missing test coverage for disconnecting peers which don't offer the desirable service flags in their VERSION message:
5f3a0574c4/src/net_processing.cpp (L3384-L3389)
This check is relevant for the connection types "outbound-full-relay", "block-relay-only" and "addr-fetch" (see `CNode::ExpectServicesFromConn(...)`). Feeler connections always disconnect, which is also tested here.
In lack of finding a proper file where this test would fit in, I created a new one. Happy to take suggestions there.
ACKs for top commit:
davidgumberg:
reACK 2f23987849
itornaza:
tested ACK 2f23987849
fjahr:
re-utACK 2f23987849
cbergqvist:
re ACK 2f23987849
stratospher:
tested ACK 2f23987. 🚀
Tree-SHA512: cf75d9d4379d0f34fa1e13152e6a8d93cd51b9573466ab3a2fec32dc3e1ac49b174bd1063cae558bc736b111c8a6e7058b1b57a496df56255221bf367d29eb5d
current check to make sure that detailed help for hidden RPC
is displayed won't work because the assertion isn't sufficient.
Even if unknown RPCs are passed, RPC names would still be present
in node.help().
Drops the mocktime added in fa4c6836c9.
Setting the mocktime in test_addpeeraddress() isn't needed
anymore as it doesn't leak into test_getrawaddrman() anymore
(since 2cc8ca19f4).
test_getrawaddrman() clear's the addrman and sets it's own
mocktime.
When trying to add an address to the IP address manager tried table,
it's first added to the new table and then moved to the tried table.
Previously, adding a conflicting address to the address manager's
tried table with test-only `addpeeraddress tried=true` RPC would
return `{ "success": true }`. However, the address would not be added
to the tried table, but would remain in the new table. This caused,
e.g., issue 28964.
This is fixed by returning `{ "success": false, "error":
"failed-adding-to-tried" }` for failed tried table additions. Since
the address remaining in the new table can't be removed (the address
manager interface does not support removing addresses at the moment
and adding this seems to be a bigger effort), an error message is
returned. This indicates to a user why the RPC failed and allows
accounting for the extra address in the new table.
Also:
To check the number of addresses in each addrman table,
the addrman checks were re-run and the log output of this check
was asserted. Ideally, logs shouldn't be used as an interface
in automated tests. To avoid asserting the logs, use the getaddrmaninfo
and getrawaddrman RPCs (which weren't implemented when the test was added).
Removing the "getnodeaddress" calls would also remove the addrman checks
from the test, which could reduce the test coverage. To avoid this,
these are kept.
432a542e27 test: fix intermittent failures with test=addrman (Martin Zumsande)
Pull request description:
The `nKey` of the addrman is generated the first time the node is started with an empty `peers.dat`. Therefore, restarting a node or turning it off and on again won't make a previously non-deterministic addrman deterministic.
This could lead to intermittent failures in `feature_asmap.py` and `rpc_net.py`
Fixes#29634
ACKs for top commit:
kevkevinpal:
ACK [432a542](432a542e27)
stratospher:
Tested ACK 432a542e27.
brunoerg:
crACK 432a542e27
0xB10C:
ACK 432a542e27
Tree-SHA512: a8e284baeb0be2df7284b8a2792cb9edc9e2d5b877a3b29ab7277ffdb75b17efa58a4d42576441eb493cd518e7c5ffdb05597b27e42b5001cf1a80e78bb04c83
38f70ba6ac RPC: Add maxfeerate and maxburnamount args to submitpackage (Greg Sanders)
Pull request description:
Resolves https://github.com/bitcoin/bitcoin/issues/28949
I couldn't manage to do it very cleanly outside of (sub)package evaluation itself, since it would change the current interface very heavily. Instead I threaded through the max fee argument and used that directly via ATMPArgs. From that perspective, this is somewhat a reversion from https://github.com/bitcoin/bitcoin/pull/19339. In a post-cluster mempool world, these checks could be consolidated to right after the given (ancestor) package is linearized/chunked, by just checking the feerate of the top chunk and rejecting the submission entirely if the top chunk is too high.
The implication here is that subpackages can be submitted to the mempool prior to hitting this new fee-based error condition.
ACKs for top commit:
ismaelsadeeq:
Re-ACK 38f70ba6ac👍🏾
glozow:
ACK 38f70ba6ac with some non-blocking nits
murchandamus:
LGTM, code review ACK 38f70ba6ac
Tree-SHA512: 38212aa9de25730944cee58b0806a3d37097e42719af8dd7de91ce86bb5d9770b6f7c37354bf418bd8ba571c52947da1dcdbb968bf429dd1dbdf8715315af18f
Add a test for a CheckBlockIndex crash that would happen before previous
"assumeutxo: Get rid of faked nTx and nChainTx values" commit.
The crash was an assert failure in the (pindex->nChainTx == pindex->nTx +
prev_chain_tx) check that would previously happen if the snapshot block was
submitted after loading the snapshot and downloading a few blocks after the
snapshot. In that case ReceivedBlockTransactions() previously would overwrite
the nChainTx value of the submitted snapshot block with a fake value based on
the previous block, so the (pindex->nChainTx == pindex->nTx + prev_chain_tx)
check would later fail on the first block after the snapshot. This test was
originally posted by Martin Zumsande <mzumsande@gmail.com> in
https://github.com/bitcoin/bitcoin/pull/29370#issuecomment-1974096225
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
Add a test for a CheckBlockIndex crash that would happen before previous
"assumeutxo: Get rid of faked nTx and nChainTx values" commit.
The crash was an assert failure in the (pindex->nChainTx == pindex->nTx +
prev_chain_tx) check that would previously happen if a snapshot was loaded, and
a block was submitted which forked from the chain before the snapshot block and
after the last downloaded background chain block. This block would not be
marked assumed-valid because it would not be an ancestor of the snapshot, and
it would have nTx set, nChainTx unset, and prev->nChainTx set with a fake
value, so the assert would fail. After the fix, prev->nChainTx is unset instead
of being set to a fake value, so the assert succeeds. This test was originally
posted by maflcko in
https://github.com/bitcoin/bitcoin/issues/29261#issuecomment-1918947945
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
The `PopulateAndValidateSnapshot` function introduced in
f6e2da5fb7 from #19806 has been setting fake
`nTx` and `nChainTx` values that can show up in RPC results (see #29328) and
make `CBlockIndex` state hard to reason about, because it is difficult to know
whether the values are real or fake.
Revert to previous behavior of setting `nTx` and `nChainTx` to 0 when the
values are unknown, instead of faking them.
This commit fixes at least two assert failures in the (pindex->nChainTx ==
pindex->nTx + prev_chain_tx) check that would happen previously. Tests for
these failures are added separately in the next two commits.
Compatibility note: This change could result in -checkblockindex failures if a
snapshot was loaded by a previous version of Bitcoin Core and not fully
validated, because fake nTx values will have been saved to the block index. It
would be pretty easy to avoid these failures by adding some compatibility code
to `LoadBlockIndex` and changing `nTx` values from 1 to 0 when they are fake
(when `(pindex->nStatus & BLOCK_VALID_MASK) < BLOCK_VALID_TRANSACTIONS`), but a
little simpler not to worry about being compatible in this case.
Add ubsan suppressions for integer overflows in the getchaintxstats RPC.
getchainstatstx line "int nTxDiff = pindex->nChainTx - past_block.nChainTx" can
trigger ubsan integer overflows when assumeutxo snapshots are loaded, from
subtracting unsigned values and assigning the result to a signed int.
The overflow behavior probably exists in current code but is hard to trigger
because it would require calling getchainstatstx at the right time with
specific parameters as background blocks are being downloaded. But the overflow
behavior becomes easier to trigger in the upcoming commit removing fake
nChainTx values, so a suppression needs to be added before then for CI to pass.
getchainstatstx should probably be improved separately in another PR to not
need this suppression, and handle edge cases and missing nChainTx values more
carefully.
5555395c15 lint: Use git --no-pager to print any output in one go (MarcoFalke)
fa5729436c lint: Fix lint-whitespace issues (MarcoFalke)
Pull request description:
The lint check has many issues:
* It uses `COMMIT_RANGE`, which is brittle code, apparently making it harder to run the CI locally, or self-hosted. See https://github.com/bitcoin/bitcoin/pull/29274#discussion_r1457739319
* The result depends on `COMMIT_RANGE`, or the number of commits passed to the script, which can cause false negatives or false positives.
* It is based on the diff output, parsing it, and printing it again, which is brittle as well.
* The output does not include line number, making it harder to act on a lint error.
Fix all issues by removing the script and replacing it with a simple call to `git grep -I --line-number ...`.
ACKs for top commit:
TheCharlatan:
Re-ACK 5555395c15
Tree-SHA512: 71ea8b6382af064beb72fb17f21a0ae9e9238c97e7fa43c2ec353fd1dd73a7bbd696ba0f0a9f65d1eff7c86cbf6cc104a992cb5450a3d50f122955e835270065
bf264e0598 test: check_mempool_result negative feerate (kevkevin)
Pull request description:
Adds test coverage in `mempool_accept.py` to check if a negative `maxfeerate` is input into `check_mempool_result`
Asserts "Amount out of range" error message and `-3` error code
Motivated by this [comment](https://github.com/bitcoin/bitcoin/pull/29434/files#r1491112250)
ACKs for top commit:
maflcko:
lgtm ACK bf264e0598
brunoerg:
nice, utACK bf264e0598
davidgumberg:
Looks great, ACK bf264e0598
Tree-SHA512: 58931b774cc887c616f2fd91af3ee65cc5db55acd8e2875c76de448c80bd4e020b057c5f4f85556431377f0d0e7553771fb285d1ec20cf64f64ec92a47776b78
0831b54dfc test: simplify test_runner.py (tdb3)
Pull request description:
Implements the simplifications to test_runner.py proposed by sipa in PR #23995.
Remove the num_running variable as it can be implied by the length of the jobs list.
Remove the i variable as it can be implied by the length of the test_results list.
Instead of counting results to determine if finished, make the queue object itself
responsible (by looking at running jobs and jobs left).
ACKs for top commit:
mzumsande:
re-ACK 0831b54
davidgumberg:
reACK 0831b54dfc
marcofleon:
re-ACK 0831b54dfc
Tree-SHA512: e5473e68d49cd779b29d97635329283ae7195412cb1e92461675715ca7eedb6519a1a93ba28d40ca6f015d270f7bcd3e77cef279d9cd655155ab7805b49638f1
1ec6684b08 test: Add test for loadtxoutset when headers are not synced (Fabian Jahr)
2bc1ecfaa9 test: Remove unnecessary sync_blocks in assumeutxo tests (Fabian Jahr)
Pull request description:
It adds a test for the change to `loadtxoutset` made in #29345. Before that change the test doesn't fail right away but times out after 10 minutes.
Also removes a `sync_blocks` call that didn't seem to do anything valuable.
ACKs for top commit:
achow101:
ACK 1ec6684b08
pablomartin4btc:
tACK 1ec6684b08
BrandonOdiwuor:
ACK 1ec6684b08
theStack:
ACK 1ec6684b08
Tree-SHA512: 1337decdf91e4a4f7213fcf8ace1d705e5c1422e0ac3872a59b5be9c33e743850cb8f5f7474750a534952eefd5cfe43fe85a54efb9bc0e47515136a2903676e5
And thread the feerate value through ProcessNewPackage to
reject individual transactions that exceed the given
feerate. This allows subpackage processing, and is
compatible with future package RBF work.
567cec9a05 doc: add release notes and help text for unix sockets (Matthew Zipkin)
bfe5192891 test: cover UNIX sockets in feature_proxy.py (Matthew Zipkin)
c65c0d0163 init: allow UNIX socket path for -proxy and -onion (Matthew Zipkin)
c3bd43142e gui: accomodate unix socket Proxy in updateDefaultProxyNets() (Matthew Zipkin)
a88bf9dedd i2p: construct Session with Proxy instead of CService (Matthew Zipkin)
d9318a37ec net: split ConnectToSocket() from ConnectDirectly() for unix sockets (Matthew Zipkin)
ac2ecf3182 proxy: rename randomize_credentials to m_randomize_credentials (Matthew Zipkin)
a89c3f59dc netbase: extend Proxy class to wrap UNIX socket as well as TCP (Matthew Zipkin)
3a7d6548ef net: move CreateSock() calls from ConnectNode() to netbase methods (Matthew Zipkin)
74f568cb6f netbase: allow CreateSock() to create UNIX sockets if supported (Matthew Zipkin)
bae86c8d31 netbase: refactor CreateSock() to accept sa_family_t (Matthew Zipkin)
adb3a3e51d configure: test for unix domain sockets (Matthew Zipkin)
Pull request description:
Closes https://github.com/bitcoin/bitcoin/issues/27252
UNIX domain sockets are a mechanism for inter-process communication that are faster than local TCP ports (because there is no need for TCP overhead) and potentially more secure because access is managed by the filesystem instead of serving an open port on the system.
There has been work on [unix domain sockets before](https://github.com/bitcoin/bitcoin/pull/9979) but for now I just wanted to start on this single use-case which is enabling unix sockets from the client side, specifically connecting to a local Tor proxy (Tor can listen on unix sockets and even enforces strict curent-user-only access permission before binding) configured by `-onion=` or `-proxy=`
I copied the prefix `unix:` usage from Tor. With this patch built locally you can test with your own filesystem path (example):
`tor --SocksPort unix:/Users/matthewzipkin/torsocket/x`
`bitcoind -proxy=unix:/Users/matthewzipkin/torsocket/x`
Prep work for this feature includes:
- Moving where and how we create `sockaddr` and `Sock` to accommodate `AF_UNIX` without disturbing `CService`
- Expanding `Proxy` class to represent either a `CService` or a UNIX socket (by its file path)
Future work:
- Enable UNIX sockets for ZMQ (https://github.com/bitcoin/bitcoin/pull/27679)
- Enable UNIX sockets for I2P SAM proxy (some code is included in this PR but not tested or exposed to user options yet)
- Enable UNIX sockets on windows where supported
- Update Network Proxies dialog in GUI to support UNIX sockets
ACKs for top commit:
Sjors:
re-ACK 567cec9a05
tdb3:
re ACK for 567cec9a05.
achow101:
ACK 567cec9a05
vasild:
ACK 567cec9a05
Tree-SHA512: de81860e56d5de83217a18df4c35297732b4ad491e293a0153d2d02a0bde1d022700a1131279b187ef219651487537354b9d06d10fde56225500c7e257df92c1
Remove the num_running variable as it can be implied by the
length of the jobs list.
Remove the i variable as it can be implied by the length of the
test_results list.
Instead of counting results to determine if finished, make the
queue object itself responsible (by looking at running jobs and
jobs left).
Originally proposed by @sipa in PR #23995.
Co-authored-by: Pieter Wuille <pieter@wuille.net>
The nKey of the addrman is generated the first time the node is
started. Therefore, restarting a node or turning it off and on
again won't make a previously non-deterministic addrman
deterministic.
Co-authored-by: 0xb10c <b10c@b10c.me>
e710cefd57 rest: read raw block in rest_block and deserialize for json (Andrew Toth)
95ce0783a6 rpc: read raw block in getblock and deserialize for verbosity > 0 (Andrew Toth)
0865ab8712 test: check more details on zmq raw block response (Andrew Toth)
38265cc14e zmq: read raw block with ReadRawBlockFromDisk (Andrew Toth)
da338aada7 blockstorage: check nPos in ReadRawBlockFromDisk before seeking back (Andrew Toth)
Pull request description:
For the `getblock` endpoint with `verbosity=0`, the `rest_block` REST endpoint for `bin` and `hex`, and zmq `NotifyBlock` we don't have to deserialize the block since we're just sending the raw data. This PR uses `ReadRawBlockFromDisk` instead of `ReadBlockFromDisk` to serve these requests, and only deserializes for `verbosity > 0` and `json` REST requests. See benchmarks in https://github.com/bitcoin/bitcoin/pull/26684.
Benchmarked using ApacheBench. Requesting block 750,000 in binary 10k times on a single core (set `-rest=1` in config):
`ab -n 10000 -c 1 "http://127.0.0.1:8332/rest/block/0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e.bin"`
On master, mean time 15ms.
On this branch, mean time 1ms.
For RPC
```
echo '{"jsonrpc": "1.0", "id": "curltest", "method": "getblock", "params": ["0000000000000000000592a974b1b9f087cb77628bb4a097d5c2c11b3476a58e", 0]}' > /tmp/data.json
ab -p /tmp/data.json -n 1000 -c 1 -A user:password "http://127.0.0.1:8332/"
```
On master, mean time 32ms
On this branch, mean time 13ms
ACKs for top commit:
achow101:
re-ACK e710cefd57
Tree-SHA512: 4cea13c7b563b2139d041b1fdcfdb793c8cc688654ae08db07e7ee6b875c5e582b8185db3ae603abbfb06d2164724f29205774620b48c493726b991999af289e
0a533613fb docs: add release notes for #27114 (brunoerg)
e6b8f19de9 test: add coverage for whitelisting manual connections (brunoerg)
c985eb854c test: add option to speed up tx relay/mempool sync (brunoerg)
66bc6e2d17 Accept "in" and "out" flags to -whitelist to allow whitelisting manual connections (Luke Dashjr)
8e06be347c net_processing: Move extra service flag into InitializeNode (Luke Dashjr)
9133fd69a5 net: Move `NetPermissionFlags::Implicit` verification to `AddWhitelistPermissionFlags` (Luke Dashjr)
2863d7dddb net: store `-whitelist{force}relay` values in `CConnman` (brunoerg)
Pull request description:
Revives #17167. It allows whitelisting manual connections. Fixes#9923
Since there are some PRs/issues around this topic, I'll list some motivations/comments for whitelisting outbound connections from them:
- Speed-up tx relay/mempool sync for testing purposes (my personal motivation for this) - In #26970, theStack pointed out that we whitelist peers to speed up tx relay for fast mempool synchronization, however, since it applies only for inbound connections and considering the topology `node0 <--- node1 <---- node2 <--- ... <-- nodeN`, if a tx is submitted from any node other than node0, the mempool synchronization can take quite long.
- https://github.com/bitcoin/bitcoin/pull/29058#issuecomment-1865155764 - "Before enabling -v2transport by default (which I'd image may happen after https://github.com/bitcoin/bitcoin/pull/24748) we could consider a way to force manual connections to be only-v1 or even only-v2 (disabling reconnect-with-v1). A possibility could be through a net permission flag, if https://github.com/bitcoin/bitcoin/pull/27114 makes it in."
- https://github.com/bitcoin/bitcoin/pull/17167#issuecomment-1168606032 - "This would allow us to use https://github.com/bitcoin/bitcoin/pull/25355 when making outgoing connections to all nodes, except to whitelisted ones for which we would use our persistent I2P address."
- Force-relay/mempool permissions for a node you intentionally connected to.
ACKs for top commit:
achow101:
ACK 0a533613fb
sr-gi:
re-ACK [0a53361](0a533613fb)
pinheadmz:
ACK 0a533613fb
Tree-SHA512: 97a79bb854110da04540897d2619eda409d829016aafdf1825ab5515334b0b42ef82f33cd41587af235b3af6ddcec3f2905ca038b5ab22e4c8a03d34f27aebe1
