This RPC allows the client to retrieve the file system locations
of the confirmed blocks and their undo data, to allow building
efficient indexes outside of Bitcoin Core.
An example usage is described here:
https://github.com/romanz/electrs/issues/308
By using the new RPC, it is possible to build an address-based
index taking ~24GB and a txindex taking ~6GB (as of Dec. 2020).
by skipping the part where we send a non-version message
before the version - this message would be interpreted as
part of the v2 handshake.
Github-Pull: #28805
Rebased-From: 35fb9930ad
- "transport_protocol_type" of inbound peer before version handshake
is "detecting" on p2p v2 nodes (as opposed to "v1" for p2p v1)
- size of a ping/pong message is 29 bytes (as opposed to 32 for p2p v1)
- for the sendmsgtopeer RPC sub-test, enforce p2p v1 connection to
have a peer id of zero
Github-Pull: #28805
Rebased-From: 2c1669c37a
By renaming the "command" send_cli arg. The old name was unsuitable
because the "addnode" RPC has its own "command" arg, leading to
ambiguity when included in kwargs.
Can be tested with
"python3 wallet_multiwallet.py --usecli --v2transport"
which fails on master because of this (python throws a TypeError).
Github-Pull: #28805
Rebased-From: 3598a1b5c9
Before, a global -v2transport provided to the test would be dropped
when restarting the node within a test and specifying any extra_args.
Fix this by adding "v2transport=1" to args (not extra_args) based
on the global parameter, and deciding for each (re)start of the node
based on this default and test-specific extra_args
(which take precedence over args) whether v2 should be used.
Github-Pull: #28805
Rebased-From: 68a9001751
Previously, starting a second bitcoind using the same datadir would
correctly fail to init and shutdown. However during shutdown the PID
file belonging to the first instance would be erroneously removed by
the second process shutting down.
Fix this to only delete the PID file if we created it.
Github-Pull: #28946
Rebased-From: 8f6ab31863
Otherwise, starting bitcoind twice may cause the `.cookie`
file generated by the first instance to be deleted by the
second instance shutdown (after failing to obtain a lock).
Github-Pull: bitcoin/bitcoin#28784
Rebased-From: 7cb9367157
Test that wallet rescans process transactions topologically, even if a
parent's entry into the mempool is later than that of its child.
This behavior is important because IsFromMe requires the ability to look
up a transaction's inputs.
Co-authored-by: furszy <matiasfurszyfer@protonmail.com>
Github-Pull: #29179
Rebased-From: df30247705
Test that wallet rescans process transactions topologically, even if a
parent's entry into the mempool is later than that of its child.
This behavior is important because IsFromMe requires the ability to look
up a transaction's inputs.
Github-Pull: #29179
Rebased-From: c3d02be536
sort_multisig test: check uncompressed keys are disallowed
sort_multisig: add test demonstrating sorting
wallet_labels: test addmultisigaddress fails if sort=true and (wallet) address is uncompressed
Verifying the wallet updates the birth time accordingly when it
detects a transaction with a time older than the oldest descriptor
timestamp.
This could happen when the user blindly imports a descriptor with
'timestamp=now'.
Github-Pull: #28920
Rebased-From: 83c66444d0
Currently the HTTP server initialization (`HTTPBindAddresses`) fails
only when *all* bindings fail. So if multiple binds are specified
(`127.0.0.1` and `::1` by defeault) and one succeeds and the other
fails, the latter is essentially ignored.
This commit changes the error behavior to fail *if not all* binds could
be performed, which I think is more in line with how software normally
handles this and what users expect.
Add a validation for cli-side commands for the follwing cases:
* duplication of cli-command (and options: at the moment a user
can specify -rpcwallet many times, only the last one will be
taken into account,
* only 1 cli-command can run at a time (eg can't run -generate and
-getinfo at the same time),
* no params starting with slash "-" will be accepted after a cli-command
(this is the case for bitcoin-cli -generate 3 -rpcwallet=xyz).
Github-Pull: #26990
Rebased-From: 755320f75f2141909e84b62f420462c1c5b193e6
The primary objective is to provide users with clearer
and more informative error messages when encountering
the RPC_WALLET_NOT_SPECIFIED error, which occurs when
multiple wallets are loadad.
This commit also rectifies the error message consistency
by bringing the error message in line with the definition
established in protocol.h ("error when there are multiple
wallets loaded").
Github-Pull: #26990
Rebased-From: bf48c40c0ee62eb6343f8aff06a88322a33f2dd6
5a0688a20d test: enable reindex readonly test on *BSD and macOS as root (Matthew Zipkin)
Pull request description:
see https://github.com/bitcoin/bitcoin/pull/27850#discussion_r1349505585
OpenBSD and FreeBSD don't have `chattr` but they do have `chflags`, use that method to make the block file immutable for the reindex_readonly test.
Written and tested on a VPS running FreeBSD:
```
FreeBSD freebsd-13-1 13.2-RELEASE-p4 FreeBSD 13.2-RELEASE-p4 GENERIC amd64
```
ACKs for top commit:
maflcko:
re-cr-lgtm-ACK 5a0688a20d
jonatack:
ACK 5a0688a20d tested on macOS only
theStack:
ACK 5a0688a20d
Tree-SHA512: 8c88d282d09c00355d22c4c504b779f60e420327a5e07bcf80fa77b97fefcb04952af9ceaf439d9033a0a2448cb26a02663fe6bddcd4a74792857cfbaf1c5162
4814e4063e test: Check tx metadata is migrated to watchonly (Andrew Chow)
d616d30ea5 wallet: Reload watchonly and solvables wallets after migration (Andrew Chow)
118f2d7d70 wallet: Copy all tx metadata to watchonly wallet (Andrew Chow)
9af87cf348 test: Check that a failed wallet migration is cleaned up (Andrew Chow)
Pull request description:
Some incomplete/incorrect state as a result of migration can be mitigated/cleaned up by simply restarting the migrated wallets. We already do this for a wallet when it is migrated, but we do not for the new watchonly and solvables wallets that may be created. This PR introduces this behavior, in addition to creating those wallets initially without an attached chain.
While implementing this, I noticed that not all `CWalletTx` metadata was being copied over to the watchonly wallet and so some data, such as time received, was being lost. This PR fixes this as a side effect of not having a chain attached to the watchonly wallet. A test has also been added.
ACKs for top commit:
ishaanam:
light code review ACK 4814e4063e
ryanofsky:
Code review ACK 4814e4063e. Just implemented the suggested orderpos, copyfrom, and path set comments since last review
furszy:
ACK 4814e406
Tree-SHA512: 0b992430df9f452cb252c2212df8e876613f43564fcd1dc00c6c31fa497adb84dfff6b5ef597590f9b288c5f64cb455f108fcc9b6c9d1fe9eb2c39e7f2c12a89
4bfaad4eca chainparams, assumeutxo: Fix signet txoutset hash (Fabian Jahr)
a503cd0f0b chainparams, assumeutxo: Fix testnet txoutset hash (Fabian Jahr)
f6213929c5 assumeutxo: Check deserialized coins for out of range values (Fabian Jahr)
66865446a7 docs: Add release notes for #28685 (Fabian Jahr)
cb0336817e scripted-diff: Rename hash_serialized_2 to hash_serialized_3 (Fabian Jahr)
351370a1d2 coinstats: Fix hash_serialized2 calculation (Fabian Jahr)
Pull request description:
Closes#28675
The last commit demonstrates that theStack's analysis [here](https://github.com/bitcoin/bitcoin/issues/28675#issuecomment-1770389468) seems to be correct. There will be more changes needed for the rest of the test suite but the `feature_assumeutxo.py` with my additional tests pass.
ACKs for top commit:
achow101:
ACK 4bfaad4eca
theStack:
Code-review ACK 4bfaad4eca
ryanofsky:
Code review ACK 4bfaad4eca
Tree-SHA512: 2f6abc92b282f7c5da46391803cf0804d13978d191d541f2509b532c538abccd0a081e46cda23d80d47206a05fa2b5d41b7ab246e6a263db7a7461d6292116ef
fa21535551 fuzz: Increase merge -rss_limit_mb (MarcoFalke)
Pull request description:
For some reason, the limit is hit. (Presumably due to `-set_cover_merge=1` eating more memory, or by simply having more fuzz inputs).
Fix it by increasing it for the merge operation.
ACKs for top commit:
dergoegge:
ACK fa21535551
hebasto:
ACK fa21535551, considering the discussion in https://github.com/bitcoin-core/qa-assets/pull/155.
Tree-SHA512: 4fed0f254eccc6fe0b53656bc345ff898b13811dc39387387317d34b521ab77cee03d82b0896dd92d253b7546b6a7e4bdcd478749f47064374ab44ad759ab9ff
The legacy serialization was vulnerable to maleation and is fixed by
adopting the same serialization procedure as was already in use for
MuHash.
This also includes necessary test fixes where the hash_serialized2 was
hardcoded as well as correction of the regtest chainparams.
Co-authored-by: Sebastian Falbesoner <sebastian.falbesoner@gmail.com>
When migrating, create the watchonly and solvables wallets without a
context. Then unload and reload them after migration completes, as we do
for the actual wallet.
There is also additional handling for a failed reload.
004903ebad test: Add Wallet Unlock Context Manager (Brandon Odiwuor)
Pull request description:
Fixes#28601, see https://github.com/bitcoin/bitcoin/pull/28403#discussion_r1325426430
Add Context Manager to manage the locking and unlocking of locked wallets with a passphrase during testing.
ACKs for top commit:
kevkevinpal:
lgtm ACK [004903e](004903ebad)
maflcko:
lgtm ACK 004903ebad
Tree-SHA512: ab234c167e71531df0d974ff9a31d444f7ce2a1d05aba5ea868cc9452f139845eeb24ca058d88f058bc02482b762adf2d99e63a6640b872cc71a57a0068abfe8
fa4c6836c9 test: Fix failing time check in rpc_net.py (MarcoFalke)
Pull request description:
This check fails on slow runners, such as s390x qemu.
Fix it by using mocktime.
See https://github.com/bitcoin/bitcoin/pull/28523#discussion_r1357980527
ACKs for top commit:
0xB10C:
ACK fa4c6836c9
pinheadmz:
ACK fa4c6836c9
brunoerg:
crACK fa4c6836c9
Tree-SHA512: 83fb534682e11e97537dc89de8c16f206f38af1a892a2d5970c02684c05eaea8fc9adba3159f16b2440ca0b3871d513a0562a6f3a38f19a5574a47be0919e42f
Instead of hardcoding the `pruneblockchain(<height>)` heights,
use 'getblockfileinfo' to obtain the highest block number of
each of the block files.
Making the test more robust and readable by stating which file
is being pruned at every point of time (the goal is to mimic
how the automatic pruning process work).
621db2f004 test: assumeutxo file with unknown block hash (Fabian Jahr)
Pull request description:
Takes care of one of the open Todos in the assumeutxo functional test. Since an unknown block could be any hash, I simply chose one placeholder, it could also be a random string though.
ACKs for top commit:
maflcko:
lgtm ACK 621db2f004
pablomartin4btc:
cr ACK 621db2f004
theStack:
ACK 621db2f004
ryanofsky:
Code review ACK 621db2f004
Tree-SHA512: ee0438ce619f7348c6f88e39b0ea7ddddb8832956d9034ecc795c6033d5d905c09d11b7d0d5afc38231b2fd091ea7c1bd0a0be99d9c32c4e6357a25d76294142
9620cb4493 assumeutxo: fail early if snapshot block hash doesn't match AssumeUTXO parameters (Sebastian Falbesoner)
Pull request description:
Right now the `loadtxoutset` RPC call treats literally all files with a minimum size of 40 bytes (=size of metadata) as potential valid snapshot candidates and the waiting loop for seeing the metadata block hash in the headers chain is always entered, e.g.:
```
$ ./src/bitcoin-cli loadtxoutset ~/.vimrc
<wait>
bitcoind log:
...
2023-10-15T14:55:45Z [snapshot] waiting to see blockheader 626174207465730a7265626d756e207465730a656c62616e65207861746e7973 in headers chain before snapshot activation
...
```
There is no point in doing any further action though if we already know from the start that the UTXO snapshot loading won't be successful. This PR adds an assumeutxo parameter check immediately after the metadata is read in, so we can fail immediately on a mismatch:
```
$ ./src/bitcoin-cli loadtxoutset ~/.vimrc
error code: -32603
error message:
Unable to load UTXO snapshot, assumeutxo block hash in snapshot metadata not recognized (626174207465730a7265626d756e207465730a656c62616e
65207861746e7973)
```
This way, users who mistakenly try to load files that are not snapshots don't have to wait 10 minutes (=the block header waiting timeout) anymore to get a negative response. If a file is loaded which is a valid snapshot (referencing to an existing block hash), but one which doesn't match the parameters, the feedback is also faster, as we don't have to wait anymore to see the hash in the headers chain before getting an error.
This is also partially fixes#28621.
ACKs for top commit:
maflcko:
lgtm ACK 9620cb4493
ryanofsky:
Code review ACK 9620cb4493. This should fix an annoyance and bad UX.
pablomartin4btc:
tACK 9620cb4493
Tree-SHA512: f88b865e9d46254858e57c024463f389cd9d8760a7cb30c190aa1723a931e159987dfc2263a733825d700fa612e7416691e4d8aab64058f1aeb0a7fa9233ac9c
faa5e061c2 fuzz: Allow multiple --m_dir args (MarcoFalke)
Pull request description:
This allows to merge the result from several servers (or just several folders) at the same time, instead of having to iterate over them.
This should also allow the fuzz engine (libFuzzer) to optimize the final merge result more, because all fuzz inputs from all folders are available at the same time.
ACKs for top commit:
dergoegge:
tACK faa5e061c2
Tree-SHA512: bf0da418b1f7b8a8af16bb7cc1e148b1ccd0f17062ce70758d1ca5b35c3eee77c0c30377d376befdd55480adfd1f1a1073cfc47118e7a710e6760e020abe24bb
348e79f7c6 lint: Include test_utxo_snapshots in lint_shell (Fabian Jahr)
Pull request description:
jamesob excluded `test_utxo_snapshots.sh` from the shell linter with this explanation: "Add the script to the shellcheck exception list since the quoted variables rule needs to be violated in order to get bitcoind to pick up on $EARLY_IBD_FLAGS." However, macrofake pointed out that single lines can be excluded from linting.
This fixes one fixable rule violation, excludes the rest of the offending lines from the linter and then removes the exclusion of the `test_utxo_snapshots.sh` file. Also adds documentation.
ACKs for top commit:
Empact:
ACK 348e79f7c6
maflcko:
lgtm ACK 348e79f7c6
pablomartin4btc:
tACK 348e79f7c6
Tree-SHA512: a904cc1cc3c94488dfbd39ea69a3ef17259f991708a797009001669448fef81eed086ecbce1ec433988d88baef293849698e2e0eb86a969b949cc7ef93af7b4b
fa858d63a0 fuzz: Merge with -set_cover_merge=1 (MarcoFalke)
Pull request description:
This should be less controversial than commit 151a2b189c. The overall size of the qa-assets repo is reduced further from 1.9GB to 1.6GB. Also, the runtime to iterate on the resulting folder is reduced further from ~1699s to ~1149s (N=1).
ACKs for top commit:
murchandamus:
crACK fa858d63a0
dergoegge:
ACK fa858d63a0
Tree-SHA512: e23fa93bd48f01d11c551b035004c678bd6d76bc24ac7d0d0a7883060804e6711763cbd0cd0ded3aad3e4c40da764decae81c2703388cc11961def3c89a4f9ba
Two recently added tests (PR #28625 / commit 2e31250027
and PR #28634 / commit 3bb51c29df)
introduced a bug by wrongly using the `assert_debug_log` helper.
Instead of passing the expected debug string in a list as expected, it
was passed as bare string, which is then interpretered as a list of
characters, very likely leading the debug log assertion pass even if the
intended message is not appearing.
In order to avoid bugs like this in the future, enforce that the
`{un}expected_msgs` parameters are lists.
4077e43bf6 test: fix usdt undeclared function errors on mantis (willcl-ark)
Pull request description:
This is one way to fix#28600
Recently usage of undeclared functions became an error rather than a warning, in C2x. https://reviews.llvm.org/D122983?id=420290
This change has migrated into the build tools of Ubuntu 23.10 which now causes the USDT tests to fail to compile, see
https://github.com/bitcoin/bitcoin/issues/28600
I think there are various potential fixes:
1. Manually declare the functions we use
2. Fix imports so that manual declarations aren't needed
3. Revert the new C2X behaviour and don't error on implicit function declarations
I would have preferred solution 2, but I believe this will require changes to the upstream bcc package. Having played with the imports I can get things working in a standalone C program, using system headers, but when building the program from a python context as we do in the test it uses its own headers (bundled with the python lib) rather than the system ones, and manually importing (some) system headers results in definition mismatches. I also investigated explicitly importing required headers from the package, which use paths like `#import </virtual/bcc/bcc_helpers.h>`, but this seems more obtuse and brittle than simply ignoring the warning.
Therefore I think that until the upstream python pacakge fixes their declarations, we should fix this by setting `-Wno-error=implicit-function-declaration` for the tracing programs.
cc maflcko 0xB10C
ACKs for top commit:
maflcko:
lgtm ACK 4077e43bf6
Tree-SHA512: 8368bb1155e920a95db128dc893267f8dab64f1ae53f6d63c6d9294e2e4e92bef8515e3697e9113228bedc51c0afdbc5bbcf558c119bf0eb3293dc2ced86b435
850670e3d6 test: don't run old binaries under valgrind (Sjors Provoost)
Pull request description:
Some, but not all, backward compatibility tests fail for me and it seems useless to run old release binaries under valgrind anyway.
Can be tested by running `test/functional/feature_txindex_compatibility.py --valgrind --timeout-factor=10` with and without this PR.
—
The previous version of this PR disabled these test entirely under valgrind. The current version does run the test, but starts the old binaries without valgrind.
ACKs for top commit:
maflcko:
lgtm ACK 850670e3d6
Tree-SHA512: ebdf461083f1292528e6619963b910f486b60b4f6b183f0aea2c8bfcafa98caeb204d138700cd288450643bcec5e49e12b89f2f7537fccdf495a2a33acd9cea0
3bb51c29df test: BIP324: add check for missing garbage terminator detection (Sebastian Falbesoner)
Pull request description:
This PR adds test coverage for the "missing garbage terminator" detection on incoming v2 transport (BIP324) connections:
04265ba937/src/net.cpp (L1205-L1209)
Note that this always happens at the same exact amount of bytes sent in (after 64 + 4095 + 16 = 4175 bytes), if at no point, the last 16 bytes of potential authentication data match the garbage, i.e. all the previous bytes after the ellswift pubkey. To keep it simple, we just send in zero-value bytes here and verify that the detection hits exactly after the last bytes is sent.
AFAICT, with this PR all the v2 transport errors that can be triggered in this simple way of "just open a socket and send in a fixed byte-string" are covered. For more advanced test, we need BIP324 cryptography in the test framework in order to perform a v2 handshake etc. (PRs #28374, #24748).
ACKs for top commit:
sipa:
utACK 3bb51c29df
laanwj:
ACK 3bb51c29df
Tree-SHA512: f88275061c7c377a3d9f2608452671afc26deb6d5bd5be596de987c7e5042555153ffe681760c33bce2b921ae04e50f349ea0128a677e6443a95a079e52cdc5f
This is unnecessary and caused test failures. The backward
compatibility tests are meant to find regressions in the
current codebase, not to detect bugs in older releases.
2e31250027 test: check that loading snapshot not matching AssumeUTXO parameters fails (Sebastian Falbesoner)
Pull request description:
This PR adds test coverage for the failed loading of an AssumeUTXO snapshot in case the referenced block hash doesn't match the parameters in the chainparams. Right now, I expect this would be the most common error-case for `loadtxoutset` out in the wild, as for mainnet the `m_assumeutxo_data` map is empty and this error condition would obviously always be triggered for any (otherwise valid, correctly encoded) snapshot. Note that this test-case is the simplest scenario and doesn't cover any of the TODO ideas mentioned at the top of the functional test yet.
ACKs for top commit:
jamesob:
ACK 2e31250027
Sjors:
utACK 2e31250027
achow101:
ACK 2e31250027
Tree-SHA512: 8bcb2d525c95fbc95f87d3e978ad717d95bddb1ff67cbe7d3b06e4783f0f1ffba32b17ef451468c39c23bc1b3ef1150baa71148c145275c386f2d4822d790d39
bfa0bd632a test: Use pathlib over os.path #28362 (ns-xvrn)
Pull request description:
In reference to issue #28362 refactoring of functional tests to use pathlib over os.path to reduce verbosity and increase the intuitiveness of managing file access.
ACKs for top commit:
maflcko:
re-ACK bfa0bd632a🐨
willcl-ark:
ACK bfa0bd632a
Tree-SHA512: fb0833c4039d09758796514e47567a93ac831cb0776ff1a7ed8299792ad132e83282ef80bea098a88ae1551d906f2a56093d3e8de240412f9080735d00d496d9
74c77825e5 test: Unit test for inferring scripts with hybrid and uncompressed keys (Andrew Chow)
f895f97014 test: Scripts with hybrid pubkeys are migrated to watchonly (Andrew Chow)
37b9b73477 descriptors: Move InferScript's pubkey validity checks to InferPubkey (Andrew Chow)
b7485f11ab descriptors: Check result of InferPubkey (Andrew Chow)
Pull request description:
`InferDescriptor` was not always checking that the pubkey it was placing into the descriptor was an allowed pubkey. For example, given a P2WPKH script that uses an uncompressed pubkey, it would produce a `wpkh()` with the uncompressed key. Additionally, the hybrid key check was only being done for `pk()` scripts, where it should've been done for all scripts.
This PR moves the key checking into `InferPubkey`. If the key is not valid for the context, then `nullptr` is returned and the inferring will fall through to the defaults of either `raw()` or `addr()`.
This also resolves an issue with migrating legacy wallets that contain hybrid pubkeys as such watchonly scripts will become `raw()` or `addr()` and go to the watchonly wallet. Note that a legacy wallet cannot sign for hybrid pubkeys. A test has been added for the migration case.
Also added unit tests for `InferDescriptor` itself as the edge cases with that function are not covered by the descriptor roundtrip test.
ACKs for top commit:
furszy:
ACK 74c77825
Sjors:
utACK 74c77825e5
Tree-SHA512: ed5f63e42a2e46120245a6b0288b90d2a6912860814c6c08fe393332add1cb364dc5eca72f16980352143570aef0c07bf1a91acd294099463bd028b6ce2fe40c
