Commit Graph

23120 Commits

Author SHA1 Message Date
fanquake
058eb69ce4
build: add missing event cflags to libbitcoin_util
The fact that this is missing is currently masked by the inclusion of
BOOST_CPPFLAGS.
2022-10-31 10:10:53 +00:00
furszy
fa84df1f03
scripted-diff: wallet: rename AvailableCoinsParams members to snake_case
-BEGIN VERIFY SCRIPT-

sed -i 's/nMinimumAmount/min_amount/g' $(git grep -l nMinimumAmount)
sed -i 's/nMaximumAmount/max_amount/g' $(git grep -l nMaximumAmount)
sed -i 's/nMinimumSumAmount/min_sum_amount/g' $(git grep -l nMinimumSumAmount)
sed -i 's/nMaximumCount/max_count/g' $(git grep -l nMaximumCount)

-END VERIFY SCRIPT-
2022-10-29 08:51:34 -03:00
furszy
61c2265629
wallet: group AvailableCoins filtering parameters in a single struct
Plus clean callers that use the params default values
2022-10-29 08:50:38 -03:00
furszy
f0f6a3577b
RPC: listunspent, add "include immature coinbase" flag
so we can return the immature coinbase UTXOs as well.
2022-10-29 08:45:12 -03:00
Andrew Chow
8781a1b6bb psbt: Include output pubkey in additional pubkeys to sign
In addition to the pubkeys in hd_keypaths and tap_bip32_keypaths, also
see if the descriptor can produce a SigningProvider for the output
pubkey.

Also slightly refactors this area to reduce code duplication.
2022-10-28 20:03:22 -04:00
Andrew Chow
323890d0d7 sign: Fill in taproot pubkey info for all script path sigs
Taproot pubkey info was not being added for multi_a signing. The filling
of this info is moved into the common function CreateTaprootScriptSig so
that any signing of taproot scripts will include the pubkey info.
2022-10-28 20:03:22 -04:00
Andrew Chow
8b050762b1
Merge bitcoin/bitcoin#26409: refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts
fa29ef00ad refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts (MacroFake)

Pull request description:

  The `std::optional` fields in the struct that fall back to chain param defaults if not provided should be initialized to `std::nullopt`. This already happens with the current code.

  However, for consistency with `check_block_index` and to silence a GCC warning, add the "missing" `{}`.

ACKs for top commit:
  achow101:
    ACK fa29ef00ad
  hebasto:
    ACK fa29ef00ad, tested on Ubuntu 22.04 + GCC 11.3.
  jonatack:
    ACK fa29ef00ad

Tree-SHA512: bdec9c56df5d601a5616e107fed48737b13b0a7242b6526092fb682b5016544a4bc08666b60304c668d44c6f7ac69d3788093d921382c1d6c577c1f9fe31fc50
2022-10-28 15:37:17 -04:00
MacroFake
1bad29fe02
Merge bitcoin/bitcoin#26377: test: Make system_tests/run_command test locale and platform agnostic
884304e6c6 test: Make `system_tests/run_command` locale agnostic (Hennadii Stepanov)

Pull request description:

  Fixes bitcoin/bitcoin#26368.

ACKs for top commit:
  Sjors:
    tACK 884304e6c6

Tree-SHA512: 76d4941e02b3b119dcf4dacbe60ef45a9dc8cf775bdb31b5291cd8147665285d41caaf1f5688abdfc9a47c393ddb535af7b11af839660d30ef30f1ca0d936133
2022-10-28 11:32:13 +02:00
MacroFake
fa29ef00ad
refactor: Silence GCC Wmissing-field-initializers in ChainstateManagerOpts 2022-10-28 11:08:20 +02:00
omahs
180eac0f73 Fix: typos
Fix: typos

Fix: typos

Fix: typos
2022-10-28 09:39:36 +02:00
Andrew Chow
f37bd15d47
Merge bitcoin/bitcoin#25685: wallet: Faster transaction creation by removing pre-set-inputs fetching responsibility from Coin Selection
3fcb545ab2 bench: benchmark transaction creation process (furszy)
a8a75346d7 wallet: SelectCoins, return early if target is covered by preset-inputs (furszy)
f41712a734 wallet: simplify preset inputs selection target check (furszy)
5baedc3351 wallet: remove fetch pre-selected-inputs responsibility from SelectCoins (furszy)
295852f619 wallet: encapsulate pre-selected-inputs lookup into its own function (furszy)
37e7887cb4 wallet: skip manually selected coins from 'AvailableCoins' result (furszy)
94c0766b0c wallet: skip available coins fetch if "other inputs" are disallowed (furszy)

Pull request description:

  #### # Context (Current Flow on Master)

  In the transaction creation process, in order to select which coins the new transaction will spend,
  we first obtain all the available coins known by the wallet, which means walking-through the
  wallet txes map, gathering the ones that fulfill certain spendability requirements in a vector.

  This coins vector is then provided to the Coin Selection process, which first checks if the user
  has manually selected any input (which could be internal, aka known by the wallet, or external),
  and if it does, it fetches them by searching each of them inside the wallet and/or inside the
  Coin Control external tx data.

  Then, after finding the pre-selected-inputs and gathering them in a vector, the Coin Selection
  process walks-through the entire available coins vector once more just to erase coins that are
  in both vectors. So the Coin Selection process doesn’t pick them twice (duplicate inputs inside
  the same transaction).

  #### # Process Workflow Changes

  Now, a new method, `FetchCoins` will be responsible for:
  1) Lookup the user pre-selected-inputs (which can be internal or external).
  2) And, fetch the available coins in the wallet (excluding the already fetched ones).

  Which will occur prior to the Coin Selection process. Which allows us to never include the
  pre-selected-inputs inside the available coins vector in the first place, as well as doing other
  nice improvements (written below).

  So, Coin Selection can perform its main responsibility without mixing it with having to fetch
  internal/external coins nor any slow and unneeded duplicate coins verification.

  #### # Summarizing the Improvements:

  1) If any pre-selected-input lookup fail, the process will return the error right away.
      (before, the wallet was fetching all the wallet available coins, walking through the
      entire txes map, and then failing for an invalid pre-selected-input inside SelectCoins)

  2) The pre-selected-inputs lookup failure causes are properly described on the return error.
      (before, we were returning an "Insufficient Funds" error for everything, even if the failure
      was due a not solvable external input)

  3) **Faster Coin Selection**: no longer need to "remove the pre-set inputs from the available coins
      vector so that Coin Selection doesn't pick them" (which meant to loop-over the entire
      available coins vector at Coin Selection time, erasing duplicate coins that were pre-selected).

      Now, the available coins vector, which is built after the pre-selected-inputs fetching,
      doesn’t include the already selected inputs in the first place.

  4) **Faster transaction creation** for transactions that only use manually selected inputs.

      We now will return early, as soon as we finish fetching the pre-selected-inputs and
      not perform the resources expensive calculation of walking-through the entire wallet
      txes map to obtain the available coins (coins that we will not use).

  ---------------------------

  Added a new bench (f6d0bb2) measuring the transaction creation process, for a wallet with ~250k UTXO, only using the pre-selected-inputs inside coin control. Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically.

  #### Result on this PR (tip f6d0bb2d):

  |               ns/op |                op/s |    err% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------:|:----------
  |        1,048,675.00 |              953.58 |    0.3% |      0.06 | `WalletCreateTransaction`

  vs

  #### Result on master (tip 4a4289e2):

  |               ns/op |                op/s |    err% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------:|:----------
  |       96,373,458.20 |               10.38 |    0.2% |      5.30 | `WalletCreateTransaction`

  The benchmark took to run in master: **96.37 milliseconds**, while in this PR: **1 millisecond**  🚀 .

ACKs for top commit:
  S3RK:
    Code Review ACK 3fcb545ab2
  achow101:
    ACK 3fcb545ab2
  aureleoules:
    reACK 3fcb545ab2

Tree-SHA512: 42f833e92f40c348007ca565a4c98039e6f1ff25d8322bc2b27115824744779baf0b0a38452e4e2cdcba45076473f1028079bbd0f670020481ec5d3db42e4731
2022-10-27 17:48:58 -04:00
Andrew Chow
551c8e9526
Merge bitcoin/bitcoin#26349: rpc: make address field optional list{transactions, sinceblock} response
eb679a7896 rpc: make `address` field optional (w0xlt)

Pull request description:

  Close https://github.com/bitcoin/bitcoin/issues/26338.

  This PR makes optional the `address` field in the response of `listtransactions` and `listsinceblock` RPC.
  And adds two tests that fail on master, but not on this branch.

ACKs for top commit:
  achow101:
    ACK eb679a7896
  aureleoules:
    ACK eb679a7896

Tree-SHA512: b267439626e2ec3134ae790c849949a4c40ef0cebd20092e8187be3db0a61941b2da10bbbba92ca880b8369f46c1aaa806d057eaa5159325f65cbec7cb33c52f
2022-10-27 13:17:39 -04:00
MacroFake
fa24239a1c
net: Avoid SetTxRelay for feeler connections 2022-10-27 16:09:33 +02:00
Hennadii Stepanov
39710f5635
Merge bitcoin-core/gui#665: Switch to the selected wallet after loading
b8b59ff9fe gui: update the screen after loading wallet (w0xlt)

Pull request description:

  Currently, the user loads a wallet and the screen does not switch to the selected wallet after loading (File -> Open Wallet -> wallet name).

  This PR changes that by making the `OpenWalletActivity::opened` signal connection a `Qt::QueuedConnection` type.

ACKs for top commit:
  jarolrod:
    ACK b8b59ff9fe
  hebasto:
    ACK b8b59ff9fe, tested on Ubuntu 22.04.

Tree-SHA512: 43cd755638b643f481014a7933a0af25df2d109e859cb5f878bc04e562950d550716fa38465140060e28526b2441688580cbcbe4ec6819566b4f95162ca5e527
2022-10-27 13:56:42 +01:00
glozow
2242de16cc
Merge bitcoin/bitcoin#26394: Fix typo in comment SHA256->SHA512
0cc23fc603 Fix typo in comment SHA256->SHA512 (Elichai Turkel)

Pull request description:

  The comment says it's the SHA-256 state, while it's actually the SHA-512 state

ACKs for top commit:
  andrewtoth:
    ACK 0cc23fc603
  aureleoules:
    ACK 0cc23fc603

Tree-SHA512: 4e390ceefb847d3bbe4f5caab390a4fdd14892fe443f58c32b08b3444fccd611cff22938c3dfa611dfd2497736f779fae4165497b4208e48aa8fc9d2236f943b
2022-10-27 11:02:22 +01:00
furszy
3fcb545ab2
bench: benchmark transaction creation process
Goal 1:
Benchmark the transaction creation process for pre-selected-inputs only.
Setting `m_allow_other_inputs=false` to disallow the wallet to include coins automatically.

Goal 2:
Benchmark the transaction creation process for pre-selected-inputs and coin selection.

-----------------------

Benchmark Setup:
1) Generates a 5k blockchain, loading the wallet with 5k transactions with two outputs each.
2) Fetch 4 random UTXO from the wallet's available coins and pre-select them as inputs inside CoinControl.

Benchmark (Goal 1):
Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=false` and
the manually selected coins.

Benchmark (Goal 2):
Call `CreateTransaction` providing the coin control, who has set `m_allow_other_inputs=true` and
the manually selected coins.
2022-10-26 15:54:31 -03:00
furszy
a8a75346d7
wallet: SelectCoins, return early if target is covered by preset-inputs 2022-10-26 15:54:31 -03:00
furszy
f41712a734
wallet: simplify preset inputs selection target check
we are already computing the preset inputs total amount inside `PreSelectedInputs::Insert`,
which internally decides whether to use the effective value or the raw output value based on
the 'subtract_fee_outputs' flag.
2022-10-26 15:54:31 -03:00
furszy
5baedc3351
wallet: remove fetch pre-selected-inputs responsibility from SelectCoins
so if there is an error in any of the pre-set coins, we can fail right away
without computing the wallet available coins set (calling `AvailableCoins`)
which is a slow operation as it goes through the entire wallet's txes map.

----------------------

And to make the Coin Selection flow cleared, have decoupled SelectCoins in two functions:

1) AutomaticCoinSelection.
2) SelectCoins.

1) AutomaticCoinSelection:
   Receives a set of coins and selects the best subset of them to
   cover the target amount.

2) SelectCoins
   In charge of select all the user manually selected coins first ("pre-set inputs"), and
   if coin_control 'm_allow_other_inputs=true', call 'AutomaticCoinSelection' to select a
   subset of coins owned by the wallet to cover for the target - preset_inputs.total_amount
   remaining value.
2022-10-26 15:54:31 -03:00
furszy
295852f619
wallet: encapsulate pre-selected-inputs lookup into its own function
First step towards decoupling the pre-selected-inputs fetching functionality
from `SelectCoins`. Which, will let us not waste resources calculating the
available coins if one of the pre-set inputs has an error.

(right now, if one of the pre-set inputs is invalid, we first walk through
the entire wallet txes map just to end up failing right after it finish)
2022-10-26 15:52:35 -03:00
furszy
37e7887cb4
wallet: skip manually selected coins from 'AvailableCoins' result
No need to walk through the entire wallet's txes map just to get
coins that we could have gotten by just doing a simple map.find(out.hash).
(Which is what we are doing inside `SelectCoins` anyway)
2022-10-26 15:52:35 -03:00
furszy
94c0766b0c
wallet: skip available coins fetch if "other inputs" are disallowed
no need to waste resources calculating the wallet available coins if
they are not going to be used.

The 'm_allow_other_inputs=true` default value change is to correct
an ugly misleading behavior:

The tx creation process was having a workaround patch to automatically
fall back to select coins from the wallet if `m_allow_other_inputs=false`
(previous default value) and no manual inputs were selected.

This could be seen in master in flows like `sendtoaddress`, `sendmany`
and even the GUI, where the `m_allow_other_inputs` value isn't customized
and the wallet still selects and adds coins to the tx internally.
2022-10-26 15:47:51 -03:00
Andrew Toth
f5ff3d773c rpc: add missing lock around chainman.ActiveTip() 2022-10-26 11:46:39 -04:00
Andrew Chow
88502ecf08
Merge bitcoin/bitcoin#23927: rpc: Pruning nodes can not fetch blocks before syncing past their height
5826bf546e test: Add test for getblockfrompeer on syncing pruned nodes (Fabian Jahr)
7fa851fba8 rpc: Pruned nodes can not fetch unsynced blocks (Fabian Jahr)

Pull request description:

  This PR prevents `getblockfrompeer` from getting used on blocks that the node has not synced past yet if the node is in running in prune mode.

  ### Problem

  While a node is still catching up to the tip that it is aware of via the headers, the user can currently use  to fetch blocks close to or at the tip. These blocks are stored in the block/rev file that otherwise contains blocks the node is receiving as part of the syncing process.

  This creates a problem for pruned nodes: The files containing a fetched block are not pruned during syncing because they contain a block close to the tip. This means the entire file (~130MB) will not be pruned until the tip has moved on far enough from the fetched block. In extreme cases with heavy pruning (like 550) and multiple blocks being fetched this could mean that the disc usage far exceeds what the user expects, potentially running out of space.

  ### Approach

  There would be certainly other approaches that could fix the problem while still allowing the current behavior, but all of the ideas I came up with seemed like overkill for a niche problem on a new RPC where it's still unclear how and how much it will be used.

  ### Testing

  So far I did not see a simple enough way to test this I am still looking into it and if it's complex will potentially add it in a follow-up. What would be needed is a way to have a node fetch headers but not sync the blocks yet, that seems like a pattern that could be generally useful.

  To manually reproduce the problematic behavior:
  1. Start a node with current `master` with `-prune=550` and an empty/new datadir, Testnet and Mainnet should both work.
  2. While the node is syncing run `getblockfrompeer` on the current tip and a few other recent blocks.
  3. Go to your datadir and observe the blocks folder: There should be a few full `blk*.dat` and `rev*.dat` files that are not being pruned. When you "pinned" a few of these files the blocks folder should be significantly above the target size of 550MB.

ACKs for top commit:
  Sjors:
    utACK 5826bf546e
  achow101:
    ACK 5826bf546e
  aureleoules:
    tACK 5826bf546e

Tree-SHA512: aa3f477ec755a9df2331c047cb10b3cd08292522bf6ad7a36a7ea36d7eba4894b84de8bd23003c9baea5ac0c53b77142c3c2819ae7528cece9d10a0d06c850d8
2022-10-26 11:27:31 -04:00
Andrew Chow
48af307481
Merge bitcoin/bitcoin#25957: wallet: fast rescan with BIP157 block filters for descriptor wallets
0582932260 test: add test for fast rescan using block filters (top-up detection) (Sebastian Falbesoner)
ca48a4694f rpc: doc: mention rescan speedup using `blockfilterindex=1` in affected wallet RPCs (Sebastian Falbesoner)
3449880b49 wallet: fast rescan: show log message for every non-skipped block (Sebastian Falbesoner)
935c6c4b23 wallet: take use of `FastWalletRescanFilter` (Sebastian Falbesoner)
70b3513904 wallet: add `FastWalletRescanFilter` class for speeding up rescans (Sebastian Falbesoner)
c051026586 wallet: add method for retrieving the end range for a ScriptPubKeyMan (Sebastian Falbesoner)
845279132b wallet: support fetching scriptPubKeys with minimum descriptor range index (Sebastian Falbesoner)
088e38d3bb add chain interface methods for using BIP 157 block filters (Sebastian Falbesoner)

Pull request description:

  ## Description

  This PR is another take of using BIP 157 block filters (enabled by `-blockfilterindex=1`) for faster wallet rescans and is a modern revival of #15845. For reviewers new to this topic I can highly recommend to read the corresponding PR review club (https://bitcoincore.reviews/15845).

  The basic idea is to skip blocks for deeper inspection (i.e. looking at every single tx for matches) if our block filter doesn't match any of the block's spent or created UTXOs are relevant for our wallet. Note that there can be false-positives (see https://bitcoincore.reviews/15845#l-199 for a PR review club discussion about false-positive rates), but no false-negatives, i.e. it is safe to skip blocks if the filter doesn't match; if the filter *does* match even though there are no wallet-relevant txs in the block, no harm is done, only a little more time is spent extra.

  In contrast to #15845, this solution only supports descriptor wallets, which are way more widespread now than back in the time >3 years ago. With that approach, we don't have to ever derive the relevant scriptPubKeys ourselves from keys before populating the filter, and can instead shift the full responsibility to that to the `DescriptorScriptPubKeyMan` which already takes care of that automatically. Compared to legacy wallets, the `IsMine` logic for descriptor wallets is as trivial as checking if a scriptPubKey is included in the ScriptPubKeyMan's set of scriptPubKeys (`m_map_script_pub_keys`): e191fac4f3/src/wallet/scriptpubkeyman.cpp (L1703-L1710)

  One of the unaddressed issues of #15845 was that [the filter was only created once outside the loop](https://github.com/bitcoin/bitcoin/pull/15845#discussion_r343265997) and as such didn't take into account possible top-ups that have happened. This is solved here by keeping a state of ranged `DescriptorScriptPubKeyMan`'s descriptor end ranges and check at each iteration whether that range has increased since last time. If yes, we update the filter with all scriptPubKeys that have been added since the last filter update with a range index equal or higher than the last end range. Note that finding new scriptPubKeys could be made more efficient than linearly iterating through the whole `m_script_pub_keys` map (e.g. by introducing a bidirectional map), but this would mean introducing additional complexity and state and it's probably not worth it at this time, considering that the performance gain is already significant.

  Output scripts from non-ranged `DescriptorScriptPubKeyMan`s (i.e. ones with a fixed set of output scripts that is never extended) are added only once when the filter is created first.

  ## Benchmark results

  Obviously, the speed-up indirectly correlates with the wallet tx frequency in the scanned range: the more blocks contain wallet-related transactions, the less blocks can be skipped due to block filter detection.

  In a [simple benchmark](https://github.com/theStack/bitcoin/blob/fast_rescan_functional_test_benchmark/test/functional/pr25957_benchmark.py), a regtest chain with 1008 blocks (corresponding to 1 week) is mined with 20000 scriptPubKeys contained (25 txs * 800 outputs) each. The blocks each have a weight of ~2500000 WUs and hence are about 62.5% full. A global constant `WALLET_TX_BLOCK_FREQUENCY` defines how often wallet-related txs are included in a block. The created descriptor wallet (default setting of `keypool=1000`, we have 8*1000 = 8000 scriptPubKeys at the start) is backuped via the `backupwallet` RPC before the mining starts and imported via `restorewallet` RPC after. The measured time for taking this import process (which involves a rescan) once with block filters (`-blockfilterindex=1`) and once without block filters (`-blockfilterindex=0`) yield the relevant result numbers for the benchmark.

  The following table lists the results, sorted from worst-case (all blocks contain wallte-relevant txs, 0% can be skipped) to best-case (no blocks contain walltet-relevant txs, 100% can be skipped) where the frequencies have been picked arbitrarily:

  wallet-related tx frequency; 1 tx per...    | ratio of irrelevant blocks  | w/o filters | with filters | speed gain
  --------------------------------------------|-----------------------------|-------------|--------------|-------------
  ~ 10 minutes (every block)                  |              0%             |   56.806s   |   63.554s    |  ~0.9x
  ~ 20 minutes (every 2nd block)              |           50% (1/2)         |   58.896s   |   36.076s    |  ~1.6x
  ~ 30 minutes (every 3rd block)              |          66.67% (2/3)       |   56.781s   |   25.430s    |  ~2.2x
  ~ 1 hour (every 6th block)                  |          83.33% (5/6)       |   58.193s   |   15.786s    |  ~3.7x
  ~ 6 hours (every 36th block)                |          97.22% (35/36)     |   57.500s   |    6.935s    |  ~8.3x
  ~ 1 day (every 144th block)                 |         99.31% (143/144)    |   68.881s   |    6.107s    | ~11.3x
    (no txs)                                  |              100%           |   58.529s   |    5.630s    | ~10.4x

  Since even the (rather unrealistic) worst-case scenario of having wallet-related txs in _every_ block of the rescan range obviously doesn't take significantly longer, I'd argue it's reasonable to always take advantage of block filters if they are available and there's no need to provide an option for the user.

  Feedback about the general approach (but also about details like naming, where I struggled a lot) would be greatly appreciated. Thanks fly out to furszy for discussing this subject and patiently answering basic question about descriptor wallets!

ACKs for top commit:
  achow101:
    ACK 0582932260
  Sjors:
    re-utACK 0582932260
  aureleoules:
    ACK 0582932260 - minor changes, documentation and updated test since last review
  w0xlt:
    re-ACK 0582932260

Tree-SHA512: 3289ba6e4572726e915d19f3e8b251d12a4cec8c96d041589956c484b5575e3708b14f6e1e121b05fe98aff1c8724de4564a5a9123f876967d33343cbef242e1
2022-10-26 11:19:19 -04:00
Elichai Turkel
0cc23fc603
Fix typo in comment SHA256->SHA512 2022-10-26 15:55:29 +03:00
dergoegge
784b023191 [net processing] Simplify use of IsContinuationOfLowWorkHeadersSync in TryLowWorkHeaderSync
`m_headers_sync` is already reset in IsContinuationOfLowWorkHeadersSync
if there is a failure, so there is no need to also reset in
TryLowWorkHeaderSync.
2022-10-26 11:12:03 +01:00
MacroFake
a1fff275e7
Merge bitcoin/bitcoin#25704: refactor: Remove almost all validation option globals
aaaa7bd0ba iwyu: Add missing includes (MacroFake)
fa9ebec096 Remove g_parallel_script_checks (MacroFake)
fa7c834b9f Move ::fCheckBlockIndex into ChainstateManager (MacroFake)
fa43188d86 Move ::fCheckpointsEnabled into ChainstateManager (MacroFake)
cccca83099 Move ::nMinimumChainWork into ChainstateManager (MacroFake)
fa29d0b57c Move ::hashAssumeValid into ChainstateManager (MacroFake)
faf44876db Move ::nMaxTipAge into ChainstateManager (MacroFake)

Pull request description:

  It seems preferable to assign globals to a class (in this case `ChainstateManager`), than to leave them dangling. This should clarify scope for code-readers, as well as clarifying unit test behaviour.

ACKs for top commit:
  dergoegge:
    Code review ACK aaaa7bd0ba
  ryanofsky:
    Code review ACK aaaa7bd0ba. No changes since last review, other than rebase
  aureleoules:
    reACK aaaa7bd0ba

Tree-SHA512: 83ec3ba0fb4f1dad95810d4bd4e578454e0718dc1bdd3a794cc4e48aa819b6f5dad4ac4edab3719bdfd5f89cbe23c2740a50fd56c1ff81c99e521c5f6d4e898d
2022-10-26 11:41:57 +02:00
MacroFake
cf288377c0
Merge bitcoin/bitcoin#26275: Fix crash on deriveaddresses when index is 2147483647 (2^31-1)
9153ff3e27 rpc: add non-regression test about deriveaddresses crash when index is 2147483647 (muxator)
addf9d6502 rpc: fix crash in deriveaddresses when derivation index is 2147483647 (muxator)

Pull request description:

  This PR is a proposal for fixing #26274 (better described there).

  The problem is due to a signed int wrapping when the `index` parameter of the `deriveaddresses` RPC call has the value `2^31-1`.

  ```C++
  for (int i = range_begin; i <= range_end; ++i) {
  ```

  * the first commit adds a "temporary" test case (`test/functional/rpc_deriveaddresses_crash.py`) that shows the crash, and can be used to generate a core dump;
  * the second commit fixes the problem giving an explicit size to the `i` variable in a for loop, from `int` to `int64_t`. The same commit also removes the ephemeral test case and adds a passing test to `test/functional/rpc_deriveaddresses.py`, in order to prevent future regressions.

  This is my first submission to this project and I do not know its conventions. Please advise if something needs to be changed.

ACKs for top commit:
  achow101:
    ACK 9153ff3e27

Tree-SHA512: 0477b57b15dc2c682cf539d6002f100d44a8c7e668041aa3340c39dcdbd40e083c75dec6896b6c076b044a01c2e5254272ae6696d8a1467539391926f270940a
2022-10-26 10:12:27 +02:00
w0xlt
eb679a7896 rpc: make address field optional 2022-10-26 01:18:28 -03:00
fanquake
28cf756971
Merge bitcoin/bitcoin#23578: Add external signer taproot support
796b020c37 wallet: add taproot support to external signer (Sjors Provoost)

Pull request description:

  Builds on #22558 (merged on 2022-06-28).

  [HWI 2.1.0](https://github.com/bitcoin-core/HWI/releases/tag/2.1.0) or newer is required to import and use taproot descriptors. Older versions will work, but won't import a taproot descriptor.

  Tested with HWI 2.1.1:
  * Trezor T (firmware v2.5.1) on Signet: signs, change detection works
  * Ledger Nano S (firmware 2.1.0, Bitcoin app 2.0.6): signs, change detection works

  Only the most basic `tr(key)` descriptor is supported, script path spending is completely untested (if it works at all).

ACKs for top commit:
  jb55:
    utACK 796b020c37
  achow101:
    ACK 796b020c37

Tree-SHA512: 6dcb7eeb45421a3bbf2bdabeacd29979867db69077d7bf192bb77faa4bfefe446487b8df07bc40f9457009a88e598bdc09f769e6106fed2833ace7ef205a157a
2022-10-26 11:10:23 +08:00
Sebastian Falbesoner
ca48a4694f rpc: doc: mention rescan speedup using blockfilterindex=1 in affected wallet RPCs 2022-10-25 15:57:39 +02:00
Sebastian Falbesoner
3449880b49 wallet: fast rescan: show log message for every non-skipped block
For that purpose, a new logging category BCLog::SCAN is introduced.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
935c6c4b23 wallet: take use of FastWalletRescanFilter
Can be reviewed with `--ignore-all-space`.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
70b3513904 wallet: add FastWalletRescanFilter class for speeding up rescans
This only supports wallet descriptors right now.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
c051026586 wallet: add method for retrieving the end range for a ScriptPubKeyMan 2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
845279132b wallet: support fetching scriptPubKeys with minimum descriptor range index
This extra method will be needed for updating the filter set for
faster wallet rescans; after an internal top-up has happened, we only
want to add the newly created scriptPubKeys.
2022-10-25 15:57:38 +02:00
Sebastian Falbesoner
088e38d3bb add chain interface methods for using BIP 157 block filters
This is useful for speeding up wallet rescans and is based on an
earlier version from PR #15845 ("wallet: Fast rescan with BIP157 block
filters"), which was never merged.

Co-authored-by: MacroFake <falke.marco@gmail.com>
2022-10-25 15:57:28 +02:00
dergoegge
e891aabf5a [net processing] Fixup TryLowWorkHeadersSync comment 2022-10-24 22:05:59 +01:00
Larry Ruane
db929893ef Faster -reindex by initially deserializing only headers
When a block is initially read from a blk*.dat file during reindexing,
it can be added to the block index only if all of its ancestor blocks
have been added, which is rare. If the block's ancestors have not been
added, the block must be re-read from disk later when it can be added.

This commit: During the initial block read, deserialize only its header,
rather than the entire block, since this is sufficient to determine
if its parent (and thus all its ancestors) has been added. This is a
performance improvement.
2022-10-24 13:02:37 -06:00
Larry Ruane
c72de9990a util: add CBufferedFile::SkipTo() to move ahead in the stream
SkipTo() reads data from the file into the CBufferedFile object
(memory), but, unlike this object's read() method, SkipTo() doesn't
transfer data into a caller's memory buffer. This is useful because
after skipping forward in the stream in this way, the user can, if
needed, rewind the stream (SetPos()) and access the object's memory
buffer including ranges that were skipped over (without needing to
read from the disk file).
2022-10-24 13:02:37 -06:00
Larry Ruane
48a68908ba Add LoadExternalBlockFile() benchmark 2022-10-24 13:02:35 -06:00
glozow
3d0fca1288
Merge bitcoin/bitcoin#26355: p2p: Handle IsContinuationOfLowWorkHeadersSync return value correctly when new headers sync is started
7ad15d1100 [net processing] Handle IsContinuationOfLowWorkHeadersSync return value correctly when new headers sync is started (dergoegge)

Pull request description:

  This PR fixes a bug in the headers sync logic that enables submitting headers to a nodes block index that don't lead to a chain that surpasses our DoS limit.

  The issue is that we ignore the return value on [the first `IsContinuationOfLowWorkHeadersSync` call after a new headers sync is started](fabc031048/src/net_processing.cpp (L2553-L2568)), which leads to us passing headers to [`ProcessNewBlockHeaders`](fabc031048/src/net_processing.cpp (L2856)) when that initial `IsContinuationOfLowWorkHeadersSync` call returns `false`. One easy way (maybe the only?) to trigger this is by sending 2000 headers where the last header has a different `nBits` value than the prior headers (which fails the pre-sync logic [here](fabc031048/src/headerssync.cpp (L189))). Those 2000 headers will be passed to `ProcessNewBlockHeaders`.

  I haven't included a test here so far because we can't test this without changing the default value for `CRegTestParams::consensus.fPowAllowMinDifficultyBlocks` or doing some more involved refactoring.

ACKs for top commit:
  sipa:
    ACK 7ad15d1100
  glozow:
    ACK 7ad15d1100

Tree-SHA512: 9aabb8bf3700401e79863d0accda0befd2a83c4d469a53f97d827e51139e2f826aee08cdfbc8866b311b153f61fdac9b7aa515fcfa2a21c5e2812c2bf3c03664
2022-10-24 15:38:37 +01:00
Hennadii Stepanov
884304e6c6
test: Make system_tests/run_command locale agnostic 2022-10-24 13:36:04 +01:00
MacroFake
fa2d01470a
test: Use type-safe NodeSeconds for TestMemPoolEntryHelper 2022-10-24 11:33:33 +02:00
MacroFake
3db23fd821
Merge bitcoin-core/gui#676: Update peers window "Transaction Relay" label and tooltip
a079103c94 gui: update peers window "Transaction Relay" label and tooltip (Jon Atack)

Pull request description:

  to current v24.0 p2p behavior.  Similar updates have been made to RPC getpeerinfo and CLI -netinfo.

Top commit has no ACKs.

Tree-SHA512: 400a794f655f799eefcb77c479fef6bcd3f81aede2af54a4a9bcb7c0c783e2e3f18bc5fd2484a79e8c30af279747a05fc0ebb69dbc47375d4c55b16ceba97b99
2022-10-24 10:42:47 +02:00
MacroFake
8fb3fd2ba4
Merge bitcoin-core/gui#673: Use fallback value for Version and User Agent during peer connection
c2a21c0670 gui: use fallback value for Version and User Agent during peer connection (Jon Atack)

Pull request description:

  During connection setup for a peer, getpeerinfo returns `"version": 0, "subver": ""` and the GUI Peers window displays 0 and an empty field, respectively.

  Give these fields the same behavior as the other fields in the GUI Peers window: display the fallback value in `src/qt/forms/debugwindow.ui` (i.e. `N/A`) until a valid result is available after the peer connection completes.

  An alternative would be to display nothing for both, as is the case currently for User Agent.

ACKs for top commit:
  jarolrod:
    ACK c2a21c0670
  furszy:
    code ACK c2a21c06

Tree-SHA512: 4f0060fa9abde120a2bb48c9dcc87894d9bb70c33e6ab43b22400a4bcd0ceff0fa098adf7f385b0a7a4cf5d7053463b36fe1232e19a8d5025eecd8db9833f73b
2022-10-24 10:40:05 +02:00
fanquake
50cc8ef5a7
Merge bitcoin/bitcoin#26302: refactor: Use type-safe time point for CWallet::m_next_resend
fa51cc9651 refactor: Use type-safe time point for CWallet::m_next_resend (MacroFake)

Pull request description:

  `GetTime` is not type-safe, thus deprecated, see 75cbbfa279/src/util/time.h (L62-L70)

ACKs for top commit:
  shaavan:
    Code Review ACK fa51cc9651
  aureleoules:
    ACK fa51cc9651

Tree-SHA512: 030de10070518580763ea75079442e2f934c54d3083be3ebe35e7f1bc6db2096745bb46d95aa1e6efe29ced30a048acfe5cd999178e6787b7647dfbec5ecb444
2022-10-24 10:11:13 +08:00
Hennadii Stepanov
45a0f4e014
Update minisketch subtree to latest upstream 2022-10-23 15:03:04 +01:00
Hennadii Stepanov
e9f1d8c272 Squashed 'src/minisketch/' changes from 47f0a2d26f..a571ba20f9
a571ba20f9 Merge sipa/minisketch#68: Add missed `#include <string>`
b9a7f7e2bc Merge sipa/minisketch#69: refactor: Drop unused `total` local variables
8a5af94edc Merge sipa/minisketch#70: build: Remove `-Qunused-arguments` workaround for clang + ccache
c36f1f03a3 Merge sipa/minisketch#72: Fix MSVC implementation of `CountBits()` function
0078bedda6 Ignore `HAVE_CLZ` macro when building with MSVC
1c772918c4 Fix MSVC implementation of `CountBits()` function
98f87c55f4 build: Remove `-Qunused-arguments` workaround for clang + ccache
11a1e25c81 refactor: Drop unused `total` local variables
ed6c8fcfd9 Add missed `#include <string>`

git-subtree-dir: src/minisketch
git-subtree-split: a571ba20f9dd1accab6a2309d066369878042ca6
2022-10-23 15:00:20 +01:00
dergoegge
7ad15d1100 [net processing] Handle IsContinuationOfLowWorkHeadersSync return value correctly when new headers sync is started 2022-10-21 11:05:34 +01:00
MacroFake
8c5c98db47
Merge bitcoin/bitcoin#26248: net: Set relay in version msg to peers with relay permission in -blocksonly mode
dddd1acf58 net: Set relay in version msg to peers with relay permission (MacroFake)

Pull request description:

  Seems odd to set the `relay` permission in -blocksonly mode and also ask the peer not to relay transactions.

ACKs for top commit:
  dergoegge:
    ACK dddd1acf58
  naumenkogs:
    ACK dddd1acf58
  mzumsande:
    ACK dddd1acf58

Tree-SHA512: 7bb0e964993ea4982747ae2801fe963ff88586e2ded03015b60ab83172b5b61f2d50e9cde9d7711b7ab207f8639467ecafc4d011ea151ec6c82c722f510f4df7
2022-10-21 11:18:48 +02:00
fanquake
afbcd227dd
build: remove threadinterrupt from libbitcoinkernel 2022-10-21 16:40:55 +08:00
fanquake
f2859c3aa8
Merge bitcoin/bitcoin#25727: util, config: error on startup if conf or reindex are set in config file
deba6fe315 test: update feature_config_args.py (josibake)
2e3826cbcd util: warn if reindex is used in conf (josibake)
5e744f4238 util: disallow setting conf in bitcoin.conf (josibake)

Pull request description:

  In help from `bitcoind -h` it specifes that `conf` can only be used from the commandline. However, if `conf` is set in a `bitcoin.conf` file, there is no error and from reading the logs it seems as if the `conf=<other file>` is being used, despite it being ignored. To recreate, you can setup a `bitcoin.conf` file in the default directory, add `conf=<some other file>.conf` and in the separate config file set whichever config value you want and verify that it is being ignored. alternatively, if you set `includeconf=<some other file>.conf` , your config in `<some other file>` will be picked up.

  This PR fixes this by having the node error when reading the config file if `conf=` is set.

  Additionally, it was mentioned in a recent [PR review club](https://bitcoincore.reviews/24858) that if `reindex=1` is set in the config file, the node will reindex on every startup, which is undesirable:
   ```irc
  17:14 <larryruane> michaelfolkson: Reindex is requested by the user (node operator) as a configuration option (command line or in the config file, tho you probably would never put it in the file, or else it would reindex on every startup!)
  ```

  This PR also has a commit to warn if `reindex=1` is set in the config file.

ACKs for top commit:
  hebasto:
    ACK deba6fe315, tested on Ubuntu 22.04.
  aureleoules:
    tACK deba6fe315
  ryanofsky:
    Code review ACK deba6fe315.

Tree-SHA512: 619fd0aa14e98af1166d6beb92651f5ba3f10d38b8ee132957f094f19c3a37313d9f4d7be2e4019f3fc9a2ca5fa42d03eb539ad820e27efec7ee58a26eb520b1
2022-10-21 16:39:44 +08:00
fanquake
085f83940d
Merge bitcoin/bitcoin#26344: wallet: Fix sendall with watchonly wallets and specified inputs
315fd4dbab test: Test for out of bounds vout in sendall (Andrew Chow)
b132c85650 wallet: Check utxo prevout index out of bounds in sendall (Andrew Chow)
708b72b715 test: Test that sendall works with watchonly spending specific utxos (Andrew Chow)
6bcd7e2a3b wallet: Correctly check ismine for sendall (Andrew Chow)

Pull request description:

  The `sendall` RPC would previously fail when used with a watchonly wallet and specified inputs. This failure was caused by checking isminetype equality with ISMINE_ALL rather than a bitwise AND as IsMine can never return ISMINE_ALL.

  Also added a test.

ACKs for top commit:
  w0xlt:
    ACK 315fd4dbab
  furszy:
    ACK 315fd4db

Tree-SHA512: fb55cf6524e789964770b803f401027319f0351433ea084ffa7c5e6f1797567a608c956b7f7c5bd542aa172c4b7b38b07d0976f5ec587569efead27266e8664c
2022-10-21 16:24:15 +08:00
Andrew Chow
b132c85650 wallet: Check utxo prevout index out of bounds in sendall 2022-10-20 13:24:57 -04:00
Andrew Chow
fabc031048
Merge bitcoin/bitcoin#26158: bench: add "priority level" to the benchmark framework
3e9d0bea8d build: only run high priority benchmarks in 'make check' (furszy)
466b54bd4a bench: surround main() execution with try/catch (furszy)
3da7cd2a76 bench: explicitly make all current benchmarks "high" priority (furszy)
05b8c76232 bench: add "priority level" to the benchmark framework (furszy)
f1593780b8 bench: place benchmark implementation inside benchmark namespace (furszy)

Pull request description:

  This is from today's meeting, a simple "priority level" for the benchmark framework.

  Will allow us to run certain benchmarks while skip non-prioritized ones in `make check`.

  By default, `bench_bitcoin` will run all the benchmarks. `make check`will only run the high priority ones,
  and have marked all the existent benchmarks as "high priority" to retain the current behavior.

  Could test it by modifying any benchmark priority to something different from "high", and
  run `bench_bitcoin -priority-level=high` and/or `bench_bitcoin -priority-level=medium,low`
  (the first command will skip the modified bench while the second one will include it).

  Note: the second commit could be avoided by having a default arg value for the priority
  level but.. an explicit set in every `BENCHMARK` macro call makes it less error-prone.

ACKs for top commit:
  kouloumos:
    re-ACK 3e9d0bea8d
  achow101:
    ACK 3e9d0bea8d
  theStack:
    re-ACK 3e9d0bea8d
  stickies-v:
    re-ACK 3e9d0bea8d

Tree-SHA512: ece59bf424c5fc1db335f84caa507476fb8ad8c6151880f1f8289562e17023aae5b5e7de03e8cbba6337bf09215f9be331e9ef51c791c43bce43f7446813b054
2022-10-20 11:05:03 -04:00
furszy
3e9d0bea8d
build: only run high priority benchmarks in 'make check' 2022-10-20 10:21:05 -03:00
furszy
466b54bd4a
bench: surround main() execution with try/catch
so we have a cleaner exit on internal runtime errors.
e.g. an unknown priority level.
2022-10-20 10:21:04 -03:00
furszy
3da7cd2a76
bench: explicitly make all current benchmarks "high" priority
no-functional changes. Only have set the priority level explicitly
on every BENCHMARK macro call.
2022-10-20 10:21:04 -03:00
furszy
05b8c76232
bench: add "priority level" to the benchmark framework
Will allow us to run certain benchmarks while skip
non-prioritized ones in 'make check'.
2022-10-20 10:21:04 -03:00
fanquake
2ac71d20b2
Merge bitcoin/bitcoin#25595: Verify PSBT inputs rather than check for fields being empty
e133264c5b Add test for PSBT input verification (Greg Sanders)
d25699280a Verify PSBT inputs rather than check for fields being empty (Greg Sanders)

Pull request description:

  In a few keys spots, PSBT finality is checked by looking for non-empty witness data.

  This complicates a couple things:
  1) Empty data can be valid in certain cases
  2) User may be passed bogus final data by a counterparty during PSBT work happening, and end up with incorrect signatures that they may not be able to check in other contexts if the UTXO doesn't exist yet in chain/mempool, timelocks, etc.

  On the whole I think these heavier checks are worth it in case someone is actually assuming the signatures are correct if our API is saying so.

ACKs for top commit:
  achow101:
    ACK e133264c5b

Tree-SHA512: 9de4fbb0be1257b081781f5df908fd55666e3acd5c4e36beb3b3f2f5a6aed69ff77068c44cde6127e159e773293fd9ced4c0bb47e693969f337e74dc8af030da
2022-10-20 08:13:14 +08:00
Andrew Chow
6bcd7e2a3b wallet: Correctly check ismine for sendall
sendall should be using a bitwise AND for sendall's IsMine check rather
than an equality as IsMine will never return ISMINE_ALL.
2022-10-19 15:13:11 -04:00
MacroFake
a97791d9fb
Merge bitcoin/bitcoin#25830: refactor: Replace m_params with chainman.GetParams()
5d3f98d278 refactor: Replace m_params with chainman.GetParams() (Aurèle Oulès)

Pull request description:

  Fixes a TODO introduced in #24595.
  Removes `m_params` from `CChainState` class and replaces it with `m_chainman.GetParams()`.

ACKs for top commit:
  MarcoFalke:
    review ACK 5d3f98d278 🌎

Tree-SHA512: de0fe31450d281cc7307c0d820495e86c93c7998e77a148db2c703da66cff1059e6560c041f1864913c42075aa24d259c2623d45e929ca0a8056ed330a9f9978
2022-10-19 10:04:34 +02:00
MacroFake
003050dfaf
Merge bitcoin/bitcoin#26286: test: Remove unused txmempool include from tests
1c48dae76f test: Use C++11 member initializers for TestMemPoolEntryHelper (MacroFake)
fad7f2239c test: Remove unused txmempool include from tests (MacroFake)

Pull request description:

  Seems odd to include this heavy header in all tests despite it only being used in a few tests.

  Can be reviewed with `--color-moved=dimmed-zebra --ignore-all-space`

ACKs for top commit:
  aureleoules:
    reACK 1c48dae76f
  hebasto:
    ACK 1c48dae76f, I have reviewed the code and it looks OK, I agree it can be merged.
  w0xlt:
    ACK 1c48dae76f

Tree-SHA512: 31f2808d04ec33bfc2409832b8e59e6c870eaa98fbcf879e1c786492c7d07134711b30f8290bdb34e1b8f7b8f2f11dae8e10c64e7eb31f584b2f5c58fcc7743b
2022-10-19 09:41:47 +02:00
MacroFake
bbe2655309
Merge bitcoin/bitcoin#26142: Use PACKAGE_NAME in messages rather than hardcoding "Bitcoin Core"
b147322a7a Use `PACKAGE_NAME` in messages rather than hardcoding "Bitcoin Core" (Hennadii Stepanov)

Pull request description:

  Usually, we do not hardcode "Bitcoin Core" in the user-faced messages.

  See:
  - bitcoin/bitcoin#18646
  - bitcoin/bitcoin#19282

  Also grammar has been improved -- singular instead of plural.

ACKs for top commit:
  jarolrod:
    ACK b147322a7a

Tree-SHA512: b135c18703dfdd7b63d4cb27d1ac48f6a9dbf69382142ae381f33bf561cbf57477a11d1c73263aa834f705206d7dd5716df2523d38ed0d4cfec8babc38bb017a
2022-10-19 09:22:22 +02:00
MacroFake
c102a558e1
Merge bitcoin/bitcoin#26179: bench: Add missed ECCVerifyHandle instance
f09d47b263 bench: Add missed `ECCVerifyHandle` instance (Hennadii Stepanov)

Pull request description:

  To clearly observe the lack of an `ECCVerifyHandle` instance,
  - apply the following diff:
  ```diff
  --- a/src/Makefile.bench.include
  +++ b/src/Makefile.bench.include
  @@ -19,11 +19,9 @@ bench_bench_bitcoin_SOURCES = \
     bench/bench.h \
     bench/bench_bitcoin.cpp \
     bench/block_assemble.cpp \
  -  bench/ccoins_caching.cpp \
     bench/chacha20.cpp \
     bench/chacha_poly_aead.cpp \
     bench/checkblock.cpp \
  -  bench/checkqueue.cpp \
     bench/crypto_hash.cpp \
     bench/data.cpp \
     bench/data.h \
  @@ -46,8 +44,7 @@ bench_bench_bitcoin_SOURCES = \
     bench/rpc_blockchain.cpp \
     bench/rpc_mempool.cpp \
     bench/strencodings.cpp \
  -  bench/util_time.cpp \
  -  bench/verify_script.cpp
  +  bench/util_time.cpp

   nodist_bench_bench_bitcoin_SOURCES = $(GENERATED_BENCH_FILES)

  ```
  - then
  ```
  $ ./autogen
  $ ./configure
  $ make clean
  $ make
  ```
  - then
  ```
  $ ./src/bench/bench_bitcoin -filter=ExpandDescriptor
  bench_bitcoin: pubkey.cpp:296: bool CPubKey::IsFullyValid() const: Assertion `secp256k1_context_verify && "secp256k1_context_verify must be initialized to use CPubKey."' failed.
  Aborted (core dumped)
  ```

ACKs for top commit:
  achow101:
    ACK f09d47b263
  w0xlt:
    ACK f09d47b263

Tree-SHA512: e1f33f88d427c57fe31d5810d12e9f46fed2911f5736208ebf7d4a968de0dd8c1f6b73a0d1093316da117dd3bcfda5dde6e41d6c95fcdb99bdea62e19df5ad20
2022-10-19 09:16:29 +02:00
Jon Atack
a079103c94 gui: update peers window "Transaction Relay" label and tooltip
to current v24.0 p2p behavior
2022-10-18 15:26:52 -07:00
MacroFake
1c48dae76f
test: Use C++11 member initializers for TestMemPoolEntryHelper
Co-authored-by: Aurèle Oulès <aurele@oules.com>
2022-10-18 17:32:44 +02:00
MacroFake
aaaa7bd0ba
iwyu: Add missing includes 2022-10-18 14:12:52 +02:00
MacroFake
fa9ebec096
Remove g_parallel_script_checks 2022-10-18 14:12:42 +02:00
MacroFake
fa7c834b9f
Move ::fCheckBlockIndex into ChainstateManager
This changes the flag for the bitcoin-chainstate executable. Previously
it was false, now it is the chain's default value (still false for the
main chain).
2022-10-18 14:11:48 +02:00
MacroFake
fa43188d86
Move ::fCheckpointsEnabled into ChainstateManager 2022-10-18 14:10:50 +02:00
MacroFake
cccca83099
Move ::nMinimumChainWork into ChainstateManager
This changes the minimum chain work for the bitcoin-chainstate
executable. Previously it was uint256{}, now it is the chain's default
minimum chain work.
2022-10-18 14:09:17 +02:00
MacroFake
fa29d0b57c
Move ::hashAssumeValid into ChainstateManager
This changes the assumed valid block for the bitcoin-chainstate
executable. Previously it was uint256{}, now it is defaultAssumeValid.
2022-10-18 14:08:49 +02:00
MacroFake
faf44876db
Move ::nMaxTipAge into ChainstateManager 2022-10-18 14:07:59 +02:00
MacroFake
fad7f2239c
test: Remove unused txmempool include from tests 2022-10-18 14:02:09 +02:00
MacroFake
ba441d493c
Merge bitcoin/bitcoin#26313: doc: consolidate library documentation to libraries.md
af781bf4b2 doc: fix typo in doc/libraries.md (fanquake)
9e9ae6101f doc: remove library commentary from src/Makefile.am (fanquake)

Pull request description:

  Deduplicate the makefile comments, in favour of doc/libraries.md. I think a single, more comprehensive source of truth is preferable. Diagrams are also useful. Came up in https://github.com/bitcoin/bitcoin/pull/26292#issuecomment-1275094478.

ACKs for top commit:
  ryanofsky:
    Code review ACK af781bf4b2, nice cleanups
  hebasto:
    ACK af781bf4b2, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: df61ed1394102221701ae2dfa42886dfabe9d9fd7f601b794e2195f93d8f7c2a1cd1c000a77d0a969b42328e8ebc0387755c57291837b283fdf376dbd98fdda1
2022-10-18 13:44:56 +02:00
Greg Sanders
d25699280a Verify PSBT inputs rather than check for fields being empty 2022-10-17 11:13:02 -04:00
Gleb Naumenko
b99ee9d22d test: Add unit tests for reconciliation negotiation 2022-10-17 12:35:44 +03:00
Gleb Naumenko
f63f1d3f4b p2p: clear txreconciliation state for non-wtxid peers
We optimistically pre-register a peer for txreconciliations
upon sending txreconciliation support announcement.
But if, at VERACK, we realize that the peer never sent
WTXIDRELAY message, we should unregister the peer
from txreconciliations, because txreconciliations rely on wtxids.
2022-10-17 12:35:44 +03:00
Gleb Naumenko
88d326c8e3 p2p: Finish negotiating reconciliation support
Once we received a reconciliation announcement support
message from a peer and it doesn't violate our protocol,
we store the negotiated parameters which will be used
for future reconciliations.
2022-10-17 12:35:44 +03:00
Gleb Naumenko
36cf6bf216 Add helper to see if a peer is registered for reconciliations 2022-10-17 12:35:44 +03:00
Gleb Naumenko
4470acf076 p2p: Forget peer's reconciliation state on disconnect 2022-10-17 12:35:44 +03:00
Gleb Naumenko
3fcf78ee6a p2p: Announce reconciliation support
If we're connecting to the peer which might support
transaction reconciliation, we announce we want to reconcile
with them.

We store the reconciliation salt so that when the peer
responds with their salt, we are able to compute the
full reconciliation salt.

This behavior is enabled with a CLI flag.
2022-10-17 12:35:43 +03:00
Gleb Naumenko
24e36fac0a log: Add tx reconciliation log category 2022-10-17 12:00:59 +03:00
fanquake
9e9ae6101f
doc: remove library commentary from src/Makefile.am
This duplicates and is less explanatory than doc/libraries.md.
2022-10-14 17:07:36 +08:00
glozow
cbb2da8fcf
add lock annotation for FeeFilterRounder::round()
Calling WITH_LOCK() on a non-recursive mutex requires not holding it beforehand.

Co-authored-by: Niklas Gögge <n.goeggi@gmail.com>
2022-10-13 14:42:59 -04:00
Andrew Chow
0384b19414
Merge bitcoin/bitcoin#24851: init: ignore BIP-30 verification in DisconnectBlock for problematic blocks
e899d4ca6f init: limit bip30 exceptions to coinbase txs (Chris Geihsler)
511eb7fdea Ignore problematic blocks in DisconnectBlock (Chris Geihsler)

Pull request description:

  Fixes https://github.com/bitcoin/bitcoin/issues/22596

  When using checklevel=4, block verification fails because of duplicate coinbase transactions involving blocks 91812 and 91722. There was already a check in place within `ConnectBlock` to ignore the problematic blocks, but `DisconnectBlock` did not contain a similar check to ignore these blocks when called from `VerifyDB`.

  By ignoring these two blocks in `DisconnectBlock`, the block verification process succeeds at checklevel=4.

  (Note to reviewers: this is my first contribution to Bitcoin Core, so any feedback is most welcome. Thanks in advance for reviewing!)

  ## Steps to reproduce:

  Use the following bitcoin.conf file and start bitcoind. I only used block data through block ~100000 so that the verification process was much faster.

  ```
  assumevalid=0
  checkblocks=0
  checklevel=4
  ```

  Without this change, you will see the following error when the blocks are verified:

  ```
  2022-04-14T02:56:44Z init message: Verifying blocks…
  2022-04-14T02:56:44Z Verifying last 101881 blocks at level 4
  2022-04-14T02:56:44Z [0%]...[10%]...[20%]...[30%]...[40%]...ERROR: VerifyDB(): *** coin database inconsistencies found (last 10160 blocks, 142571 good transactions before that)

  2022-04-14T02:57:01Z : Corrupted block database detected.
  Please restart with -reindex or -reindex-chainstate to recover.
  : Corrupted block database detected.
  Please restart with -reindex or -reindex-chainstate to recover.
  ```

  With this change, you will see this instead:

  ```
  2022-04-14T02:32:29Z init message: Verifying blocks…
  2022-04-14T02:32:29Z Verifying last 101746 blocks at level 4
  2022-04-14T02:32:29Z [0%]...[10%]...[20%]...[30%]...[40%]...[50%]...[60%]...[70%]...[80%]...[90%]...[DONE].
  2022-04-14T02:32:48Z No coin database inconsistencies in last 101746 blocks (226126 transactions)
  ```

ACKs for top commit:
  laanwj:
    Code review ACK e899d4ca6f
  achow101:
    ACK e899d4ca6f
  jamesob:
    (Biased) ACK e899d4ca6f ([`jamesob/ackr/24851.2.seejee.init_ignore_bip_30_verif`](https://github.com/jamesob/bitcoin/tree/ackr/24851.2.seejee.init_ignore_bip_30_verif))

Tree-SHA512: d2f6d25e9619aee32c1a73fe846b1b587698eaa5a4994fa6424f1038f45654f9fd52b74a69843cc84d90168d74827130ccf8e9201502f5d52281acdb20429291
2022-10-13 14:15:28 -04:00
Andrew Chow
92be831847
Merge bitcoin/bitcoin#25412: rest: add /deploymentinfo endpoint
a8250e30f1 doc: add release note about `/rest/deploymentinfo` (brunoerg)
5c96020024 doc: add `/deploymentinfo` in REST-interface (brunoerg)
3e44bee08e test: add coverage for `/rest/deploymentinfo` (brunoerg)
91497031cb rest: add `/deploymentinfo` (brunoerg)

Pull request description:

  #23508 added a new RPC named `getdeploymentinfo`, it moved the softfork section from `getblockchaininfo` into this new one. In the REST interface, we have an endpoint named`/rest/chaininfo.json` (which refers to `getblockchaininfo`), so, this PR adds a new REST endpoint named `/deploymentinfo` which refers to `getdeploymentinfo`.

  You can use it by passing a block hash, e.g: '/rest/deploymentinfo/<BLOCKHASH>.json' or you can use it without passing a block hash to get the 'deploymentinfo' for the last block.

ACKs for top commit:
  jonatack:
    re-ACK a8250e30f1 rebase-only since my last review at c65f82bb
  achow101:
    ACK a8250e30f1
  stickies-v:
    re-ACK a8250e30f1

Tree-SHA512: 0735183b6828d51a72ed0e2be5a09b314ac4693f548982c6e9adaa0ef07a55aa428d3b2d1b1de70b83169811a663a8624b686166e5797f624dcc00178b9796e6
2022-10-13 13:30:55 -04:00
Andrew Chow
0bac04b758
Merge bitcoin/bitcoin#24407: fees: make the class FeeFilterRounder thread-safe
8173f160e0 style: rename variables to match coding style (Vasil Dimov)
8b4ad203d0 fees: make FeeFilterRounder::feeset const (Vasil Dimov)
e7a5bf6be7 fees: make the class FeeFilterRounder thread-safe (Vasil Dimov)

Pull request description:

  Make the class `FeeFilterRounder` thread-safe so that its methods can be called concurrently by different threads on the same object. Currently it has just one method (`round()`).

  The second commit is optional, but it improves readability, showing that the `feeset` member will never be changed, thus does not need protection from concurrent access.

ACKs for top commit:
  jonatack:
    re-ACK 8173f160e0
  laanwj:
    Code review ACK 8173f160e0
  promag:
    Code review ACK 8173f160e0

Tree-SHA512: 94b809997c485c0d114fa702d0406b980be8eaaebcfefa56808ed670aa943959c2f16cfd0ef72b4752fe2a409a23af1b4b7f2f236e51212957759569e3bbbefd
2022-10-13 11:57:07 -04:00
Andrew Chow
1dec90d95b
Merge bitcoin/bitcoin#25526: wallet: avoid double keypool TopUp() call on descriptor wallets
bfb9b94ebe wallet: remove duplicate descriptor type check in GetNewDestination (furszy)
76b982a4a5 wallet: remove unused `nAccountingEntryNumber` field (furszy)
599ff5adfc wallet: avoid double TopUp() calls on descriptor wallets (furszy)

Pull request description:

  Found it while was digging over a `getnewaddress` timeout on the functional test suite.

  ### Context:

  We are calling `TopUp()` twice in the following flows for descriptor wallets:

  A) `CWallet::GetNewDestination`:
     1) Calls spk_man->TopUp()
     2) Calls spk_man->GetNewDestination() --> which, after the basic script checks, calls TopUp() again.

  B) `CWallet::GetReservedDestination`:
     1) Calls spk_man->TopUp()
     2) Calls spk_man->GetReservedDestination() --> which calls to GetNewDestination (which calls to TopUp again).

  ### Changes:

  Move `TopUp()` responsibility from the wallet class to each scriptpubkeyman.
  So each spkm can decide to call it or not after perform the basic checks
  for the new destination request.

  Aside from that, remove the unused `nAccountingEntryNumber` wallet field. And a duplicated descriptor type check in `GetNewDestination`

ACKs for top commit:
  aureleoules:
    re-ACK bfb9b94ebe.
  achow101:
    ACK bfb9b94ebe
  theStack:
    Code-review ACK bfb9b94ebe

Tree-SHA512: 3ab73f37729e50d6c6a4434f676855bc1fb404619d63c03e5b06ce61c292c09c59d64cb1aa3bd9277b06f26988956991d62c90f9d835884f41ed500b43a12058
2022-10-13 11:27:38 -04:00
Andrew Chow
cb9764b686
Merge bitcoin/bitcoin#26109: rpc, doc: getpeerinfo updates
a3789c700b Improve getpeerinfo pingtime, minping, and pingwait help docs (Jon Atack)
df660ddb1c Update getpeerinfo/-netinfo/TxRelay#m_relay_txs relaytxes docs (for v24 backport) (Jon Atack)
1f448542e7 Always return getpeerinfo "minfeefilter" field (for v24 backport) (Jon Atack)
9cd6682545 Make getpeerinfo field order consistent with its help (for v24 backport) (Jon Atack)

Pull request description:

  Various updates and fixups, mostly targeting v24. Please refer to the commit messages for details.

ACKs for top commit:
  achow101:
    ACK a3789c700b
  brunoerg:
    ACK a3789c700b
  vasild:
    ACK a3789c700b

Tree-SHA512: b8586a9b83c1b18786b5ac1fc1dba91573c13225fc2cfc8d078f4220967c95056354f6be13327f33b4fcf3e9d5310fa4e1bdc93102cbd6574f956698993a54bf
2022-10-13 11:07:33 -04:00
Andrew Chow
bc2b1f0fe2
Merge bitcoin/bitcoin#23549: Add scanblocks RPC call (attempt 2)
626b7c8493 fuzz: add scanblocks as safe for fuzzing (James O'Beirne)
94fe5453c7 test: rpc: add scanblocks functional test (Jonas Schnelli)
6ef2566b68 rpc: add scanblocks - scan for relevant blocks with descriptors (Jonas Schnelli)
a4258f6e81 rpc: move-only: consolidate blockchain scan args (James O'Beirne)

Pull request description:

  Revives #20664. All feedback from the previous PR has either been responded to inline or incorporated here.

  ---

  Major changes from Jonas' PR:
  - consolidated arguments for scantxoutset/scanblocks
  - substantial cleanup of the functional test

  Here's the range-diff (`git range-diff master jonasschnelli/2020/12/filterblocks_rpc jamesob/2021-11-scanblocks`): https://gist.github.com/jamesob/aa4a975344209f0316444b8de2ec1d18

  ### Original PR description

  > The `scanblocks` RPC call allows one to get relevant blockhashes from a set of descriptors by scanning all blockfilters in a given range.
  >
  > **Example:**
  >
  > `scanblocks start '["addr(<bitcoin_address>)"]' 661000` (returns relevant blockhashes for `<bitcoin_address>` from blockrange 661000->tip)
  >
  > ## Why is this useful?
  > **Fast wallet rescans**: get the relevant blocks and only rescan those via `rescanblockchain getblockheader(<hash>)[height] getblockheader(<hash>)[height])`. A future PR may add an option to allow to provide an array of blockhashes to `rescanblockchain`.
  >
  > **prune wallet rescans**: (_needs additional changes_): together with a call to fetch blocks from the p2p network if they have been pruned, it would allow to rescan wallets back to the genesis block in pruned mode (relevant #15946).
  >
  > **SPV mode** (_needs additional changes_): it would be possible to build the blockfilterindex from the p2p network (rather then deriving them from the blocks) and thus allow some sort of hybrid-SPV mode with moderate bandwidth consumption (related #9483)

ACKs for top commit:
  furszy:
    diff re-ACK 626b7c8

Tree-SHA512: f84e4dcb851b122b39e9700c58fbc31e899cdcf9b587df9505eaf1f45578cc4253e89ce2a45d1ff21bd213e31ddeedbbcad2c80810f46755b30acc17b07e2873
2022-10-13 10:48:16 -04:00
Andrew Chow
6912a28f08
Merge bitcoin/bitcoin#25667: assumeutxo: snapshot initialization
bf95976061 doc: add note about snapshot chainstate init (James O'Beirne)
e4d7995286 test: add testcases for snapshot initialization (James O'Beirne)
cced4e7336 test: move-only-ish: factor out LoadVerifyActivateChainstate() (James O'Beirne)
51fc9241c0 test: allow on-disk coins and block tree dbs in tests (James O'Beirne)
3c361391b8 test: add reset_chainstate parameter for snapshot unittests (James O'Beirne)
00b357c215 validation: add ResetChainstates() (James O'Beirne)
3a29dfbfb2 move-only: test: make snapshot chainstate setup reusable (James O'Beirne)
8153bd9247 blockmanager: avoid undefined behavior during FlushBlockFile (James O'Beirne)
ad67ff377c validation: remove snapshot datadirs upon validation failure (James O'Beirne)
34d1590331 add utilities for deleting on-disk leveldb data (James O'Beirne)
252abd1e8b init: add utxo snapshot detection (James O'Beirne)
f9f1735f13 validation: rename snapshot chainstate dir (James O'Beirne)
d14bebf100 db: add StoragePath to CDBWrapper/CCoinsViewDB (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11) (parent PR: https://github.com/bitcoin/bitcoin/pull/15606)

  ---

  Half of the replacement for #24232. The original PR grew larger than expected throughout the review process.

  This change adds the ability to initialize a snapshot-based chainstate during init if one is detected on disk. This is of course unused as of now (aside from in unittests) given that we haven't yet enabled actually loading snapshots.

  Don't be scared! There are some big move-only commits in here.

  Accompanying changes include:

  - moving the snapshot coinsdb directory from being called `chainstate_[base blockhash]` to `chainstate_snapshot`, since we only support one snapshot in use at a time. This simplifies some logic, but it necessitates writing that base blockhash out to a file within the coinsdb dir. See [discussion here](https://github.com/bitcoin/bitcoin/pull/24232#discussion_r832762880).
  - adding a simple fix in `FlushBlockFile()` that avoids a crash when attemping to flush to disk before `LoadBlockIndexDB()` is called, which happens when calling `MaybeRebalanceCaches()` during multiple chainstate init.
  - improving the unittest to allow testing with on-disk chainstates - necessary to test a simulated restart and re-initialization.

ACKs for top commit:
  naumenkogs:
    utACK bf95976061
  ariard:
    Code Review ACK bf9597606
  ryanofsky:
    Code review ACK bf95976061. Changes since last review: rebasing, switching from CAutoFile to AutoFile, adding comments, switching from BOOST_CHECK to Assert in test util, using chainman.GetMutex() in tests, destroying one ChainstateManager before creating a new one in tests
  fjahr:
    utACK bf95976061
  aureleoules:
    ACK bf95976061

Tree-SHA512: 15ae75caf19f8d12a12d2647c52897904d27b265a7af6b4ae7b858592eeadb8f9da6c2394b6baebec90adc28742c053e3eb506119577dae7c1e722ebb3b7bcc0
2022-10-13 10:19:27 -04:00
glozow
147d64dbdf
Merge bitcoin/bitcoin#25858: psbt: Only include PSBT_OUT_TAP_TREE when the output has a script path
9e386afb67 tests: Test that PSBT_OUT_TAP_TREE is included correctly (Andrew Chow)
30ff25cf37 psbt: Only include m_tap_tree if it has scripts (Andrew Chow)
0577d423ad psbt: Change m_tap_tree to store just the tuples (Andrew Chow)
22c051ca70 tests: Test that PSBT_OUT_TAP_TREE is combined correctly (Andrew Chow)
7df6e1bb77 psbt: Fix merging of m_tap_tree (Andrew Chow)
0652dc53b2 [BugFix]: Do not allow deserializing PSBT with empty PSBT_OUT_TAP_TREE (Jeremy Rubin)

Pull request description:

  PSBT_OUT_TAP_TREE should not be included for outputs that do not have such a tree. This should be disallowed during parsing, as well as prior to serialization when the field is populated during updating.

  Also added some test cases.

  Alternative to #25856

ACKs for top commit:
  instagibbs:
    ACK 9e386afb67
  darosior:
    ACK 9e386afb67

Tree-SHA512: ce5c02a69752d176dbd967c1e8d30129b1905c8f186aeeef034576c1de82059271a1ee846bd040f5be4e66bb77ba711dcf14ac1e597c5707d7e7e2293f6cfefb
2022-10-13 09:40:27 -04:00
MacroFake
fa51cc9651
refactor: Use type-safe time point for CWallet::m_next_resend 2022-10-13 14:49:18 +02:00
fanquake
aa6fb37acc
Merge bitcoin/bitcoin#26205: wallet: #25768 follow ups
b01682a812 refactor: revert m_next_resend to not be std::atomic (stickies-v)
9245f45670 wallet: only update m_next_resend when actually resending (stickies-v)
7fbde8af5c refactor: carve out tx resend timer logic into ShouldResend (stickies-v)
01f3534632 refactor: remove unused locks for ResubmitWalletTransactions (stickies-v)
c6e8e11fb0 wallet: fix capitalization in docstring (stickies-v)

Pull request description:

  This PR addresses the outstanding comments/issues from #25768:

  - capitalization [typo](https://github.com/bitcoin/bitcoin/pull/25768#discussion_r958572522) in docstring
  - remove [unused locks](01f3534632) that we previously needed for `ReacceptWalletTransactions()`
  - before #25768, only `ResendWalletTransactions()` would reset `m_next_resend` (formerly called `nNextResend`). By unifying it with `ReacceptWalletTransactions()` into `ResubmitWalletTransactions()`, the number of callsites that would reset the `m_next_resend` timer increased
    - since `m_next_resend` is only used in case of `relay=true` (formerly `ResendWalletTransactions()`), this is unintuitive
    - it leads to [unexpected behaviour](https://github.com/bitcoin/bitcoin/pull/25768#issuecomment-1252619427) such as transactions potentially never being rebroadcasted.
    - it makes the ResubmitWalletTransactions()` logic [more complicated than strictly necessary](https://github.com/bitcoin/bitcoin/pull/25768#discussion_r962828563)
    - since #25768, we relied on an earlier call of `ResubmitWalletTransactions(relay=false, force=true)` to initialize `m_next_resend()`, I think we can more elegantly do that by just providing `m_next_resend` with a default value
    - just to highlight: this commit introduces behaviour change

  Note: the `if (!fBroadcastTransactions)` in `CWallet:ShouldResend()` is duplicated on purpose, since it potentially avoids the slightly more expensive `if (!chain().isReadyToBroadcast())` check afterwards. I don't have a strong view on it, so happy to remove that additional check to reduce the diff, too.

ACKs for top commit:
  aureleoules:
    ACK b01682a812
  achow101:
    ACK b01682a812

Tree-SHA512: ac5f1d8858f8dd736dd1480f385984d660c1916b62a42562317020e8f9fd6a30bd8f23d973d47e4c9480d744c5ba39fdbefd69568a5eb0589a8422d7e5971c1c
2022-10-13 12:09:44 +08:00
fanquake
422efcad36
Merge bitcoin/bitcoin#26188: test: silence TSAN false positive in coinstatsindex_initial_sync
861cb3fadc test: move SyncWithValidationInterfaceQueue() before Stop() in txindex_tests (Vasil Dimov)
6526dc3b78 test: silence TSAN false positive in coinstatsindex_initial_sync (Vasil Dimov)

Pull request description:

  Silence false positives from TSAN about unsynchronized calls to `BaseIndex::~BaseIndex()` and `BaseIndex::SetBestBlockIndex()`. They are synchronized, but beyond the comprehension of TSAN - by `SyncWithValidationInterfaceQueue()`, called from `BaseIndex::BlockUntilSyncedToCurrentChain()`.

  Fixes https://github.com/bitcoin/bitcoin/issues/25365

ACKs for top commit:
  MarcoFalke:
    review ACK 861cb3fadc
  ryanofsky:
    Code review ACK 861cb3fadc. Just comment change since last review.

Tree-SHA512: 8c30fdf2fd11d54e9adfa68a67185ab820bd7bd9f7f3ad6456e7e6d219fa9cf6d34b41e98e723eae86cb0c1baef7f3fc57b1b011a13dc3fe3d78334b9b5596de
2022-10-13 11:42:27 +08:00
glozow
7e1007a3c6
Merge bitcoin/bitcoin#25421: net: convert standalone IsSelectableSocket() and SetSocketNonBlocking() to Sock methods
b527b54950 net: convert standalone SetSocketNonBlocking() to Sock::SetNonBlocking() (Vasil Dimov)
29f66f7682 moveonly: move SetSocketNonBlocking() from netbase to util/sock (Vasil Dimov)
b4bac55679 net: convert standalone IsSelectableSocket() to Sock::IsSelectable() (Vasil Dimov)
5db7d2ca0a moveonly: move IsSelectableSocket() from compat.h to sock.{h,cpp} (Vasil Dimov)

Pull request description:

  _This is a piece of #21878, chopped off to ease review._

  * convert standalone `IsSelectableSocket()` to `Sock::IsSelectable()`
  * convert standalone `SetSocketNonBlocking()` to `Sock::SetNonBlocking()`

  This further encapsulates syscalls inside the `Sock` class and makes the callers mockable.

ACKs for top commit:
  jonatack:
    ACK b527b54950 review/debug build/unit tests at each commit, cross-referenced the changes with `man select` and `man errno`, ran a signet node on the last commit with ip4/ip6//tor/i2p/cjdns and network connections were nominal
  dergoegge:
    Code review ACK b527b54950

Tree-SHA512: af783ce558c7a89e173f7ab323fb3517103d765c19b5d14de29f64706b4e1fea3653492e8ea73ae972699986aaddf2ae72c7cfaa7dad7614254283083b7d2632
2022-10-12 15:49:02 -04:00