bitcoin/src
Andrew Chow 4ea3a8b71d
Merge bitcoin/bitcoin#25806: wallet: group outputs only once, decouple it from Coin Selection
6a302d40df wallet: single output groups filtering and grouping process (furszy)
bd91ed1cb2 wallet: unify outputs grouping process (furszy)
55962001da test: coinselector_tests refactor, use CoinsResult instead of plain std::vector (furszy)
34f54a0a3a wallet: decouple outputs grouping process from each ChooseSelectionResult (furszy)
461f0821a2 refactor: make OutputGroup::m_outputs field a vector of shared_ptr (furszy)
d8e749bb84 test: wallet, add coverage for outputs grouping process (furszy)
06ec8f9928 wallet: make OutputGroup "positive_only" filter explicit (furszy)

Pull request description:

  The idea originates from https://github.com/bitcoin/bitcoin/pull/24845#issuecomment-1130310321.

  Note:
  For clarity, it's recommended to start reviewing from the end result to understand the structure of the flow.

  #### GroupOutputs function rationale:
  If "Avoid Partial Spends" is enabled, the function gathers outputs with the same script together inside a container. So Coin Selection can treats them as if them were just one possible input and either select them all or not select them.

  #### How the Inputs Fetch + Selection process roughly works:

  ```
  1. Fetch user’s manually selected inputs.
  2. Fetch wallet available coins (walks through the entire wallet txes map) and insert them into a set of vectors (each vector store outputs from a single type).
  3. Coin Selection Process:
     Call `AttemptSelection` 8 times. Each of them expands the coin eligibility filter (accepting a larger subset of coins in the calculation) until it founds a solutions or completely fails if no solutions gets founds after the 8 rounds.

     Each `AttemptSelection` call performs the following actions:
       - For each output type supported by the wallet (P2SH, P2PK, P2WPKH, P2WSH and a combination of all of them):
         Call ‘ChooseSelectionResult’ providing the respective, filtered by type, coins vector. Which:
             I. Groups the outputs vector twice (one for positive only and a second one who includes the negative ones as well).
                - GroupOutputs walks-through the entire inputted coins vector one time at least, + more if we are avoiding partial spends, to generate a vector of OutputGroups.
             II. Then performs every coin selection algorithm using the recently created vector of OutputGroup: (1) BnB, (2) knapsack and (3) SRD.
             III. Then returns the best solution out of them.
  ```

  We perform the general operation of gathering outputs, with the same script, into a single container inside:
  Each coins selection attempt (8 times —> each coin eligibility filter), for each of the outputs vector who were filtered by type (plus another one joining all the outputs as well if needed), twice (one for the positive only outputs effective value and a second one for all of them).

  So, in the worst case scenario where no solution is found after the 8 Coin Selection attempts, the `GroupOutputs` function is called 80 times (8 * 5 * 2).

  #### Improvements:

  This proposal streamlines the process so that the output groups, filtered by coin eligibility and type, are created in a single loop outside of the Coin Selection Process.

  The new process is as follows:

  ```
  1. Fetch user’s manually selected inputs.
  2. Fetch wallet available coins.
  3. Group outputs by each coin eligibility filter and each different output type found.
  4. Coin Selection Process:
     Call AttemptSelection 8 times. Each of them expands the coin eligibility filter (accepting different output groups) until it founds a solutions or completely fails if no solutions gets founds after the 8 rounds.

     Each ‘AttemptSelection’ call performs the following actions:
        - For each output type supported by the wallet (P2SH, P2PK, P2WPKH, P2WSH and all of them):
            A. Call ‘ChooseSelectionResult’ providing the respective, filtered by type, output group. Which:
               I. Performs every coin selection algorithm using the provided vector of OutputGroup: (1) BnB, (2) knapsack and (3) SRD.
               II. Then returns the best solution out of them.
  ```

  Extra Note:
  The next steps after this PR will be to:
  1) Merge `AvailableCoins` and `GroupOutputs` processes.
  2) Skip entire coin selection rounds if no new coins are added into the subsequent round.
  3) Remove global feerates from the OutputGroup class.
  4) Remove secondary "grouped" tx creation from `CreateTransactionInternal` by running Coin Selection results over the aps grouped outputs vs non-aps ones.

ACKs for top commit:
  S3RK:
    ReACK 6a302d4
  achow101:
    ACK 6a302d40df
  theStack:
    re-ACK 6a302d40df 🥥

Tree-SHA512: dff849063be328e7d9c358ec80239a6db2cd6131963b511b83699b95b337d3106263507eaba0119eaac63e6ac21c6c42d187ae23d79d9220b90c323d44b01d24
2023-03-06 18:51:34 -05:00
..
bench wallet: single output groups filtering and grouping process 2023-03-06 09:45:40 -03:00
common Use DataStream where possible 2023-01-26 10:44:05 +01:00
compat compat: use STDIN_FILENO over 0 2023-01-04 12:00:25 +00:00
config
consensus Merge bitcoin/bitcoin#26345: refactor: modernize the implementation of uint256.* 2023-02-06 13:56:51 -05:00
crc32c Update crc32c subtree to latest upstream master 2022-08-13 13:55:38 +01:00
crypto Merge bitcoin/bitcoin#26153: Reduce wasted pseudorandom bytes in ChaCha20 + various improvements 2023-02-15 14:58:47 +00:00
index refactor, dbwrapper: Add DBParams and DBOptions structs 2023-02-10 04:39:11 -04:00
init refactor: Stop using gArgs global in system.cpp 2023-02-27 14:21:13 -05:00
interfaces Remove reindex special case from the progress bar label 2023-02-07 11:02:01 +01:00
ipc scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
kernel kernel: add missing include 2023-02-22 15:46:21 +00:00
leveldb Update leveldb-subtree subtree to latest upstream 2022-09-29 16:43:30 +01:00
logging Use steady clock for logging timer 2023-01-31 18:48:50 +01:00
minisketch Update minisketch subtree to latest upstream 2022-10-23 15:03:04 +01:00
node Merge bitcoin/bitcoin#26533: prune: scan and unlink already pruned block files on startup 2023-02-28 09:54:10 -05:00
policy clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
primitives scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
qt Merge bitcoin/bitcoin#27169: Update translations for 25.0 soft translation string freeze 2023-03-01 14:31:06 +01:00
rpc Merge bitcoin/bitcoin#27155: doc: Expand scantxoutset help text to cover tr() and miniscript 2023-03-06 11:15:16 -05:00
script Fixed a couple of typos in comments to make linter happy 2023-03-03 19:06:02 -03:00
secp256k1 Update secp256k1 subtree to upstream libsecp256k1 version 0.2.0 2022-12-12 23:40:36 -05:00
support util: Remove duplicate include 2023-02-23 17:58:40 -05:00
test Fixed a couple of typos in comments to make linter happy 2023-03-03 19:06:02 -03:00
univalue Merge bitcoin/bitcoin#26612: refactor: RPC: pass named argument value as string_view 2023-03-03 15:23:43 +01:00
util util: add missing include and fix function signature 2023-03-03 22:19:00 +00:00
wallet Merge bitcoin/bitcoin#25806: wallet: group outputs only once, decouple it from Coin Selection 2023-03-06 18:51:34 -05:00
zmq clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
.bear-tidy-config tidy: enable modernize-use-nullptr 2022-04-26 10:43:33 +01:00
.clang-format
.clang-tidy Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
addrdb.cpp Merge bitcoin/bitcoin#26847: p2p: track AddrMan totals by network and table, improve precision of adding fixed seeds 2023-01-31 16:08:44 -05:00
addrdb.h [net] Move asmap into NetGroupManager 2022-04-20 14:29:29 +01:00
addrman_impl.h addrman, refactor: combine two size functions 2023-01-26 18:11:13 -05:00
addrman.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
addrman.h addrman: Use std::nullopt instead of {} 2023-02-01 10:18:08 -05:00
arith_uint256.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
arith_uint256.h Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
attributes.h
banman.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
banman.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
base58.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
base58.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
bech32.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
bech32.h
bitcoin-chainstate.cpp refactor, validation: Add ChainstateManagerOpts db options 2023-02-10 04:39:11 -04:00
bitcoin-cli-res.rc
bitcoin-cli.cpp refactor: Stop using gArgs global in system.cpp 2023-02-27 14:21:13 -05:00
bitcoin-tx-res.rc
bitcoin-tx.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
bitcoin-util-res.rc
bitcoin-util.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
bitcoin-wallet-res.rc
bitcoin-wallet.cpp refactor: Stop using gArgs global in system.cpp 2023-02-27 14:21:13 -05:00
bitcoind-res.rc
bitcoind.cpp refactor: Stop using gArgs global in system.cpp 2023-02-27 14:21:13 -05:00
blockencodings.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
blockencodings.h script: remove out-of-date snprintf TODO 2023-02-15 14:42:28 -08:00
blockfilter.cpp refactor: use Hash helper for double-SHA256 calculations 2023-01-31 19:34:35 +01:00
blockfilter.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
chain.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
chain.h refactor: add kernel/cs_main.* 2023-01-05 09:05:14 +00:00
chainparams.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
chainparams.h
chainparamsbase.cpp
chainparamsbase.h
chainparamsseeds.h Merge bitcoin/bitcoin#25853: net: update testnet torv3 hardcoded seeds for 24.x 2022-09-09 10:07:39 +02:00
checkqueue.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
clientversion.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
clientversion.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
coins.cpp Add deterministic mode to CCoinsViewCache 2023-02-02 09:00:15 -05:00
coins.h Add deterministic mode to CCoinsViewCache 2023-02-02 09:00:15 -05:00
compressor.cpp
compressor.h
core_io.h rpc: decode Miniscript descriptor when possible in decodescript 2023-02-03 18:15:42 +01:00
core_memusage.h
core_read.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
core_write.cpp rpc: decode Miniscript descriptor when possible in decodescript 2023-02-03 18:15:42 +01:00
cuckoocache.h clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
dbwrapper.cpp refactor, dbwrapper: Add DBParams and DBOptions structs 2023-02-10 04:39:11 -04:00
dbwrapper.h refactor, dbwrapper: Add DBParams and DBOptions structs 2023-02-10 04:39:11 -04:00
deploymentinfo.cpp
deploymentinfo.h
deploymentstatus.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
deploymentstatus.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
dummywallet.cpp
external_signer.cpp Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
external_signer.h Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
flatfile.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
flatfile.h
fs.cpp Merge bitcoin/bitcoin#25898: util: remove WSL 1 workaround in fs 2023-02-16 12:33:26 +00:00
fs.h clang-tidy: Fix performance-move-const-arg in headers 2023-01-18 15:47:06 +00:00
hash.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
hash.h Merge bitcoin/bitcoin#26649: refactor: Use AutoFile and HashVerifier (without ser-type and ser-version) where possible 2023-01-30 15:57:12 +00:00
headerssync.cpp Move headerssync logging to BCLog::NET 2022-08-30 12:09:04 -04:00
headerssync.h Track headers presync progress and log it 2022-08-29 08:10:35 -04:00
httprpc.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
httprpc.h
httpserver.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
httpserver.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
i2p.cpp Merge bitcoin/bitcoin#26837: I2P network optimizations 2023-02-22 17:58:41 +00:00
i2p.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
indirectmap.h
init.cpp refactor: Stop using gArgs global in system.cpp 2023-02-27 14:21:13 -05:00
init.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
key_io.cpp
key_io.h
key.cpp refactor: use Hash helper for double-SHA256 calculations 2023-01-31 19:34:35 +01:00
key.h clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
logging.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
logging.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
Makefile.am build: allow libitcoinkernel dll builds now that exports are fixed 2023-02-22 21:23:10 +00:00
Makefile.bench.include Merge bitcoin/bitcoin#16981: Improve runtime performance of --reindex 2022-11-15 19:23:39 -05:00
Makefile.crc32c.include build: Create .la library for crc32c 2022-04-26 16:25:38 -04:00
Makefile.leveldb.include Use more specific path when including memenv.h header 2022-06-23 15:33:01 +02:00
Makefile.minisketch.include build: Build minisketch test in make check, not in make 2022-07-21 10:42:09 +01:00
Makefile.qt_locale.include qt: Periodic translation updates from Transifex 2023-02-27 13:53:29 +00:00
Makefile.qt.include build: extract $(BOOST_CPPFLAGS) from $(BITCOIN_INCLUDES) 2022-09-13 17:15:17 +01:00
Makefile.qttest.include build: extract $(BOOST_CPPFLAGS) from $(BITCOIN_INCLUDES) 2022-09-13 17:15:17 +01:00
Makefile.test_fuzz.include [fuzz] Move ConsumeNetAddr to fuzz/util/net.h 2022-11-17 14:52:45 +00:00
Makefile.test_util.include Merge bitcoin/bitcoin#26940: test: create random and coins utils, add amount helper, dedupe add_coin 2023-02-17 17:28:14 -05:00
Makefile.test.include test: wallet, add coverage for outputs grouping process 2023-03-06 09:45:40 -03:00
Makefile.univalue.include
mapport.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
mapport.h Remove configure-time setting of DEFAULT_UPNP 2023-01-28 15:24:13 +00:00
memusage.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
merkleblock.cpp
merkleblock.h
net_permissions.cpp
net_permissions.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
net_processing.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
net_processing.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
net_types.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
net_types.h
net.cpp Merge bitcoin/bitcoin#26837: I2P network optimizations 2023-02-22 17:58:41 +00:00
net.h Merge bitcoin/bitcoin#26837: I2P network optimizations 2023-02-22 17:58:41 +00:00
netaddress.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
netaddress.h Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
netbase.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
netbase.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
netgroup.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
netgroup.h [netgroupman] Remove NetGroupManager::GetAsmap() 2022-04-20 14:35:53 +01:00
netmessagemaker.h
noui.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
noui.h
outputtype.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
outputtype.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
pow.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
pow.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
prevector.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
protocol.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
protocol.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
psbt.cpp Merge bitcoin/bitcoin#24149: Signing support for Miniscript Descriptors 2023-02-16 10:01:33 +00:00
psbt.h Merge bitcoin/bitcoin#26997: psbt: s/transcation/transaction/ 2023-02-16 09:39:50 +00:00
pubkey.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
pubkey.h Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
random.cpp random: consolidate WIN32 #ifdefs 2023-02-17 15:01:50 +00:00
random.h Merge bitcoin/bitcoin#26153: Reduce wasted pseudorandom bytes in ChaCha20 + various improvements 2023-02-15 14:58:47 +00:00
randomenv.cpp Merge bitcoin/bitcoin#26826: refactor: remove windows-only compat.h usage in randomenv 2023-01-23 16:36:27 +01:00
randomenv.h
rest.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
rest.h
reverse_iterator.h doc: remove usages of C++11 2023-01-12 13:42:44 +00:00
scheduler.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
scheduler.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
serialize.h clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
shutdown.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
shutdown.h
signet.cpp
signet.h
span.h clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
streams.h Merge bitcoin/bitcoin#26992: refactor: Remove unused CDataStream SerializeMany constructor 2023-02-02 10:47:37 +00:00
sync.cpp
sync.h doc: Export threadsafety.h from sync.h 2023-01-25 09:33:26 +01:00
threadsafety.h
timedata.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
timedata.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
tinyformat.h clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
torcontrol.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
torcontrol.h clang-tidy: Fix modernize-use-default-member-init in headers 2023-01-31 11:50:10 +00:00
txdb.cpp refactor, txdb: Use DBParams struct in CBlockTreeDB 2023-02-10 04:39:11 -04:00
txdb.h refactor, txdb: Use DBParams struct in CBlockTreeDB 2023-02-10 04:39:11 -04:00
txmempool.cpp Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
txmempool.h Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
txorphanage.cpp Merge bitcoin/bitcoin#26551: p2p: Track orphans by who provided them 2023-01-26 10:36:18 +00:00
txorphanage.h Merge bitcoin/bitcoin#26551: p2p: Track orphans by who provided them 2023-01-26 10:36:18 +00:00
txrequest.cpp
txrequest.h
uint256.cpp refactor: modernize the implementation of uint256.* 2022-12-10 14:34:44 -06:00
uint256.h Merge bitcoin/bitcoin#26345: refactor: modernize the implementation of uint256.* 2023-02-06 13:56:51 -05:00
undo.h
validation.cpp Merge bitcoin/bitcoin#23897: refactor: Move calculation logic out from CheckSequenceLocksAtTip() 2023-02-28 16:53:02 +00:00
validation.h Merge bitcoin/bitcoin#23897: refactor: Move calculation logic out from CheckSequenceLocksAtTip() 2023-02-28 16:53:02 +00:00
validationinterface.cpp Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
validationinterface.h refactor: add kernel/cs_main.* 2023-01-05 09:05:14 +00:00
version.h
versionbits.cpp refactor: use braced init for integer constants instead of c style casts 2023-01-03 19:31:29 -06:00
versionbits.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
walletinitinterface.h scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
warnings.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
warnings.h