Commit Graph

259 Commits

Author SHA1 Message Date
John Newbery
f572f2b204 [addrman] Set m_asmap in CAddrMan initializer list
This allows us to make it const.
2021-08-27 10:55:41 +01:00
Russell Yanofsky
e4709c7b56 Start using init makeNode, makeChain, etc methods
Use interfaces::Init::make* methods instead of interfaces::Make*
functions, so interfaces can be constructed differently in different
executables without having to change any code. (So for example
bitcoin-gui can make an interfaces::Node pointer that communicates with
a bitcoin-node subprocess, while bitcoin-qt can make an interfaces::Node
pointer that starts node code in the same process.)
2021-08-17 03:05:15 -05:00
fanquake
803ef70fd9
Merge bitcoin/bitcoin#20233: addrman: Make consistency checks a runtime option
a4d78546b0 [addrman] Make addrman consistency checks a runtime option (John Newbery)
10aac24145 [tests] Make deterministic addrman use nKey = 1 (John Newbery)
fa9710f62c [addrman] Add deterministic argument to CAddrMan ctor (John Newbery)
ee458d84fc Add missing const to CAddrMan::Check_() (MarcoFalke)

Pull request description:

  CAddrMan has internal consistency checks. Currently, these are only run when the program is compiled with the  `DEBUG_ADDRMAN` option. This option is not enabled on any of our CI builds, and it's likely that no-one is running them at all.

  This PR makes consistency checks a (hidden) runtime option that can be enabled with `-checkaddrman`, where `-checkaddrman=n` will result in the consistency checks running every n operations (similar to `-checkmempool=n`). We set the ratio to 1/100 for our unit tests, and leave it disabled by default for all networks. Additionally, a consistency check failure now asserts, rather than logging and continuing. This matches the behavior of CTxMemPool and TxRequestTracker, where a failed consistency check asserts.

ACKs for top commit:
  jonatack:
    ACK a4d78546b0 per `git diff 00fd089 a4d7854`, tested by adding logging similar to #22479 and running with `-checkaddrman=<n>` for various values 0/1/10/100 etc, tested the updated docs with `bitcoind -help-debug | grep -A2 "checkaddrman\|checkmempool"` and verified rebased on master that compiling with `CPPFLAGS="-DDEBUG_ADDRMAN"` no longer causes the build to error.
  mzumsande:
    Code-review ACK a4d78546b0
  theStack:
    Code-review ACK a4d78546b0

Tree-SHA512: eaee003f7a99154822c5b5efbc62008d32c1efbecc6fec6e183427f6b2ae5d30b3be7924e3a7271b1a1de91517f5bd2a70011d45358c3105c6a0702f12b70f7c
2021-08-13 17:03:01 +08:00
John Newbery
a4d78546b0 [addrman] Make addrman consistency checks a runtime option
Currently addrman consistency checks are a compile time option, and are not
enabled in our CI. It's unlikely anyone is running these consistency checks.

Make them a runtime option instead, where users can enable addrman
consistency checks every n operations (similar to mempool tests). Update
the addrman unit tests to do internal consistency checks every 100
operations (checking on every operations causes the test runtime to
increase by several seconds).

Also assert on a failed addrman consistency check to terminate program
execution.
2021-08-12 10:41:11 +01:00
Samuel Dobson
b1a672d158
Merge bitcoin/bitcoin#22337: wallet: Use bilingual_str for errors
92993aa5cf Change SignTransaction's input_errors to use bilingual_str (Andrew Chow)
171366e89b Use bilingual_str for address fetching functions (Andrew Chow)
9571c69b51 Add bilingual_str::clear() (Andrew Chow)

Pull request description:

  In a couple of places in the wallet, errors are `std::string`. In order for these errors to be translated, change them to use `bilingual_str`.

ACKs for top commit:
  hebasto:
    re-ACK 92993aa5cf, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/22337#pullrequestreview-694542729) review, verified with
  klementtan:
    Code review ACK 92993aa5cf
  meshcollider:
    Code review ACK 92993aa5cf

Tree-SHA512: 5400e419dd87db8c49b67ed0964de2d44b58010a566ca246f2f0760ed9ef6a9b6f6df7a6adcb211b315b74c727bfe8c7d07eb5690b5922fa5828ceef4c83461f
2021-08-09 14:45:12 +12:00
John Newbery
fa9710f62c [addrman] Add deterministic argument to CAddrMan ctor
Removes the need for tests to update nKey and insecure_rand after constructing
a CAddrMan.
2021-08-05 17:10:30 +01:00
Larry Ruane
703b1e612a Close minor startup race between main and scheduler threads
Don't schedule class PeerManagerImpl's background tasks from its
constructor, but instead do that from a separate method,
StartScheduledTasks(), that can be called later at the end of startup,
after other things, such as the active chain, are initialzed.
2021-07-30 16:34:09 -06:00
MarcoFalke
faa54e3757
Move pblocktree global to BlockManager 2021-07-15 13:54:09 +02:00
James O'Beirne
617661703a
validation: make CChainState::m_mempool optional
Since we now have multiple chainstate objects, only one of them is active at any given
time. An active chainstate has a mempool, but there's no point to others having one.

This change will simplify proposed assumeutxo semantics. See the discussion here:
https://github.com/bitcoin/bitcoin/pull/15606#pullrequestreview-692965905

Co-authored-by: Russell Yanofsky <russ@yanofsky.org>
2021-07-13 11:11:35 -04:00
Andrew Chow
92993aa5cf Change SignTransaction's input_errors to use bilingual_str 2021-07-01 12:57:53 -04:00
fanquake
8071ec179d
Merge bitcoin/bitcoin#21789: refactor: Remove ::Params() global from CChainState
fa0d9211ef refactor: Remove chainparams arg from CChainState member functions (MarcoFalke)
fa38947125 refactor: Remove ::Params() global from inside CChainState member functions (MarcoFalke)

Pull request description:

  The `::Params()` global is verbose and confusing. Also it makes tests a bit harder to write because they'd have to mock a global.

  Fix all issues by simply using a member variable that points to the right params.

  (Can be reviewed with `--word-diff-regex=.`)

ACKs for top commit:
  jnewbery:
    ACK fa0d9211ef
  kiminuo:
    utACK fa0d9211
  theStack:
    ACK fa0d9211ef 🍉

Tree-SHA512: 44676b19c9ed471ccb536331d3029bad192d7d50f394fd7b8527ec431452aeec8c4494164b9cf8e16e0123c4463b16be864366c6b599370032c17262625a0356
2021-06-29 11:22:57 +08:00
Vasil Dimov
d197977ae2
banman: save the banlist in a JSON format on disk
Save the banlist in `banlist.json` instead of `banlist.dat`.

This makes it possible to store Tor v3 entries in the banlist on disk
(and any other addresses that cannot be serialized in addrv1 format).

Only read `banlist.dat` if it exists and `banlist.json` does not
exist (first start after an upgrade).

Supersedes https://github.com/bitcoin/bitcoin/pull/20904
Resolves https://github.com/bitcoin/bitcoin/issues/19748
2021-06-21 14:39:44 +02:00
MarcoFalke
fa0d9211ef
refactor: Remove chainparams arg from CChainState member functions
Passing this is confusing and redundant with the m_params member.
2021-06-13 09:43:54 +02:00
fanquake
a55904a80c
Merge bitcoin/bitcoin#21866: [Bundle 7/7] validation: Farewell, global Chainstate!
6f994882de validation: Farewell, global Chainstate! (Carl Dong)
972c5166ee qt/test: Reset chainman in ~ChainstateManager instead (Carl Dong)
6c3b5dc0c1 scripted-diff: tree-wide: Remove all review-only assertions (Carl Dong)
3e82abb8dd tree-wide: Remove stray review-only assertion (Carl Dong)
f323248aba qt/test: Use existing chainman in ::TestGUI (can be scripted-diff) (Carl Dong)
6c15de129c scripted-diff: wallet/test: Use existing chainman (Carl Dong)
ee0ab1e959 fuzz: Initialize a TestingSetup for test_one_input (Carl Dong)
0d61634c06 scripted-diff: test: Use existing chainman in unit tests (Carl Dong)
e197076219 test: Pass in CoinsTip to ValidateCheckInputsForAllFlags (Carl Dong)
4d99b61014 test/miner_tests: Pass in chain tip to CreateBlockIndex (Carl Dong)
f0dd5e6bb4 test/util: Use existing chainman in ::PrepareBlock (Carl Dong)
464c313e30 init: Use existing chainman (Carl Dong)

Pull request description:

  Based on:  #21767

  à la Mr. Sandman
  ```
  Mr. Chainman, bring me a tip (bung, bung, bung, bung)
  Make it the most work that I've ever seen (bung, bung, bung, bung)
  Rewind old tip till we're at the fork point (bung, bung, bung, bung)
  Then tell it that it's time to call Con-nectTip

  Chainman, I'm so alone (bung, bung, bung, bung)
  No local objects to call my own (bung, bung, bung, bung)
  Please make sure I have a ref
  Mr. Chainman, bring me a tip!
  ```

  This is the last bundle in the #20158 series. Thanks everyone for their diligent review.
  I would like to call attention to https://github.com/bitcoin/bitcoin/issues/21766, where a few leftover improvements were collated.

  - Remove globals:
    - `ChainstateManager g_chainman`
    - `CChainState& ChainstateActive()`
    - `CChain& ChainActive()`
  - Remove all review-only assertions.

ACKs for top commit:
  jamesob:
    reACK 6f994882de based on the contents of
  ariard:
    Code Review ACK 6f99488.
  jnewbery:
    utACK 6f994882de
  achow101:
    Code Review ACK 6f994882de
  ryanofsky:
    Code review ACK 6f994882de.

Tree-SHA512: 4052ea79360cf0efd81ad0ee3f982e1d93aab1837dcec75f875a56ceda085de078bb3099a2137935d7cc2222004ad88da94b605ef5efef35cb6bc733725debe6
2021-06-12 11:29:31 +08:00
Carl Dong
6f994882de validation: Farewell, global Chainstate! 2021-06-10 15:05:25 -04:00
Carl Dong
0d61634c06 scripted-diff: test: Use existing chainman in unit tests
-BEGIN VERIFY SCRIPT-
git ls-files -- src/test \
    | grep -v '^src/test/fuzz' \
    | xargs sed -i -E \
            -e 's@g_chainman\.m_blockman@m_node.chainman->m_blockman@g' \
            -e 's@([^:])(Chain(state|)Active)@\1::\2@g' \
            -e 's@::Chain(state|)Active\(\)@m_node.chainman->ActiveChain\1()@g'
-END VERIFY SCRIPT-
2021-06-10 15:04:39 -04:00
Russell Yanofsky
493fb47c57 Make SetupServerArgs callable without NodeContext
bitcoin-gui code needs to call SetupServerArgs but will not have a
NodeContext object if it is communicating with an external bitcoin-node
process.
2021-06-10 09:58:45 -05:00
MarcoFalke
f63fc53c2a
Merge bitcoin/bitcoin#21767: [Bundle 6/n] Prune g_chainman usage in auxiliary modules
7a799c9c2b index: refactor-only: Reuse CChain ref (Carl Dong)
db33cde80f index: Add chainstate member to BaseIndex (Carl Dong)
f4a47a1feb bench: Use existing chainman in AssembleBlock (Carl Dong)
91226eb917 bench: Use existing NodeContext in DuplicateInputs (Carl Dong)
e6b4aa6eb5 miner: Pass in chainman to RegenerateCommitments (Carl Dong)
9ecade1425 rest: Add GetChainman function and use it (Carl Dong)
fc1c282845 rpc/blockchain: Use existing blockman in gettxoutsetinfo (Carl Dong)

Pull request description:

  Overall PR: #20158 (tree-wide: De-globalize ChainstateManager)

  The first 2 commits are fixups addressing review for the last bundle: #21391

  NEW note:
  1. I have opened #21766 which keeps track of potential improvements where the flaws already existed before the de-globalization work, please post on that issue about these improvements, thanks!

  Note to reviewers:
  1. This bundle may _apparently_ introduce usage of `g_chainman` or `::Chain(state|)Active()` globals, but these are resolved later on in the overall PR. [Commits of overall PR](https://github.com/bitcoin/bitcoin/pull/20158/commits)
  2. There may be seemingly obvious local references to `ChainstateManager` or other validation objects which are not being used in callers of the current function in question, this is done intentionally to **_keep each commit centered around one function/method_** to ease review and to make the overall change systematic. We don't assume anything about our callers. Rest assured that once we are considering that particular caller in later commits, we will use the obvious local references. [Commits of overall PR](https://github.com/bitcoin/bitcoin/pull/20158/commits)
  3. When changing a function/method that has many callers (e.g. `LookupBlockIndex` with 55 callers), it is sometimes easier (and less error-prone) to use a scripted-diff. When doing so, there will be 3 commits in sequence so that every commit compiles like so:
  1. Add `new_function`, make `old_function` a wrapper of `new_function`, divert all calls to `old_function` to `new_function` **in the local module only**
  2. Scripted-diff to divert all calls to `old_function` to `new_function` **in the rest of the codebase**
  3. Remove `old_function`

ACKs for top commit:
  jarolrod:
    ACK  7a799c9
  ariard:
    Code Review ACK 7a799c9
  fjahr:
    re-ACK 7a799c9c2b
  MarcoFalke:
    review ACK 7a799c9c2b 🌠
  ryanofsky:
    Code review ACK 7a799c9c2b. Basically no change since last review except fixed rebase conflicts and a new comment about REST Ensure()
  jamesob:
    conditional ACK 7a799c9c2b ([`jamesob/ackr/21767.1.dongcarl.bundle_6_n_prune_g_chai`](https://github.com/jamesob/bitcoin/tree/ackr/21767.1.dongcarl.bundle_6_n_prune_g_chai))

Tree-SHA512: 531c00ddcb318817457db2812d9a9d930bc664e58e6f7f1c746350732b031dd624270bfa6b9f49d8056aeb6321d973f0e38e4ff914acd6768edd8602c017d10e
2021-06-01 13:34:18 +02:00
W. J. van der Laan
7257e50dba
Merge bitcoin/bitcoin#20833: rpc/validation: enable packages through testmempoolaccept
13650fe2e5 [policy] detect unsorted packages (glozow)
9ef643e21b [doc] add release note for package testmempoolaccept (glozow)
c4259f4b7e [test] functional test for packages in RPCs (glozow)
9ede34a6f2 [rpc] allow multiple txns in testmempoolaccept (glozow)
ae8e6df709 [policy] limit package sizes (glozow)
c9e1a26d1f [fuzz] add ProcessNewPackage call in tx_pool fuzzer (glozow)
363e3d916c [test] unit tests for ProcessNewPackage (glozow)
cd9a11ac96 [test] make submit optional in CreateValidMempoolTransaction (glozow)
2ef187941d [validation] package validation for test accepts (glozow)
578148ded6 [validation] explicit Success/Failure ctors for MempoolAcceptResult (glozow)
b88d77aec5 [policy] Define packages (glozow)
249f43f3cc [refactor] add option to disable RBF (glozow)
897e348f59 [coins/mempool] extend CCoinsViewMemPool to track temporary coins (glozow)
42cf8b25df [validation] make CheckSequenceLocks context-free (glozow)

Pull request description:

  This PR enables validation dry-runs of packages through the `testmempoolaccept` RPC. The expectation is that the results returned from `testmempoolaccept` are what you'd get from test-then-submitting each transaction individually, in that order (this means the package is expected to be sorted in topological order, for now at least). The validation is also atomic: in the case of failure, it immediately halts and may return "unfinished" `MempoolAcceptResult`s for transactions that weren't fully validated. The API for 1 transaction stays the same.

  **Motivation:**
  - This allows you to test validity for transaction chains (e.g. with multiple spending paths and where you don't want to broadcast yet); closes #18480.
  - It's also a first step towards package validation in a minimally invasive way.
  - The RPC commit happens to close #21074 by clarifying the "allowed" key.

  There are a few added restrictions on the packages, mostly to simplify the logic for areas that aren't critical to main package use cases:
  - No package can have conflicts, i.e. none of them can spend the same inputs, even if it would be a valid BIP125 replacement.
  - The package cannot conflict with the mempool, i.e. RBF is disabled.
  - The total count of the package cannot exceed 25 (the default descendant count limit), and total size cannot exceed 101KvB (the default descendant size limit).

  If you're looking for review comments and github isn't loading them, I have a gist compiling some topics of discussion [here](https://gist.github.com/glozow/c3acaf161c95bba491fce31585b2aaf7)

ACKs for top commit:
  laanwj:
    Code review re-ACK 13650fe2e5
  jnewbery:
    Code review ACK 13650fe2e5
  ariard:
    ACK 13650fe

Tree-SHA512: 8c5cbfa91a6c714e1c8710bb281d5ff1c5af36741872a7c5df6b24874d6272b4a09f816cb8a4c7de33ef8e1c2a2c252c0df5105b7802f70bc6ff821ed7cc1a2f
2021-05-27 22:40:24 +02:00
Carl Dong
e6b4aa6eb5 miner: Pass in chainman to RegenerateCommitments
Pass in chainman instead of prev_block so that we can enforce the
block.hashPrevBlock refers to prev_block invariant in the function
itself.

We should probably rethink BlockAssembler's API and somehow include
commitment regeneration functionality in there. Something like a variant
of CreateNewBlock that takes in a std::vector<TxRef> and return a CBlock
instead of CBlockTemplate. That could avoid reaching for
LookupBlockIndex at all.
2021-05-27 13:50:11 -04:00
glozow
cd9a11ac96 [test] make submit optional in CreateValidMempoolTransaction
This allows us to easily create transaction chains for package
validation. We don't test_accept if submit=false because we want to be
able to make transactions that wouldn't pass ATMP (i.e. a child
transaction in a package would fail due to missing inputs).
2021-05-24 14:42:10 +01:00
Kiminuo
4d8189f620 scripted-diff: Change ArgsManager.GetDataDirPath() to ArgsManager.GetDataDirBase() in tests
-BEGIN VERIFY SCRIPT-
git ls-files src/test/*_tests.cpp src/test/util/setup_common.cpp | xargs sed -i 's/.GetDataDirPath()/.GetDataDirBase()/g';
-END VERIFY SCRIPT-
2021-05-24 10:29:57 +02:00
MarcoFalke
2e30e328a7
Merge bitcoin/bitcoin#19064: refactor: Cleanup thread ctor calls
792be53d3e refactor: Replace std::bind with lambdas (Hennadii Stepanov)
a508f718f3 refactor: Use appropriate thread constructor (Hennadii Stepanov)
30e4448215 refactor: Make TraceThread a non-template free function (Hennadii Stepanov)

Pull request description:

  This PR does not change behavior.
  Its goal is to improve readability and maintainability of the code.

ACKs for top commit:
  jnewbery:
    utACK 792be53d3e
  jonatack:
    tACK 792be53d3e
  MarcoFalke:
    cr ACK 792be53d3e

Tree-SHA512: a03142f04f370f6bc02bd3ddfa870819b51740fcd028772241d68c84087f95a2d78207cbd5edb3f7c636fcf2d76192d9c59873f8f0af451d3b05c0cf9cf234df
2021-05-12 08:51:32 +02:00
Hennadii Stepanov
a508f718f3
refactor: Use appropriate thread constructor 2021-04-29 18:39:01 +03:00
Hennadii Stepanov
30e4448215
refactor: Make TraceThread a non-template free function
Also it is moved into its own module.
2021-04-25 12:28:44 +03:00
Kiminuo
bb8d1c6e02 Change ClearDataDirPathCache() to ArgsManager.ClearPathCache(). 2021-04-18 12:07:00 +02:00
Kiminuo
511ce3a26b BasicTestingSetup: Add ArgsManager. 2021-04-18 11:59:28 +02:00
MarcoFalke
0dd7b23489
Merge #21391: [Bundle 5/n] Prune g_chainman usage in RPC modules
586190f0b4 rpc/rest: Take and reuse local Chain/ChainState obj (Carl Dong)
bc3bd36902 rpc: style: Improve BuriedForkDescPushBack signature (Carl Dong)
f99913969f rpc: Remove unnecessary casting of block height (Carl Dong)
6a3d192020 rpc: Tidy up local references (see commit message) (Carl Dong)
038854f31e rest/rpc: Remove now-unused old Ensure functions (Carl Dong)
6fb65b49f4 scripted-diff: rest/rpc: Use renamed EnsureAny*() (Carl Dong)
1570c7ee98 rpc: Add renamed EnsureAny*() functions (Carl Dong)
306b1cd3ee rpc: Add alt Ensure* functions acepting NodeContext (Carl Dong)
d7824acdb9 rest: Use existing NodeContext (Carl Dong)
3f08934799 rest: Pass in NodeContext to rest_block (Carl Dong)
7be0671b95 rpc/rawtx: Use existing NodeContext (Carl Dong)
60dc05afc6 rpc/mining: Use existing NodeContext (Carl Dong)
d485e815e2 rpc/blockchain: Use existing NodeContext (Carl Dong)
d0abf0bf42 rpc/*,rest: Add review-only assertion to EnsureChainman (Carl Dong)
cced0f46c9 miner: Pass in previous CBlockIndex to RegenerateCommitments (Carl Dong)

Pull request description:

  Overall PR: #20158 (tree-wide: De-globalize ChainstateManager)

  Based on:
  - [x] #21270 | [Bundle 4/n] Prune g_chainman usage in validation-adjacent modules
  - [x] #21525 | [Bundle 4.5/n] Followup fixups to bundle 4

  Note to reviewers:
  1. This bundle may _apparently_ introduce usage of `g_chainman` or `::Chain(state|)Active()` globals, but these are resolved later on in the overall PR. [Commits of overall PR](https://github.com/bitcoin/bitcoin/pull/20158/commits)
  2. There may be seemingly obvious local references to `ChainstateManager` or other validation objects which are not being used in callers of the current function in question, this is done intentionally to **_keep each commit centered around one function/method_** to ease review and to make the overall change systematic. We don't assume anything about our callers. Rest assured that once we are considering that particular caller in later commits, we will use the obvious local references. [Commits of overall PR](https://github.com/bitcoin/bitcoin/pull/20158/commits)
  3. When changing a function/method that has many callers (e.g. `LookupBlockIndex` with 55 callers), it is sometimes easier (and less error-prone) to use a scripted-diff. When doing so, there will be 3 commits in sequence so that every commit compiles like so:
  	1. Add `new_function`, make `old_function` a wrapper of `new_function`, divert all calls to `old_function` to `new_function` **in the local module only**
  	2. Scripted-diff to divert all calls to `old_function` to `new_function` **in the rest of the codebase**
  	3. Remove `old_function`

ACKs for top commit:
  ryanofsky:
    Code review ACK 586190f0b4. Since last review, no changes to existing commits, just some simple new commits added: three new commits renaming std::any Ensure functions (scripted diff commit and manual pre/post commits), and one new commit factoring out a repeated `ActiveChain()` call made in a loop. Thanks for the updates!
  jnewbery:
    utACK 586190f0b4
  MarcoFalke:
    review ACK 586190f0b4 🍯

Tree-SHA512: 64b677fb50141805b55c3f1afe68fcd298f9a071a359bdcd63256d52e334f83e462f31fb3ebee9b630da8f1d912a03a128cfc38179e7aaec29a055744a98478c
2021-04-17 17:37:37 +02:00
MarcoFalke
fa40d6a1c4
test: Reset mocktime in the common setup
Doing it there will reduce code bloat and also ensure no test can "forget" to reset it
2021-04-14 17:38:07 +02:00
MarcoFalke
fa6183d776
test: Remove option to make TestChain100Setup non-deterministic
Seems odd to have an option for non-deterministic tests
when the goal should be for all tests to be deterministic.

Can be reviewed with `--ignore-all-space`.
2021-04-08 08:59:00 +02:00
MarcoFalke
fa732bccb3
test: Use compressed keys in TestChain100Setup
coinbaseKey.MakeNewKey(true); creates a compressed key and there is no reason
for the deterministic setup to use uncompressed ones.
2021-04-08 08:58:44 +02:00
Carl Dong
cced0f46c9 miner: Pass in previous CBlockIndex to RegenerateCommitments 2021-04-05 11:13:51 -04:00
MarcoFalke
80a699fda9
Merge #21525: [Bundle 4.5/n] Followup fixups to bundle 4
693414d271 node/ifaces: ChainImpl: Use an accessor for ChainMan (Carl Dong)
98c4e252f0 node/ifaces: NodeImpl: Use an accessor for ChainMan (Carl Dong)
7e8b5ee814 validation: Make BlockManager::LookupBlockIndex const (Carl Dong)
88aead263c node: Avoid potential UB by asserting assumptions (Carl Dong)
1dd8ed7a84 net_processing: Move comments to declarations (Carl Dong)
07156eb387 node/coinstats: Replace #include with fwd-declaration (Carl Dong)
7b8e976cd5 miner: Add chainstate member to BlockAssembler (Carl Dong)
e62067e7bc Revert "miner: Pass in chainstate to BlockAssembler::CreateNewBlock" (Carl Dong)
eede0647b0 Revert "scripted-diff: Invoke CreateNewBlock with chainstate" (Carl Dong)
0c1b2bc549 Revert "miner: Remove old CreateNewBlock w/o chainstate param" (Carl Dong)

Pull request description:

  Chronological history of this changeset:
  1. Bundle 4 (#21270) got merged
  2. Posthumous reviews were posted
  3. These changes were prepended in bundle 5
  4. More reviews were added in bundle 5
  5. Someone suggested that we split the prepended changes up to another PR
  6. This is that PR

  In the future, I will just do posthumous review changes in another PR instead. I apologize for the confusion.

  Addresses posthumous reviews on bundle 4:
    - From jnewbery:
      - https://github.com/bitcoin/bitcoin/pull/21270#issuecomment-796738048
        - I didn't fix this one, but I added a `TODO` comment so that we don't lost track of it
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r592291225
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r592296942
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r592299738
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r592301704
    - From MarcoFalke:
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r593096212
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r593097032
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r593097867
      - https://github.com/bitcoin/bitcoin/pull/21270#discussion_r593100570

  Addresses reviews on bundle 5:
  - Checking chainman existence before locking cs_main
    - MarcoFalke
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r596601776
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r596601876
  - Appropriate locking, usage of chainman, and control flow in `src/node/interfaces.cpp`
    - MarcoFalke
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r596601383
    - jnewbery
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r597029360
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r597029921
    - ryanofsky
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r597163828
  - Style/comment formatting changes
    - jnewbery
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r597026552
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r597027186
  - Making LookupBlockIndex const
    - jnewbery
      - https://github.com/bitcoin/bitcoin/pull/21391#discussion_r597035062

ACKs for top commit:
  MarcoFalke:
    review ACK 693414d271 🛐
  ryanofsky:
    Code review ACK 693414d271. I reviewed this previously as part of #21391. I am a fan of the increasingly complicated bundle numbering, and kind of hope there in the next round there is some way we can get bundles 5.333333 and 5.666667!
  jamesob:
    ACK 693414d271 ([`jamesob/ackr/21525.1.dongcarl.bundle_4_5_n_followup_f`](https://github.com/jamesob/bitcoin/tree/ackr/21525.1.dongcarl.bundle_4_5_n_followup_f))

Tree-SHA512: 9bdc199f70400d01764e1bd03c25bdb6cff26dcef60e4ca3b649baf8d017a2dfc1f058099067962b4b6ccd32d078002b1389d733039f4c337558cb70324c0ee3
2021-04-01 10:58:53 +02:00
Carl Dong
7b8e976cd5 miner: Add chainstate member to BlockAssembler 2021-03-24 15:40:56 -04:00
Carl Dong
eede0647b0 Revert "scripted-diff: Invoke CreateNewBlock with chainstate"
This reverts commit 46b7f29340.
2021-03-24 15:40:56 -04:00
John Newbery
392a95d393 [net_processing] Keep addrman reference in PeerManager 2021-03-20 10:24:40 +00:00
John Newbery
1c25adf6d2 [net] Construct addrman outside connman
node.context owns the CAddrMan. CConnman holds a reference to
the CAddrMan.
2021-03-20 10:24:36 +00:00
MarcoFalke
e0bc27a14c
Merge #21404: refactor: Remove MakeUnique<T>()
1a6323bdbe doc: update developer notes for removal of MakeUnique (fanquake)
3ba2840e7e scripted-diff: remove MakeUnique<T>() (fanquake)

Pull request description:

  Since requiring C++17, this is just pointless abstraction. I think we should just "tear the band-aid off" and remove it. Similar to the changes happening in #21366.

  Also, having a comment saying this is deprecated doesn't prevent it's usage in new code. i.e : https://github.com/bitcoin/bitcoin/pull/20946#discussion_r561949731.

  The repository is fairly quiet at the moment, so any potential complaints about having to rebase should be minimal. Might as well get this over and done with.

ACKs for top commit:
  jnewbery:
    utACK 1a6323bdbe
  practicalswift:
    cr ACK 1a6323bdbe: patch looks correct
  ajtowns:
    ACK 1a6323bdbe -- code review only
  glozow:
    ACK 1a6323bdbe looks correct

Tree-SHA512: 4a14b9611b60b9b3026b54d6f5a2dce4c5d9b63a7b93d7de1307512df736503ed84bac66e7b93372c76e3117f49bf9f29cd473d3a47cb41fb2775bc10234736f
2021-03-12 08:34:15 +01:00
fanquake
3ba2840e7e
scripted-diff: remove MakeUnique<T>()
-BEGIN VERIFY SCRIPT-
git rm src/util/memory.h
sed -i -e 's/MakeUnique/std::make_unique/g' $(git grep -l MakeUnique src)
sed -i -e '/#include <util\/memory.h>/d' $(git grep -l '#include <util/memory.h>' src)
sed -i -e '/util\/memory.h \\/d' src/Makefile.am
-END VERIFY SCRIPT-
2021-03-11 13:45:14 +08:00
Carl Dong
2c3ba00693 miner: Pass in blockman to ::RegenerateCommitments
REQUIRES ATTENTION
2021-03-08 15:54:31 -05:00
Carl Dong
46b7f29340 scripted-diff: Invoke CreateNewBlock with chainstate
-BEGIN VERIFY SCRIPT-
find_regex='(\.|->)CreateNewBlock\(' \
    && git grep -l -E "$find_regex" -- src \
        | grep -v '^src/miner\.\(cpp\|h\)$' \
        | xargs sed -i -E 's@'"$find_regex"'@\0::ChainstateActive(), @g'
-END VERIFY SCRIPT-
2021-03-08 15:54:31 -05:00
Carl Dong
4927c9e699 validation: Remove global ::LoadGenesisBlock 2021-02-22 11:48:39 -05:00
Carl Dong
3704433c4f scripted-diff: Invoke ::AcceptToMemoryPool with chainstate
-BEGIN VERIFY SCRIPT-
find_regex='\bAcceptToMemoryPool\(' \
    && git grep -l -E "$find_regex" -- src \
        | grep -v '^src/validation\.\(cpp\|h\)$' \
        | xargs sed -i -E 's@'"$find_regex"'@\0::ChainstateActive(), @g'
-END VERIFY SCRIPT-
2021-02-18 14:49:06 -05:00
Wladimir J. van der Laan
b805dbb0b9
Merge #19809: log: Prefix log messages with function name and source code location if -logsourcelocations is set
b4511e2e2e log: Prefix log messages with function name if -logsourcelocations is set (practicalswift)

Pull request description:

  Prefix log messages with function name if `-logfunctionnames` is set.

  Yes, exactly like `-logthreadnames` but for function names instead of thread names :)

  This is a small developer ergonomics improvement: I've found this to be a cheap/simple way to correlate log output and originating function.

  For me it beats the ordinary cycle of 1.) try to figure out a regexp matching the static part of the dynamic log message, 2.) `git grep -E 'Using .* MiB out of .* requested for signature cache'`, 3.) `mcedit filename.cpp` (`openemacs filename.cpp` works too!) and 4.) search for log message and scroll up to find the function name :)

  Without any logging parameters:

  ```
  $ src/bitcoind -regtest
  2020-08-25T03:29:04Z Using RdRand as an additional entropy source
  2020-08-25T03:29:04Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
  2020-08-25T03:29:04Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
  2020-08-25T03:29:04Z Loaded best chain: hashBestChain=0fff88f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22ff height=0 date=2011-02-02T23:16:42Z progress=1.000000
  2020-08-25T03:29:04Z block tree size = 1
  2020-08-25T03:29:04Z nBestHeight = 0
  2020-08-25T03:29:04Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
  2020-08-25T03:29:04Z 0 addresses found from DNS seeds
  ```

  With `-logthreadnames` and `-logfunctionnames`:

  ```
  $ src/bitcoind -regtest -logthreadnames -logfunctionnames
  2020-08-25T03:29:04Z [init] [ReportHardwareRand] Using RdRand as an additional entropy source
  2020-08-25T03:29:04Z [init] [InitSignatureCache] Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
  2020-08-25T03:29:04Z [init] [InitScriptExecutionCache] Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
  2020-08-25T03:29:04Z [init] [LoadChainTip] Loaded best chain: hashBestChain=0fff88f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22ff height=0 date=2011-02-02T23:16:42Z progress=1.000000
  2020-08-25T03:29:04Z [init] [AppInitMain] block tree size = 1
  2020-08-25T03:29:04Z [init] [AppInitMain] nBestHeight = 0
  2020-08-25T03:29:04Z [loadblk] [LoadMempool] Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
  2020-08-25T03:29:04Z [dnsseed] [ThreadDNSAddressSeed] 0 addresses found from DNS seeds
  ```

ACKs for top commit:
  laanwj:
    Code review ACK b4511e2e2e
  MarcoFalke:
    review ACK b4511e2e2e 🌃

Tree-SHA512: d100f5364630c323f31d275259864c597f7725e462d5f4bdedcc7033ea616d7fc0d16ef1b2af557e692f4deea73c6773ccfc681589e7bf6ba970b9ec169040c7
2021-02-18 14:37:51 +01:00
Amiti Uttarwar
9a3bbe8fc5 [test] Introduce a unit test helper to create a valid mempool transaction. 2021-02-16 12:23:00 -08:00
James O'Beirne
31d225274f
tests: add deterministic chain generation unittest fixture 2021-02-12 07:53:36 -06:00
Wladimir J. van der Laan
d0d256536c
Merge #21016: refactor: remove boost::thread_group usage
dc8be12510 refactor: remove boost::thread_group usage (fanquake)

Pull request description:

  Post #18710, there isn't much left using `boost::thread_group`, so should just be able to replace it with the standard library. This also removes the last use of `boost::thread_interrupted`.

  After this change, last piece of Boost Thread we'd be using is `boost::shared_mutex`. See the commentary [here](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) as to why it may be non-trivial to swap that for `std::shared_mutex` in the near future.

  Closes #17307

ACKs for top commit:
  laanwj:
    Code review re-ACK dc8be12510
  MarcoFalke:
    review ACK dc8be12510 🔁
  jonatack:
    Non-expert code review ACK dc8be12510, also checked range-diff since last review and that local debug build is clean with gcc 10.2.1-6 on Debian

Tree-SHA512: 5510e2d760cce824234207dc86b1551ca8f21cbf3a2ce753c0254a0d03ffd83c94e449aec202fb7bd76e6fc64df783a6b70a736b0add9ece3734bb9c8ce8fc2f
2021-02-01 13:27:28 +01:00
fanquake
dc8be12510
refactor: remove boost::thread_group usage 2021-01-29 15:39:44 +08:00
Carl Dong
5f8cd7b3a5 validation: Remove global ::ActivateBestChain
Instead use CChainState::ActivateBestChain, which is what the global one
calls anyway.
2021-01-28 14:15:26 -05:00
Wladimir J. van der Laan
b386d37360
Merge #18710: Add local thread pool to CCheckQueue
bb6fcc75d1 refactor: Drop boost::thread stuff in CCheckQueue (Hennadii Stepanov)
6784ac471b bench: Use CCheckQueue local thread pool (Hennadii Stepanov)
dba30695fc test: Use CCheckQueue local thread pool (Hennadii Stepanov)
01511776ac Add local thread pool to CCheckQueue (Hennadii Stepanov)
0ef938685b refactor: Use member initializers in CCheckQueue (Hennadii Stepanov)

Pull request description:

  This PR:
  - gets rid of `boost::thread_group` in the `CCheckQueue` class
  - allows thread safety annotation usage in the `CCheckQueue` class
  - is alternative to #14464 (https://github.com/bitcoin/bitcoin/pull/18710#issuecomment-616618525, https://github.com/bitcoin/bitcoin/pull/18710#issuecomment-617291612)

  Also, with this PR (I hope) it could be easier to resurrect a bunch of brilliant ideas from #9938.

  Related: #17307

ACKs for top commit:
  laanwj:
    Code review ACK bb6fcc75d1
  LarryRuane:
    ACK bb6fcc75d1
  jonatack:
    Code review ACK bb6fcc75d1 and verified rebase to master builds cleanly with unit/functional tests green

Tree-SHA512: fddeb720d5a391b48bb4c6fa58ed34ccc3f57862fdb8e641745c021841c8340e35c5126338271446cbd98f40bd5484f27926aa6c3e76fa478ba1efafe72e73c1
2021-01-25 20:21:19 +01:00
practicalswift
b4511e2e2e log: Prefix log messages with function name if -logsourcelocations is set 2021-01-15 09:57:32 +00:00
Anthony Towns
a568b82feb net_processing: split PeerManager into interface and implementation classes 2021-01-09 23:27:45 +10:00
MarcoFalke
f13e03cda2
Merge #20584: Declare de facto const reference variables/member functions as const
31b136e580 Don't declare de facto const reference variables as non-const (practicalswift)
1c65c075ee Don't declare de facto const member functions as non-const (practicalswift)

Pull request description:

  _Meta: This is the second and final part of the `const` refactoring series (part one: #20581). **I promise: no more refactoring PRs from me in a while! :)** I'll now go back to focusing on fuzzing/hardening!_

  Changes in this PR:
  * Don't declare de facto const member functions as non-const
  * Don't declare de facto const reference variables as non-const

  Awards for finding candidates for the above changes go to:
  * `clang-tidy`'s [`readability-make-member-function-const`](https://clang.llvm.org/extra/clang-tidy/checks/readability-make-member-function-const.html)  check ([list of `clang-tidy` checks](https://clang.llvm.org/extra/clang-tidy/checks/list.html))
  * `cppcheck`'s `constVariable` check ([list of `cppcheck` checks](https://sourceforge.net/p/cppcheck/wiki/ListOfChecks/))

  See #18920 for instructions on how to analyse Bitcoin Core using Clang Static Analysis, `clang-tidy` and `cppcheck`.

ACKs for top commit:
  ajtowns:
    ACK 31b136e580
  jonatack:
    ACK 31b136e580
  theStack:
    ACK 31b136e580 ❄️

Tree-SHA512: f58f8f00744219426874379e9f3e9331132b9b48e954d24f3a85cbb858fdcc98009ed42ef7e7b4619ae8af9fc240a6d8bfc1c438db2e97b0ecd722a80dcfeffe
2021-01-07 09:05:09 +01:00
John Newbery
68334b3944 [net processing] Add m_ignores_incoming_txs to PeerManager and use internally 2020-12-09 18:13:37 +00:00
Carl Dong
81137c60fe test: Add new ChainTestingSetup and use it
Previously, the validation_chainstatemanager_tests test suite
instantiated its own duplicate ChainstateManager on which tests were
performed.

This wasn't a problem for the specific actions performed in
that suite. However, the existence of this duplicate ChainstateManager
and the fact that many of our validation static functions reach for
g_chainman, ::Chain(state|)Active means we may end up acting on two
different CChainStates should we write more extensive tests in the
future.

This change adds a new ChainTestingSetup which performs all
initialization previously done by TestingSetup except:

1. RPC command registration
2. ChainState initialization
3. Genesis Activation
4. {Ban,Conn,Peer}Man initialization

Means that we will no longer need to initialize a duplicate
ChainstateManger in order to test the initialization codepaths of
CChainState and ChainstateManager.

Lastly, this change has the additional benefit of allowing for
review-only assertions meant to show correctness to work in future work
de-globalizing g_chainman.

In the test chainstatemanager_rebalance_caches, an additional
LoadGenesisBlock call is added as MaybeReblanaceCaches eventually calls
FlushBlockFile, which tries to access vinfoBlockFile[nLastBlockFile],
which is out of bounds when LoadGenesisBlock hasn't been called yet.

-----

Note for the future:

The class con/destructor inheritance structure we have for these
TestingSetup classes is probably not the most suitable abstraction. In
particular, for both TestingSetup and ChainTestingSetup, we need to stop
the scheduler first before anything else. Otherwise classes depending on
the scheduler may be referenced by the scheduler after said classes are
freed. This means that there's no clear parallel between our teardown
code and C++'s destructuring order for class hierarchies.

Future work should strive to coalesce (as much as possible) test and
non-test init codepaths and perhaps structure it in a more fail-proof
way.
2020-12-08 15:00:25 -05:00
practicalswift
1c65c075ee Don't declare de facto const member functions as non-const 2020-12-06 18:44:25 +00:00
Antoine Poinsot
86ff2cf202
Remove the remaining fee estimation globals
This moves the CBlockPolicyEstimator to the NodeContext, which get rids
of two globals and allows us to conditionally create the
CBlockPolicyEstimator (and to remove a circular dep).

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-12-03 12:56:37 +01:00
Elle Mouton
e3310692d0 refactor: Make CTxMemPool::m_check_ratio a const and a constructor argument
Since m_check_ratio is only set once and since the CTxMemPool object is
no longer a global variable, m_check_ratio can be passed into the
constructor of CTxMemPool. Since it is only read from after
initialization, m_check_ratio can also be made a const and hence no
longer needs to be guarded by the cs mutex.
2020-10-23 14:41:30 +02:00
Hennadii Stepanov
01511776ac
Add local thread pool to CCheckQueue 2020-09-24 06:55:33 +03:00
Carl Dong
74f73c783d
validation: Pass in chainman to UnloadBlockIndex 2020-09-15 14:11:34 -04:00
John Newbery
58bd369b0d scripted-diff: [net processing] Rename PeerLogicValidation to PeerManager
-BEGIN VERIFY SCRIPT-
sed -i 's/PeerLogicValidation/PeerManager/g' $(git grep -l PeerLogicValidation ./src ./test)
sed -i 's/peer_logic/peerman/g' $(git grep -l peer_logic ./src ./test)
-END VERIFY SCRIPT-

PeerLogicValidation was originally net_processing's implementation to
the validation interface. It has since grown to contain much of
net_processing's logic. Therefore rename it to reflect its
responsibilities.

Suggested in
https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618.
2020-09-07 11:15:48 +01:00
John Newbery
2297b26b3c [net_processing] Pass chainparams to PeerLogicValidation constructor
Keep a references to chainparams, rather than calling the global
Params() function every time it's needed. This is fine, since
globalChainParams does not get updated once it's been set, and it's
available at the point of constructing the PeerLogicValidation object.
2020-09-07 11:13:58 +01:00
MarcoFalke
fafb381af8
Remove mempool global 2020-09-05 16:24:56 +02:00
MarcoFalke
fa0572d0f3
Pass mempool reference to chainstate constructor 2020-08-28 10:42:04 +02:00
fanquake
6a2ba62685
Merge #19779: Remove gArgs global from init
fa9d5902f7 scripted-diff: gArgs -> args (MarcoFalke)
fa33bc2dab init: Capture copy of blocknotify setting for BlockNotifyCallback (MarcoFalke)
fa40017706 init: Pass reference to ArgsManager around instead of relying on global (MarcoFalke)

Pull request description:

  The gArgs global has several issues:

  * gArgs is used by each process (bitcoind, bitcoin-qt, bitcoin-wallet, bitcoin-cli, bitcoin-tx, ...), but it is hard to determine which arguments are actually used by each process. For example arguments that have never been registered, but are still used, will always return the fallback value.
  * Tests may run several sub-tests, which need different settings. So globals will have to be overwritten, but that is fragile on its own: e.g. https://github.com/bitcoin/bitcoin/pull/19704#issuecomment-678259092 or #19511

  The goal is to remove gArgs, but as a first step in that direction this pull will change gArgs in init to use a passed-in reference instead.

ACKs for top commit:
  ryanofsky:
    Code review ACK fa9d5902f7. Looks good. Nice day to remove some globals, and add some lambdas 👍
  fanquake:
    ACK fa9d5902f7 - I'm not as familiar with the settings & argument handling code, but this make sense, and is a step in the right direction towards a reduction in the usage of globals. Not a huge fan of the clang-formatting in the scripted diff.
  jonasschnelli:
    Concept ACK fa9d5902f7

Tree-SHA512: ed00db5f826566c7e3b4d0b3d2ee0fc1a49a6e748e04e5c93bdd694ac7da5598749e73937047d5fce86150d764a067d2ca344ba4ae3eb2704cc5c4fa0d20940f
2020-08-26 15:18:38 +08:00
fanquake
92735e45ba
Merge #19775: test: Activate segwit in TestChain100Setup
fad84b7e14 test: Activate segwit in TestChain100Setup (MarcoFalke)
fa11ff2980 test: Pass empty tx pool to block assembler (MarcoFalke)
fa96574b0d test: Move doxygen comment to header (MarcoFalke)

Pull request description:

  This fixes not only a TODO in the code, but also prevents a never ending source of uninitialized reads. E.g.

  * #18376
  * https://github.com/bitcoin/bitcoin/pull/19704#issuecomment-678259092
  * ...

ACKs for top commit:
  jnewbery:
    utACK fad84b7e14

Tree-SHA512: 64cf16a59656d49e022b603f3b06441ceae35a33a4253b4382bc8a89a56e08ad5412c8fa734d0fc7b58586f40ea6d57b348a3b4838bc6890a41ae2ec3902e378
2020-08-26 13:17:35 +08:00
MarcoFalke
fa40017706
init: Pass reference to ArgsManager around instead of relying on global 2020-08-24 07:45:17 +02:00
MarcoFalke
fad84b7e14
test: Activate segwit in TestChain100Setup 2020-08-21 18:44:52 +02:00
MarcoFalke
fa11ff2980
test: Pass empty tx pool to block assembler 2020-08-21 18:44:50 +02:00
MarcoFalke
fa96574b0d
test: Move doxygen comment to header
Also, unrelated formatting fixups.

Can be reviewed with --word-diff-regex=.
2020-08-21 18:44:27 +02:00
John Newbery
c556770b5e [net_processing] Change PeerLogicValidation to hold a connman reference
Hold a reference to connman rather than a pointer because:

- PeerLogicValidation can't run without a connman
- The pointer never gets reseated

The alternative is to always assert that the pointer is non-null before
dereferencing.

Change the name from connman to m_connman at the same time to conform
with current style guidelines.
2020-08-12 14:25:28 +01:00
MarcoFalke
4b705b1c98
Merge #19098: test: Remove duplicate NodeContext hacks
edc316020e test: Remove duplicate NodeContext hacks (Russell Yanofsky)

Pull request description:

  Qt tests currently are currently using two NodeContext structs at the same time, one in interfaces::NodeImpl::m_context, and the other in BasicTestingSetup::m_node, and the tests have hacks transferring state between them.

  Fix this by getting rid of the NodeImpl::m_context struct and making it a pointer. This way a common BitcoinApplication object can be used for all qt tests, but they can still have their own testing setups.

  Non-test code is changing but non-test behavior is still the same as before.

  Motivation for this PR is to be able to remove the "std::move(test.m_node.connman)" and mempool hacks for swapping individual NodeContext members in Qt tests, because followup PR #19099 adds yet another member (wallet_client) that needs to be swapped. After this change, the whole NodeContext struct can be swapped instead of individual members, so the workarounds are less fragile and invasive.

ACKs for top commit:
  MarcoFalke:
    crACK edc316020e 🌮
  promag:
    ACK edc316020e.

Tree-SHA512: c1650e4127f43a4020304ca7c13b5d9122fb5723aacd8fa1cf855d03c6052fcfb7685810aa2a5ef708561015f0022fecaacbad479295104ca45d2c17579466a4
2020-08-07 08:07:37 +02:00
MarcoFalke
fac674db20
Pass mempool pointer to UnloadBlockIndex 2020-07-29 12:29:51 +02:00
MarcoFalke
2f71a1ea35
Merge #18637: coins: allow cache resize after init
f19fdd47a6 test: add test for CChainState::ResizeCoinsCaches() (James O'Beirne)
8ac3ef4699 add ChainstateManager::MaybeRebalanceCaches() (James O'Beirne)
f36aaa6392 Add CChainState::ResizeCoinsCaches (James O'Beirne)
b223111da2 txdb: add CCoinsViewDB::ChangeCacheSize (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: #15606
  Issue: #15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

  ---

  In the assumeutxo implementation draft (#15056), once a UTXO snapshot is loaded, a new chainstate object is created after initialization. This means that we have to reclaim some of the cache that we've allocated to the original chainstate (per `dbcache=`) to repurpose for the snapshot chainstate.

  Furthermore, it makes sense to have different cache allocations depending on which chainstate is more active. While the snapshot chainstate is working to get to the network tip (and the background validation chainstate is idle), it makes sense that the snapshot chainstate should have the majority of cache allocation. And contrariwise once the snapshot has reached network tip, most of the cache should be given to the background validation chainstate.

  This set of changes (detailed in the commit messages) allows us to dynamically resize the various coins caches. None of the functionality introduced here is used at the moment, but will be in the next AU PR (which introduces `ActivateSnapshot`).

  `ChainstateManager::MaybeRebalanceCaches()` defines the (somewhat normative) cache allocations between the snapshot and background validation chainstates. I'd be interested in feedback if anyone has thoughts on the proportions I've set there.

ACKs for top commit:
  ajtowns:
    weak utACK f19fdd47a6 -- didn't find any major problems, but not super confident that I didn't miss anything
  fjahr:
    Code review ACK f19fdd4
  ryanofsky:
    Code review ACK f19fdd47a6. Only change since last review is constructor cleanup (no change in behavior). I think the suggestions here from ajtowns and others are good, but shouldn't delay merging the PR (and hold up assumeutxo)

Tree-SHA512: fffb7847fb6993dd4a1a41cf11179b211b0b20b7eb5f7cf6266442136bfe9d43b830bbefcafd475bfd4af273f5573500594aa41fff03e0ed5c2a1e8562ff9269
2020-07-29 07:53:19 +02:00
Russell Yanofsky
edc316020e test: Remove duplicate NodeContext hacks
Qt tests currently are currently using two NodeContext structs at the same
time, one in interfaces::NodeImpl::m_context, and the other in
BasicTestingSetup::m_node, and the tests have hacks transferring state between
them.

Fix this by getting rid of the NodeImpl::m_context struct and making it a
pointer. This way a common BitcoinApplication object can be used for all qt
tests, but they can still have their own testing setups.

Non-test code is changing but non-test behavior is still the same as before.

Motivation for this PR is to be able to remove the
"std::move(test.m_node.connman)" and mempool hacks for swapping individual
NodeContext members in Qt tests, because followup PR #19099 adds yet another
member (wallet_client) that needs to be swapped. After this change, the whole
NodeContext struct can be swapped instead of individual members, so the
workarounds are less fragile and invasive.
2020-07-13 04:34:27 -04:00
MarcoFalke
5ec19df687
Merge #19277: util: Add Assert identity function
fab80fef61 refactor: Remove unused EnsureChainman (MarcoFalke)
fa34587f1c scripted-diff: Replace EnsureChainman with Assert in unit tests (MarcoFalke)
fa6ef701ad util: Add Assert identity function (MarcoFalke)
fa457fbd33 move-only: Move NDEBUG compile time check to util/check (MarcoFalke)

Pull request description:

  The utility function is primarily useful to dereference pointer types, which are known to be not null at that time.

  For example, the ArgsManager is known to exist when the wallets are started: https://github.com/bitcoin/bitcoin/pull/18923/files#diff-fdb2a1a1d8bc790fcddeb6cf5a42ac55R503 . Instead of silently relying on that assumption, `Assert` can be used to abort the program and avoid UB should the assumption ever be violated.

ACKs for top commit:
  promag:
    Tested ACK fab80fef61.
  ryanofsky:
    Code review ACK fab80fef61

Tree-SHA512: 830fba10152ba17d47c4dd42809c7e26f9fe6d38e17a2d5b3f054fd644a5c4c9841286ac421ec9bb28cea9f5faeb659740fcf00de6cc589d423fee7694c42d16
2020-07-04 08:44:45 -04:00
James O'Beirne
f36aaa6392 Add CChainState::ResizeCoinsCaches
Also adds CCoinsViewCache::ReallocateCache() to attempt to free
memory that the cacheCoins's allocator may be hanging onto when
downsizing the cache.

Adds `CChainState::m_coins{tip,db}_cache_size_bytes` data members
so that we can reference cache size on a per-chainstate basis for
flushing.
2020-07-01 14:44:28 -04:00
Wladimir J. van der Laan
e1b20e2285
Merge #19028: test: Set -logthreadnames in unit tests
99993489da test: Set -logthreadnames in unit tests (MarcoFalke)
fa4ea997b4 init: Setup scheduler in tests and init in exactly the same way (MarcoFalke)

Pull request description:

  Generally the unit tests are single threaded, with the exception of the script check threads, the schedule, and optionally indexer threads.

  Like the functional tests, the thread name can serve additional debug information, so set `-logthreadnames` in unit tests.

  Can be tested with

  ```
  ./src/test/test_bitcoin -l test_suite -t validation_tests/test_combiner_all -- DEBUG_LOG_OUT

ACKs for top commit:
  laanwj:
    ACK 99993489da

Tree-SHA512: 3bdbfc211da146da64b50b0826246aff5c611a84b69ab896a55b3c9d1adc92c5975da36ab92aee577df82e229c4326b477f4105bfdd1a5df4c9a0b018cf61602
2020-07-01 16:54:54 +02:00
MarcoFalke
fa34587f1c
scripted-diff: Replace EnsureChainman with Assert in unit tests
-BEGIN VERIFY SCRIPT-
sed --regexp-extended -i -e 's/EnsureChainman\((m?_?node)\)\./Assert(\1.chainman)->/g' $(git grep -l EnsureChainman)
-END VERIFY SCRIPT-
2020-06-15 07:39:26 -04:00
MarcoFalke
fa9819695a
test: Remove unused scheduler.h include from the common setup
The common setup is included in virtually all tests, so it should be
as slim as possible.
2020-05-28 09:00:56 -04:00
MarcoFalke
fa1d97b256
validation: Make ProcessNewBlock*() members of ChainstateManager 2020-05-21 09:56:16 -04:00
MarcoFalke
fa05fdf0f1
net: Pass chainman into PeerLogicValidation 2020-05-21 09:55:58 -04:00
MarcoFalke
fa7b626d7a
node: Add chainman alias for g_chainman 2020-05-21 09:55:51 -04:00
MarcoFalke
99993489da
test: Set -logthreadnames in unit tests 2020-05-21 08:13:23 -04:00
MarcoFalke
fa4ea997b4
init: Setup scheduler in tests and init in exactly the same way 2020-05-21 08:13:00 -04:00
Russell Yanofsky
b3f7f375ef refactor: Remove g_rpc_node global
This commit does not change behavior
2020-05-13 16:20:13 -04:00
Russell Yanofsky
6fca33b2ed refactor: Pass NodeContext to RPC and REST methods through util::Ref
This commit does not change behavior
2020-05-13 16:20:13 -04:00
MarcoFalke
fa488f131f
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-04-16 13:33:09 -04:00
MarcoFalke
fa0cbd48c4
test: Add optional extra_args to testing setup 2020-04-15 15:13:04 -04:00
MarcoFalke
fad4fa7e2f
node: Add args alias for gArgs global 2020-04-15 15:05:18 -04:00
MarcoFalke
10358a381a
Merge #17737: Add ChainstateManager, remove BlockManager global
c9017ce3bc protect g_chainman with cs_main (James O'Beirne)
2b081c4568 test: add basic tests for ChainstateManager (James O'Beirne)
4ae29f5f0c use ChainstateManager to initialize chainstate (James O'Beirne)
5b690f0aae refactor: move RewindBlockIndex to CChainState (James O'Beirne)
89cdf4d569 validation: introduce unused ChainstateManager (James O'Beirne)
8e2ecfe249 validation: add CChainState.m_from_snapshot_blockhash (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: #15606
  Issue: #15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

  ---

  This changeset introduces `ChainstateManager`, which is responsible for creating and managing access to multiple chainstates. Until we allow chainstate creation from UTXO snapshots (next assumeutxo PR?) it's basically unnecessary, but it is a prerequisite for background IBD support.

  Changes are also made to the initialization process to make use of `g_chainman` and thus clear the way for multiple chainstates being loaded on startup.

  One immediate benefit of this change is that we no longer have the `g_blockman` global, but instead have the ChainstateManager inject a reference of its shared BlockManager into any chainstate it creates.

  Another immediate benefit is that uses of `ChainActive()` and `ChainstateActive()` are now covered by lock annotations. Because use of `g_chainman` is annotated to require cs_main, these two functions subsequently follow.

  Because of whitespace changes, this diff looks bigger than it is. E.g., 4813167d98 is most easily reviewed with
  ```sh
  git show --color-moved=dimmed_zebra -w 4813167d98
  ```

ACKs for top commit:
  MarcoFalke:
    re-ACK c9017ce3bc 📙
  fjahr:
    Code Review Re-ACK c9017ce3bc
  ariard:
    Code Review ACK c9017ce
  ryanofsky:
    Code review ACK c9017ce3bc. No changes since last review other than a straight rebase

Tree-SHA512: 3f250d0dc95d4bfd70852ef1e39e081a4a9b71a4453f276e6d474c2ae06ad6ae6a32b4173084fe499e1e9af72dd9007f4a8a375c63ce9ac472ffeaada41ab508
2020-04-10 13:02:01 -04:00
MarcoFalke
4eb1eeb02c
Merge #18504: build: Drop bitcoin-tx and bitcoin-wallet dependencies on libevent
01a3392b1b Drop bitcoin-wallet dependency on libevent (Russell Yanofsky)
0660119ac3 Drop unintended bitcoin-tx dependency on libevent (Russell Yanofsky)

Pull request description:

  This fixes compile errors trying to build bitcoin-tx and bitcoin-wallet without libevent, which were reported by Luke Dashjr in https://github.com/bitcoin/bitcoin/issues/18465

  The fix avoiding `bitcoin-tx` dependency on libevent just adds a conditional build rule. This is implemented in the first commit (more details in commit description).

  The fix avoiding `bitcoin-wallet` dependency on libevent requires minor code changes, because `bitcoin-wallet` (unlike `bitcoin-tx`) links against code that calls `urlDecode` / `evhttp_uridecode`. This fix is implemented in the second commit (again details in the commit description).

ACKs for top commit:
  jonasschnelli:
    utACK 01a3392b1b.

Tree-SHA512: d2245e912ab494cccceeb427a1eca8e55b01a0006ff93eebcfb5461ae7cecd1083ac2de443d9db036b18bdc6f0fb615546caaa20c585046f66d234937f74870a
2020-04-10 12:52:37 -04:00
MarcoFalke
fa6a008434
fuzz: Add process_messages harness 2020-04-05 10:46:24 +08:00
Russell Yanofsky
01a3392b1b Drop bitcoin-wallet dependency on libevent
Don't require urlDecode function in wallet code since urlDecode implementation
currently uses libevent. Just call urlDecode indirectly though URL_DECODE
function pointer constant if available.

In bitcoind and bitcoin-qt, URL_DECODE is implemented and used to interpret RPC
wallet requests. In bitcoin-wallet, URL_DECODE is null to avoid depending on
libevent.
2020-04-02 08:35:10 -04:00
Wladimir J. van der Laan
2e97d80017
Merge #18134: Replace std::to_string with locale-independent alternative
d056df033a Replace std::to_string with locale-independent alternative (Ben Woosley)

Pull request description:

  Addresses #17866 following practicalswift's suggestion:
  https://github.com/bitcoin/bitcoin/issues/17866#issuecomment-584287299

  ~Used ::ToString to avoid aliasing issues. Left uses in QT and test.~

ACKs for top commit:
  practicalswift:
    ACK d056df033a
  laanwj:
    ACK d056df033a

Tree-SHA512: 9e6966a9cdd14f4a1a40d9f0fa7c402aed22b2f1ad8681708e22b050d51a91c5d62220a9ec4c425be2d57acf5c964fca87a5e981b5cbff048bc3b6720dae92b7
2020-03-25 20:11:47 +01:00
James O'Beirne
2b081c4568 test: add basic tests for ChainstateManager
Feedback incorporated from Russell Yanofsky.

Co-authored-by: MarcoFalke <falke.marco@gmail.com>
2020-03-17 14:03:40 -04:00
James O'Beirne
4ae29f5f0c use ChainstateManager to initialize chainstate
This allows us to easily initialize multiple chainstates on startup in future
commits. It retires the g_chainstate global in lieu of g_chainman.
2020-03-17 14:03:40 -04:00
Ben Woosley
d056df033a
Replace std::to_string with locale-independent alternative 2020-03-14 12:23:01 -07:00
MarcoFalke
fa7fea3654
refactor: Remove mempool global from net
This refactor does two things:
* Pass mempool in to PeerLogicValidation
* Pass m_mempool around where needed
2020-03-12 09:23:56 -04:00
practicalswift
fd1dae10b4 tests: Add fuzzing harness for ProcessMessage(...) 2020-03-11 06:57:55 +00:00
Anthony Towns
306f71b4eb scheduler: don't rely on boost interrupt on shutdown
Calling interrupt_all() will immediately stop the scheduler, so it's
safe to invoke stop() beforehand, and this removes the reliance on boost
to interrupt serviceQueue().
2020-03-06 23:13:31 +10:00
Jeffrey Czyz
0aed17ef28 Refactor FormatStateMessage into ValidationState 2020-02-27 17:59:07 -08:00
Amiti Uttarwar
7c8b6e5b52 [lib] add scheduler to node context
- also update test setup & access point in denial of service test
2020-02-17 14:49:34 -08:00
MarcoFalke
fa37e0a68b
test: Show debug log on unit test failure 2020-01-02 18:00:05 -05:00
MarcoFalke
faa92a2297
rpc: Remove mempool global from miner 2019-12-23 06:12:10 +07:00
MarcoFalke
fac07f2038
node: Add reference to mempool in NodeContext
Currently it is an alias to the global ::mempool and should be used as
follows.

* Node code (validation and transaction relay) can use either ::mempool
  or node.mempool, whichever seems a better fit.
* RPC code should use the added convenience getter EnsureMempool, which
  makes sure the mempool exists before use. This prepares the RPC code
  to a future where the mempool might be disabled at runtime or compile
  time.
* Test code should use m_node.mempool directly, as the mempool is always
  initialized for tests.
2019-11-15 13:40:00 -05:00
MarcoFalke
772673dfbe
Merge #16978: test: Seed test RNG context for each test case, print seed
fae43a97ca test: Seed test RNG context for each test case, print seed (MarcoFalke)

Pull request description:

  Debugging failing unit tests is hard if the failure is non-deterministic and the seed is not known.

  Fix that by printing the seed and making it possible to set the seed from outside.

ACKs for top commit:
  davereikher:
    Tested ACK fae43a97ca

Tree-SHA512: 33d848dd1f4180d3664ecf60e9810c2a93590c05276b2c46b1e4fe6e376b45916a46b90c803bb602750ab666da3a05ce499e550024685a90b8cc38fab6667cb8
2019-11-07 10:18:40 -05:00
MarcoFalke
7d14e35f3f
Merge #17342: refactor: Clean up nScriptCheckThreads
5506ecfe7a [refactor] Replace global int nScriptCheckThreads with bool (John Newbery)
d9957623b4 [tests] Don't use TestingSetup in the checkqueue_tests (John Newbery)

Pull request description:

  The meaning of this value is confusing. Refactor it and add comments.

ACKs for top commit:
  sipa:
    ACK 5506ecfe7a
  promag:
    ACK 5506ecfe7a, only change was addressing my nits.
  laanwj:
    Code review ACK 5506ecfe7a
  MarcoFalke:
    ACK 5506ecfe7a 🥐

Tree-SHA512: 78536727c98d2c23f3c0f3f169131474fef9a4486ae65029011caf06eab30f6f70ff73a65b2fb04a5d969fc1150858d1c6ea4767f04d48c1eea6b829316d0e63
2019-11-07 10:07:11 -05:00
MarcoFalke
faec28252c
scripted-diff: test: Move setup_common to test library
-BEGIN VERIFY SCRIPT-
 # Move files
 for f in $(git ls-files src/test/lib/); do git mv $f src/test/util/; done
 git mv src/test/setup_common.cpp                     src/test/util/
 git mv src/test/setup_common.h                       src/test/util/
 # Replace Windows paths
 sed -i -e 's|\\setup_common|\\util\\setup_common|g' $(git grep -l '\\setup_common')
 sed -i -e 's|src\\test\\lib\\|src\\test\\util\\|g'  build_msvc/test_bitcoin/test_bitcoin.vcxproj
 # Everything else
 sed -i -e 's|/setup_common|/util/setup_common|g'    $(git grep -l 'setup_common')
 sed -i -e 's|test/lib/|test/util/|g'                $(git grep -l 'test/lib/')
 # Fix include guard
 sed -i -e 's|BITCOIN_TEST_SETUP_COMMON_H|BITCOIN_TEST_UTIL_SETUP_COMMON_H|g' ./src/test/util/setup_common.h
 sed -i -e 's|BITCOIN_TEST_LIB_|BITCOIN_TEST_UTIL_|g'                     $(git grep -l 'BITCOIN_TEST_LIB_')
-END VERIFY SCRIPT-
2019-11-06 11:56:41 -05:00