Commit Graph

4869 Commits

Author SHA1 Message Date
MarcoFalke
fa776e61cd
Add importmempool RPC
test_importmempool_union contributed by glozow

Co-authored-by: glozow <gloriajzhao@gmail.com>
2023-08-07 11:33:34 +02:00
fanquake
d096743150
Merge bitcoin/bitcoin#28213: scripted-diff: Specify Python major version explicitly on Windows
6a7686b446 scripted-diff: Specify Python major version explicitly on Windows (Hennadii Stepanov)

Pull request description:

  On Windows, it is the accepted practice to use `py.exe` launcher:
  - https://learn.microsoft.com/en-us/windows/python/faqs#what-is-py-exe-
  - https://docs.python.org/3/using/windows.html#python-launcher-for-windows

  One of its features is the correct handling of shebang lines like the one we use: `#!/usr/bin/env python3`.

  However, Windows OS app execution aliases might [interfere](https://learn.microsoft.com/en-us/windows/python/faqs#why-does-running-python-exe-open-the-microsoft-store-) with the launcher's behaviour. Such aliases are enabled on Windows 11 by default:

  ![image](https://github.com/bitcoin/bitcoin/assets/32963518/407837ec-e89a-4bc1-98b1-db983002065a)

  For example, on a fresh Windows 11 Pro installation with the Python installed from the [Chocolatey](https://community.chocolatey.org/packages/python/3.11.4) package manager, one will get the following error:
  ```
  >py -3 test\functional\rpc_signer.py
  2023-08-03T19:41:13.353000Z TestFramework (INFO): PRNG seed is: 2694758731106548661
  2023-08-03T19:41:13.353000Z TestFramework (INFO): Initializing test directory C:\Users\hebasto\AppData\Local\Temp\bitcoin_func_test_mldbzzw3
  2023-08-03T19:41:14.538000Z TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
    File "C:\Users\hebasto\bitcoin\test\functional\test_framework\util.py", line 140, in try_rpc
      fun(*args, **kwds)
    File "C:\Users\hebasto\bitcoin\test\functional\test_framework\coverage.py", line 50, in __call__
      return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\hebasto\bitcoin\test\functional\test_framework\authproxy.py", line 129, in __call__
      raise JSONRPCException(response['error'], status)
  test_framework.authproxy.JSONRPCException: RunCommandParseJSON error: process(py C:\Users\hebasto\bitcoin\test\functional\mocks\signer.py enumerate) returned 9009: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
   (-1)

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "C:\Users\hebasto\bitcoin\test\functional\test_framework\test_framework.py", line 131, in main
      self.run_test()
    File "C:\Users\hebasto\bitcoin\test\functional\rpc_signer.py", line 72, in run_test
      assert_raises_rpc_error(-1, 'fingerprint not found',
    File "C:\Users\hebasto\bitcoin\test\functional\test_framework\util.py", line 131, in assert_raises_rpc_error
      assert try_rpc(code, message, fun, *args, **kwds), "No exception raised"
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\hebasto\bitcoin\test\functional\test_framework\util.py", line 146, in try_rpc
      raise AssertionError(
  AssertionError: Expected substring not found in error message:
  substring: 'fingerprint not found'
  error message: 'RunCommandParseJSON error: process(py C:\Users\hebasto\bitcoin\test\functional\mocks\signer.py enumerate) returned 9009: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
  '.
  2023-08-03T19:41:14.592000Z TestFramework (INFO): Stopping nodes
  2023-08-03T19:41:14.799000Z TestFramework (WARNING): Not cleaning up dir C:\Users\hebasto\AppData\Local\Temp\bitcoin_func_test_mldbzzw3
  2023-08-03T19:41:14.799000Z TestFramework (ERROR): Test failed. Test logging available at C:\Users\hebasto\AppData\Local\Temp\bitcoin_func_test_mldbzzw3/test_framework.log
  2023-08-03T19:41:14.799000Z TestFramework (ERROR):
  2023-08-03T19:41:14.799000Z TestFramework (ERROR): Hint: Call C:\Users\hebasto\bitcoin\test\functional\combine_logs.py 'C:\Users\hebasto\AppData\Local\Temp\bitcoin_func_test_mldbzzw3' to consolidate all logs
  2023-08-03T19:41:14.799000Z TestFramework (ERROR):
  2023-08-03T19:41:14.799000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
  2023-08-03T19:41:14.799000Z TestFramework (ERROR): https://github.com/bitcoin/bitcoin/issues
  2023-08-03T19:41:14.799000Z TestFramework (ERROR):

  ```

  This PR resolves this issue by explicitly specifying the Python major version and makes testing of self-compiled binaries more straightforward.

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 6a7686b446
  stickies-v:
    utACK 6a7686b446

Tree-SHA512: 5681141e222bc833c6250cb79fe3a1c8e02255eb2c86010bc0f8239afcdfed784ed7788c8579209d931bd357f58d5655cf33ffeb2f46b1879f37cdc30e7a7c91
2023-08-04 15:09:34 +02:00
fanquake
da3816e4e8
Merge bitcoin/bitcoin#27832: doc: Clarify -datacarriersize, add -datacarriersize=2 tests
faafc35a77 doc: Clarify that -datacarriersize applies to the full raw scriptPubKey, not the data push (MarcoFalke)
55550e7fe7 test: Add -datacarriersize=2 tests (MarcoFalke)

Pull request description:

  Clarify with a test that `-datacarriersize` applies to the serialized size of the scriptPubKey, not the size of the pushed data. So for example,

  * `-datacarriersize=2` will reject a `raw(6a01aa)`, even though only one byte is pushed
  * `-datacarriersize=0` (or `-datacarrier=0`) will reject a `raw(6a)`, even though no byte is pushed
  * `-datacarriersize=0` (or `-datacarrier=0`) will reject a `raw(6a00)`, even though zero bytes are pushed

ACKs for top commit:
  ajtowns:
    ACK faafc35a77
  instagibbs:
    ACK faafc35a77

Tree-SHA512: f01ace02798f596ac2a02461e9f2a6ef91b3b37c976ea0b3bc860e2d3efb0ace0fd8b779dd18249cee7f84ebbe5fd21d8506afd3a15edadc00b843ff3b4aacc7
2023-08-03 17:46:43 +01:00
Hennadii Stepanov
6a7686b446
scripted-diff: Specify Python major version explicitly on Windows
Using `py.exe` launcher might by fragile depending on how Python was
installed. Specifying the Python version explicitly fixes test errors
like this:
```
RunCommandParseJSON error: process(py C:\Users\hebasto\bitcoin\test\functional\mocks\signer.py enumerate) returned 9009: Python was not found...
```

-BEGIN VERIFY SCRIPT-
sed -i 's|"py "|"py -3 "|g' $(git grep -l '"py "' -- test/functional)
-END VERIFY SCRIPT-
2023-08-03 14:57:45 +01:00
glozow
6fa49937e4 test: Check tx from disconnected block is immediately requestable
Check that peers can immediately request txs from blocks that have been
reorged out and are now in our mempool.
2023-08-03 21:33:22 +10:00
Hennadii Stepanov
703b758e18
qa: Close SQLite connection properly
Connection object used as context manager only commits or rollbacks
transactions, so the connection object should be closed manually.

Fixes the following error on Windows:
```
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: ...
```
2023-08-02 19:29:01 +01:00
fanquake
2fa60f0b68
Merge bitcoin/bitcoin#27452: test: cover addrv2 anchors by adding TorV3 to CAddress in messages.py
ba8ab4fc54 test: cover addrv2 support in anchors.dat with a TorV3 address (Matthew Zipkin)
b4bee4bbf4 test: add keep_alive option to socks5 proxy in test_framework (Matthew Zipkin)
5aaf988ccc test: cover TorV3 address in p2p_addrv2_relay (Matthew Zipkin)
80f64a3d40 test: add support for all networks in CAddress in messages.py (brunoerg)

Pull request description:

  Closes https://github.com/bitcoin/bitcoin/issues/27140

  Adds test coverage for https://github.com/bitcoin/bitcoin/pull/20516 to ensure that https://github.com/bitcoin/bitcoin/issues/20511 is completed and may be closed.

  This PR adds a test case to `feature_anchors.py` where an onion v3 address is set as a blocks-only relay peer and then shutdown, ensuring that the address is saved to anchors.dat in addrv2 format. We then ensure that bitcoin attempts to reconnect to that anchor address on restart.

  To compute the addrv2 serialization of the onion v3 address, I added logic to `CAddress` in `messages.py`. This new logic is covered by extending `p2p_addrv2_relay.py` to include an onion v3 address. Future work will be adding coverage for ipv6, torv2 and cjdns in these modules and also `feature_proxy.py`

  Also includes de/serialization unit test for `CAddress` in test framework.

ACKs for top commit:
  jonatack:
    ACK ba8ab4fc54
  brunoerg:
    crACK ba8ab4fc54
  willcl-ark:
    ACK ba8ab4fc54

Tree-SHA512: 7220e30d7cb975903d9ac575a7215a08e8f784c24c5741561affcbde12fb92cbf8704cb42e66494b788ba6ed4bb255fb0cc327e4f2190fae50c0ed9f336c0ff0
2023-08-02 12:57:30 +01:00
fanquake
2dea6c5ca0
Merge bitcoin/bitcoin#27572: test: dedup file hashing using sha256sum_file helper
2c0c6f4477 test: dedup file hashing using `sha256sum_file` helper (Sebastian Falbesoner)

Pull request description:

  Rather than doing the open/read/hash-steps manually in the affected functional tests, we can just use the `sha256sum_file` helper from the utils module instead.

  Note that for the tool_wallet.py test, the used hash is changed from sha1 to sha256, but as the only purpose is to detect file content changes, this doesn't matter. Also, the optimization using `memoryview` is overkill here, as the opened file has only a size of 24KiB and determining the hash via the helper doesn't take longer than a few hundred micro-seconds on my machine.

ACKs for top commit:
  kristapsk:
    ACK 2c0c6f4477

Tree-SHA512: 64fe21650b56a50e9f1a95f6ef27d88d8bfbb621e5be456f327ef8dbb5596b529d03976c200f3fd68da48cc427de9f257b403f3228e38cf1df918006674fac68
2023-08-02 11:53:35 +01:00
fanquake
1b5cbf71df
Merge bitcoin/bitcoin#28144: test: fix intermittent failure in p2p_getaddr_caching.py
8a20f765cc test: drop duplicate getaddrs from p2p_getaddr_caching (Martin Zumsande)
feb0096139 test: fix intermittent failure in p2p_getaddr_caching (Martin Zumsande)

Pull request description:

  Fixes #28133

  In the consistency check, it's not enough to check that our address/port is unique, only the combination of source and target must be unique. Otherwise, the OS may reuse ports for connections to different `-addrbind`, which was happening in the failed runs.

  While at it, the second commit cleans up duplicate `getaddr` messages in `p2p_getaddr_caching.py` that do nothing but generate `Ignoring repeated "getaddr"` log messages (and cleans up some whitespace the python linter complains about).

ACKs for top commit:
  vasild:
    ACK 8a20f765cc

Tree-SHA512: eabe4727d7887f729074076f6333a918bba8cb34b8e3baaa83f167b441b0daa24f7c4824abcf03a9538a2ef14b2d826ff19aeffcb93a6c20735253a9678aac9c
2023-08-01 16:59:37 +01:00
fanquake
eb95368e0c
Merge bitcoin/bitcoin#28166: test, rpc: invalid sighashtype coverage
90c8f79e94 test: remove redundant test values (Jon Atack)
c3f203387d test: use common assert_signing_completed_successfully helper (Jon Atack)
647d95aae9 test: add coverage for passing an invalid sighashtype (Jon Atack)

Pull request description:

  Add test coverage for passing an invalid sighashtype to RPCs signrawtransactionwithwallet, signrawtransactionwithkey, walletprocesspsbt, and descriptorprocesspsbt.

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 90c8f79e94 🎥
  brunoerg:
    light crACK 90c8f79e94

Tree-SHA512: 3861658487edd0d9a377390acf3d43f45c3dd9e324894f0fdb8f5312b618301a55479b1f70c61daee0b20785e768ffde6fa5abe6af190b73c0d0e017f3976704
2023-08-01 16:56:15 +01:00
MarcoFalke
fa8685597e
index: Drop legacy -txindex check 2023-08-01 15:27:23 +02:00
MarcoFalke
fa69148a0a
scripted-diff: Use blocks_path where possible
-BEGIN VERIFY SCRIPT-
  sed -i 's|].chain_path, .blocks.|].blocks_path|g' $(git grep -l chain_path)
-END VERIFY SCRIPT-
2023-08-01 15:26:55 +02:00
fanquake
e5a9f2fb62
Merge bitcoin/bitcoin#28194: test: python E721 and flake8 updates
bee2d57a65 script: update flake8 to 6.1.0 (Jon Atack)
38c3fd846b test: python E721 updates (Jon Atack)

Pull request description:

  Update our functional tests per [E721](https://www.flake8rules.com/rules/E721.html) enforced by [flake8 6.1.0](https://flake8.pycqa.org/en/latest/release-notes/6.1.0.html), and update our CI lint task to use that release.  This makes the following linter output on current master with flake8 6.1.0 green.

  ```
  $ ./test/lint/lint-python.py ; ./test/lint/lint-spelling.py
  test/functional/p2p_invalid_locator.py:35:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
  test/functional/test_framework/siphash.py:34:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
  test/functional/test_framework/siphash.py:64:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
  src/test/fuzz/descriptor_parse.cpp:88: occurences ==> occurrences
  ^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in test/lint/spelling.ignore-words.txt
  ```

ACKs for top commit:
  MarcoFalke:
    lgtm ACK bee2d57a65

Tree-SHA512: f3788a543ca98e44eeeba1d06c32f1b11eec95d4aef068aa1b6b5c401261adfa3fb6c6d6c769f3fe6839d78e74a310d5c926867e7c367d6513a53d580fd376f3
2023-08-01 09:42:07 +01:00
fanquake
8535802f1d
Merge bitcoin/bitcoin#28070: test: Drop 22.x node from TxindexCompatibilityTest
fafe43cb6c scripted-diff: Use blocks_path where possible (MarcoFalke)
fa060c15fb test: Add blocks_path property to TestNode (MarcoFalke)
faba4fc325 test: Drop 22.x node from TxindexCompatibilityTest (MarcoFalke)
fa7f65b0f8 test: Use clean chain in MempoolCompatibilityTest (MarcoFalke)

Pull request description:

  The node in this test was never really needed, because the compatibility tests shouldn't be used to test previous releases. (The test suite of the previous release itself should be used for that). So remove it.

  Also, other test changes. (See individual commits)

ACKs for top commit:
  theStack:
    Code-review ACK fafe43cb6c

Tree-SHA512: 289f54695bf5310663ab38ebf1aa457f53d0aafae56e6657be0e75bf96b303165bad417dc7eaf4c40f0639aa92ce139e5bacb318a2eabab1f8e23a811cabe0cc
2023-08-01 09:38:49 +01:00
Jon Atack
38c3fd846b test: python E721 updates 2023-07-31 12:13:46 -06:00
Hennadii Stepanov
ab498d913c
qa, doc: Fix comment
This change is a follow-up for:
- https://github.com/bitcoin/bitcoin/pull/9956
- https://github.com/bitcoin/bitcoin/pull/10096
2023-07-30 11:10:59 +01:00
Jon Atack
90c8f79e94 test: remove redundant test values
as they are parsed identically.

See AmountFromValue() / ParseFixedPoint() / UniValue#getValStr()
2023-07-27 06:54:32 -06:00
Jon Atack
c3f203387d test: use common assert_signing_completed_successfully helper 2023-07-27 06:37:55 -06:00
Jon Atack
647d95aae9 test: add coverage for passing an invalid sighashtype
in RPCs descriptorprocesspbst, walletprocesspbst, signrawtransactionwithkey,
and signrawtransactionwithwallet.
2023-07-27 06:37:54 -06:00
brunoerg
108c6255bc test: remove unused totalOut code
In `wallet_fundrawtransaction`, `totalOut` is used in
some functions to check if the change is correct. In
other ones, it has been created but never used.
2023-07-26 16:05:37 -03:00
brunoerg
0fc3deee9a test: remove unecessary decoderawtransaction calls
In `wallet_fundrawtransaction`, there are some unecessary
variables/calls to `decoderawtransaction`. They have not
been used.
2023-07-26 15:57:43 -03:00
fanquake
f033a981ed
Merge bitcoin/bitcoin#28139: test: create wallet specific for test_locked_wallet case
c648bdbda2 test: create wallet specific for test_locked_wallet case (furszy)

Pull request description:

  Coming from https://github.com/bitcoin/bitcoin/pull/28089#discussion_r1265478128.

  Several test cases are relying on the node1 default wallet, which thanks to 'test_locked_wallet' is encrypted.
  And can be only accessed within a specific timeframe (100ms), a duration internally set by the same test.

  This situation introduces a potential race condition, where other tests must complete their operations within
  the specified 100ms window to pass (otherwise the wallet gets re-locked and they fail).

  This can be seen running the test in valgrind (https://github.com/bitcoin/bitcoin/pull/28089), where other test cases fail due the wallet re-locking
  itself after the 100ms.

ACKs for top commit:
  MarcoFalke:
    lgtm ACK c648bdbda2
  ishaanam:
    utACK c648bdbda2

Tree-SHA512: 01cde5a4a0cb3405adb9ea3c1f73841f3fa237d1162268ed06f0d49ca38541006b423a029e0b5e5955e1aa7e018c4600d894e555a68cf17ff60a4b8be58f4aa9
2023-07-26 10:19:56 +01:00
fanquake
8fba5dfc10
Merge bitcoin/bitcoin#27529: test: fix feature_addrman.py on big-endian systems
53c990ad34 test: fix `feature_addrman.py` on big-endian systems (Sebastian Falbesoner)

Pull request description:

  The test `feature_addrman.py` currently serializes the addrdb without specifying endianness for `int`s, so the machine's native byte order is used (see https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment) and the generated `peers.dat` would be invalid on big-endian systems (our internal (de)serializers always use little-endian, see `ser_{read,write}data32`). Fix this by explicitly specifying little-endian serialization via the `<` character in `struct.pack(...)`.

  This is not detected by CI as we unfortunately don't run functional tests on big-endian systems there (I think we definitely should!).

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 53c990ad34   🔚

Tree-SHA512: 513af6f1f785a713e7a8ef3a57fcd3fe2520a7d537f63a9c8e1f4bdea4c2f605fd4c35001623d6b13458883dbc256f24943684ab8f224055c22bf8d8eeee5fe2
2023-07-26 09:42:20 +01:00
fanquake
54fe963a53
Merge bitcoin/bitcoin#28035: test: Ignore UTF-8 errors in assert_debug_log
fa3d72960b lint: Ignore check_fileopens failure on **kwargs (MarcoFalke)
fa6bb85cd2 test: Ignore UTF-8 errors in assert_debug_log (MarcoFalke)
fa63326fbc test: Fix debug_log_size helper (MarcoFalke)

Pull request description:

  Fix two bugs, see commit messages.

ACKs for top commit:
  theStack:
    utACK fa3d72960b

Tree-SHA512: 4a29bdf954bf62bb7676c2a41b03ad017bc86d535b2bd912c96bd41d1621beb06d840b53c211480ad51974e8b293bbae620060d2528d269159f32c0b44e47712
2023-07-26 09:35:51 +01:00
Martin Zumsande
8a20f765cc test: drop duplicate getaddrs from p2p_getaddr_caching
python p2p instances will automatically send a getaddr msg after
connecting, the explicit message was a duplicate that was being ignored.
2023-07-25 10:34:29 -04:00
Martin Zumsande
feb0096139 test: fix intermittent failure in p2p_getaddr_caching
Only the combined addr:port of source and destination
must be unique. If the destination is different, the same addr:port
for the source may be used by the OS.
2023-07-25 10:34:29 -04:00
MarcoFalke
fafe43cb6c
scripted-diff: Use blocks_path where possible
-BEGIN VERIFY SCRIPT-
  sed -i 's|].chain_path / "blocks"|].blocks_path|g' $(git grep -l chain_path)
-END VERIFY SCRIPT-
2023-07-25 15:51:36 +02:00
MarcoFalke
fa060c15fb
test: Add blocks_path property to TestNode 2023-07-25 15:51:22 +02:00
MarcoFalke
faba4fc325
test: Drop 22.x node from TxindexCompatibilityTest
* The node was only used to migrate the legacy txindex. But now that it
  is known to be working and that 22.x is EOL, it can be dropped.
* Also, fix a typo to properly check the txindex of node [1], not [2].
2023-07-25 15:51:19 +02:00
MarcoFalke
fa7f65b0f8
test: Use clean chain in MempoolCompatibilityTest
The test creates enough blocks itself, so there is no need to have more.

Also, remove os import.
2023-07-25 15:24:04 +02:00
furszy
c648bdbda2
test: create wallet specific for test_locked_wallet case
Other tests are also relying on the node1 default wallet,
which thanks to 'test_locked_wallet' is encrypted.
And can only be accessed within a specific timeframe (100ms)
set internally by the same test.

This make other tests susceptible to races. They can only
perform their operations successfully within the specified
time.

This can be seen running the test in valgrind, where other
test cases fail due the wallet re-locking itself after the
100ms.
2023-07-24 11:03:05 -03:00
Luke Dashjr
5e3e83b005 RPC/Mining: Document template_request better for getblocktemplate 2023-07-22 01:29:11 +00:00
Andrew Chow
b3022af0e2
Merge bitcoin/bitcoin#28108: test: fix intermittent failure in wallet_resendwallettransactions.py
e667bd68a1 test: fix intermittent failure in wallet_resendwallettransactions.py (Martin Zumsande)

Pull request description:

  Fixes #28094

  The test bumps the mocktime for ~2 weeks and then triggers eviction from the mempool. But this bump will also cause a new resubmit, and if the timing is such that this resubmit happens right after the eviction and before the check that the tx was evicted, the test can fail as in #28094:

  ```
  node0 2023-07-17T21:31:23.809483Z (mocktime: 2023-08-02T09:46:27Z) [httpworker.1] [validation.cpp:267] [LimitMempoolSize] [mempool] Expired 2 transactions from the memory pool
  node0 2023-07-17T21:31:23.810079Z (mocktime: 2023-08-02T09:46:27Z) [scheduler] [wallet/wallet.h:895] [WalletLogPrintf] [default wallet] ResubmitWalletTransactions: resubmit 2 unconfirmed transactions
  node0 2023-07-17T21:31:23.810474Z (mocktime: 2023-08-02T09:46:27Z) [httpworker.2] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getmempoolentry user=__cookie__
  2023-07-17T21:31:23.811000Z TestFramework (ERROR): Assertion failed (...) AssertionError: No exception raised
  ```
  Fix this by flushing out the current resubmit call before triggering mempool eviction.

ACKs for top commit:
  MarcoFalke:
    Nice. lgtm ACK e667bd68a1
  achow101:
    ACK e667bd68a1
  jonatack:
    Light "this looks like the other tests in this file" ACK e667bd68a1

Tree-SHA512: 027c2177ecd8bea80ec388ec2564f8fcbc717efd2722304b16fc0e9fa7ad216af61977c4e360b8135de68586cf13b0aa729ffa4fa27bad655092c3a55f73933c
2023-07-20 11:39:24 -04:00
Andrew Chow
7edce77ff3
Merge bitcoin/bitcoin#28067: descriptors: do not return top-level only funcs as sub descriptors
dd9633b516 test: wallet, add coverage for watch-only raw sh script migration (furszy)
cc781a2180 descriptor: InferScript, do not return top-level only func as sub descriptor (furszy)
286e0c7d5e wallet: loading, log descriptor parsing error details (furszy)

Pull request description:

  Linked to #28057.

  Currently, the `InferScript` function returns an invalid descriptor when it tries to infer a p2sh-p2pkh script whose pubkey is not known by the wallet.

  This behavior occurs because the inference process bypasses the `pkh` subscript when the pubkey is not contained by the wallet (no pubkey provider), interpreting it as a `sh(addr(ADDR))` descriptor. Then, the failure arises because the `addr()` function is restricted to being used only at the top level.

  For reviewers, would recommend to start by examining the functional test to understand the context and the circumstances on which this can result in a fatal error (e.g. during the migration process).

ACKs for top commit:
  achow101:
    ACK dd9633b516
  darosior:
    utACK dd9633b516

Tree-SHA512: 61e763206c604c372019d2c36e31684f3dddf81f8b154eb9aba5cd66d8d61bda457ed4e591613eb6ce6c76cf7c3f11764abc6cd727a7c2b6414f1065783be032
2023-07-20 11:16:45 -04:00
fanquake
79954903b2
Merge bitcoin/bitcoin#27620: test: miner: add coverage for -blockmintxfee setting
bbbb89d238 test: miner: add coverage for `-blockmintxfee` setting (Sebastian Falbesoner)

Pull request description:

  This PR adds missing test coverage for the `-blockmintxfee` option, which can be used by miners to specify the lowest fee-rate for transactions to be included in blocks. The setting was introduced in PR #9380 (commit daec955fd6), with the rationale to decouple different minimum fees from `-minrelaytxfee`. According to the PR description it _"should be set by miners to reflect their marginal cost of transmitting extra bytes."_.

  On each iteration, the test creates and submits two txs using MiniWallet: one with the the minimum fee-rate as specified for `-blockmintxfee` and a second one with a fee-rate a little below that (-0.01 sats/vbyte). Then it checks that  only the first one is picked for the block template and accordingly also only exists in the block that is mined after. This is repeatedly done for a fixed (but obviously somewhat arbitrary) list of different `-blockmintxfee` settings on a single node, including the default and zero (i.e. no minimum fee a.k.a. "include even zero-fee txs") settings.

ACKs for top commit:
  ryanofsky:
    Code review ACK bbbb89d238, nice test
  brunoerg:
    reACK bbbb89d238
  glozow:
    ACK bbbb89d238, sorry for the late re-review!

Tree-SHA512: 7b72612971e6a1667b4b3913ec27109953fd17a1020a4bde6941a93666b2e10a23fb6fe7df23471a5671ffe31e42cd992d2efb8b31903915b3dfc1d6478df757
2023-07-20 15:33:54 +01:00
furszy
dd9633b516
test: wallet, add coverage for watch-only raw sh script migration 2023-07-20 11:04:52 -03:00
furszy
cc781a2180
descriptor: InferScript, do not return top-level only func as sub descriptor
e.g. sh(addr(ADDR)) or sh(raw(HEX)) are invalid descriptors.

Making sh and wsh top level functions to return addr/raw descriptors when
the subscript inference fails.
2023-07-20 11:04:52 -03:00
fanquake
04afe55e29
Merge bitcoin/bitcoin#26467: bumpfee: Allow the user to choose which output is change
e8c31f135c tests: Test for bumping single output transaction (Andrew Chow)
4f4d4407e3 test: Test bumpfee reduce_output (Andrew Chow)
7d83502d3d bumpfee: Allow original change position to be specified (Andrew Chow)

Pull request description:

  When bumping the transaction fee, we will try to find the change output of the transaction in order to have an output whose value we can modify so that we can meet the target feerate. However we do not always find the change output which can cause us to unnecessarily add an additional output to the transaction. We can avoid this issue by outsourcing the determination of change to the user if they so desire.

  This PR adds a `orig_change_pos` option to bumpfee which the user can use to specify the index of the change output.

  Fixes #11233
  Fixes #20795

ACKs for top commit:
  ismaelsadeeq:
    re ACK e8c31f135c
  pinheadmz:
    re-ACK e8c31f135c
  furszy:
    Code review ACK e8c31f13

Tree-SHA512: 3a230655934af17f7c1a5953fafb5ef0d687c21355cf284d5e98fece411f589cd69ea505f06d6bdcf82836b08d268c366ad2dd30ae3d71541c9cdf94d1f698ee
2023-07-20 09:55:04 +01:00
MarcoFalke
fa6bb85cd2
test: Ignore UTF-8 errors in assert_debug_log
read() fails in text mode when the unicode hasn't been fully written
yet. Fixes: "wallet_importdescriptors.py: can't decode bytes in position
228861-228863: unexpected end of data"
(https://github.com/bitcoin/bitcoin/issues/28030)
2023-07-20 09:15:29 +02:00
MarcoFalke
fa63326fbc
test: Fix debug_log_size helper
debug_log_bytes returned "an opaque number when in text mode"
(https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects),
not the number of bytes.

Fix this by using binary mode or text mode (with the same encoding)
consistently when opening the file for ftell() and read().
2023-07-20 09:15:04 +02:00
Martin Zumsande
e667bd68a1 test: fix intermittent failure in wallet_resendwallettransactions.py
Before, it was possible that a resend was triggered right between
eviction the txns from the mempool and the check that they were evicted.
2023-07-19 14:14:59 -04:00
Matthew Zipkin
ba8ab4fc54
test: cover addrv2 support in anchors.dat with a TorV3 address 2023-07-19 13:24:44 -04:00
fanquake
0be2f5481c
Merge bitcoin/bitcoin#27986: test: remove race in the user-agent reception check
20b49460b3 test: remove race in the user-agent reception check (Vasil Dimov)

Pull request description:

  In `add_p2p_connection()` we connect to `bitcoind` from the Python client and check that it has received our version string.

  This check looked up the last/newest entry from `getpeerinfo` RPC, assuming that it must be the connection we have just opened. But this will not be the case if a new inbound or outbound connection is made to/from `bitcoind` in the meantime.

  Instead of the last entry in `getpeerinfo`, check all and find the one which corresponds to our connection using our outgoing address:port tuple which is unique.

ACKs for top commit:
  jonatack:
    re-ACK 20b49460b3
  MarcoFalke:
    Concept ACK 20b49460b3

Tree-SHA512: 61fd3359ef11ea830021ede0e745497a7b60690c32d21c47cd12ff79f78907bb45e922c9d01e5d7ff614a8cd5e4560d39a3fc86b01b200429773a23ace3917e4
2023-07-19 12:32:30 +01:00
Andrew Chow
bc88f3ab90
Merge bitcoin/bitcoin#27997: Descriptors: rule out unspendable miniscript descriptors
c7db88af71 descriptor: assert we never parse a sane miniscript with no pubkey (Antoine Poinsot)
a49402a9ec qa: make sure we don't let unspendable Miniscript descriptors be imported (Antoine Poinsot)
639e3b6c97 descriptor: refuse to parse unspendable miniscript descriptors (Antoine Poinsot)
e3280eae1b miniscript: make GetStackSize() and GetOps() return optionals (Antoine Poinsot)

Pull request description:

  `IsSane()` in Miniscript does not ensure a Script is actually spendable. This is an issue as we would accept any sane Miniscript when parsing a descriptor. Fix this by explicitly checking a Miniscript descriptor is both sane and spendable when parsing it.

  This bug was exposed due to a check added in #22838 (https://github.com/bitcoin/bitcoin/pull/22838#discussion_r1226859880) that triggered a fuzz crash (https://github.com/bitcoin/bitcoin/pull/22838#issuecomment-1612510057).

ACKs for top commit:
  sipa:
    utACK c7db88af71
  achow101:
    ACK c7db88af71

Tree-SHA512: e79bc9f7842e98a4e8f358f05811fca51b15b4b80a171c0d2b17cf4bb1f578a18e4397bc2ece9817d392e0de0196ee6a054b7318441fd3566dd22e1f03eb64a5
2023-07-17 19:16:09 -04:00
MarcoFalke
faf8be7c32
test: Disable known broken USDT test 2023-07-17 13:49:00 +02:00
Andrew Chow
357e3f6aa4
Merge bitcoin/bitcoin#28025: test: refactor: deduplicate legacy ECDSA signing for tx inputs
5cf44275c8 test: refactor: deduplicate legacy ECDSA signing for tx inputs (Sebastian Falbesoner)

Pull request description:

  There are several instances in functional tests and the framework (MiniWallet, feature_block.py, p2p_segwit.py) where we create a legacy ECDSA signature for a certain transaction's input by doing the following steps:

  1. calculate the `LegacySignatureHash` with the desired sighash type
  2. create the actual digital signature by calling `ECKey.sign_ecdsa` on the signature message hash calculated above
  3. put the DER-encoded result as CScript data push into tx input's scriptSig

  Create a new helper `sign_input_legacy` which hides those details and takes only the necessary parameters (tx, input index, relevant scriptPubKey, private key, sighash type [SIGHASH_ALL by default]). For further convenience, the signature is prepended to already existing data-pushes in scriptSig, in order to avoid rehashing the transaction after calling the new signing function.

ACKs for top commit:
  dimitaracev:
    ACK `5cf4427`
  achow101:
    ACK 5cf44275c8
  pinheadmz:
    ACK 5cf44275c8

Tree-SHA512: 8f0e4fb2c3e0f84fac5dbc4dda87973276242b0f628034272a7f3e45434c1e17dd1b26a37edfb302dcaf380dbfe98b0417391ace5e0ac9720155d8fba702031e
2023-07-11 17:25:40 -04:00
fanquake
21ed784614
Merge bitcoin/bitcoin#28028: test: Check expected_stderr after stop
faf902858d test: Check expected_stderr after stop (MarcoFalke)

Pull request description:

  This fixes a bug where stderr wasn't checked for the shutdown sequence.

  Fix that by waiting for the shutdown to finish and then check stderr.

ACKs for top commit:
  theStack:
    ACK faf902858d

Tree-SHA512: a70cd1e6cda84d542782e41e8b59741dbcd472c0d0575bcef5cbfd1418473ce94efe921481d557bae3fbbdd78f1c49c09c48872883c052d87c5c9a9a51492692
2023-07-11 10:14:48 +01:00
Matthew Zipkin
b4bee4bbf4
test: add keep_alive option to socks5 proxy in test_framework
The Socks5 server we use in the test framework would disconnect
by default immediately after the handshake and sometimes would
not register as a connected peer by bitcoind.
2023-07-10 10:07:45 -04:00
Matthew Zipkin
5aaf988ccc
test: cover TorV3 address in p2p_addrv2_relay 2023-07-10 10:07:45 -04:00
brunoerg
80f64a3d40
test: add support for all networks in CAddress in messages.py
Also removes TorV2 from messages.py
See https://github.com/bitcoin/bitcoin/pull/22050

Co-authored-by: Matthew Zipkin <pinheadmz@gmail.com>
2023-07-10 10:07:40 -04:00
furszy
fcbdaeef4d
init: don't start indexes sync thread prematurely
By moving the 'StartIndexes()' call into the 'initload'
thread, we can remove the threads active wait. Optimizing
the available resources.

The only difference with the current state is that now the
indexes threads will only be started when they can process
work and not before it.
2023-07-10 10:50:50 -03:00
MarcoFalke
faf902858d
test: Check expected_stderr after stop 2023-07-10 13:45:50 +02:00
furszy
04575106b2
scripted-diff: rename 'loadblk' thread name to 'initload'
The thread does not only load blocks, it loads the mempool and,
in a future commit, will start the indexes as well.

Also, renamed the 'ThreadImport' function to 'ImportBlocks'
And the 'm_load_block' class member to 'm_thread_load'.

-BEGIN VERIFY SCRIPT-

sed -i "s/ThreadImport/ImportBlocks/g" $(git grep -l ThreadImport -- ':!/doc/')
sed -i "s/loadblk/initload/g" $(git grep -l loadblk -- ':!/doc/release-notes/')
sed -i "s/m_load_block/m_thread_load/g" $(git grep -l m_load_block)

-END VERIFY SCRIPT-
2023-07-07 19:31:27 -03:00
Sebastian Falbesoner
bbbb89d238 test: miner: add coverage for -blockmintxfee setting
Co-authored-by: glozow <gloriajzhao@gmail.com>
2023-07-07 15:56:24 +02:00
furszy
7ecc29a0b7
test: wallet, add coverage for addressbook migration 2023-07-06 16:11:55 -03:00
Vasil Dimov
20b49460b3
test: remove race in the user-agent reception check
In `add_p2p_connection()` we connect to `bitcoind` from the Python
client and check that it has received our version string.

This check looked up the last/newest entry from `getpeerinfo` RPC,
assuming that it must be the connection we have just opened. But this
will not be the case if a new inbound or outbound connection is made
to/from `bitcoind` in the meantime.

Instead of the last entry in `getpeerinfo`, check all and find the one
which corresponds to our connection using our outgoing address:port
tuple which is unique.

Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Co-authored-by: Jon Atack <jon@atack.com>
2023-07-06 17:48:15 +02:00
glozow
ecf3baffc0
Merge bitcoin/bitcoin#27869: wallet: Give deprecation warning when loading a legacy wallet
8fbb6e99bf wallet: Give deprecation warning when loading a legacy wallet (Andrew Chow)

Pull request description:

  Next step in legacy wallet deprecation.

ACKs for top commit:
  S3RK:
    reACK 8fbb6e99bf
  jonatack:
    re-ACK 8fbb6e99bf

Tree-SHA512: 902984b09452926cf199f06e5fb56e4985325cdd5e0dcc829992158488f42d5fbc33e9a30a29303feac24c8315193e8d31712022e2a0503abd6b67169a0027f4
2023-07-06 10:47:41 +01:00
Sebastian Falbesoner
5cf44275c8 test: refactor: deduplicate legacy ECDSA signing for tx inputs
There are several instances in functional tests and the framework
(MiniWallet, feature_block.py, p2p_segwit.py) where we create a legacy
ECDSA signature for a certain transaction's input by doing the following
steps:
    1) calculate the `LegacySignatureHash` with the desired sighash type
    2) create the actual digital signature by calling `ECKey.sign_ecdsa`
       on the signature message hash calculated above
    3) put the DER-encoded result as CScript data push into
       tx input's scriptSig

Create a new helper `sign_input_legacy` which hides those details and
takes only the necessary parameters (tx, input index, relevant
scriptPubKey, private key, sighash type [SIGHASH_ALL by default]). For
further convenience, the signature is prepended to already existing
data-pushes in scriptSig, in order to avoid rehashing the transaction
after calling the new signing function.
2023-07-03 17:33:41 +02:00
Antoine Poinsot
a49402a9ec
qa: make sure we don't let unspendable Miniscript descriptors be imported 2023-07-01 12:12:26 +02:00
fanquake
61d59fed74
Merge bitcoin/bitcoin#24005: test: add python implementation of Elligator swift
4f4d039a98 test: add ellswift test vectors from BIP324 (stratospher)
a31287718a test: Add ellswift unit tests (stratospher)
714fb2c02a test: Add python ellswift implementation to test framework (stratospher)

Pull request description:

  Built on top of https://github.com/bitcoin/bitcoin/pull/26222.

  This PR introduces Elligator swift encoding and decoding in the functional test framework. It's used in #24748 for writing p2p encryption tests.

ACKs for top commit:
  sipa:
    ACK 4f4d039a98
  theStack:
    ACK 4f4d039a98 🐊

Tree-SHA512: 32bc8e88f715f2cd67dc04cd38db92680872072cb3775478e2c30da89aa2da2742992779ea14da2f1faca09228942cfbd86d6957402b24bf560244b389e03540
2023-06-30 19:30:49 +01:00
fanquake
3367e1c850
Merge bitcoin/bitcoin#28009: script, test: python typing and linter updates
6c97757a48 script: appease spelling linter (Jon Atack)
1316119ce7 script: update ignored-words.txt (Jon Atack)
146c861da2 script: update linter dependencies (Jon Atack)
92408224a4 test: fix PEP484 no implicit optional argument types errors (Jon Atack)
f86a301433 script, test: add missing python type annotations (Jon Atack)

Pull request description:

  With these updates, `./test/lint/lint-python.py` and `./test/lint/lint-spelling.py` should be green again for developers using relatively recent Python dependencies, in particular mypy 0.991 (released 11/2022) and later. Please see the commit messages for details.

ACKs for top commit:
  fanquake:
    ACK 6c97757a48

Tree-SHA512: 8a46a4d36d5978affdcecf4f2ace20ca1b52d483e098304911a2169afe60ccb9b042fa90c04b762d94f3ce53d2cafe6f24476ae839867a770c7f31e7e7242d99
2023-06-30 16:20:37 +01:00
Jon Atack
92408224a4 test: fix PEP484 no implicit optional argument types errors
Fix warnings for these files when ./test/lint/lint-python.py is run using
mypy 0.991 (released 11/2022) and later:

$ test/lint/lint-python.py
test/functional/test_framework/coverage.py:23: error: Incompatible default for argument "coverage_logfile" (default has type "None", argument has type "str")  [assignment]
test/functional/test_framework/coverage.py:23: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
test/functional/test_framework/util.py:318: error: Incompatible default for argument "timeout" (default has type "None", argument has type "int")  [assignment]
test/functional/test_framework/util.py:318: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
test/functional/test_framework/util.py:318: error: Incompatible default for argument "coveragedir" (default has type "None", argument has type "str")  [assignment]
test/functional/interface_rest.py:67: error: Incompatible default for argument "query_params" (default has type "None", argument has type "dict[str, Any]")  [assignment]
test/functional/interface_rest.py:67: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True

Verified using https://github.com/hauntsaninja/no_implicit_optional

For details, see:

https://mypy-lang.blogspot.com/2022/11/mypy-0990-released.html
2023-06-29 16:14:07 -06:00
Jon Atack
f86a301433 script, test: add missing python type annotations
Fix warnings for these files when ./test/lint/lint-python.py is run using
mypy 0.991 (released 11/2022) and later:

"By default the bodies of untyped functions are not checked, consider using
--check-untyped-defs [annotation-unchecked]"

For details, see:

https://mypy-lang.blogspot.com/2022/11/mypy-0990-released.html
2023-06-29 16:13:51 -06:00
stratospher
4f4d039a98 test: add ellswift test vectors from BIP324
The test vector input file is taken from:
1. https://github.com/bitcoin/bips/blob/master/bip-0324/xswiftec_inv_test_vectors.csv
2. https://github.com/bitcoin/bips/blob/master/bip-0324/ellswift_decode_test_vectors.csv

Co-authored-by: theStack <sebastian.falbesoner@gmail.com>
2023-06-29 23:32:56 +05:30
stratospher
a31287718a test: Add ellswift unit tests
remove util also since unit tests there were removed in #27538

Co-authored-by: theStack <sebastian.falbesoner@gmail.com>
2023-06-29 23:32:56 +05:30
stratospher
714fb2c02a test: Add python ellswift implementation to test framework
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
2023-06-29 23:32:54 +05:30
MarcoFalke
fabd34873c
test: Rename EncodeDecimal to serialization_fallback
The new name better explains that the function handles fallbacks,
without listing all in the function name.
2023-06-29 19:51:43 +02:00
fanquake
e8543629ae
Merge bitcoin/bitcoin#27884: test: Use TestNode datadir_path or chain_path where possible
aaaa3aefbd test: Use TestNode *_path properties where possible (MarcoFalke)
dddd89962b test: Allow pathlib.Path as RPC argument via authproxy (MarcoFalke)
fa41614a0a scripted-diff: Use wallets_path and chain_path where possible (MarcoFalke)
fa493fadfb test: Use wallet_dir lambda in wallet_multiwallet test where possible (MarcoFalke)

Pull request description:

  It seems inconsistent, fragile and verbose to:

  * Call `get_datadir_path` to recreate the path that already exists as field in TestNode
  * Call `os.path.join` with the hardcoded chain name or `self.chain` to recreate the TestNode `chain_path` property
  * Sometimes even use the hardcoded node dir name (`"node0"`)

  Fix all issues by using the TestNode properties.

ACKs for top commit:
  willcl-ark:
    re-ACK aaaa3aefbd
  theStack:
    Code-review ACK aaaa3aefbd 🌊

Tree-SHA512: e4720278085beb8164e1fe6c1aa18f601558a9263494ce69a83764c1487007de63ebb51d1b1151862dc4d5b49ded6162a5c1553cd30ea1c28627d447db4d8e72
2023-06-29 09:51:53 +01:00
Andrew Chow
626d346469
Merge bitcoin/bitcoin#26222: Introduce secp256k1 module with field and group classes to test framework
d4fb58ae8a test: EC: optimize scalar multiplication of G by using lookup table (Sebastian Falbesoner)
1830dd8820 test: add secp256k1 module with FE (field element) and GE (group element) classes (Pieter Wuille)

Pull request description:

  This PR rewrites a portion of `test_framework/key.py`, in a compatible way, by introducing classes that encapsulate field element and group element logic, in an attempt to be more readable and reusable.

  To maximize readability, the group element logic does not use Jacobian coordinates. Instead, group elements just store (affine) X and Y coordinates directly. To compensate for the performance loss this causes, field elements are represented as fractions. This undoes most, but not all, of the performance loss, and there is a few % slowdown (as measured in `feature_taproot.py`, which heavily uses this).

  The upside is that the implementation for group laws (point doubling, addition, subtraction, ...) is very close to the mathematical description of elliptic curves, and this extends to potential future extensions (e.g. ElligatorSwift as needed by #27479).

ACKs for top commit:
  achow101:
    ACK d4fb58ae8a
  theStack:
    re-ACK d4fb58ae8a
  stratospher:
    tested ACK d4fb58a. really liked how this PR makes the secp256k1 code in the tests more intuitive and easier to follow!

Tree-SHA512: 9e0d65d7de0d4fb35ad19a1c19da7f41e5e1db33631df898c6d18ea227258a8ba80c893dab862b0fa9b0fb2efd0406ad4a72229ee26d7d8d733dee1d56947f18
2023-06-28 16:27:55 -04:00
Andrew Chow
caff95a023
Merge bitcoin/bitcoin#27896: Remove the syscall sandbox
32e2ffc393 Remove the syscall sandbox (fanquake)

Pull request description:

  After initially being merged in #20487, it's no-longer clear that an internal syscall sandboxing mechanism is something that Bitcoin Core should have/maintain, especially when compared to better maintained/supported alterantives, i.e [firejail](https://github.com/netblue30/firejail).

  There is more related discussion in #24771.

  Note that given where it's used, the sandbox also gets dragged into the kernel.

  If it's removed, this should not require any sort of deprecation, as this was only ever an opt-in, experimental feature.

  Closes #24771.

ACKs for top commit:
  davidgumberg:
     crACK 32e2ffc393
  achow101:
    ACK 32e2ffc393
  dergoegge:
    ACK 32e2ffc393

Tree-SHA512: 8cf71c5623bb642cb515531d4a2545d806e503b9d57bfc15a996597632b06103d60d985fd7f843a3c1da6528bc38d0298d6b8bcf0be6f851795a8040d71faf16
2023-06-27 18:19:21 -04:00
Sebastian Falbesoner
d4fb58ae8a test: EC: optimize scalar multiplication of G by using lookup table
On my machine, this speeds up the functional test feature_taproot.py by
a factor of >1.66x (runtime decrease from 1m16.587s to 45.334s).

Co-authored-by: Pieter Wuille <pieter@wuille.net>
2023-06-27 09:34:52 -04:00
Pieter Wuille
1830dd8820 test: add secp256k1 module with FE (field element) and GE (group element) classes
These are primarily designed for ease of understanding, not performance.
2023-06-27 09:34:48 -04:00
Andrew Chow
e8c31f135c tests: Test for bumping single output transaction 2023-06-26 17:56:50 -04:00
Andrew Chow
4f4d4407e3 test: Test bumpfee reduce_output 2023-06-26 17:56:50 -04:00
Andrew Chow
8847497161
Merge bitcoin/bitcoin#27631: test: avoid sporadic MINIMALDATA failure in feature_taproot.py (fixes #27595)
54877253c8 test: avoid sporadic MINIMALDATA failure in feature_taproot.py (fixes #27595) (Sebastian Falbesoner)

Pull request description:

  The functional test feature_taproot.py fails in some rare cases on the execution of the following `"branched_codesep"` spending script (can be reproduced via `$ ./test/functional/feature_taproot.py --randomseed 9048710178866422833` on master / 137a98c5a2):

  9d85c03620/test/functional/feature_taproot.py (L741)

  The problem occurs if the first data-push (having random content with a random length in the range [0, 510]) has a length of 1 and the single byte has value of [1...16] or [-1]; in this case, the data-push is not minimally encoded by test framework's CScript class (i.e. doesn't use the special op-codes OP_1...OP_16 or OP_1NEGATE) and the script interpreter throws an SCRIPT_ERR_MINIMALDATA error:

  ```
  test_framework.authproxy.JSONRPCException: non-mandatory-script-verify-flag (Data push larger than necessary) (-26)
  ```

  Background: the functional test framework's CScript class translates passed bytes/bytearrays always to data pushes using OP_PUSHx/OP_PUSHDATA{1,2,4} op-codes (see `CScript.__coerce_instance(...)`). E.g. the expression `CScript(bytes([1]))` yields `bytes([OP_PUSH1, 1])` instead of the minimal-encoded `bytes([OP_1])`.

  Fix this by adapting the random-size range to [2,...], i.e. never pass byte-arrays below length two to be pushed.

  Closes #27595.

ACKs for top commit:
  instagibbs:
    ACK 54877253c8
  sipa:
    utACK 54877253c8
  achow101:
    ACK 54877253c8

Tree-SHA512: 3ffad89b2c3985c20702242192e744c9b10188bff880efaf3c38424a00fa07bd4608d8c948678ff9cdbb4e1e5b06696c7f55407ee10bb05edbb3ee03aa599cdc
2023-06-23 18:54:06 -04:00
Andrew Chow
035ae61c5a
Merge bitcoin/bitcoin#27577: p2p: give seednodes time before falling back to fixed seeds
30778124b8 net: Give seednodes time before falling back to fixed seeds (Martin Zumsande)

Pull request description:

  `-seednode` is an alternative bootstrap mechanism - when choosing it, we make a `AddrFetch` connection to the specified peer, gather addresses from them, and then disconnect. Presumably, if users specify a seednode they prefer addresses from that node over fixed seeds.

  However, when disabling dns seeds and specifiying `-seednode`, `CConnman::ProcessAddrFetch()`  immediately removes the entry from `m_addr_fetches` (before the seednode could give us addresses) - and once `m_addr_fetches`  is empty, `ThreadOpenConnections` will add fixed seeds, resulting in a "race" between the fixed seeds and seednodes filling up AddrMan.

  This PR suggests to check for any provided `-seednode` arg instead of using the size of `m_addr_fetches`, thus delaying the querying of fixed seeds for 1 minute when specifying any seednode (as we already do for `addnode` peers).
  That way, we actually give the seednodes a chance for  to provide us with addresses before falling back to fixed seeds.

  This can be tested with `bitcoind -debug=net -dnsseed=0 -seednode=(...)` on a node without `peers.dat` and observing the debug log.

ACKs for top commit:
  ajtowns:
    utACK 30778124b8
  achow101:
    ACK 30778124b8
  dergoegge:
    Code review ACK 30778124b8
  sr-gi:
    ACK [3077812](30778124b8) with a tiny nit, feel free to ignore it

Tree-SHA512: 96446eb34c0805f10ee158a00a3001a07029e795ac40ad5638228d426e30e9bb836c64ac05d145f2f9ab23ec5a528f3a416e3d52ecfdfb0b813bd4b1ebab3c01
2023-06-23 17:39:58 -04:00
Andrew Chow
8fbb6e99bf wallet: Give deprecation warning when loading a legacy wallet 2023-06-23 16:37:22 -04:00
stickies-v
9f55773a37
test: refactor: usdt_mempool: store all events
Even though we expect these functions to only produce one event,
we still keep a counter to check if that's true. By simply storing
all the events, we can remove the counters and make debugging
easier, by allowing pdb to access the events.
2023-06-23 16:47:43 +01:00
stickies-v
bc43270450
test: refactor: remove unnecessary nonlocal
Since we're only mutating, and not reassigning, we don't need to
declare `events` as `nonlocal`.
2023-06-23 16:47:43 +01:00
stickies-v
326db63a68
test: log sanity check assertion failures 2023-06-23 16:47:43 +01:00
stickies-v
f5525ad680
test: store utxocache events
By storing the events instead of doing the comparison inside the
handle_utxocache_* functions, we simplify the overall logic and
potentially making debugging easier, by allowing pdb to access the
events.

Mostly a refactor, but changes logging behaviour slightly by not
raising and not calling self.log.exception("Assertion failed")
2023-06-23 16:46:46 +01:00
stickies-v
f1b99ac94f
test: refactor: deduplicate handle_utxocache_* logic
Carve out the comparison logic into a helper function to avoid
code duplication.
2023-06-23 16:46:32 +01:00
stickies-v
ad90ba36bd
test: refactor: rename inbound to is_inbound
Makes it easier to recognize this variable represents a flag.
2023-06-23 16:46:15 +01:00
stickies-v
afc0224cdb
test: refactor: remove unnecessary blocks_checked counter
Since we already store all the blocks in `events`, keeping an
additional counter is redundant.
2023-06-23 16:46:02 +01:00
MarcoFalke
fa748c6f2a
test: Fix intermittent issue in mining_getblocktemplate_longpoll.py 2023-06-23 15:21:29 +02:00
fanquake
a7261da479
Merge bitcoin/bitcoin#27831: test: handle failed assert_equal() assertions in bcc callback functions
61f4b9b7ad Manage exceptions in bcc callback functions (virtu)

Pull request description:

  Address #27380 (and similar future issues) by handling failed `assert_equal()` assertions in bcc callback functions

  ### Problem

  Exceptions are not propagated in ctype callback functions used by bcc. This means an AssertionError exception raised by `assert_equal()` to signal a failed assertion is not getting caught and properly logged. Instead, the error is logged to stdout and execution of the callback stops.

  The current workaround to check whether all `assert_equal()` assertions in a callback succeeded is to increment a success counter after the assertions (which only gets incremented if none exception is raised and stops execution). Then, outside the callback, the success counter can be used to check whether a callback executed successfully.

  One issue with the described workaround is that when an exception occurs, there is no way of telling which of the `assert_equal()` statements caused the exception; moreover, there is no way of inspecting how the pieces of data that got compared in `assert_equal()` differed (often a crucial clue when debugging what went wrong).

  This problem is happening in #27380: Sporadically, in the `mempool:rejected` test, execution does not reach the end of the callback function and the success counter is not incremented. Thus, the test fails when comparing the counter to its expected value of one. Without knowing which of the asserts failed any why it failed, this issue is hard to debug.

  ### Solution

  Two fixes come to mind. The first involves having the callback function make event data accessible outside the callback and inspecting the event using `assert_equal()` outside the callback. This solution still requires a counter in the callback in order  to tell whether a callback was actually executed or if instead the call to perf_buffer_poll() timed out.

  The second fix entails wrapping all relevant `assert_equal()` statements inside callback functions into try-catch blocks and manually logging AssertionErrors. While not as elegant in terms of design, this approach can be more pragmatic for more complex tests (e.g., ones involving multiple events, events of different types, or the order of events).

  The solution proposed here is to select the most pragmatic fix on a case-by-case basis: Tests in `interface_usdt_net.py`, `interface_usdt_mempool.py` and `interface_usdt_validation.py` have been refactored to use the first approach, while the second approach was chosen for `interface_usdt_utxocache.py` (partly to provide a reference for the second approach, but mainly because the utxocache tests are the most intricate tests, and refactoring them to use the first approach would negatively impact their readability). Lastly, `interface_usdt_coinselection.py` was kept unchanged because it does not use `assert_equal()` statements inside callback functions.

ACKs for top commit:
  0xB10C:
    Reviewed the changes since my last review. ACK 61f4b9b7ad. I've tested that the combined log contains both exceptions by modifying `interface_usdt_utxocache.py`.
  willcl-ark:
    utACK 61f4b9b
  stickies-v:
    utACK 61f4b9b7a

Tree-SHA512: 85cdaabf370d4f09a9eab6af9ce7c796cd9d08cb91f38f021f71adda34c5f643331022dd09cadb95be2185dad6016c95cbb8942e41e4fbd566a49bf431c5141a
2023-06-22 16:08:15 +01:00
Martin Zumsande
30778124b8 net: Give seednodes time before falling back to fixed seeds
Before, we'd remove a seednode from the list right after connecting
to it, leading to a race with loading the fixed seed and connecting
to them.
2023-06-21 15:11:00 -04:00
fanquake
7d65e3372f
Merge bitcoin/bitcoin#27733: test: refactor: introduce generate_keypair helper with WIF support
1a572ce7d6 test: refactor: introduce `generate_keypair` helper with WIF support (Sebastian Falbesoner)

Pull request description:

  In functional tests it is a quite common scenario to generate fresh elliptic curve keypairs, which is currently a bit cumbersome as it involves multiple steps, e.g.:

      privkey = ECKey()
      privkey.generate()
      privkey_wif = bytes_to_wif(privkey.get_bytes())
      pubkey = privkey.get_pubkey().get_bytes()

  Simplify this by providing a new `generate_keypair` helper function that returns the private key either as `ECKey` object or as WIF-string (depending on the boolean `wif` parameter) and the public key as byte-string; these formats are what we mostly need (currently we don't use `ECPubKey` objects from generated keypairs anywhere).

  With this, most of the affected code blocks following the pattern above can be replaced by one-liners, e.g.:

      privkey, pubkey = generate_keypair(wif=True)

  Note that after this commit, the only direct uses of `ECKey` remain in situations where we want to set the private key explicitly, e.g. in MiniWallet (test/functional/test_framework/wallet.py) or the test for the signet miner script (test/functional/tool_signet_miner.py).

ACKs for top commit:
  instagibbs:
    ACK 1a572ce7d6
  kevkevinpal:
    reACK [1a572ce](1a572ce7d6)
  stratospher:
    ACK 1a572ce7. neat to have this since keypair generation is done in lots of places.

Tree-SHA512: ceb695ba7b34dc9f65357b55be03e67609e7e13a178083d405284eff4d8d3c5cea4fb0b6632658604a533f38ebfefc33e0c375995cc21ebc7843442ad764287b
2023-06-21 10:45:25 +01:00
MarcoFalke
aaaa3aefbd
test: Use TestNode *_path properties where possible
Seems odd to place the burden on test writers to hardcode the chain or
datadir path for the nodes under test.
2023-06-21 08:49:18 +02:00
MarcoFalke
dddd89962b
test: Allow pathlib.Path as RPC argument via authproxy
Also, add datadir_path property to TestNode
2023-06-21 08:48:52 +02:00
MarcoFalke
fa41614a0a
scripted-diff: Use wallets_path and chain_path where possible
Instead of passing the datadir and chain name to os.path.join, just use
the existing properties, which are the same.

-BEGIN VERIFY SCRIPT-
 sed -i --regexp-extended 's|\.datadir, self\.chain, .wallets.|.wallets_path|g' $(git grep -l '\.datadir, self\.chain,')
 sed -i --regexp-extended 's|\.datadir, self\.chain,|.chain_path,|g'            $(git grep -l '\.datadir, self\.chain,')
-END VERIFY SCRIPT-
2023-06-21 08:48:34 +02:00
MarcoFalke
fa493fadfb
test: Use wallet_dir lambda in wallet_multiwallet test where possible
Seems odd to hardcode all parent directory names in the path for no good
reason.

Also, add wallet_path property to TestNode.

Also, rework wallet_backup.py test for scripted-diff in the next commit.
2023-06-21 08:47:54 +02:00
Ryan Ofsky
ee22ca59a2
Merge bitcoin/bitcoin#26740: wallet: Migrate wallets that are not in a wallet dir
a1e653828b test: Add test for migrating default wallet and plain file wallet (Andrew Chow)
bdbe3fd76b wallet: Generated migrated wallet's path from walletdir and name (Andrew Chow)

Pull request description:

  This PR fixes an assertion error that is hit during the setup of the new database during migration of a wallet that was not contained in a wallet dir. Also added a test for this case as well as one for migrating the default wallet.

ACKs for top commit:
  ryanofsky:
    Code review ACK a1e653828b
  furszy:
    ACK a1e65382

Tree-SHA512: 96b218c0de8567d8650ec96e1bf58b0f8ca4c4726f5efc6362453979b56b9d569baea0bb09befb3a5aed8d16d29bf75ed5cd8ffc432bbd4cbcad3ac5574bc479
2023-06-20 16:10:44 -04:00
Andrew Chow
e4bbfb2d49
Merge bitcoin/bitcoin#27632: Raise on invalid -debug and -loglevel config options
daa5a658c0 refactor: rename BCLog::BLOCKSTORE to BLOCKSTORAGE (Jon Atack)
cf622b214b doc: release note re raising on invalid -debug/debugexclude/loglevel (Jon Atack)
6cb1c66041 init: remove config option names from translated -loglevel strings (Jon Atack)
2547829272 test: -loglevel raises on invalid values (Jon Atack)
a9c295888b init: raise on invalid loglevel config option (Jon Atack)
b0c3995393 test: -debug and -debugexclude raise on invalid values (Jon Atack)
4c3c19d943 init: raise on invalid debug/debugexclude config options (Jon Atack)

Pull request description:

  and rename BCLog::BLOCKSTORE to BLOCKSTORAGE so the enum is the same as its value like the other BCLog enums.

  Per discussion in bitcoin-core-dev IRC today from https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-05-11#921458.

ACKs for top commit:
  achow101:
    ACK daa5a658c0
  ryanofsky:
    Code review ACK daa5a658c0. Just translated string template cleanup since last review
  pinheadmz:
    re-ACK daa5a658c0

Tree-SHA512: 4c107a93d8e8ce4e2ee81d44aec672526ca354ec390b241221067f68204beac8b4ba7a65748bcfa124ff2245c4307fa9243ec4fe0b464d0fa69c787fb322c3cc
2023-06-20 13:55:18 -04:00
glozow
f80db62b2d
Merge bitcoin/bitcoin#27622: Fee estimation: avoid serving stale fee estimate
d2b39e09bc test: ensure old fee_estimate.dat not read on restart and flushed (ismaelsadeeq)
cf219f29f3 tx fees, policy: read stale fee estimates with a regtest-only option (ismaelsadeeq)
3eb241a141 tx fees, policy: do not read estimates of old fee_estimates.dat (ismaelsadeeq)
5b886f2b43 tx fees, policy: periodically flush fee estimates to fee_estimates.dat (ismaelsadeeq)

Pull request description:

  Fixes #27555

  The issue arises when an old `fee_estimates.dat` file is sometimes read during initialization.
  Or after an unclean shutdown, the latest fee estimates are not flushed to `fee_estimates.dat`.
  If the fee estimates in the old file are old, they can cause transactions to become stuck in the mempool.
  This  PR ensures that nodes do not use stale estimates from the old file during initialization. If  `fee_estimates.dat`
  has not been updated for 60 hours or more, it is considered stale and will not be read during initialization. To avoid
  having old estimates, the `fee_estimates.dat` file will be flushed periodically every hour. As mentioned #27555

  > "The immediate improvement would be to store fee estimates to disk once an hour or so to reduce the chance of having an old file. From there, this case could probably be detected, and refuse to serve estimates until we sync."

  In addition, I will follow-up PR to persist the `mempoolminfee` across restarts.

ACKs for top commit:
  willcl-ark:
    ACK d2b39e09bc
  instagibbs:
    reACK d2b39e09bc
  glozow:
    ACK d2b39e09bc. One nit if you follow up.

Tree-SHA512: 4f6e0c296995d0eea5cf80c6aefdd79b7295a6a0ba446f2166f32afc105fe4f831cfda1ad3abd13c5c752b4fbea982cf4b97eaeda2af1fd7184670d41edcfeec
2023-06-20 16:48:29 +01:00
Sebastian Falbesoner
1a572ce7d6 test: refactor: introduce generate_keypair helper with WIF support
In functional tests it is a quite common scenario to generate fresh
elliptic curve keypairs, which is currently a bit cumbersome as it
involves multiple steps, e.g.:

    privkey = ECKey()
    privkey.generate()
    privkey_wif = bytes_to_wif(privkey.get_bytes())
    pubkey = privkey.get_pubkey().get_bytes()

Simplify this by providing a new `generate_keypair` helper function that
returns the private key either as `ECKey` object or as WIF-string
(depending on the boolean `wif` parameter) and the public key as
byte-string; these formats are what we mostly need (currently we don't
use `ECPubKey` objects from generated keypairs anywhere).

With this, most of the affected code blocks following the pattern above
can be replaced by one-liners, e.g.:

    privkey, pubkey = generate_keypair(wif=True)

Note that after this commit, the only direct uses of `ECKey` remain in
situations where we want to set the private key explicitly, e.g. in
MiniWallet (test/functional/test_framework/wallet.py) or the test for
the signet miner script (test/functional/tool_signet_miner.py).
2023-06-19 17:38:14 +02:00
virtu
61f4b9b7ad Manage exceptions in bcc callback functions
Exceptions are not propagated in ctype callback functions used by bcc.
This means an AssertionError exception raised by check_equal() to signal
a failed assertion is not getting caught and properly logged. Instead,
the error is logged to stdout and execution of the handler stops.

The current workaround to check whether all check_equal() assertions in
a callback succeeded is to increment a success counter after the
assertions (which only gets incremented if none exception is raised and
stops execution). Then, outside the callback, the success counter can be
used to check whether a callback executed successfully.

One issue with the described workaround is that when an exception
occurs, there is no way of telling which of the check_equal() statements
caused the exception; moreover, there is no way of inspecting how the
pieces of data that got compared in check_equal() differed (often
a crucial clue when debugging what went wrong).

Two fixes to this problem come to mind. The first involves having the
callback function make event data accessible outside the callback and
inspecting the event using check_equal() outside the callback. This
solution still requires a counter in the callback to tell whether
a callback was actually executed or if instead the call to
perf_buffer_poll() timed out.

The second fix entails wrapping all relevant check_equal() statements
inside callback functions into try-catch blocks and manually logging
AssertionErrors. While not as elegant in terms of design, this approach
can be more pragmatic for more complex tests (e.g., ones involving
multiple events, events of different types, or the order of events).

The solution proposed here is to select the most pragmatic fix on
a case-by-case basis: Tests in interface_usdt_net.py,
interface_usdt_mempool.py and interface_usdt_validation.py have been
refactored to use the first approach, while the second approach was
chosen for interface_usdt_utxocache.py (partly to provide a reference
for the second approach, but mainly because the utxocache tests are the
most intricate tests, and refactoring them to use the first approach
would negatively impact their readability). Lastly,
interface_usdt_coinselection.py was kept unchanged because it does not
use check_equal() statements inside callback functions.
2023-06-19 14:38:32 +02:00
fanquake
7f0b79ea13
Merge bitcoin/bitcoin#27906: doc: test: update TestShell instructions
14405e8d4d doc: test: update TestShell instructions (ismaelsadeeq)

Pull request description:

  Fixes  #27904

  From  #27904 and IRC.
  Update [Testshell instructions ](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test-shell.md#2-importing-testshell-from-the-bitcoin-core-repository)

  E.g `TestShell.setup()` throws
  ```
  AttributeError: type object 'TestShell' has no attribute 'setup'
  ```
  Parentheses are missing, it should be `TestShell().setup()`

ACKs for top commit:
  Sjors:
    utACK 14405e8d4d
  brunoerg:
    crACK 14405e8d4d
  hernanmarino:
    utACK 14405e8d4d

Tree-SHA512: ffe5fa1103a3b00ef0ee99879adae967b0da07cb8f8451c4c261b0a70b3b666af7aeaacd6f46f85a84ee5e9c7c7ed49700209b5b1f124d7a76efc420ad5c9cd9
2023-06-18 12:48:46 +02:00
ismaelsadeeq
14405e8d4d doc: test: update TestShell instructions
add missing parentheses `TestShell.method` should be `TestShell().method`.
2023-06-16 22:55:36 +01:00
Andrew Chow
f0758d8a66
Merge bitcoin/bitcoin#27757: rpc: remove deprecated "warning" field from {create,load,restore,unload}wallet
5524fa00fa doc: add release note about removal of `deprecatedrpc=walletwarningfield` flag (Sebastian Falbesoner)
5c77db7354 Restorewallet/createwallet help documentation fixups/improvements (Jon Atack)
a00ae31fcc rpc: remove deprecated "warning" field from {create,load,restore,unload}wallet (Sebastian Falbesoner)

Pull request description:

  The "warning" string field for wallet creating/loading RPCs (`createwallet`, `loadwallet`, `unloadwallet` and `restorewallet`) has been deprecated with the configuration option `-deprecatedrpc=walletwarningfield` in PR #27279 (released in v25.0). For the next release v26.0, the field and the configuration option can be removed.

ACKs for top commit:
  achow101:
    ACK 5524fa00fa
  jonatack:
    ACK 5524fa00fa

Tree-SHA512: 8212f72067d08095304018b8a95d2ebef630004b65123483fbbfb078cc5709c2d825bbc35b16ea5f6b28ae7377347382d7e9afaf7bdbf0575d2c229d970784de
2023-06-16 15:11:44 -04:00