`p2p_dns_seeds.py` would try to connect to the DNS server configured on
the machine and resolve `dummySeed.invalid`.
To block that configure an unavailable proxy which will be used also to
connect to the name server. The test needs 2 successful connections to
other peers (two Python `P2PInterface`s) and they work in spite of the
unavailable proxy because they are on `127.0.0.1` (`NET_UNROUTABLE`) and
the proxy is not used for that.
Github-Pull: #31646
Rebased-From: 2ed161c5ce
`feature_config_args.py` uses a proxy address of `1.2.3.4`. This results
in actually trying to open TCP connections over the internet to
`1.2.3.4:9050`.
The test does not need those to succeed so use `127.0.0.1:1` instead.
Also avoid `-noconnect=0` because that is interpreted as `-connect=1`
which is interpreted as `-connect=0.0.0.1` and a connection to
`0.0.0.1:18444` is attempted.
Github-Pull: #31646
Rebased-From: a5746dc559
Previously passing a directory path as -conf would lead to an ifstream being opened for it, and would not trigger any errors.
Github-Pull: #31212
Rebased-From: e85abe92c7
-noconf would previously lead to an ifstream "successfully" being opened to the ".bitcoin"-directory (not a file). (Guards against the general case of directories as configs are added in grandchild commit to this one).
Other users of AbsPathForConfigVal() in combination with negated args have been updated earlier in this PR ("args: Support -nopid" and "args: Support -norpccookiefile...").
Github-Pull: #31212
Rebased-From: 483f0dacc4
This ensures we don't needlessly start the node, and reduces implicit dependencies between test functions.
test_seed_peers() - Move assert calling RPC to verify correct chain after our own function actually started the node.
Github-Pull: #31212
Rebased-From: 312ec64cc0
Replaces belt & suspenders check for initialization in RPCAuthorized() with not allowing empty passwords further down.
Github-Pull: #31212
Rebased-From: 39cbd4f37c
1bf9b70658 docs: Add 379 and 387 to bips.md (Ava Chow)
35ef34eab7 docs: Remove release 28.0 release notes fragments (Ava Chow)
f5cf43bb91 build: Bump to 28.99 (Ava Chow)
Pull request description:
* Bump to 28.99 in preparation for the 28.x branching
* Remove current release note fragments. They've been moved to [draft release notes]((https://github.com/bitcoin-core/bitcoin-devwiki/wiki/28.0-Release-Notes-Draft).) in the wiki.
* Updated bips.md with missing BIPs that were implemented a while ago.
ACKs for top commit:
maflcko:
lgtm ACK 1bf9b70658
tdb3:
re ACK 1bf9b70658
Tree-SHA512: 9d5b05330268bb233369d78f06011dbdcdcb87770c499ae743ac4833735ba657b6611ff1e332c9d4added854b694c5a3a961d911247303f7ddaaaa72149d5574
b061b35105 seeds: Regenerate mainnet seeds (virtu)
02dc45c506 seeds: Pull nodes from Luke's seeder (virtu)
7a2068a0ff seeds: Pull nodes from virtu's crawler (virtu)
Pull request description:
This builds on #30008 and adds data [exported](https://github.com/virtu/seed-exporter) by [my crawler](https://github.com/virtu/p2p-crawler) an additional source for seed nodes. Data covers all supported network types.
[edit: Added Luke's seeder as input as well.]
### Motivation
- Further decentralizes the seed node selection process (in the long term potentially enabling an _n_-source threshold for nodes to prevent a single source from entering malicious nodes)
- No longer need to manually curate seed node list for any network type: See last paragraph of OP in #30008. My crawler has been [discovering the handful of available cjdns nodes](https://21.ninja/reachable-nodes/nodes-by-net-type/) for around two months, all but one of which meet the reliability criteria.
- Alignment of uptime requirements for Onion and I2P nodes with those of clearnet nodes to 50%: If I'm reading the code correctly, seeders appear to optimize for up-to-dateness by using [lower connection timeouts](3c1a63c672/src/crawl.rs (L349)) than [Bitcoin Core](bc87ad9854/src/netbase.cpp (L40C27-L40C48)) to maximize throughput. Since my crawler does not have the same timeliness requirements, it opts for accuracy by using generous timeouts. As a result, its data contains additional eligible Onion (and other darknet nodes), as is shown in the histogram below. Around 4500 Onion nodes are discovered so far (blue); my data adds ~6400 more (orange); ~ 1500 nodes take longer than the default 20-second Bitcoin Core timeout and won't qualify as "good".
Here's the current results with 512 nodes for all networks except cjdns:
<details>
<summary>Using the extra data</summary>
```
IPv4 IPv6 Onion I2P CJDNS Pass
10335 2531 11545 1589 10 Initial
10335 2531 11545 1589 10 Skip entries with invalid address
5639 1431 11163 1589 8 After removing duplicates
5606 1417 11163 1589 8 Enforce minimal number of blocks
5606 1417 11163 1589 8 Require service bit 1
4873 1228 11163 1589 8 Require minimum uptime
4846 1225 11161 1588 8 Require a known and recent user agent
4846 1225 11161 1588 8 Filter out hosts with multiple bitcoin ports
512 512 512 512 8 Look up ASNs and limit results per ASN and per net
```
</details>
<details>
<summary>Before</summary>
```
IPv4 IPv6 Onion I2P CJDNS Pass
5772 1323 443 0 2 Initial
5772 1323 443 0 2 Skip entries with invalid address
4758 1110 443 0 2 After removing duplicates
4723 1094 443 0 2 Enforce minimal number of blocks
4723 1094 443 0 2 Require service bit 1
3732 867 443 0 2 Require minimum uptime
3718 864 443 0 2 Require a known and recent user agent
3718 864 443 0 2 Filter out hosts with multiple bitcoin ports
512 409 443 0 2 Look up ASNs and limit results per ASN and per net
```
</details>
### To dos
- [x] Remove manual nodes and update README
- [x] Mark nodes with connection times exceeding Bitcoin Core's default as bad in [exporter](https://github.com/virtu/seed-exporter): [done](https://github.com/virtu/seed-exporter/pull/12)
- [x] Regenerate mainnet seeds
- [x] Rebase, then remove WIP label once #30008 gets merged
ACKs for top commit:
achow101:
ACK b061b35105
fjahr:
utACK b061b35105
Tree-SHA512: 63e86220787251c7e8d2d5957bad69352e19ae17d7b9b2d27d8acddfec5bdafe588edb68d77d19c57f25f149de723e2eeadded0c8cf13eaca22dc33bd8cf92a0
1b41d45d46 wallet: bugfix: ensure atomicity in settings updates (ismaelsadeeq)
Pull request description:
This PR fixes#30620.
As outlined in the issue, creating two wallets with `load_on_startup=true` simultaneously results in only one wallet being added to the startup file.
The current issue arises because the wallet settings update process involves:
1. Obtaining the settings value while acquiring the settings lock.
2. Modifying the settings value.
3. Overwriting the settings value while acquiring the settings lock again.
This sequence is not thread-safe. Different threads could modify the same base value simultaneously, overwriting data from other workers without realizing it.
The PR attempts to fix this by modifying the chain interface's `updateRwSetting` method to accept a function that will be called with the settings reference. This function will either update or delete the setting and return an enum indicating whether the settings need to be overwritten in this or not.
Additionally, this PR introduces two new methods to the chain interface:
- `overwriteRwSetting`: This method replaces the setting with a new value.
Used in `VerifyWallets`
- `deleteRwSettings`: This method completely erases a specified setting.
This method is currently used only in `overwriteRwSetting`.
These changes ensure that updates are race-free across all clients.
ACKs for top commit:
achow101:
ACK 1b41d45d46
furszy:
self-code-ACK 1b41d45d46
Tree-SHA512: 50cda612b782aeb5e03e2cf63cc44779a013de1c535b883b57af4de22f24b0de80b4edecbcda235413baec0a12bdf0e5750fb6731c9e67d32e742d8c63f08c13
fa5b58ea01 test: Avoid intermittent block download timeout in p2p_ibd_stalling (MarcoFalke)
Pull request description:
Fixes#30704
The goal of the test is to check the stalling timeout, not the block download timeout.
On extremely slow hardware (for example qemu virtual hardware), downloading the 1023 blocks may take longer than the block download timeout.
Fix it by pinning the time using mocktime, and only advance it when testing the stalling timeout.
ACKs for top commit:
tdb3:
CR ACK fa5b58ea01
brunoerg:
utACK fa5b58ea01
Tree-SHA512: 9a9221f264bea52be5e9fe81fd319f5a6970cd315cc5e9f5e2e049c5d84619b19b9f6f075cda8d34565c2d6c17a88fb57e195c66c271e40f73119a77caecb6d7
Regenerate mainnet seeds from new sources without the need for hardcoded
data. Result has 512 nodes from each network type except cjdns, for
which only eight nodes were found that match the seed node criteria.
Pull additional nodes from virtu's crawler. Data includes sufficient
Onion and I2P nodes to align the uptime requirements for these networks
to that of clearnet nodes (i.e., 50%). Data also includes more than
three times the number of CJDNS nodes currently hardcoded into
nodes_main_manual.txt, so hardcoded nodes becomes obsolete.
41ad84a00c seeds: Use fjahr's more up to date asmap (Ava Chow)
d8fd1e0faf seeds: Fixed seeds update (Ava Chow)
f1f24d7214 seeds: Add testnet4 fixed seeds file (Ava Chow)
8ace71c737 seeds: Remove manual onion and i2p seeds (Ava Chow)
ed5b86cbe4 seeds: Add testnet instructions (Ava Chow)
0676515397 seeds: Also pull from achow101 seeder (Ava Chow)
5bab3175a6 makeseeds: Configurable minimum blocks for testnet4's smaller chain (Ava Chow)
d2465dfac6 makeseeds: Shuffle ips after parsing (Ava Chow)
af550b3a0f makeseeds: Support CJDNS (Ava Chow)
d5a8c4c4bd makeseeds: Update user agent regex (Ava Chow)
Pull request description:
The [DNS seeder](https://github.com/achow101/dnsseedrs) that I wrote collects statistics on node reliability in the same way that sipa's seeder does, and also outputs this information in the same file format. Thus it can also be used in our fixed seeds update scripts. My seeder additionally crawls onion v3, i2p, and cjdns, so will now be able to set those fixed seeds automatically rather than curating manual lists.
In doing this update, I've found that `makeseeds.py` is missing newer versions from the regex as well as cjdns support; both of these have been updated.
I also noticed that the testnet fixed seeds are all manually curated and sipa's seeder does not appear to publish any testnet data. Since I am also running the seeder for testnet, I've added the commands to generate testnet fixed seeds from my seeder's data too.
Lastly, I've updated all of the fixed seeds. However, since my seeder has not found any cjdns nodes that met the reliability criteria (possibly due to connectivity issues present in those networks), I've left the previous manual seeds for that network.
ACKs for top commit:
fjahr:
re-ACK 41ad84a00c
virtu:
ACK [41ad84a](41ad84a00c)
Tree-SHA512: 6ba0141f053d9d6ae7d8c9574f061be38f3e65b28de1d6660c1885ab942623b5a0ec70754b4fcfc5d98fe970f5f179a940d5880b5061ed698f7932500e01d3ee
5b4f34006d devtools, utxo-snapshot: Fix block height out of range (pablomartin4btc)
Pull request description:
<details>
<summary>Fixing a <a href="https://github.com/bitcoin/bitcoin/pull/28553#pullrequestreview-2251032570">bug</a> in <code>utxo_snapshot.sh</code>.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n):
Disabling network activity
false
error code: -8
error message:
Block height out of range
```
And the user will see the following in the node and it would stay there if not reset:
```
2024-08-21T14:44:13Z UpdateTip: new best=00000000000000afa0cd000a16e244f56032735d41acd32ac00337aceb2a5240 height=235382 version=0x00000002 log2_work=69.987697 tx=17492185 date='2013-05-09T23:54:32Z' progress=0.016219 cache=71.0MiB(571085txo)
2024-08-21T14:44:13Z UpdateTip: new best=0000000000000087c5e0b820afff496b95ba44ad64640c73b234d3261d3f99d2 height=235383 version=0x00000002 log2_work=69.987750 tx=17492341 date='2013-05-09T23:54:47Z' progress=0.016219 cache=71.0MiB(571291txo)
2024-08-21T14:44:13Z UpdateTip: new best=000000000000014a4b5fddf3c8abb6209247255ca9e8df786b271dd1b2ac82a6 height=235384 version=0x00000002 log2_work=69.987804 tx=17492344 date='2013-05-10T00:20:18Z' progress=0.016219 cache=71.0MiB(571297txo)
2024-08-21T14:44:13Z SetNetworkActive: false
```
</details>
This is a "temporary" fix until #29553 gets merged, which will remove the script entirely.
Handle the "Block height out of range" error gracefully by checking if the node has synchronized to or beyond the required block height, otherwise without this validation the node would keep the network disabled if the user selected that option.
<details>
<summary>Provide a user-friendly message if the block height is out of range and exit the script cleanly.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Error: The node has not yet synchronized to block height 840001.
Please wait until the node has synchronized past this block height and try again.
```
</details>
ACKs for top commit:
achow101:
ACK 5b4f34006d
fjahr:
tACK 5b4f34006d
Tree-SHA512: 2b71286b627872d7cfdb367e29361afa3806a7ef9d65075b93892b735ff2ab729069e2f7259d30262909e73cef17fb7dca231615cc1863968cd042f4a2a4f901
e1d5dd732d test: check xor.dat recreated when missing (tdb3)
d1610962bf test: add null block xor key (tdb3)
1ad999b9da refactor: lift NUM_XOR_BYTES (tdb3)
d8399584dd refactor: move read_xor_key() to TestNode (tdb3)
d43948c3ef refactor: use unlink rather than os.remove (tdb3)
c8176f758b test: add blocks_key_path (tdb3)
Pull request description:
Builds on PR #30657.
Refactors `read_xor_key()` from `util.py` to `test_node.py` (comment https://github.com/bitcoin/bitcoin/pull/30657#discussion_r1723358327)
Adds a check that `xor.dat` is created when missing (comment https://github.com/bitcoin/bitcoin/pull/30657#discussion_r1717724161)
Help states:
```
-blocksxor
Whether an XOR-key applies to blocksdir *.dat files. The created XOR-key
will be zeros for an existing blocksdir or when `-blocksxor=0` is
set, and random for a freshly initialized blocksdir. (default: 1)
```
ACKs for top commit:
maflcko:
ACK e1d5dd732d
achow101:
ACK e1d5dd732d
theStack:
re-ACK e1d5dd732d
brunoerg:
reACK e1d5dd732d
Tree-SHA512: 325912ef646ec88e0a58e1ece263a2b04cbc06497e8fe5fcd603e509e80c6bcf84b09dd52dfac60e23013f07fc2b2f6db851ed0598649c3593f452c4a1424bd9
- Settings updates were not thread-safe, as they were executed in
three separate steps:
1) Obtain settings value while acquiring the settings lock.
2) Modify settings value.
3) Overwrite settings value while acquiring the settings lock.
This approach allowed concurrent threads to modify the same base value
simultaneously, leading to data loss. When this occurred, the final
settings state would only reflect the changes from the last thread
that completed the operation, overwriting updates from other threads.
Fix this by making the settings update operation atomic.
- Add test coverage for this behavior.
Co-authored-by: furszy <matiasfurszyfer@protonmail.com>
60055f1abc test: replace deprecated secp256k1 context flags usage (Sebastian Falbesoner)
Pull request description:
The flags `SECP256K1_CONTEXT_{SIGN,VERIFY}` have been marked as deprecated since libsecp256k1 version 0.2 (released in December 2022), with the recommendation to use SECP256K1_CONTEXT_NONE instead, see https://github.com/bitcoin-core/secp256k1/pull/1126 and 1988855079/CHANGELOG.md (L132). Note that in contrast to other deprecated functions/variables, these defines don't have a deprecated attribute and hence don't lead to a compiler warning (see https://github.com/bitcoin-core/secp256k1/pull/1126#discussion_r922105271), so they are not easily detected.
ACKs for top commit:
TheCharlatan:
ACK 60055f1abc
ismaelsadeeq:
utACK 60055f1abc
tdb3:
light CR and test ACK 60055f1abc
Tree-SHA512: d93cf49e018a58469620c0d2f50242141f22dabc70afb2a7cd64e416f4f55588714510ae5a877376dd1e6b6f7494261969489af4b18a1c9dff0d0dfdf93f1fa8
fa5aeab3cb test: Avoid duplicate curl call in get_previous_releases.py (MarcoFalke)
Pull request description:
Seems odd having to translate `404` to "Binary tag was not found". Also, it seems odd to write a for-loop over a list with one item.
Fix both issues by just using a single call to `curl --fail ...`.
Can be tested with: `test/get_previous_releases.py -b v99.99.99`
Before:
```
Releases directory: releases
Fetching: https://bitcoincore.org/bin/bitcoin-core-99.99.99/bitcoin-99.99.99-x86_64-linux-gnu.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 286k 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
Binary tag was not found
```
After:
```
Releases directory: releases
Fetching: https://bitcoincore.org/bin/bitcoin-core-99.99.99/bitcoin-99.99.99-x86_64-linux-gnu.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 286k 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
curl: (22) The requested URL returned error: 404
ACKs for top commit:
fanquake:
ACK fa5aeab3cb
brunoerg:
utACK fa5aeab3cb
tdb3:
tested ACK fa5aeab3cb
Tree-SHA512: d5d31e0bccdd9de9b4a8ecf2e69348f4e8cee773050c8259b61db1ce5de73f6fbfffbe8c4d2571f7bef2de29cb42fd244573deebfbec614e487e76ef41681b9c
cccc5bfd35 test: Enable detect_leaks=1 in ASAN_OPTIONS explicitly (MarcoFalke)
Pull request description:
It should be enabled by default, but being explicit can't hurt.
ACKs for top commit:
fanquake:
ACK cccc5bfd35
Tree-SHA512: ed284abd05c7a99c30b509844aa75785a5ccb506d8296a71347b4c328750a6a4ed1f87e7a3ec36ab17f27b467c033cc8ca5eb5e2b951f2ae7473327c5eb1ddae
