Commit Graph

528 Commits

Author SHA1 Message Date
Hennadii Stepanov
1a0d8e178c
build: Re-enable external signer on Windows 2023-01-06 10:53:36 +00:00
MarcoFalke
bf3b589413
Merge bitcoin/bitcoin#26791: ci: Properly set COMMIT_RANGE in lint task
fa5cbf2290 ci: Properly set COMMIT_RANGE in lint task (MarcoFalke)

Pull request description:

  Currently the variable holds (apart from the commits in the pull request) all commits to master since the pull was opened.

  This is problematic, because already merged commits are linted in unrelated pulls, leading to:

  * Wasted resources. For example, currently the lint task may take 9 minutes, when it should take 1. See https://cirrus-ci.com/task/6032782770569216?logs=lint#L1449
  * False failures. For example, when a "wrong" commit is in master it can lead to some pulls failing unrelatedly, and others not.

  Now that the CI has the `/merge` commit (since commit fad7281d78), `COMMIT_RANGE` can simply be set to `HEAD~..HEAD` to only hold the changes in the pull.

ACKs for top commit:
  fanquake:
    ACK fa5cbf2290

Tree-SHA512: e85fca4ca9d2615ddd2544403485e06885769a3f70bca297e23eefda2a1d28f47c5271f6adfa6ce0e5e972335c78098b76e0db4b109f59d0986bf508cef7528f
2023-01-04 13:56:45 +01:00
Jon Atack
1e5e87cec3 script: update python linter dependencies 2023-01-03 11:05:09 -08:00
MarcoFalke
fa5cbf2290
ci: Properly set COMMIT_RANGE in lint task 2023-01-02 14:05:23 +01:00
MarcoFalke
d8bdee0fc8
Merge bitcoin/bitcoin#26775: ci: Revert tsan task changes
fabb6af850 ci: Remove duplicate CC and CXX from tsan task (MarcoFalke)
fa5d9a0e24 Revert "ci: Use clang-15 in tsan task" (MarcoFalke)
faa835e7e5 Revert "test: Drop no longer needed `race:epoll_ctl` TSan suppression" (MarcoFalke)

Pull request description:

  Looks like there are still bugs in clang-15, so we need to roll back all the way to the previously used version (clang-13).

ACKs for top commit:
  hebasto:
    ACK fabb6af850, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: d62203049847ab9095ee3fc89e18bdd721d1d9d5a7ef7a9f524c80e6be58d1d9f6aa2f14533df1ea77eb59597fba6fa9b987b17eb03b2c3f7cb577ab59cd59c0
2023-01-01 10:26:02 +01:00
MarcoFalke
fabb6af850
ci: Remove duplicate CC and CXX from tsan task 2022-12-30 09:50:22 +01:00
MarcoFalke
fa5d9a0e24
Revert "ci: Use clang-15 in tsan task"
This reverts commit faa00ca78e.
2022-12-30 09:49:51 +01:00
MarcoFalke
65de8eeeca
Merge bitcoin/bitcoin#26770: ci: Move --enable-c++20 from "tidy" task back to "ASan..." one
afc6052430 ci: Move `--enable-c++20` from "tidy" task back to "ASan..." one (Hennadii Stepanov)

Pull request description:

  This PR reverts cc7335edc8 from https://github.com/bitcoin/bitcoin/pull/25528 partially.

  C++20 has introduced some new headers, and it is premature to consider them when using the IWYU tool.

  Required for https://github.com/bitcoin/bitcoin/pull/26763 and https://github.com/bitcoin/bitcoin/pull/26766.

  Related discussions:
  - https://github.com/bitcoin/bitcoin/pull/25528#discussion_r1058906785
  - https://github.com/bitcoin/bitcoin/pull/26763#discussion_r1058860943

ACKs for top commit:
  MarcoFalke:
    review only ACK afc6052430

Tree-SHA512: 9c1d3317d0f7a94d46d1e442da1a91669cd9ed1e62a41579edc96e1f5447551b536d32eeb222caf277e85228482753be545e0a874208cb24f9a8491fce3b6d9f
2022-12-29 20:49:21 +01:00
Hennadii Stepanov
afc6052430
ci: Move --enable-c++20 from "tidy" task back to "ASan..." one
This change reverts cc7335edc8 partially.

C++20 has introduced some new headers, and it is premature to consider
them when using the IWYU tool.
2022-12-29 18:39:00 +00:00
MarcoFalke
faa00ca78e
ci: Use clang-15 in tsan task 2022-12-29 16:30:13 +01:00
Hennadii Stepanov
306ccd4927
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
- 2021: f47dda2c58
- 2020: fa0074e2d8
- 2019: aaaaad6ac9
2022-12-24 23:49:50 +00:00
fanquake
d3a84347e8
ci: remove --prefix from msan job 2022-12-20 17:17:35 +00:00
pasta
935acdcc79
refactor: modernize the implementation of uint256.*
- Constructors of uint256 to utilize Span instead of requiring a std::vector
- converts m_data into a std::array
- Prefers using `WIDTH` instead of `sizeof(m_data)`
- make all the things constexpr
- replace C style functions with c++ equivalents
    - memset -> std::fill
    - memcpy -> std::copy
        Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
    - memcmp -> std::memcmp
2022-12-10 14:34:44 -06:00
Hennadii Stepanov
574e50addf
ci: Use CONFIG_SITE variable and --prefix option properly
This change fixes scripts when they are being run locally with a pre-
existed `$DEPENDS_DIR/$HOST` directory.
2022-12-10 19:13:20 +00:00
MarcoFalke
9e59d21fbe
Merge bitcoin/bitcoin#26592: ci: only run USDT interface tests on CirrusCI
2811f40f30 ci: only run USDT interface tests on CirrusCI (0xb10c)

Pull request description:

  As mentioned in #26571, the task running the USDT interface tests fail when run in docker. cc7335edc8 in #25528 added that the tests are run in a **VM** in Cirrus CI. Running them locally in docker containers might not work:

  - We use [bcc] as tracing toolkit which requires the kernel headers to compile the BPF bytecode. As docker containers use the hosts kernel and don't run their own, there is a potential for mismatches between kernel headers available in the container and the host kernel. This results in a failure loading the BPF byte code.
  - Privilges are required to load the BPF byte code into the kernel. Normally, the docker containers aren't run with these.
  - We currently use an untrusted third-party PPA to install the bpfcc-tools package on Ubuntu 22.04. Using this on a local dev system could be a security risk.

  To not hinder the ASan + LSan + UBSan part of the CI task, the USDT tests are disabled on non-CirrusCI runs.

  [bcc]: https://github.com/iovisor/bcc

Top commit has no ACKs.

Tree-SHA512: 7c159b59630b36d5ed927b501fa0ad6f54ff3d98d0902f975cc8122b4147a7f828175807d40a470a85f0f3b6eeb79307d3465a287dbd2f3d75b803769ad0b6e7
2022-12-02 10:52:52 +01:00
0xb10c
2811f40f30
ci: only run USDT interface tests on CirrusCI
As mentioned in #26571, the task running the USDT interface tests
fail when run in docker. cc7335edc8
in #25528 added that the tests are run in a **VM** in Cirrus CI.
Running them locally in docker containers might not work:

- We use [bcc] as tracing toolkit which requires the kernel headers
  to compile the BPF bytecode. As docker containers use the hosts
  kernel and don't run their own, there is a potential for mismatches
  between kernel headers available in the container and the host
  kernel. This results in a failure loading the BPF byte code.
- Privilges are required to load the BPF byte code into the kernel.
  Normally, the docker containers aren't run with these.
- We currently use an untrusted third-party PPA to install the
  bpfcc-tools package on Ubuntu 22.04. Using this on a local dev
  system could be a security risk.

To not hinder the ASan + LSan + UBSan part of the CI task, the USDT
tests are disabled on non-CirrusCI runs.

[bcc]: https://github.com/iovisor/bcc
2022-11-28 21:26:26 +01:00
MarcoFalke
d415b7261c
Merge bitcoin/bitcoin#26588: ci: Skip COMMIT_RANGE if no CIRRUS_PR
fad1c55301 lint: Skip COMMIT_RANGE if no CIRRUS_PR (MarcoFalke)

Pull request description:

  It doesn't make sense to run this for non-PRs, because:

  * There are known whitespace "violations" in previous commits, so the lint may fail
  * Once the changes are merged, it is too late to fix them up (force pushes are illegal)
  * It isn't possible to determine which commits to run on if there is no reference branch (target branch of the pull request)

  Moreover, the test fails on non-master:

  * https://github.com/bitcoin/bitcoin/runs/8664441400

  Fix all issues by skipping it.

ACKs for top commit:
  hebasto:
    ACK fad1c55301, also tested in my personal Cirrus account.

Tree-SHA512: be15f00e2b2a9069583833545883e0e5968a33d2455dad59e6fb47c1102b4dd16ef932e9ba945e29e9d941e6c17bd531a02c66b0491097801be6bda476875537
2022-11-28 17:18:29 +01:00
MarcoFalke
fad1c55301
lint: Skip COMMIT_RANGE if no CIRRUS_PR 2022-11-28 11:09:30 +01:00
MarcoFalke
c239d3dac9
Merge bitcoin/bitcoin#26574: ci: use ci_exec_root for clang install
54dd8f51ce ci: use ci_exec_root for clang install (josibake)

Pull request description:

  fixes a bug introduced in #25900 ; see https://github.com/bitcoin/bitcoin/pull/25900#issuecomment-1327311069

  the general idea of #25900 was to use a non-root user as much as possible to avoid modifying the user's local filesystem. however, it appears the root user is needed to correctly install clang.

ACKs for top commit:
  hebasto:
    ACK 54dd8f51ce, tested on Ubuntu 22.04.

Tree-SHA512: beb01d4b6127fbba3c8d18e85cf7ec7d1b2ec93ea05c475ab51bcaa04ef1b0591d886f1a7e0732c5ae86806013f022c0b44027380d2b0cfb1bfdc843e40f99b4
2022-11-28 10:20:21 +01:00
josibake
54dd8f51ce
ci: use ci_exec_root for clang install 2022-11-25 14:13:29 +01:00
MarcoFalke
fa3b2cf277
fuzz: Move-only net utils 2022-11-23 17:26:01 +01:00
MarcoFalke
85892f77c9
Merge bitcoin/bitcoin#25900: ci: run docker wrapper with a non-root user
849f20a6d3 ci: create and use non-root user for docker image (josibake)

Pull request description:

  Previously, everything in the ci docker image ran as the root user. This would lead to certain directories (`ci/scratch`, `depends`) being owned by `root` after running the ci locally which would lead to annoying behavior such as subsequent guix builds failing due to `depends/` being owned by root.

  This PR adds a non-root user in the container and chowns the mounted working directory. All the `docker exec` commands now run as the non-root user, except for the few that still need to run as root (mainly, installing packages).

  To test this I checked out a fresh copy of the repo, applied my changes, ran the CI, and verified all the local file permissions were unchanged after the CI was finished running.

ACKs for top commit:
  hebasto:
    ACK 849f20a6d3, tested on Ubuntu 22.04 by running commands as follows:

Tree-SHA512: 734dca0f36157fce5fab243b4ff657fc17ba980e8e4e4644305f41002ff21bd5cef02c306ea1e0b5c841d4c07c095e8e4be16722e6a38c890717c60a3f5ec62a
2022-11-22 12:46:40 +01:00
fanquake
1b680948d4
Merge bitcoin/bitcoin#26292: util: move threadinterrupt into util/
b89530483d util: move threadinterrupt into util (fanquake)

Pull request description:

  Alongside thread and threadnames. It's part of libbitcoin_util.

ACKs for top commit:
  ryanofsky:
    Code review ACK b89530483d. No changes since last review other than rebase
  theuni:
    ACK b89530483d.

Tree-SHA512: 0421f4d1881ec295272446804b27d16bf63e6b62b272f8bb52bfecde9ae6605e8109ed16294690d3e3ce4b15cc5e7c4046f99442df73adb10bdf069d3fb165aa
2022-11-22 09:52:53 +00:00
josibake
849f20a6d3
ci: create and use non-root user for docker image
Running all commands as the root user in the docker image
will change local file permissions in the ci and depends directory.

Add a non-root user to the container and use this user whenever
possible when running docker exec commands.
2022-11-21 18:11:28 +01:00
MacroFake
2222ec71fd
util: Move error message formatting of NonFatalCheckError to cpp
This allows to strip down the header file
2022-11-16 12:21:33 +01:00
fanquake
b89530483d
util: move threadinterrupt into util 2022-11-01 10:14:49 +00:00
fanquake
3a0b352c63
refactor: move url.h/cpp from lib util to lib common 2022-10-31 10:17:04 +00:00
MacroFake
bd478890c5
Merge bitcoin/bitcoin#26388: ci: Use macos-ventura-xcode:14.1 image for "macOS native" task
da16893474 ci: Use `macos-ventura-xcode:14.1` image for "macOS native" task (Hennadii Stepanov)
702836530f ci: Make `getopt` path architecture agnostic (Hennadii Stepanov)

Pull request description:

  The "macOS native" CI task always uses the recent OS image.

  This PR updates it up to the recent macOS release.

  Cirrus Labs [stopped](https://github.com/bitcoin/bitcoin/pull/25160#issuecomment-1162829773) updating macOS images for `x86_64`, therefore, an `arm64` image been used.

  Also `make test-security-check` has been dropped as it ["isn't even expected to pass"](https://github.com/bitcoin/bitcoin/issues/26386#issuecomment-1290318628) on `arm64` in CI.

ACKs for top commit:
  Sjors:
    utACK da16893

Tree-SHA512: 36785d33b7f11b3cdbc53bcfbf97d88bf821fad248c825982dd9f8e3413809a4ef11190eaf950e60fdf479b62ff66920c35d9ea42d534723f015742eec7e19b6
2022-10-27 16:15:20 +02:00
Hennadii Stepanov
da16893474
ci: Use macos-ventura-xcode:14.1 image for "macOS native" task 2022-10-25 13:39:03 +01:00
Hennadii Stepanov
702836530f
ci: Make getopt path architecture agnostic 2022-10-25 09:49:07 +01:00
MacroFake
aaaa7bd0ba
iwyu: Add missing includes 2022-10-18 14:12:52 +02:00
MacroFake
9ca39d69df
Merge bitcoin/bitcoin#26254: iwyu: Add zmq source files
13afcc0cd4 iwyu: Add zmq source files (Hennadii Stepanov)

Pull request description:

ACKs for top commit:
  fanquake:
    ACK 13afcc0cd4

Tree-SHA512: 7af95e991fc2782aeba2edfef0a2f75f9c361058295586adb062087aa31c47cfcce2425aee9dd5153e18e018cf1f9272c9617c671b7262db55f241526c3fcb15
2022-10-10 18:08:45 +02:00
Hennadii Stepanov
13afcc0cd4
iwyu: Add zmq source files 2022-10-10 15:44:02 +01:00
MacroFake
fa04376554
Remove clang-format from lint task
clang-format could be used in scripted diffs, but remained largely
unused.
2022-10-05 10:52:42 +02:00
fanquake
a23f8c8978
Merge bitcoin/bitcoin#26234: ci: Allow PIP_PACKAGES on centos
fa6054e952 ci: Allow PIP_PACKAGES on centos (MacroFake)
fac085a05c ci: Remove unused package (MacroFake)

Pull request description:

  This was added in 7fc5e865b9 but I can't see a reason why this should be forbidden.

  This is also needed for other changes (bumping the minimum python version).

ACKs for top commit:
  hebasto:
    re-ACK fa6054e952

Tree-SHA512: e8ead9ee00079024eb1e8c6e7b31c78cf2a3392159b444765c2ea9a58bed2a7550bf71083210692a45bb8ed7896cb882b72bf70baa13a2384864b2b510b73005
2022-10-04 21:39:29 +01:00
MacroFake
fa6054e952
ci: Allow PIP_PACKAGES on centos
This was added in 7fc5e865b9 but I can't
see a reason why this should be forbidden.
2022-10-04 11:53:31 +02:00
MacroFake
fac085a05c
ci: Remove unused package
Address feedback from https://github.com/bitcoin/bitcoin/pull/24561/files#r985719812
2022-10-04 11:51:33 +02:00
Hennadii Stepanov
ac1d99240a
ci: Move git config commands into script where they are used 2022-10-04 08:51:55 +01:00
MacroFake
d8ded8bc08 ci: Use git2.34 for lint task 2022-09-28 19:58:13 +02:00
Jon Atack
8f2010de6e Bump codespell version to 2.2.1 2022-09-15 12:53:32 +02:00
MacroFake
fa875349e2
Fix iwyu 2022-08-20 09:33:01 +02:00
MacroFake
7d82f86341
Merge bitcoin/bitcoin#25650: script: default to necessary tags in test/get_previous_releases.py
21a9e94dbb ci: remove hardcoded tag list from ci scripts (josibake)
d530ba390e doc: update test/README.md (josibake)
614d4682ba script: default to necessary tags in get_previous_releases.py (josibake)

Pull request description:

  Almost every time I need to use this script, I forget the tag list is needed and that a specific set of tags is needed for the backwards compatibility tests to work. I end up wasting time reading through the script and googling to find the tag list before remembering it is in `test/README.md`

  I assume (hope) I'm not the only one this happens to, so I figured it would make more sense to have the script default to downloading/building the necessary tags. This has the added benefit of making the script the source of truth: the script already needs to be updated with the SHA256_SUM of the binary for every new tag that is added, so it makes sense to use `SHA256_SUMS` list as the necessary tag list. This means there is less risk of the README and the script drifting (i.e updating the readme with a new tag and forgetting to update the script, or updating the script and forgetting to update the README). Now all that needs to happen is to update the `SHA256_SUMS` list in the script and everything Just Works (TM)

ACKs for top commit:
  Sjors:
    re-tACK 21a9e94dbb

Tree-SHA512: 97b488227a89a6827584edd251820a7074fad75dfd7f26f1aa5f858e2521d2e02effd0f11e6dc4676e1155d3d5aba6ff94a4b58ffef80dc201376afd5927deb9
2022-08-05 10:51:06 +02:00
Carl Dong
41c5201a90 validationcaches: Add and use ValidationCacheSizes
Also:

- Make DEFAULT_MAX_SIG_CACHE_SIZE into constexpr
  DEFAULT_MAX_SIG_CACHE_BYTES to utilize the compile-time integer
  arithmetic overflow checking available to constexpr.
- Fix comment (MiB instead of MB) for DEFAULT_MAX_SIG_CACHE_BYTES.
- Pass in max_size_bytes parameter to InitS*Cache(), modify log line to
  no longer allude to maxsigcachesize being split evenly between the two
  validation caches.
- Fix possible integer truncation and add a comment.

[META] I've kept the integer types as int64_t in order to not introduce
       unintended behaviour changes, in the next commit we will make
       them size_t.
2022-08-03 12:03:27 -04:00
MacroFake
ddddd6913b
sort after scripted-diff 2022-08-02 15:31:05 +02:00
MacroFake
fac812ca83
scripted-diff: Move mempool_args to src/node
It is part of the node library. Also, it won't be moved to the kernel
lib, as it will be pruned of ArgsManager.

-BEGIN VERIFY SCRIPT-
 # Move module
 git mv src/mempool_args.cpp src/node/
 git mv src/mempool_args.h   src/node/
 # Replacements
 sed -i 's:mempool_args\.h:node/mempool_args.h:g'     $(git grep -l mempool_args)
 sed -i 's:mempool_args\.cpp:node/mempool_args.cpp:g' $(git grep -l mempool_args)
 sed -i 's:MEMPOOL_ARGS_H:NODE_MEMPOOL_ARGS_H:g'      $(git grep -l MEMPOOL_ARGS_H)
-END VERIFY SCRIPT-
2022-08-02 15:31:01 +02:00
MacroFake
fa468bdfb6
Return optional error from ApplyArgsManOptions
Also pass in a (for now unused) reference to the params.

Both changes are needed for the next commit.
2022-08-02 15:21:50 +02:00
MacroFake
eeb5a94e27
Merge bitcoin/bitcoin#25528: ci: run USDT interface tests in the CI
cc7335edc8 ci: run USDT interface test in a VM (0xb10c)
dba6f82342 test: adopt USDT utxocache interface tests (0xb10c)
220a5a2841 test: hook into PID in tracing tests (0xb10c)

Pull request description:

  Changes a CI task that runs test the previously not run `test/functional/interface_usdt_*.py` functional tests (added in https://github.com/bitcoin/bitcoin/pull/24358).

  This task is run as CirussCI `compute_engine_instance` VM as hooking into the tracepoints is not possible in CirrusCI docker containers (https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845). We use an unoffical PPA and untrusted  `bpfcc-tools` package in the CI as the Ubuntu jammy and Debian bullseye packages are outdated. We hope use an official package when new Ubuntu/Debian releases are available for the use with Google Compute Engine.

  We make sure to hook into `bitcoind` binaries in USDT interface tests via their PID, instead of their path. This makes sure multiple functional tests running in parallel don't interfere with each other.

  The utxocache USDT interface tests is adopted to a change of the functional test framework that wasn't detected as the tests weren't run in the CI. As the tracepoints expose internals, it can happen that we need to adopt the interface test when internals change. This is a bit awkward, and if it happens to frequently, we should consider generalizing the tests a bit more. For now it's fine, I think.

  See the individual commit messages for more details on the changes.

  Fixes https://github.com/bitcoin/bitcoin/issues/24782
  Fixes https://github.com/bitcoin/bitcoin/issues/23296

  I'd like to hear from reviewers:
  - Are we OK with using the [`hadret/bpfcc`](https://launchpad.net/~hadret/+archive/ubuntu/bpfcc) PPA for now? There is a clear plan when to drop it and as is currently, it could only impact the newly added VM task.
  - ~~Adding a new task increases CI runtime and costs. Should an existing `container` CI task be ported to a VM and reused instead?~~ Yes, see https://github.com/bitcoin/bitcoin/pull/25528#issuecomment-1179509525

ACKs for top commit:
  MarcoFalke:
    cr ACK cc7335edc8

Tree-SHA512: b7fddccc0a77d82371229d048abe0bf2c4ccaa45906497ef3040cf99e7f05561890aef4c253c40e4afc96bb838c9787fae81c8454c6fd9db583276e005a4ccb3
2022-08-01 11:27:29 +02:00
fanquake
3fe58a98d3
tidy: run clang-tidy in quiet mode 2022-07-29 16:12:45 +01:00
fanquake
5671217477
Merge bitcoin/bitcoin#24974: refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono)
fa74e726c4 refactor: Make FEELER_SLEEP_WINDOW type safe (std::chrono) (MacroFake)
fa3b3cb9b5 Expose underlying clock in CThreadInterrupt (MacroFake)

Pull request description:

  This gets rid of the `value*1000` manual conversion.

ACKs for top commit:
  naumenkogs:
    utACK fa74e726c4
  dergoegge:
    Code review ACK fa74e726c4

Tree-SHA512: 90409c05c25f0dd2f1c4dead78f707ebfd78b7d84ea4db9fcefd9c4958a1a3338ac657cd9e99eb8b47d52d4485fa3c947dce4ee1559fb56ae65878685e1ed9a3
2022-07-26 15:09:21 +01:00
MacroFake
fa2247a9f9
refactor: Make CTransaction constructor explicit
It involves calculating two hashes, so the performance impact should be
made explicit.

Also, add the module to iwyu.
2022-07-25 12:16:54 +02:00
josibake
21a9e94dbb
ci: remove hardcoded tag list from ci scripts 2022-07-21 12:02:08 +02:00
Hennadii Stepanov
6d58117a31
build: Build minisketch test in make check, not in make 2022-07-21 10:42:09 +01:00
fanquake
cc5b39e44e
ci: better pin to dwarf4 in valgrind job
Use `-gdwarf` and also set CFLAGS. I was seeing Valgrind issues otherwise.
2022-07-21 10:16:46 +01:00
Ryan Ofsky
1e761a0169 ci: Enable IWYU in src/kernel directory
Suggested https://github.com/bitcoin/bitcoin/pull/25308#discussion_r892505713
2022-07-19 16:54:52 -04:00
MacroFake
1111ddeedf
Remove unused includes from dbwrapper.h 2022-07-19 14:32:53 +02:00
MacroFake
fa869ce2c2
Add missing includes to node/chainstate
This is needed for the next commit
2022-07-19 14:12:14 +02:00
glozow
821f5c824f
Merge bitcoin/bitcoin#25487: [kernel 3b/n] Decouple {Dump,Load}Mempool from ArgsManager
cb3e9a1e3f Move {Load,Dump}Mempool to kernel namespace (Carl Dong)
aa30676541 Move DEFAULT_PERSIST_MEMPOOL out of libbitcoinkernel (Carl Dong)
06b88ffb8a LoadMempool: Pass in load_path, stop using gArgs (Carl Dong)
b857ac60d9 test/fuzz: Invoke LoadMempool via CChainState (Carl Dong)
b3267258b0 Move FopenFn to fsbridge namespace (Carl Dong)
ae1e8e3756 mempool: Use NodeClock+friends for LoadMempool (Carl Dong)
f9e8e5719f mempool: Improve comments for [GS]etLoadTried (Carl Dong)
813962da0b scripted-diff: Rename m_is_loaded -> m_load_tried (Carl Dong)
413f4bb52b DumpMempool: Pass in dump_path, stop using gArgs (Carl Dong)
bd4407817e DumpMempool: Use std::chrono instead of weird int64_t arthmetics (Carl Dong)
c84390b741 test/mempool_persist: Test manual savemempool when -persistmempool=0 (Carl Dong)

Pull request description:

  This is part of the `libbitcoinkernel` project: #24303, https://github.com/bitcoin/bitcoin/projects/18

  -----

  This PR moves `{Dump,Load}Mempool` into its own `kernel/mempool_persist` module and introduces `ArgsManager` `node::` helpers in `node/mempool_persist_args`to remove the scattered calls to `GetBoolArg("-persistmempool", DEFAULT_PERSIST_MEMPOOL)`.

  More context can be gleaned from the commit messages.

  -----

  One thing I was reflecting on as I wrote this was that in the long run, I think we should probably invert the validation <-> mempool relationship. Instead of mempool not depending on validation, it might make more sense to have validation not depend on mempool. Not super urgent since `libbitcoinkernel` will include both validation and mempool, but perhaps something for the future.

ACKs for top commit:
  glozow:
    re ACK cb3e9a1e3f via `git range-diff 7ae032e...cb3e9a1`
  MarcoFalke:
    ACK cb3e9a1e3f 🔒
  ryanofsky:
    Code review ACK cb3e9a1e3f

Tree-SHA512: 979d7237c3abb5a1dd9b5ad3dbf3b954f906a6d8320ed7b923557f41a4472deccae3e8a6bca0018c8e7a3c4a93afecc502acd1e26756f2054f157f1c0edd939d
2022-07-18 16:09:27 +01:00
fanquake
70d807c355
refactor: integrate no_nul into univalue unitester 2022-07-18 09:15:18 +01:00
Carl Dong
cb3e9a1e3f Move {Load,Dump}Mempool to kernel namespace
Also:
1. Add the newly introduced kernel/mempool_persist.cpp to IWYU CI script
2. Add chrono mapping for iwyu
2022-07-15 12:26:20 -04:00
MacroFake
fa3b3cb9b5
Expose underlying clock in CThreadInterrupt
Overloading sleep_for is not needed, as

* seconds and minutes can be converted to milliseconds by the compiler,
  not needing a duration_cast
* std::condition_variable::wait_for will convert milliseconds to the
  duration type of the underlying clock

So simply expose the clock.
2022-07-13 15:20:49 +02:00
0xb10c
cc7335edc8
ci: run USDT interface test in a VM
Our CI tasks are run by CirrusCI in Docker containers in a Google
Compute Engine based Kubernetes environment. These containers have
limited capabilities - especially CAP_SYS_ADMIN is missing. See
https://github.com/bitcoin/bitcoin/issues/23296#issuecomment-1024920845

We need elevated privileges to hook into the USDT tracepoints. We use a
CirrusCI "compute_engine_instance" (a VM, not a container) where we have
the required privileges. The ubunut-mininmal-2204-lts was choosen with
debian-11 being an alternative. Both pack an outdated 'bpfcc-tools'
package (v0.18.0) from 2020. This version prints warnings to stderr
during BPF bytecode compilation, which causes our functional test runner
to fail. This is fixed in newer verison.

Until debian-12 or a newer Ubuntu release is avaliable as image in GCE
(https://cloud.google.com/compute/docs/images/os-details), we use a
third-party and untrusted PPA that releases up-to-date versions of the
package.

The official iovisor (authors of BCC) PPA is outdated too. An
alternative would be to compile BCC from source in the CI.

Co-authored-by: MacroFake <falke.marco@gmail.com>
2022-07-08 19:41:58 +02:00
fanquake
07f2c25d04
refactor: add most of src/util to iwyu
These files change infrequently, and not much header shuffling is required.

We don't add everything in src/util/ yet, because IWYU makes some
dubious suggestions, which I'm going to follow up with upstream.
2022-07-08 11:06:01 +01:00
MacroFake
bac52a1cfe
Merge bitcoin/bitcoin#25447: fuzz: add low-level target for txorphanage
6eb0909cb7 fuzz: add low-level target for txorphanage (chinggg)

Pull request description:

  This adds a low-level fuzz target for orphan transaction handling by creating random transactions and calling all functions in `TxOrphanage`.
  It cannot simulate real-world `orphan/unorphan` scenarios effectively since it does not maintain any state about the node and the chain. A high-level fuzz target which construct well-designed transaction graphs will be added later.

ACKs for top commit:
  MarcoFalke:
    review ACK 6eb0909cb7 🐈

Tree-SHA512: b4d64f5941df77d13981f75ec170cef6ffabe782797c982ede7f34134be01dc0026dd7c0bee614bc1d64715e90a933d2a8c95974d402e32eaba8e24cc928299e
2022-07-07 14:03:52 +02:00
chinggg
6eb0909cb7 fuzz: add low-level target for txorphanage 2022-07-06 22:13:54 +08:00
sogoagain
31346a3196 [ci] apply cache size limit and print ccache statistics in "ARM64 Android APK" 2022-07-04 22:18:41 +09:00
Hennadii Stepanov
0bb7a1f71d
ci: Improve naming related to "macOS 12 native x86_64" task 2022-06-22 10:15:16 +02:00
fanquake
d873ff96e5
refactor: cleanups post unsubtree'ing univalue
Mostly changes to remove src/univalue exceptions from the various linters,
and the required code changes to make them happy. As well as minor doc
changes.
2022-06-15 12:56:44 +01:00
MacroFake
2ab4a80480
Merge bitcoin/bitcoin#25254: Move minRelayTxFee to policy/settings
fa4068b4e2 Move minRelayTxFee to policy/settings (MacroFake)

Pull request description:

  Seems a bit confusing to put policy stuff into validation, so fix that.

  Also fix includes via `iwyu`.

ACKs for top commit:
  ariard:
    ACK fa4068b, the includes move compiles well locally.
  ryanofsky:
    Code review ACK fa4068b4e2. Make sense to move the global variable to policy/settings and the default constant to policy/policy. Ariard points out other constants that could be moved, which seems fine, but it seems like moving the global variable to be with other related global variables is more significant.

Tree-SHA512: adf9619002610d1877f3aef0a9e6115fc4c2ad64135a3e5100824c650b560c47f47ac28894c6214a50a7888355252a9f6f7cec98c23a771a1964160ef1ca77de
2022-06-07 11:31:10 +02:00
MacroFake
f66633d9cb
Merge bitcoin/bitcoin#25288: test: Reliably don't start itself (lint-all.py runs all tests twice)
f26a496dfd test: clean up all-lint.py (Martin Leitner-Ankerl)
64d72c4c87 test: rename lint-all.py to all-lint.py (Martin Leitner-Ankerl)

Pull request description:

  When running `./test/lint/lint-all.py`, the script runs all tests but
  also calls itself because the comparison with `__file__` doesn't work.

  Comparing resolved paths gives reliable comparison, and lint-all.py doesn't call itself any more

ACKs for top commit:
  laanwj:
    Code review ACK f26a496dfd

Tree-SHA512: b44abdd685f7b48a6a9f48e96d97138b635c31c1c7ab543cb5636b5f49690ccd56fa6fec01ae7fcc16af01a613372ee77632f70c32059919b373aa8051953791
2022-06-07 10:37:34 +02:00
Martin Leitner-Ankerl
64d72c4c87 test: rename lint-all.py to all-lint.py
That way it is impossible for the script to call itself.
2022-06-07 10:22:45 +02:00
Hennadii Stepanov
c47944f4e9
ci: Reuse some configure options in "ARM64 Android APK" task 2022-05-31 18:49:53 +02:00
Hennadii Stepanov
7739438811
ci, android: Update NDK up to r23c 2022-05-31 18:45:13 +02:00
Hennadii Stepanov
ca0c3e5077
ci, android: Update Command-line Tools from 2.1 up to 7.0 2022-05-31 18:43:11 +02:00
Hennadii Stepanov
8790da3c1e
ci: Drop unneeded packages in "ARM64 Android APK" task 2022-05-31 15:48:25 +02:00
MacroFake
fa4068b4e2
Move minRelayTxFee to policy/settings
Also fix includes using iwyu
2022-05-31 15:05:57 +02:00
Hennadii Stepanov
3ffc190321
ci: Install documented packages for "Win64" CI task 2022-05-28 11:34:11 +02:00
Jon Atack
345647c4da ci: add DEBUG_LOCKCONTENTION to CI task containing DEBUG_LOCKORDER 2022-05-20 12:48:17 +02:00
Sjors Provoost
dba1231672
test: previous releases: add v23.0
Starting from v23.0 there is a separate macOS release for x86_64 and aarch64.

Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
2022-05-06 10:00:47 +02:00
MacroFake
2c56404088
Merge bitcoin/bitcoin#25029: rpc: Move fee estimation RPCs to separate file
fa753abd7c rpc: Move fee estimation RPCs to separate file (MacroFake)

Pull request description:

  Fee estimation is generally used by wallets when creating txs. It doesn't have anything to do with creating or submitting blocks.

ACKs for top commit:
  pk-b2:
    ACK fa753abd7c
  brunoerg:
    crACK fa753abd7c

Tree-SHA512: 81e0edc936198a0baf0f5bfa8cfedc12db51759c7873bb0082dfc5f0040d7f275b35f639c6f5b86fa1ea03397b0d5e757c2ce1b6b16f1029880a39b9c3aaceda
2022-05-03 08:17:50 +02:00
MacroFake
fa847ed2f6
ci: Clone iwyu only if missing 2022-04-30 09:23:56 +02:00
MacroFake
fa753abd7c
rpc: Move fee estimation RPCs to separate file
Can be reviewed with --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2022-04-29 16:25:06 +02:00
fanquake
91ac12be44
Merge bitcoin/bitcoin#25013: Remove cs_main from verifymessage, move msg utils to new file
fa60169811 rpc: Move signmessage RPC util to new file (MacroFake)
fa9425177e Remove cs_main from verifymessage (MacroFake)

Pull request description:

  The `verifymessage` RPC has several issues:

  * It takes `cs_main` for no reason, blocking progress on removing the `cs_main` global mutex.
  * It is located in a file called `misc`, which is not a very helpful name.

  Fix all issues.

ACKs for top commit:
  vincenzopalazzo:
    ACK fa60169811

Tree-SHA512: c71a1f481b828e0a544405fecbbc7ca44e66ea46b498d7aed1f1c584d6a99724deb13e89d90b9d5cdeecbce293e6a41e9f7ae299543f6d761bf9e7a839b6c7f3
2022-04-29 12:20:10 +01:00
fanquake
dd17c42a16
Merge bitcoin/bitcoin#24322: [kernel 1/n] Introduce initial libbitcoinkernel
035fa1f07a build: Remove LIBTOOL_APP_LDFLAGS for bitcoin-chainstate (Cory Fields)
3f0595095d docs: Add libbitcoinkernel_la_SOURCES explanation (Carl Dong)
94ad45deb2 ci: Build libbitcoinkernel (Carl Dong)
26b2e7ffb3 build: Extract the libbitcoinkernel library (Carl Dong)
1df44dd20c b-cs: Define G_TRANSLATION_FUN in bitcoinkernel.cpp (Carl Dong)
83a0bb7cc9 build: Separate lib_LTLIBRARIES initialization (Carl Dong)
c1e16cb31f build: Create .la library for bitcoincrypto (Carl Dong)
8bdfe057c7 build: Create .la library for leveldb (Carl Dong)
05d1525b6d build: Create .la library for crc32c (Carl Dong)
64caf94479 build: Remove vestigial LIBLEVELDB_SSE42 (Carl Dong)
1392e8e2d8 build: Don't add unrelated libs to LIBTEST_* (Carl Dong)

Pull request description:

  Part of: #24303

  This PR introduces a `libbitcoinkernel` static library linking in the minimal list of files necessary to use our consensus engine as-is. `bitcoin-chainstate` introduced in #24304 now will link against `libbitcoinkernel`.

  Most of the changes are related to the build system.

  Please read the commit messages for more details.

ACKs for top commit:
  theuni:
    This may be my favorite PR ever. It's a privilege to ACK 035fa1f07a.

Tree-SHA512: b755edc3471c7c1098847e9b16ab182a6abb7582563d9da516de376a770ac7543c6fdb24238ddd4d3d2d458f905a0c0614b8667aab182aa7e6b80c1cca7090bc
2022-04-28 15:14:32 +01:00
laanwj
85aea18ae6
Merge bitcoin/bitcoin#24982: tests: Port lint-all.sh to lint-all.py
29f44fed36 Converting `lint-all.sh` to `lint-all.py`. (hiago)

Pull request description:

  This PR is converting `test/lint/lint-all.sh` to `test/lint/lint-assertions.py`. It's an item of https://github.com/bitcoin/bitcoin/issues/24783.

ACKs for top commit:
  laanwj:
    Tested ACK 29f44fed36

Tree-SHA512: 5427936aaa8e009613048448cbd79d9225675bdcadcf4fbb70fd091e0aab85e350ef1678da1b388f70d62ca16f40409409f90da3f6bbf057480522cd50fde811
2022-04-28 12:28:30 +02:00
MacroFake
fa60169811
rpc: Move signmessage RPC util to new file
Can be reviewed with --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2022-04-28 11:19:29 +02:00
MacroFake
9446de160f
Merge bitcoin/bitcoin#24831: tidy: add include-what-you-use
9b0a13a289 tidy: Add include-what-you-use (fanquake)
74cd038e30 refactor: fix includes in src/init (fanquake)
c79ad935f0 refactor: fix includes in src/compat (fanquake)

Pull request description:

  We recently added a [`clang-tidy` job](https://github.com/bitcoin/bitcoin/blob/master/ci/test/00_setup_env_native_tidy.sh) to the CI, which generates a compilation database. We can leverage that now existing database to begin running [include-what-you-use](https://include-what-you-use.org/) over the codebase.

  This PR demonstrates using a mapping_file to indicate fixups / includes that may differ from IWYU suggestions. In this case, I've added some fixups for glibc includes that I've [upstreamed changes for](https://github.com/include-what-you-use/include-what-you-use/pull/1026):
  ```bash
  # Fixups / upstreamed changes
  [
    { include: [ "<bits/termios-c_lflag.h>", private, "<termios.h>", public ] },
    { include: [ "<bits/termios-struct.h>", private, "<termios.h>", public ] },
    { include: [ "<bits/termios-tcflow.h>", private, "<termios.h>", public ] },
  ]
  ```

  The include "fixing" commits of this PR:
  * Adds missing includes.
  * Swaps C headers for their C++ counterparts.
  * Removes the pointless / unmaintainable `//for abc, xyz` comments. When using IWYU, if anyone wants to see / generate those comments, to see why something is included, it is trivial to do so (IWYU outputs them by default). i.e:
  ```cpp
  // The full include-list for compat/stdin.cpp:
  #include <compat/stdin.h>
  #include <poll.h>                  // for poll, pollfd, POLLIN
  #include <termios.h>               // for tcgetattr, tcsetattr
  #include <unistd.h>                // for isatty, STDIN_FILENO
  ```

  TODO:
  - [ ] Qt mapping_file. There is one in the IWYU repo, but it's for Qt 5.11. Needs testing.
  - [ ] Boost mapping_file. There is one in the IWYU repo, but it's for Boost 1.75. Needs testing.

  I'm not suggesting we turn this on the for entire codebase, or immediately go-nuts refactoring all includes. However I think our dependency includes are now slim enough, and our CI infrastructure in place such that we can start doing this in some capacity, and just automate away include fixups / refactorings etc.

ACKs for top commit:
  MarcoFalke:
    review ACK 9b0a13a289
  jonatack:
    ACK 9b0a13a289 reviewed changes and run CI output in https://cirrus-ci.com/task/4750910332076032

Tree-SHA512: 00beab5a5f2a6fc179abf08321a15391ecccaa91ab56f3c50c511e7b29a0d7c95d8bb43eac2c31489711086f6f77319d43d803cf8ea458e7cd234a780d9ae69e
2022-04-28 10:06:26 +02:00
Carl Dong
94ad45deb2 ci: Build libbitcoinkernel 2022-04-27 17:36:39 -04:00
fanquake
9c96f1008b
tidy: enable modernize-use-nullptr 2022-04-26 10:43:33 +01:00
hiago
29f44fed36 Converting lint-all.sh to lint-all.py.
Converting `lint-all.sh` to `lint-all.py`.
2022-04-26 06:25:01 -03:00
fanquake
9b0a13a289
tidy: Add include-what-you-use 2022-04-20 14:14:52 +01:00
Hennadii Stepanov
a3a2bd9e8a
ci: Drop no longer needed package-specific flags 2022-04-16 08:59:13 +02:00
Hennadii Stepanov
86c2889518
ci: Make log verbose in error case only
This change silences depends build using LOG=1.
2022-04-14 13:44:29 +02:00
fanquake
864772c4f7
lint: mypy 0.942 2022-04-07 12:50:54 +01:00
fanquake
38031adee8
lint: flake8 4.0.1 2022-04-07 12:50:10 +01:00
fanquake
eaf712c801
lint: codespell 2.1.0 2022-04-07 12:49:51 +01:00
MarcoFalke
faa7ae8242
ci: Build all optional tools in tidy task 2022-04-06 12:48:38 +02:00
fanquake
372f1a3c25
Merge bitcoin/bitcoin#24753: ci: Add clang-tidy task
fab24f8c35 ci: Add clang-tidy task (MarcoFalke)

Pull request description:

  Fixes #24747

ACKs for top commit:
  fanquake:
    ACK fab24f8c35
  vincenzopalazzo:
    Code Review ACK fab24f8c35

Tree-SHA512: 87a5d67a23d540cea09925a6c186303c1249d4ca244b95940c5214860bf7a849fa12cc4fcf1bb9270b1004407124b3487902969d8ae9d2b14dae1fdfb57cc7e8
2022-04-06 10:10:41 +01:00
laanwj
f421de5be6
Merge bitcoin/bitcoin#24236: Remove utxo db upgrade code
fa9112aac0 Remove utxo db upgrade code (MarcoFalke)

Pull request description:

  It is not possible to upgrade Bitcoin Core pre-segwit (pre-0.13.1) to a recent version without a full IBD from scratch after  commit 19a56d1519 (released in version 22.0).

  Any Bitcoin Core version with the new database format after commit 1088b02f0c (released in version 0.15), can upgrade to any version that is supported as of today.

  This leaves the versions 0.13.1-0.14.x. Even though those versions are unsupported, some users with an existing datadir may want to upgrade to a recent version. However, it seems reasonable to simply ask them to `-reindex` to run a full IBD from scratch. This allows us to remove the utxo db upgrade code.

ACKs for top commit:
  Sjors:
    re-ACK fa9112aac0
  laanwj:
    Code review ACK fa9112aac0

Tree-SHA512: 4243bb35df9ac4892f9fad30fe486d338745952bcff4160bcb0937c772d57b13b800647da14695e21e3655e85ee0d95fa3dc7789ee309d59ad84f422297fecb8
2022-04-05 15:38:14 +02:00
MarcoFalke
fab24f8c35
ci: Add clang-tidy task 2022-04-04 11:57:06 +02:00
fanquake
b0740fdcb8
ci: use DWARF-4 for Valgrind CI job
clang-14 defaults to using DWARF-5, which breaks vlagrinds (3.18) ability
to parse debug info. Valgrind claims to support DWARF-5 from version
3.18 onwards, but maybe that only works when building with GCC.

Explicitly use DWARF-4 for now. Note that from 11.0 GCC also defaults to
using DWARF-5.

https://releases.llvm.org/14.0.0/tools/clang/docs/ReleaseNotes.html#dwarf-support-in-clang
https://www.gnu.org/software/gcc/gcc-11/changes.html
https://valgrind.org/docs/manual/dist.news.html
2022-04-04 10:39:55 +01:00