jamtis changes by jeffro256 · Pull Request #26 · UkoeHB/monero

jeffro256 · 2023-12-03T16:41:32Z

nominal address tag protection for LWs (extra address key)
flexible view tags
churning and pocketchange protection for LWs (auxiliary enote records)

jeffro256 · 2023-12-06T05:31:10Z

For unmodified seraphis_lib, running the test_remote_scanner_client_scan_sp performance test, I got the following result:

test_remote_scanner_client_scan_sp (23 calls) - OK: 13391 µs/call
test_remote_scanner_client_scan_sp (23 calls) - OK: 13478 µs/call
test_remote_scanner_client_scan_sp (23 calls) - OK: 14521 µs/call

On this branch, I get:

test_remote_scanner_client_scan_sp (23 calls) - OK: 1790 ms/call
test_remote_scanner_client_scan_sp (23 calls) - OK: 1811 ms/call
test_remote_scanner_client_scan_sp (23 calls) - OK: 1802 ms/call

This means that for the different test modes ALL_FAKE, ONE_FAKE_TAG_MATCH, ONE_OWNED, the new client-side view scanning takes 134x, 134x, and 124x, respectively, times more CPU cycles than unmodified seraphis_lib. This puts it on par with the dense/sparse changes #15.

jeffro256 · 2023-12-11T08:24:03Z

Latest commit adds a unit test case which performs a successful Janus attack with knowledge of one address private key. Will push fix soon...

jeffro256 · 2023-12-13T22:06:32Z

Currently making modifications to the "Implementing Seraphis" document to reflect the changes in this PR. Will release a PR to the Seraphis repo when done.

* nominal address tag protection for LWs (extra address key) * flexible view tags * churning and pocketchange protection for LWs (auxiliary enote records) PR implementing changes is here: UkoeHB/monero#26

jeffro256 · 2023-12-21T19:13:20Z

Jamtis changes are documented here: UkoeHB/Seraphis#6

jeffro256 · 2024-02-14T19:03:12Z

Rebased

UkoeHB · 2024-02-22T04:24:35Z

Needs to be updated to reflect the notational changes made in UkoeHB/Seraphis#6.

jeffro256 · 2024-02-23T21:56:28Z

Did a simple rebase again to fix merge conflicts with the derived view balance key PR. Will tackle updating notation today

jeffro256 · 2024-02-24T20:49:32Z

How does the notation look on first glance?

UkoeHB

13/72 files done.

One thing you need to do is check that all the KDF transcripts are <= 128 bytes (i.e. one blake2b block). If > 128 then the cost to hash increases a lot. You can test this by editing SpKDFTranscript so the constructor prints the domain separator, and the .size() method prints its size. Then when you run the seraphis unit test suite all the sizes will print.

If you find any that crept over the line, then the relevant domain separator needs to be shaved down.

jeffro256 · 2024-02-25T07:09:46Z

The idea behind there being two try_find_sp_enotes_in_tx is that the first one does the meat of the work, but does the least amount of allocations and copying possible. All it really needs to allocate is the output vector of bools. This ideally would be used in actual production code which scans on behalf of users, but doesn't need to send out chunk data yet, and just caches some bools persistently (maybe as unsigned ints for compactness).

jeffro256 · 2024-02-26T21:11:18Z

In regards to the KDF transcripts being smaller than 128 bytes, I applied this diff:

diff --git a/src/seraphis_crypto/sp_transcript.h b/src/seraphis_crypto/sp_transcript.h
index 17db163d6..f0e192fb3 100644
--- a/src/seraphis_crypto/sp_transcript.h
+++ b/src/seraphis_crypto/sp_transcript.h
@@ -100,7 +100,7 @@ public:
 
     /// access the transcript data
     const void* data() const { return m_transcript.data(); }
-    std::size_t size() const { return m_transcript.size(); }
+    std::size_t size() const { if (m_transcript.size() > 128) { std::cout << "BIG TRANSCRIPT: " << m_transcript.data(); } return m_transcript.size(); }
 
 private:
 //member variables

I then ran ./build/Linux/jamtis_lw_take_2/debug/tests/unit_tests/unit_tests --gtest_filter=seraphis* | grep jamtis_. The only domain separator that showed up was jamtis_input_context_standardlegacy_input_KI, so I don't think that these changes should've made any transcripts go over 128 bytes.

UkoeHB

Part 2 (20/72 files done)

jeffro256 · 2024-03-07T16:43:31Z

Rebased

* SpTxCoinbaseV1: remove block_reward field Not storing/serializing `block_reward` saves us a few bytes on coinbase transactions, and makes it so that you can't initialize a coinbase transaction that has a block reward not matching its output sum.

jeffro256 · 2024-03-22T16:33:53Z

Rebased again

…structs (monero-project#31)

--------- Co-authored-by: UkoeHB <37489173+UkoeHB@users.noreply.github.com>

UkoeHB

Part 3 (44/72 files done)

Skipping over the files with huge diffs for now.

DangerousFreedom1984 · 2024-03-28T11:54:47Z

Wouldn't it be better to use

sr1 instead of q
sr2 instead of baked_key ?

DangerousFreedom1984 · 2024-03-28T23:07:31Z

Okay, I reviewed the new changes and compared to the new seraphis draft paper and it is looking good to me. Very nice work!

jeffro256 · 2024-04-07T12:26:28Z

@DangerousFreedom1984 It might be better to use that notation in order to match the implementation paper. It also might be a little easier to read them wrong when skimming. I don't have a super strong opinion either way.

* direct & compact tx serialization txs are [de]serialized directly from their classes and sizes of containers are not serialized if they can be implied.

Implement async wallet scanner. Adds a new functional test for direct wallet2 -> live RPC daemon interactions. This sets up a framework to test pointing the Seraphis wallet lib to a live daemon. Tests sending and scanning: - a normal transfer - a sweep single (0 change) - to a single subaddress - to 3 subaddresses (i.e. scanning using additional pub keys) * scan machine: option to force reorg avoidance increment first pass - when pointing to a daemon that does not support returning empty blocks when the client requests too high of a height, we have to be careful in our scanner to always request blocks below the chain tip, in every request. - by forcing the reorg avoidance increment on first pass, we make sure clients will always include the reorg avoidance increment when requesting blocks from the daemon, so the client can expect the request for blocks should *always* return an ok height. * core tests: check conversion tool on all legacy enote version types Stil TODO: - check complete scanning on all enote types - hit every branch condition for all enote versions * conn pool mock: epee http client connection pool - Enables concurrent network requests using the epee http client. - Still TODO for production: 1) close_connections 2) require the pool respect max_connections * enote finding context: IN LegacyUnscannedChunk, OUT ChunkData - finds owned enotes by legacy view scanning a chunk of blocks * async: function to remove minimum element from token queue - Useful when we want to remove elements of the token queue in an order that is different than insertion order. * async scanner: scan via RPC, fetching & scanning parallel chunks *How it works* Assume the user's wallet must start scanning blocks from height 5000. 1. The scanner begins by launching 10 RPC requests in parallel to fetch chunks of blocks as follows: ``` request 0: { start_height: 5000, max_block_count: 20 } request 1: { start_height: 5020, max_block_count: 20 } ... request 9: { start_height: 5180, max_block_count: 20 } ``` 2. As soon as any single request completes, the wallet immediately parses that chunk. - This is all in parallel. For example, as soon as request 7 responds, the wallet immediately begins parsing that chunk in parallel to any other chunks it's already parsing. 3. If a chunk does not include a total of max_block_count blocks, and the chunk is not the tip of the chain, this means there was a "gap" in the chunk request. The scanner launches another parallel RPC request to fill in the gap. - This gap can occur because the server will **not** return a chunk of blocks greater than 100mb (or 20k txs) via the /getblocks.bin` RPC endpoint ([`FIND_BLOCKCHAIN_SUPPLEMENT_MAX_SIZE`](https://github.com/monero-project/monero/blob/053ba2cf07649cea8134f8a188685ab7a5365e5c/src/cryptonote_core/blockchain.cpp#L65)) - The gap is from `(req.start_height + res.blocks.size())` to `(req.start_height + req.max_block_count)`. 4. As soon as the scanner finishes parsing the chunk, it immediately submits another parallel RPC request. 5. In parallel, the scanner identifies a user's received (and spent) enotes in each chunk. - For example, as soon as request 7 responds and the wallet parses it, the wallet scans that chunk in parallel to any other chunks it's already scanning. 6. Once a single chunk is fully scanned locally, the scanner launches a parallel task to fetch and scan the next chunk. 7. Once the scanner reaches the tip of the chain (the terminal chunk), the scanner terminates. *Some technical highlights* - The wallet scanner is backwards compatible with existing daemons (though it would need to point to an updated daemon to realize the perf speed-up). - On error cases such as the daemon going offline, the same wallet errors that wallet2 uses (that the wallet API expects) are propagated up to the higher-level Seraphis lib. - The implementation uses an http client connection pool (reusing the epee http client) to support parallel network requests ([related](seraphis-migration/wallet3#58)). - A developer using the scanner can "bring their own blocks/network implementation" to the scanner by providing a callback function of the following type as a param to the async scanner constructor: `std::function<bool(const cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::request, cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::response)>` --------- Co-authored-by: jeffro256 <jeffro256@tutanota.com>

--------- Co-authored-by: SNeedlewoods <sneedlewoods_1@protonmail.com>

This PR removes "universal"-style indexing for legacy CLSAG rings, and replaces it with a reference set scheme that uses (amount, index in amount) indexing pairs to reference on-chain enotes. This is the same method that Cryptonote txs use, and is how the current Monero Core LMDB database is referenced. Doing things this way means that the database will not have to be re-indexed, saving at a very minimum 1.6 GB (100M on-chain enotes * (16 bytes for extra table keys)) of storage space, and an expensive database migration involving moving all existing enote data to a new table. We change the MockLedgerContext to support this indexing scheme. In practice, serialized txs under this method shouldn't take up much more space than pre-PR if compressed clever-ly, and assuming most ring members will RingCT enotes. We also add LegacyEnoteOriginContext for contextualized enote records so we can better keep tracked of scanned legacy enotes under the legacy indexing scheme. Co-authored-by: SNeedlewoods <sneedlewoods_1@protonmail.com>

* nominal address tag protection for LWs (extra address key) * flexible view tags * churning and pocketchange protection for LWs (auxiliary enote records)

jeffro256 · 2024-05-24T19:08:14Z

Rebased to fix conflicts with #23, #39, and #42

UkoeHB · 2024-08-02T19:34:03Z

@jeffro256 Sorry the rebase broke this again. Sadly it looks like the work here may end up never being merged (largely my fault for not finishing review earlier this year).

jeffro256 marked this pull request as draft December 3, 2023 16:45

jeffro256 mentioned this pull request Dec 3, 2023

jamtis: implement dense/sparse view privacy changes #15

Closed

jeffro256 force-pushed the jamtis_lw_take_2 branch 3 times, most recently from 55cad5e to 3cf1863 Compare December 4, 2023 15:04

jeffro256 marked this pull request as ready for review December 6, 2023 05:19

jeffro256 mentioned this pull request Dec 21, 2023

Jamtis changes UkoeHB/Seraphis#6

Merged

UkoeHB force-pushed the seraphis_lib branch 2 times, most recently from d493a6d to 279be2b Compare February 9, 2024 00:11

jeffro256 force-pushed the jamtis_lw_take_2 branch 3 times, most recently from af37a83 to 5ecea4f Compare February 14, 2024 19:01

jeffro256 force-pushed the jamtis_lw_take_2 branch 2 times, most recently from f73feab to 1761394 Compare February 14, 2024 19:05

jeffro256 force-pushed the jamtis_lw_take_2 branch from 1761394 to 2c96c0e Compare February 23, 2024 21:55

UkoeHB requested changes Feb 25, 2024

View reviewed changes

jeffro256 commented Feb 25, 2024

View reviewed changes

UkoeHB requested changes Feb 25, 2024

View reviewed changes

Comment thread src/seraphis_core/jamtis_enote_utils.cpp Outdated

Comment thread tests/unit_tests/seraphis_knowledge_proofs.cpp Outdated

UkoeHB requested changes Feb 25, 2024

View reviewed changes

Comment thread src/seraphis_core/jamtis_enote_utils.cpp Outdated

UkoeHB requested changes Mar 5, 2024

View reviewed changes

UkoeHB and others added 2 commits March 6, 2024 02:44

fix typo in header

8d14b42

free function serialization DSL (monero-project#32)

111e58a

UkoeHB force-pushed the seraphis_lib branch from 70b0404 to 111e58a Compare March 6, 2024 08:44

jeffro256 force-pushed the jamtis_lw_take_2 branch 2 times, most recently from 66f8605 to 98b443b Compare March 7, 2024 16:43

UkoeHB and others added 2 commits March 21, 2024 09:48

blake2b whitespace fixes

60f0ea9

jeffro256 force-pushed the jamtis_lw_take_2 branch from 98b443b to c987366 Compare March 22, 2024 16:33

Jeffro and others added 2 commits March 22, 2024 14:14

seraphis_crypto: split BulletproofPlus2 into proof-only and complete …

7c80674

…structs (monero-project#31)

Remaining tx ridigity issues (monero-project#34)

b7d8632

--------- Co-authored-by: UkoeHB <37489173+UkoeHB@users.noreply.github.com>

UkoeHB requested changes Mar 28, 2024

View reviewed changes

UkoeHB and others added 8 commits April 23, 2024 16:39

cleanup

f0eac25

cleanup

6674edc

Direct & compact tx serialization (monero-project#39)

c742260

* direct & compact tx serialization txs are [de]serialized directly from their classes and sizes of containers are not serialized if they can be implied.

serialization: serialize tools::variant (monero-project#49)

38b0fc2

async: default threadpool cannot be inlined

e45a5ac

Identify non-standard received legacy enotes (monero-project#43)

111fe51

--------- Co-authored-by: SNeedlewoods <sneedlewoods_1@protonmail.com>

jeffro256 force-pushed the jamtis_lw_take_2 branch from f35a8ed to 9003a71 Compare May 24, 2024 18:37

jamtis changes

537ba29

* nominal address tag protection for LWs (extra address key) * flexible view tags * churning and pocketchange protection for LWs (auxiliary enote records)

jeffro256 force-pushed the jamtis_lw_take_2 branch from 9003a71 to 537ba29 Compare May 24, 2024 19:06

UkoeHB force-pushed the seraphis_lib branch from ace9228 to d2edf42 Compare August 2, 2024 18:49

Conversation

jeffro256 commented Dec 3, 2023

Uh oh!

jeffro256 commented Dec 6, 2023

Uh oh!

jeffro256 commented Dec 11, 2023

Uh oh!

jeffro256 commented Dec 13, 2023

Uh oh!

jeffro256 commented Dec 21, 2023

Uh oh!

jeffro256 commented Feb 14, 2024

Uh oh!

UkoeHB commented Feb 22, 2024

Uh oh!

jeffro256 commented Feb 23, 2024

Uh oh!

jeffro256 commented Feb 24, 2024

Uh oh!

UkoeHB left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeffro256 Feb 25, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeffro256 commented Feb 26, 2024

Uh oh!

UkoeHB left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeffro256 commented Mar 7, 2024

Uh oh!

jeffro256 commented Mar 22, 2024

Uh oh!

UkoeHB left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DangerousFreedom1984 commented Mar 28, 2024

Uh oh!

DangerousFreedom1984 commented Mar 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffro256 commented Apr 7, 2024

Uh oh!

jeffro256 commented May 24, 2024

UkoeHB left a comment •

edited

Loading

UkoeHB left a comment •

edited

Loading

DangerousFreedom1984 commented Mar 28, 2024 •

edited

Loading