feat(peer-store): introduce libp2p-peer-store #5724

drHuangMHT · 2024-12-07T07:36:19Z

Description

Introduce libp2p-peer-store for a peer store implementation.

Related: #4103

Notes & open questions

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

drHuangMHT · 2024-12-07T07:49:28Z

This is a very basic implementation, would love to hear more ideas on how to implement this!

elenaf9

Thanks for starting this effort @drHuangMHT!

If I understand the current implementation correctly, the purpose of this store is just to track connected peers, and explicitly added addresses. However, I think we have to keep in mind that protocols like kademlia or identify very frequently report all addresses that they learn as NewExternalAddrOfPeer. So with the current MemoryStore implementation, the store would grow unbounded with all of these addresses. I think there needs to be some GC strategy.

Also, after reading #4103 (comment), I think a peer store implementation could also do much more than just track explicitly added addresses, e.g.:

track how valuable a know address is, by using infos about how the address info was received, if we connected to it already, etc
implement a TTL for address records
track other meta data for a peer, e.g. public key

I don't think this needs to be implemented in this PR. But I think we could forward more of the already present info to the Store trait, so that a custom implementation can have more sophisticated logic.

elenaf9 · 2024-12-07T15:41:06Z

misc/peer-store/src/store.rs

+    /// Peers that are currently connected.
+    connected_peers: HashSet<PeerId>,


Is there a case where using a HashSet<PeerId> to track connected peers is unsuitable for a specific use case? If now, how about moving this into the Behavior, so that the Store only concerns the "address-book" part of this behavior?

The idea of Store trait is to allow on-disk storage, now I think about it, this info will be changing in real time so it should be kept in memory anyway. Will move it into the behaviour itself.

can we have MemoryStore<T=()> so that we are able to store data for peers (like scoring etc):

can we have MemoryStore<T=()> so that we are able to store data for peers (like scoring etc):

MemoryStore is more of a reference implementation, I don't think it is necessary to include a generic parameter for customization since we are maintaining its internals.

misc/peer-store/src/store.rs

misc/peer-store/src/lib.rs

misc/peer-store/src/store.rs

misc/peer-store/src/behaviour.rs

misc/peer-store/src/store.rs

dariusc93 · 2024-12-07T18:03:40Z

misc/peer-store/src/behaviour.rs

+        _role_override: libp2p_core::Endpoint,
+        _port_use: libp2p_core::transport::PortUse,
+    ) -> Result<libp2p_swarm::THandler<Self>, libp2p_swarm::ConnectionDenied> {
+        self.store.on_peer_connect(&peer);


It might make more sense to remove this for FromSwarm events since a connection could be denied later on (ie connection limits, banned peer, etc.), so that way the store isnt exactly cluttered.

That makes sense. Will favor the swarm event.

can we instead define Store::handle_* methods that are called here and in the other NetworkBehaviour::handle_*
so that it allows us to further manage our peers?

can we instead define Store::handle_* methods that are called here and in the other NetworkBehaviour::handle_* so that it allows us to further manage our peers?

What are those specifically? The store no longer record connected peers.

dariusc93 · 2024-12-07T18:04:52Z

misc/peer-store/src/store.rs

+/// - contains all observed addresses of peers;
+pub trait Store {
+    /// Called when a peer connects.
+    fn on_peer_connect(&mut self, peer: &PeerId);


If we are tracking peer connections too, should we also keep tabs on the ConnectionId?

We can, but how to store it? I think there can be multiple connections from a single peer, no?

misc/peer-store/src/store.rs

misc/peer-store/src/behaviour.rs

simplify Store trait, don't report conencted unless confirmed by swarm event

…ture denial

drHuangMHT · 2024-12-09T06:32:48Z

Oops that use<'a> syntax was stabilized in rust 1.82, the compiler on 1.80 wasn't happy with it.

jxs

Thanks for starting this @drHuangMHT, already looking good! left some notes to allow further customization.
cc @elenaf9

jxs · 2024-12-11T12:30:08Z

misc/peer-store/Cargo.toml

+[package]
+name = "libp2p-peer-store"
+version = "0.1.0"
+edition = "2021"


you can add yourself as the author here :)

misc/peer-store/src/behaviour.rs

jxs · 2024-12-11T16:00:28Z

misc/peer-store/src/store.rs

+    /// Peers that are currently connected.
+    connected_peers: HashSet<PeerId>,


can we have MemoryStore<T=()> so that we are able to store data for peers (like scoring etc):

jxs · 2024-12-11T16:02:30Z

misc/peer-store/src/behaviour.rs

+        _role_override: libp2p_core::Endpoint,
+        _port_use: libp2p_core::transport::PortUse,
+    ) -> Result<libp2p_swarm::THandler<Self>, libp2p_swarm::ConnectionDenied> {
+        self.store.on_peer_connect(&peer);


can we instead define Store::handle_* methods that are called here and in the other NetworkBehaviour::handle_*
so that it allows us to further manage our peers?

Store now handles FromSwarm directly; rename some on_* methods; allow removing address; update records on FromSwarm::ConnectionEstablished

drHuangMHT · 2024-12-16T06:22:29Z

track how valuable a know address is, by using infos about how the address info was received, if we connected to it already, etc

This can get very complicated very soon, for example how to record the activitiy on the address(there will be some overhead) and how to make it machine-readable(scoring system). The discussion will be quite lengthy.

implement a TTL for address records

I didn't see a good way to implement TTL(garbage collect) for records, any pointer?

track other meta data for a peer, e.g. public key

I don't see the remote public key being available through swarm itself, be it in the form of PeerRecord, SignedEnvelope or just PublicKey. But the pubkey is indeed a very important metadata of a peer.

…into peer-store

drHuangMHT added 5 commits December 6, 2024 14:48

template implementation

1affefe

implement in-memory store

8d4930a

manifest update

9cb59c3

formatting

fbc1344

docs

2bcfa7f

elenaf9 reviewed Dec 7, 2024

View reviewed changes

dariusc93 reviewed Dec 7, 2024

View reviewed changes

drHuangMHT added 5 commits December 8, 2024 11:14

return iterator of references instead of heap allocation

6df6ee5

move conencted_peers out of Store

99a6bfd

simplify Store trait, don't report conencted unless confirmed by swarm event

use capped LruCache instead of uncapped HashMap for address records

5c7ba32

update address book when a connection is established regardless of fu…

b6dcd59

…ture denial

provide address for dial

cbb1906

jxs reviewed Dec 11, 2024

View reviewed changes

apply suggestions

6270259

Store now handles FromSwarm directly; rename some on_* methods; allow removing address; update records on FromSwarm::ConnectionEstablished

drHuangMHT added 3 commits December 17, 2024 20:36

garbage collect records, test

cc91ab5

documentation and formatting

f9b040e

clippy lint

a9597da

drHuangMHT requested review from jxs, dariusc93 and elenaf9 December 18, 2024 02:30

drHuangMHT added 6 commits December 20, 2024 14:36

simplify Store trait

0e6b280

Merge branch 'master' into peer-store

fe14c42

manifest and changelog

92ac2fb

Merge branch 'peer-store' of https://github.com/drHuangMHT/rust-libp2p …

913ed1f

…into peer-store

export at libp2p crate root

b8a7114

changelog for libp2p

5f628fb

elenaf9 mentioned this pull request Dec 23, 2024

feat(request-response): Add support for custom dial options when making request to disconnected peer #5692

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(peer-store): introduce libp2p-peer-store #5724

feat(peer-store): introduce libp2p-peer-store #5724

drHuangMHT commented Dec 7, 2024

drHuangMHT commented Dec 7, 2024

elenaf9 left a comment •

edited

Loading

elenaf9 Dec 7, 2024

drHuangMHT Dec 8, 2024

jxs Dec 11, 2024

drHuangMHT Dec 16, 2024

dariusc93 Dec 7, 2024

drHuangMHT Dec 8, 2024

jxs Dec 11, 2024

drHuangMHT Dec 16, 2024

dariusc93 Dec 7, 2024

drHuangMHT Dec 8, 2024

drHuangMHT commented Dec 9, 2024

jxs left a comment

jxs Dec 11, 2024

jxs Dec 11, 2024

jxs Dec 11, 2024

drHuangMHT commented Dec 16, 2024 •

edited

Loading

		/// Peers that are currently connected.
		connected_peers: HashSet<PeerId>,

feat(peer-store): introduce libp2p-peer-store #5724

Are you sure you want to change the base?

feat(peer-store): introduce libp2p-peer-store #5724

Conversation

drHuangMHT commented Dec 7, 2024

Description

Notes & open questions

Change checklist

drHuangMHT commented Dec 7, 2024

elenaf9 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drHuangMHT commented Dec 9, 2024

jxs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drHuangMHT commented Dec 16, 2024 • edited Loading

elenaf9 left a comment •

edited

Loading

drHuangMHT commented Dec 16, 2024 •

edited

Loading