Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions src/llmq/net_signing.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -320,8 +320,11 @@ void NetSigning::WorkThreadDispatcher()
}
}

// Collect pending sig shares synchronously and dispatch each batch to a worker for parallel BLS verification
while (!workInterrupt) {
// Collect pending sig shares synchronously and dispatch each batch to a worker for parallel BLS verification.
// Batches awaiting verification are bounded so that under flood shares back up in the capped
// pending maps instead of migrating into the unbounded worker pool task queue.
static constexpr int MAX_UNVERIFIED_BATCHES{4};
while (!workInterrupt && unverified_batches < MAX_UNVERIFIED_BATCHES) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bound verification work by share count

Fresh evidence beyond the earlier queue-cap concern is that CollectPendingSigSharesToVerify(32, ...) limits unique (nodeId, signHash) pairs, not actual shares; if peers fill the pending maps with many quorum members for the same few sessions, one collected “batch” can contain up to the 10k pending-share cap. This line then allows four such batches to be queued/in flight after they have been erased from the capped pending maps, so flood traffic can still move tens of thousands of BLS shares into verification work despite the intended global cap. Cap the number of shares collected per dispatch or keep queued/in-flight shares charged against the cap.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 355da45d4a4. CollectPendingSigSharesToVerify now caps each dispatch by actual shares added (MAX_SHARES_PER_BATCH = 32) instead of unique sessions. With four unverified batches, at most 128 shares can sit in/on the worker pool outside the 10,000-share pending maps; randomized peer round-robin fairness is preserved. Focused LLMQ tests and neighboring suites pass.

std::unordered_map<NodeId, std::vector<CSigShare>> sigSharesByNodes;
std::unordered_map<std::pair<Consensus::LLMQType, uint256>, CQuorumCPtr, StaticSaltedHasher> quorums;

Expand All @@ -332,7 +335,13 @@ void NetSigning::WorkThreadDispatcher()
break;
}

++unverified_batches;
worker_pool.push([this, sigSharesByNodes = std::move(sigSharesByNodes), quorums = std::move(quorums)](int) mutable {
// Ensures unverified_batches is decremented on every exit path, including exceptions.
struct UnverifiedBatchGuard {
std::atomic<int>& count;
~UnverifiedBatchGuard() { --count; }
} guard{unverified_batches};
ProcessPendingSigShares(std::move(sigSharesByNodes), std::move(quorums));
});
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Expand Down
2 changes: 2 additions & 0 deletions src/llmq/net_signing.h
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

#include <thread>

#include <atomic>
#include <memory>

class CSporkManager;
Expand Down Expand Up @@ -72,6 +73,7 @@ class NetSigning final : public NetHandler, public CValidationInterface
std::thread shares_cleaning_thread;
std::thread shares_dispatcher_thread;
mutable ctpl::thread_pool worker_pool;
std::atomic<int> unverified_batches{0};

CThreadInterrupt workInterrupt;
};
Expand Down
35 changes: 33 additions & 2 deletions src/llmq/signing_shares.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@ namespace {
constexpr size_t MAX_SESSIONS_PER_PEER_FACTOR{4};
constexpr size_t MIN_SESSIONS_PER_PEER{100};

// Incoming QSIGSHARE/QBSIGSHARES traffic is cheap to admit but drains only at BLS verification
// speed, so unverified shares are bounded and over-cap shares dropped without misbehaviour scoring.
constexpr size_t MAX_PENDING_SIG_SHARES_PER_NODE{1000};
constexpr size_t MAX_PENDING_SIG_SHARES_TOTAL{10000};

size_t GetMaxSessionsForPeer(const Consensus::LLMQParams& params)
{
return std::max<size_t>(size_t(params.size) * MAX_SESSIONS_PER_PEER_FACTOR, MIN_SESSIONS_PER_PEER);
Expand Down Expand Up @@ -422,7 +427,7 @@ bool CSigSharesManager::ProcessMessageBatchedSigShares(const CNode& pfrom, const
LOCK(cs);
auto& nodeState = nodeStates[pfrom.GetId()];
for (const auto& s : sigSharesToProcess) {
nodeState.pendingIncomingSigShares.Add(s.GetKey(), s);
TryAddPendingIncomingSigShare(pfrom.GetId(), nodeState, s);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve request state when admission fails

When TryAddPendingIncomingSigShare returns false because the per-node or global pending-share cap is full, this loop silently drops a requested QBSIGSHARES response even though line 401 already removed the peer's requestedSigShares entry and the request path clears session.announced.inv once the request is sent. In that over-cap scenario the share is not verified and is not requested from the same peer again, so recovery can stall if no other peer advertises that quorum member; keep the request/announcement state until the share is admitted, or requeue it on admission failure.

Useful? React with 👍 / 👎.

}
return true;
}
Expand Down Expand Up @@ -467,14 +472,39 @@ bool CSigSharesManager::ProcessMessageSigShare(NodeId fromId, const CSigShare& s
}

auto& nodeState = nodeStates[fromId];
nodeState.pendingIncomingSigShares.Add(sigShare.GetKey(), sigShare);
TryAddPendingIncomingSigShare(fromId, nodeState, sigShare);
}

LogPrint(BCLog::LLMQ_SIGS, "CSigSharesManager::%s -- signHash=%s, id=%s, msgHash=%s, member=%d, node=%d\n", __func__,
signHash.ToString(), sigShare.getId().ToString(), sigShare.getMsgHash().ToString(), sigShare.getQuorumMember(), fromId);
return true;
}

bool CSigSharesManager::TryAddPendingIncomingSigShare(NodeId nodeId, CSigSharesNodeState& nodeState,
const CSigShare& sigShare)
{
AssertLockHeld(cs);

if (nodeState.banned) {
return false;
}
if (nodeState.pendingIncomingSigShares.Size() >= MAX_PENDING_SIG_SHARES_PER_NODE) {
LogPrint(BCLog::LLMQ_SIGS, "CSigSharesManager::%s -- per-node pending sig shares cap reached (%d), dropping sigShare. node=%d\n",
__func__, MAX_PENDING_SIG_SHARES_PER_NODE, nodeId);
return false;
}
size_t total{0};
for (const auto& [_, ns] : nodeStates) {
total += ns.pendingIncomingSigShares.Size();
}
if (total >= MAX_PENDING_SIG_SHARES_TOTAL) {
LogPrint(BCLog::LLMQ_SIGS, "CSigSharesManager::%s -- global pending sig shares cap reached (%d), dropping sigShare. node=%d\n",
__func__, MAX_PENDING_SIG_SHARES_TOTAL, nodeId);
return false;
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: Global pending-sig-share cap check is O(peers) per admission

TryAddPendingIncomingSigShare iterates every entry in nodeStates while holding cs to compute the global total on every QSIGSHARE/QBSIGSHARES admission. The per-node check is O(1) thanks to CountedBucketMap::Size(), but this loop reintroduces O(peers) work on the hot admission path. Under a flood with many connected peers, cs is held longer per accepted share, partially defeating the O(1) counting refactor and making the backpressure path itself scale with peer count. Maintain a manager-level running total that is updated alongside per-node Add/Erase operations so the global cap remains O(1). Correctness is unaffected, so this is non-blocking.

source: ['claude']

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 355da45d4a4 as part of the same accounting correction. A manager-level m_pending_sig_shares_total is now updated on every add/erase/session removal/node removal/ban path, so the global admission cap is O(1) instead of walking all peers under cs. The erase primitives return exact removed counts, with regression coverage.

return nodeState.pendingIncomingSigShares.Add(sigShare.GetKey(), sigShare);
}
Comment on lines +483 to +508

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: Cap/drop/banned enforcement in TryAddPendingIncomingSigShare has no direct unit coverage

The PR's stated purpose is bounding pending incoming sig shares, but the enforcement logic in TryAddPendingIncomingSigShare — the per-node cap at MAX_PENDING_SIG_SHARES_PER_NODE (line 491), the global cap at MAX_PENDING_SIG_SHARES_TOTAL (line 502), and the banned-state refusal (line 488) — is not exercised by any added test. A grep of llmq_utils_tests.cpp finds no reference to TryAddPendingIncomingSigShare, MAX_PENDING_SIG_SHARES, or the banned path; the new tests only cover the supporting CountedBucketMap size tracking and RemoveSession behavior. So the machinery the caps rely on (Size() accuracy) is tested, but the threshold/drop-without-misbehavior/banned-refusal behavior that is the point of the change is not. TryAddPendingIncomingSigShare is private and needs a CSigSharesManager plus quorum setup to drive, which makes this materially harder than the node-state-level tests already added, so this is a suggestion rather than a blocker.

source: ['claude']


bool CSigSharesManager::CollectPendingSigSharesToVerify(
size_t maxUniqueSessions, std::unordered_map<NodeId, std::vector<CSigShare>>& retSigShares,
std::unordered_map<std::pair<Consensus::LLMQType, uint256>, CQuorumCPtr, StaticSaltedHasher>& retQuorums)
Expand Down Expand Up @@ -1425,6 +1455,7 @@ void CSigSharesManager::MarkAsBanned(NodeId nodeId)
sigSharesRequested.Erase(k);
});
nodeState.requestedSigShares.Clear();
nodeState.pendingIncomingSigShares.Clear();
nodeState.banned = true;
}

Expand Down
148 changes: 105 additions & 43 deletions src/llmq/signing_shares.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
#include <evo/types.h>
#include <llmq/signhash.h>
#include <llmq/signing.h>
#include <util/std23.h>

#include <random.h>
#include <saltedhasher.h>
Expand Down Expand Up @@ -160,49 +159,124 @@ class CBatchedSigShares
[[nodiscard]] std::string ToInvString() const;
};

/**
* Two-level (signHash -> quorumMember) map with a running entry count, so Size() is O(1)
* instead of a fold over all sign hash buckets. All structural mutations go through the
* counted methods; Buckets() is for lookups and in-place value updates only.
*/
template<typename T>
class SigShareMap
class CountedBucketMap
{
public:
using BucketMap = Uint256HashMap<std::unordered_map<uint16_t, T>>;

private:
Uint256HashMap<std::unordered_map<uint16_t, T>> internalMap;
BucketMap m_data;
size_t m_num_entries{0};

public:
bool Add(const SigShareKey& k, const T& v)
BucketMap& Buckets() { return m_data; }
const BucketMap& Buckets() const { return m_data; }
[[nodiscard]] size_t Size() const { return m_num_entries; }

bool Emplace(const SigShareKey& k, const T& v)
{
auto& m = internalMap[k.first];
return m.emplace(k.second, v).second;
if (!m_data[k.first].emplace(k.second, v).second) {
return false;
}
++m_num_entries;
return true;
}

void Erase(const SigShareKey& k)
{
auto it = internalMap.find(k.first);
if (it == internalMap.end()) {
auto it = m_data.find(k.first);
if (it == m_data.end()) {
return;
}
it->second.erase(k.second);
m_num_entries -= it->second.erase(k.second);
if (it->second.empty()) {
internalMap.erase(it);
m_data.erase(it);
}
}

void EraseBucket(const uint256& signHash)
{
auto it = m_data.find(signHash);
if (it == m_data.end()) {
return;
}
m_num_entries -= it->second.size();
m_data.erase(it);
}

template<typename F>
void EraseIf(F&& f)
{
for (auto it = m_data.begin(); it != m_data.end(); ) {
SigShareKey k;
k.first = it->first;
for (auto jt = it->second.begin(); jt != it->second.end(); ) {
k.second = jt->first;
if (f(k, jt->second)) {
jt = it->second.erase(jt);
--m_num_entries;
} else {
++jt;
}
}
if (it->second.empty()) {
it = m_data.erase(it);
} else {
++it;
}
}
}

void Clear()
{
internalMap.clear();
m_data.clear();
m_num_entries = 0;
}
};

template<typename T>
class SigShareMap
{
private:
CountedBucketMap<T> internalMap;

public:
bool Add(const SigShareKey& k, const T& v)
{
return internalMap.Emplace(k, v);
}

void Erase(const SigShareKey& k)
{
internalMap.Erase(k);
}

void Clear()
{
internalMap.Clear();
}

[[nodiscard]] bool Has(const SigShareKey& k) const
{
auto it = internalMap.find(k.first);
if (it == internalMap.end()) {
auto& m = internalMap.Buckets();
auto it = m.find(k.first);
if (it == m.end()) {
return false;
}
return it->second.count(k.second) != 0;
}

T* Get(const SigShareKey& k)
{
auto it = internalMap.find(k.first);
if (it == internalMap.end()) {
auto& m = internalMap.Buckets();
auto it = m.find(k.first);
if (it == m.end()) {
return nullptr;
}

Expand All @@ -226,72 +300,58 @@ class SigShareMap

const T* GetFirst() const
{
if (internalMap.empty()) {
auto& m = internalMap.Buckets();
if (m.empty()) {
return nullptr;
}
return &internalMap.begin()->second.begin()->second;
return &m.begin()->second.begin()->second;
}

[[nodiscard]] size_t Size() const
{
return std23::ranges::fold_left(internalMap, size_t{0},
[](size_t s, const auto& p) { return s + p.second.size(); });
return internalMap.Size();
}

[[nodiscard]] size_t CountForSignHash(const uint256& signHash) const
{
auto it = internalMap.find(signHash);
if (it == internalMap.end()) {
auto& m = internalMap.Buckets();
auto it = m.find(signHash);
if (it == m.end()) {
return 0;
}
return it->second.size();
}

[[nodiscard]] bool Empty() const
{
return internalMap.empty();
return internalMap.Buckets().empty();
}

const std::unordered_map<uint16_t, T>* GetAllForSignHash(const uint256& signHash) const
{
auto it = internalMap.find(signHash);
if (it == internalMap.end()) {
auto& m = internalMap.Buckets();
auto it = m.find(signHash);
if (it == m.end()) {
return nullptr;
}
return &it->second;
}

void EraseAllForSignHash(const uint256& signHash)
{
internalMap.erase(signHash);
internalMap.EraseBucket(signHash);
}

template<typename F>
void EraseIf(F&& f)
{
for (auto it = internalMap.begin(); it != internalMap.end(); ) {
SigShareKey k;
k.first = it->first;
for (auto jt = it->second.begin(); jt != it->second.end(); ) {
k.second = jt->first;
if (f(k, jt->second)) {
jt = it->second.erase(jt);
} else {
++jt;
}
}
if (it->second.empty()) {
it = internalMap.erase(it);
} else {
++it;
}
}
internalMap.EraseIf(f);
}

template<typename F>
void ForEach(F&& f)
{
for (auto& p : internalMap) {
for (auto& p : internalMap.Buckets()) {
SigShareKey k;
k.first = p.first;
for (auto& p2 : p.second) {
Expand Down Expand Up @@ -484,6 +544,8 @@ class CSigSharesManager : public llmq::CRecoveredSigsListener
bool GetSessionInfoByRecvId(NodeId nodeId, uint32_t sessionId, CSigSharesNodeState::SessionInfo& retInfo)
EXCLUSIVE_LOCKS_REQUIRED(!cs);
static CSigShare RebuildSigShare(const CSigSharesNodeState::SessionInfo& session, const std::pair<uint16_t, CBLSLazySignature>& in);
bool TryAddPendingIncomingSigShare(NodeId nodeId, CSigSharesNodeState& nodeState, const CSigShare& sigShare)
EXCLUSIVE_LOCKS_REQUIRED(cs);

void RemoveSigSharesForSession(const uint256& signHash) EXCLUSIVE_LOCKS_REQUIRED(cs);

Expand Down
Loading
Loading