NIP-91: AND operator for filters

[hzrd149]

Duplicate PR

This PR is a clone of PR #1365 in order to reopen that PR and continue the discussion and tracking of implementations.

---

Filter operand that has optimization benefits for relays, users and developers. Likely contentious.

This NIP is the result of convergent efforts. I wrote this NIP because it seemed obvious but didn't submit because I didn't have an implementation, @v0l wrote an implementation(*) around the same time. @lez connected the dots.

Rendered NIP

Discourse

• issue

Implementations

• nostr-rs-relay docker image
• satellite-node
• netstr
• worker-relay
• applesauce
• CodyTseng/nostr-relay
• nostr-relay-nestjs
• nostr-relay-tray
• nostr-relay-sqlite
• amethyst
• nostr.land
• nostr-idb
• nostr-rs-relay
• wok

pending

• strfry PR #1
• strfry PR #2
• nostr-tools

Rationale

• Reduce bandwidth for all, meme AND cat objectively consumes less bandwidth than meme OR cat
• Reduce clock-time for relays, indexing with AND is fast for all common index formats, and faster compared to OR for some index formats. (See section below)
• Reduce client-side caching requirements
• Reduce centralization vectors by reducing or even eliminating the need for centralized REST, GraphQL APIs or specialized relay "feed" endpoints.
• Give relays the option to be more useful at the protocol level while improving efficiency for all parties.

Considerations

• New field for NIP-11.limitations: max_tags_per_and and max_tags_and
• Benchmarking should be conducted to validate that bandwidth and protocol usability as benefits supercede implementation and clock-time cost.

Index Efficiency

Index Type	AND Operation Efficiency	OR Operation Efficiency	Notes
B-Tree	High	Moderate	B-Tree indexes are very efficient for AND operations, especially with compound indexes. For OR operations, they are less efficient than for AND, as the database engine might need to traverse multiple paths.
Bitmap	High	High	Bitmap indexes excel in both AND and OR operations, particularly for columns with low cardinality. They utilize fast bitwise operations, making them ideal for read-heavy environments.
Hash	Not Applicable	Not Applicable	Hash indexes are designed for equality checks and do not directly support range-based queries or optimize for AND/OR operations efficiently.
Full-Text	High	High	Optimized for text search, full-text indexes efficiently handle both AND and OR conditions, making them suitable for complex text queries.

[staab]

should I merge this again 😂

[vitorpamplona]

should I merge this again 😂

Wait!!! We need to mindlessly debate until everybody gets tired and implement the original concept anyway.

[hzrd149]

I don't have anything to debate, but I felt it was necessary to open another PR because the old one was "merged" and the changes were reverted

So this is a second attempt to merge this NIP or at least continue whatever conversation was happening and tracking implementations

[Semisol]

I support this proposal like last time, mostly.

There seems to be no good reason to object this, and it is backwards compatible by just coying & content into # as well and doing client-side postfiltering

New field for NIP-11.limitations: max_tags_per_and and max_tags_and

Replace this with one max_and_entries. Internally, the performance difference of

• &p: [a, b] + &e: [c]
• &p: [a, b, c]

is zero as the different number of tags doesn't matter, only total conditions to check.

---

FYI: Nostr.land is going to implement next release, as stated in last discussion, not sure why it got marked as implemented yet.

[scsibug]

This was already implemented in nostr-rs-relay, but is now in the 0.10.0 release.

[mattn]

Any update on this ?

[vitorpamplona]

We need to merge. There is already too much support for this not to be "official".

[alexgleason]

What client specifically needs this?

[dskvr]

@alexgleason There are clients listed in the PR body.

---

TLDR: NIP-91 allows relays to give the client the needle it is looking for instead of arbitrarily limiting the protocol so that a relay must return an entire haystack n times so that a client can find the needle.

Recap for anyone watching

• This NIP presents absolutely zero risk to any relay, client or library. It is completely backwards compatible and not a single properly implemented relay will be affected.
• NIP-91 codifies a common implementation pattern that clients have to do all the time into a filter operand and helps slowly moves the network to simpler implementation patterns, less bandwidth consumption for all parties and generally more useful and performant clients. It presents zero risk to NIP-01. It is idempotent by design.
• My original assertion 2 years ago was: It benefits all parties, relay operators, client developers and users. In 2 years there has been exactly 0 arguments to disprove this, only actual implementations (proof of work) that definitively prove the original assertions. Most relay developers have reported that NIP-91 is more performant (which again, benefits everyone)
• NIP-91 presents ZERO slippery-slope risk, it is intentionally limited in scope and follows existing philosophy for operands. This is far from a "crappy barely functional mysql" it just allows for relays to opt-in to handling intersections so that a request's filter can consume less clock-time and less bandwidth and in turn simplify client development
• There are use cases littered through both PRs. AND operands have existed for decades and have never been this contentious. NIP-17 is the most obvious one, but since there are infinite schemas, and there many single-letter tags where you want to know an event has two or more of them simultaneously defined, the answer is there are an infinite number of use cases.

Summary

1. Solves a real problem for ALL parties.
2. Is opt-in, voluntary and idempotent.
3. Has amongst the widest support of the majority of NIPs before they were merged
4. nostr-rs-relay has NIP-91 support and is amongst the most widely deployed relay softwares.

[alexgleason]

I understand what problems it solves technically. We need to start thinking about Nostr from a product standpoint.

I want to know what amazing projects people are actively working on that are blocked by not having this.

Knowing that will help me justify why I should be upgrading my relay and all my tooling right now.

[dskvr]

It’s hard to quantify what’s blocked because the missing functionality discourages people from exploring the use cases in the first place.

Practical Examples of where it is needed:

&p [NIP-17]: return only messages involving all parties, not every event involving any of the desired #p tags
&t return kind 1 notes matching specific combinations of hashtags
&t [NIP-35]: return “4k movies,” not torrents tagged “4k” or “movie.”

IMO the real question is:

What innovations are we arbitrarily discouraging from existing?
Not: What in-flight application developments specifically are we blocking?

Once developers realize this basic operation isn’t available, they either settle for local intersection, which will still happen with NIP-91, or abandon the idea entirely.

NIP-91 moves us toward optimizing this case. It likely enables richer meta-driven interactions while reducing bandwidth, battery use, relay CPU time, and client complexity.

Every single one of those benefits mentioned are upstream from any product discussion and therefor could have a direct effect on the resulting product.