Research generic project code search capabilities #3331

Background

XBuilder needs a generic way to search code across user projects. The first expected consumer is the admin interface, but this issue tracks the generic project code search capability itself.

The backend capability should stay generic and should not be specific to spx or any single analysis workflow.

Scope

• Search source files from XBuilder projects
• Include both text search and syntax-level structural matching in the initial capability
• Clarify whether the searchable scope should include latest project versions, project releases, or both
• Compare text search, syntax-level structural matching, and semantic filtering
• Identify how project metadata, permissions, file versions, content hashes, and index freshness should be represented
• Evaluate possible indexing backends such as PostgreSQL trigram indexes, Zoekt, Tantivy, or OpenSearch
• Evaluate possible structural query syntaxes and matching engines, including Tree-sitter queries and ast-grep-like patterns
• Evaluate the role of Tree-sitter, tree-sitter-xgo, and XGo semantic analysis

Current findings

• Project and release file collections are stored as path -> universal URL maps. See the files schema and examples in docs/openapi.yaml
• Text file content may be stored inline as data: URLs, while larger files or binary assets may be referenced through kodo:// URLs
• The indexing pipeline should resolve file content through universal URL handling rather than assuming a single storage backend
• PostgreSQL is a good fit for metadata, permissions, index state, file versions, content hashes, and extracted facts
• PostgreSQL may be enough for an early low-volume implementation, but full project-wide code search may need a dedicated code search index
• Text search can provide candidate retrieval for syntax-level structural matching
• Tree-sitter can provide concrete syntax trees for language-aware structural matching
• tree-sitter-xgo is likely the preferred foundation for XGo structural matching if the first version uses Tree-sitter-compatible query or pattern tooling
• XGo semantic analysis is still needed for resolved symbols and classfile-specific semantics
• Syntax-level structural matching and semantic filtering should be treated as separate layers

Open questions

• What is the first indexed scope: latest project versions, project releases, or both?
• What file types should be indexed initially?
• Should the index store full source content, snippets, or only references back to the original file collection?
• What is the minimum useful syntax-level structural matching capability for the first version?
• Should the first structural query syntax be ast-grep-like, Tree-sitter query based, or a smaller internal subset?
• What should be done if tree-sitter-xgo is not mature enough when the backend search work starts?
• Which index backend is appropriate for the expected data volume and update rate?
• How should index freshness and rebuild failures be surfaced to admins?

References

• PostgreSQL pg_trgm
• Zoekt
• Tree-sitter
• ast-grep pattern syntax

Non-goals

• Do not define a framework-specific API usage reporting feature here
• Do not make the backend API specific to spx
• Do not require resolved-symbol matching as part of the first syntax-level structural search milestone