diff --git a/AGENTS.md b/AGENTS.md index 8fabf029..649ea0a7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ Top-level modules: - `example_rules/` — sample SML rules and YAML config. - `example_atproto_rules/` — sample SML rules paired with `example_atproto_plugins/`. -Reference files: `docs/DEVELOPMENT.md` (setup), `example_plugins/src/register_plugins.py` (plugin patterns), `example_plugins/src/services/labels_service.py` (labels service example). +Reference files: `docs/development/local.md` (setup), `example_plugins/src/register_plugins.py` (plugin patterns), `example_plugins/src/services/labels_service.py` (labels service example). ## Design @@ -159,7 +159,7 @@ uv tool run fawltydeps --check-unused --pyenv .venv - Do not disable lint or type rules to silence errors. Fix the underlying issue, or use a narrowly-scoped `# noqa: ` / `# type: ignore[]` with a comment explaining why. - Before adding a new dependency, check it for known CVEs and confirm the license is compatible with `LICENSE.md`. - Do not commit generated protobuf files from an untrusted toolchain; always regenerate via `./gen-protos.sh`. -- Default Docker bindings are `127.0.0.1`; do not change bind addresses without explicit instruction (see `docs/DEVELOPMENT.md` §6). +- Default Docker bindings are `127.0.0.1`; do not change bind addresses without explicit instruction (see `docs/development/local.md` §6). ## Code review diff --git a/CHANGELOG.md b/CHANGELOG.md index 80dbbfb6..720c5c6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ For more information about each release including git tags and artifacts, see [R - `GetActionId()` stdlib UDF for retrieving the current action ID from execution context ([#327](https://github.com/roostorg/osprey/pull/327) by [@julietshen](https://github.com/julietshen)) - `ParseInt` UDF for converting numeric strings to integers ([#190](https://github.com/roostorg/osprey/pull/190) by [@bealsbe](https://github.com/bealsbe)) - `StringSlice` UDF for extracting substrings by index range ([#189](https://github.com/roostorg/osprey/pull/189) by [@bealsbe](https://github.com/bealsbe)) -- `InExperiment` UDF for checking experiment membership ([#203](https://github.com/roostorg/osprey/pull/203) by [@bealsbe](https://github.com/bealsbe)) +- `Experiment` and `ExperimentWhen` UDFs for checking experiment membership ([#203](https://github.com/roostorg/osprey/pull/203) by [@bealsbe](https://github.com/bealsbe)) - Per-action health metrics in the executor ([#191](https://github.com/roostorg/osprey/pull/191) by [@cmttt](https://github.com/cmttt)) - Option to suppress cached errors to reduce metric bloat ([#180](https://github.com/roostorg/osprey/pull/180) by [@lithium-powered](https://github.com/lithium-powered)) - Experimental asyncio-native worker with metrics and engine/coordinator improvements ([#341](https://github.com/roostorg/osprey/pull/341) by [@cmttt](https://github.com/cmttt)) diff --git a/README.md b/README.md index 0a1ea1a3..2ed28b3c 100644 --- a/README.md +++ b/README.md @@ -2,20 +2,31 @@ # Osprey -**Automate the obvious and investigate the ambiguous.** High-performance safety rules engine for real-time event processing at scale. +**Automate the obvious and investigate the ambiguous.** Osprey is a safety rules engine and investigation console for real-time event processing at scale. -- take action based on user behavior -- combine actions with human written rules -- let operators query human actions and past decisions -- perform investigations or write new rules based on decisions +Your platform streams events to Osprey, and human-written rules evaluate each one as it arrives—taking automatic action, applying labels to the entities involved, and sending verdicts and custom effects to your own systems. Analysts then query and chart the results to spot patterns, investigate, and turn what they find into new rules. Originally developed internally at [Discord](https://discord.com/) to combat spam, abuse, botting, and scripting across its platform, Osprey has been open-sourced to help other platforms facing similar challenges. -Osprey is an event stream decisions engine and analysis UI designed to investigate and take automatic action on events and their properties as they happen in real-time. Originally developed internally at [Discord](https://discord.com/) to combat spam, abuse, botting, and scripting across its platform, Osprey has been open-sourced to help other platforms facing similar challenges. +![The Query page with an SML filter for post-creation events and its history of past queries, beside a timeseries chart showing matching event volume in fifteen-minute buckets](docs/images/query-and-charts.png) -![Osprey UI sample](docs/images/query-and-charts.png) +Rules are written in SML, Osprey's structured rule language, and extended with user-defined functions (UDFs). Osprey tracks state across events by labeling entities when you provide a labels service backend; see [labels_service.py](./example_plugins/src/services/labels_service.py) for a Postgres-backed example. -Osprey is a library for processing actions through human-written rules and outputting verdicts & custom effects back to configurable output sinks. It evaluates events using structured rule logic (SML) that is extendable via user-defined functions (UDFs). Osprey can also track state across events by labelling entities if implementers provide a labels service backend (see [labels_service.py](./example_plugins/src/services/labels_service.py) for a Postgres-backed labels service example). +Osprey is built for engineers and Trust & Safety teams who want to explore, test, and integrate it into their platform for incident response and investigations. [Read more about user research and personas](docs/research-personas.md). -Osprey is built for engineers and Trust & Safety teams who want to explore, test, and integrate its core capabilities into their platform for incident response and Trust & Safety investigation. [Read more about user research and personas](docs/user_personas.md). +## Try it + +If you have Docker with Compose v2, one command brings up the full stack with sample data and opens the UI on a pre-filled query: + +```sh +./demo.sh +``` + +Or, without cloning the repo first: + +```sh +curl -sSL https://raw.githubusercontent.com/roostorg/osprey/main/demo.sh | bash +``` + +The [Getting Started guide](https://roostorg.github.io/osprey/latest/development/) explains what the demo starts, what to try in the UI, and how to shut it all down. The rest of the documentation lives at [roostorg.github.io/osprey/latest](https://roostorg.github.io/osprey/latest/). ## Adopters @@ -28,13 +39,11 @@ Using Osprey and want to add your project/organization to this list? [Open a pul ## Development -- See the [development guide](./docs/development/) for comprehensive development setup and workflow documentation -- All code changes should pass linting (Ruff) and type checking (MyPy) -- Pre-commit hooks automatically run on each commit to maintain code quality +See the [development guide](./docs/development/) for development setup and workflow documentation, including the linting, type checking, and pre-commit hooks that changes are expected to pass. -## Join Us +## Join us -Writing code is not the only way to help the project. Reviewing pull requests, answering questions to help others on mailing lists or issues, providing feedback from a domain expert perspective, organizing and teaching tutorials, working on the website, improving the documentation, are all priceless contributions. +Writing code is not the only way to help the project. Reviewing pull requests, answering questions in issues and discussions, providing feedback from a domain expert perspective, teaching tutorials, and improving the documentation are all priceless contributions. - Join us in [our Discord server](https://discord.gg/5Csqnw2FSQ) - Join our [newsletter](https://roost.tools/#get-started) for more announcements and information @@ -42,7 +51,7 @@ Writing code is not the only way to help the project. Reviewing pull requests, a _[ROOST](https://roost.tools) (Robust Open Online Safety Tools) is a non-profit organization that brings together expertise, resources, and investments from major technology companies and philanthropies to build scalable, interoperable safety infrastructure for the AI era._ -### Feedback Wanted +### Feedback wanted This is a working system, not a prototype. Try it locally, connect your data, write some rules, and tell us what's missing for your use case. We're particularly interested in: @@ -51,8 +60,8 @@ This is a working system, not a prototype. Try it locally, connect your data, wr - Missing detection capabilities or response actions you need - API improvements that would make adoption easier for your team -Your experimentation feedback will directly shape future priorities and help us build the most useful Trust & Safety tooling for the community. +Your feedback will directly shape future priorities and help us build the most useful Trust & Safety tooling for the community. ## Recognition -Discord uses Osprey to quickly detect and remove new types of harm that put users at risk. Rather than leaving other platforms to build similar tools from scratch, ROOST and Discord have open-sourced this powerful rule engine in collaboration with [internet.dev](https://internet.dev/) to make it available for anyone who needs it. +Discord uses Osprey to quickly detect and remove new types of harm that put users at risk. Rather than leaving other platforms to build similar tools from scratch, ROOST and Discord have open-sourced Osprey in collaboration with [internet.dev](https://internet.dev/) to make it available for anyone who needs it. diff --git a/demo.sh b/demo.sh index b00e4565..d833bcaf 100755 --- a/demo.sh +++ b/demo.sh @@ -298,17 +298,14 @@ echo -e " • Osprey UI: ${GREEN}${UI_URL}${NC}" echo -e " • Druid Console: ${GREEN}http://localhost:8888${NC}" echo -e " • Osprey API: ${GREEN}http://localhost:5004${NC}" echo "" -echo -e "${BLUE}Demo Rules Active:${NC}" -echo -e " • ContainsHello - Bans users who say 'hello'" -echo -e " • LazyPostRule - Labels posts with 'lazy' as low_effort" -echo -e " • QuickPostRule - Labels posts with 'quick' as potential_bot" -echo -e " • FoxPostRule - Bans users who say 'fox' (spam pattern)" +echo -e "${BLUE}Demo Rule Active:${NC}" +echo -e " • ContainsHello - Bans users who post the word 'hello' and labels them 'meow'" echo "" echo -e "${BLUE}What to Demo:${NC}" echo -e " 1. Event Stream - See processed events with rule matches" echo -e " 2. TopN Panel - See users grouped by labels/bans" echo -e " 3. Timeseries - See event volume over time" -echo -e " 4. Query Filter - Try: LazyPostRule == True" +echo -e " 4. Query Filter - Try: ContainsHello == True" echo -e " 5. Rules Viz - View the rule dependency graph" echo "" echo -e "${YELLOW}Opening Osprey UI in browser...${NC}" diff --git a/docs/README.md b/docs/README.md index dd0ae1e5..d34f98c8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,14 +1,22 @@ # Welcome -Welcome to the Osprey documentation! Visit [Osprey on GitHub](https://github.com/roostorg/osprey#readme) for source code and project information. These docs are split into a few guides, depending on who you are and what you're looking for: +Osprey is an open source rules engine and investigation console for trust and safety teams: your platform streams events to it, your rules evaluate each one in real time, and your analysts query, chart, and act on the results. Originally built at Discord and running in production at Bluesky, Osprey is now developed in the open by [ROOST](https://roost.tools) and the community; visit [Osprey on GitHub](https://github.com/roostorg/osprey#readme) for source code and project information. -- [User Guide](user/): learn about Osprey, its functionality, and the user interface -- [Development Guide](development/): get Osprey running and learn how to navigate the code +These docs are split into a few guides, depending on who you are and what you're looking for: -See additional topics for reference: +- [User Guide](user/): learn the investigation UI—querying events, labeling entities, and running bulk jobs. -- [Writing Rules](rules.md) -- [User Research & Personas](research-personas.md) +- [Development Guide](development/): run a local development environment and change Osprey itself. + +- [Integration Guide](integration/): connect Osprey to your platform—get data in and out, and extend it with plugins. + +- [Concepts](concepts.md): learn about the basic concepts and terminology used in Osprey and these docs. + +The docs are versioned; other versions are available at the [documentation site index](https://roostorg.github.io/osprey/). + +[![The Query page with an SML filter for post-creation events and its history of past queries, beside a timeseries chart showing matching event volume in fifteen-minute buckets](images/query-and-charts.png)](images/query-and-charts.png) + +The fastest way to get a feel for Osprey is the one-command demo in [Getting Started](development/), which brings up the full stack on live sample data in a few minutes. ## Contributing diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 7ad17d18..97d47845 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -10,23 +10,32 @@ - [Query Syntax](user/investigate/query-syntax.md) - [Manage](user/manage.md) - [Operate](user/operate.md) - - [Appearance](user/appearance.md) --- - [Development Guide](development/README.md) - - [Workflow](development/workflow.md) - - [Tools Overview](development/tools.md) + - [Local Development](development/local.md) + - [Development Workflow](development/workflow.md) + - [Development Tools](development/tools.md) + - [CLI Reference](development/cli-reference.md) - [Troubleshooting](development/troubleshooting.md) - [IDE Setup](development/ide.md) - [Release Process](development/releases.md) --- +- [Integration Guide](integration/README.md) + - [Data Flow](integration/data-flow.md) + - [Integrations & Plugins](integration/integrations.md) + +--- + # Concepts & More -- [Writing Rules](rules.md) -- [Integrations & Plugins](integrations.md) +- [Concepts](concepts.md) +- [Writing Rules](rules/README.md) + - [Examples](rules/examples.md) + - [Why a Bespoke Language?](rules/why-sml.md) - [User Research & Personas](research-personas.md) --- diff --git a/docs/book.toml b/docs/book.toml index c663872b..202fd41f 100644 --- a/docs/book.toml +++ b/docs/book.toml @@ -6,11 +6,19 @@ src = "." [build] build-dir = "./book" +create-missing = false [output.html] -site-url = "/osprey/" +default-theme = "roost" +preferred-dark-theme = "roost" +additional-css = ["theme/css/roost.css", "theme/css/roost-hljs.css", "theme/css/content.css"] edit-url-template = "https://github.com/roostorg/osprey/edit/main/docs/{path}" git-repository-url = "https://github.com/roostorg/osprey" +site-url = "/osprey/" [output.html.print] enable = false + +[output.html.redirect] +"/integrations.html" = "integration/integrations.html" +"/rules.html" = "rules/index.html" diff --git a/docs/concepts.md b/docs/concepts.md new file mode 100644 index 00000000..89c50258 --- /dev/null +++ b/docs/concepts.md @@ -0,0 +1,58 @@ +# Concepts + +Osprey watches a stream of events happening on your platform, runs each one through your rules in real time, and records what it found so analysts can query and act on it. This page defines the terms the rest of these docs (and the UI itself) use. They're listed in the order data flows through Osprey, and later concepts build on earlier ones, so it's worth a read from start to finish. + +Throughout the docs are examples of moderating a small social network where users create posts; the same concepts adapt to any platform where people interact. + +## Events (and actions) + +Anything that happens on your platform—e.g. someone registers an account, creates a post, sends a message, or reacts to content—can be sent to Osprey as an **event**, as it happens. Each event arrives with a name, like `create_post`, and whatever JSON data your platform sent along, e.g. the author's user ID and the post text "hello world." + +> [!NOTE] +> You may also come across the term **action**: Osprey's submission API and rules engine historically call each incoming event an action, which is why the query bar filters on `ActionName` even though the UI's live feed is the Event Stream. In this context, they're the same: one event in, one `ActionName` recorded. + +## Features + +A **feature** is a named value that your rules extract from each event: `PostText`, `UserId`, `AccountAgeSeconds`. Features are what you query on; every feature your rules define is queryable by name, unless its name starts with an underscore, which keeps it private to the rule file that defined it. + +For a "hello world" post, your rules might extract `UserId`, `PostText`, and `EventType` as features. + +## Entities + +An **entity** is a feature that uniquely identifies something on your platform: a user ID, an email address, an IP address. Entities are declared with a type (like `User`) so Osprey knows which values refer to the same thing across many events. That identity is what makes [labels](#labels) possible, and it's how selecting an entity in the UI can open a view of everything Osprey knows about it. + +In our example, `UserId` would be declared as an entity: the same user posts many times, and you want Osprey to remember them. + +## Rules + +A **rule** is a named condition over features, written in Osprey's rule language, SML. Rules are evaluated against every event as it arrives. On their own, rules just return `True` or `False` (which is itself queryable, like any feature), or `None` when a signal it depends on is missing. Rules can be wired to [effects](#effects). + +In our example, a rule might cover "the event is a post creation and the post text contains 'hello'." + +Rules live as code in your Osprey deployment alongside the Osprey code itself. See [Writing Rules](rules/) for more detail. + +## Effects + +An **effect** is what a matched rule does beyond evaluating; e.g. ban the user, add a label to an entity, declare a verdict. Effects are recorded on the event's result, so an investigator can always see afterward exactly what Osprey did and why. + +When the demo's `ContainsHello` rule matches our "hello world" post, it fires two effects: a ban for the author, and a `meow` label added to them. + +## Labels + +A **label** is a tag on an entity that persists across events—Osprey's memory. Rules add and remove labels as an effect, and can also _check_ labels, so past decisions inform future ones: "flag this post if its author was previously labeled a spammer." You can also add or remove labels by hand from the UI, one entity at a time or in bulk. + +Labels have a name, the entity types they apply to, and a connotation (positive, negative, or neutral). See [Labels](user/investigate/labels.md) for more information about how they're used for investigations. + +## Verdicts + +A **verdict** is Osprey's answer when a caller is waiting for one. Most deployments feed Osprey from a queue and read results the same way, but a service can also submit an event synchronously and get a response; rules declare verdict strings, and those—along with any labels applied—are what the caller gets back. If your deployment only consumes results asynchronously, you may never handle a verdict directly, though you can spot them among an event's recorded features in the UI. + +## Results, and where they surface + +Every processed event produces an execution result: the extracted features, the rules that matched, and the effects that fired. Results are indexed for querying—that's what the whole [Investigate](user/investigate/) side of the UI reads. The query bar filters results by feature (`ContainsHello == True`), charts aggregate them over time, Top N groups them by any feature, and the Event Stream shows them one by one as they happen. Clicking any entity opens its details: current labels and its history on your platform. + +How results get from the engine to those views (and into your own systems) is plumbing your developers control; the [Data Flow](integration/data-flow.md) page covers it. + +## Where to go from here + +Try these ideas out on live sample data by running the [demo](development/), then head to [Investigate](user/investigate/) to learn the query workflow, or [Writing Rules](rules/) when you're ready to automate a decision of your own. diff --git a/docs/development/README.md b/docs/development/README.md index 80d4956e..d1986f01 100644 --- a/docs/development/README.md +++ b/docs/development/README.md @@ -1,159 +1,61 @@ -# Development Guide +# Getting Started -Set up a development environment for Osprey. +Quickly see Osprey working with sample data and a real rule using the demo script. It needs no code or configuration, and everything cleans up with a single command afterward. For a full development environment—running services individually, making changes, and debugging—see [Local Development](local.md). -## Prerequisites +To run the demo: -- **Operating System**: macOS, Linux, or Windows (with WSL recommended) -- **[Python](https://www.python.org/) 3.11 or higher** (check with `python --version`) -- **[Git](https://git-scm.com/)** for version control -- **[uv](https://docs.astral.sh/uv/)** for Python package management -- **[Node.js](https://nodejs.org/en/download/) 22+** for the UI (Corepack ships with Node and auto-resolves pnpm from `osprey_ui/package.json`'s `packageManager` field; no separate pnpm install needed) +1. **Ensure you have prerequisites** installed: Docker with the Compose v2 plugin, and the Docker daemon running. The script checks these before doing anything. -## Project Setup + ```sh + docker --version && docker compose version + ``` -### 1. Clone the Repository + The stack also needs a number of free ports; the script checks them up front and names any that are taken. The ones you'll actually visit are `5002` (Osprey UI) and `8888` (Druid console); the worker and UI API use `5001` and `5004`. -```bash -git clone git@github.com:roostorg/osprey.git -cd osprey -``` - -### 2. Install Dependencies - -```bash -# Install all dependencies including development tools -uv sync -``` - -This command will: - -- Create a virtual environment automatically -- Install all production dependencies -- Install development dependencies (ruff, mypy, pre-commit) automatically -- Use the locked versions from `uv.lock` for reproducible builds - -**Note**: `uv sync` includes development dependencies by default. Use `uv sync --no-dev` if you only want production dependencies. - -### 3. Set Up Pre-commit Hooks - -```bash -uv run pre-commit install -``` - -This installs git hooks that automatically run code quality checks before each commit. - -### 4. Verify Setup - -Run these commands to ensure everything is working correctly: - -```bash -# Check linting configuration -uv run ruff check - -# Check formatting -uv run ruff format --diff - -# Run type checking -uv run mypy . - -# Test pre-commit hooks -uv run pre-commit run --all-files -``` - -**Expected Results:** - -- Ruff should report "All checks passed!" or show specific issues to fix -- MyPy should run without errors -- Pre-commit should run all hooks successfully - -### 5. Getting Started - -```bash -docker compose up -d -``` - -or using the wrapper script - -```bash -./start.sh -``` +2. **Run the script** from a clone of the repo: -This starts up many services, including: -- **Osprey Worker**: The main engine that processes input events given the rules and UDFs - - **Test Data Producer**: Optional with `--profile test_data` -- **Osprey UI**: Frontend service that hosts the react code for the web interface and communicates to the UI API -- **Osprey UI API**: Backend service that provides data and functionality to the web interface -- **Kafka** (KRaft mode): Message streaming for user generated events -- **Postgres**: A database that the Worker, UI API, and Druid use for various reasons, such as the Postgres-backed Labels Service (in the example plugins) -- **Druid**: A database that consumes Osprey Worker outputs to power the UI API for real-time querying + ```sh + ./demo.sh + ``` -Alternatively, you can start Osprey with `osprey-coordinator`, refer to the [Coordinator README](../example_docker_compose/run_osprey_with_coordinator/README.md) for more information + Or without cloning anything first—this clones the repo into `./osprey-demo` for you: -### 6. (Optional) Open ports for the UI/UI API + ```sh + curl -sSL https://raw.githubusercontent.com/roostorg/osprey/main/demo.sh | bash + ``` -By default, the `docker-compose.yaml` binds running services to `127.0.0.1`. If you are running the docker compose on a headless machine, you may need to modify this configuration and/or make changes to your firewall, specifically for ports `5002` and `5004`. + The script starts the whole stack (Kafka, Druid, PostgreSQL, MinIO, and Osprey's own worker, API, and UI—more than a dozen containers in all) plus a test-data producer that sends one synthetic post event per second. If it finds services from a previous run, it asks before stopping them; volumes from earlier demo runs are removed either way, so each demo starts from clean state. Expect the first run to spend a few minutes pulling and building images. -For example, if you use Tailscale to access your Osprey instance, you may change `127.0.0.1:5002:5002` to `:5002:5002`. Alternatively, if you wish for your instance to be accessible from the public internet, you may set it simply to `5002:5002` to bind to `0.0.0.0`. +3. **Wait for the "Demo Ready!" banner.** The script waits for every service to report healthy, then for the first events to flow through, so this takes a couple of minutes. When it's done, it opens the UI in your browser (or prints the URL to open yourself), pre-filled with a query for the last day of events. -Be aware that some firewalls like iptables/UFW do _not_ prevent access to ports being used by Docker networking. Not explicitly setting a bind address with only UFW as a firewall will not prevent access from the public internet unless [properly configured](https://github.com/chaifeng/ufw-docker). +## What to try -### 7. Access the Application +The demo ruleset has one rule, `ContainsHello`: any post containing the word "hello" gets its author banned and labeled `meow`. The producer builds five-word posts from a small word pool, so roughly a third of the generated posts trigger it. -The UI will automatically connect to the backend services running in Docker containers. +- **Watch the event stream.** The right panel shows events as they're processed. Click one that matched `ContainsHello` and look at its extracted features and effects. +- **Filter with a query.** Enter `ContainsHello == True` in the query bar to see only the posts that fired the rule; see [Query Syntax](../user/investigate/query-syntax.md) for what else you can express. +- **Group with Top N.** The pre-filled query groups by `UserId`, showing which synthetic users have been banned the most. +- **Open the Rules Visualizer** from the navigation bar to see the dependency graph between the demo's features and rule. -- Osprey UI: [localhost:5002](http://localhost:5002) -- Backend API: [localhost:5004](http://localhost:5004) -- Worker Service: [localhost:5001](http://localhost:5001) +The [User Guide](../user/) covers the full investigation workflow, and [Writing Rules](../rules/) explains the SML behind `ContainsHello` and how to write your own rules. -## Plugins +## Stopping the demo -In Osprey, UDFs and output sinks are designed to be easily portable. This is done through a plugin system based on pluggy. An example plugin package has been provided for reference, see `example_plugins/register_plugins.py`: +From the repo folder (or `osprey-demo/` if the script cloned for you): -```python -@hookimpl_osprey -def register_udfs() -> Sequence[Type[UDFBase[Any, Any]]]: - # Register custom user-defined functions - -@hookimpl_osprey -def register_output_sinks(config: Config) -> Sequence[BaseOutputSink]: - # Define output destinations - # By default it prints the execution results to the console - -@hookimpl_osprey -def register_ast_validators() -> None: - # Register AST validators +```sh +docker compose --profile test_data down -v ``` -### Available hooks - -Implement any subset of these in your plugin's `register_plugins.py`: - -| Hook | Returns | Notes | -| --- | --- | --- | -| `register_udfs` | `Sequence[Type[UDFBase]]` | Custom user-defined functions. | -| `register_output_sinks` | `Sequence[BaseOutputSink]` | Where execution results go. | -| `register_ast_validators` | `Sequence[Type[BaseValidator]]` | Extra SML validators. | -| `register_action_proto_deserializer` | `ActionProtoDeserializer \| None` | Custom action proto → JSON. | -| `register_input_stream` | `BaseInputStream` | Single-provider (`firstresult`). | -| `register_execution_result_store` | `ExecutionResultStore` | Single-provider (`firstresult`). | -| `register_labels_service_or_provider` | `LabelsServiceBase \| LabelsProvider` | Single-provider (`firstresult`). | - -## Rules - -Rules are written in SML, some examples are provided in `example_rules/` with YAML config, the rules are mounted to the worker processes when the containers start via environment variables. ex: - -```bash -OSPREY_RULES=./example_rules uv run python3.11 osprey_worker/src/osprey/worker/cli/sinks.py run-rules-sink -``` +This stops every container and deletes the volumes, including all generated demo data. -For more about rules, see [Writing Rules](../rules.md). +## If something goes wrong -## Test Data +The script is a convenience wrapper; you can run the same stack directly and watch the logs: -Generate sample JSON actions: -```bash -docker compose --profile test_data up osprey-kafka-test-data-producer -d +```sh +docker compose --profile test_data up -d +docker compose logs --follow osprey-worker ``` -Produces user login events with timestamps, user IDs, and IP addresses to `osprey.actions_input` topic. +Then open once things settle. [Troubleshooting](troubleshooting.md) covers common failure modes, and [Local Development](local.md) documents each service the demo starts. diff --git a/docs/development/cli-reference.md b/docs/development/cli-reference.md new file mode 100644 index 00000000..8ad47c18 --- /dev/null +++ b/docs/development/cli-reference.md @@ -0,0 +1,109 @@ +# CLI Reference + +Osprey includes the following command-line tools installed as console scripts; run them with `uv run ` from the repo root, or `` inside an environment where the package is installed. + +## osprey-cli + +General-purpose worker administration including pushing rules, opening a debug shell, and applying labels manually. + +### osprey-cli push-rules + +```sh +osprey-cli push-rules RULES_PATH [--dry-run/--no-dry-run] [--suppress-warnings] +``` + +Validates the rules at `RULES_PATH` and pushes them. Use `--dry-run` to validate without pushing. Exits non-zero if validation fails. + +### osprey-cli compute-and-upload-dependencies-mapping + +```sh +osprey-cli compute-and-upload-dependencies-mapping RULES_PATH [--suppress-warnings] +``` + +Computes the dependency graph for the rules at `RULES_PATH` and uploads it; this is what powers the [Rules Visualizer](../user/manage.md#rules-visualizer). + +### osprey-cli shell + +```bash +osprey-cli shell [-i / --auto-import / --no-auto-import] +``` + +Opens an interactive shell (IPython if installed, otherwise a fallback `code.InteractiveConsole`) with `labels`, `access_audit_log`, and `stored_execution_result` storage modules pre-imported, plus `EntityT`, `EntityLabelMutation`, and `LabelStatus`. With `--auto-import` (the default), it also imports every model class it can find under `osprey_lib`. Useful for interactively inspecting stored data. + +### osprey-cli apply-label + +```bash +osprey-cli apply-label ENTITY_TYPE ENTITY_ID LABEL_NAME LABEL_STATUS \ + [--reason REASON] [--description DESCRIPTION] [--expire-instantly] +``` + +Manually applies a label to a single entity. Mainly intended for debugging or importing individual labels from an external source. `--reason` defaults to `CliLabelMutationWithoutEffects`; `--description` defaults to `"Manually changed from the command line for debugging."`. `--expire-instantly` makes the label expire almost immediately rather than persisting. + +This requires a labels provider to be configured for the Osprey instance; it fails with an assertion error otherwise. + +### osprey-cli bulk-apply-label + +```bash +osprey-cli bulk-apply-label ENTITY_TYPE ENTITY_IDS_FILE_PATH LABEL_NAME LABEL_STATUS \ + [--reason REASON] [--description DESCRIPTION] [--expire-instantly] +``` + +Same as `apply-label`, but reads entity IDs (one per line) from `ENTITY_IDS_FILE_PATH` and applies the label to all of them, printing progress as it goes. This is the CLI path for importing bulk label lists from external sources; see also the UI-driven [Bulk Actions](../user/operate.md#bulk-actions) workflow. + +## osprey-stress + +Installed by `osprey_worker` (`osprey.worker.stress.cli:main`). Runs an end-to-end stress test against a live Osprey worker: produces synthetic events, consumes the resulting execution results, and reports drop rate and latency. Useful for validating dependency bumps, measuring throughput regressions, and gating CI on pipeline health. + +### osprey-stress run + +```bash +osprey-stress run \ + --events 10000 --rate 1000 \ + --threshold-drop-rate 0.01 --threshold-p95-ms 500 \ + --report json +``` + +Produces `--events` synthetic events (default `1000`) at `--rate` events/second (default `100.0`) to the input topic, then waits up to `--drain-seconds` (default `30.0`) for the worker to finish processing before reporting. + +Common flags: +- `--report {human,json}`: output format (default `human`) +- `--verbose`: emit periodic progress lines to stderr, with `--verbose-interval-seconds` (default `2.0`) controlling frequency +- `--bootstrap-servers` (default `localhost:9092`), `--input-topic` (default `osprey.actions_input`), `--output-topic` (default `osprey.execution_results`) +- `--threshold-drop-rate` and `--threshold-p95-ms`: if set, the command exits non-zero when the observed drop rate or p95 latency breaches the threshold, so it can gate a CI job + +### osprey-stress measure + +```bash +osprey-stress measure [--duration SECONDS] [--report {human,json}] +``` + +Reserved for open-loop measurement against externally-produced events, such as the JetStream input stream sample ([#236](https://github.com/roostorg/osprey/pull/236)). The measurement layer isn't implemented yet, so today it prints a stub message and exits non-zero; use `run` for closed-loop synthetic testing in the meantime. + +## osprey-async-cli + +Installed by `osprey_async_worker` (`osprey.async_worker.cli.main:cli`). **Experimental**: the asyncio-native worker prototype (no gevent, no monkey-patching), for validating whether an asyncio-based executor can replace the gevent one. Not intended for production use yet. + +### osprey-async-cli run + +```bash +osprey-async-cli run --rules-path PATH [--input-file PATH] [--max-concurrent 12] \ + [--with-plugins] [--input-source {file,kafka}] \ + [--kafka-topic TOPIC] [--kafka-bootstrap-servers SERVERS] \ + [--kafka-group-id GROUP] [--kafka-offset-reset {latest,earliest}] +``` + +Runs the async worker against a static rules directory (`--rules-path`, required). By default it uses stdlib UDFs only and prints results to stdout; `--with-plugins` loads the full async plugin system (UDFs, validators, and output sinks registered via the `osprey_async_plugin` entry-point group), which requires external services to be available. + +Input source is controlled by `--input-source`: +- `file` (default): reads JSONL actions from `--input-file`, or runs with no input if omitted (useful for just validating the worker boots) +- `kafka`: consumes from `--kafka-topic` (default `osprey.actions_input`) via `--kafka-bootstrap-servers` (default `localhost:9092`) + +### osprey-async-cli benchmark + +```bash +osprey-async-cli benchmark --rules-path PATH --input-file PATH \ + [--max-concurrent 12] [--iterations 1000] [--warmup 50] +``` + +Benchmarks the async executor against the gevent executor (if `gevent` is importable) using the same rules and input data, running `--warmup` iterations first, then `--iterations` timed iterations, and prints a throughput/latency comparison. + diff --git a/docs/development/ide.md b/docs/development/ide.md index 554da48e..71132e95 100644 --- a/docs/development/ide.md +++ b/docs/development/ide.md @@ -20,3 +20,13 @@ If using VS Code, we recommend you install these extensions for the best develop "python.analysis.typeCheckingMode": "basic" } ``` + +## SML files + +There's no dedicated editor tooling for SML yet, but since SML is a legal subset of Python syntax, associating `.sml` files with Python gets you reasonable highlighting. In VS Code: + +```json +"files.associations": { + "*.sml": "python" +} +``` diff --git a/docs/development/local.md b/docs/development/local.md new file mode 100644 index 00000000..2ae6caea --- /dev/null +++ b/docs/development/local.md @@ -0,0 +1,153 @@ +# Local Development + +Set up a full local development environment for Osprey. To just get up and running quickly with sample data, see [Getting Started](./) instead. + +## Prerequisites + +- **Operating System**: macOS, Linux, or Windows (with WSL recommended) +- **[Python](https://www.python.org/) 3.11 or higher** (check with `python --version`) +- **[Git](https://git-scm.com/)** for version control +- **[uv](https://docs.astral.sh/uv/)** for Python package management +- **[Node.js](https://nodejs.org/en/download/) 22+** for the UI (Corepack ships with Node and auto-resolves pnpm from `osprey_ui/package.json`'s `packageManager` field; no separate pnpm install needed) + +## Project Setup + +### 1. Clone the Repository + +```bash +git clone git@github.com:roostorg/osprey.git +cd osprey +``` + +### 2. Install Dependencies + +```bash +# Install all dependencies including development tools +uv sync +``` + +This creates a virtual environment and installs production and development dependencies (ruff, mypy, pre-commit) at the locked versions from `uv.lock`. Use `uv sync --no-dev` if you only want production dependencies. + +### 3. Set Up Pre-commit Hooks + +```bash +uv run pre-commit install +``` + +This installs git hooks that automatically run code quality checks before each commit. + +### 4. Verify Setup + +Run these commands to ensure everything is working correctly: + +```bash +# Check linting configuration +uv run ruff check + +# Check formatting +uv run ruff format --diff + +# Run type checking +uv run mypy . + +# Test pre-commit hooks +uv run pre-commit run --all-files +``` + +Ruff reports "All checks passed!" (or the specific issues to fix), and mypy and the pre-commit hooks run without errors. + +### 5. Start the Services + +```bash +docker compose --profile test_data up -d +``` + +or using the wrapper script + +```bash +./start.sh --profile test_data up -d +``` + +The `test_data` profile includes a producer that generates sample events; without it the stack still runs, but the UI shows no data until you send events yourself. [Getting Started](./) covers the one-command demo that wraps all of this. + +This starts up many services, including: +- **Osprey Worker**: The main engine that processes input events given the rules and UDFs + - **Test Data Producer**: The `--profile test_data` sample event generator +- **Osprey UI**: Frontend service that hosts the React code for the web interface and communicates with the UI API +- **Osprey UI API**: Backend service that provides data and functionality to the web interface +- **Kafka** (KRaft mode): Message streaming for user-generated events +- **Postgres**: A database that the Worker, UI API, and Druid use for various reasons, such as the Postgres-backed Labels Service (in the example plugins) +- **Druid**: A database that consumes Osprey Worker outputs to power the UI API for real-time querying +- **MinIO**: S3-compatible object storage; the default execution result store in this stack (`OSPREY_EXECUTION_RESULT_STORAGE_BACKEND=minio`) + +Alternatively, you can start Osprey with `osprey-coordinator`; see the [Coordinator README](https://github.com/roostorg/osprey/tree/main/example_docker_compose/run_osprey_with_coordinator) for more information. + +### 6. (Optional) Open ports for the UI/UI API + +By default, the `docker-compose.yaml` binds running services to `127.0.0.1`. If you are running the docker compose on a headless machine, you may need to modify this configuration and/or make changes to your firewall, specifically for ports `5002` and `5004`. + +For example, if you use Tailscale to access your Osprey instance, you may change `127.0.0.1:5002:5002` to `:5002:5002`. Alternatively, if you wish for your instance to be accessible from the public internet, you may set it simply to `5002:5002` to bind to `0.0.0.0`. + +Be aware that some firewalls like iptables/UFW do _not_ prevent access to ports being used by Docker networking. Not explicitly setting a bind address with only UFW as a firewall will not prevent access from the public internet unless [properly configured](https://github.com/chaifeng/ufw-docker). + +### 7. Access the Application + +The UI will automatically connect to the backend services running in Docker containers. + +- Osprey UI: [localhost:5002](http://localhost:5002) +- Backend API: [localhost:5004](http://localhost:5004) +- Worker Service: [localhost:5001](http://localhost:5001) + +## Plugins + +In Osprey, UDFs and output sinks are designed to be easily portable through a plugin system based on pluggy. An example plugin package is provided for reference; see `example_plugins/src/register_plugins.py`. For a full walkthrough with examples (UDFs, sinks, hash-based lookups, ML models, labels service), see [Integrations & Plugins](../integration/integrations.md). + +```python +@hookimpl_osprey +def register_udfs() -> Sequence[Type[UDFBase[Any, Any]]]: + # Register custom user-defined functions + +@hookimpl_osprey +def register_output_sinks(config: Config) -> Sequence[BaseOutputSink]: + # Define output destinations + # By default it prints the execution results to the console + +@hookimpl_osprey +def register_ast_validators() -> Sequence[Type[BaseValidator]]: + # Register AST validators +``` + +### Available hooks + +Implement any subset of these in your plugin's `register_plugins.py`: + +| Hook | Returns | Notes | +| --- | --- | --- | +| `register_udfs` | `Sequence[Type[UDFBase]]` | Custom user-defined functions. | +| `register_output_sinks` | `Sequence[BaseOutputSink]` | Where execution results go. | +| `register_ast_validators` | `Sequence[Type[BaseValidator]]` | Extra SML validators. | +| `register_action_proto_deserializer` | `ActionProtoDeserializer \| None` | Custom action proto → JSON. | +| `register_input_stream` | `BaseInputStream` | Single-provider (`firstresult`). | +| `register_execution_result_store` | `ExecutionResultStore` | Single-provider (`firstresult`). | +| `register_labels_service_or_provider` | `LabelsServiceBase \| LabelsProvider` | Single-provider (`firstresult`). | +| `register_validation_exporter` | `BaseValidationResultExporter \| None` | Optional; publishes experiment/bucket metadata after validation. Single-provider (`firstresult`). | +| `register_label_output_sink` | `BaseOutputSink \| None` | Optional; custom label-mutation sink, replacing the default `LabelOutputSink`. Single-provider (`firstresult`). | + +## Rules + +Rules are written in SML; examples with YAML config are provided in `example_rules/`. Rules are mounted into the worker processes when the containers start, controlled via environment variables; for example: + +```bash +OSPREY_RULES_PATH=./example_rules uv run python osprey_worker/src/osprey/worker/cli/sinks.py run-rules-sink +``` + +For more about rules, see [Writing Rules](../rules/). + +## Test Data + +Generate sample JSON actions: +```bash +docker compose --profile test_data up osprey-kafka-test-data-producer -d +``` + +This produces synthetic post-creation events with timestamps, user IDs, and IP addresses to the `osprey.actions_input` topic; it's the same producer the `test_data` profile starts. diff --git a/docs/development/tools.md b/docs/development/tools.md index 037eb5bd..55313a27 100644 --- a/docs/development/tools.md +++ b/docs/development/tools.md @@ -4,14 +4,7 @@ Familiarize yourself with these development tools to better understand how to wo ## Ruff -Linting and formatting. Replaces Black, isort, Flake8, and other tools. Configuration in `pyproject.toml` under `[tool.ruff]`. - -Key rules enabled: - -- `E`: pycodestyle errors -- `F`: pyflakes -- `I`: isort (import sorting) -- `B006`: flake8-bugbear (mutable default arguments) +Linting and formatting. Replaces Black, isort, Flake8, and other tools. Configuration lives in `pyproject.toml` under `[tool.ruff]`; the enabled rules are pycodestyle errors (`E`), pyflakes (`F`), import sorting (`I`), and flake8-bugbear's mutable-default-arguments check (`B006`). Commands: @@ -31,14 +24,7 @@ uv run ruff check path/to/file.py ## MyPy -Static type checking for Python. Configuration in `pyproject.toml` under `[tool.mypy]`. - -Key features: - -- Pydantic plugin support -- SQLAlchemy plugin support -- Relaxed strict mode (matching legacy codebase) -- Ignores protobuf generated files +Static type checking for Python. Configuration in `pyproject.toml` under `[tool.mypy]` loads the Pydantic and SQLAlchemy plugins, relaxes strict mode to match the older parts of the codebase, and skips generated protobuf files. Commands: @@ -73,7 +59,7 @@ uv run pre-commit run --all-files uv run pre-commit run ruff # Update hook versions -uv run pre-commit autoupgrade +uv run pre-commit autoupdate # Bypass hooks (emergency only) git commit --no-verify diff --git a/docs/development/troubleshooting.md b/docs/development/troubleshooting.md index e23c2b57..fdd3b928 100644 --- a/docs/development/troubleshooting.md +++ b/docs/development/troubleshooting.md @@ -31,7 +31,7 @@ The UI defaults to querying the last 24 hours. If the selected time range is too - Narrow the time range to 1–4 hours centered on when you generated test data - Click the edit icon next to the displayed time range to switch to a custom date/time picker -- Note that Druid's Kafka consumer uses `auto.offset.reset: latest`; it only picks up events produced after `docker compose up` first ran, so events from before that point will not appear regardless of the time range +- Druid's Kafka consumer uses `auto.offset.reset: latest`; it only picks up events produced after `docker compose up` first ran, so events from before that point will not appear regardless of the time range ## Kafka topic disk growth diff --git a/docs/development/workflow.md b/docs/development/workflow.md index 23ed6baf..ff0b2e99 100644 --- a/docs/development/workflow.md +++ b/docs/development/workflow.md @@ -1,92 +1,37 @@ # Development Workflow -## Branch Management +Branch from `main` and name your branch `github_username/short-description` (e.g. `caidanw/fix-database-timeout`). -- **Branch naming convention**: Use `github_username/description` format (e.g., `caidanw/feature-auth`, `caidanw/fix-database-timeout`) -- **Base branch**: Always branch from `main` -- **Create new branch**: `git checkout -b username/feature-name` +## What runs on every commit -## Code Quality Standards +Pre-commit hooks run automatically when you commit; the config is `.pre-commit-config.yaml` at the repo root. The hooks check for filename case conflicts, leftover merge-conflict markers, and forgotten Python debugger calls; validate JSON and TOML; normalize line endings to LF; lint and format Python with Ruff; and type-check with mypy. Changes under `osprey_ui/` also go through Prettier and ESLint. -### Automated Checks +Hooks stop at the first failure, and several fix files in place—if a hook modifies a file, re-stage it and commit again. -Every commit automatically runs: +## Checking your work before you push -1. **Trailing whitespace removal** -2. **End-of-file fixing** -3. **YAML/JSON/TOML validation** -4. **Ruff linting and formatting** +CI runs the same hooks against the whole repo, so you can catch failures ahead of time with: -### Manual Checks +```bash +uv run pre-commit run --all-files +``` -Before pushing, run: +Or run the individual tools directly: ```bash -# Comprehensive linting check +# Lint and format uv run ruff check - -# Format all code uv run ruff format -# Type checking (on specific files/modules) -uv run mypy osprey_worker/src/osprey_worker/lib -# Or you can type check every module (this will happen in CI) -uv run mypy . - -# Run all pre-commit hooks -uv run pre-commit run --all-files +# Type check a specific module, or everything with `uv run mypy .` +uv run mypy osprey_worker/src/osprey/worker/lib ``` -## Commit Standards +## Commit messages -Follow [Conventional Commits](https://www.conventionalcommits.org/) format: +Follow the [Conventional Commits](https://www.conventionalcommits.org/) format. Real examples from this repo's history: ``` -feat: add user authentication system -fix: resolve database connection timeout -docs: update API documentation -refactor: simplify rule evaluation logic +fix(ui): show the event-stream timezone once, not twice +build(deps): remove unused Discord-era Python dependencies ``` - -**Examples:** - -- `feat:` - New features -- `fix:` - Bug fixes -- `docs:` - Documentation changes -- `refactor:` - Code refactoring -- `test:` - Adding or updating tests -- `chore:` - Maintenance tasks - -## Making Changes - -1. **Create a new branch:** - - ```bash - git checkout -b username/feature-name - ``` - -2. **Make your changes** - -3. **Run quality checks:** - - ```bash - uv run ruff check --fix - uv run ruff format - ``` - -4. **Test your changes** (if tests exist) - -5. **Commit your changes:** - - ```bash - git add . - git commit -m "feat: descriptive commit message" - ``` - - Pre-commit hooks will run automatically and may fix formatting issues. - -6. **Push your branch:** - - ```bash - git push origin username/feature-name - ``` diff --git a/docs/integration/README.md b/docs/integration/README.md new file mode 100644 index 00000000..f09278f7 --- /dev/null +++ b/docs/integration/README.md @@ -0,0 +1,8 @@ +# Integration Guide + +Osprey is designed to plug into your platform rather than replace it: your systems send events in, your rules decide, and your systems act on what comes out. This guide covers that seam. + +- [Data Flow](data-flow.md): the conceptual picture—how events get into Osprey, and how results and effects get back out +- [Integrations & Plugins](integrations.md): the how-to—writing UDFs, input and output sinks, a labels service, and ML model integration through the plugin system + +If you're still evaluating Osprey, start with the one-command demo in [Getting Started](../development/); if you want to change Osprey itself rather than connect to it, see the [Development Guide](../development/). diff --git a/docs/integration/data-flow.md b/docs/integration/data-flow.md new file mode 100644 index 00000000..a727eb94 --- /dev/null +++ b/docs/integration/data-flow.md @@ -0,0 +1,97 @@ +# Data Flow + +How events actually reach Osprey, how results come back out, and what to change (not fork) if your platform doesn't look like the default docker-compose setup. + +## Architecture at a glance + +At its core, Osprey is a pipeline: events come in, get evaluated against your SML rules, and verdicts/effects go out. + +``` + Kafka topic(s) / PubSub / gRPC + │ + ▼ + Osprey Coordinator (Rust, optional) + │ + ▼ + Osprey Worker (Python) + evaluates rules against the + event, produces an + ExecutionResult + │ + ▼ + Output sink(s) + execution result store + (stdout, Kafka, Postgres, GCS, MinIO, BigTable, or your own plugin) + │ + ▼ + Druid + UI API → Osprey UI +``` + +For the worker's rule-evaluation architecture (how a single event is scored against your rules), see the diagram at the top of [Writing Rules](../rules/). For the coordinator's internals (priority queues, bidirectional streaming vs. synchronous API), see the [Coordinator README](https://github.com/roostorg/osprey/tree/main/example_docker_compose/run_osprey_with_coordinator), which has a more detailed component diagram and a working example. + +## Getting data in + +There are three ways to get an event into Osprey: + +### 1. Kafka (the default) + +By default, the worker consumes from the Kafka topic `osprey.actions_input` (`OSPREY_KAFKA_INPUT_STREAM_TOPIC` in `docker-compose.yaml`; see `KafkaInputStream` in `osprey_worker/src/osprey/worker/sinks/sink/input_stream.py`). Each message is a JSON blob shaped like: + +```json +{ + "send_time": "", + "data": { + "action_id": 123, + "action_name": "user_login", + "data": { "...": "your event's actual fields" } + } +} +``` + +`action_name` is what your rules match against; the inner `data` object is whatever fields your event has, and those become the values your UDFs and feature extractors read. + +### 2. Google Cloud PubSub + +An alternate input source, selected via `InputStreamSource.PUBSUB` (`osprey_worker/src/osprey/worker/sinks/input_stream_chooser.py`). Configured with `PUBSUB_OSPREY_PROJECT_ID` and `PUBSUB_OSPREY_RULES_SINK_SUBSCRIPTION`. + +### 3. The Coordinator's synchronous gRPC API + +If you run Osprey with the optional Rust coordinator (`osprey_coordinator/`), external services can submit a single action directly and get an immediate response, without going through Kafka at all, which is useful when a caller needs a synchronous verdict rather than firing into a queue. This is the Sync Action API on port `19951` (bidirectional streaming for workers is a separate port, `19950`). See the [Coordinator README](https://github.com/roostorg/osprey/tree/main/example_docker_compose/run_osprey_with_coordinator) for a working `grpcurl` example and full setup. + +### Running against your own platform, without forking + +If your events don't come from Kafka/PubSub/the coordinator, or don't arrive as that JSON envelope, you don't need to fork Osprey. Two plugin hooks handle exactly this: + +- **`register_action_proto_deserializer`**: convert your own protobuf `Action` message into the JSON dict shape the engine expects. +- **`register_input_stream`**: swap in an entirely custom input source (a different queue system, a database poller, whatever you have). + +Both are `pluggy` hooks, registered the same way as UDFs and output sinks. See [Integrations & Plugins](integrations.md) for the plugin mechanics and `example_plugins/src/register_plugins.py` for a working example. + +## Getting data out + +Once a rule evaluates, the result flows out through two, largely independent mechanisms: + +### Output sinks + +An `ExecutionResult` is handed to every registered `BaseOutputSink` (`register_output_sinks`). Stdlib ships: +- `StdoutOutputSink`: prints to stdout (the default if nothing else is configured; useful for local dev) +- `KafkaOutputSink`: writes to `osprey.execution_results` by default (`OSPREY_KAFKA_OUTPUT_TOPIC`) +- `StoredExecutionResultOutputSink`: persists via whichever `ExecutionResultStore` is configured (see below) + +You can register your own via the same hook; it's the extension point for pushing results into a review queue, a webhook, or any other external system. + +### Execution result storage + +Separately, `ExecutionResultStore` backends persist full execution results for later querying. The active backend is chosen via `OSPREY_EXECUTION_RESULT_STORAGE_BACKEND` (`osprey_worker/src/osprey/worker/_stdlibplugin/execution_result_store_chooser.py`): + +| Value | Backend | +| --- | --- | +| `bigtable` | Google Cloud BigTable | +| `gcs` | Google Cloud Storage | +| `minio` | MinIO (S3-compatible; `OSPREY_MINIO_ENDPOINT`, `OSPREY_MINIO_ACCESS_KEY`, `OSPREY_MINIO_SECRET_KEY`, `OSPREY_MINIO_EXECUTION_RESULTS_BUCKET`) | +| `postgres` | Postgres | +| `plugin` | Your own `register_execution_result_store` implementation | +| `none` (default) | No persistence | + +### How results become queryable in the UI + +Druid consumes the worker's Kafka output and powers the UI API's real-time querying; this is what backs the [Investigate](../user/investigate/) query interface and the [Rules](../user/manage.md#rules-registry)/[Features Registries](../user/manage.md#features-registry). If you're running without Kafka output enabled, Druid has nothing to index and the query UI will be empty even though rules are still evaluating correctly. diff --git a/docs/integrations.md b/docs/integration/integrations.md similarity index 68% rename from docs/integrations.md rename to docs/integration/integrations.md index 25f85e4d..4d906558 100644 --- a/docs/integrations.md +++ b/docs/integration/integrations.md @@ -1,8 +1,8 @@ # Integrations & Plugins -Osprey is designed to be extended without modifying the core codebase; you can wire up your own logic such as detection functions, output destinations, entity state storage, and ML models through plugin packages that Osprey discovers at startup. +Osprey is designed to be extended without modifying the core codebase; you can wire up your own logic such as detection functions, output destinations, entity state storage, and ML models through plugin packages that Osprey discovers at startup. A plugin package implements any subset of the hooks in the [Available hooks](../development/local.md#available-hooks) table; this page walks through the integration points adopters ask about most. -See the [`example_plugins/` directory](https://github.com/roostorg/osprey/tree/main/example_plugins) for reference. +See the [`example_plugins/` directory](https://github.com/roostorg/osprey/tree/main/example_plugins) for a working reference package. ## How plugins are loaded @@ -25,7 +25,7 @@ Each entry point resolves to a module that contains hook functions decorated wit ## Writing UDFs -A user-defined function (UDF) is a Python class that can be called from your rules. UDFs encapsulate reusable detection logic such as text matching, DNS lookups, hash comparisons, or ML inference and make it available under a named function in the rules language. +A user-defined function (UDF) is a Python class that can be called from your rules. UDFs encapsulate reusable detection logic—text matching, DNS lookups, hash comparisons, ML inference—and expose it as a named function in the rules language. See [Writing Rules § User Defined Functions](../rules/#user-defined-functions-udfs) for the language-level view. ### Anatomy of a UDF @@ -89,19 +89,25 @@ def register_udfs(): return [TextContains, BanUser] ``` -## Built-in UDFs: hash lookups +Give each UDF a category from `UdfCategories` (`osprey_worker/src/osprey/engine/stdlib/udfs/categories.py`, e.g. `STRING`, `HASH`, `ENTITY`, `HTTP`) so it's grouped sensibly in the [UDF Registry](../user/manage.md#udf-registry). -Osprey's standard library includes hash UDFs (`HashMd5`, `HashSha1`, `HashSha256`, `HashSha512`) under the `HASH` category. They take a string `input` and return the hex digest. Use them in rules to compare hashed values against known-bad hash sets without storing raw data: +## Hash-based lookups + +Osprey's standard library includes the `Hash*` UDF family (`HashMd5`, `HashSha1`, `HashSha256`, `HashSha512`, under the `HASH` category and available without registration), which takes a string `input` and returns the hex digest. Compose these with SML's `in` operator or `HasLabel` to check values against known-bad sets without storing raw data: ```python -HashSha256(input=Username) == "e3b0c44298fc1c149afbf4c8996fb924..." +# Check a hashed value against a small inline set +IsKnownBadHash = HashSha256(input=SomeValue) in ['abc123...', 'def456...'] + +# Or check membership via a label that was set by some other process +IsKnownBadActor = HasLabel(entity=SomeEntity, label='KnownBad') ``` -These are available without registration. +Inline sets suit small, stable lists. There's no bulk-import or lookup-table primitive, so for a large external list (millions of hashes, updated frequently), write a custom UDF (see above) that queries your own store. ## Configuring input sinks -An input sink is where events _enter_ Osprey. Osprey ships with built-in sources (Kafka, Google Pub/Sub, the Osprey Coordinator, and a synthetic generator for local testing) selected via the `InputStreamSource` config value. If none of those fit your platform, you can register a custom input stream as a plugin. +An input sink is where events _enter_ Osprey. Osprey ships with built-in sources (Kafka, Google Pub/Sub, the Osprey Coordinator, and a synthetic generator for local testing) selected via the `InputStreamSource` config value. If none of those fit your platform, you can register a custom input stream as a plugin. For the conceptual picture, see [Data Flow § Getting data in](data-flow.md#getting-data-in). ### Built-in sources @@ -115,7 +121,7 @@ Source | Config `SYNTHETIC` |   | Generates random fake events; useful for local dev without any upstream system `PLUGIN` |   | Delegates to your registered `register_input_stream` hook -Set `InputStreamSource.KAFKA` (or whichever fits your existing infrastructure) if you already have events flowing through Kafka or Pub/Sub. Otherwise, implement a custom input stream and set `InputStreamSource.PLUGIN` in your config. +Set `InputStreamSource.KAFKA` if you already have events flowing through Kafka, or `InputStreamSource.PUBSUB` for Google Pub/Sub (see the table above). Otherwise, implement a custom input stream and set `InputStreamSource.PLUGIN` in your config. If your events arrive as protobuf rather than JSON, there's also a `register_action_proto_deserializer` hook for supplying your own deserializer. ### Writing a custom input stream @@ -158,11 +164,11 @@ def register_input_stream(config): ## Configuring output sinks -An output sink receives every `ExecutionResult` after rule evaluation and decides what to do with it, i.e. log it, forward it to a queue, call a webhook, or write to a database. +An output sink receives every `ExecutionResult` after rule evaluation and decides what to do with it, e.g. log it, forward it to a queue, call a webhook, or write to a database. For the conceptual picture, see [Data Flow § Getting data out](data-flow.md#getting-data-out); if what you want is to persist results in a backend other than the built-in BigTable/GCS/MinIO/Postgres options, the `register_execution_result_store` hook covers that instead. ### Sync output sink -Subclass `BaseOutputSink` and implement three methods; for example: +Subclass `BaseOutputSink` and implement its methods; for example: ```python from osprey.worker.sinks.sink.output_sink import BaseOutputSink @@ -175,7 +181,7 @@ class MyOutputSink(BaseOutputSink): return True def push(self, result: ExecutionResult) -> None: - # Do something with the result — send to a queue, call an API, etc. + # Do something with the result—send to a queue, call an API, etc. pass def stop(self) -> None: @@ -226,9 +232,9 @@ def register_async_output_sinks(config): return [ExampleAsyncOutputSink()] ``` -## Connecting to a review tool via a labels service +## Labels service -Osprey tracks state across events through entity labels: arbitrary tags attached to users, accounts, or other entities. Labels are read during rule evaluation and written by rules with label effects. To persist labels across process restarts (and share them between workers), you provide a `LabelsServiceBase` implementation backed by your own storage. +Osprey tracks state across events through entity labels: arbitrary tags attached to users, accounts, or other entities (e.g., "this user has 3 prior violations"). Labels are read during rule evaluation and written by rules with label effects. To persist labels across process restarts (and share them between workers), you provide a `LabelsServiceBase` implementation backed by your own storage via the `register_labels_service_or_provider` hook. The example implementation in [`example_plugins/src/services/labels_service.py`](https://github.com/roostorg/osprey/blob/main/example_plugins/src/services/labels_service.py) uses PostgreSQL, e.g.: @@ -237,7 +243,7 @@ from osprey.worker.lib.storage.labels import LabelsServiceBase class PostgresLabelsService(LabelsServiceBase): def initialize(self) -> None: - # Called once at startup — open connections here + # Called once at startup—open connections here ... def read_labels(self, entity) -> EntityLabels: @@ -259,11 +265,17 @@ def register_labels_service_or_provider(config): return PostgresLabelsService() ``` -A labels service backed by your existing datastore lets Osprey decisions feed directly into your review tool: label an entity "flagged", and your review queue queries for that label. +## Connecting to a review tool + +Osprey doesn't currently support direct integration with a review tool; however, these extension points can help you integrate: + +- `register_output_sinks`—push execution results into a review queue as they're produced. +- `register_label_output_sink`—a sink specifically for label mutations, replacing the default `LabelOutputSink`. +- A labels service backed by your existing datastore (previous section)—label an entity "flagged" from a rule, and your review queue queries your own store for that label. ## Plugging in your own ML model -ML models integrate as UDFs. Wrap your model's `predict` call in `execute`. Since a UDF's `__init__` receives `validation_context` and `arguments` from the framework, override it to accept and forward both, then do your model loading after the `super().__init__()` call; for example: +ML models can be integrated as UDFs. For an in-process model, wrap your model's `predict` call in `execute`. Since a UDF's `__init__` receives `validation_context` and `arguments` from the framework, override it to accept and forward both, then do your model loading after the `super().__init__()` call; for example: ```python class Arguments(ArgumentsBase): @@ -284,7 +296,13 @@ The returned score is then available in rules, e.g.: MySpamClassifier(text=MessageContent) > 0.85 ``` -Osprey constructs one UDF instance per call site when the rules are compiled, not per event, so the model isn't reloaded for every action processed. Keep in mind this means per _call site_, not per _class_: if you call the same UDF from multiple rules, each call site gets its own instance, and each one loads its own copy of the model. **For a large model, prefer calling the UDF from a single rule (or share the loaded weights via a module-level cache) rather than invoking it from many places.** +Osprey constructs one UDF instance per call site when the rules are compiled, not per event, so the model isn't reloaded for every event processed. That's one instance per _call site_, not per _class_: if you call the same UDF from multiple rules, each call site gets its own instance, and each one loads its own copy of the model. **For a large model, prefer calling the UDF from a single rule (or share the loaded weights via a module-level cache) rather than invoking it from many places.** + +For a model served remotely, the same pattern applies with `execute()` calling out over HTTP, gRPC, or your model server's SDK (you bring the client code). Because remote model calls are often slow or costly, gate them so they only run when relevant, using [Writing Rules' `Require(..., require_if=...)` pattern](../rules/#workflow-structure-and-file-placement): + +```python +Require(rule='ai_services/my_ai_service.sml', require_if=ActionName == 'register') +``` ## Packaging your plugin @@ -309,4 +327,4 @@ register_plugins = "register_plugins" Install it into the same environment as Osprey and it will be discovered automatically on the next startup. -See also: [Writing Rules](rules.md) +See also: [Writing Rules](../rules/) diff --git a/docs/research-personas.md b/docs/research-personas.md index 4509130c..41fa2d90 100644 --- a/docs/research-personas.md +++ b/docs/research-personas.md @@ -1,10 +1,10 @@ -# User Research Overview +# User Research & Personas -Osprey is designed for users who need to investigate and take automated action on events in real-time. By interviewing prospective Osprey users, ROOST has drafted the following user personas for developers to better understand who might be interested in using this tool. We hope this research helps the open source community prioritize features to build moving forward! +Osprey is designed for users who need to investigate and take automated action on events in real time. By interviewing prospective Osprey users, ROOST has drafted the following user personas for developers to better understand who might be interested in using this tool. We hope this research helps the open source community prioritize features to build moving forward. ## T&S Engineers -Most engineers who are interested in potentially using Osprey are like Alice – they do not currently have a viable rules engine due to cost, lack of flexibility, or limited available options. However, they need a tool for real-time event processing to handle adverse events at scale. They also want to optimize how Osprey works with the rest of their tech stack. +Most engineers who are interested in potentially using Osprey are like Alice—they do not currently have a viable rules engine due to cost, lack of flexibility, or limited available options. However, they need a tool for real-time event processing to handle adverse events at scale. They also want to optimize how Osprey works with the rest of their tech stack. ![user_persona_engineer](images/user_persona_engineer.png) @@ -12,7 +12,7 @@ Most engineers who are interested in potentially using Osprey are like Alice – - When I manage rules, I want a flexible and intuitive developer experience (including intuitive logic and easy to learn programming syntax) so that I can make changes quickly and without being reliant on third party tools -- When I assess new tools, I clear info on ease of deployment and integration with common tools (e.g., Splunk), so that I know if the tool will work within our stack +- When I assess new tools, I want clear info on ease of deployment and integration with common tools (e.g., Splunk), so that I know if the tool will work within our stack - When I deploy changes, I want to ensure they will be as effective as possible, to avoid future duplication of work diff --git a/docs/rules.md b/docs/rules.md deleted file mode 100644 index 53e1494d..00000000 --- a/docs/rules.md +++ /dev/null @@ -1,493 +0,0 @@ -# Osprey Rules - -![images/rules_architecture.png](images/rules_architecture.png) - -## Creating Rules - -Osprey rules are written in SML (“Some Madeup Language”) which is a subset of Python with additional restrictions to simplify rule writing. You may write rules that are specific to -single event types on a network, or ones that are applied to multiple event types. - -By themselves, rules only create variables; without a corresponding `WhenRules()` function call, the rule will have no effects outside of evaluation and query functionality. - -Rules currently support the following concepts through the `Rule(...)` function of the same name: - -- **Name** - - `Rule_Name = Rule(...)` - - The name of the rule also functions as a conventional "RuleId" and the name of the bool that can be used to query individual rule hits in the Osprey Query UI. As a result, changing the name of a rule after activation may affect historical query results in the UI if not logged externally. - -- **Logic** - - `when_all=[]` - - The actual logic that will be used to evaluate Osprey rules is all encompassed as single comma-delimited list of signals within the `when_all` parameter of the `Rule(...)` function and supports the use of Labels, Plugins, UDFs and other values to help enrich heuristics. - - At present, when evaluating UDFs or abstracted variables, any `NULL` evaluations in the series will cause the entire rule function to evaluate as `NULL`, which may be undesirable. - -- **Description** - - `description=f''` - - There is an additional string description field that is able to be emitted alongside the rule itself to external systems such as logging and ticketing systems to help enrich work-streams that may benefit from plain-language context on what the rule criteria is and what the rule may intend to do. - - It may be helpful to include dynamic variables as well to help enrich operational workflows that may need to identify specific values related to the trigger criteria. - - -Here's an example of a simple rule using various signal evaluations and out-of-the-box UDFs: - -```python -My_Rule_Name_v2 = Rule( - when_all=[ - # Primary Signal - MyFirstValue == True, - HasLabel(entity=MyEntityName, label='MyLabel'), - ListLength(list=UsersValues) == 5, - # Secondary Signal - RegexMatch(target=MyStringValue, pattern='(hello|world)'), - MySecondValue >= 3, - MyThirdValue != Null, - # Guardrail Signal - (_LocalValue in [1, 2, 3, 5]) or (GlobalValue in ['hello', 'howdy']), - not HasLabel(entity=MySecondEntityName, label='MySecondLabel'), - ], - description=f"{UserA} performed {ActionB} in this way. Emit warning", -) -``` - -## Rule Structuring - -You will likely find it useful to maintain two subdirectories inside of your main rules directory - a `rules` directory where actual logic will be added and a `models` directory for defining the various features that occur in any or specific event types. For example, your structure may look something like this: - -```bash -example-rules/ -| rules/ -| | record/ -| | | post/ -| | | | first_post_link.sml -| | | | index.sml -| | | like/ -| | | | like_own_post.sml -| | | | index.sml -| | account/ -| | | signup/ -| | | | high_risk_signup.sml -| | | | index.sml -| | index.sml -| models/ -| | record/ -| | | post.sml -| | | like.sml -| | account/ -| | | signup.sml -| main.sml -``` - -The `main.sml` file at the root of your rules directory serves as the entrypoint. It uses `Import` and `Require` statements to control which other files are loaded and when, allowing you to compose together logic across the project. This sort of structure lets you define rules and models that are specific to certain event types so that only the necessary rules are run for various event types. For example, you likely have some rules that should only be run on a `post` event, since only a `post` will have features like `text` or `mention_count`. - -Inside of each directory, you may maintain an `index.sml` file that will define the conditional logic in which the rules inside that directory are actually included for execution. Although you could handle all of this conditional logic inside of a single file, maintaining separate `index.sml`s per directory greatly helps with neat organization. See [Workflow Structure and File Placement](#workflow-structure-and-file-placement) for more on `Import` and `Require`. - -## Models - -Before you actually write a rule, you'll need to define a "model" for an event type. For this example, we will assume that you run a social media website that lets users create posts, either at the "top level" or as a reply to another top level post. Each post may include text, mentions of other users on your network, and an optional link embed in the post. Let's say that the event's JSON structure looks like this: - -```json -{ - "eventType": "userPost", - "user": { - "userId": "user_id_789", - "handle": "carol", - "postCount": 3, - "accountAgeSeconds": 9002 - }, - "postId": "abc123xyz", - "replyId": null, - "text": "Is anyone online right now? @alice or @bob, you there? If so check this video out", - "mentionIds": ["user_id_123", "user_id_456"], - "embedLink": "https://youtube.com/watch?id=1" -} -``` - -Inside of our `models/record` directory, we should now create a `post.sml` file where we will define the features for a post. - -```python -PostId: Entity[str] = EntityJson( - type='PostId', - path='$.postId', -) - -PostText: str = JsonData( - path='$.text', -) - -MentionIds: List[str] = JsonData( - path='$.mentionIds', -) - -EmbedLink: Optional[str] = JsonData( - path='$.embedLink', - required=False, -) - -ReplyId: Entity[str] = JsonData( - path='$.replyId', - required=False, -) -``` - -The [`JsonData` UDF](#user-defined-functions-udfs) lets us take the event's JSON and define features based on the contents of that JSON. These features can then be referenced in other rules that we import the `models/record/post.sml` model into. If you have any values inside your JSON object that may not always be present, you can set `required` to `False`, and these features will be `None` whenever the feature is not present. - -Note that we did not actually create any features for things like `userId` or `handle`. That is because these values will be present in *any* event. It wouldn't be very nice to have to copy these features into each event type's model. Therefore, we will actually create a `base.sml` model that defines these features which are always present. Inside of `models/base.sml`, let's define these. - -```python -EventType = JsonData( - path='$.eventType', -) - -UserId: Entity[str] = EntityJson( - type='UserId', - path='$.user.userId', -) - -Handle: Entity[str] = EntityJson( - type='Handle', - path='$.user.handle', -) - -PostCount: int = JsonData( - path='$.user.postCount', -) - -AccountAgeSeconds: int = JsonData( - path='$.user.accountAgeSeconds', -) -``` - -Here, instead of simply using `JsonData`, we instead use the `EntityJson` UDF for the `UserID`. This is covered in the [UDFs section](#user-defined-functions-udfs), but as a rule of thumb, you likely will want to have values for things like a user's ID set to be entities. This will help more later, such as when doing data explorations within the Osprey UI. - -### Model Hierarchy - -In practice, you may find it useful to create a hierarchy of base models: - -- `base.sml` for features present in every event (user IDs, handles, account stats, etc.) -- `account_base.sml` for features that appear only in account related events, but always appear in each account related event. Similarly, you may add one like `record_base.sml` for those features which appear in all record events. - -This type of hierarchy prevents duplication (which Osprey does not allow) and ensures features are defined at the appropriate level of abstraction. - -## Effects with WhenRules - -The `WhenRules()` function allows for creating effects that trigger external services, create declarations, or modify internal labels by listing `Rule` objects in sequence within the -`rules_any` parameter of `WhenRules()`. By default, operators and designers may utilize UDFs with predefined effects such as `DeclareVerdict()`, `LabelAdd()`, or `LabelRemove()` upon -positive rule evaluation. - -Below is an example of the use of a WhenRules() block to verify and email and reject a request. - -```python -WhenRules( - rules_any=[ - Enabled_Rule_1, - Enabled_Rule_2, - # Disabled_Rule_1, - ], - then=[ - # Verdicts - DeclareVerdict(verdict='reject'), - # Labels - LabelAdd(entity=UserId, label='recently_challenged', expires_after=TimeDelta(days=7)), - LabelAdd(entity=UserId, label='verify', apply_if=NotVerified), - LabelAdd(entity=Email, label='pending_verify'), - LabelAdd(entity=Domain, label='recently_seen', expires_after=TimeDelta(days=7)), - ], -) -``` - -`WhenRules()` must be placed after rule declaration within a file, and it may become difficult to interpret outcomes of rules that are too distributed. Therefore, it may be beneficial -to place any effects toward the bottom of workflows. - -## Output Sinks - -After all rules are evaluated for an input event, a set of output sinks takes the resulting `ExecutionResult` and performs additional work based on that data. These may be defined -as part of a plugin for performing domain specific work. - -Some default use cases include a `StdoutOutputSink` which simply outputs the result to the log, a `KafkaOutputSink` which pipes data to Kafka (used for Osprey UI), or the -`LabelOutputSink` which can add some stateful data to be used in future rules executions. - -```python -class StdoutOutputSink(BaseOutputSink): - """An output sink that prints to standard out!""" - - def __init__(self, log_sampler: Optional[DynamicLogSampler] = None): - pass - - def will_do_work(self, result: ExecutionResult) -> bool: - return True - - def push(self, result: ExecutionResult) -> None: - print(f'result: {result.extracted_features_json} {result.verdicts}') - - def stop(self) -> None: - pass -``` - -Passing data to these output sinks is standardized through the use of `Effects`, which are outputs of some functions, usually UDFs. - -```python -def push(self, result: ExecutionResult) -> None: - users_to_ban = result.effects[BanUserEffect] - ban_users(users_to_ban) -``` - -## User Defined Functions (UDFs) - -User Defined Functions (UDFs) are plugins written in Python that enable users of Osprey to extend and customize their use of the Osprey SML. UDFs are implemented as Python functions and are registered -as a plugin. They extend the `UDFBase` abstract base class with a set of arguments and an output. These will be executed whenever called in SML. - -```python -# example_plugins/text_contains.py -class TextContainsArguments(ArgumentsBase): - text: str - phrase: str - case_sensitive = False - -class TextContains(UDFBase[TextContainsArguments, bool]): - def execute(self, execution_context: ExecutionContext, arguments: TextContainsArguments) -> bool: - escaped = re.escape(arguments.phrase) - pattern = rf'\b{escaped}\b' - flags = 0 if arguments.case_sensitive else re.IGNORECASE - regex = re.compile(pattern, flags) - return bool(regex.search(arguments.text)) - -# example_plugins/register_plugins.py -@hookimpl_osprey -def register_udfs(): - return [TextContains] -``` - -Usage in SML: - -```python -# example_rules/post_contains_hello.sml -ContainsHello = Rule( - when_all=[ - EventType == 'create_post', - TextContains(text=PostText, phrase='hello'), - ], - description='Post contains the word "hello"', -) -``` - -### Effect UDFs - -Plugins may also define external effects, which are useful for performing functionality in your primary service. Effects are simply passed to output sinks at the end of a rule run. -These UDFs have an output that extends `EffectBase`, and can be called as a result of a `WhenRules`. - -```python -# example_plugins/src/ban_user.py -class BanUser(UDFBase[BanUserArguments, BanUserEffect]): - category = UdfCategories.ENGINE - - def execute(self, execution_context: ExecutionContext, arguments: BanUserArguments) -> BanUserEffect: - return BanUserEffect( - entity=arguments.entity, - comment=arguments.comment, - ) - -# example_rules/post_contains_hello.sml -WhenRules( - rules_any=[ContainsHello], - then=[BanUser(entity=UserId, comment='User said "hello"')], -) -``` - -UDF outputs can also implement the `CustomExtractedFeature` interface - which get persisted in the outputs for the UI. `EffectToCustomExtractedFeatureBase` can also be used when effects need additional processing for use in the UI. - -## Labels - -Labels are a standard plugin that enable stateful rules, and touch many parts of Osprey. They are effectively tags on various entities, which may be arbitrarily defined. For more about labels, see [User Guide → Investigate → Labels](user/investigate/labels.md). - -### Creating Entities - -Labels are applied to Entities, which are dynamically interpreted from outputs of the UDF `EntityJson`, usually applied to pieces of data that are generally consistent across actions such as User ID or email. - -```python -# user.sml -UserId: Entity[str] = EntityJson( - type='User', - path='$.user_id' -) -``` - -It is possible to create new UDFs that also create entities by having the output of UDF set to `EntityT`. - -### Adding Labels - -Labels may be added in a `WhenRules()` clause. This will cause the labels output sink to tag the given entity with the given label at the end of the rules run. - -```python -WhenRules( - rules_any=[ - Sent_Too_Many_DMs, - ], - then=[ - LabelAdd(entity=UserId, label='likely_spammer') - ], -) -``` - -### Using Labels - -Since labels may be retrieved during a rule run, they can be effectively used as state for your rules. - -```python -Should_Warn_User_Of_Spammer = Rule( - when_all=[ - HasLabel(entity=UserId, label='likely_spammer'), - This_Is_A_New_DM, - ], -) -``` - -Labels will also be shown in the UI for entities, and can also be set manually. Note that since the UI only searches across actions, `HasLabel()` will not work in the Query UI. -Instead, you may use `DidAddLabel`, which will be true when the given action added a label to a specific entity. - -```python -# UI Query -DidAddLabel(entity_type="UserId", label_name="likely_spammer") -``` - -## Notable Gotchas - -### Nulls - -Nulls are the case where a rule or variable in SML does not exist. This can occur for many reasons - either a piece of data is missing or a rule didn't run. Unlike many programming languages, generally rules with null valued variables will not evaluate that rule (and thus, downstream rules will not evaluate either). The exception cases are when nulls are explicitly checked in a rule. For example: - -```python -Thing: int = JsonData(path='$.property_that_doesnt_exist') - -# Evaluates to False -MyFirstRule = Rule(when_all=[ - Thing != Null, -]) - -# Skips evaluation and sets to Null -MySecondRule = Rule(when_all=[ - Thing > 1, -]) - -# Skips evaluation and sets to Null -MyThirdRule = Rule(when_all=[ - MySecondRule, -]) -``` - -### Workflow Structure and File Placement - -SML files can be composed to make your rules easier to understand. The `Import` statement allows you to include rules and variables found in other files. - -```python -# models/action_name.sml -ActionName = "foo" - -# main.sml -Import( - rules=[ - 'models/action_name.sml', - 'models/http_request.sml', - ] -) - -MyRule = Rule(when_all=[ActionName == "foo"]) -``` - -`Require` allows you to selectively run other SML scripts. Requires supports templating and conditionals, allowing scripts to be filtered out if necessary. This is important in situations where some rules or UDFs are particularly expensive to run (such as making a call to an AI service, for example). - -```python -# main.sml -Require(rule=f'actions/{ActionName}.sml') # will execute 'actions/foo.sml' - -Require(rule='ai_services/my_ai_service.sml', require_if=ActionName == "register") -``` - -## Full Example - -The following is a complete walkthrough of writing a rule using the project structure described above. The goal is to flag accounts whose first post mentions at least one user and includes a link. - -### Writing the Rule - -We'll create `rules/record/post/first_post_link.sml` for the rule logic. This file defines both the conditions that cause the rule to evaluate to `True` and the actions to take when it does. - -```python -# First, import the models that you will need inside of this rule -Import( - rules=[ - 'models/base.sml', - 'models/record/post.sml', - ], -) - -# Next, define a variable that uses the `Rule` UDF -FirstPostLinkRule = Rule( - # Set the conditions in which this rule will be `True` - when_all=[ - PostCount == 1, # if this is the user's first post - EmbedLink != None, # if there is a link inside of the post - ListLength(list=MentionIds) >= 1, # if there is at least one mention in the post - ], - description='First post for user includes a link embed', -) - -# Finally, set which effect UDFs will be triggered -WhenRules( - rules_any=[FirstPostLinkRule], - then=[ - # This is a custom effect UDF that we have implemented - ReportRecord( - entity=PostId, - comment='This was the first post by a user and included a link', - severity=3, - ), - ], -) -``` - -### Wiring Up the Rule - -We want this rule to run *only* when the event is a post event. Using the project structure described above, this involves three files. - -First, `main.sml` at the project root includes a single `Require` statement pointing to the top-level rules index: - -```python -Require( - rule='rules/index.sml', -) -``` - -Next, `rules/index.sml` conditionally requires the post rules when the event type matches: - -```python -Import( - rules=[ - 'models/base.sml', - ], -) - -Require( - rule='rules/record/post/index.sml', - require_if=EventType == 'userPost', -) -``` - -Finally, `rules/record/post/index.sml` requires the new rule: - -```python -Import( - rules=[ - 'models/base.sml', - 'models/record/post.sml', - ], -) - -Require( - rule='rules/record/post/first_post_link.sml', -) -``` diff --git a/docs/rules/README.md b/docs/rules/README.md new file mode 100644 index 00000000..d33a036d --- /dev/null +++ b/docs/rules/README.md @@ -0,0 +1,327 @@ +# Writing Rules + +Rules are how you teach Osprey what to look for and what to do when it finds it. This page covers SML ("Some Madeup Language"), Osprey's rule language: defining features with models, writing rules over them, and wiring rules to effects. + +See also: + - [Examples](examples.md) to apply all the concepts to complete, runnable rulesets + - [Why a Bespoke Language?](why-sml.md) for more about SML and why it exists + +![A user's action flows from your production service into Osprey as an event, where rules call UDFs and produce execution output; output sinks return verdicts to your service—the user gets banned—and send features to the Osprey UI](../images/rules_architecture.png) + +## Creating Rules + +Osprey rules are written in SML, a subset of Python with additional restrictions to simplify rule writing. You may write rules that are specific to single event types on a network, or ones that are applied to multiple event types. + +By themselves, rules only create variables; without a corresponding `WhenRules()` function call, the rule will have no effects outside of evaluation and query functionality. + +Rules currently support the following concepts through the `Rule(...)` function of the same name: + +- **Name** + + `Rule_Name = Rule(...)` + + The rule's name doubles as its identifier and as the boolean feature you can query in the Osprey UI to see individual rule hits. Renaming a rule after it's active disconnects it from its historical query results, so name carefully. + +- **Logic** + + `when_all=[]` + + The rule's logic is the list of signals in the `when_all` parameter; a signal can be a comparison over features, a label check, a UDF call, or another rule. + + If any signal in the list evaluates to `None`, the whole rule evaluates to `None`; see [None values](#none-values) below. + +- **Description** + + `description=f''` + + A plain-language description of what the rule looks for, emitted alongside the rule to external systems like logging and ticketing. It's an f-string, so you can interpolate feature values to show responders exactly what triggered the rule. + + +Here's an example of a simple rule using various signal evaluations and out-of-the-box UDFs: + +```python +My_Rule_Name_v2 = Rule( + when_all=[ + # Primary Signal + MyFirstValue == True, + HasLabel(entity=MyEntityName, label='MyLabel'), + ListLength(list=UsersValues) == 5, + # Secondary Signal + RegexMatch(target=MyStringValue, pattern='(hello|world)'), + MySecondValue >= 3, + MyThirdValue != None, + # Guardrail Signal + (_LocalValue in [1, 2, 3, 5]) or (GlobalValue in ['hello', 'howdy']), + not HasLabel(entity=MySecondEntityName, label='MySecondLabel'), + ], + description=f"{UserA} performed {ActionB} in this way. Emit warning", +) +``` + +## Rule Structuring + +You will likely find it useful to maintain two subdirectories inside of your main rules directory: a `rules` directory where actual logic will be added, and a `models` directory for defining the various features that occur in any or specific event types. For example, your structure may look something like this: + +```bash +example-rules/ +| rules/ +| | record/ +| | | post/ +| | | | first_post_link.sml +| | | | index.sml +| | | like/ +| | | | like_own_post.sml +| | | | index.sml +| | account/ +| | | signup/ +| | | | high_risk_signup.sml +| | | | index.sml +| | index.sml +| models/ +| | record/ +| | | post.sml +| | | like.sml +| | account/ +| | | signup.sml +| main.sml +``` + +The `main.sml` file at the root of your rules directory serves as the entry point. It uses `Import` and `Require` statements to control which other files are loaded and when, allowing you to compose together logic across the project. This sort of structure lets you define rules and models that are specific to certain event types so that only the necessary rules are run for various event types. For example, you likely have some rules that should only be run on a `post` event, since only a `post` will have features like `text` or `mention_count`. + +Inside of each directory, you may maintain an `index.sml` file that will define the conditional logic in which the rules inside that directory are actually included for execution. Although you could handle all of this conditional logic inside of a single file, maintaining separate `index.sml`s per directory greatly helps with neat organization. + +See [Workflow Structure and File Placement](#workflow-structure-and-file-placement) for more on `Import` and `Require`. + +## Models + +Before you actually write a rule, you'll need to define a "model" for an event type. For this example, we will assume that you run a social media website that lets users create posts, either at the "top level" or as a reply to another top level post. Each post may include text, mentions of other users on your network, and an optional link embed in the post. Let's say that the event's JSON structure looks like this: + +```json +{ + "eventType": "userPost", + "user": { + "userId": "user_id_789", + "handle": "carol", + "postCount": 3, + "accountAgeSeconds": 9002 + }, + "postId": "abc123xyz", + "replyId": null, + "text": "Is anyone online right now? @alice or @bob, you there? If so check this video out", + "mentionIds": ["user_id_123", "user_id_456"], + "embedLink": "https://youtube.com/watch?id=1" +} +``` + +Inside of our `models/record` directory, we should now create a `post.sml` file where we will define the features for a post. + +```python +PostId: Entity[str] = EntityJson( + type='PostId', + path='$.postId', +) + +PostText: str = JsonData( + path='$.text', +) + +MentionIds: List[str] = JsonData( + path='$.mentionIds', +) + +EmbedLink: Optional[str] = JsonData( + path='$.embedLink', + required=False, +) + +ReplyId: Entity[str] = EntityJson( + type='PostId', + path='$.replyId', + required=False, +) +``` + +The [`JsonData` UDF](#user-defined-functions-udfs) lets us take the event's JSON and define features based on the contents of that JSON. These features can then be referenced in other rules that we import the `models/record/post.sml` model into. If you have any values inside your JSON object that may not always be present, you can set `required` to `False`, and these features will be `None` whenever the feature is not present. + +We did not create features for values like `userId` or `handle` because they're present in *any* event; copying them into every event type's model would get tedious. Instead, let's define them once in a `base.sml` model. Inside of `models/base.sml`: + +```python +EventType = JsonData( + path='$.eventType', +) + +UserId: Entity[str] = EntityJson( + type='UserId', + path='$.user.userId', +) + +Handle: Entity[str] = EntityJson( + type='Handle', + path='$.user.handle', +) + +PostCount: int = JsonData( + path='$.user.postCount', +) + +AccountAgeSeconds: int = JsonData( + path='$.user.accountAgeSeconds', +) +``` + +Here we use the `EntityJson` UDF for `UserId` rather than plain `JsonData`. UDFs are covered [below](#user-defined-functions-udfs), but as a rule of thumb, identifiers like user IDs should be entities; it pays off later when exploring data in the Osprey UI. + +### Model Hierarchy + +In practice, you may find it useful to create a hierarchy of base models: + +- `base.sml` for features present in every event (user IDs, handles, account stats, etc.) +- `account_base.sml` for features that appear only in account related events, but always appear in each account related event. Similarly, you may add one like `record_base.sml` for those features which appear in all record events. + +This type of hierarchy prevents duplication (which Osprey does not allow) and ensures features are defined at the appropriate level of abstraction. + +## Effects with WhenRules + +`WhenRules()` wires rules to effects: list `Rule` objects in its `rules_any` parameter, and when any of them evaluates true, the effects in `then=` fire. Osprey ships effect UDFs like `DeclareVerdict()`, `LabelAdd()`, and `LabelRemove()`; effects can also trigger external services through your output sinks. + +Below is an example of a `WhenRules()` block that rejects the request and labels the user, email, and domain for verification follow-up: + +```python +WhenRules( + rules_any=[ + Enabled_Rule_1, + Enabled_Rule_2, + # Disabled_Rule_1, + ], + then=[ + # Verdicts + DeclareVerdict(verdict='reject'), + # Labels + LabelAdd(entity=UserId, label='recently_challenged', expires_after=TimeDelta(days=7)), + LabelAdd(entity=UserId, label='verify', apply_if=NotVerified), + LabelAdd(entity=Email, label='pending_verify'), + LabelAdd(entity=Domain, label='recently_seen', expires_after=TimeDelta(days=7)), + ], +) +``` + +`WhenRules()` must appear after the rules it references. Effects scattered throughout a file are hard to trace, so keep them together toward the bottom. + +After evaluation, effects and the rest of the execution result are handed to your deployment's output sinks. [Data Flow](../integration/data-flow.md) covers where results go from there, and [Integrations & Plugins](../integration/integrations.md#configuring-output-sinks) covers adding output sinks of your own. + +## User Defined Functions (UDFs) + +Nearly every function on this page—`Rule`, `JsonData`, `EntityJson`—is a UDF: a function implemented in Python and made available to SML. Osprey ships a standard library of them, and your deployment's developers can register custom ones through plugins; writing a UDF in Python is covered in [Integrations & Plugins § Writing UDFs](../integration/integrations.md#writing-udfs). + +The authoritative list of what's callable in _your_ deployment—with signatures, descriptions, and categories—is the in-app [UDF Registry](../user/manage.md#udf-registry). The standard library includes general-purpose helpers like `RegexMatch`, `ListLength`, `ParseInt` (numeric string to integer), `StringSlice` (substring by index range), and the `Hash*` family, plus engine functions like `GetActionName()` and `GetActionId()` for the name and ID of the event being processed, and `Experiment`/`ExperimentWhen` for bucketing entities so a rule can roll out to a slice of traffic. + +Custom UDFs are called from SML like any other function. The demo ruleset's one rule is built on a custom `TextContains` UDF: + +```python +# example_rules/rules/post_contains_hello.sml +ContainsHello = Rule( + when_all=[ + EventType == 'create_post', + TextContains(text=PostText, phrase='hello'), + ], + description='Post contains the word "hello"', +) +``` + +### Effect UDFs + +Some UDFs don't return a value to compare against; they produce an **effect**, a structured output that Osprey's output sinks act on after evaluation—banning a user, reporting a post. Effect UDFs are called in the `then=` list of a `WhenRules()` block: + +```python +# example_rules/rules/post_contains_hello.sml +WhenRules( + rules_any=[ContainsHello], + then=[BanUser(entity=UserId, comment='User said "hello"')], +) +``` + +Implementing an effect UDF (and the output sink that consumes it) is plugin work; see [Integrations & Plugins](../integration/integrations.md#udfs-with-side-effects). + +## Labels + +Labels are a standard plugin that enable stateful rules, and touch many parts of Osprey. They are effectively tags on various entities, which may be arbitrarily defined. Rules can add and remove labels as effects and check them as conditions, so past decisions inform future ones; [Examples § Labels as state](examples.md#labels-as-state) walks through that pattern end to end. For how labels behave in the UI, see [User Guide → Investigate → Labels](../user/investigate/labels.md). + +### Creating Entities + +Labels are applied to entities: features created with the `EntityJson` UDF, usually for values that stay consistent across events, like a user ID or email address. + +```python +# user.sml +UserId: Entity[str] = EntityJson( + type='User', + path='$.user_id' +) +``` + +A custom UDF can also create entities by declaring `EntityT` as its output type. + +## Notable Gotchas + +### None values + +A rule or variable in SML is `None` when it doesn't exist; a piece of data was missing, or a rule didn't run. Unlike in many programming languages, a rule with a `None` signal is skipped entirely (and so are rules downstream of it) unless the rule checks for `None` explicitly. For example: + +```python +Thing: int = JsonData(path='$.property_that_doesnt_exist') + +# Evaluates to False +MyFirstRule = Rule( + when_all=[ + Thing != None, + ], + description=f'Thing is present', +) + +# Skips evaluation and sets to None +MySecondRule = Rule( + when_all=[ + Thing > 1, + ], + description=f'Thing is greater than 1', +) + +# Skips evaluation and sets to None +MyThirdRule = Rule( + when_all=[ + MySecondRule, + ], + description=f'MySecondRule matched', +) +``` + +### Workflow Structure and File Placement + +SML files can be composed to make your rules easier to understand. The `Import` statement includes rules and variables from other files. + +```python +# models/action_name.sml +ActionName = "foo" + +# main.sml +Import( + rules=[ + 'models/action_name.sml', + 'models/http_request.sml', + ] +) + +MyRule = Rule(when_all=[ActionName == "foo"], description=f'Action is foo') +``` + +`Require` selectively runs other SML scripts. It supports templating and conditionals, so scripts can be skipped entirely; that matters when a rule or UDF is expensive to run, like a call to an AI service. + +```python +# main.sml +Require(rule=f'actions/{ActionName}.sml') # will execute 'actions/foo.sml' + +Require(rule='ai_services/my_ai_service.sml', require_if=ActionName == "register") +``` + +--- + +See [Examples](examples.md) to put these concepts together into complete rulesets you can run and adapt. diff --git a/docs/rules/examples.md b/docs/rules/examples.md new file mode 100644 index 00000000..58dc48af --- /dev/null +++ b/docs/rules/examples.md @@ -0,0 +1,219 @@ +# Examples + +Everything [Writing Rules](./) covers, applied to complete rulesets: first the demo's real, runnable ruleset file by file, then two patterns you'll reach for as your own ruleset grows—labels as state, and a multi-signal rule organized across files. The examples use the same small social network as the rest of these docs. + +## The demo ruleset, file by file + +The [demo](../development/) runs the actual ruleset in [`example_rules/`](https://github.com/roostorg/osprey/tree/main/example_rules), which is small enough to read in one sitting: two models, one rule, and one label. The platform being moderated has a single (admittedly strict) policy: users may never say "hello." + +Events arrive as JSON like this—an event name (`action_name`), an ID, and whatever the platform sent: + +```json +{ + "action_id": 1, + "action_name": "create_post", + "data": { + "user_id": "user_1923", + "event_type": "create_post", + "post": { "text": "hello world" } + } +} +``` + +**`main.sml`** is the entrypoint. This ruleset is small enough to skip conditional `index.sml` files entirely: import the base model, require the one rule file. + +```python +Import(rules=['models/base.sml']) + +Require(rule='rules/post_contains_hello.sml') +``` + +**`models/base.sml`** defines the features present on every event. `UserId` and `EventType` are declared as entities, so Osprey tracks them as _things_ that persist across events—that's what lets a label stick to a user later. `coerce_type=True` (the default) converts a mismatched value, like a numeric user ID, to the declared type instead of erroring. + +```python +UserId: Entity[str] = EntityJson( + type='User', + path='$.user_id', + coerce_type=True +) + +EventType: Entity[str] = EntityJson( + type='EventType', + path='$.event_type', + coerce_type=True +) + +ActionName=GetActionName() + +ActionId=GetActionId() +``` + +The last two lines pull the event's name and ID from Osprey itself rather than the JSON payload: `GetActionName()` and `GetActionId()` are stdlib UDFs, and exposing them as features makes them queryable in the UI. + +**`models/post.sml`** adds the one feature specific to posts: + +```python +PostText: Entity[str] = EntityJson( + type='PostText', + path='$.post.text', + coerce_type=True +) +``` + +**`rules/post_contains_hello.sml`** imports both models, defines the rule, and wires it to effects—the whole loop in one file: + +```python +Import( + rules=[ + 'models/base.sml', + 'models/post.sml', + ] +) + +ContainsHello = Rule( + when_all=[ + EventType == 'create_post', + TextContains(text=PostText, phrase='hello') + ], + description='Post contains the word "hello"', +) + +WhenRules( + rules_any=[ContainsHello], + then=[ + BanUser(entity=UserId, comment='User said "hello"'), + LabelAdd(entity=UserId, label='meow'), + ], +) +``` + +`TextContains` and `BanUser` aren't stdlib—they're custom UDFs from [`example_plugins/`](https://github.com/roostorg/osprey/tree/main/example_plugins), which is the reference for [writing your own](../integration/integrations.md#writing-udfs). `LabelAdd` is stdlib. + +**`config/labels.yaml`** declares the label the rule applies: which entity types it's valid for, and its connotation. + +```yaml +labels: + meow: + valid_for: [User] + connotation: positive + description: testing label +``` + +**What you'll see in the UI** once events flow: each processed post appears in the Event Stream with `UserId`, `EventType`, `PostText`, and `ContainsHello` among its extracted features; querying `ContainsHello == True` filters to the posts that matched; and clicking through to a matched post's author shows the `meow` label on their `User` entity, with the `BanUser` effect recorded on the event. [Getting Started](../development/) walks through this UI tour on live demo data. + +## Labels as state + +Rules can't see previous events directly, but labels persist on entities across events—so a label added by one rule becomes a condition another rule checks later. Say you've flagged users who send too many DMs: + +```python +WhenRules( + rules_any=[ + Sent_Too_Many_DMs, + ], + then=[ + LabelAdd(entity=UserId, label='likely_spammer') + ], +) +``` + +From then on, _every_ event by that user carries the state, and other rules can act on it—here, on a different event type entirely: + +```python +Should_Warn_User_Of_Spammer = Rule( + when_all=[ + HasLabel(entity=UserId, label='likely_spammer'), + This_Is_A_New_DM, + ], + description=f'Likely spammer {UserId} started a new DM', +) +``` + +Labels also show on the entity in the UI, where they can be added and removed by hand. One asymmetry to know about: `HasLabel()` works in rules but not in the query bar, because queries search events rather than current entity state. To find events where a label was applied, query `DidAddLabel(entity_type="User", label_name="likely_spammer")` instead; see [Query Syntax](../user/investigate/query-syntax.md) for details. + +## A multi-signal rule, organized across files + +The following is a complete walkthrough of writing a rule using the project structure described in [Writing Rules § Rule Structuring](./#rule-structuring). The goal is to flag accounts whose first post mentions at least one user and includes a link—three signals that are individually innocent but suspicious together. + +### Writing the rule + +We'll create `rules/record/post/first_post_link.sml` for the rule logic. This file defines both the conditions that cause the rule to evaluate to `True` and the actions to take when it does. + +```python +# First, import the models that you will need inside of this rule +Import( + rules=[ + 'models/base.sml', + 'models/record/post.sml', + ], +) + +# Next, define a variable that uses the `Rule` UDF +FirstPostLinkRule = Rule( + # Set the conditions in which this rule will be `True` + when_all=[ + PostCount == 1, # if this is the user's first post + EmbedLink != None, # if there is a link inside of the post + ListLength(list=MentionIds) >= 1, # if there is at least one mention in the post + ], + description='First post for user includes a link embed', +) + +# Finally, set which effect UDFs will be triggered +WhenRules( + rules_any=[FirstPostLinkRule], + then=[ + # This is a custom effect UDF that we have implemented + ReportRecord( + entity=PostId, + comment='This was the first post by a user and included a link', + severity=3, + ), + ], +) +``` + +### Wiring up the rule + +We want this rule to run _only_ when the event is a post event. Using the project structure described above, this involves three files. + +First, `main.sml` at the project root includes a single `Require` statement pointing to the top-level rules index: + +```python +Require( + rule='rules/index.sml', +) +``` + +Next, `rules/index.sml` conditionally requires the post rules when the event type matches: + +```python +Import( + rules=[ + 'models/base.sml', + ], +) + +Require( + rule='rules/record/post/index.sml', + require_if=EventType == 'userPost', +) +``` + +Finally, `rules/record/post/index.sml` requires the new rule: + +```python +Import( + rules=[ + 'models/base.sml', + 'models/record/post.sml', + ], +) + +Require( + rule='rules/record/post/first_post_link.sml', +) +``` + +--- + +From here, try modifying the demo ruleset while it's running—add a phrase to `TextContains`, or a second rule over `PostText`—or head to [Integrations & Plugins](../integration/integrations.md) when a pattern needs a UDF that doesn't exist yet. diff --git a/docs/rules/why-sml.md b/docs/rules/why-sml.md new file mode 100644 index 00000000..54339882 --- /dev/null +++ b/docs/rules/why-sml.md @@ -0,0 +1,13 @@ +# Why a Bespoke Language? + +Why does Osprey use the bespoke SML ("Some Madeup Language") instead of supporting plain Python? The short answer is that SML's restrictions are what make Osprey's rule engine safe to run untrusted, third-party-authored logic in production—and fast enough to do it in real time. + +- **The grammar is intentionally small.** The SML parser (`osprey_worker/src/osprey/engine/ast/py_ast.py`) only understands a restricted set of Python AST nodes: assignments, calls, comparisons, boolean/binary operators, literals, and f-strings. Anything outside that set raises a syntax error; i.e. there's no `for`/`while`, `def`/`class`, or arbitrary `import`. This helps rule out unbounded loops and recursion by construction, meaning a rule can't hang or DoS the worker. + +- **Every name is assign-once.** The `UniqueStoredNames` validator rejects re-declaring the same name anywhere in the ruleset, and `NoUnusedLocals` rejects declaring a name that's never read. Combined, every rule and feature compiles down to a single, static, acyclic dependency graph (`execution_graph.py`) rather than an imperative script with mutable state. + +- **The static graph makes the engine fast and introspectable.** Because nothing can have side effects or run in a loop, the executor can safely evaluate independent branches of the graph in parallel using gevent greenlets (`executor.py`); the UI can also render the exact dependency graph for any rule or feature, which powers the [Rules Visualizer](../user/manage.md#rules-visualizer) and the cross-referencing in the [Rules](../user/manage.md#rules-registry) and [Features Registry](../user/manage.md#features-registry) pages. + +- **Types are checked before rules run.** `validate_static_types.py` and `validate_call_rvalue.py` catch type errors and misused return values at validation/push time, not as a runtime `AttributeError` in production the first time a rare code path executes. + +Since SML is a legal subset of Python syntax, pointing your editor at Python-mode gets you reasonable syntax highlighting. See [IDE Setup](../development/ide.md) for current editor setup. diff --git a/docs/theme/css/content.css b/docs/theme/css/content.css new file mode 100644 index 00000000..ad0973e9 --- /dev/null +++ b/docs/theme/css/content.css @@ -0,0 +1,47 @@ +/* Content styling overrides */ + +.content img { + background-color: color-mix(in srgb, var(--bg) 80%, white); + border: 0.5em solid color-mix(in srgb, var(--bg) 80%, white); + box-shadow: 0 0.25em 0.5em 0.25em rgba(0 0 0 / 0.1 ); + box-sizing: border-box; + margin-block: 0.5em; +} + +.blockquote-tag { + background: var(--quote-bg); + border-radius: 0.25em; + padding: 1em; +} + +.menu-title { + font-weight: 600; +} + +table { + width: 100%; +} + +table td, +table thead th { + padding: 0.25em 0.5em; +} + +table td { + text-wrap: balance; +} + +dt { + margin-block: 1.5em 0; +} + +dd { + margin-inline-start: 1em; + margin-block: 0.25em; +} + +hr { + margin-block: 2em; + opacity: 0.25; +} + diff --git a/docs/theme/css/roost-hljs.css b/docs/theme/css/roost-hljs.css new file mode 100644 index 00000000..7fadf1fd --- /dev/null +++ b/docs/theme/css/roost-hljs.css @@ -0,0 +1,199 @@ +/*! + Theme: a11y-light + Author: @ericwbailey + Maintainer: @ericwbailey + + Based on the Tomorrow Night Eighties theme: https://github.com/isagalaev/highlight.js/blob/master/src/styles/tomorrow-night-eighties.css +*/ + +.roost .hljs { + background: #fefefe; + color: #545454; +} + +/* Comment */ +.roost .hljs-comment, +.roost .hljs-quote { + color: #696969; +} + +/* Red */ +.roost .hljs-variable, +.roost .hljs-template-variable, +.roost .hljs-tag, +.roost .hljs-name, +.roost .hljs-selector-id, +.roost .hljs-selector-class, +.roost .hljs-regexp, +.roost .hljs-deletion { + color: #d91e18; +} + +/* Orange */ +.roost .hljs-number, +.roost .hljs-built_in, +.roost .hljs-literal, +.roost .hljs-type, +.roost .hljs-params, +.roost .hljs-meta, +.roost .hljs-link { + color: #aa5d00; +} + +/* Yellow */ +.roost .hljs-attribute { + color: #aa5d00; +} + +/* Green */ +.roost .hljs-string, +.roost .hljs-symbol, +.roost .hljs-bullet, +.roost .hljs-addition { + color: #008000; +} + +/* Blue */ +.roost .hljs-title, +.roost .hljs-section { + color: #007faa; +} + +/* Purple */ +.roost .hljs-keyword, +.roost .hljs-selector-tag { + color: #7928a1; +} + +.roost .hljs-emphasis { + font-style: italic; +} + +.roost .hljs-strong { + font-weight: bold; +} + +@media screen and (-ms-high-contrast: active) { + .roost .hljs-addition, + .roost .hljs-attribute, + .roost .hljs-built_in, + .roost .hljs-bullet, + .roost .hljs-comment, + .roost .hljs-link, + .roost .hljs-literal, + .roost .hljs-meta, + .roost .hljs-number, + .roost .hljs-params, + .roost .hljs-string, + .roost .hljs-symbol, + .roost .hljs-type, + .roost .hljs-quote { + color: highlight; + } + + .roost .hljs-keyword, + .roost .hljs-selector-tag { + font-weight: bold; + } +} + +@media (prefers-color-scheme: dark) { + /*! + Theme: a11y-dark + Author: @ericwbailey + Maintainer: @ericwbailey + + Based on the Tomorrow Night Eighties theme: https://github.com/isagalaev/highlight.js/blob/master/src/styles/tomorrow-night-eighties.css + */ + + .roost .hljs { + background: #2b2b2b; + color: #f8f8f2; + } + + /* Comment */ + .roost .hljs-comment, + .roost .hljs-quote { + color: #d4d0ab; + } + + /* Red */ + .roost .hljs-variable, + .roost .hljs-template-variable, + .roost .hljs-tag, + .roost .hljs-name, + .roost .hljs-selector-id, + .roost .hljs-selector-class, + .roost .hljs-regexp, + .roost .hljs-deletion { + color: #ffa07a; + } + + /* Orange */ + .roost .hljs-number, + .roost .hljs-built_in, + .roost .hljs-literal, + .roost .hljs-type, + .roost .hljs-params, + .roost .hljs-meta, + .roost .hljs-link { + color: #f5ab35; + } + + /* Yellow */ + .roost .hljs-attribute { + color: #ffd700; + } + + /* Green */ + .roost .hljs-string, + .roost .hljs-symbol, + .roost .hljs-bullet, + .roost .hljs-addition { + color: #abe338; + } + + /* Blue */ + .roost .hljs-title, + .roost .hljs-section { + color: #00e0e0; + } + + /* Purple */ + .roost .hljs-keyword, + .roost .hljs-selector-tag { + color: #dcc6e0; + } + + .roost .hljs-emphasis { + font-style: italic; + } + + .roost .hljs-strong { + font-weight: bold; + } + + @media screen and (-ms-high-contrast: active) { + .roost .hljs-addition, + .roost .hljs-attribute, + .roost .hljs-built_in, + .roost .hljs-bullet, + .roost .hljs-comment, + .roost .hljs-link, + .roost .hljs-literal, + .roost .hljs-meta, + .roost .hljs-number, + .roost .hljs-params, + .roost .hljs-string, + .roost .hljs-symbol, + .roost .hljs-type, + .roost .hljs-quote { + color: highlight; + } + + .roost .hljs-keyword, + .roost .hljs-selector-tag { + font-weight: bold; + } + } +} diff --git a/docs/theme/css/roost.css b/docs/theme/css/roost.css new file mode 100644 index 00000000..2f3b6bc3 --- /dev/null +++ b/docs/theme/css/roost.css @@ -0,0 +1,142 @@ +.roost, +html:not(.js) { + color-scheme: light dark; + background-color: var(--roost-charcoal); + + --roost-gray: rgb(187 193 190); + --roost-charcoal: hsl(80, 22%, 13%); + --roost-yellow: rgb(238 238 0); + --roost-peach: rgb(247 187 128); + --roost-muddy: rgb(229 217 136); + + --color-scheme: light; + --bg: rgba(250 250 250); + --fg: var(--roost-charcoal); + --sidebar-header-border-color: oklch(from var(--roost-yellow) calc(l * 0.95) c h); + --sidebar-resize-indicator-space: 0px; + --sidebar-bg: var(--roost-charcoal); + --sidebar-fg: rgba(255 255 255 / 0.9); + --sidebar-non-existant: #5c6773; + --sidebar-active: var(--roost-muddy); + --sidebar-spacer: var(--roost-muddy); + + --icons: white; + --icons-hover: var(--roost-yellow); + --links: inherit; + + --theme-popup-bg: var(--bg); + --theme-popup-border: color-mix(var(--fg) 15%, transparent); + --theme-hover: color-mix(var(--fg) 4%, transparent); + + --quote-bg: color-mix(var(--fg) 3%, transparent); + --quote-border: color-mix(var(--fg) 10%, transparent); + + --searchbar-bg: color-mix(var(--fg) 3%, transparent); + --searchbar-fg: inherit; + --searchbar-border-color: color-mix(var(--fg) 10%, transparent); + --searchbar-shadow-color: var(--roost-peach); + --searchbar-margin-block-start: 2em; + + --warning-border: color-mix(orange, var(--fg)); + + --table-border-color: color-mix(var(--fg) 7%, var(--bg)); + --table-header-bg: color-mix(var(--fg) 10%, var(--bg)); + --table-alternate-bg: color-mix(var(--fg) 3%, var(--bg)); + + --scrollbar: color-mix(var(--fg), transparent); + + --inline-code-color: inherit; + + --searchresults-header-fg: color-mix(var(--roost-peach) 75%, var(--fg)); + --searchresults-border-color: color-mix(var(--fg), transparent); + --searchresults-li-bg: color-mix(var(--roost-yellow) 20%, transparent); + --search-mark-bg: color-mix(var(--roost-yellow), transparent); + --footnote-highlight: var(--search-mark-bg); + + --copy-button-filter: none; + --copy-button-filter-hover: invert(20%); + + --blockquote-note-color: color-mix(blue, var(--fg)); + --blockquote-tip-color: color-mix(lime, var(--fg)); + --blockquote-important-color: color-mix(fuchsia, var(--fg)); + --blockquote-warning-color: var(--warning-border); + --blockquote-caution-color: color-mix(red, var(--fg)); + + --overlay-bg: color-mix(var(--fg), transparent); +} + +html:not(.js) { + --sidebar-resize-indicator-width: 0px; +} + +@media (prefers-color-scheme: dark) { + .roost, + html:not(.js) { + --color-scheme: dark; + --bg: color-mix(var(--roost-charcoal) 90%, black); + --fg: rgba(255 255 255 / 0.9); + --copy-button-filter: invert(80%); + --copy-button-filter-hover: invert(100%); + } +} + +/* Theme overrides */ + +.roost body { + background-color: var(--bg); + min-height: calc(100vh + var(--menu-bar-height)); +} + +.roost #mdbook-menu-bar { + background-color: var(--roost-charcoal); + border: none; + box-shadow: 0 0.25em 1em rgba(0 0 0 / 0.25); +} + +.roost #mdbook-theme-toggle { + x-display: none; +} + +.roost #mdbook-searchbar { + border-radius: 999px; + transition: none; +} + +.roost #mdbook-searchbar:focus-visible { + outline: 1px solid color-mix(var(--searchbar-shadow-color), transparent); +} + +.roost ul#mdbook-searchresults li { + margin-block: 1.25em; +} + +.roost #mdbook-sidebar { + z-index: 102; +} + +.roost #mdbook-sidebar-resize-handle { + background-color: var(--sidebar-bg); +} + +.roost :not(pre):not(a) > .hljs { + background: color-mix(var(--fg) 10%, transparent); +} + +.roost .nav-wide-wrapper { + --icons: color-mix(var(--fg) 25%, transparent); + --icons-hover: var(--fg); +} + +.roost .content { + padding: 2em 1em 1em; +} + +.roost .content a { + font-weight: bold; + text-decoration: underline; +} + +.roost a:link.mobile-nav-chapters { + color: var(--roost-yellow); +} + diff --git a/docs/theme/fonts/fonts.css b/docs/theme/fonts/fonts.css index 5180f001..3f6bd375 100644 --- a/docs/theme/fonts/fonts.css +++ b/docs/theme/fonts/fonts.css @@ -7,13 +7,10 @@ font-style: normal; } + h1, h2, h3, h4, h5, h6 { font-family: "Funnel Display", sans-serif; font-optical-sizing: auto; font-weight: 700; font-style: normal; } - -.content a { - text-decoration: underline; -} diff --git a/docs/theme/index.hbs b/docs/theme/index.hbs new file mode 100644 index 00000000..1c790abe --- /dev/null +++ b/docs/theme/index.hbs @@ -0,0 +1,367 @@ + + + + + + {{ title }} + {{#if is_print }} + + {{/if}} + {{#if base_url}} + + {{/if}} + + + + {{> head}} + + + + + + {{#if favicon_svg}} + + {{/if}} + {{#if favicon_png}} + + {{/if}} + + + + {{#if print_enable}} + + {{/if}} + + + + + + + + + + + {{#each additional_css}} + + {{/each}} + + {{#if mathjax_support}} + + + {{/if}} + + + + + + + +
+
+

Keyboard shortcuts

+
+

Press ← or → to navigate between chapters

+ {{#if search_enabled}} +

Press S or / to search in the book

+ {{/if}} +

Press ? to show this help

+

Press Esc to hide this help

+
+
+
+
+ + + + + + + + + + + + + +
+ +
+ {{> header}} +
+ + + {{#if search_enabled}} + + {{/if}} + + + + +
+
+ {{{ content }}} +
+ + +
+
+ + + +
+ + + + + + + + {{#if live_reload_endpoint}} + + + {{/if}} + + {{#if playground_line_numbers}} + + {{/if}} + + {{#if playground_copyable}} + + {{/if}} + + {{#if playground_js}} + + + + + + {{/if}} + + {{#if search_js}} + + + + {{/if}} + + + + + + + {{#each additional_js}} + + {{/each}} + + {{#if is_print}} + {{#if mathjax_support}} + + {{else}} + + {{/if}} + {{/if}} + + {{#if fragment_map}} + + {{/if}} + +
+ + diff --git a/docs/user/README.md b/docs/user/README.md index b79ff014..4dcc376c 100644 --- a/docs/user/README.md +++ b/docs/user/README.md @@ -1,27 +1,21 @@ # User Guide -Osprey is a web-based investigation and management console for safety teams. Query event data in real time, visualize trends, label entities, manage rules and features, and run bulk operations. +Osprey is a web-based investigation and management console for safety teams. Query event data in real time, visualize trends, label entities, manage rules and features, and run bulk operations. If terms like events, features, and labels are new to you, start with [Concepts](../concepts.md). -Osprey's tools are categorized into three primary functions: +[![The Osprey UI during an investigation: a query filtering events where SuspiciousDisplayName is True, timeseries charts of matching event volume, and the live event stream with one entity's label popup open showing a negative spam_display_name label](../images/osprey-suspicious-display-name.png)](../images/osprey-suspicious-display-name.png) -**Investigate**: query events and examine entity behavior +The sidebar groups Osprey's tools by task, and this guide follows the same sections: -- [Query](investigate/#query): the main investigation page -- [Query History](investigate/#query-history): browse and re-run past queries -- [Saved Queries](investigate/#saved-queries): manage frequently used queries +- **[Investigate](investigate/)**: query events in real time, chart the results, and drill into individual events and entities. Query history and saved queries let you revisit and share past investigations. -**Manage**: understand and navigate your rule and feature configuration + [![Two Top N tables for a query, grouping matching events by event type and by post text](../images/multiple-top-charts.png)](../images/multiple-top-charts.png) -- [Rules Visualizer](manage.md#rules-visualizer): graph view of label/rule relationships -- [UDF Registry](manage.md#udf-registry): API reference for all available functions -- [Features](manage.md#features): inventory of all features in the system -- [Rules](manage.md#rules): inventory of all rules in the system +- **[Manage](manage.md)**: browse the rules, features, and UDFs configured in your deployment, and visualize how rules and labels relate. -**Operate**: run and review bulk operations + [![The UDF Registry listing available functions with type signatures and descriptions, grouped by category](../images/udf-documentation.png)](../images/udf-documentation.png) -- [Bulk Actions](operate.md#bulk-actions): start and monitor bulk labeling jobs -- [Bulk Job History](operate.md#bulk-job-history): review past bulk jobs and their results +- **[Operate](operate.md)**: run bulk labeling jobs over query results and review past jobs. -The sidebar can be collapsed to a narrow icon-only strip using the toggle at the bottom. Its state persists between sessions. + [![The Bulk Edit Labels form for roughly 9,800 entities matching a query, with label name, status, reason, and expiration fields](../images/bulk-label.png)](../images/bulk-label.png) -The interface supports light and dark themes, see [Appearance](appearance.md) for details. \ No newline at end of file +The sidebar can be collapsed to an icon-only strip with the toggle at the bottom, and its state persists between sessions. The interface follows your OS or browser color scheme preference, or you can select a light or dark style using the toggle in the navigation bar. diff --git a/docs/user/appearance.md b/docs/user/appearance.md deleted file mode 100644 index 710f9769..00000000 --- a/docs/user/appearance.md +++ /dev/null @@ -1,19 +0,0 @@ -# Appearance - -Osprey supports light and dark themes. By default, the interface follows your OS or browser colour-scheme preference. You can override this at any time using the theme toggle in the navigation bar. - -## Changing the theme - -The theme toggle is in the top-right area of the navigation bar. It offers three options: - -| Option | Icon | Behaviour | -|--------|------|-----------| -| **Light** | ☀ Sun | Always use the light theme | -| **System** | 🖥 Desktop | Follow the OS/browser preference *(default)* | -| **Dark** | ☾ Moon | Always use the dark theme | - -Your choice is saved in the browser and persists across sessions. Clearing browser storage resets it to **System**. - -## System preference - -When set to **System**, Osprey listens for OS colour-scheme changes in real time. Switching your OS between light and dark mode while Osprey is open takes effect immediately, no page reload required. \ No newline at end of file diff --git a/docs/user/investigate/README.md b/docs/user/investigate/README.md index 9b25f036..598ad05a 100644 --- a/docs/user/investigate/README.md +++ b/docs/user/investigate/README.md @@ -6,17 +6,17 @@ The Investigate section is where most day-to-day work happens. It contains the m The Query page is the home page of Osprey. It's a live investigation workspace with three panels. -![Osprey Home](../../images/osprey-home.png) +![The Query page on first load: an empty query input with a Last Day time range on the left above the query history, and empty Timeseries Chart, Top N Results, and Event Stream panels awaiting data](../../images/osprey-home.png) ### Query input The left panel is where you write and run queries. Osprey uses SML syntax—the same language used to write rules—to filter and search event data. See [Query Syntax](query-syntax.md) for a full reference. -![Query Box](../../images/query-box.png) +![The query input with an SML filter for post-creation events by one user, a Last 2 Weeks time range, and a Submit Query button](../../images/query-box.png) As you type, the input offers autocomplete suggestions for feature names, action names, and UDFs. Hovering over any UDF name shows a tooltip with its description. -![Query UDF Hover](../../images/query-udf-hover.png) +![Autocomplete in the query input suggesting the ContainsHello UDF after typing "Contains", with a tooltip describing it as "Post contains the word hello"](../../images/query-udf-hover.png) The active query is reflected in the page URL, making it easy to share a specific investigation with a teammate, though be aware this may expose sensitive query parameters. @@ -24,7 +24,7 @@ The active query is reflected in the page URL, making it easy to share a specifi Every query runs against a time window. You can choose a preset interval—from the last second up to the last three months—or set a custom date range using the date picker. -![Query Time Range](../../images/query-time-range.png) +![The time range dropdown under the query input, open to presets ranging from Last 12 Hours to Last 3 Months](../../images/query-time-range.png) The entire page updates dynamically when the query or time range changes, and interacting with any other panel (charts, event stream) can also update the query in turn. @@ -32,35 +32,36 @@ The entire page updates dynamically when the query or time range changes, and in The center panels show two types of visualizations: -![Charts](../../images/charts.png) +![A Timeseries Chart of daily matching event counts above a Top N Results table grouping events by PostText, with Download CSV, Bulk Label, PoP, Yeet Table, and Precision controls along the table footer](../../images/charts.png) **Timeseries** displays how many matching events occurred over time. You can set the granularity—minute, fifteen minutes, half hour, hour, day, week, or month—and hover over individual bars to see the count for that period. -![Time Series Hover](../../images/hover-time-series.png) +![Hovering over a timeseries bar, showing a tooltip with that bucket's date, time, and event count](../../images/hover-time-series.png) -You can add additional timeseries charts to compare different time granularities side-by-side. Charts you no longer need can be removed ("yeeted"). +You can add additional timeseries charts to compare different time granularities side by side. Charts you no longer need can be removed with the **Yeet** button. -![Multiple Time Series](../../images/multiple-time-series.png) +![Two timeseries charts for the same query stacked vertically: daily granularity above, per-minute granularity below, with a Yeet button beside the second chart](../../images/multiple-time-series.png) **Top N** shows a table of the top results for the current query, grouped by a dimension you choose. You can: - Add or remove dimension columns -- Adjust the number of results shown (precision) -- Enable **Period over Period (PoP)** to compare current results against a past time window and see the delta +- Adjust the number of results shown with **Precision** +- Toggle **PoP** (period over period) to compare current results against a past time window and see the delta - Export the table as a CSV +- Remove the table with the **Yeet Table** button -![Top N Charts](../../images/top-n-charts.png) +![A Top N Results table grouping matching events by UserId, sorted by count](../../images/top-n-charts.png) -![Period over Period](../../images/pop.png) +![A Top N Results table with PoP enabled, adding delta and percent-change columns comparing each count against the previous period, with values redacted](../../images/pop.png) ### Event stream -The right panel is Osprey's live feed. It shows individual events matching the current query in near-real time, and can also be used to search historical events. +The right panel is Osprey's live feed. It shows individual events matching the current query in near real time, and can also be used to search historical events. -![Event Stream](../../images/event-stream.png) +![The Event Stream filtered to one UserId, showing payment events as cards listing each extracted feature, with sensitive values redacted](../../images/event-stream.png) -The stream can be displayed in card format or list format. You can customize which fields appear per event by selecting **Summary Features**, helpful when different team members care about different metadata. +The stream can be displayed in card format or list format. On first load, each event card shows all of its extracted features (or the summary features configured for that event type, if any), so the stream is useful before you've set anything up. You can customize which fields appear per event with **Select Summary Features**, helpful when different team members care about different metadata. -![Summary Features](../../images/summary-features.png) +![The Set Features modal for the event stream, with a toggle for Set Custom Features and a filterable checklist of features grouped by model and rule](../../images/summary-features.png) Selecting an entity in the event stream (such as a user ID or IP address) opens the [Entity Details](labels.md#entity-details) view. Hold Ctrl (⌘ Cmd on macOS) while clicking to select multiple events for bulk labeling. @@ -70,27 +71,30 @@ Selecting an event opens a detail view at `/events/:eventId` showing all extract Every query you run is automatically saved to your history. -![Query History](../../images/query-history.png) +![A history entry highlighted in the sidebar's History tab, showing its query text and original time range beside the results it produced](../../images/query-history.png) Hovering over a query in the sidebar shows the Top N dimensions that were active during that session. The full Query History page (accessible from the sidebar) shows a searchable list of all queries run across your team. You can filter by user email, view the original query text, and re-run any past query using the same time range it was originally run with. -![Query History Page](../../images/query-history-page.png) +![The Query History page listing each query with its text and time range, buttons to view the original query or re-run it, a filter-by-user dropdown, and a Saved Queries sidebar](../../images/query-history-page.png) ## Saved Queries For queries you return to frequently, Osprey lets you save them by name. -![Save Query](../../images/query-history-save.png) +![The menu on a query history entry, with View Original Query and Save Query options](../../images/query-history-save.png) -The Saved Queries page (accessible from the sidebar) shows a grid of all saved queries with the query text, the user who saved it, and when it was saved. You can filter by user email. +The Saved Queries page (accessible from the sidebar) lists all saved queries with the query text, the user who saved it, and when it was saved. You can filter by user email. -![Saved Queries Page](../../images/saved-queries-page.png) +![The Saved Queries page listing a saved query with its text and metadata, and an open menu of Run Original Query, Run Query Using Interval, Show Saved Query History, and Delete Saved Query options](../../images/saved-queries-page.png) -From the grid, you can: -- **Run** a saved query to load it into the Query page -- **Rename** it via an edit modal -- **Delete** it (with a confirmation step) +From each saved query's menu, you can: +- **Run Original Query** to re-run it over its original time range +- **Run Query Using Interval** to run it over its saved interval (e.g. Last Day) relative to now +- **Show Saved Query History** to see past runs of the query +- **Delete Saved Query**, with a confirmation step + +To rename a saved query, run it and select the edit icon next to its name on the Query page. Saved queries also have a direct URL: `/saved-query/:savedQueryId/latest` automatically loads and executes the query. diff --git a/docs/user/investigate/labels.md b/docs/user/investigate/labels.md index 574e2764..a1d03b71 100644 --- a/docs/user/investigate/labels.md +++ b/docs/user/investigate/labels.md @@ -2,7 +2,7 @@ Labels are annotations you apply to entities: users, IP addresses, emails, and other tracked objects. They're the bridge between human judgment and Osprey's automated rule system; a label you apply manually can feed into rules that act on future events automatically. -Labels have three polarities: +Labels have one of three connotations: - **Negative**: harmful or problematic (e.g. `spammer`, `bot`, `banned`, `suspicious`) - **Positive**: trusted or verified (e.g. `verified`, `trusted`, `premium_user`) @@ -14,7 +14,7 @@ A reason is required whenever you apply a label. Selecting an entity anywhere in the UI navigates to an entity view showing every label that has ever been applied to that entity, grouped by label name. -![User Entity](../../images/osprey-user-entity.png) +![The entity view for a user, listing their negative labels—including identity_evasion with its description and an automatically added RapidHandleChange entry—beside the query page's charts and event stream](../../images/osprey-user-entity.png) Each label entry shows: - The label value and type @@ -26,12 +26,12 @@ Each label entry shows: From a Top N table, hover over an entity row and select **Edit Labels**: -![Add Labels](../../images/add-labels.png) +![The label popup for a PostText entity in a Top N table, with keyboard hints for adding the entity to the query and an Edit Labels button](../../images/add-labels.png) From the event stream, select any entity to open its label drawer. -![Empty Label](../../images/empty-label.png) +![The label drawer for a user entity, with a label name search, reason and expiration fields, and empty negative, positive, and neutral label sections](../../images/empty-label.png) -![Complete Label](../../images/complete-label.png) +![An entity view with negative labels applied, one expanded to show its description and the rule event that produced it, plus neutral labels below; identifying values are redacted](../../images/complete-label.png) -For more about how labels are used, see [Writing Rules → Labels](../../rules.md#labels). +For more about how labels are used, see [Writing Rules → Labels](../../rules/#labels). diff --git a/docs/user/investigate/query-syntax.md b/docs/user/investigate/query-syntax.md index 9aca6cd2..d2ad2a80 100644 --- a/docs/user/investigate/query-syntax.md +++ b/docs/user/investigate/query-syntax.md @@ -1,30 +1,13 @@ # Query Syntax -Osprey uses SML (“Some Madeup Language,” a subset of Python with additional restrictions) for queries. Queries filter events by matching against features, actions, and labels. - -## Core concepts - -**Actions** are events that happen in your system, like a user creating a post or sending a message. Your query filters which action types to look at. - -**Features** are named variables extracted from events. All features are in a global namespace; any feature exported by Osprey rules is queryable. Prefixing a variable with `_` keeps it local to a rule file and excludes it from querying. - -```py -UserId: Entity[int] = EntityJson(type='User', path='$.user.id', coerce_type=True) -UserEmail: str = JsonData(type='Email', path='$.user.email', required=False) -``` - -Both `UserId` and `UserEmail` above are features. - -**Entities** are a special kind of feature. They can have effects applied to them: labels, classifications, signals. Clicking an entity in the UI navigates to its [Entity Details](labels.md#entity-details) view. - -**Effects** are outcomes triggered when one or more rules evaluate to true; for example, applying a `spammer` label to a `UserId` entity. +Osprey uses SML (“Some Madeup Language,” a subset of Python with additional restrictions) for queries. Queries match against **features**—the named values your rules extract from each event—including **entities** and **labels**. If those terms are new, read [Concepts](../../concepts.md) first. ## Basic comparisons ```py EventType == "create_post" UserId == 12345 -MessageText != Null +MessageText != None ``` ## Combining conditions @@ -40,29 +23,29 @@ EventType == "user_login" and LoginAttempts >= 3 EventType in ["create_post", "send_message"] ``` -## Using UDFs +## Query functions -UDFs (see [UDF Registry](../manage.md#udf-registry)) extend what you can express in a query: +Queries support a small, fixed set of built-in functions. Unlike the UDFs you use in rules (which are pluggable), this set does not grow when you add plugins: + +- `RegexMatch(target=..., pattern=...)`: regex match against a feature +- `DidAddLabel(...)` and `DidRemoveLabel(...)`: events where a label was added or removed (see [Label queries](#label-queries)) +- `DidDeclareVerdict(...)`: events where a verdict was declared ```py -# Text search -TextContains(text=PostContent, phrase="spam") +# Regex match against a feature RegexMatch(target=MessageText, pattern="(buy|sell|deal)") - -# List operations -ListLength(list=UserConnections) > 10 ``` > [!NOTE] -> If you query a UDF that doesn't exist, Osprey will silently fail with a 500 error. Use the UDF Registry to confirm a function name before using it. +> These are the only functions that work in queries. The [UDF Registry](../manage.md#udf-registry) lists every UDF, but most are rules-only and will fail with a silent 500 error if you use them in a query. ## Label queries -The query UI searches across actions (events), not entity state, so `HasLabel()` won't work here. Use `DidAddLabel()` instead, which matches events where a label was added: +The query UI searches across events, not entity state, so `HasLabel()` won't work here. Use `DidAddLabel()` instead, which matches events where a label was added: ```py # Find events that added a specific label -DidAddLabel(entity_type="UserId", label_name="likely_spammer") +DidAddLabel(entity_type="User", label_name="likely_spammer") DidAddLabel(entity_type="IpAddress", label_name="suspicious") ``` @@ -72,14 +55,14 @@ DidAddLabel(entity_type="IpAddress", label_name="suspicious") # Suspicious login attempts EventType == "user_login" and LoginAttempts >= 5 -# Posts containing specific words -EventType == "create_post" and TextContains(text=PostContent, phrase="urgent") +# Posts matching a pattern +EventType == "create_post" and RegexMatch(target=PostContent, pattern="urgent") # Users who were flagged -DidAddLabel(entity_type="UserId", label_name="flagged") +DidAddLabel(entity_type="User", label_name="flagged") # Complex: messages matching a pattern, from users without a verified label EventType == "send_message" and RegexMatch(target=MessageText, pattern="(click|link|urgent)") and -not DidAddLabel(entity_type="UserId", label_name="verified") +not DidAddLabel(entity_type="User", label_name="verified") ``` diff --git a/docs/user/manage.md b/docs/user/manage.md index 36ed4225..671d8e3b 100644 --- a/docs/user/manage.md +++ b/docs/user/manage.md @@ -4,11 +4,11 @@ The Manage section provides visibility into your Osprey configuration: the rules ## Rules Visualizer -The Rules Visualizer shows how labels, rules, and other labels relate to one another in a dependency graph. It's useful for understanding what will fire when a particular label is applied, or what conditions must be true for a label to be produced. +The Rules Visualizer shows how rules and labels relate to one another in a dependency graph. It's useful for understanding what will fire when a particular label is applied, or what conditions must be true for a label to be produced. -![Rules Visualizer](../images/rules-visualizer.png) +![The Rules Visualizer page with its search box open, offering Actions and Labels categories, before anything is selected](../images/rules-visualizer.png) -To use it, select a feature or label from the search interface. A graph appears showing the upstream and downstream relationships for your selection. You can toggle upstream and downstream visibility independently. +To use it, search for an action or label in the search box. A graph appears showing the upstream and downstream relationships for your selection. You can toggle upstream and downstream visibility independently. Node types in the graph: - **Red ellipse**: a label that is upstream of a rule (an input condition) @@ -21,7 +21,7 @@ Hovering over a node shows its source file path. The graph supports zoom and pan The UDF Registry is an auto-generated API reference for every user-defined function (UDF) available in Osprey. It updates dynamically as UDFs are added or modified in code, so it always reflects what's actually available. -![UDF Documentation](../images/udf-documentation.png) +![The UDF Registry listing available functions with type signatures and descriptions, grouped by category](../images/udf-documentation.png) UDFs are organized by category and are searchable. Each entry shows: - Function signature with syntax highlighting @@ -29,11 +29,11 @@ UDFs are organized by category and are searchable. Each entry shows: - Parameter names, types, and descriptions - Return type -Use this page as your reference when writing queries or rules: especially to confirm a function's exact name and parameter order before using it. (Querying a UDF that doesn't exist causes a silent 500 error.) +Use this page as your reference when writing rules, to confirm a function's exact name and parameter order. Note that most UDFs are for rules only: the query box supports just four functions (`RegexMatch`, `DidAddLabel`, `DidRemoveLabel`, `DidDeclareVerdict`), so using any other UDF in a query currently fails with a silent 500 error. -## Features +## Features Registry -The Features page lists every feature defined in your Osprey deployment. Features are named variables extracted from events; they're what you query against and what rules operate on. +The Features Registry lists every feature defined in your Osprey deployment. Features are named variables extracted from events; they're what you query against and what rules operate on. The list is paginated (50 per page) and can be filtered and sorted: @@ -45,9 +45,9 @@ The list is paginated (50 per page) and can be filtered and sorted: Each row shows the feature's name, category, extraction function(s), reference count (how many rules use it), description, owner, and last modified date. -## Rules +## Rules Registry -The Rules page lists every rule loaded in your Osprey deployment. +The Rules Registry lists every rule loaded in your Osprey deployment. The list is paginated (50 per page) and can be filtered and sorted: diff --git a/docs/user/operate.md b/docs/user/operate.md index 73884b32..dea46f1b 100644 --- a/docs/user/operate.md +++ b/docs/user/operate.md @@ -13,17 +13,21 @@ Bulk Actions let you apply a label to every entity matching the current query; u There are two ways to start a bulk action: -1. **Bulk Actions** → **Create New Job** to open the job creation modal +1. **Bulk Actions** → **Create New Job** to create a job from an uploaded file of entities 2. **From the Query page**: use the bulk label drawer in the chart column, which pre-fills the query from your current session -![Bulk Label](../images/bulk-label.png) +![The Bulk Edit Labels form for roughly 9,800 entities matching a query, with label name, status, reason, and expiration fields](../images/bulk-label.png) Before submitting, Osprey shows a count of how many unique entities will be labeled. Review this number carefully to understand the impact of your bulk action. -Each job requires: -- The entity type and label to apply +> For scripted or bulk labeling from the command line (e.g., importing label lists from external sources), see `apply_label` and `bulk_apply_label` in the [CLI Reference](../development/cli-reference.md#osprey-cli-apply_label). + +Each labeling job takes: +- The label to apply and its status - A reason (required for all labeling operations) -- The label polarity (negative, positive, or neutral) +- An optional expiration + +Standard bulk jobs are limited to 100,000 entities; the form has a checkbox to remove the limit for larger jobs. ### Monitoring jobs @@ -35,7 +39,7 @@ The page polls for updates automatically; you don't need to refresh to see progr The Bulk Job History page provides a detailed view of past bulk jobs. -![Bulk Job History](../images/bulk-job-history.png) +![The empty Bulk Job History page: a task ID search field on the left and a Find Single Bulk Job panel on the right](../images/bulk-job-history.png) The page has two columns: - Recent bulk jobs with status, progress, and summary statistics; filterable by task ID