ir!: the validation constraints WA Web enforces, not just field shapes (schemaVersion 2.0.0)

README.md

@@ -14,12 +14,31 @@ WhatsApp Web ships its whole protocol (IQ stanzas, protobuf schemas, GraphQL ope
 | **appstate** | `appstate/index.json` (+ Rust) | app-state (syncd) action schemas + indexing |
 | **abprops** | `abprops/index.json` (+ Rust) | the A/B-props feature-flag registry (~1.7k flags: code, type, default) |
 | **enums** | `enums/index.json` (+ Rust) | the wire-enum catalog (nack codes, chat/receipt types, …) |
-| **notif** | `notif/index.json` (+ Rust) | the incoming stanza-dispatch catalog: `<notification type="…">` kinds + handlers + typed content shapes |
+| **notif** | `notif/index.json` (+ Rust) | the incoming stanza-dispatch catalog: `<notification type="…">` kinds + handlers + typed content shapes + the payload **action unions** (`w:gp2`'s 40+ group actions) |
 | **tokens** | `tokens/index.json` (+ `tokens.json`) | the binary-protocol token dictionaries (single-byte + 4 double-byte), wire-indexable |
 | **wasm** | `wasm/index.json` | the WebAssembly surface: emscripten payload names (`liboqs_wasm_wrapper.wasm`, …) + the bootloader (`bx`) handles the JS resolves their bytes through, with consuming modules |
 
 Every domain ships a JSON Schema under `generated/schema/`, and a top-level `generated/manifest.json` stamps the WhatsApp version, per-domain counts, content hashes, and extraction diagnostics.
 
+### Validation constraints, not just field shapes
+
+Field shapes tell you how to *read* a stanza. They are not enough to *produce* one, nor to explain why a real client rejected the one you produced. So the IR also carries the rules WA Web's own parsers enforce — symmetric by construction, so they serve a parser and an emitter equally:
+
+- **Echo rules** (`assertions[].kind == "reference"`, and `referencePath` on a field) — an answer's `from` must equal the **request's** `to`, its `id` the request's `id`. `referencePath` is the argument list of WA's `attrStringFromReference`, so `["to"]` means "the request's `to`" and `["account","action"]` means "the `action` attribute of the request's `<account>` child" — no name-matching required. An emitter that hardcodes `from="s.whatsapp.net"` makes every answer to a `g.us` request unparseable.
+- **Pinned values** (`literalValue` on a field) — `type="admin"` on a successful promote, `matched="true"`/`"false"` on a blocklist update, `code=429` on a rate-limit error. `required` separates the two forms: a required pin is a hard discriminator (must be present and exact); an optional one is pinned only when present (may be omitted, must never be contradicted).
+- **The per-RPC error vocabulary** (`errorArms`, plus `errorEnvelope` for a two-level error) — a **closed** set, and it differs per RPC: `BatchGetGroupInfo` accepts `400 bad-request` and `429 rate-overlimit` and **rejects `404 item-not-found`**, even though that mixin exists and other RPCs use it. Each arm pairs the `code` with the `text` that must accompany it, so an emitter cannot combine one arm's code with another's text and produce a stanza no branch accepts; an arm that range-checks instead of pinning carries `codeMin`/`codeMax`. The variant's `kind` says which side is at fault (`client_error` / `server_error`), derived from the codes rather than from the parser's name.
+- **Response enums** (`enumRef` on a field) — the legal values behind an `attrStringEnum`, resolved the same way the request side already resolves them, instead of a bare `"type": "enum"`.
+- **The accessor's decoded type**, kept faithful: every `attr*` / `content*` spelling WA's parsers use is classified (a `maybeAttrX` derives from `attrX`, so a flavour cannot be covered for one spelling and missed for the other), and the JID flavours stay distinct — a PN user JID and a LID user JID are different identities for the same person and must never collapse into one `string`.
+- **Notification action unions** (`notifications[].actions`) — the payload inside the envelope. The `wireTag → actionType` mapping is **many-to-one** (`not_ephemeral` normalises into `ephemeral` with `duration: 0`, so branching on `not_ephemeral` is dead code) and field names are rebound (the disappearing-message timer arrives in `expiration`, but the action field is `duration`). Neither is derivable from the wire.
+
+Note the contract version: this raised `schemaVersion` to **2.0.0**, and it is a real major bump rather than a cautious one. Three changes need action from a 1.x consumer:
+
+- `AssertionKind` gained a `reference` variant, widening the value space of an existing field — a closed-enum consumer rejects the document rather than ignoring it (validating the current `iq/index.json` against the 1.0 schema fails 579 times).
+- `ResponseVariantKind` gained `client_error` and `server_error`; a variant that was `error` may now be either. Match on all three, or use the `is_error()` grouping.
+- A response variant's `errorCodes` / `errorTexts` / `errorCodeMin` / `errorCodeMax` / `errorClass` are **gone**, replaced by `errorArms` (+ `errorEnvelope`). The flat lists were removed rather than kept alongside because they were unsound: two independent lists cannot say which code goes with which text, and 117 variants admitted combinations the parser rejects.
+
+Anything the extractor sees but cannot resolve structurally is counted under `manifest.diagnostics.iq.dropsByReason` rather than omitted, so "no constraint here" and "a constraint we failed to extract" never look alike. `manifest.diagnostics.iq.constraints` and `diagnostics.notif.actions` are floor-guarded: a WA refactor that hides one of these constructs fails the update instead of silently emptying a field.
+
 ## Quick start
 
 ```sh
@@ -35,7 +54,8 @@ cargo run --release -p whatspec -- update --cache .wa-cache
 # Seed that same cache from the lock-pinned GitHub Release (no live bundle fetch):
 cargo run --release -p whatspec -- restore --from-lock generated/bundles.lock.json --cache .wa-cache
 
-# Also resolve, download and store the client's wasm payloads (~41 MB), as <url-hash>.wasm:
+# Also resolve, download and store the client's wasm payloads (~41 MB), as <url-hash>.wasm
+# (a cache filename is a location label, not a content address — see the note below):
 cargo run --release -p whatspec -- update --cache .wa-cache --wasm-out ./wasm
 
 # Restore a locked wasm set (its own lock + its own content-addressed release asset):
@@ -56,6 +76,7 @@ WhatsApp only serves the *current* bundle version — old bundle URLs 404 — so
 
 - **`generated/bundles.lock.json`** records the content SHA-256 (+ size, and origin URL when known) of every bundle that produced the committed `generated/`, plus a one-line, order-invariant `setHash` fingerprint of the whole set.
 - The **bytes** live in a durable, WhatsApp-independent store: a rolling **`bundle-store`** GitHub Release whose assets are **content-addressed** — `bundles-<version>-<setHash>.tar.xz` (pure-Rust xz, roughly half the size of gzip; legacy `.tar.gz` assets are still read) — so a set is never overwritten with different bytes and every past commit's lock keeps resolving the exact archive it pins. Published automatically by the update workflow.
+  Note the two different hashes: a **release asset** is named after its *contents* (`setHash`), while a **cache** filename is named after its *URL* (the JS last segment, or `sha256(url)` for wasm) — there, the content hash lives in the cache's `manifest.json` and is what integrity is verified against. A cache filename never proves what is inside the file.
 - **`whatspec restore --from-lock generated/bundles.lock.json --out <dir>`** pulls that archive, verifies every bundle's SHA-256 against the lock, and writes a directory ready for `update --bundles`. Use **`--cache <dir>`** instead to seed the exact version directly into the reusable `update --cache` layout; cache metadata and integrity files are written by the same `BundleCache` implementation as a live fetch. **`scripts/regen.sh`** wraps restore + `update --check` into a one-shot, offline determinism check — also run in CI, so every commit's committed IR (each `index.json` + `WAProto.proto`) is proven reproducible from its pinned inputs.
 
 > Bootstrap: the lock and the first archive are created by the first run of the update workflow (or a manual `update --save-bundles <dir>` followed by `scripts/publish-bundles.sh <dir>`). Until then the CI reproducibility gate stays dormant.

crates/wa-codegen/src/emit.rs

@@ -38,20 +38,21 @@ pub(crate) fn emit_field_parse(f: &ParsedField, node_var: &str, indent: &str) ->
     let fmsg = rust_lit_inner(wire);
     let method = f.method.as_str();
 
-    if method == wap::CONTENT_BYTES {
-        return vec![format!(
-            "{indent}let {name} = {node_var}.content_bytes().map(|b| b.to_vec()).unwrap_or_default();"
-        )];
-    }
-    if method == wap::CONTENT_STRING {
-        return vec![format!(
-            "{indent}let {name} = {node_var}.content_str().unwrap_or_default().to_string();"
-        )];
-    }
-    if method == wap::CONTENT_INT {
-        return vec![format!(
-            "{indent}let {name} = {node_var}.content_str().and_then(|s| s.parse().ok()).unwrap_or_default();"
-        )];
+    // Every remaining content spelling reads the node body and is typed by the canonical
+    // classifier, so a newly-recognized one (`contentUint`, `contentEnum`,
+    // `contentBytesRange`, `contentLiteralBytes`) is parsed rather than silently skipped.
+    if wap::is_content_method(method) {
+        return match wap::method_field_type(method) {
+            ParsedFieldType::Bytes => vec![format!(
+                "{indent}let {name} = {node_var}.content_bytes().map(|b| b.to_vec()).unwrap_or_default();"
+            )],
+            ParsedFieldType::Integer => vec![format!(
+                "{indent}let {name} = {node_var}.content_str().and_then(|s| s.parse().ok()).unwrap_or_default();"
+            )],
+            _ => vec![format!(
+                "{indent}let {name} = {node_var}.content_str().unwrap_or_default().to_string();"
+            )],
+        };
     }
     if !wap::is_attr_method(method) {
         return Vec::new();
@@ -269,16 +270,29 @@ fn emit_struct_reads(
                 indent,
             );
             let lit = rust_lit(tag);
-            let bytes = ct == wa_ir::ContentType::Bytes;
+            // Two spellings per kind: one that yields an `Option` for the `maybeChild`
+            // chain, one that unwraps for the required branch. Kept as literal text per
+            // kind so adding the integer reading leaves the existing bytes/string output
+            // byte-for-byte unchanged.
+            let (opt_read, req_read) = match ct {
+                wa_ir::ContentType::Bytes => (
+                    "content_bytes().map(|b| b.to_vec())",
+                    "content_bytes().map(|b| b.to_vec()).unwrap_or_default()",
+                ),
+                wa_ir::ContentType::Integer => (
+                    "content_str().and_then(|s| s.parse().ok())",
+                    "content_str().and_then(|s| s.parse().ok()).unwrap_or_default()",
+                ),
+                _ => (
+                    "content_str().map(|s| s.to_string())",
+                    "content_str().unwrap_or_default().to_string()",
+                ),
+            };
             if f.method == "maybeChild" {
                 lines.push(format!(
                     "{indent}let {id} = {base}.get_optional_child({lit})"
                 ));
-                lines.push(if bytes {
-                    format!("{indent}    .and_then(|n| n.content_bytes().map(|b| b.to_vec()));")
-                } else {
-                    format!("{indent}    .and_then(|n| n.content_str().map(|s| s.to_string()));")
-                });
+                lines.push(format!("{indent}    .and_then(|n| n.{opt_read});"));
             } else {
                 lines.push(format!(
                     "{indent}let {id}_node = {base}.get_optional_child({lit})"
@@ -287,11 +301,7 @@ fn emit_struct_reads(
                     "{indent}    .ok_or_else(|| anyhow::anyhow!(\"missing <{}>\"))?;",
                     rust_lit_inner(tag)
                 ));
-                lines.push(if bytes {
-                    format!("{indent}let {id} = {id}_node.content_bytes().map(|b| b.to_vec()).unwrap_or_default();")
-                } else {
-                    format!("{indent}let {id} = {id}_node.content_str().unwrap_or_default().to_string();")
-                });
+                lines.push(format!("{indent}let {id} = {id}_node.{req_read};"));
             }
             inits.push(format!("{indent}    {id},"));
             continue;

crates/wa-codegen/src/fields.rs

@@ -123,14 +123,17 @@ pub(crate) fn is_child_field(f: &ParsedField) -> bool {
 }
 
 /// A field the codegen materializes as a top-level attr-style field: an attribute
-/// accessor (via [`wap::is_attr_method`]), a `contentBytes` leaf, or a `hasAttr`
-/// presence marker (the latter filtered out by callers before emission).
+/// accessor (via [`wap::is_attr_method`]), **any** content leaf (via
+/// [`wap::is_content_method`]), or a `hasAttr` presence marker (the latter filtered out
+/// by callers before emission).
+///
+/// The content half is derived, not enumerated. Naming three spellings here was the
+/// fourth site of that mistake in this change, and the most misleading: the scanners
+/// recovered live `contentUint` fields, the IR carried them, and the reference codegen
+/// dropped them on the floor — so the generated Rust came out byte-identical and the
+/// unchanged output read as "nothing broke" rather than "the new data is being ignored".
 pub(crate) fn is_attr_field(f: &ParsedField) -> bool {
-    wap::is_attr_method(&f.method)
-        || f.method == wap::CONTENT_BYTES
-        || f.method == wap::CONTENT_STRING
-        || f.method == wap::CONTENT_INT
-        || f.method == wap::HAS_ATTR
+    wap::is_attr_method(&f.method) || wap::is_content_method(&f.method) || f.method == wap::HAS_ATTR
 }
 
 /// If `f` is a `child`/`maybeChild` whose body is just a content accessor
@@ -145,15 +148,24 @@ pub(crate) fn child_content_type(f: &ParsedField) -> Option<ContentType> {
     if kids.is_empty() || !kids.iter().all(is_content_method) {
         return None;
     }
+    // Bytes wins a mixed group (the widest reading); otherwise the kids' own decoded
+    // type decides. Collapsing everything that was not bytes onto `String` is what made
+    // `<registration>`, `<type>` and `<id>` — three `contentUint` siblings — share a
+    // single generated `content` field, two of the three silently dropped.
     Some(if kids.iter().any(|c| c.method == wap::CONTENT_BYTES) {
         ContentType::Bytes
+    } else if kids
+        .iter()
+        .all(|c| wap::method_field_type(&c.method) == wa_ir::ParsedFieldType::Integer)
+    {
+        ContentType::Integer
     } else {
         ContentType::String
     })
 }
 
 fn is_content_method(f: &ParsedField) -> bool {
-    f.method == wap::CONTENT_STRING || f.method == wap::CONTENT_BYTES
+    wap::is_content_method(&f.method)
 }
 
 fn children_of(f: &ParsedField) -> &[ParsedField] {
@@ -253,10 +265,10 @@ pub(crate) fn collect_response_fields(
             // `child("x").contentString()` → a single `x: String` field (named by
             // the child tag), not a stray `content` attr that collapses siblings.
             if let Some(ct) = f.content_type.or_else(|| child_content_type(f)) {
-                let base = if ct == ContentType::Bytes {
-                    "Vec<u8>"
-                } else {
-                    "String"
+                let base = match ct {
+                    ContentType::Bytes => "Vec<u8>",
+                    ContentType::Integer => "u64",
+                    _ => "String",
                 };
                 let wrapped = if f.method == "maybeChild" {
                     format!("Option<{base}>")

crates/wa-codegen/src/lib.rs

@@ -318,6 +318,61 @@ mod tests {
         assert!(c.contains("let body = body_node.content_str().unwrap_or_default().to_string();"));
     }
 
+    #[test]
+    fn integer_content_children_get_their_own_named_fields() {
+        // Two things were wrong at once, and the second hid the first: the codegen's
+        // content predicate admitted only three spellings, so live `contentUint` fields
+        // were dropped and the generated Rust came out unchanged — which read as "nothing
+        // broke". Once they arrived, every one of them was named `content` and collapsed
+        // into a single field, so two of three values were still lost.
+        let uint_child = |tag: &str| {
+            let mut c = parsed("child", tag, ParsedFieldType::String);
+            c.tag = Some(tag.into());
+            c.children = Some(vec![parsed(
+                "contentUint",
+                "content",
+                ParsedFieldType::Integer,
+            )]);
+            c
+        };
+        let ir = IqIr {
+            wa_version: "0.0.0".into(),
+            stanzas: vec![IqStanzaDef {
+                module_name: "WAWebDigest".into(),
+                namespace: "encrypt".into(),
+                iq_type: IqType::Get,
+                target: IqTarget::Server,
+                parser_name: "p".into(),
+                exported_function: Some("digest".into()),
+                all_exports: vec!["digest".into()],
+                request: IqRequestDef {
+                    namespace: "encrypt".into(),
+                    iq_type: IqType::Get,
+                    target: IqTarget::Server,
+                    children: vec![],
+                },
+                response: ParsedResponse {
+                    parser_name: "p".into(),
+                    assertions: vec![],
+                    fields: vec![uint_child("registration"), uint_child("id")],
+                    ..Default::default()
+                },
+            }],
+            unparseable: vec![],
+        };
+        let c = &generate_iq(&ir);
+        assert!(c.contains("pub registration: u64,"), "registration field");
+        assert!(c.contains("pub id: u64,"), "id field");
+        assert!(
+            !c.contains("pub content: u64,"),
+            "must not collapse both onto `content`"
+        );
+        assert!(
+            c.contains("content_str().and_then(|s| s.parse().ok())"),
+            "and the value is actually parsed"
+        );
+    }
+
     #[test]
     fn generates_spec_with_request_and_response() {
         let ir = IqIr {

crates/wa-codegen/src/notif_export.rs

@@ -295,6 +295,7 @@ mod tests {
                             kind: AssertionKind::Tag,
                             name: Some("notification".into()),
                             value: None,
+                            reference_path: None,
                         }],
                         fields: vec![ParsedField {
                             method: "attrString".into(),
@@ -305,6 +306,7 @@ mod tests {
                         }],
                         ..Default::default()
                     }),
+                    actions: Vec::new(),
                 },
                 NotificationDef {
                     notif_type: "encrypt".into(),
@@ -319,6 +321,7 @@ mod tests {
                         }],
                     }],
                     content: None,
+                    actions: Vec::new(),
                 },
             ],
         }