mjain/make program cache infallible - #32
Closed
mjain-jump wants to merge 576 commits into
Closed
Conversation
…z#13193) Improves naming of fields in VotePayload
* perf: skip writing untouched accounts * Simplify by passing through touched_flags * Replace Box<[Cell<bool>]> with Box<[bool]> once owned in LoadedTransaction * Use safe, but in-place into_iter collect * Move Box of Cell conversion to fn from() function * Include fee payer in execute_timings.details.touched_account_count
* Remove bench-tps bench-tps has been eclipsed by solana-transaction-bench tool * Remove bench-tps from multinode scripts and documentation
* Give write permissions to release jobs Release jobs use default GITHUB_TOKEN for several jobs: to create version bump PRs and to create releases. These operations require the token to have `content: write` * Also give write access for pull requests The workflow creates version bump PRs. Co-authored-by: Yihau Chen <a122092487@gmail.com> --------- Co-authored-by: Yihau Chen <a122092487@gmail.com>
xdp: add 802.1Q VLAN encapsulation for the tx loop
AF_XDP transmits raw frames directly to the bound NIC and bypasses the
kernel VLAN driver, so traffic the kernel would normally tag (via a
`vlanN` sub-interface) leaves untagged when XDP is enabled. This adds
802.1Q encapsulation in userspace, modeled on the existing GRE path: when
a route resolves through a VLAN sub-interface the tx loop prepends the
4-byte tag. The wire layout matches what the kernel VLAN driver would
have produced: [dst|src MAC] [TPID 0x8100] [TCI {pcp, dei=0, vid}]
[EtherType] [IPv4] [UDP] [payload].
VLAN sub-interfaces are discovered from netlink link attributes
(IFLA_LINKINFO kind "vlan", IFLA_VLAN_ID), the same way GRE tunnel info
is parsed, so no configuration is needed and the route monitor picks up
link changes automatically. Only 802.1Q is handled; 802.1ad (QinQ)
sub-interfaces are skipped. PCP is emitted as 0, matching the kernel's
default egress QoS mapping.
Since the XDP sender is shared, this tags both retransmit and broadcast
traffic that egresses through a VLAN.
* refactor(ci): add channel-info auto-resolver to xtask Adds `cargo xtask channel-info` that derives EDGE/BETA/STABLE channels from each `vX.Y` branch's workspace.package.version (stage gating on the top branch). Pin-file path (`ci/channel-info.sh`) stays authoritative; the new resolver runs alongside it via a soft-failing `channel-info-divergence` buildkite step that diffs both outputs and annotates on mismatch. Per anza-xyz#12450 sequencing step 1: observe parity over a window before cutting over and removing the pin override. * refactor(ci): parse workspace version via semver serde * refactor(ci): replace Xy tuple with BranchVersion struct * refactor(ci): drop EnvInputs, pass branch/channel directly * refactor(ci): use git ls-remote + reqwest for channel-info * style(ci): cargo fmt
Long ago, several dummy values were inserted into the transaction status and address signature columns. Those values are no longer in use and code was added to remove those dummy entries if they were detected. It has been many release and multiple years since then so we can reasonably delete the cleanup step
…xyz#13175) Updates perf/src/packet.rs to use wincode instead of bincode
The column was originally populated with bincode values but converted to protobuf values with 359707c in Oct 2020. Values have been written only as protobuf since. Fallback to bincode was left to support the transition as well as to support reading old Blockstores. The transition period has long passed, and there are numerous other breakages that prevent one from reading an old Blockstore with recent Agave. Thus, there is no benefit in supporting the fallback to bincode logic in the Rewards so column so rip it out
* Reapply "Collapse unflushed roots (anza-xyz#13174)" (anza-xyz#13189) * Remove unflushed root debug assert - Assert is invalid because any new root adds all of its parents to unflushed roots - Sometimes that will include slots thaht have already been flushed, re-adding them to to the write cache - They will get cleared out benignly on the next cycle of flush_slots
This code was present for when bincode columns were switched to protobuf. The only remaining caller is test code so remove it
* rm -rf docs/** * rm .github/workflows/docs.yml * update docs/README.md * update .mergify.yml * update .github/dependabot.yml
* chore(deps): bump prost from 0.14.3 to 0.14.4 in /dev-bins Bumps [prost](https://github.com/tokio-rs/prost) from 0.14.3 to 0.14.4. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Changelog](https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md) - [Commits](tokio-rs/prost@v0.14.3...v0.14.4) --- updated-dependencies: - dependency-name: prost dependency-version: 0.14.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update all workspaces --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump serde_with from 3.20.0 to 3.21.0 Bumps [serde_with](https://github.com/jonasbb/serde_with) from 3.20.0 to 3.21.0. - [Release notes](https://github.com/jonasbb/serde_with/releases) - [Commits](jonasbb/serde_with@v3.20.0...v3.21.0) --- updated-dependencies: - dependency-name: serde_with dependency-version: 3.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update all workspaces --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump bitflags from 2.12.1 to 2.13.0 in /programs/sbf Bumps [bitflags](https://github.com/bitflags/bitflags) from 2.12.1 to 2.13.0. - [Release notes](https://github.com/bitflags/bitflags/releases) - [Changelog](https://github.com/bitflags/bitflags/blob/main/CHANGELOG.md) - [Commits](bitflags/bitflags@2.12.1...2.13.0) --- updated-dependencies: - dependency-name: bitflags dependency-version: 2.13.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update all workspaces --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…/ci/xtask (anza-xyz#13205) chore(deps): bump toml in /ci/xtask Bumps [toml](https://github.com/toml-rs/toml) from 0.9.12+spec-1.1.0 to 1.0.6+spec-1.1.0. - [Commits](toml-rs/toml@toml-v0.9.12...toml-v1.0.6) --- updated-dependencies: - dependency-name: toml dependency-version: 1.0.6+spec-1.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…urn (anza-xyz#13504) feat: migrate to spl-token-2022-interface 3.x and parse PermissionedBurn Bumps the workspace from spl-token-2022-interface 2.x to 3.1.0 (plus spl-token-metadata-interface 1.0 and proof-extraction 0.6) and adds parsing for the PermissionedBurn mint extension that landed in 3.0. account-decoder couldn't parse PermissionedBurn (ExtensionType 28) on the old interface, and since one unrecognized extension makes get_extension_types fail and drops the whole extensions array, mints carrying it returned no parsed extensions at all over jsonParsed. account-decoder now emits permissionedBurnConfig with its authority. The 2.x to 3.x bump moves the optional-pubkey fields to MaybeNull and relocates the ZK pod/proof types, so this also includes the mechanical fixups across account-decoder, rpc, and transaction-status: OptionalNonZeroPubkey to MaybeNull via into(), and the PodElGamalPubkey/proof-data paths moving to solana-zk-sdk-pod and solana-zk-elgamal-proof-interface. transaction-status gains the new 3.x bits: the PermissionedBurn authority and extension type variants, and parsing for the new UnwrapLamports instruction plus the permissioned burn Initialize/Burn/BurnChecked instructions. ConfidentialBurn and the Batch instruction carry proof/nested-instruction complexity and are stubbed as not-parsable for focused follow-up PRs.
…#13512) feat: parse the token Batch instruction Replaces the not-parsable stub for the Batch instruction with a parser that walks the inner instructions (each prefixed with a u8 account count and u8 data length) and recurses through parse_token, emitting them under an "instructions" array. Since parse_token serves both spl-token and spl-token-2022, this covers Batch on both programs. Nested batches are rejected, matching the program.
…n instruction (anza-xyz#13511) feat: parse the permissioned burn ConfidentialBurn instruction Replaces the not-parsable stub for PermissionedBurnInstruction::ConfidentialBurn with full parsing. It mirrors the confidential mint-burn Burn parser since they share the BurnInstructionData layout, with the permissioned burn authority sitting between the optional proof accounts and the owner/delegate.
* chore(deps): bump arrayvec from 0.7.6 to 0.7.7 in /dev-bins Bumps [arrayvec](https://github.com/bluss/arrayvec) from 0.7.6 to 0.7.7. - [Release notes](https://github.com/bluss/arrayvec/releases) - [Changelog](https://github.com/bluss/arrayvec/blob/master/CHANGELOG.md) - [Commits](bluss/arrayvec@0.7.6...0.7.7) --- updated-dependencies: - dependency-name: arrayvec dependency-version: 0.7.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update all workspaces --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump log from 0.4.32 to 0.4.33 in /ci/xtask Bumps [log](https://github.com/rust-lang/log) from 0.4.32 to 0.4.33. - [Release notes](https://github.com/rust-lang/log/releases) - [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md) - [Commits](rust-lang/log@0.4.32...0.4.33) --- updated-dependencies: - dependency-name: log dependency-version: 0.4.33 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update all workspaces --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update mergify config for v4.2 backport label v4.2 is an alpha candidate; v4.0 and v4.1 stay active. Add v4.2 backport rules plus its alpha-branch warning, and drop the inactive v3.1 rules. Move the shared backport assignee into a `shared` anchor so future oldest-branch removals don't have to relocate it. * mergify: use single backport warning for v4.2
…abi (anza-xyz#13555) * runtime: make snapshot serialization use newtype structs with frozen-abi Replace the inline tuples used to serialize the accounts db fields and the overall bank snapshot with named newtype structs (SerializableAccountsDb and SerializableBankSnapshot) that derive Serialize and carry StableAbi/abi_digest frozen-abi tests. Drop the now-redundant serialize_account_storage_ms datapoint (covered by bank_serialize_us) and report num_storages in snapshot_bank instead. * runtime: remove redundant BankAbiTestWrapper frozen-abi test The snapshot wire ABI is now frozen by the StableAbi abi_digest tests on SerializableBankSnapshot and its component structs (SerializableVersionedBank, SerializableAccountsDb, ExtraFieldsToSerialize), which hash serialized bytes over many random samples. BankAbiTestWrapper froze the same structure via the legacy AbiExample/api_digest path through a real Bank; it added no ABI coverage the type-level digests don't already provide, and was the last snapshot user of that legacy mechanism.
* Remove roots tracker * Moved append vec stats to account_db-stores
PR feedback. Add wincode and bincode as abi serializers for RepairProtocol adn BlockIdRepairResponse. Add frozen_abi to BlockIdrepairResponse. Drop new equivalence units tests in favor of frozen abi round-trip equivalence and wire tests. Add stableAbi to AncestorHashesResponse. Drop serde from ShredRepairType. Gate serde and partialEq derives behind frozen-abi. Fix frozen-abi sampling. Replace bincode-bound Packet::deserialize_slice() and perf::packet::from_data() with new wincode equivalents Add api_digest and abi_digest to RepairResponse. Drop RepairRequestConfig type. Use wincode packet config for serialization in packet_from_data() Derive StableAbiSample for RepairResponse and AncestorHashesResponse
The only caller that raised EmptyEntryBatch variant was updated with BlockAborted in 870b198. So, remove the now unused variant
Introduces GenesisCert type
* chore(deps): bump memmap2 from 0.9.10 to 0.9.11 in /dev-bins Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.10 to 0.9.11. - [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md) - [Commits](RazrFalcon/memmap2-rs@v0.9.10...v0.9.11) --- updated-dependencies: - dependency-name: memmap2 dependency-version: 0.9.11 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update all workspaces --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…om 0.6.0 to 0.6.1 (anza-xyz#13585)
nit: look up CertificateType from Certificate
…a-xyz#13180) CompletedDataSetsService is only spawned when --full-rpc-api is set, so a plugin that enables deshred transaction notifications gets nothing unless the node also runs the full rpc api. Spawn it whenever the deshred transaction notifier is present as well. The service feeds two independent sinks: RPC signatureSubscribe notifications and the deshred notifier, and only the former needs RpcSubscriptions, so it now takes an Option and the creation no longer hinges on an RPC port. A geyser node started without --rpc-port still gets deshred notifications. The gate is on the notifier itself rather than on a plugin being loaded, so plugins that don't subscribe don't pay for the extra blockstore reads. Co-authored-by: sam0x17 <sam@durosoft.com>
…IteratorView (anza-xyz#13596) Replace the `serialize_iter_as_map`/`serialize_iter_as_seq` helpers with a single `SerializableExactIteratorView<I>` that wraps a cloneable `ExactSizeIterator` and serializes it as a length-prefixed sequence, re-creating the iterator via `Clone` on each serialization so nothing is materialized. `SerializableAccountsDb::new` streams the storages slice as a `slot -> [entry]` map through it, keeping the closure iterator as nested return-position `impl Trait` so the concrete return type stays nameable. serde maps and sequences share the same wire encoding, so the bincode output is unchanged (verified by the serde_snapshot roundtrip tests). Drop the now-unused `utils` module.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.mergify.yml(Revise RPC API update instructions in.mergify.ymlanza-xyz/agave#12609)filter_executable_program_accounts()(Refactor -filter_executable_program_accounts()anza-xyz/agave#12589)RewardCommission.commission_bpson SIMD-0232 (runtime: gateRewardCommission.commission_bpson SIMD-0232 anza-xyz/agave#12588)SwitchBankEventchannel with a single shared cell (refactor: replaceSwitchBankEventchannel with a single shared cell anza-xyz/agave#12593)TimeoutConfig(refactor: extract timeout constituent parameters intoTimeoutConfiganza-xyz/agave#12564)--do-not-require-vote-historyfor ag startup/set-identity (validator: add--do-not-require-vote-historyfor ag startup/set-identity anza-xyz/agave#12750)ProgramCache::extract()(Refactor - program owner inProgramCache::extract()anza-xyz/agave#12683)delta_first_slice <= delta_block(nit: add assert fordelta_first_slice <= delta_blockanza-xyz/agave#12765)delta_timeout(fix(votor): prevent overflow when scalingdelta_timeoutanza-xyz/agave#12772)muton test stake creator (accounts-db: Avoidmuton test stake creator anza-xyz/agave#12551)context.slot) (rpc-client: Add method to get latest blockhash with response context (notablycontext.slot) anza-xyz/agave#12826)account_ownercheck toProgramCacheEntry::eq()(Refactor - Addsaccount_ownercheck toProgramCacheEntry::eq()anza-xyz/agave#12804)CertificateandCertificateTypeinto their own module (movesCertificateandCertificateTypeinto their own module anza-xyz/agave#12840)Bank::prune_program_cache()(Refactor -Bank::prune_program_cache()anza-xyz/agave#12838)NonZeroto represent total stake (consensus_pool: UseNonZeroto represent total stake anza-xyz/agave#12946)Blocktype alias to aBlocktype for better readability and type safety (Converts theBlocktype alias to aBlocktype for better readability and type safety anza-xyz/agave#12929)BLSOpand removesCopyderivation onVoteMessage(Cleans upBLSOpand removesCopyderivation onVoteMessageanza-xyz/agave#12952)translate_signers_candtranslate_signers_rust. (Unifytranslate_signers_candtranslate_signers_rust. anza-xyz/agave#13095)DepositandWithdraw(transaction-status: fix parsed account labels for confidential transferDepositandWithdrawanza-xyz/agave#13149)GenericTraversal(Remove needless collect inGenericTraversalanza-xyz/agave#13182)SlotMetafor repairs to avoid bitvec alloc (Add trimmed version ofSlotMetafor repairs to avoid bitvec alloc anza-xyz/agave#13181)VotePayloadandCertPayload(Improves naming of fields inVotePayloadandCertPayloadanza-xyz/agave#13193)perf/src/packet.rsto use wincode instead of bincode (Updatesperf/src/packet.rsto use wincode instead of bincode anza-xyz/agave#13175)Indexinfind_missing_data_indexes(Leverage blockstoreIndexinfind_missing_data_indexesanza-xyz/agave#13185)prepare_program_cache_for_upcoming_feature_set()(Fix - Take loading lock inprepare_program_cache_for_upcoming_feature_set()anza-xyz/agave#13203)Certificatein genesis account (Do not storeCertificatein genesis account anza-xyz/agave#13346)NonZero<u64>(Stores validator and total stake asNonZero<u64>anza-xyz/agave#13355)Votein a single batch (bls-sigverifies votes of the sameVotein a single batch anza-xyz/agave#13344)prepare_one_program_for_upcoming_feature_set()(#13406)GenesisCerttype (#13574)CertificateTypefromCertificate(#13599)