Implement the execution of the streams, messages and operations in persistent instances over the block. by MathieuDutSik · Pull Request #6032 · linera-io/linera-protocol

MathieuDutSik · 2026-04-15T19:05:50Z

Motivation

When executing a message, an operation, or a stream then we are instantiating, processing, and then finalizing.
So, the finalization is done many times, which is inefficient.

Proposal

By having persistent instances for contracts, we can address many issues with the execution model: The contract are running, and when we finish the block, we save. That covers many scenario:

Contract A executes a message, Contract B executes another message which lead to executing A.
Multiple operations are grouped.

However, that framework fails on the problem of checkpointing:

We need to be able to checkpoint before executing an IncomingBundle. That execution could fail.
Checkpointing the ChainStateView is not enough. One needs to checkpoint the state of the Wasm machine.
So we use snapshots of the Wasm running instance.

Snapshots:

We implement them for wasmer and wasmtime.
They are small. Something like 1.12 M for matching engine.
They are better than software solutions like introducing a save in the linera-sdk.
They are super fact and do not cost any fuel. Doing a load / save cost 25k for a counter contract.
They help enforce a strict alignment of the execution for the staging loop.

The execution on the web does not work because of the lack of shared memory (the same problem that forces the preloading of all relevant applications before running). So, we use a different scheme for the web:

We implement the serialization for the Wasm snapshots.
We use those snapshots when we execute a transaction.
After a transaction, we wind down the Wasm execution instance.

With the changes:

For a given application, load is called once inside a block, and store is called once. Slight break of the SDK.
This is breaking the testnet_conway.

The flash loan application:

Contract A calls the bank in order to get tokens.
Contract B repays the bank in tokens.
If the terminate function of the bank fails to terminate correctly then it means that the loans was not repaid correctly and the block cannot be created.

Test Plan

CI should cover all scenarios.

A benchmark function for the matching_engine is introduced: 50 bids, 50 asks, all sent to the same application. The processing of the inbox is benchmarked. The result is 760ms for the old code and 413ms for the new code.

If accepted, then the documentation has to be updated since the counter application is changing.

Release Plan

It cannot be backported to testnet_conway for the following reasons:

The API of the contract is changed.
The cost structure is changed.

Links

None.

deuszx · 2026-04-16T12:22:36Z

This looks promising but it'd be good to have some metric that we can use to confirm/gauge how much of the improvement we're getting.

MathieuDutSik · 2026-04-22T08:57:57Z

This looks promising but it'd be good to have some metric that we can use to confirm/gauge how much of the improvement we're getting.

During a simulation for the matching engine with one user creating 50 messages, another user 50 messages, and then measuring the process_inbox we are achieving a 2x speedup. More precisely 760.286667ms vs 380.791458ms.

However, that implementation over the block does not work for the checkpointing. This is because if we use persistent instances, then when accessing an older state, we have to access the older state. But we would lose the modification to the view.

I leave it here as a draft since it is possible to run without auto_retry with BundleFailurePolicy::Abort. So, it could be interesting in some context due to the massive potential benefit.

afck · 2026-05-04T09:01:07Z

+                            .expect("Failed to restore Wasm global from snapshot");
+                    }
+                }
+                _ => {}


Should this be an error? Is it expected that we'd encounter multiple Extern::Memory instances?

For Linera, no since we have just one (the heap). However, there are use-cases where we have several, and maybe it would be better not to restruct ourselves initially.

afck · 2026-05-04T09:02:43Z

+        for (name, ext) in exports {
+            match ext {
+                Extern::Memory(mem) if !memory_restored => {
+                    mem.data_mut(&mut self.store)[..snapshot.memory_data.len()]


Do we need to check here if the memory data is large enough?

Yes, added a correction.

afck · 2026-05-04T09:06:22Z

+                    .copy_to_vec()
+                    .expect("Failed to copy Wasm memory")
+            })
+            .unwrap_or_default();


Should it be an error if we don't have exactly one memory?

I am not sure.
I added error handling to the snapshot operations. However, a priori we could have several memory on Wasm, so this looks like an unnecessary check. There are plenty of other possible failure scenarios.

afck · 2026-05-04T09:29:25Z

+
+    /// Creates a snapshot of the Wasm instance's mutable state (memory and globals).
+    ///
+    /// Returns `None` for non-Wasm contract implementations.


I think this really shouldn't have a default implementation: It needs to work for all VMs.

(If we want to postpone e.g. the EVM implementation I'd rather add an error and a TODO comment in the EVM-specific code.)

I agree that we should not have a default implementation for the snapshots.

Well execution can fail for a block that contains REVM message. That is true as it is for Wasm.

But the difference is that REVM does not have snapshots. So, we save to storage even if we are not at the end of the block. So, the snapshots functionality becomes a noop.

afck · 2026-05-04T09:29:35Z

+    }
+
+    /// Restores the Wasm instance's mutable state from a snapshot.
+    fn restore_snapshot(&mut self, _snapshot: &(dyn std::any::Any + Send)) {}


Yes, done similarly.

afck · 2026-05-04T09:30:22Z

+    /// Creates a snapshot of the Wasm instance's mutable state (memory and globals).
+    ///
+    /// Returns `None` for non-Wasm contract implementations.
+    fn create_snapshot(&mut self) -> Option<Box<dyn std::any::Any + Send>> {


And why not make the snapshot an associated type instead of Any?

It is not easy.
The problem is that we have to store the snapshots of many possible execution engines: Wasmer, Wasmtime, REVM (that one is trivial).
We could have a type that encapsulates the Box<dyn Any+Send> but I am afraid there is no totally nice solutions.

afck · 2026-05-04T09:33:34Z

    /// Creates a new BlockExecutionTracker.
+    ///
+    /// The `runtime_channels` argument is `Some` when a block-level contract runtime thread
+    /// is available (native). On web, WASM modules cannot be shipped to the runtime worker


Suggested change

/// is available (native). On web, WASM modules cannot be shipped to the runtime worker

/// is available (native). On web, Wasm modules cannot be shipped to the runtime worker

Ok, corrected.
Actually, the code somehow assumes that the execution engine is Wasm, which is not accurate. But that is already visible with the metrics.

afck · 2026-05-04T09:40:40Z

+    /// The `runtime_channels` argument is `Some` when a block-level contract runtime thread
+    /// is available (native). On web, WASM modules cannot be shipped to the runtime worker
+    /// via shared memory, so the block-level runtime is not spawned and `runtime_channels`
+    /// is `None` — per-transaction fallback runtimes are used instead.


I don't understand; does that mean that in the web client, the staging loop will produce a different outcome than the final block execution?

That was the big problem. But I think I found a solution.

There are now indeed two execution paths:

One for native where we have shared memory.

One for the web.

This is forced because of the lack of shared memory in wasmer. The same problem that forces us to preload the contracts being used. There is a test that those two paths are coherent.

For the Native, we have persistent Wasm instance running over the whole execution of the block. For the web we create the Wasm instance during the scope of the execution. We work with snapshots in order to reinstate the Wasm instance when needed.

So, snapshots are used for both code execution paths, just in different ways.

And using the web approach for all would be less efficient, would it? (Because we'd needlessly snapshot-and-restore.)

Add the motion for the REVM.

…..)".

MathieuDutSik force-pushed the user_action_bundle_messages_c branch 2 times, most recently from 976da24 to 8858863 Compare April 22, 2026 07:35

MathieuDutSik force-pushed the user_action_bundle_messages_c branch from 8858863 to be8a4ac Compare April 22, 2026 09:14

MathieuDutSik changed the title ~~Implement the execution of the streams, messages and operations in persistent instances.~~ Implement the execution of the streams, messages and operations in persistent instances over the block. Apr 22, 2026

MathieuDutSik force-pushed the user_action_bundle_messages_c branch 4 times, most recently from 29c6138 to e7b3e5b Compare April 28, 2026 20:44

MathieuDutSik marked this pull request as ready for review May 2, 2026 10:57

MathieuDutSik requested review from afck and ma2bd May 2, 2026 10:58

MathieuDutSik mentioned this pull request May 2, 2026

Introduce persistent execution over an IncomingBundle #6093

Closed

MathieuDutSik force-pushed the user_action_bundle_messages_c branch 2 times, most recently from f1d9930 to 356f869 Compare May 4, 2026 05:50

afck reviewed May 4, 2026

View reviewed changes

MathieuDutSik force-pushed the user_action_bundle_messages_c branch 2 times, most recently from 3b49879 to a8b725d Compare May 5, 2026 05:34

MathieuDutSik added 11 commits May 6, 2026 11:59

Implement the new scheme for executing Wasm instances.

68f005e

Add the motion for the REVM.

Unify the execution platform between web and non-web.

f74fea6

Reformatting.

f9947e1

Remove the performance test.

54e34fd

Revert the changes with LoadContract / LoadService.

ecc0914

Some update so that wasmtime pass nicely.

83f639d

Rename the snapshot and related to something VM agnostic.

d475468

Address clippy issues.

11f9fc7

Add the serialization of the snapshots.

af3dcb5

Rewrite a little the type access, "Value" is ambiguous.

f822a42

Add some forgotten new files.

b0341f6

MathieuDutSik added 14 commits May 6, 2026 11:59

First step of the changes.

29f92fe

Some clippy corrections.

dc92ed8

Next step.

7885be7

Add the current_snapshots entry.

e7581b6

One step.

394134c

Some update.

d66a4b3

Next step.

58bbdb2

Introduce the next execution scheme.

6bcf4ae

Change the execution path.

8c93543

Add some execution scheme.

c3e5490

Some simplification of the comparison code.

6552057

Make the online test pass.

a1c92d6

Reformat.

f09e068

Some upgrade from clippy corrections.

5dd5eb8

MathieuDutSik force-pushed the user_action_bundle_messages_c branch from 28476cd to 5dd5eb8 Compare May 6, 2026 10:13

MathieuDutSik added 7 commits May 6, 2026 13:54

Implement memory expansion.

dc1cbcc

Improve the support for the snapshots.

bd90219

Make the snapshot operations return error instead of using "expect(..…

a34422f

…..)".

Switch to a BTreeMap for the snapshot entries.

e66c241

Remove the FlashLoanInitial.

b6304b9

Avoid the possible overflow.

8ba7bb3

Extend the scope of the test.

e38ec39

ma2bd mentioned this pull request May 18, 2026

Multi-leader jitter #6289

Merged

afck mentioned this pull request Jun 15, 2026

Use a single UserAction to execute an entire bundle of messages #5698

Open

	/// is available (native). On web, WASM modules cannot be shipped to the runtime worker
	/// is available (native). On web, Wasm modules cannot be shipped to the runtime worker

Uh oh!

Conversation

MathieuDutSik commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Proposal

Test Plan

Release Plan

Links

Uh oh!

deuszx commented Apr 16, 2026

Uh oh!

MathieuDutSik commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MathieuDutSik commented Apr 15, 2026 •

edited

Loading

MathieuDutSik commented Apr 22, 2026 •

edited

Loading