-
Notifications
You must be signed in to change notification settings - Fork 24
Provide information about build progress #1001
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
6916c50
Build progress tracking initial implementation
ekharkunov b056ac4
Merge remote-tracking branch 'origin/dev' into feature/build-progress
ekharkunov c7fb15a
Merge remote-tracking branch 'origin/dev' into feature/build-progress
ekharkunov 432e482
Guard progress-reconnect-attempts property against malformed values
ekharkunov ceae152
Merge remote-tracking branch 'origin/dev' into feature/build-progress
ekharkunov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # Live build progress | ||
|
|
||
| The extender streams live build progress over Server-Sent Events (SSE) so clients can show | ||
| what a build is doing (downloading the SDK, resolving dependencies, compiling file N of M, | ||
| linking, packaging) instead of a silent wait. | ||
|
|
||
| Progress is **advisory**: `/job_status` polling remains the source of truth for build | ||
| completion, and terminal progress events are only emitted after the result files | ||
| (`build.zip`/`error.txt`) are in place. Everything is backward compatible — old clients | ||
| never call the endpoint, and new clients fall back to polling when the server has no | ||
| progress support. | ||
|
|
||
| ## Endpoint | ||
|
|
||
| ``` | ||
| GET /job_progress?jobId=<jobId> | ||
| Accept: text/event-stream | ||
| ``` | ||
|
|
||
| * Live job: streams `progress` events. A snapshot of the current state is sent immediately | ||
| on subscribe (early events always precede the first subscriber, because the build is | ||
| dispatched before the jobId is returned). | ||
| * Reconnects can pass the standard `Last-Event-ID` header; missed events are replayed from | ||
| a per-job ring buffer when possible, otherwise a state snapshot is sent. | ||
| * Job already finished (result files on disk): a single terminal event, then the stream closes. | ||
| * Unknown job, or `extender.progress.enabled: false`: `404`. | ||
| * Comment lines (`:ka`) are heartbeats sent every `heartbeat-interval` to keep idle | ||
| connections alive through proxies. | ||
|
|
||
| Event payload (JSON, `id:` field = `seq`): | ||
|
|
||
| ```json | ||
| { | ||
| "jobId": "job1234567890", | ||
| "seq": 17, | ||
| "ts": 1720512345678, | ||
| "stage": "COMPILING", | ||
| "detail": "extension1: compiling source files", | ||
| "percent": 55, | ||
| "extension": "extension1", | ||
| "currentFile": 12, | ||
| "totalFiles": 34, | ||
| "terminal": false | ||
| } | ||
| ``` | ||
|
|
||
| Stages, in pipeline order: `RECEIVED`, `QUEUED`, `SDK`, `DEPENDENCIES`, `MANIFESTS`, | ||
| `PLATFORM`, `COMPILING`, `LINKING`, `PACKAGING`, and the terminals `SUCCESS`/`ERROR`. | ||
| `REMOTE_BUILDING` is a coarse stage used by a frontend when its remote builder runs an | ||
| older server without progress support. `percent` is a 0-100 estimate and never decreases. | ||
| `extension`/`currentFile`/`totalFiles` are only present while compiling. | ||
|
|
||
| ## Frontend / remote builder setups | ||
|
|
||
| A frontend instance relays the remote builder's progress stream to its own subscribers | ||
| under the frontend's jobId, so external clients only ever talk to the frontend. If the | ||
| remote builder runs an older server version the frontend degrades to coarse | ||
| `REMOTE_BUILDING` ticks. | ||
|
|
||
| If the server sits behind a buffering reverse proxy, response buffering must be disabled | ||
| for `/job_progress` (e.g. nginx `proxy_buffering off` or the `X-Accel-Buffering: no` | ||
| response header), otherwise events arrive in bursts or not at all. The built-in heartbeats | ||
| keep idle timeouts (Jetty and load balancers) from closing quiet streams. | ||
|
|
||
| ## Server configuration (`application.yml`) | ||
|
|
||
| ```yaml | ||
| extender: | ||
| progress: | ||
| enabled: true # false restores the old behavior exactly | ||
| sse-timeout: 1800000 # SseEmitter timeout, ms | ||
| heartbeat-interval: 15000 # keepalive comment cadence, ms | ||
| event-buffer-size: 256 # per-job replay buffer for Last-Event-ID | ||
| max-subscribers-per-job: 8 | ||
| registry-ttl: 1200000 # sweep for jobs that died without a terminal event | ||
| cleanup-period: 20000 | ||
| ``` | ||
|
|
||
| ## Client API | ||
|
|
||
| `ExtenderClient` gained an overload that reports progress while the existing polling flow | ||
| runs unchanged: | ||
|
|
||
| ```java | ||
| extenderClient.build(platform, sdkVersion, sourceResources, destination, log, | ||
| (stage, detail, percent, currentFile, totalFiles) -> { | ||
| // called on a background thread; currentFile/totalFiles are -1 outside COMPILING | ||
| System.out.printf("[%3d%%] %s %s%n", percent, stage, detail); | ||
| }); | ||
| ``` | ||
|
|
||
| System properties: | ||
|
|
||
| * `com.defold.extender.client.progress-enabled` (default `true`) — set `false` to never | ||
| open the progress stream. | ||
| * `com.defold.extender.client.progress-reconnect-attempts` (default `5`) — reconnect | ||
| attempts for a dropped stream. | ||
|
|
||
| ## Trying it with curl | ||
|
|
||
| ```sh | ||
| JOB=$(curl -s -X POST -F "file=@upload.zip" http://localhost:9000/build_async/x86_64-linux/<sdk-sha1>) | ||
| curl -sN "http://localhost:9000/job_progress?jobId=$JOB" | ||
| # reconnect mid-build and replay everything after event 10: | ||
| curl -sN -H "Last-Event-ID: 10" "http://localhost:9000/job_progress?jobId=$JOB" | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
167 changes: 167 additions & 0 deletions
167
client/src/main/java/com/defold/extender/client/ExtenderProgressConsumer.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| package com.defold.extender.client; | ||
|
|
||
| import org.apache.http.HttpResponse; | ||
| import org.apache.http.HttpStatus; | ||
| import org.apache.http.client.HttpClient; | ||
| import org.apache.http.client.config.RequestConfig; | ||
| import org.apache.http.client.methods.HttpGet; | ||
| import org.apache.http.util.EntityUtils; | ||
| import org.json.simple.JSONObject; | ||
| import org.json.simple.parser.JSONParser; | ||
|
|
||
| import java.io.BufferedReader; | ||
| import java.io.IOException; | ||
| import java.io.InputStreamReader; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.logging.Level; | ||
| import java.util.logging.Logger; | ||
|
|
||
| /** | ||
| * Consumes the server's /job_progress SSE stream on a background thread | ||
| * and forwards events to an ExtenderProgressListener. | ||
| * | ||
| * Strictly advisory: any failure (404 from an old server, dropped | ||
| * connection, malformed data) is swallowed after bounded reconnect | ||
| * attempts. The poll loop in ExtenderClient.build_async remains the sole | ||
| * authority on build completion and calls stop() when the build is done. | ||
| */ | ||
| class ExtenderProgressConsumer implements Runnable { | ||
| private static final Logger logger = Logger.getLogger(ExtenderProgressConsumer.class.getName()); | ||
|
|
||
| /** Creates GET requests carrying the client's auth and custom headers. */ | ||
| interface GetRequestFactory { | ||
| HttpGet create(String url) throws IOException; | ||
| } | ||
|
|
||
| private static final long RECONNECT_BACKOFF_MS = 2000; | ||
|
|
||
| private final HttpClient httpClient; | ||
| private final String jobProgressUrl; | ||
| private final GetRequestFactory requestFactory; | ||
| private final ExtenderProgressListener listener; | ||
| private final int maxReconnectAttempts; | ||
|
|
||
| private volatile boolean stopped = false; | ||
| private volatile HttpGet currentRequest = null; | ||
| private long lastEventId = -1; | ||
|
|
||
| ExtenderProgressConsumer(HttpClient httpClient, String extenderBaseUrl, String jobId, | ||
| GetRequestFactory requestFactory, ExtenderProgressListener listener) { | ||
| this.httpClient = httpClient; | ||
| this.jobProgressUrl = String.format("%s/job_progress?jobId=%s", extenderBaseUrl, jobId); | ||
| this.requestFactory = requestFactory; | ||
| this.listener = listener; | ||
| this.maxReconnectAttempts = Integer.parseInt( | ||
| System.getProperty("com.defold.extender.client.progress-reconnect-attempts", "5")); | ||
| } | ||
|
|
||
| /** Stops the consumer and unblocks the stream read. Safe to call more than once. */ | ||
| void stop() { | ||
| stopped = true; | ||
| HttpGet request = currentRequest; | ||
| if (request != null) { | ||
| request.abort(); | ||
| } | ||
| } | ||
|
|
||
| @Override | ||
| public void run() { | ||
| try { | ||
| int attempts = 0; | ||
| while (!stopped && attempts < maxReconnectAttempts) { | ||
| attempts++; | ||
| try { | ||
| if (stream()) { | ||
| return; // unsupported by server or terminal event seen | ||
| } | ||
| } catch (IOException e) { | ||
| if (stopped) { | ||
| return; | ||
| } | ||
| logger.log(Level.FINE, "Build progress stream dropped, reconnecting: " + e.getMessage()); | ||
| } | ||
| Thread.sleep(RECONNECT_BACKOFF_MS); | ||
| } | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| } catch (Exception e) { | ||
| // progress must never break the build | ||
| logger.log(Level.FINE, "Build progress consumer stopped: " + e.getMessage()); | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Opens the SSE stream and forwards events until it ends. | ||
| * Returns true when the consumer is done for good (server has no | ||
| * progress support, or a terminal event arrived); false to reconnect. | ||
| */ | ||
| private boolean stream() throws IOException { | ||
| HttpGet request = requestFactory.create(jobProgressUrl); | ||
| request.setHeader("Accept", "text/event-stream"); | ||
| if (lastEventId >= 0) { | ||
| request.setHeader("Last-Event-ID", Long.toString(lastEventId)); | ||
| } | ||
| // the stream stays open for the whole build: disable the socket | ||
| // read timeout for this request; stop() aborts it when the build is done | ||
| request.setConfig(RequestConfig.custom().setSocketTimeout(0).build()); | ||
| currentRequest = request; | ||
| try { | ||
| HttpResponse response = httpClient.execute(request); | ||
| if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) { | ||
| // old server or progress disabled; polling still reports completion | ||
| EntityUtils.consumeQuietly(response.getEntity()); | ||
| return true; | ||
| } | ||
| try (BufferedReader reader = new BufferedReader( | ||
| new InputStreamReader(response.getEntity().getContent(), StandardCharsets.UTF_8))) { | ||
| String eventId = null; | ||
| StringBuilder data = new StringBuilder(); | ||
| String line; | ||
| while (!stopped && (line = reader.readLine()) != null) { | ||
| if (line.isEmpty()) { | ||
| // end of one SSE event | ||
| if (data.length() > 0 && dispatch(eventId, data.toString())) { | ||
| return true; // terminal event | ||
| } | ||
| eventId = null; | ||
| data.setLength(0); | ||
| } else if (line.startsWith("id:")) { | ||
| eventId = line.substring(3).trim(); | ||
| } else if (line.startsWith("data:")) { | ||
| data.append(line.substring(5).trim()); | ||
| } | ||
| // "event:" names and ":" comments (heartbeats) are ignored | ||
| } | ||
| } | ||
| return stopped; | ||
| } finally { | ||
| currentRequest = null; | ||
| } | ||
| } | ||
|
|
||
| /** Forwards one event to the listener. Returns true for terminal events. */ | ||
| private boolean dispatch(String eventId, String data) { | ||
| boolean terminal = false; | ||
| try { | ||
| JSONObject json = (JSONObject) new JSONParser().parse(data); | ||
| if (eventId != null) { | ||
| lastEventId = Long.parseLong(eventId); | ||
| } | ||
| String stage = (String) json.get("stage"); | ||
| String detail = (String) json.get("detail"); | ||
| Number percent = (Number) json.get("percent"); | ||
| Number currentFile = (Number) json.get("currentFile"); | ||
| Number totalFiles = (Number) json.get("totalFiles"); | ||
| Boolean isTerminal = (Boolean) json.get("terminal"); | ||
| terminal = isTerminal != null && isTerminal; | ||
| listener.onProgress(stage, detail, | ||
| percent != null ? percent.intValue() : 0, | ||
| currentFile != null ? currentFile.intValue() : -1, | ||
| totalFiles != null ? totalFiles.intValue() : -1); | ||
| } catch (Exception e) { | ||
| // a malformed event or a listener bug must not kill the stream | ||
| logger.log(Level.FINE, "Ignoring bad progress event: " + e.getMessage()); | ||
| } | ||
| return terminal; | ||
| } | ||
| } | ||
24 changes: 24 additions & 0 deletions
24
client/src/main/java/com/defold/extender/client/ExtenderProgressListener.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| package com.defold.extender.client; | ||
|
|
||
| /** | ||
| * Receives live build-progress updates while ExtenderClient.build(...) is | ||
| * waiting for the server to finish a build. | ||
| * | ||
| * Progress is advisory: it may stop arriving at any time (old server, | ||
| * dropped connection) while the build itself keeps running. Completion is | ||
| * always determined by the build call returning or throwing. | ||
| * | ||
| * Callbacks are invoked on a background thread, never on the thread that | ||
| * called build(...). | ||
| */ | ||
| public interface ExtenderProgressListener { | ||
| /** | ||
| * @param stage Current build stage, e.g. "SDK", "DEPENDENCIES", | ||
| * "COMPILING", "LINKING", "PACKAGING", "SUCCESS", "ERROR" | ||
| * @param detail Human-readable detail line, e.g. the extension being compiled. May be null. | ||
| * @param percent Overall progress estimate 0-100, never decreasing. | ||
| * @param currentFile Files compiled so far for the current extension, or -1 when not compiling. | ||
| * @param totalFiles Total files to compile for the current extension, or -1 when not compiling. | ||
| */ | ||
| void onProgress(String stage, String detail, int percent, int currentFile, int totalFiles); | ||
|
ekharkunov marked this conversation as resolved.
Dismissed
|
||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.