feat(store): implement pluggable cloud storage for HStore#3081
Open
vaijosh wants to merge 6 commits into
Open
Conversation
…age with S3 support out of the box
dosubot
Bot
added
size:XXL
vaijosh
added a commit
to vaijosh/hugegraph
that referenced
this pull request
Jul 8, 2026
apache#3081 -added multipart upload for the S3 storage provider
vaijosh
force-pushed
the
Hstore+CloudStorage
branch
from
6a3f496 to
8b6d597
Compare
apache#3081 -added multipart upload for the S3 storage provider
vaijosh
force-pushed
the
Hstore+CloudStorage
branch
from
8b6d597 to
4275426
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3081 +/- ##
============================================
- Coverage 36.80% 30.62% -6.18%
- Complexity 338 395 +57
============================================
Files 805 810 +5
Lines 68587 69157 +570
Branches 9029 9119 +90
============================================
- Hits 25241 21180 -4061
- Misses 40653 45555 +4902
+ Partials 2693 2422 -271 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
apache#3081 - Improved test coverage.
VGalaxies
reviewed
Jul 11, 2026
VGalaxies
left a comment
VGalaxies
left a comment
Contributor
There was a problem hiding this comment.
Review summary
- Blocking: yes
- Summary: The change cannot currently guarantee recoverable RocksDB state, may silently disable cloud storage, and contains S3 retry and deployment defects.
- Evidence:
- Static analysis of
git diff origin/master...HEAD - shell syntax and POM XML checks passed
git diff --checkreported whitespace-only errors
- Static analysis of
| if (provider == null) { | ||
| return; | ||
| } | ||
| preHydrateDbFiles(provider, dbName, dbPath); |
Contributor
There was a problem hiding this comment.
High: Recreated databases can hydrate deleted data
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListener.java:144
Evidence
- The listener hydrates the existing remote prefix during every open but implements neither
onDBDeleteBeginnoronDBDeleted; directory deletion outside RocksDB generates no table-file deletion callbacks.
Impact
- Recreating a graph or partition at the same path can download and ingest stale data from the deleted database.
Requested fix
- Delete or tombstone the remote database generation during database destruction and make hydration honor that generation marker.
| } | ||
| String remoteKey = toRelativeKey(filePath); | ||
| try { | ||
| provider.uploadFile(filePath, remoteKey); |
Contributor
There was a problem hiding this comment.
High: Cloud upload blocks the RocksDB event thread
hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListener.java:217
Evidence
RocksDBFactoryinvokes listeners inline, and this callback synchronously performs the entire S3 upload, including multipart retries and sleeps. RocksDB documents that these callbacks run on the triggering thread and should not perform extended work.
Impact
- Slow or unavailable object storage can block flush/compaction threads and cause write stalls.
Requested fix
- Hand uploads to a bounded executor and pin each SST using a checkpoint, hard link, or spool copy before returning from the callback.
| .uploadId(uploadId).partNumber(partNumber) | ||
| .contentLength(partLen) | ||
| .build(), | ||
| RequestBody.fromInputStream(partStream, partLen)); |
Contributor
There was a problem hiding this comment.
High: Multipart request bodies cannot be replayed for SDK retries
hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProvider.java:538
Evidence
RequestBody.fromInputStream()receivesLimitedInputStream, which has no mark/reset support. AWS SDK retries require a replayable stream and can fail withIllegalStateException, which the surrounding retry loop does not catch.
Impact
- A transient failure during a large upload can abort the multipart operation instead of retrying the affected part.
Requested fix
- Supply a
ContentStreamProviderthat opens, seeks, and bounds a fresh file stream for each SDK attempt.
| .build(), | ||
| RequestBody.fromInputStream(partStream, partLen)); | ||
| return resp.eTag(); | ||
| } catch (SdkClientException e) { |
Contributor
There was a problem hiding this comment.
High: S3 service failures bypass the provider retry contract
hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProvider.java:540
Evidence
- Provider operations catch only
SdkClientException; S3 service responses useS3Exception/AwsServiceException, which are separate exception types. The outer part retry catches onlyIOException.
Impact
- 429/5xx responses bypass part-level retries, while callers can receive unchecked exceptions despite the provider’s
IOExceptioncontract.
Requested fix
- Catch service exceptions, classify retryable versus permanent status codes, and translate them consistently into retryable or non-retryable cloud exceptions.
| log.warn("S3 multipart abort failed: key={} uploadId={} reason={}", | ||
| fullKey, uploadId, abortEx.getMessage()); | ||
| } | ||
| throw new IOException( |
Contributor
There was a problem hiding this comment.
Medium: Multipart wrapping erases the direct-DLQ marker
hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProvider.java:506
Evidence
uploadOnePartWithRetry()throwsCloudStorageNonRetryableException, but the enclosing catch wraps it as a plainIOException; the queue recognizes non-retryable failures only by type.
Impact
multipart-exhausted-direct-dlq=truestill permits expensive whole-file retries.
Requested fix
- Abort the multipart upload, then rethrow
CloudStorageNonRetryableExceptionunchanged.
| https://central.sonatype.com/artifact/org.yaml/snakeyaml/1.28 -> Apache 2.0 | ||
| https://central.sonatype.com/artifact/org.yaml/snakeyaml/2.2 -> Apache 2.0 | ||
| https://central.sonatype.com/artifact/org.zeroturnaround/zt-zip/1.14 -> Apache 2.0 | ||
| https://central.sonatype.com/artifact/software.amazon.awssdk/annotations/2.25.60 -> Apache 2.0 |
Contributor
There was a problem hiding this comment.
Medium: Release metadata lists different dependency versions
install-dist/release-docs/LICENSE:761
Evidence
- The provider pins AWS SDK
2.33.8, andknown-dependencies.txtrecords2.33.8with Netty4.1.126.Final; LICENSE adds AWS2.25.60and Netty4.1.108.Final.
Impact
- Binary release licensing metadata does not describe the actual bundled artifacts.
Requested fix
- Regenerate the inventory from the packaged distribution and update LICENSE/NOTICE to the exact resolved versions and components.
| wait_svc "server" 180 | ||
| log "waiting for graph backend..." | ||
| wait_http "$GRAPH_API_BASE/graph/vertices" 60 | ||
| log "✓ SUCCESS: Cloud storage infrastructure ready" |
Contributor
There was a problem hiding this comment.
Medium: The advertised SST test never verifies cloud storage
docker/cloud-storage/scripts/test-graph-queries-and-sst.sh:378
Evidence
- The script reports success after service and HTTP health checks; it never loads graph data, triggers or counts SST uploads, checks MinIO, tests recovery, or produces the reports promised by the README.
Impact
- The validation passes even when the provider is disabled by the entrypoint namespace defect.
Requested fix
- Fail unless graph writes succeed, SST objects appear in every expected bucket, and a recovery read succeeds from cloud-backed state.
apache#3081 - Improved review comments. - Synced METADATA, CURRENT AND OPTIONS to cloud so that we can recover the DB after cluster crash scenarios. - Improved the resiliancy and reduced the data loss possibilities.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR introduces a cloud storage architecture for HugeGraph Store (HStore). In cloud-native environments, storage nodes are ephemeral, making reliance on local disk storage a single point of failure. This implementation decouples the storage layer from local disk dependencies, allowing SST files to be offloaded to durable, scalable cloud storage providers.
Key Changes
Pluggable Architecture
CloudStorageProviderSPI interface to enable extensible storage backends.CloudStorageProviderFactoryfor seamless provider discovery and lifecycle management.S3 Provider Implementation
hg-store-cloud-s3module, leveraging AWS SDK v2 for production-ready S3/S3 compatible storage interaction.Lifecycle & Event Integration
CloudStorageEventListenerwithRocksDBFactory. This hooks into critical SST lifecycle events (onTableFileCreated,onTableFileDeleted) to ensure synchronized state between local RocksDB and cloud storage.onDBCreated) to backfill pre-existing files and read-miss on-demand hydration to ensure data availability.Infrastructure & Testing
docker/cloud-storage/using MinIO.test-graph-queries-and-sst.sh) to verify end-to-end data durability and query consistency.Implementation Highlights
META-INF/services.RocksdbEventListenerwithinRocksDBFactoryto intercept file operations without modifying core RocksDB logic.onReadMissto prevent redundant hydration requests during high-concurrency read scenarios.Verifying These Changes
CloudStorageConfigTest), factory registration, and event listener logic.docker-composesetup with MinIO.application.ymlbindings for credentials, bucket management, and sync intervals.Impact
LICENSEandNOTICEaccordingly.cloud.storagenamespace toapplication.yml(disabled by default).Documentation
hugegraph-store/docs/pluggable-cloud-storage-architecture.md.