feat(cli): support remote GCS state fetching by dwnoble · Pull Request #174 · datacommonsorg/datacommons

dwnoble · 2026-07-17T11:47:20Z

Adds support for fetching Terraform state outputs directly from GCS, allowing DCP admin CLI commands to be executed from anywhere outside of the local initialized Terraform folder.

Key Changes:

Added global options under the admin command group:
- --project-id: GCP project ID used to locate the state bucket.
- --instance-name: DCP instance name used to locate the state bucket.
- --tf-state-location: Direct GCS URI pointing to the Terraform state file.
Context-Aware Remote State Fetching:
- Updated get_terraform_output to check Click context parameters. If specified, it fetches the state JSON from GCS and extracts values directly. If not, it falls back to local terraform output -json.
Colorized Error Monkeypatch:
- Custom styled Click exceptions to colorize all CLI error outputs to stderr in bold red.

Usage Examples:

Execute configuration tasks from outside your deployment directory:

# Example A: Query ingestion configuration using project ID and instance name
datacommons admin \
  --project-id datcom-website-dev \
  --instance-name dan-ns \
  ingest show-config

# Example B: Initialize database using a direct state location
datacommons admin \
  --tf-state-location gs://tf-state-dan-ns-datcom-website-dev/terraform/state/dan-ns/default.tfstate \
  init-db

(Note: Chained off 'deprecate-namespace' branch/PR)

…name or tf-state-location

gemini-code-assist

Code Review

This pull request transitions the configuration from using 'namespace' to 'instance_name' across Terraform files and the admin CLI, while maintaining backward compatibility. It also adds support for fetching Terraform outputs directly from GCS remote state files. The review feedback highlights several opportunities to make the remote state retrieval more robust, such as properly handling whitespace-only state locations, ensuring the parsed JSON state is a dictionary, and passing the project ID to the GCS client, along with adding corresponding assertions in the unit tests.

gemini-code-assist · 2026-07-17T11:49:40Z

+    if tf_state_location:
+        gcs_uri = tf_state_location.strip()
+        if not gcs_uri.endswith(".tfstate"):
+            gcs_uri = gcs_uri.rstrip("/") + "/default.tfstate"


If tf_state_location is a whitespace-only string (e.g., " "), bool(tf_state_location) evaluates to True, which bypasses the elif project_id and instance_name fallback. It then strips to an empty string and appends /default.tfstate, resulting in an invalid GCS URI error. We should strip the string first and check if it is non-empty to allow falling back to the project_id and instance_name logic if they are provided.

Suggested change

if tf_state_location:

gcs_uri = tf_state_location.strip()

if not gcs_uri.endswith(".tfstate"):

gcs_uri = gcs_uri.rstrip("/") + "/default.tfstate"

tf_state_location_stripped = tf_state_location.strip() if tf_state_location else ""

if tf_state_location_stripped:

gcs_uri = tf_state_location_stripped

if not gcs_uri.endswith(".tfstate"):

gcs_uri = gcs_uri.rstrip("/") + "/default.tfstate"

gemini-code-assist · 2026-07-17T11:49:40Z

+        try:
+            client = storage.Client()
+        except Exception as e:


When initializing the storage.Client, it is highly recommended to pass the project_id if it is available. This ensures that the client is explicitly bound to the target GCP project for quota and billing purposes, rather than relying on the ambient default project of the active gcloud credentials.

Suggested change

try:

client = storage.Client()

except Exception as e:

try:

client = storage.Client(project=project_id) if project_id else storage.Client()

except Exception as e:

gemini-code-assist · 2026-07-17T11:49:40Z

+        try:
+            state_data = json.loads(content)
+        except json.JSONDecodeError:
+            raise click.ClickException(f"Failed to parse Terraform state at '{gcs_uri}' as valid JSON.")
+
+        outputs = state_data.get("outputs", {})


If the downloaded GCS file is valid JSON but not a dictionary (e.g., a JSON list or primitive), calling state_data.get("outputs", {}) will raise an AttributeError. We should defensively verify that state_data is a dictionary.

try: state_data = json.loads(content) except json.JSONDecodeError: raise click.ClickException(f"Failed to parse Terraform state at '{gcs_uri}' as valid JSON.") if not isinstance(state_data, dict): raise click.ClickException(f"Invalid Terraform state format at '{gcs_uri}': expected a JSON object.") outputs = state_data.get("outputs", {})

gemini-code-assist · 2026-07-17T11:49:41Z

    if key not in outputs:
        from pathlib import Path
-
+        location_desc = tf_state_location or f"GCP project: '{project_id}' / instance: '{instance_name}'" if use_remote else f"'{Path.cwd()}'"


The inline conditional expression combined with or has tricky operator precedence (since if-else has lower precedence than or in Python, it evaluates as (tf_state_location or ...) if use_remote else ...). While correct, this is highly prone to readability issues and future bugs. Splitting this into a clear if/else block is much more maintainable.

if use_remote: location_desc = tf_state_location or f"GCP project: '{project_id}' / instance: '{instance_name}'" else: location_desc = f"'{Path.cwd()}'"

gemini-code-assist · 2026-07-17T11:49:41Z

+    assert result.exit_code == 0
+    assert "VAL=gcs-resolved-val" in result.output
+    mock_client_inst.bucket.assert_called_once_with("tf-state-mock-instance-mock-project")
+    mock_bucket.blob.assert_called_once_with("terraform/state/mock-instance/default.tfstate")


Assert that the storage.Client is initialized with the correct project_id to ensure the client is explicitly bound to the target GCP project.

Suggested change

assert result.exit_code == 0

assert "VAL=gcs-resolved-val" in result.output

mock_client_inst.bucket.assert_called_once_with("tf-state-mock-instance-mock-project")

mock_bucket.blob.assert_called_once_with("terraform/state/mock-instance/default.tfstate")

assert result.exit_code == 0

assert "VAL=gcs-resolved-val" in result.output

mock_storage_client.assert_called_once_with(project="mock-project")

mock_client_inst.bucket.assert_called_once_with("tf-state-mock-instance-mock-project")

mock_bucket.blob.assert_called_once_with("terraform/state/mock-instance/default.tfstate")

gemini-code-assist · 2026-07-17T11:49:41Z

+    mock_client_inst.reset_mock()
+    mock_bucket.reset_mock()
+    result = runner.invoke(


Reset mock_storage_client before the second test case to prevent assertions from being affected by the first test case.

Suggested change

mock_client_inst.reset_mock()

mock_bucket.reset_mock()

result = runner.invoke(

mock_storage_client.reset_mock()

mock_client_inst.reset_mock()

mock_bucket.reset_mock()

result = runner.invoke(

gemini-code-assist · 2026-07-17T11:49:41Z

+    assert result.exit_code == 0
+    assert "VAL=gcs-resolved-val" in result.output
+    mock_client_inst.bucket.assert_called_once_with("custom-bucket")
+    mock_bucket.blob.assert_called_once_with("custom-prefix/state.tfstate")


Assert that the storage.Client is initialized without arguments when only tf-state-location is provided.

Suggested change

assert result.exit_code == 0

assert "VAL=gcs-resolved-val" in result.output

mock_client_inst.bucket.assert_called_once_with("custom-bucket")

mock_bucket.blob.assert_called_once_with("custom-prefix/state.tfstate")

assert result.exit_code == 0

assert "VAL=gcs-resolved-val" in result.output

mock_storage_client.assert_called_once_with()

mock_client_inst.bucket.assert_called_once_with("custom-bucket")

mock_bucket.blob.assert_called_once_with("custom-prefix/state.tfstate")

gmechali · 2026-07-17T14:29:52Z

        tfvars_content = tfvars_example
        tfvars_content = tfvars_content.replace('"$$PROJECT_ID$$"', f'"{project_id}"')
-        tfvars_content = tfvars_content.replace('"$$NAMESPACE$$"', f'"{namespace}"')
+        tfvars_content = tfvars_content.replace('"$$NAMESPACE$$"', f'"{instance_name}"')


Any reason to leave the NAMESPACE here?

gmechali · 2026-07-17T14:30:52Z

    """Initialize Terraform scaffolding for Data Commons administration/infrastructure."""
    click.secho("Data Commons Admin Init", fg="cyan", bold=True)

+    # For backward compatibility, fallback to namespace if instance_name is empty


I think we rip the bandaid off tbh. Or we ll carry this over far longer than we need.
No one is really using this yet, if so, they can continue using previous version of the CLI.

gmechali · 2026-07-17T14:31:09Z

+    project_id: str | None,
+    instance_name: str | None,
+) -> dict:
+    from google.cloud import storage


nit: move imports to top of file?

gmechali · 2026-07-17T14:34:46Z

+
+    if use_remote:
+        outputs = _get_outputs_from_gcs(tf_state_location, project_id, instance_name)
+    else:


can we move this elsewhere and put it in a helper for validation to use the CLI (or at least this function)?

There's no point in fetching the terraform state and doig all that logic above if it's gonna fail here.

gmechali · 2026-07-17T14:37:21Z

+        blob = bucket.blob(blob_name)
+
+        try:
+            content = blob.download_as_text()


one criticism from Jorge is that he doesn't want us to bring in the terraform state locally as it opens up the possibility of someone running a tf command they shouldnt.

For this specific scenario, we can just read the file from GCS and parse the output without copying the data over in the local terraform state right?

dwnoble added 2 commits July 17, 2026 03:48

refactor(infra): deprecate namespace in favor of instance-name

17f0207

feat(cli): support remote GCS state fetching via project-id/instance-…

84acd19

…name or tf-state-location

gemini-code-assist Bot reviewed Jul 17, 2026

View reviewed changes

dwnoble requested a review from gmechali July 17, 2026 11:52

gmechali reviewed Jul 17, 2026

View reviewed changes

dwnoble requested a review from clincoln8 July 17, 2026 18:21

Uh oh!

Conversation

dwnoble commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Changes:

Usage Examples:

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gmechali Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gmechali Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gmechali Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gmechali Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

gmechali Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dwnoble commented Jul 17, 2026 •

edited

Loading