resource_spanner_instance passes incorrect billingProject during force_destroy

[Noremac201]

Community Note

• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

https://github.com/hashicorp/terraform-provider-google/blob/main/google/services/spanner/resource_spanner_instance.go#L776

https://github.com/hashicorp/terraform-provider-google/blob/main/google/services/spanner/resource_spanner_instance.go#L38

Method signatures are backwards.

Terraform Version & Provider Version(s)

All google-provider-versions

Affected Resource(s)

resource_spanner_instance

Terraform Configuration

No response

Debug Output

No response

Expected Behavior

No response

Actual Behavior

No response

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

[Noremac201]

I'll open a PR for this fix.

[ggtisc]

Hi @Noremac201

I've tried to replicate this issue with the following code which is similar to the terraform registry example (link here), but everything worked fine without errors during creation, deletion and forcing deletion.

resource "google_spanner_instance" "spanner_instance_21007" {
  config       = "regional-us-central1"
  display_name = "spanner-instance-21007"
  num_nodes    = 2
  edition      = "STANDARD"
  default_backup_schedule_type = "AUTOMATIC"
  labels = {
    "foo" = "bar"
  }
}

Could you please share with us the code you are using WITHOUT USING MODULES, VARIABLES or LOCALS to ensure this is an issue as well as being clearer with the steps?

You commented this happens with force_destroy, could you also share with us the command you are using? example:

terraform destroy -target=google_spanner_instance.spanner_instance_21007 -auto-approve

For sensitive data you could use examples like:

1. project = "project-21007"
2. member = "user:[email protected]"

[Noremac201]

Hey @ggtisc , thanks for looking into this.

This is only reproducible if an instance has a database that has a backup is attempted to be deleted. Sorry I wasn't clear since I was going to send a PR myself.

So the steps are:

1. Deploy the following TF code:
2. Either wait for an automated scheduled backup to occur, or create one on your own outside of terraform (this is important). This is how automated backups works.
3. Attempt to delete with force_destroy enabled.

provider "google" {
  project     = "xyz"
  region      = "us-central1"
}

resource "google_spanner_instance" "example" {
  config       = "regional-us-central1"
  display_name = "Test Spanner Instance"
  num_nodes    = 2
  edition      = "STANDARD"
  force_destroy = "true"
  default_backup_schedule_type = "AUTOMATIC"
  labels = {
    "foo" = "bar"
  }
}

resource "google_spanner_database" "database" {
  instance = google_spanner_instance.example.name
  name     = "my-database"
  ddl = [
    "CREATE TABLE t1 (t1 INT64 NOT NULL,) PRIMARY KEY(t1)",
    "CREATE TABLE t2 (t2 INT64 NOT NULL,) PRIMARY KEY(t2)",
  ]
  deletion_protection = false
}

Edit* Removed error message as it was incorrect, I will paste the correct one once the backup finishes.

[Noremac201]

Hmm, ok that is very odd, I'll continue to try to replicate this, I haven't been able to as you said.

The real place that I found this was in KCC's K8s Config Connector with this error:

    message: |-
      Delete call failed: error deleting resource: [{0 googleapi: Error 400: Project 'projects/Terraform/ (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/kcc/controller-manager' not found or deleted.
      Details:
      [
        {
          "@type": "type.googleapis.com/google.rpc.ErrorInfo",
          "domain": "googleapis.com",
          "metadata": {
            "consumer": "projects/Terraform/ (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/kcc/controller-manager",
            "containerInfo": "projects/Terraform/ (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google-beta/kcc/controller-manager",
            "service": "spanner.googleapis.com"
          },
          "reason": "USER_PROJECT_DENIED"
        }
      ]  []}]

Which in this case it's very obvious that billing project and userAgent are swapped somewhere, and the fact that the code says they're swapped here seems to confirm it as well.

I'm not sure how it's not reproducible here though..

[ggtisc]

Maybe some update occurred during this short time. In any case, if it is caused by 3rd party software, whether directly or orchestrated, we would not be able to intervene, but if it is directly related to terraform and google providers and it happens again, please let us know.

[Noremac201]

Sorry to necro-post, this has been submitted.

Debug logs show this:

 319 2025-01-23T13:41:31.076-0800 [DEBUG] provider.terraform-provider-google_v6.17.0_x5: ---[ REQUEST ]---------------------------------------
 320 2025-01-23T13:41:31.076-0800 [DEBUG] provider.terraform-provider-google_v6.17.0_x5: DELETE /v1/projects/xyz/instances/tfgen-spanid-20250123201705372/backups/test1?alt=json HTTP/1.1
 321 2025-01-23T13:41:31.076-0800 [DEBUG] provider.terraform-provider-google_v6.17.0_x5: Host: spanner.googleapis.com
 322 2025-01-23T13:41:31.076-0800 [DEBUG] provider.terraform-provider-google_v6.17.0_x5: User-Agent: xyx
 323 2025-01-23T13:41:31.077-0800 [DEBUG] provider.terraform-provider-google_v6.17.0_x5: Content-Type: application/json
 324 2025-01-23T13:41:31.077-0800 [DEBUG] provider.terraform-provider-google_v6.17.0_x5: Accept-Encoding: gzip

So in TF's case for some reason useragent is set as the same as project number, which is how this bug was uncaught in TF for so long. However, when user-agent is used properly, the issue happens.