AI Software Engineering Agent Setup

📋 Related Issue: Issue #187 - AI Agent Test Environment Provisioning

Overview

This document provides comprehensive instructions for provisioning the AI Software Engineering Agent service account (ai-swe-agent@construction-code-expert-test.iam.gserviceaccount.com) with all required permissions for automated development, testing, and deployment in the test environment.

Service Account Creation

Step 1: Set Environment Variables

export GCP_PROJECT_ID="construction-code-expert-test"
export SERVICE_ACCOUNT_ID="ai-swe-agent"
export SERVICE_ACCOUNT_DISPLAY_NAME="AI Software Engineer Agent"
export SERVICE_ACCOUNT_EMAIL="${SERVICE_ACCOUNT_ID}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"

Step 2: Create the Service Account

gcloud iam service-accounts create "${SERVICE_ACCOUNT_ID}" \
    --project="${GCP_PROJECT_ID}" \
    --display-name="${SERVICE_ACCOUNT_DISPLAY_NAME}"

Step 3: Create and Download Service Account Key

# Define key file path
SECRETS_FOLDER_PATH=".secrets/agent-credentials"
OUTPUT_KEY_FILE_PATH="${SECRETS_FOLDER_PATH}/${GCP_PROJECT_ID}.${SERVICE_ACCOUNT_ID}.json"
mkdir -p "${SECRETS_FOLDER_PATH}"

# Create and download the key
gcloud iam service-accounts keys create "${OUTPUT_KEY_FILE_PATH}" \
    --iam-account="${SERVICE_ACCOUNT_EMAIL}" \
    --project="${GCP_PROJECT_ID}"

IAM Permissions

The AI agent requires the following permissions for full deployment and testing functionality:

Core Application Permissions

# 1. GCS Bucket Access - for reading/writing architectural plans and generated content
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/storage.objectAdmin"

# 2. Firestore Access - for task progress tracking and real-time UI updates
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/datastore.user"

# 3. Vertex AI Access - for Gemini Pro model inference, embeddings, and vector search
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/aiplatform.user"

# 4. Firebase Admin - for user authentication and RBAC
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/firebase.admin"

Cloud Run Permissions

# 5. Cloud Run Developer - for deploying and managing Cloud Run services
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/run.developer"

# 6. Cloud Run Invoker - for triggering Cloud Run Jobs
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/run.invoker"

Build and Deployment Permissions

# 7. Cloud Build Editor - for building container images
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/cloudbuild.builds.editor"

# 8. Cloud Build Viewer - for viewing build logs
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/cloudbuild.builds.viewer"

# 9. Artifact Registry Writer - for pushing Docker images
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/artifactregistry.writer"

# 10. Storage Admin - for Cloud Build artifacts
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/storage.admin"

Service Management Permissions

# 11. Service Management Admin - for managing Cloud Endpoints (ESPv2)
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/servicemanagement.admin"

# 12. Service Usage Admin - for enabling/disabling APIs
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/serviceusage.serviceUsageAdmin"

# 13. Service Usage Consumer - for using Google Cloud services
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/serviceusage.serviceUsageConsumer"

Additional Permissions

# 14. Service Account User - for impersonating other service accounts during deployment
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/iam.serviceAccountUser"

# 15. Service Account Token Creator - for token generation during testing
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/iam.serviceAccountTokenCreator"

# 16. BigQuery Data Editor - for LLM log traces and analytics (future)
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/bigquery.dataEditor"

# 17. Logging Writer - for writing deployment logs
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/logging.logWriter"

# 18. Logging Viewer - for viewing logs
gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
    --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
    --role="roles/logging.viewer"

Secret Manager Access

The AI agent needs read access to specific secrets for retrieving API keys during builds, deployments, and testing:

# Grant access to Google Maps API key (Issue #236)
gcloud secrets add-iam-policy-binding google-maps-api-key \
  --project=${GCP_PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/secretmanager.secretAccessor"

# Grant access to Stripe secret key (Issues #202, #216)
gcloud secrets add-iam-policy-binding stripe-secret-key \
  --project=${GCP_PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/secretmanager.secretAccessor"

# Grant access to Stripe publishable key (Issues #202, #216)
# Note: Publishable key is optional in Secret Manager - can also be in setvars.sh (it's public)
gcloud secrets add-iam-policy-binding stripe-publishable-key \
  --project=${GCP_PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/secretmanager.secretAccessor" 2>/dev/null || \
  echo "⚠️  stripe-publishable-key not in Secret Manager - may be in env config files instead"

# Grant access to Stripe webhook secret (Issues #202, #216)
# Note: This secret may not exist yet if webhook endpoint hasn't been created
gcloud secrets add-iam-policy-binding stripe-webhook-secret \
  --project=${GCP_PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/secretmanager.secretAccessor" 2>/dev/null || \
  echo "⚠️  stripe-webhook-secret doesn't exist yet - create it after setting up webhook endpoint"

# Verify access
gcloud secrets get-iam-policy google-maps-api-key --project=${GCP_PROJECT_ID}
gcloud secrets get-iam-policy stripe-secret-key --project=${GCP_PROJECT_ID}
gcloud secrets get-iam-policy stripe-publishable-key --project=${GCP_PROJECT_ID} 2>/dev/null
gcloud secrets get-iam-policy stripe-webhook-secret --project=${GCP_PROJECT_ID} 2>/dev/null

Use Cases:

Google Maps (Issue #236): Retrieve API key for address autocomplete and geocoding
Stripe Payment (Issues #202, #216): Load payment processing credentials for billing tests
- stripe-secret-key: Backend secret (sk_test_* or sk_live_*)
- stripe-webhook-secret: Webhook signature verification (whsec_*)
- Note: Stripe publishable key (pk_test_, pk_live_) does NOT need Secret Manager - it's safe to commit in frontend config files
Future integrations: Other third-party API keys

Stripe Key Storage Summary:

Backend Keys (Secret Manager required):
  • stripe-secret-key          → sk_test_xxx (for dev/test) or sk_live_xxx (for prod)
  • stripe-webhook-secret       → whsec_xxx (get from webhook endpoint)

Frontend Keys (Can commit to git):
  • STRIPE_PUBLISHABLE_KEY      → pk_test_xxx or pk_live_xxx
                                  (stored in env/*/firebase/m3/setvars.sh)
                                  (public key, domain-restricted, safe to commit)

Cross-Environment Access (for local development/testing):

When running tests locally against a different environment's secrets (e.g., test project accessing dev secrets):

# Example: Grant test project's service account access to dev project secrets
export DEV_PROJECT="construction-code-expert-dev"
export TEST_PROJECT="construction-code-expert-test"
export TEST_SERVICE_ACCOUNT="ai-swe-agent@${TEST_PROJECT}.iam.gserviceaccount.com"

# Grant test service account access to dev Stripe secrets
gcloud secrets add-iam-policy-binding stripe-secret-key \
  --project=${DEV_PROJECT} \
  --member="serviceAccount:${TEST_SERVICE_ACCOUNT}" \
  --role="roles/secretmanager.secretAccessor"

gcloud secrets add-iam-policy-binding stripe-webhook-secret \
  --project=${DEV_PROJECT} \
  --member="serviceAccount:${TEST_SERVICE_ACCOUNT}" \
  --role="roles/secretmanager.secretAccessor" 2>/dev/null || \
  echo "⚠️  stripe-webhook-secret doesn't exist in ${DEV_PROJECT} yet"

Security Note: Only grant cross-project access when necessary (e.g., CI/CD pipelines, testing). Keep production secrets isolated.

Dev Container Configuration

To enable the agent to use the service account credentials in the dev container, configure the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Multi-Architecture Support

The devcontainer supports both ARM64 (Apple Silicon Macs) and AMD64 (Intel/AMD) architectures using Docker BuildKit's TARGETARCH automatic argument.

How TARGETARCH Works

Scenario	How TARGETARCH is set	Value
Devcontainer (local dev)	Defaults to Docker daemon's host architecture	`arm64` (M1/M2/M3) or `amd64` (Intel)
docker build (no flag)	Defaults to Docker daemon's host arch	varies
docker build --platform linux/amd64	Explicitly set via `--platform` flag	`amd64`
Cloud Build	Runs on GCE (x86-64 infrastructure)	`amd64`
GitHub Actions (default runners)	x86-64 runners	`amd64`

Architecture-Aware JAVA_HOME

The Dockerfile uses TARGETARCH to set JAVA_HOME dynamically:

# temurin-23-jdk installs to /usr/lib/jvm/temurin-23-jdk-{arch}
ARG TARGETARCH
ENV JAVA_HOME=/usr/lib/jvm/temurin-23-jdk-${TARGETARCH}

This ensures:

ARM64 Macs (M1/M2/M3): JAVA_HOME=/usr/lib/jvm/temurin-23-jdk-arm64
Intel/AMD64 machines: JAVA_HOME=/usr/lib/jvm/temurin-23-jdk-amd64

Key Insight: Local vs CI/CD Builds

The devcontainer Dockerfile (.devcontainer/Dockerfile) is built locally by VS Code/Cursor when you open the workspace. It's NOT pushed to Artifact Registry or used in CI/CD pipelines.

For CI/CD and production deployments, we use different Dockerfiles in env/ that are built by Cloud Build, which runs on amd64 infrastructure.

Related Issue: Issue #418 - Discovered during Copilot code review

Update `.devcontainer/devcontainer.json`

{
  "containerEnv": {
    "GOOGLE_APPLICATION_CREDENTIALS": "/workspaces/construction-code-expert/.secrets/agent-credentials/construction-code-expert-test.ai-swe-agent.json"
  }
}

After updating, rebuild the dev container for changes to take effect.

Verify Authentication

# Check authenticated account
gcloud auth list

# Verify project is set
gcloud config get-value project

# Test access to a GCS bucket
gsutil ls gs://construction-code-expert-test/

# Test access to Secret Manager
gcloud secrets list --project=construction-code-expert-test

Permissions Summary

Permission	Role	Purpose
Storage	`storage.objectAdmin`	Read/write architectural plans and files
Firestore	`datastore.user`	Task tracking and real-time updates
Vertex AI	`aiplatform.user`	Gemini Pro model inference
Firebase	`firebase.admin`	User authentication and RBAC
Cloud Run	`run.developer`, `run.invoker`	Deploy services and trigger jobs
Cloud Build	`cloudbuild.builds.editor`	Build Docker images
Artifact Registry	`artifactregistry.writer`	Push container images
Service Management	`servicemanagement.admin`	Manage Cloud Endpoints (ESPv2)
Service Usage	`serviceusage.serviceUsageAdmin`	Enable/disable APIs
IAM	`iam.serviceAccountUser`, `iam.serviceAccountTokenCreator`	Service account impersonation and token generation
Secret Manager	`secretmanager.secretAccessor`	Read API keys: google-maps-api-key (Issue #236), stripe-secret-key, stripe-webhook-secret (Issues #202, #216)
Logging	`logging.logWriter`, `logging.viewer`	Write and view logs
BigQuery	`bigquery.dataEditor`	LLM traces and analytics

Testing Agent Permissions

After granting permissions, verify the agent can perform key operations:

Test Cloud Run Deployment

# Verify can deploy Cloud Run service
gcloud run services list --project=${GCP_PROJECT_ID}

Test Secret Manager Access

# Verify can read Google Maps API key
gcloud secrets versions access latest \
  --secret=google-maps-api-key \
  --project=${GCP_PROJECT_ID}

# Verify can read Stripe secret key (Issues #202, #216)
gcloud secrets versions access latest \
  --secret=stripe-secret-key \
  --project=${GCP_PROJECT_ID}

# Verify can read Stripe webhook secret (optional - may not exist yet)
gcloud secrets versions access latest \
  --secret=stripe-webhook-secret \
  --project=${GCP_PROJECT_ID} 2>/dev/null || \
  echo "⚠️  stripe-webhook-secret not found - this is OK if webhook endpoint hasn't been created yet"

Test Firebase Token Generation

# Generate test token
./firebase-token-generator/generate-token.sh --env test ai-swe-agent-test@codetricks.org

Test GCS Access

# List bucket contents
gsutil ls gs://construction-code-expert-test/inputs/architectural-plans/

Troubleshooting

Permission Denied Errors

If the agent encounters permission denied errors:

Verify service account is active:

gcloud iam service-accounts describe ${SERVICE_ACCOUNT_EMAIL} \
  --project=${GCP_PROJECT_ID}

Check IAM policy bindings:

gcloud projects get-iam-policy ${GCP_PROJECT_ID} \
  --flatten="bindings[].members" \
  --format="table(bindings.role)" \
  --filter="bindings.members:${SERVICE_ACCOUNT_EMAIL}"

Verify credentials are loaded:

echo $GOOGLE_APPLICATION_CREDENTIALS
cat $GOOGLE_APPLICATION_CREDENTIALS | jq .client_email

API Not Enabled

If you see "API not enabled" errors, enable the required API:

# Enable the API (example: Secret Manager)
gcloud services enable secretmanager.googleapis.com \
  --project=${GCP_PROJECT_ID}

Security Considerations

Principle of Least Privilege: Only grant permissions actually needed by the agent
Read-Only When Possible: Use viewer/accessor roles instead of admin when read-only access suffices
Rotate Keys Regularly: Service account keys should be rotated periodically
Audit Access: Regularly review IAM policy bindings to ensure they're still needed
Separate Environments: The agent only has access to the test environment, not prod

Future Permissions

As new features are added requiring additional GCP services or secrets, update this document with the required permissions. Follow the same pattern:

# Template for new secret access
gcloud secrets add-iam-policy-binding <secret-name> \
  --project=${GCP_PROJECT_ID} \
  --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
  --role="roles/secretmanager.secretAccessor"

References

Issue #187 - AI Agent Test Environment Provisioning
Issue #236 - Google Maps Integration (Secret Manager access)
Google Cloud IAM Roles
Service Account Best Practices

Overview​

Service Account Creation​

Step 1: Set Environment Variables​

Step 2: Create the Service Account​

Step 3: Create and Download Service Account Key​

IAM Permissions​

Core Application Permissions​

Cloud Run Permissions​

Build and Deployment Permissions​

Service Management Permissions​

Additional Permissions​

Secret Manager Access​

Dev Container Configuration​

Multi-Architecture Support​

How TARGETARCH Works​

Architecture-Aware JAVA_HOME​

Key Insight: Local vs CI/CD Builds​

Update .devcontainer/devcontainer.json​

Verify Authentication​

Permissions Summary​

Testing Agent Permissions​

Test Cloud Run Deployment​

Test Secret Manager Access​

Test Firebase Token Generation​

Test GCS Access​

Troubleshooting​

Permission Denied Errors​

API Not Enabled​

Security Considerations​

Future Permissions​

References​