API Integration Testing with gRPC
This document provides comprehensive guidance on end-to-end (E2E) integration testing of the PermitProof backend using gRPC calls from Java test code.
Overview
The project uses Java-based gRPC integration tests to validate complete user journeys by making actual gRPC network calls to running servers. These tests verify the entire stack including network serialization, message size limits, authentication, and business logic.
Key Achievement
🎯 Real Network Testing: We've implemented comprehensive integration tests that make actual gRPC calls to deployed environments, testing the complete request-response cycle including authentication, authorization, and data persistence.
Available Integration Tests
The gRPC integration test suite is located in src/test/java/ and includes the following test categories:
Core Integration Tests
ArchitecturalPlanWriteServiceGrpcIntegrationTest.java- Project creation, file upload (small and large files), server connectivityComplianceReportIntegrationTest.java- Complete compliance report workflow with GCS and Firestore validationCostAnalysisIntegrationTest.java- Cost tracking and task monitoring throughout project lifecycleProjectCopyFunctionalityTest.java- Project duplication with sharing settings
Authentication & Authorization Tests
FirebaseRbacTest.java- Role-based access control validationAdminRolePreservationIntegrationTest.java- Admin role persistence across operationsInviteUserToProjectTest.java- User invitation and permission granting
Full Pipeline E2E Test
EndToEndIntegrationTest.java- Complete workflow validation (create → upload → ingest → applicability → compliance)
Quick Start: Running E2E Tests
The fastest way to run the full E2E integration test:
# Using the helper script (recommended)
./cli/utils/run-e2e-test.sh crsr
# Or with custom options
./cli/utils/run-e2e-test.sh crsr --prefix=SmokeTest
Stage Skip Options
For faster iteration during development, you can skip individual stages:
| Option | Description |
|---|---|
--skip-upload | Skip PDF upload stage |
--skip-ingestion | Skip page ingestion (Cloud Run Job) |
--skip-applicability | Skip applicability analysis (Cloud Run Job) |
--skip-compliance | Skip compliance report generation |
# Example: Skip compliance for faster iteration
./cli/utils/run-e2e-test.sh crsr --skip-compliance
# Example: Skip multiple stages
./cli/utils/run-e2e-test.sh crsr --skip-ingestion --skip-applicability
Resume from Existing Project
To rerun specific stages on an existing project:
# Resume from an existing project, skip early stages
./cli/utils/run-e2e-test.sh crsr \
--baseline-project=IntegrationTest-commit.abc123-time.10.30.00-PST \
--skip-ingestion \
--skip-applicability
Configuration Reference
| Option | Description |
|---|---|
--prefix=<name> | Prefix for new project names (default: IntegrationTest) |
--owners=<emails> | Comma-separated emails to share project with |
--books=<ids> | ICC book IDs for applicability (default: 2217) |
--baseline-project=<id> | Use existing project instead of creating new |
--baseline-file-id=<id> | Use existing file ID (implies --skip-upload) |
Advanced Maven pass-through (for edge cases):
./cli/utils/run-e2e-test.sh crsr -Dsome.custom.property=value
Prerequisites
-
ICC Book Data: Sync to target environment:
gsutil -m cp -r gs://construction-code-expert-repo/api/icc/content/ \
gs://construction-code-expert-<env>/api/icc/content/ -
Test User:
ai-swe-agent-test@codetricks.orgmust exist in Firebase -
Environment Deployed: Latest backend changes must be deployed
End-to-End Test Scenarios
Implemented Scenarios
-
Complete Project Workflow (
CostAnalysisIntegrationTest)- Copy baseline project with sharing settings
- Get project input files
- Start page ingestion task
- Monitor task completion via Firestore
- Validate cost analysis data
- Verify task status and progress tracking
-
Compliance Report Generation (
ComplianceReportIntegrationTest)- Project copying with GCS file structure verification
- Compliance report generation via gRPC
- GCS output file validation
- Firestore task tracking with cost analysis
- Multi-page compliance report validation
-
File Upload Operations (
ArchitecturalPlanWriteServiceGrpcIntegrationTest)- Small file upload (< 1MB)
- Large file upload (> 50MB)
- Message size limit testing (100MB)
- Network serialization validation
- Server configuration verification
-
Project Management
- Project creation with metadata
- Project copying with file structure
- Project deletion (soft and hard delete)
- Sharing settings management
-
Access Control Validation
- Firebase authentication token generation
- Role-based permission checking
- Admin role preservation
- User invitation workflow
Potential Future Scenarios
The following scenarios can be implemented using the existing gRPC infrastructure:
-
Multi-User Collaboration Workflows
- Share project with multiple users at different permission levels
- Verify concurrent access and modifications
- Test permission inheritance and revocation
- Validate real-time collaboration features
-
Code Analysis Pipeline
- Upload architectural plans
- Ingest multiple pages in parallel
- Run code applicability analysis on all pages
- Generate comprehensive compliance reports
- Download and validate report artifacts
-
Background Task Orchestration
- Start multiple async tasks concurrently
- Monitor task progress via Firestore
- Handle task failures and retries
- Verify task cancellation
- Test task timeout scenarios
-
Billing and Cost Tracking
- Validate cost accumulation across operations
- Test billing profile updates
- Verify transaction history
- Check balance deductions
- Test payment processing integration
-
Performance and Load Testing
- Concurrent project operations
- Large batch file uploads
- Stress testing with multiple users
- Memory leak detection
- Response time validation
-
Error Handling and Resilience
- Network failure scenarios
- Invalid input validation
- Authorization failures
- Resource exhaustion handling
- Graceful degradation testing
Running gRPC Integration Tests
Prerequisites
- Environment Setup: Test environment must be deployed with latest backend changes
- Authentication: Firebase Web API key and service account credentials configured
- Test Data: Baseline projects (e.g.,
SanJose-baseline) must exist in test environment - Network Access: Connectivity to Cloud Run services (test or local)
Local Development Testing
# Set Java environment (dev container)
export JAVA_HOME=/usr/lib/jvm/temurin-23-jdk-arm64
# Start local gRPC server (in one terminal)
./cli/start-local-server.sh
# Set authentication token (in another terminal)
export CODEPROOF_DEMO_USER_FIREBASE_TOKEN="your_firebase_token"
# Run specific integration test
mvn test -Dtest=ArchitecturalPlanWriteServiceGrpcIntegrationTest#testServerConnectivity
# Run small file upload test
mvn test -Dtest=ArchitecturalPlanWriteServiceGrpcIntegrationTest#testUploadSmallFileViaGrpc
# Run large file upload test with custom parameters
mvn test -Dtest=ArchitecturalPlanWriteServiceGrpcIntegrationTest#testUploadLargeFileViaGrpc \
-Dpdf.path="/path/to/your/large-file.pdf" \
-Dproject.id="test-large-upload-$(date +%s)" \
-Dupload.filename="test-plan.pdf" \
-Dauth.token="$CODEPROOF_DEMO_USER_FIREBASE_TOKEN"
# Run project creation test
mvn test -Dtest=ArchitecturalPlanWriteServiceGrpcIntegrationTest#testCreateProjectViaGrpc
Testing Against Deployed Environment
Recommended: Using the Helper Script
# Full E2E test on any environment
./cli/utils/run-e2e-test.sh crsr # Cursor environment
./cli/utils/run-e2e-test.sh agy # Antigravity environment
./cli/utils/run-e2e-test.sh gcli # Gemini CLI environment
# With custom options
./cli/utils/run-e2e-test.sh crsr --prefix=Nightly
# Skip stages for faster iteration
./cli/utils/run-e2e-test.sh crsr --skip-compliance
# Show all available options
./cli/utils/run-e2e-test.sh --help
Manual: Using Maven Directly
# Set environment variables
export FIREBASE_WEB_API_KEY="your_firebase_web_api_key"
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
# Run compliance report integration test against test environment
mvn test \
-Dtest=ComplianceReportIntegrationTest \
-Dserver.host=construction-code-expert-test-fhj3jauezq-uc.a.run.app \
-Dserver.port=443 \
-Dproject.id=construction-code-expert-test \
-DFIREBASE_WEB_API_KEY="$FIREBASE_WEB_API_KEY"
# Run cost analysis integration test
mvn test \
-Dtest=CostAnalysisIntegrationTest \
-Dserver.host=construction-code-expert-test-fhj3jauezq-uc.a.run.app \
-Dserver.port=443 \
-Dproject.id=construction-code-expert-test
# Run all integration tests
mvn test -Dtest="*IntegrationTest"
Using Helper Scripts
# Run compliance report integration test with all prerequisites
./run-compliance-integration-test.sh
# This script automatically:
# - Verifies environment variables are set
# - Checks Google Application Default Credentials
# - Compiles the project
# - Runs the integration test with proper configuration
Running Specific Test Suites
# Run only authentication/RBAC tests
mvn test -Dtest="*RbacTest,*AccessControlTest"
# Run only file upload tests
mvn test -Dtest="ArchitecturalPlanWriteServiceGrpcIntegrationTest#testUpload*"
# Run only project management tests
mvn test -Dtest="ProjectCopyFunctionalityTest,ArchitecturalPlanWriteServiceGrpcIntegrationTest#testCreateProjectViaGrpc"
# Run tests with specific tag or pattern
mvn test -Dtest="*IntegrationTest" -Dmaven.test.failure.ignore=false
Authentication in gRPC Integration Tests
How Authentication Works
The gRPC integration tests use Firebase authentication with bearer tokens:
- Token Generation: Tests generate Firebase custom tokens using the
firebase-token-generator/generate-token.shscript or Firebase Web API - gRPC Interceptor: Tests inject bearer tokens into gRPC call headers using client interceptors
- Server Validation: Backend validates tokens and enforces RBAC policies
- Test User: Tests use
ai-swe-agent-test@codetricks.orgwith appropriate permissions
Creating Authenticated gRPC Stubs
Example from ArchitecturalPlanWriteServiceGrpcIntegrationTest.java:
/**
* Creates a blocking stub with authorization header
*/
private ArchitecturalPlanWriteServiceGrpc.ArchitecturalPlanWriteServiceBlockingStub
createAuthenticatedStub(String authToken) {
ClientInterceptor authInterceptor = new ClientInterceptor() {
@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions,
Channel next) {
return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(
next.newCall(method, callOptions)) {
@Override
public void start(ClientCall.Listener<RespT> responseListener, Metadata headers) {
// Add Bearer token to authorization header
headers.put(
Metadata.Key.of("authorization", Metadata.ASCII_STRING_MARSHALLER),
"Bearer " + authToken
);
super.start(responseListener, headers);
}
};
}
};
return blockingStub.withInterceptors(authInterceptor)
.withDeadlineAfter(60, TimeUnit.SECONDS);
}
Token Generation Methods
Method 1: Using Firebase Web API (Recommended for Tests)
Example from CostAnalysisIntegrationTest.java:
private void setupAuthentication() throws Exception {
String firebaseWebApiKey = System.getProperty("FIREBASE_WEB_API_KEY");
if (firebaseWebApiKey == null || firebaseWebApiKey.isEmpty()) {
firebaseWebApiKey = System.getenv("FIREBASE_WEB_API_KEY");
}
if (firebaseWebApiKey == null || firebaseWebApiKey.isEmpty()) {
throw new IllegalStateException("FIREBASE_WEB_API_KEY must be set");
}
// Generate custom token using firebase-token-generator script
ProcessBuilder pb = new ProcessBuilder(
"./firebase-token-generator/generate-token.sh",
TEST_USER_EMAIL
);
pb.redirectErrorStream(true);
Process process = pb.start();
BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream())
);
String customToken = reader.readLine();
if (customToken == null || customToken.isEmpty()) {
throw new IllegalStateException("Failed to generate custom token");
}
// Exchange custom token for ID token via Firebase Web API
authToken = exchangeCustomTokenForIdToken(customToken, firebaseWebApiKey);
}
private String exchangeCustomTokenForIdToken(String customToken, String apiKey)
throws Exception {
// Make HTTP POST to Firebase Auth REST API
String url = "https://identitytoolkit.googleapis.com/v1/accounts:signInWithCustomToken?key=" + apiKey;
// ... HTTP request implementation ...
return idToken;
}
Method 2: Using Environment Variable
String authToken = System.getProperty("auth.token");
if (authToken == null || authToken.isEmpty()) {
authToken = System.getenv("CODEPROOF_DEMO_USER_FIREBASE_TOKEN");
}
if (authToken == null || authToken.isEmpty()) {
System.out.println("Skipping test - no auth token provided");
return;
}
Setting Up gRPC Channel
@Before
public void setUp() throws Exception {
// Generate authentication token
setupAuthentication();
// Initialize gRPC channel for test environment
String serverHost = System.getProperty("server.host", "localhost");
int serverPort = Integer.parseInt(System.getProperty("server.port", "8080"));
ManagedChannelBuilder<?> channelBuilder = ManagedChannelBuilder
.forAddress(serverHost, serverPort);
// Use TLS for production/test environments
if (serverPort == 443) {
channelBuilder.useTransportSecurity();
} else {
channelBuilder.usePlaintext();
}
channel = channelBuilder
.maxInboundMessageSize(100 * 1024 * 1024) // 100MB
.build();
// Create authenticated stubs
ClientInterceptor authInterceptor = createAuthInterceptor();
planService = ArchitecturalPlanServiceGrpc
.newBlockingStub(channel)
.withInterceptors(authInterceptor);
}
@After
public void tearDown() throws Exception {
if (channel != null) {
channel.shutdown().awaitTermination(5, TimeUnit.SECONDS);
}
}
Example Test Pattern with Authentication
@Test
public void testCompleteWorkflow() throws Exception {
// Step 1: Copy baseline project
CopyArchitecturalPlanRequest copyRequest = CopyArchitecturalPlanRequest.newBuilder()
.setSourceProjectId("SanJose-baseline")
.setTargetProjectId("test-project-" + UUID.randomUUID())
.setCopySharingSettings(true)
.build();
CopyArchitecturalPlanResponse copyResponse = writeService.copyArchitecturalPlan(copyRequest);
assertTrue("Project copy should succeed", copyResponse.getSuccess());
String projectId = copyResponse.getTargetProjectId();
// Step 2: Start page ingestion task
StartAsyncIngestFileRequest ingestRequest = StartAsyncIngestFileRequest.newBuilder()
.setProjectId(projectId)
.setFilename("building-plan.pdf")
.addPageNumbers(1)
.addPageNumbers(2)
.build();
StartAsyncIngestFileResponse ingestResponse = asyncWriteService
.startAsyncIngestFile(ingestRequest);
assertTrue("Ingestion should start successfully", ingestResponse.getSuccess());
String taskId = ingestResponse.getTaskId();
// Step 3: Monitor task completion via Firestore
Map<String, Object> taskData = waitForTaskCompletion(taskId);
assertEquals("Task should complete", "COMPLETE", taskData.get("status"));
// Step 4: Validate results
GetArchitecturalPlanRequest getRequest = GetArchitecturalPlanRequest.newBuilder()
.setArchitecturalPlanId(projectId)
.build();
ArchitecturalPlan plan = planService.getArchitecturalPlan(getRequest);
assertNotNull("Project should exist", plan);
assertTrue("Project should have ingested pages", plan.getPagesCount() > 0);
}
CI/CD Integration
Maven Test Execution
Integration tests can be run as part of the Maven build lifecycle:
# Run all tests (unit + integration)
mvn test
# Run only integration tests
mvn test -Dtest="*IntegrationTest"
# Skip tests during build
mvn clean package -DskipTests
# Run tests with specific profile
mvn test -P integration-tests
GitHub Actions Workflow
Example workflow for running gRPC integration tests:
name: gRPC Integration Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
schedule:
- cron: '0 2 * * *' # Run daily at 2 AM
jobs:
grpc-integration-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Java
uses: actions/setup-java@v3
with:
java-version: '23'
distribution: 'temurin'
- name: Setup Google Cloud credentials
env:
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
echo "$GCP_SA_KEY" > $HOME/gcp-key.json
export GOOGLE_APPLICATION_CREDENTIALS=$HOME/gcp-key.json
- name: Setup Firebase credentials
env:
FIREBASE_WEB_API_KEY: ${{ secrets.FIREBASE_WEB_API_KEY }}
run: |
echo "FIREBASE_WEB_API_KEY=$FIREBASE_WEB_API_KEY" >> $GITHUB_ENV
- name: Run gRPC integration tests
run: |
mvn test \
-Dtest="*IntegrationTest" \
-Dserver.host=construction-code-expert-test-fhj3jauezq-uc.a.run.app \
-Dserver.port=443 \
-Dproject.id=construction-code-expert-test \
-DFIREBASE_WEB_API_KEY=$FIREBASE_WEB_API_KEY
- name: Upload test results
if: always()
uses: actions/upload-artifact@v3
with:
name: test-results
path: target/surefire-reports/
Cloud Build Integration
For Google Cloud Build, add a test step to your cloudbuild.yaml:
steps:
# ... other build steps ...
- name: 'maven:3.9-eclipse-temurin-23'
id: 'integration-tests'
entrypoint: 'bash'
args:
- '-c'
- |
mvn test \
-Dtest="*IntegrationTest" \
-Dserver.host=construction-code-expert-test-fhj3jauezq-uc.a.run.app \
-Dserver.port=443 \
-Dproject.id=construction-code-expert-test \
-DFIREBASE_WEB_API_KEY=$$FIREBASE_WEB_API_KEY
env:
- 'GOOGLE_APPLICATION_CREDENTIALS=/workspace/.secrets/service-account.json'
secretEnv:
- 'FIREBASE_WEB_API_KEY'
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/firebase-web-api-key/versions/latest
env: 'FIREBASE_WEB_API_KEY'
Common Gotchas and Troubleshooting
1. Authentication Token Expiration
Problem: Tests fail with UNAUTHENTICATED status after running for a while
Solution:
- Firebase ID tokens expire after 1 hour
- Regenerate tokens for long-running test suites
- Implement token refresh logic in test setup
- Use custom tokens with longer expiration for test environments
// Check token expiration before each test
@Before
public void refreshTokenIfNeeded() throws Exception {
if (isTokenExpired(authToken)) {
setupAuthentication(); // Regenerate token
}
}
2. Message Size Limits
Problem: RESOURCE_EXHAUSTED error when uploading large files
Solution:
- Configure client-side message size:
.maxInboundMessageSize(100 * 1024 * 1024) - Verify server-side limits match client configuration
- Check ESPv2 gateway limits if using REST API transcoding
- Consider chunking for files > 100MB
channel = ManagedChannelBuilder.forAddress(serverHost, serverPort)
.usePlaintext()
.maxInboundMessageSize(100 * 1024 * 1024) // 100MB
.maxOutboundMessageSize(100 * 1024 * 1024) // 100MB
.build();
3. Firestore Access Issues
Problem: Tests fail to read/write Firestore documents
Solution:
- Verify
GOOGLE_APPLICATION_CREDENTIALSis set correctly - Ensure service account has Firestore permissions
- Check Firestore project ID matches test environment
- Use Application Default Credentials in dev container
// Initialize Firestore with explicit project ID
firestore = FirestoreOptions.newBuilder()
.setProjectId(TEST_PROJECT_ID)
.build()
.getService();
4. Test Data Dependencies
Problem: Tests fail because baseline projects don't exist
Solution:
- Document required baseline projects in test javadoc
- Create baseline projects as part of test setup
- Use project copying to create test fixtures
- Implement cleanup to remove test projects after execution
@Before
public void ensureBaselineProjectExists() throws Exception {
try {
GetArchitecturalPlanRequest request = GetArchitecturalPlanRequest.newBuilder()
.setArchitecturalPlanId("SanJose-baseline")
.build();
planService.getArchitecturalPlan(request);
} catch (StatusRuntimeException e) {
if (e.getStatus().getCode() == Status.Code.NOT_FOUND) {
fail("Baseline project 'SanJose-baseline' does not exist. Please create it first.");
}
}
}
5. Network Connectivity Issues
Problem: Tests timeout or fail to connect to gRPC server
Solution:
- Verify server host and port are correct
- Check firewall rules and network policies
- Test basic connectivity with
testServerConnectivity() - Use IPv4 explicitly if IPv6 causes issues:
-Djava.net.preferIPv4Stack=true
# Test basic connectivity first
mvn test -Dtest=ArchitecturalPlanWriteServiceGrpcIntegrationTest#testServerConnectivity \
-Dserver.host=localhost \
-Dserver.port=8080
6. Async Task Timeouts
Problem: Tests timeout waiting for background tasks to complete
Solution:
- Increase task timeout for LLM-intensive operations
- Poll Firestore more frequently for task status
- Implement exponential backoff for polling
- Add detailed logging to understand task progress
private Map<String, Object> waitForTaskCompletion(String taskId) throws Exception {
int maxAttempts = 60; // 10 minutes with 10-second intervals
int attempt = 0;
while (attempt < maxAttempts) {
DocumentSnapshot doc = firestore.collection("tasks")
.document(taskId)
.get()
.get();
if (doc.exists()) {
Map<String, Object> data = doc.getData();
String status = (String) data.get("status");
System.out.println("Task " + taskId + " status: " + status +
" (attempt " + (attempt + 1) + "/" + maxAttempts + ")");
if ("COMPLETE".equals(status) || "FAILED".equals(status)) {
return data;
}
}
Thread.sleep(10000); // Wait 10 seconds
attempt++;
}
throw new TimeoutException("Task did not complete within timeout period");
}
7. GCS File Access Issues
Problem: Tests fail to read/write files from Google Cloud Storage
Solution:
- Verify service account has Storage Object Admin role
- Check bucket names and paths are correct
- Ensure GCS bucket exists in test environment
- Use proper file naming conventions
// Validate GCS file exists after upload
Storage storage = StorageOptions.newBuilder()
.setProjectId(TEST_PROJECT_ID)
.build()
.getService();
String bucketName = "construction-code-expert-test-data";
String blobName = "projects/" + projectId + "/input/" + filename;
Blob blob = storage.get(bucketName, blobName);
assertNotNull("File should exist in GCS", blob);
assertTrue("File should have content", blob.getSize() > 0);
Test Configuration Parameters
The following system properties can be used to configure integration tests:
| Parameter | Description | Default | Example |
|---|---|---|---|
server.host | gRPC server hostname | localhost | construction-code-expert-test-fhj3jauezq-uc.a.run.app |
server.port | gRPC server port | 8080 | 443 |
project.id | GCP project ID for test environment | N/A | construction-code-expert-test |
auth.token | Firebase ID token for authentication | $CODEPROOF_DEMO_USER_FIREBASE_TOKEN | eyJhbGciOiJSUzI1... |
pdf.path | Path to PDF file for upload tests | Hardcoded path | /path/to/test.pdf |
upload.filename | Filename to use when uploading | Original filename | test-plan.pdf |
FIREBASE_WEB_API_KEY | Firebase Web API key for token exchange | N/A | AIzaSyC... |
GOOGLE_APPLICATION_CREDENTIALS | Path to service account JSON | N/A | /path/to/sa.json |
Best Practices
- Use Unique Project IDs: Generate unique test project IDs using UUID to avoid conflicts
- Clean Up Test Data: Always delete test projects in
@Aftermethods - Document Prerequisites: Clearly document required baseline projects and environment setup
- Test Isolation: Each test should be independent and not rely on other tests
- Meaningful Assertions: Use descriptive assertion messages to aid debugging
- Comprehensive Logging: Add detailed logging to understand test flow and failures
- Handle Async Operations: Implement proper waiting and polling for background tasks
- Validate End-to-End: Don't just check gRPC responses, validate actual data persistence
- Test Error Scenarios: Include negative test cases for error handling
- Use Helper Methods: Extract common patterns into reusable helper methods
Related Documentation
- Developer Playbook - General development workflows
- gRPC API Testing with grpcurl - Command-line gRPC testing
- REST API Testing - REST API testing with curl
- gRPC Playbook - gRPC development guide
- Background Tasks Architecture - Async task processing
- Firebase RBAC Integration - Authentication and authorization