Skip to content

feat: Add support for new GCP and Azure features [COMP-1463, COMP-1464]#606

Merged
cristianrcv merged 14 commits intomasterfrom
feat/COMP-1464-add-support-for-new-gcp-and-azure-features
Apr 16, 2026
Merged

feat: Add support for new GCP and Azure features [COMP-1463, COMP-1464]#606
cristianrcv merged 14 commits intomasterfrom
feat/COMP-1464-add-support-for-new-gcp-and-azure-features

Conversation

@cristianrcv
Copy link
Copy Markdown
Contributor

@cristianrcv cristianrcv commented Apr 10, 2026

Summary GCP

Closes COMP-1463 — Make new platform features available via CLI.

  • WIF credentials: --mode=workload-identity with --service-account-email, --workload-identity-provider, --token-audience options for Google credentials (PR #10346)
  • Network tags: --network-tags (comma-separated) with --network / --subnetwork options and GCP format validation (PR #10416)
  • Machine types: --head-job-machine-type (single) and --compute-jobs-machine-type (comma-separated, wildcard support), mutually exclusive with instance templates (PRs #10488, #10532)
  • Boot disk image: --boot-disk-image with format validation for project paths, family paths, and batch short names (PR #10484)
  • Fusion Snapshots: --fusion-snapshots toggle requiring --fusion-v2 (PR #10751)

All features include CLI-side validation matching backend/frontend rules.

Summary Azure

  • Add Azure Entra (service principal) credentials support (tw credentials add azure-entra)
  • Add managed identity options for head/pool client IDs and resource IDs in Forge and Manual Azure Batch CEs
  • Add VNet/subnet support (--subnet-id) for private network isolation
  • Add dual pool mode (--dual-pool) with separate head and worker pool configuration for Forge, and --worker-pool for Manual mode
  • Add job max wall clock time (--job-max-wall-clock-time)
  • Replace JobCleanupPolicy enum with 3 boolean toggles: --delete-jobs-on-completion, --delete-tasks-on-completion, --terminate-jobs-on-completion

Note: Requires tower-java-sdk bump to >= 1.133.0 for new model fields. Code will not compile until the SDK is updated.

Platform PRs implemented

  • #10519 — Entra credentials for Azure Batch Forge and Fusion v2
  • #10577 — Separate Azure managed identities for head and compute jobs
  • #10541 — Private VNet/subnet support for Azure Batch CEs
  • #10636 — Separate head and worker pools for Azure Batch
  • #10708 — jobMaxWallClockTime for Azure Batch CEs
  • #10681 — Replace JobCleanupPolicy enum with boolean toggles

ramonamela and others added 7 commits April 10, 2026 12:14
- WIF (Workload Identity Federation) support for Google credentials
  with --mode=workload-identity, --service-account-email,
  --workload-identity-provider, and --token-audience options
- Network tags (--network-tags) with VPC requirement and GCP format
  validation, plus --network and --subnetwork options
- Machine type selection: --head-job-machine-type (single) and
  --compute-jobs-machine-type (comma-separated list with wildcard
  support), mutually exclusive with instance templates
- Boot disk image (--boot-disk-image) with format validation for
  projects/*/global/images/*, family paths, and batch-* short names
- Fusion Snapshots (--fusion-snapshots) toggle requiring Fusion v2

All features include CLI validation matching backend/frontend rules.
Code will compile once tower-java-sdk is bumped to include the new
fields on GoogleSecurityKeys and GoogleBatchConfig.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Upgrade tower-java-sdk from 1.114.0 to 1.133.0 to include new fields
  on GoogleSecurityKeys and GoogleBatchConfig
- Add bash integration test script for all COMP-1463 features

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Azure Entra (service principal) credentials support (add/update)
- Add managed identity options (head/pool client IDs and resource IDs)
  for both Forge and Manual Azure Batch compute environments
- Add VNet/subnet support (--subnet-id) for private network isolation
- Add dual pool mode (--dual-pool) with separate head and worker pool
  configuration for Forge mode, and --worker-pool for Manual mode
- Add job max wall clock time option (--job-max-wall-clock-time)
- Replace JobCleanupPolicy enum with 3 boolean toggles:
  --delete-jobs-on-completion, --delete-tasks-on-completion,
  --terminate-jobs-on-completion

Note: requires tower-java-sdk bump to >= 1.133.0 for new model fields.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Bump tower-java-sdk from 1.114.0 to 1.133.0 for new Azure Batch
  model fields (managed identity, subnet, dual pool, wall clock time,
  cleanup toggles)
- Add bash integration test script for all new Azure Batch CLI features

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…DK 1.133.0 bump

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix AzBatchConfig.deleteJobsOnCompletion mock response: changed from
  string "on_success" to boolean true after field type change in SDK
- Switch request body matchers from exact string to json() partial
  matching for pipeline and label mocks to handle new serialized fields
  in WorkflowLaunchRequest, AssociatePipelineLabelsRequest, and
  CreateLabelRequest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eflection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@cristianrcv cristianrcv force-pushed the feat/COMP-1464-add-support-for-new-gcp-and-azure-features branch from fda15a9 to 07e1f8b Compare April 10, 2026 10:14
@cristianrcv cristianrcv requested review from a team and jonmarti April 10, 2026 10:22
…aalVM reflection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ramonamela and others added 6 commits April 13, 2026 18:04
Simplify test coverage to Forge flows, fix ws_flag ordering so it comes after the subcommand, auto-fill /subnets/default when SUBNET_ID is a VNet ID, and extract cleanup into a standalone clean-azure-batch-test-resources.sh that wipes all test-* compute envs and credentials.

[COMP-1464] [test]
… GraalVM reflection

Register AzBatchForgePlatform inner classes (HeadPoolOptions, WorkerPoolOptions, ManagedIdentityOptions), AzBatchManualPlatform.ManagedIdentityOptions, Azure Entra credential commands/providers, ManageCmd for pipeline versions, plus additional SDK 1.133.0 model classes (Role DTOs, GitHubAppSecurityKeys, DataStudio/Dataset update requests, AzBatchPoolConfig, ValidateUrl) exposed by the native Azure Batch integration tests.

[COMP-1464] [graalvm]
…ed-identity-head-client-id

Align the head-job managed identity client ID flag with its resource-id counterpart so all head/pool option pairs share the same naming scheme (--managed-identity-{head,pool}-{client-id,resource-id}). Applies to both forge and manual AzBatch platforms; AzCloud is left untouched because it has no head/pool split.

Addresses Stefano's review comment on PR #606.

[COMP-1464]
…DK field

Reverting 459650f. The CLI should mirror the API field names; the SDK model AzBatchConfig has managedIdentityClientId (head) alongside managedIdentityHeadResourceId, managedIdentityPoolClientId, managedIdentityPoolResourceId. Renaming the CLI flag to add a 'head' prefix broke that 1:1 mapping. Added a note on the Option description acknowledging the asymmetry is inherited from the SDK.

Audited all other COMP-1464 flags against the tower-java-sdk 1.133.0 AzBatchConfig, AzBatchForgeConfig, AzBatchPoolConfig and AzureEntraKeys models — remaining flags already match their API counterparts (only inverted-boolean idioms like --no-auto-scale / --preserve-resources differ, and those predate COMP-1464).

[COMP-1464]
…ed-identity-head-client-id

Restoring the rename originally applied in 459650f (reverted in d059503). Per reviewer discussion on PR #606, the CLI-side head/pool symmetry (--managed-identity-{head,pool}-{client-id,resource-id}) is clearer for users than strictly mirroring the SDK's asymmetric field names, so prefer the consistent CLI naming even though the underlying API keeps managedIdentityClientId for the head job.

[COMP-1464]
…ror messages

IllegalArgumentException was not handled by the CLI error handler, causing
full stack traces to be printed for validation errors. Switching to
TowerRuntimeException produces clean one-line ERROR messages instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jonmarti
Copy link
Copy Markdown
Contributor

GCP QA Test Results (2026-04-15)

Target: api.cloud.dev-seqera.io (24.2.0-auth0-RC1) | Workspace: unified-compute / new-ce-forms | CLI: 0.25.0 (branch feat/COMP-1464-add-support-for-new-gcp-and-azure-features)

WIF Credentials

# Test Expected Result Error Message
1 Create WIF credential (basic) Success PASS
2 Create WIF credential (with --token-audience) Success PASS
3 WIF missing --service-account-email Fail PASS Option '--service-account-email' is required when using '--mode=workload-identity'.
4 WIF missing --workload-identity-provider Fail PASS Option '--workload-identity-provider' is required when using '--mode=workload-identity'.
5 WIF invalid email format Fail PASS Invalid service account email format. Expected format: <name>@<project>.iam.gserviceaccount.com
6 WIF invalid provider format Fail PASS Invalid Workload Identity Provider format. Expected: projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{POOL}/providers/{PROVIDER}
7 WIF mode with --key (mutually exclusive) Fail PASS Option '--key' cannot be used with '--mode=workload-identity'. Workload Identity mode uses federated authentication without a key file.
8 SA key mode with WIF options Fail PASS Options '--service-account-email', '--workload-identity-provider', and '--token-audience' can only be used with '--mode=workload-identity'.
9 Invalid --mode value Fail PASS Invalid Google credential mode 'invalid-mode'. Allowed values: 'service-account-key', 'workload-identity'.
10 Default SA key mode without --key Fail PASS Option '--key' is required when using service account key mode.

Google Batch Compute Environment

# Test Expected Result Error Message
11 Network tags without --network Fail PASS Network tags require VPC configuration: set the '--network' option to use network tags.
12 Invalid network tag (uppercase) Fail PASS Invalid network tag 'InvalidTag': must start with a lowercase letter, end with a letter or number, and contain only lowercase letters, numbers, and hyphens.
13 Invalid network tag (trailing hyphen) Fail PASS Invalid network tag 'bad-tag-': must start with a lowercase letter, end with a letter or number, and contain only lowercase letters, numbers, and hyphens.
14 Wildcard on head job machine type Fail PASS Wildcard machine type families are not supported for the head job -- select a specific machine type instead.
15 Head machine type + template (mutually exclusive) Fail PASS Head job machine type and head job instance template are mutually exclusive -- specify only one.
16 Compute machine type + template (mutually exclusive) Fail PASS Compute jobs machine type and compute jobs instance template are mutually exclusive -- specify only one.
17 Invalid boot disk image format Fail PASS Invalid boot disk image format. Use projects/{PROJECT}/global/images/{IMAGE}, projects/{PROJECT}/global/images/family/{FAMILY}, or a Batch image name (e.g., batch-debian).
18 Fusion snapshots without --fusion-v2 Fail PASS Fusion Snapshots requires Fusion v2 to be enabled (--fusion-v2).
19 Full CE: network + machine types + boot image Success PASS
20 Full CE: fusion-v2 + snapshots + wave Success PASS

Bug Fix

During QA, validation errors were printing full Java stack traces instead of clean error messages. Fixed in 6b4db1f by replacing IllegalArgumentException with TowerRuntimeException in GoogleProvider, GoogleBatchPlatform,
and AwsProvider. Tests 3-5 were re-run to confirm clean output.

@cristianrcv cristianrcv merged commit 8687b32 into master Apr 16, 2026
11 checks passed
georgi-seqera added a commit that referenced this pull request Apr 17, 2026
* feat: add new Google Batch CE and credential features [COMP-1463]

- WIF (Workload Identity Federation) support for Google credentials
  with --mode=workload-identity, --service-account-email,
  --workload-identity-provider, and --token-audience options
- Network tags (--network-tags) with VPC requirement and GCP format
  validation, plus --network and --subnetwork options
- Machine type selection: --head-job-machine-type (single) and
  --compute-jobs-machine-type (comma-separated list with wildcard
  support), mutually exclusive with instance templates
- Boot disk image (--boot-disk-image) with format validation for
  projects/*/global/images/*, family paths, and batch-* short names
- Fusion Snapshots (--fusion-snapshots) toggle requiring Fusion v2

All features include CLI validation matching backend/frontend rules.
Code will compile once tower-java-sdk is bumped to include the new
fields on GoogleSecurityKeys and GoogleBatchConfig.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump tower-java-sdk to 1.133.0 and add integration test script

- Upgrade tower-java-sdk from 1.114.0 to 1.133.0 to include new fields
  on GoogleSecurityKeys and GoogleBatchConfig
- Add bash integration test script for all COMP-1463 features

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: update CLI to support new Azure Batch features [COMP-1464]

- Add Azure Entra (service principal) credentials support (add/update)
- Add managed identity options (head/pool client IDs and resource IDs)
  for both Forge and Manual Azure Batch compute environments
- Add VNet/subnet support (--subnet-id) for private network isolation
- Add dual pool mode (--dual-pool) with separate head and worker pool
  configuration for Forge mode, and --worker-pool for Manual mode
- Add job max wall clock time option (--job-max-wall-clock-time)
- Replace JobCleanupPolicy enum with 3 boolean toggles:
  --delete-jobs-on-completion, --delete-tasks-on-completion,
  --terminate-jobs-on-completion

Note: requires tower-java-sdk bump to >= 1.133.0 for new model fields.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: bump tower-java-sdk to 1.133.0 and add integration test script

- Bump tower-java-sdk from 1.114.0 to 1.133.0 for new Azure Batch
  model fields (managed identity, subnet, dual pool, wall clock time,
  cleanup toggles)
- Add bash integration test script for all new Azure Batch CLI features

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: pass resolveSymlink param to generateDownloadUrlDataLink after SDK 1.133.0 bump

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update tests for SDK 1.133.0 serialization changes

- Fix AzBatchConfig.deleteJobsOnCompletion mock response: changed from
  string "on_success" to boolean true after field type change in SDK
- Switch request body matchers from exact string to json() partial
  matching for pipeline and label mocks to handle new serialized fields
  in WorkflowLaunchRequest, AssociatePipelineLabelsRequest, and
  CreateLabelRequest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: register new SDK 1.133.0 model methods and classes for GraalVM reflection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: register DataStudioStartRequest name JsonNullable methods for GraalVM reflection

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(azure-batch): refine Azure Batch integration test script

Simplify test coverage to Forge flows, fix ws_flag ordering so it comes after the subcommand, auto-fill /subnets/default when SUBNET_ID is a VNet ID, and extract cleanup into a standalone clean-azure-batch-test-resources.sh that wipes all test-* compute envs and credentials.

[COMP-1464] [test]

* fix: register Azure Batch forge options and additional SDK models for GraalVM reflection

Register AzBatchForgePlatform inner classes (HeadPoolOptions, WorkerPoolOptions, ManagedIdentityOptions), AzBatchManualPlatform.ManagedIdentityOptions, Azure Entra credential commands/providers, ManageCmd for pipeline versions, plus additional SDK 1.133.0 model classes (Role DTOs, GitHubAppSecurityKeys, DataStudio/Dataset update requests, AzBatchPoolConfig, ValidateUrl) exposed by the native Azure Batch integration tests.

[COMP-1464] [graalvm]

* refactor(azure-batch): rename --managed-identity-client-id to --managed-identity-head-client-id

Align the head-job managed identity client ID flag with its resource-id counterpart so all head/pool option pairs share the same naming scheme (--managed-identity-{head,pool}-{client-id,resource-id}). Applies to both forge and manual AzBatch platforms; AzCloud is left untouched because it has no head/pool split.

Addresses Stefano's review comment on PR #606.

[COMP-1464]

* revert(azure-batch): restore --managed-identity-client-id to mirror SDK field

Reverting 459650f. The CLI should mirror the API field names; the SDK model AzBatchConfig has managedIdentityClientId (head) alongside managedIdentityHeadResourceId, managedIdentityPoolClientId, managedIdentityPoolResourceId. Renaming the CLI flag to add a 'head' prefix broke that 1:1 mapping. Added a note on the Option description acknowledging the asymmetry is inherited from the SDK.

Audited all other COMP-1464 flags against the tower-java-sdk 1.133.0 AzBatchConfig, AzBatchForgeConfig, AzBatchPoolConfig and AzureEntraKeys models — remaining flags already match their API counterparts (only inverted-boolean idioms like --no-auto-scale / --preserve-resources differ, and those predate COMP-1464).

[COMP-1464]

* refactor(azure-batch): rename --managed-identity-client-id to --managed-identity-head-client-id

Restoring the rename originally applied in 459650f (reverted in d059503). Per reviewer discussion on PR #606, the CLI-side head/pool symmetry (--managed-identity-{head,pool}-{client-id,resource-id}) is clearer for users than strictly mirroring the SDK's asymmetric field names, so prefer the consistent CLI naming even though the underlying API keeps managedIdentityClientId for the head job.

[COMP-1464]

* feat: add a command to update studio details

---------

Co-authored-by: ramonamela <25862624+ramonamela@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Cristian Ramon-Cortes <cristian.ramon-cortes@seqera.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants