Add SwiftV2 long-running pipeline with scheduled tests #4142

sivakami-projects · 2025-11-24T16:53:34Z

Pipeline to run repeated tests on long running Swiftv2 AKS clusters.

Test pipeline - Tests are scheduled to run every 3 hours on central us euap. Link to Pipeline
Recent test run

Testing Approach
Test Lifecycle (per stage):
Create 8 pod scenarios with PodNetwork, PodNetworkInstance, Pods
Run 9 connectivity tests (HTTP-based)
Run private endpoint tests (storage access)
Delete all resources (Phase 1: Pods, Phase 2: PNI/PN/Namespaces)

Node Selection:
Tests filter by workload-type=$WORKLOAD_TYPE AND nic-capacity labels
Ensures isolation between different workload type stages
Currently: WORKLOAD_TYPE=swiftv2-linux

Files Changed
Pipeline Configuration
pipeline.yaml: Main pipeline with schedule trigger
long-running-pipeline-template.yaml: Stage definitions with VM SKU constants

Setup Scripts
create_aks.sh: AKS cluster creation with node labeling
create_vnets.sh: Customer VNet creation
create_peerings.sh: VNet peering mesh
create_storage.sh: Storage accounts with public access disabled (SA1 only)
create_nsg.sh: NSG rule application with retry logic
create_pe.sh: Private endpoint and DNS zone setup

Test Code
datapath.go: Enhanced with node label filtering, private endpoint testing
datapath_create_test.go: Resource creation scenarios
datapath_connectivity_test.go: HTTP connectivity validation
datapath_private_endpoint_test.go: Private endpoint access/isolation tests
datapath_delete_test.go: Resource cleanup

Documentation
README.md:

Reason for Change:

Issue Fixed:

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
relevant PR labels added

Notes:

- Implemented scheduled pipeline running every 1 hour with persistent infrastructure - Split test execution into 2 jobs: Create (with 20min wait) and Delete - Added 8 test scenarios across 2 AKS clusters, 4 VNets, different subnets - Implemented two-phase deletion strategy to prevent PNI ReservationInUse errors - Added context timeouts on kubectl commands with force delete fallbacks - Resource naming uses RG name as BUILD_ID for uniqueness across parallel setups - Added SkipAutoDeleteTill tags to prevent automatic resource cleanup - Conditional setup stages controlled by runSetupStages parameter - Auto-generate RG name from location or allow custom names for parallel setups - Added comprehensive README with setup instructions and troubleshooting - Node selection by agentpool labels with usage tracking to prevent conflicts - Kubernetes naming compliance (RFC 1123) for all resources fix ginkgo flag. Add datapath tests. Delete old test file. Add testcases for provate endpoint. Ginkgo run specs only on specified files. update pipeline params. Add ginkgo tags Add datapath tests. Add ginkgo build tags. remove wait time. set namespace. update pod image. Add more nsg rules to block subnets s1 and s2 test change. Change delegated subnet address range. Use delegated interface for network connectivity tests. Datapath test between clusters. test. test private endpoints. fix private endpoint tests. Set storage account names in putput var. set storage account name. fix pn names. update pe update pe test. update sas token generation. Add node labels for sw2 scenario, cleanup pods on any test failure. enable nsg tests. update storage. Add rules to nsg. disable private endpoint negative test. disable public network access on storage account with private endpoint. wait for default nsg to be created. disable negative test on private endpoint. private endpoint depends on aks cluster vnets, change pipeline job dependencies. Add node labels for each workload type and nic capacity. make sku constant. Update readme, set schedule for long running cluster on test branch.

Copilot

Pull request overview

This PR adds a comprehensive long-running test pipeline for SwiftV2 pod networking on Azure AKS. The pipeline creates persistent infrastructure (2 AKS clusters, 4 VNets, storage accounts with private endpoints, NSGs) and runs scheduled tests every 3 hours to validate pod-to-pod connectivity, network security group isolation, and private endpoint access across multi-tenant scenarios.

Key Changes:

Adds scheduled pipeline with conditional infrastructure setup (runSetupStages parameter)
Implements 8 pod test scenarios across 2 clusters and 4 VNets with different NIC capacities
Includes 9 connectivity tests and 5 private endpoint tests with tenant isolation validation

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
.pipelines/swiftv2-long-running/pipeline.yaml	Main pipeline with 3-hour scheduled trigger and runSetupStages parameter
.pipelines/swiftv2-long-running/template/long-running-pipeline-template.yaml	Two-stage template: setup (conditional) and datapath tests with 4 jobs
.pipelines/swiftv2-long-running/scripts/*.sh	Infrastructure setup scripts for AKS, VNets, storage, NSGs, and private endpoints
test/integration/swiftv2/longRunningCluster/datapath*.go	Test implementation split into create, connectivity, private endpoint, and delete tests
test/integration/swiftv2/helpers/az_helpers.go	Azure CLI and kubectl helper functions for resource management
test/integration/manifests/swiftv2/long-running-cluster/*.yaml	Kubernetes resource templates for PodNetwork, PNI, and Pods
go.mod, go.sum	Updates to support Ginkgo v2 testing framework
hack/aks/Makefile	Updates for SwiftV2 cluster creation with multi-tenancy tags
.pipelines/swiftv2-long-running/README.md	Comprehensive documentation of pipeline architecture and test scenarios

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

go.mod

.pipelines/swiftv2-long-running/scripts/create_vnets.sh

.pipelines/swiftv2-long-running/scripts/create_aks.sh

.pipelines/swiftv2-long-running/scripts/create_storage.sh

test/integration/swiftv2/longRunningCluster/datapath_create_test.go

test/integration/swiftv2/longRunningCluster/datapath_delete_test.go

test/integration/swiftv2/longRunningCluster/datapath_connectivity_test.go

test/integration/swiftv2/longRunningCluster/datapath_delete_test.go

kmurudi · 2025-12-02T23:32:53Z

.pipelines/swiftv2-long-running/scripts/create_aks.sh

+    sleep_seconds=15
+    retry_count=0
+
+    while [[ $retry_count -lt $max_retries ]]; do


what is the min & max wait time for resources creation step?, does it wait 20mins max for cluster completion?

These retries are for stamp infra vnets. Cluster creation job has no timeout set.

kmurudi · 2025-12-02T23:35:10Z

test/integration/manifests/swiftv2/long-running-cluster/pod.yaml

+        echo "Pod IP: $(hostname -i)";
+        echo "Starting HTTP server on port 8080";
+        
+        # Create a simple HTTP server directory


do we need to define this index page instead of using a library image? please mention the use case if we are adding details to print

test/integration/manifests/swiftv2/long-running-cluster/podnetwork.yaml

kmurudi · 2025-12-02T23:43:02Z

test/integration/swiftv2/longRunningCluster/datapath_connectivity_test.go

+		// Format: pn-<rg>-<vnet-prefix>-<subnet-name>
+		// Example: pn-sv2-long-run-centraluseuap-a1-s1
+		getNamespace := func(vnetName, subnetName string) string {
+			// Extract vnet prefix (a1, a2, a3, b1, etc.) from cx_vnet_a1 -> a1


nit: rename vnet names to be vn1, vn2 - to give some clarity on naming of pod

kmurudi · 2025-12-02T23:44:07Z

test/integration/swiftv2/longRunningCluster/datapath_delete_test.go

+		ginkgo.Fail(fmt.Sprintf("Missing required environment variables: RG='%s', BUILD_ID='%s'", rg, buildId))
+	}
+
+	ginkgo.It("deletes PodNetwork, PodNetworkInstance, and Pods", ginkgo.NodeTimeout(0), func() {


does it check for any pending resources like MTPNC etc? sometimes mtpnc deletion gets stuck in ICMs, would be good to check

Let me add that.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

…st.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

…ity_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

…st.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

sivakami added 2 commits November 24, 2025 08:38

Update readme file.

873c05e

sivakami-projects marked this pull request as ready for review November 24, 2025 17:07

sivakami-projects requested a review from a team as a code owner November 24, 2025 17:07

sivakami-projects requested review from Copilot and ravisraju November 24, 2025 17:07

Copilot started reviewing on behalf of sivakami-projects November 24, 2025 17:07 View session

Copilot finished reviewing on behalf of sivakami-projects November 24, 2025 17:09

Copilot AI reviewed Nov 24, 2025

View reviewed changes

sivakami added 2 commits November 24, 2025 09:37

fix syntax for pe test.

3395415

Create NSG rules with unique priority.

b34b332

kmurudi reviewed Dec 2, 2025

View reviewed changes

sivakami-projects and others added 5 commits December 5, 2025 10:43

Update go.mod

e9f50e6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

Update test/integration/swiftv2/longRunningCluster/datapath_create_te…

8364bf5

…st.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

Update test/integration/swiftv2/longRunningCluster/datapath_delete_te…

1d2ed59

…st.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

Update test/integration/swiftv2/longRunningCluster/datapath_connectiv…

efbfb02

…ity_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

Update test/integration/swiftv2/longRunningCluster/datapath_delete_te…

04a22a0

…st.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add SwiftV2 long-running pipeline with scheduled tests #4142

Add SwiftV2 long-running pipeline with scheduled tests #4142

sivakami-projects commented Nov 24, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmurudi Dec 2, 2025

Uh oh!

sivakami-projects Dec 5, 2025

Uh oh!

kmurudi Dec 2, 2025

Uh oh!

Uh oh!

kmurudi Dec 2, 2025

Uh oh!

kmurudi Dec 2, 2025

Uh oh!

sivakami-projects Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add SwiftV2 long-running pipeline with scheduled tests #4142

Are you sure you want to change the base?

Add SwiftV2 long-running pipeline with scheduled tests #4142

Conversation

sivakami-projects commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmurudi Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

sivakami-projects Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

kmurudi Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kmurudi Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

kmurudi Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

sivakami-projects Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sivakami-projects commented Nov 24, 2025 •

edited

Loading