Bootstrapping the SDAF Control Plane & Deployer VM Tooling

Part 2: Hardening the Execution Runner, Dynamic UAMI Governance, and Zero-Exposure Bastion Operations

---

1. The Role of the Control Plane: Beyond "Run from My Laptop"

In small-scale cloud proof-of-concepts, engineers frequently run Terraform commands and Ansible playbooks directly from their local laptops or through generic CI/CD runners. While this approach suffices for stateless web services or short-lived microservices, it is an anti-pattern for enterprise SAP deployments.

An automated SAP S/4HANA 2022 deployment is an intensive, stateful, and long-running orchestration:

• Massive Media Staging: Extracting, verifying, and staging over 100 GB of SAP installation media—including the SAP Software Provisioning Manager (SWPM/SAPinst), SAP Kernel archives, SAP HANA Database 2.0 binaries, and S/4HANA export dumps.
• Unbroken Multi-Hour Execution: Executing an 8-stage unattended configuration and software installation sequence requiring 4 to 6 hours of unbroken network communication between the orchestrator and target virtual machines.
• Strict Concurrency & State Locking: Managing concurrent Terraform remote state leases and locks across multiple decoupled layers (Control Plane, Workload Zone, SAP System, and Shared Storage).

If an engineer's laptop drops Wi-Fi, sleeps, or re-negotiates a VPN handshake midway through an unattended hdblcm HANA database installation or SWPM database load, the deployment state becomes corrupted, leaving orphan Azure resources, broken state locks, and half-configured SAP database tables.

Figure 1: Architectural blueprint of the SDAF Control Plane, dynamic UAMI role assignment, in-guest real-time telemetry, and zero-exposure Bastion tunneling.

To solve this, Microsoft's SAP Deployment Automation Framework (SDAF) introduces a dedicated, isolated execution tier: The Control Plane.

The Control Plane decouples the automation engine from the target SAP workload through two primary building blocks:

1. The Deployer VM: A hardened Linux execution runner provisioned in its own dedicated virtual network (10.x.100.0/25). It executes Terraform IaC, runs Ansible playbooks, and mediates all private management operations over private IP routing.
2. The SAP Library: A centralized foundation comprising Azure Storage accounts and an Azure Key Vault. It persists Terraform remote state files (.terraform.tfstate), hosts the multi-gigabyte SAP media library container (sapbits), and safeguards automation credentials without exposing them to local developer machines.

---

2. A Cloud & Security Primer: Core Microsoft Azure Concepts for SAP Engineers

For engineers transitioning from traditional on-premises SAP Basis administration or non-Azure cloud platforms, the terminology surrounding Azure identity, resource governance, and networking can be confusing.

Before diving into the implementation details, let us break down the five foundational Azure concepts that govern how our Control Plane operates:

2.1 Azure Resource Manager (ARM)

Azure Resource Manager (ARM) is the centralized management and control layer for Microsoft Azure. Every time you create a virtual machine, modify a network security group, or execute a Terraform plan using the azurerm provider, your request is submitted as an authenticated JSON payload to the ARM REST API endpoint (management.azure.com).

ARM serves three essential purposes:

• Authentication & Validation: Verifies the identity of the requester and confirms they have adequate permissions.
• Resource Providers: Dispatches tasks to specialized resource engines (such as Microsoft.Compute for VMs, Microsoft.Network for VNets and NSGs, Microsoft.Storage for storage accounts, and Microsoft.KeyVault for secrets).
• Declarative Resource Management: Ensures resources are tracked with unified Azure Resource IDs (e.g., /subscriptions/<id>/resourceGroups/<rg>/providers/...), consistent tagging, and coordinated lifecycle management.

2.2 Service Principals (SPN)

In Microsoft Entra ID (formerly Azure Active Directory), human users authenticate using usernames, passwords, and multi-factor authentication (MFA). Automated scripts and CI/CD pipelines, however, cannot perform interactive MFA prompts.

To enable automation, Azure uses an Application Registration and its corresponding Service Principal (often called SPN - Service Principal Name):

• A Service Principal is essentially a "service account" identity in your Entra ID tenant.
• It authenticates non-interactively using an Application (Client) ID, a Directory (Tenant) ID, and an authentication credential—typically a Client Secret (a generated cryptographic password) or a Certificate.

[!WARNING] The Operational Risk of Service Principals in Automation:
In enterprise environments, hardcoded Service Principal client secrets represent a significant operational risk. Client secrets typically expire after 1 to 2 years, leading to unexpected pipeline outages when they silently lapse. Furthermore, engineers frequently make the mistake of leaving credentials in configuration files or shell scripts, risking credential leaks.

2.3 Managed Identities (MSI): SAMI vs. UAMI

To eliminate the need for static client secrets altogether, Microsoft introduced Managed Identities for Azure Resources (MSI).

With a Managed Identity, Azure automatically handles identity registration, token requests, and cryptographic rotation in Entra ID behind the scenes. The application or script running on the VM simply requests an OAuth 2.0 token from the local Azure Instance Metadata Service (IMDS) at http://169.254.169.254/metadata/identity/oauth2/token. No password or secret ever exists on disk or in .tfvars files.

There are two distinct types of Managed Identities:

1. System-Assigned Managed Identity (SAMI):
2. Enabled directly on an individual Azure resource (e.g., a specific VM).
3. Tied directly to the resource lifecycle: If the VM is deleted, recreated, or re-imaged, Azure automatically deletes the identity from Entra ID along with all associated role assignments.
4. User-Assigned Managed Identity (UAMI):
5. Created as an independent, standalone Azure resource in a resource group.
6. Independent lifecycle: It can be assigned to one or more virtual machines. If a VM is stopped, decommissioned, or completely rebuilt, the UAMI remains intact, preserving its Entra ID Object ID and all previously granted permissions.

Because automation runners occasionally need to be rebuilt or updated without having security teams re-approve subscription-level permissions, SDAF uses a User-Assigned Managed Identity (SDAF-DEV-UAMI).

2.4 Azure Role-Based Access Control (Azure RBAC) vs. Key Vault Access Policies

Authorization in Azure determines what an authenticated identity is permitted to do. Azure provides two distinct authorization mechanisms:

Azure RBAC

Azure RBAC is an enterprise-grade authorization engine built on three components:

• Security Principal: Who is requesting access (User, Group, Service Principal, or Managed Identity).
• Role Definition: What operations are permitted (e.g., Contributor, Reader, Storage Blob Data Contributor, Key Vault Secrets Officer).
• Scope: Where the permissions apply. Azure scopes form an inheritance hierarchy: $$\text{Management Group} \longrightarrow \text{Subscription} \longrightarrow \text{Resource Group} \longrightarrow \text{Resource}$$

Key Vault Access Policies (Legacy Mode)

Historically, Azure Key Vault used an internal, vault-specific authorization model called Access Policies. Instead of using Azure RBAC role assignments, an Access Policy lists specific secret, key, or certificate permissions directly on the Key Vault resource itself. As we will explore in Section 4, mixing Access Policies with modern Managed Identities in Terraform creates a subtle collision trap.

2.5 Azure Bastion and Native Client Tunneling

Traditionally, accessing private Linux and Windows VMs in Azure required deploying an internet-facing "Jumpbox" VM with a Public IP, exposing SSH (port 22) or RDP (port 3389) through Network Security Groups. This created public attack surfaces, required constant operating system patching, and introduced third-party jumpbox licensing costs.

Azure Bastion is a fully managed Platform-as-a-Service (PaaS) offering provisioned inside a dedicated subnet named AzureBastionSubnet (/26 or larger). Bastion provides secure administrative access directly over encrypted TLS (HTTPS port 443) without assigning public IPs to any workload VMs.

Furthermore, with Azure Bastion Native Client Tunneling, operators can use the Azure CLI (az network bastion tunnel) on their local workstations to open secure, local loopback tunnels directly to private VM endpoints. This enables native tools—including SSH clients and the SAP GUI (Port 3200)—to communicate securely with private VMs as if they were local ports on localhost.

---

3. Deployer Bootstrap Mechanics: The OS Pivot from SLES to Ubuntu

When our team prepared the automation architecture, our initial design called for provisioning the Deployer VM on SUSE Linux Enterprise Server (SLES) 15 SP5.

Why SLES Was Chosen Initially: Alignment with SAP Target VMs

In enterprise SAP architecture, operating system choice is heavily governed by strict vendor certification standards. Both SAP HANA 2.0 SPS07 and SAP S/4HANA (NetWeaver ABAP application servers) require specialized Linux distributions certified under SAP Notes:

• SAP Note 2684254: SAP HANA DB: Recommended OS settings for SLES 15 / SLES for SAP Applications 15
• SAP Note 2578899: SUSE Linux Enterprise Server 15: Installation notes

Our target workload virtual machines—the HANA database server (teldevhana on Standard_E64s_v5) and the application server (erpdevapp1 on Standard_D8ads_v5)—were mandated to run on SLES 15 SP5 for SAP Applications.

Naturally, our engineering team sought architectural uniformity:

• Single-Vendor Synergy: Standardizing on SLES across the entire landscape meant using one unified Linux distribution across both the Control Plane and Workload Plane.
• Licensing Uniformity: Taking advantage of SUSE Bring-Your-Own-Subscription (BYOS) licensing and Azure Hybrid Benefit across all virtual machines.
• Operational Simplicity: Allowing our SAP Basis and Linux administrators to maintain one set of security baselines, kernel configurations, and patching procedures.

The Problem: When Enterprise Stability Clashes with Modern DevOps Tooling

While SLES 15 SP5 is an outstanding enterprise platform for running SAP HANA in-memory databases, it proved problematic as an automation execution runner.

During our early pilot runs (DEPLOYER_REBUILD_PLAN.md), critical friction points emerged:

1. Ansible Core Version Divergence:
   Upstream Microsoft SDAF playbooks are developed against modern Ansible Core 2.16+, leveraging advanced collection resolution, optimized loop constructs, and Python 3.10+ semantics. SLES 15 SP5’s enterprise software repositories packaged Ansible Core 2.12, which failed to resolve SDAF’s dynamic collection dependencies in /opt/ansible/collections.

2. Python Packaging & Wheel Incompatibilities:
   Auxiliary operational tooling—such as Azure SDK libraries, automated download monitors, and Model Context Protocol (MCP) servers—relied on standard Python wheels. Compiling these on SLES required adding non-standard development repositories, creating package dependency conflicts with core SUSE system packages.

3. PowerShell Core Availability:
   Running automated pre-flight quality check suites (QualityCheck.ps1) required Microsoft PowerShell 7 Core (pwsh) and the Azure Az module. On SLES, installing PowerShell Core required out-of-repo RPM configurations that complicated repeatable automation.

The Architectural Rebuild: Standardizing on Ubuntu 22.04 LTS Gen2

We recognized that the Control Plane (Deployer) and the Workload Plane (SAP VMs) have fundamentally different operational roles:

• The SAP VMs are transaction-processing application servers that require SAP-certified operating systems (SLES 15 SP5).
• The Deployer VM is a DevOps execution runner whose primary job is executing Terraform, Ansible, and Python scripts with zero friction.

To establish complete alignment with Microsoft's primary upstream engineering baselines, we executed an automated migration plan transitioning the deployer to Ubuntu Server 22.04 LTS Gen2.

We scripted the automated VM rebuild via deploy_ubuntu_deployer.sh:

#!/bin/bash
# deploy_ubuntu_deployer.sh - Provisioning SDAF Ubuntu 22.04 Deployer VM
set -e

# Target configuration
SUBSCRIPTION="<your-subscription-id>"
RESOURCE_GROUP="rg-sap-control-dev"
LOCATION="indonesiacentral"
VM_NAME="sdaf-deployer-ubuntu"
VM_SIZE="Standard_D4s_v5"   # 4 vCPU, 16 GB RAM
ADMIN_USER="azureadm"
IMAGE="Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest"
OS_DISK_SIZE=128

echo "Setting active Azure subscription context..."
az account set --subscription "$SUBSCRIPTION"

echo "Deploying Ubuntu 22.04 LTS Deployer VM: ${VM_NAME}..."
az vm create \
  --resource-group "$RESOURCE_GROUP" \
  --name "$VM_NAME" \
  --location "$LOCATION" \
  --size "$VM_SIZE" \
  --image "$IMAGE" \
  --admin-username "$ADMIN_USER" \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --vnet-name "vnet-sap-control-dev" \
  --subnet "DeployerSubnet" \
  --nsg "nsg-sap-deployer-dev" \
  --os-disk-size-gb $OS_DISK_SIZE \
  --os-disk-name "${VM_NAME}-osdisk" \
  --storage-sku Premium_LRS \
  --tags \
    Environment=Dev \
    Role=Deployer \
    Framework=SDAF \
    OS=Ubuntu-22.04 \
    Deployment=AutomatedSAPDeployment

echo "✓ Ubuntu Deployer VM successfully provisioned."

Idempotent Deployer Runtime Configuration

Following VM provisioning, we bootstrapped the deployer runtime using Microsoft’s automated configuration script (configure_deployer.sh):

# Executed in-guest on the Deployer VM
mkdir -p ~/Azure_SAP_Automated_Deployment && cd $_
wget https://raw.githubusercontent.com/Azure/sap-automation/main/deploy/scripts/configure_deployer.sh
chmod +x configure_deployer.sh
./configure_deployer.sh

This installation script establishes a standardized, idempotent execution environment:

• Terraform Core: Installs Terraform 1.6+, supporting robust AzureRM provider state locking and cross-subscription resource management.
• Ansible Core: Installs Ansible Core 2.16.x using system Python 3.10+, placing official SDAF collections under /opt/ansible/collections.
• Azure CLI & PowerShell Core: Installs Azure CLI 2.50+ alongside PowerShell 7 (pwsh), providing native support for both shell scripts and PowerShell Az cmdlets.
• Persistent Profile Environment: Writes /etc/profile.d/deploy_server.sh, ensuring that every interactive terminal session or background cron daemon automatically sources correct environment paths, Ansible collection directories, and Azure subscription variables.

---

4. Authentication Architecture: UAMI Governance & The Key Vault Access Policy Trap

Enterprise security architecture forbids storing plaintext passwords or long-lived Service Principal secrets in deployment files. In our implementation, we assigned our User-Assigned Managed Identity (SDAF-DEV-UAMI) to the Deployer VM, eliminating hardcoded passwords.

However, during iterative testing, we encountered a severe Terraform state lifecycle issue: The Key Vault Access Policy Trap.

Anatomy of the Collision: How local.run_as_msi Triggers a Lockout

In SDAF’s upstream Terraform modules (sap_deployer/role_assignments.tf), the framework evaluates whether Terraform is executing under a Managed Identity or an external Service Principal using internal logic:

# Internal snippet from sap-automation/deploy/terraform/terraform-units/modules/sap_deployer/role_assignments.tf
locals {
  run_as_msi = length(var.deployer.user_assigned_identity_id) == 0 ? (
    var.bootstrap || var.options.use_spn ? false :
      azurerm_user_assigned_identity.deployer[0].principal_id == data.azurerm_client_config.current.object_id
  ) : (
    data.azurerm_user_assigned_identity.deployer[0].principal_id == data.azurerm_client_config.current.object_id
  )
}

How the Failure Manifests:

1. The Bootstrap Phase:
   During initial bootstrap (var.bootstrap = true), Terraform sets local.run_as_msi = false. It creates the Deployer Key Vault (azurerm_key_vault.kv_user) and provisions an inline access policy resource named kv_user_pre_deployer. This policy grants the executing identity write permissions to upload newly generated SSH private keys and master deployment passwords into the Key Vault.

2. The Subsequent Run Phase:
   When an engineer runs Terraform again (for example, applying the Workload Zone or deploying the SAP System), var.bootstrap is false. Because the Deployer VM is now running with the UAMI assigned, Terraform compares the executing principal ID with the UAMI principal ID, evaluating local.run_as_msi = true.

3. The Silent Erasure:
   Under Access Policy mode, Terraform assumes that kv_user_pre_deployer was only a temporary bootstrap policy. Terraform issues an execution plan that destroys the access policy: ```text # azurerm_key_vault_access_policy.kv_user_pre_deployer will be destroyed

4. resource "azurerm_key_vault_access_policy" "kv_user_pre_deployer" {

   • certificate_permissions = []
   • key_permissions = []
   • secret_permissions = ["Get", "List", "Set", "Delete", "Purge"]
   • tenant_id = ""
   • object_id = "" } ```

5. The Immediate 403 Lockout:
   The moment this access policy is stripped, subsequent module steps lose permissions to read or write Key Vault secrets. The deployment halts with: text Error: Key Vault Secret "DEV-IDCE-DEP01-sshkey": Status=403 Code="Forbidden" Message="The user, group or application does not have secrets get permission on key vault"

The Solution: The spn_id Bridge and Dynamic RBAC Governance

To permanently resolve this collision without resorting to insecure subscription-level owner permissions, we implemented a two-part architectural fix:

Part 1: Populating spn_id in .tfvars

In both DEV-IDCE-DEP01-INFRASTRUCTURE.tfvars and DEV-IDCE-SAP_LIBRARY.tfvars, we explicitly populated the spn_id parameter with the Client ID of our Managed Identity:

# WORKSPACES/DEPLOYER/DEV-IDCE-DEP01-INFRASTRUCTURE/DEV-IDCE-DEP01-INFRASTRUCTURE.tfvars
# ----------------------------------------------------------------------------
# Identity & Key Vault Authorization Configuration
# ----------------------------------------------------------------------------
enable_rbac_authorization          = true
enable_purge_control_for_keyvaults = false # Allows fast purge in test/dev cycles

# The SPN ID Bridge: Setting spn_id to the UAMI Client ID locks kv_user_pre_deployer
# This signals to Terraform that the identity is persistent, preventing policy destruction
spn_id = "<uami-client-id>"

By explicitly setting spn_id = "<uami-client-id>", the Terraform module's access policy conditions continue to match, preserving the execution identity's policy binding across iterative updates.

Part 2: Granular Least-Privilege Role Provisioning (grant_uami_permissions_dynamic.sh)

While Key Vault Access Policies handle secret CRUD within the vault, Azure Resource Manager operations require RBAC permissions at the resource level.

Rather than granting wide, risky Contributor permissions across the entire Azure subscription, we authored grant_uami_permissions_dynamic.sh. This script dynamically inspects the deployment resource groups to identify newly created Key Vaults and Storage Accounts, assigning granular roles directly at the resource scope:

#!/bin/bash
# grant_uami_permissions_dynamic.sh - Dynamic Least-Privilege RBAC Provisioning
set -e

UAMI_ID="<uami-principal-id>"
SUBSCRIPTION_ID="<your-subscription-id>"
DEPLOYER_RG="DEV-IDCE-DEP01-INFRASTRUCTURE"
LIBRARY_RG="DEV-IDCE-SAP_LIBRARY"

echo "=== Step 1: Detect Deployer Key Vault & Grant Secrets Officer ==="
DEPLOYER_KV=$(az keyvault list --resource-group "$DEPLOYER_RG" --query "[0].name" -o tsv)
az role assignment create \
  --assignee "$UAMI_ID" \
  --role "Key Vault Secrets Officer" \
  --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$DEPLOYER_RG/providers/Microsoft.KeyVault/vaults/$DEPLOYER_KV" \
  2>&1 | grep -v "already exists" || true
echo "  ✓ Key Vault Secrets Officer granted on: $DEPLOYER_KV"

echo "=== Step 2: Detect SAP Media Storage & Grant Blob Data Contributor ==="
SAPLIB_SA=$(az storage account list --resource-group "$LIBRARY_RG" --query "[?contains(name, 'saplib')].name" -o tsv | head -1)
az role assignment create \
  --assignee "$UAMI_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$LIBRARY_RG/providers/Microsoft.Storage/storageAccounts/$SAPLIB_SA" \
  2>&1 | grep -v "already exists" || true
echo "  ✓ Storage Blob Data Contributor granted on: $SAPLIB_SA"

echo "=== Step 3: Detect Terraform State Storage & Grant Blob Data Contributor ==="
TFSTATE_SA=$(az storage account list --resource-group "$LIBRARY_RG" --query "[?contains(name, 'tfstate')].name" -o tsv | head -1)
az role assignment create \
  --assignee "$UAMI_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$LIBRARY_RG/providers/Microsoft.Storage/storageAccounts/$TFSTATE_SA" \
  2>&1 | grep -v "already exists" || true
echo "  ✓ Storage Blob Data Contributor granted on: $TFSTATE_SA"

echo "✓ Least-privilege UAMI RBAC assignments completed successfully."

This dynamic approach offers two significant advantages:

1. Resilience to Random Naming: SDAF generates randomized 3-character suffixes for storage accounts and Key Vaults (e.g., devidcesaplibb6f). Dynamic querying ensures scripts never fail due to naming drift.
2. Audit Compliance: Access is restricted strictly to the required storage containers and Key Vaults, meeting enterprise zero-trust governance criteria.

---

5. Deployer VM Operational Toolset: Telemetry & In-Guest Auditing

Operating a headless cloud deployment presents significant visibility hurdles. When an Ansible playbook is orchestrating a 4-hour SAP database import or pulling 45 GB of installation archives from Azure Blob storage, relying on periodic SSH logins and manual tail -f log inspection is inefficient and error-prone.

To provide continuous visibility and programmatic auditing, our team engineered two custom operational utilities:

5.1 Real-Time In-Guest Web Monitor (ansible_monitor)

We built a lightweight, self-contained Python Flask service (ansible_monitor/app.py) running as a systemd service on Deployer port 5000.

The service exposes an operational dashboard that tracks:

• Active Process Tracking: Continuously parses /proc and ps aux to detect running ansible-playbook and terraform processes, extracting process IDs (PIDs), start times, and execution durations converted to the local team timezone (GMT+7).
• SAP Media Download Velocity Gauge: Uses the Azure Storage Python SDK and the VM's Managed Identity to monitor blob byte transfers inside the sapbits container against the expected 44.5 GB target (111 files), reporting transfer speed and completion percentage.
• Playbook Stage Progression: Scans log files in ~/ansible/dev/logs/ for milestone completion banners, indicating whether the run is in OS configuration, HANA installation, Central Services setup, or database load.

# Extract from deployer_artifacts/ansible_monitor/app.py
@app.route('/api/status')
def get_status():
    running_processes = get_ansible_processes()
    media_download_status = check_blob_storage_progress(
        storage_account=SAP_STORAGE_ACCOUNT,
        container=SAP_CONTAINER,
        expected_gb=44.5
    )
    latest_logs = glob.glob('/home/azureadm/ansible/dev/logs/*.log')
    return jsonify({
        'processes': running_processes,
        'download_progress': media_download_status,
        'latest_log_update': get_latest_log_timestamp(latest_logs),
        'timezone': 'GMT+7'
    })

5.2 Model Context Protocol (MCP) Server: deployer-probe

To catalog and verify the 525 operational artifacts residing on the live remote Deployer without opening interactive terminal sessions or risking accidental file modifications, we developed a dedicated Model Context Protocol (MCP) server: tools/deployer_probe_mcp.py.

The Model Context Protocol (MCP) is an open standard that allows AI assistants and developer tools to securely discover and read structured data from external systems.

Our deployer-probe MCP server runs locally on developer workstations, exposing strictly read-only primitives over SSH:

• deployer_list_directory: Recursively inspects file trees while automatically filtering out excluded upstream repositories (e.g., sap-automation/).
• deployer_read_file: Fetches byte-range-limited contents of configuration files, execution logs, and playbook traces.
• deployer_file_stat: Extracts file sizes, line counts, modification timestamps, and document titles.
• deployer_grep: Fast pattern searching across historical deployment logs.

# Extract from tools/deployer_probe_mcp.py
def run_remote_command(cmd: str, timeout: int = 45) -> tuple[int, str, str]:
    """Execute a strictly read-only command on the remote deployer via SSH."""
    ssh_cmd = [
        "ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10",
        "-o", "StrictHostKeyChecking=accept-new",
        "azureadm@deployer", cmd
    ]
    proc = subprocess.run(ssh_cmd, capture_output=True, text=True, timeout=timeout)
    return proc.returncode, proc.stdout, proc.stderr

This read-only MCP toolchain allowed us to index all 788 implementation artifacts in info-index.md with complete auditability, without giving autonomous tools destructive shell access.

---

6. Secure Operations: Zero Public Exposure via Azure Bastion Native Tunneling

Enterprise cloud security policies mandate that production workloads must not have direct public IP addresses attached to their virtual network interfaces.

While SDAF provides a configuration flag to deploy a public IP on the Deployer VM for pilot sandboxes (deployer_enable_public_ip = true), our production-ready deployment isolated the Deployer completely within the private management subnet 10.x.100.64/28.

To enable operators to access both the Deployer CLI and the SAP S/4HANA GUI without creating public jumpbox VMs or opening inbound firewall rules, we leveraged Azure Bastion Native Client Tunneling.

6.1 Private Deployer CLI Access

Using Azure CLI native tunneling, an engineer opens an encrypted tunnel through the enterprise Bastion host directly to the Deployer VM:

# Open an encrypted tunnel to the private Deployer VM
az network bastion tunnel \
  --name "vnet-core-network-id-Bastion" \
  --resource-group "rg-core-network-01" \
  --subscription "<your-subscription-id>" \
  --target-resource-id "/subscriptions/<your-subscription-id>/resourceGroups/rg-sap-control-dev/providers/Microsoft.Compute/virtualMachines/sdaf-deployer-ubuntu" \
  --resource-port 22 \
  --port 2222

In a separate terminal window, the engineer connects securely via localhost:

ssh -p 2222 -i ~/.ssh/id_rsa azureadm@127.0.0.1

6.2 Zero-Exposure SAP GUI Port Forwarding (sapgui_bastion_tunnel.sh)

When the deployment of Central Services (ASCS) and the Primary Application Server (PAS) completes, SAP Basis engineers must log in using the SAP GUI desktop client to verify client 000 configuration, check transaction SICK, and validate transport routes.

Traditionally, this required establishing a remote desktop connection to a Windows jumpbox inside the VNet. Instead, we automated direct port forwarding via sapgui_bastion_tunnel.sh:

#!/bin/bash
# sapgui_bastion_tunnel.sh - Secure SAP GUI Tunneling via Azure Bastion
# Maps local port 3200 to target SAP PAS instance 00 (Port 3200)

BASTION_NAME="vnet-core-network-id-Bastion"
BASTION_RG="rg-core-network-01"
BASTION_SUB="<your-subscription-id>"
TARGET_VM_ID="/subscriptions/<your-subscription-id>/resourceGroups/rg-sap-network-dev/providers/Microsoft.Compute/virtualMachines/erpdevapp1"
SAP_INSTANCE="00"
REMOTE_PORT="3200" # 32XX where XX represents SAP instance number
LOCAL_PORT="${1:-3200}"

echo "Establishing Azure Bastion tunnel to SAP PAS (Instance $SAP_INSTANCE)..."
echo "Configure SAP GUI:"
echo "  - Application Server: localhost"
echo "  - Instance Number:    $SAP_INSTANCE"
echo "  - System ID:          D01"
echo "  - Port:               $LOCAL_PORT"

az network bastion tunnel \
  --name "$BASTION_NAME" \
  --resource-group "$BASTION_RG" \
  --subscription "$BASTION_SUB" \
  --target-resource-id "$TARGET_VM_ID" \
  --resource-port "$REMOTE_PORT" \
  --port "$LOCAL_PORT"

With this tunnel active, an engineer's desktop SAP GUI connects directly to localhost:3200. The traffic is encapsulated across Azure Bastion's encrypted TLS tunnel straight to the private SAP application server (10.x.1.x), eliminating external attack surfaces without compromising operational convenience.

---

7. Control Plane Pre-Flight Verification Checklist

Before triggering the deployment of the SAP Workload Zone or provisioning compute instances, execute the following pre-flight verification sequence directly on the Deployer VM:

Verification Stage	Command / Test	Expected Output / Acceptance Criteria
1. Toolchain Integrity	ansible --version && terraform --version	Ansible Core 2.16.x, Terraform 1.6+, Python 3.10+
2. UAMI Authentication	az account show --query "{User: user.name, Sub: id}"	Resolves to SDAF-DEV-UAMI without requiring interactive login
3. Key Vault Access	az keyvault secret list --vault-name <DeployerKV>	Successfully lists secrets without 403 Forbidden errors
4. State Storage Leasing	az storage container list --account-name <tfstateSA>	Returns container tfstate with LeaseStatus: unlocked/available
5. VNet Peering	az network vnet peering list -g <DeployerRG> --vnet-name <DeployerVNet>	Peering to Workload VNet shows PeeringState: Connected
6. Workload Reachability	nc -zvw3 10.x.1.x 22	TCP port 22 reachable on SAP target VM private IP

---

8. Summary & Architectural Lessons Learned

Bootstrapping an enterprise-grade SDAF Control Plane establishes the bedrock upon which the entire automated SAP deployment rests:

• Decoupling the Roles: Recognize that the Deployer VM is a DevOps runner while SAP VMs are application servers. Using Ubuntu 22.04 LTS for the Deployer provided full toolchain compatibility (Ansible Core 2.16+, Terraform 1.6+), while our workload virtual machines remained strictly compliant on SAP-certified SLES 15 SP5.
• Dynamic Governance: User-Assigned Managed Identity (SDAF-DEV-UAMI) eliminates plaintext secrets, while setting spn_id = "<uami-client-id>" prevents Terraform from destroying Key Vault access policies on re-apply.
• In-Guest Observability: Telemetry tools like ansible_monitor and read-only MCP servers (deployer-probe) provide real-time visibility into multi-hour deployments without terminal lock-in.
• Zero-Exposure Operations: Azure Bastion native client tunneling provides secure, private administrative pathways for both CLI execution and desktop SAP GUI connectivity.

With our Control Plane hardened and peered to the brownfield workload network, our next major hurdle was shared storage architecture.

---

Coming Up in Part 3:

The Storage Architecture Dilemma: Pivoting from ANF to Azure Files NFS — How we handled the sudden absence of Azure NetApp Files quota in a new cloud region, re-engineered SAP shared storage (/sapmnt, /usr/sap/trans, /hana/shared) on Azure Files Premium NFS 4.1, and solved brownfield subnet firewall locks.