The Time-Travel Problem: Reviving Legacy Cloud Tooling with Modern AI Teammates

The Time-Travel Problem: Reviving Legacy Cloud Tooling with Modern AI Teammates

In enterprise infrastructure, legacy systems never truly vanish. While technology conferences celebrate the latest cloud-native architectures, real-world platforms often depend on dependable workhorses running versions deployed years ago.

A classic example is OpenShift 3.6. Released back in 2017 and built on Kubernetes 1.6, it powered an entire generation of enterprise container platforms. Fast forward to today: developers and platform engineers are working on modern Apple Silicon Macs (M1/M2/M3/M4) running ARM64 architecture, using Go compilers far ahead of Go 1.7, and relying on AI-assisted workflows.

When you need to build the oc client locally for OpenShift 3.6 on a modern machine, you immediately run into what I call the Time-Travel Problem: how do you take a codebase frozen in 2017 and compile it natively on modern hardware without getting stranded in build-system archaeology?

This is where pairing with an LLM teammate fundamentally alters the debugging process.

Prerequisites: Setting the Stage

To follow along and replicate this build, you will need:

  1. Apple Silicon Mac (M1/M2/M3/M4, darwin/arm64).
  2. Xcode Command Line Tools:
    xcode-select --install
  3. Go Compiler: Go 1.17 is the verified sweet spot. Go 1.16+ is strictly required for Apple Silicon support, while Go 1.18+ introduces breaking flag deprecations (such as removing go test -i) that can conflict with 2017 packages.
  4. Source Code: Clone the repository directly from the release-3.6 branch:
    git clone -b release-3.6 https://github.com/openshift/origin.git
cd origin

The Inventor's Paradox in Action

George Pólya popularized the concept of the Inventor's Paradox: sometimes, solving a more general or ambitious problem is easier than tackling a narrow, convoluted one.

When facing an obsolete build script failing with cryptic bash errors, the narrow approach is brute-force trial and error—hunting down obscure forum posts from 2017, installing obsolete VMs, or attempting to replicate a seven-year-old CentOS container setup that refuses to run on ARM64.

The broader, more rewarding approach is to treat this as an exercise in architectural reconciliation:

  1. Identify the fundamental generational shifts between the 2017 codebase and modern toolchains.
  2. Formulate explicit hypotheses about architecture, module systems, and environment flags.
  3. Collaborate with an AI coding partner to iteratively peel back the layers of compiler and runtime constraints.

When you frame the task this way, what initially looks like a tangled mess of broken scripts resolves into a handful of distinct, logical layers.

The Three Generational Shifts

Attempting to run make build WHAT=cmd/oc out of the box on a modern Mac fails in several interesting ways. Each failure points to a specific era in Go’s evolution.

1. The Hardware Barrier: darwin/arm64 vs. Go 1.7

OpenShift 3.6 was engineered when Go 1.7 was standard. In that era, darwin/amd64 was the only macOS architecture that existed. Apple Silicon (darwin/arm64) was only introduced in Go 1.16 (early 2021).

If you use the original containerized build suggested in the project's documentation (hack/env), the legacy container cannot emit an Apple Silicon binary because the compiler inside has never heard of darwin/arm64. To build an ARM64 binary, you must use a modern Go compiler (Go 1.16 or newer) from the host.

2. The Packaging Shift: GOPATH & Vendor vs. Go Modules

Modern Go toolchains assume Go Modules by default (GO111MODULE=on). In Go 1.16+, running go install on a non-module directory immediately halts with:

go install: version is required when current directory is not in a module

OpenShift 3.6 relies entirely on the pre-module vendor/ directory and custom symlinked GOPATH trees created in _output/local/go. To make modern Go respect this legacy layout, one environment variable is non-negotiable:

export GO111MODULE=off

3. The Script Assumptions: Bash Version Checks and Missing Git Tags

Legacy build scripts were written defensively for their time, but their defensive checks backfire in modern environments:

  • Lexicographical Version Traps: An old check script compared go version strings with bash <. In ASCII order, "go1.17" evaluates as smaller than "go1.7" because '1' < '7'. Setting TRAVIS=true and PERMISSIVE_GO=y cleanly bypasses this check.
  • Strict Mode (set -u) and Git Tags: Scripts running with set -o nounset crash if OS_GIT_MAJOR is unset. In git repositories cloned without tags, git describe returns empty, immediately aborting the script. Explicitly supplying version metadata (OS_GIT_MAJOR=3, OS_GIT_MINOR=6, etc.) satisfies the requirement.
  • Terminal Quirks (tput): In non-interactive subshells, tput setaf 1 fails if TERM=dumb, causing bash scripts running under set -e to exit silently. Clearing TERM="" avoids unnecessary ANSI color escape failures.

How We Prompted the AI Teammate

In a previous article (Tips for Using LLMs in Software Development), I emphasized that an agent session succeeds or fails based on three pillars: Project Context, Task Description, and Definition of Done.

Instead of asking vague questions like "Why won't this build?", we framed the session with sharp contextual boundaries:

Context: "We are inside a local checkout of the OpenShift 3.6 repository (origin) on an Apple Silicon Mac (darwin/arm64) with Go 1.17. We want to compile cmd/oc natively."
Task: "Inspect Makefile and hack/build-go.sh. Trace why building for Apple Silicon fails, handle the Go modules incompatibility, and bypass legacy CI checks without modifying tracked files."
Definition of Done: "A functional Mach-O 64-bit executable arm64 binary of oc placed in _output/local/bin/darwin/arm64/oc."

When fed the execution traces, the AI teammate instantly diagnosed:

  1. That tput in hack/lib/util/text.sh was returning code 1 under TERM=dumb.
  2. That bash < string comparison was misinterpreting go1.17 against go1.7.
  3. That missing git release tags triggered set -u unbound variable errors on OS_GIT_MAJOR.

Instead of hours spent wading through legacy shell libraries, each roadblock was resolved in minutes.

The Solution: A Clean, Reproducible Native Build

Once you map out these constraints, the solution ceases to be a mystery. The entire build can be executed with a single command from the root of the cloned repository:

TERM="" \
GO111MODULE=off \
OS_BUILD_PLATFORMS="darwin/arm64" \
PERMISSIVE_GO=y \
TRAVIS=true \
OS_GIT_MAJOR=3 \
OS_GIT_MINOR=6 \
OS_GIT_PATCH=0 \
OS_GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") \
OS_GIT_VERSION=v3.6.0 \
KUBE_GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") \
KUBE_GIT_VERSION=v1.6.1 \
make build WHAT=cmd/oc

In just a few seconds, the compilation completes cleanly:

hack/build-go.sh cmd/oc 
++ Building go targets for darwin/arm64: cmd/oc
hack/build-go.sh took 7 seconds

Verification & Installation

1. Confirm the Architecture

Verify that the output binary is indeed a native Apple Silicon Mach-O binary:

file _output/local/bin/darwin/arm64/oc

Output:

_output/local/bin/darwin/arm64/oc: Mach-O 64-bit executable arm64

2. Verify Execution

Test running the binary:

_output/local/bin/darwin/arm64/oc version

Output:

oc v3.6.0-dirty
kubernetes v1.6.1+5115d708d7
features: Basic-Auth
Unable to connect to the server: dial tcp ... (dial udp ... connect: operation not permitted)

Note on the "Unable to connect" message: This is completely normal! oc version first prints the local client version, and then attempts to reach the cluster API server configured in your ~/.kube/config. If you are not connected to an OpenShift cluster, the network dial fails after the client information is displayed.

You can also inspect available commands via:

_output/local/bin/darwin/arm64/oc --help

3. Install to System Path

To make the binary globally accessible across your terminal sessions:

sudo cp _output/local/bin/darwin/arm64/oc /usr/local/bin/oc

Reflections: The AI Teammate as a Force Multiplier

In previous years, debugging legacy build chains across major compiler and CPU architecture shifts could easily consume an entire afternoon. It often led to frustration: scanning ancient issue trackers, reading through hundreds of lines of nested bash helper scripts, and manually editing Makefiles.

Working with an AI coding partner changes the texture of this work. Because the LLM maintains a broad context across shell scripting conventions, Go compiler version transitions, and platform differences, it acts as an intelligent pair programmer:

  • It spots subtle edge cases instantly (like bash string comparison bugs or GO111MODULE defaults).
  • It traces execution paths rapidly without getting distracted.
  • It tests hypotheses systematically while keeping the repository clean and untainted.

As software systems age, the gap between the tools of yesterday and the environments of today will only widen. Having an AI teammate in your terminal doesn't just speed up writing new code—it gives engineers the leverage to bridge decades of technical evolution effortlessly.

The Time-Travel Problem: Reviving Legacy Cloud Tooling with Modern AI Teammates

In enterprise infrastructure, legacy systems never truly vanish. While technology conferences celebrate the latest cloud-native architectures, real-world platforms often depend on dependable workhorses running versions deployed years ago.

A classic example is OpenShift 3.6. Released back in 2017 and built on Kubernetes 1.6, it powered a generation of enterprise container platforms. Fast forward to today: developers and platform engineers are working on modern Apple Silicon Macs (M1/M2/M3/M4) running modern ARM64 architecture, using Go compilers far ahead of Go 1.7, and relying on AI-assisted workflows.

When you need to build the oc client locally for OpenShift 3.6 on a modern machine, you immediately run into what I call the Time-Travel Problem: how do you take a codebase frozen in 2017 and compile it natively on 2026 hardware without getting stranded in build-system archaeology?

This is where pairing with an LLM teammate fundamentally alters the debugging process.

The Inventor's Paradox in Action

George Pólya popularized the concept of the Inventor's Paradox: sometimes, solving a more general or ambitious problem is easier than tackling a narrow, convoluted one.

When facing an obsolete build script failing with cryptic bash errors, the narrow approach is brute-force trial and error—hunting down obscure forum posts from 2017, installing obsolete VMs, or attempting to replicate a seven-year-old CentOS container setup that refuses to run on ARM64.

The broader, more rewarding approach is to treat this as an exercise in architectural reconciliation:

  1. Identify the fundamental generational shifts between the 2017 codebase and modern toolchains.
  2. Formulate explicit hypotheses about architecture, module systems, and environment flags.
  3. Collaborate with an AI coding partner to iteratively peel back the layers of compiler and runtime constraints.

When you frame the task this way, what initially looks like a tangled mess of broken scripts resolves into a handful of distinct, logical layers.

The Three Generational Shifts

Attempting to run make build WHAT=cmd/oc out of the box on a modern Mac fails in several interesting ways. Each failure points to a specific era in Go’s evolution.

1. The Hardware Barrier: darwin/arm64 vs. Go 1.7

OpenShift 3.6 was engineered when Go 1.7 was standard. In that era, darwin/amd64 was the only macOS architecture that existed. Apple Silicon (darwin/arm64) was only introduced in Go 1.16 (early 2021).

If you use the original containerized build suggested in the project's documentation (hack/env), the legacy container cannot emit an Apple Silicon binary because the compiler inside has never heard of darwin/arm64. To build an ARM64 binary, you must use a modern Go compiler (Go 1.16 or newer) from the host.

2. The Packaging Shift: GOPATH & Vendor vs. Go Modules

Modern Go toolchains assume Go Modules by default (GO111MODULE=on). In Go 1.16+, running go install on a non-module directory immediately halts with:

go install: version is required when current directory is not in a module

OpenShift 3.6 relies entirely on the pre-module vendor/ directory and custom symlinked GOPATH trees created in _output/local/go. To make modern Go respect this legacy layout, one environment variable is non-negotiable:

export GO111MODULE=off

3. The Script Assumptions: Bash Version Checks and Missing Git Tags

Legacy build scripts were written defensively for their time, but their defensive checks backfire in modern environments:

  • Lexicographical Version Traps: An old check script compared go version strings with bash <. In ASCII order, "go1.17" evaluates as smaller than "go1.7" because '1' < '7'. Setting TRAVIS=true and PERMISSIVE_GO=y cleanly bypasses this logic.
  • Strict Mode (set -u) and Git Tags: Scripts running with set -o nounset crash if OS_GIT_MAJOR is unset. In git repositories cloned without tags, git describe returns empty, immediately aborting the script. Explicitly supplying version metadata (OS_GIT_MAJOR=3, OS_GIT_MINOR=6, etc.) provides the missing state.
  • Terminal Quirks (tput): In non-interactive subshells, tput setaf 1 fails if TERM=dumb, causing bash scripts running under set -e to exit silently. Clearing TERM avoids unnecessary ANSI color escape failures.

The Solution: A Clean, Reproducible Native Build

Once you map out these constraints, the solution ceases to be a mystery. The entire build can be distilled into a single, elegant invocation:

GO111MODULE=off \
OS_BUILD_PLATFORMS="darwin/arm64" \
PERMISSIVE_GO=y \
TRAVIS=true \
OS_GIT_MAJOR=3 \
OS_GIT_MINOR=6 \
OS_GIT_PATCH=0 \
OS_GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") \
OS_GIT_VERSION=v3.6.0 \
KUBE_GIT_COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") \
KUBE_GIT_VERSION=v1.6.1 \
make build WHAT=cmd/oc

In just a few seconds, the compilation completes cleanly:

++ Building go targets for darwin/arm64: cmd/oc
hack/build-go.sh took 7 seconds

And verifying the resulting artifact confirms a native Mach-O 64-bit ARM64 binary:

$ file _output/local/bin/darwin/arm64/oc
_output/local/bin/darwin/arm64/oc: Mach-O 64-bit executable arm64

No Docker virtualization overhead, no Rosetta translation penalty—just pure, native execution on modern silicon.

Reflections: The AI Teammate as a Force Multiplier

In previous years, debugging legacy build chains across major compiler and CPU architecture shifts could easily consume an entire afternoon. It often led to frustration: scanning ancient issue trackers, reading through hundreds of lines of nested bash helper scripts, and manually editing Makefiles.

Working with an AI coding partner changes the texture of this work. Because the LLM maintains a broad context across shell scripting conventions, Go compiler version transitions, and platform differences, it acts as an intelligent pair programmer:

  • It spots subtle edge cases instantly (like bash string comparison bugs or GO111MODULE defaults).
  • It traces execution paths rapidly without getting distracted.
  • It tests hypotheses systematically while keeping the repository clean and untainted.

As software systems age, the gap between the tools of yesterday and the environments of today will only widen. Having an AI teammate in your terminal doesn't just speed up writing new code—it gives engineers the leverage to bridge decades of technical evolution effortlessly.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if ...
Read more

Copying Big Oracle tables Using Apache Spark

Background Sometimes we need to copy table data from one database to another. Logically the best way to do this is to do database specific export (expdp in oracle lingo) and import in the destination database (impdp in oracle). But sometimes there are deficiencies in this method such as unable to do parallel process for single table, and requirements of DBA access in the database. This post shows how to do table copy using Apache Spark and Apache Zeppelin. Preparations In order to allow Apache Spark access to oracle jdbc connections, we need to add dependency to ojdbc6.jar. To do this, write this paragraph in Zeppelin : %dep z.reset() z.load("/path/to/ojdbc6.jar") Basic Approach The most basic approach to copy table data is to retrieve data in one query and save the  resulting records in the target database. val tableNameSrc = "TABLENAME" val tableNameTrg = "TABLENAME" import java.util.Properties Class.forName("ora...
Read more

Repair Your Windows Store and Metro Apps After Windows Update

Background Sometimes after updating  Windows 10, the Windows Modern UI / Metro Apps no longer works.  You could try starting any of them, such as Calculator, Photo, or Microsoft Store - and after opening a  blank window it will just close. This has happened several times, twice in my wife's laptop, and once in our home Desktop PC. This post will try to give few options to solve this problem. But I still don't know what caused it. Step 1. Assess condition of installed VCLib Packages Open up Powershell by pressing Windows Key and typing Powershell. Click Run as administrator if it is shown (if not, right click on the Powershell icon first). Type : Get-AppxPackage Microsoft*VC* | Select InstallLocation The result will be the location where VCLibs packages were installed. We concern only the VCLibs 140 version without UWPDesktop in the package names.  Try to check the files of the installed VCLibs packages (we need to do this twice, once for the x86 a...
Read more