Debugging Openshift Docker Registry using Visual Studio Code

Overview

Red Hat  has augmented docker's registry with some enhancement before deploying it in Openshift cluster. In some cases we might want to debug it, to find out reasons why some stuff doesn't work or for educational purposes.
In this post we will describe steps required to do debugging for the docker registry (a.k.a docker distribution) from Visual Studio Code. We are going to run the openshift docker registry in a separate deployment config.

Repository identification

First, docker registry is created using Go Language; and in this case, the source code for the enhanced docker registry lives in github repository https://github.com/openshift/origin for some versions and in https://github.com/openshift/docker-distribution for later versions. In this post I will use the version in openshift 3.6 ( https://github.com/openshift/origin/tree/v3.6.1 ) because of selfish reasons our sandbox/development openshift cluster uses this version.
The main source code file lives in cmd/dockerregistry path (https://github.com/openshift/origin/tree/v3.6.1/cmd/dockerregistry) which we need to build using go debug-specific options.

S2I Base image

To be able to build the docker registry, we need a base image; for this purpose I build from the go 1.15 version in https://github.com/sclorg/golang-container. The steps for building this image is as follows :

cd 1.15
docker build -t golang:scl-1.15 -f Dockerfile.fedora .

To push this to internal registry use :

oc login [openshift-cluster-context-url]
[ enter username and password here ]

docker tag golang:scl-1.15 [internal-repo-url]/openshift/golang:scl-1.15
docker login -u [username] -p `oc whoami -t` [internal-repo-url]
docker push [internal-repo-url]/openshift/golang:scl-1.15

Building the dockerregistry binary

First, clone the https://github.com/openshift/origin/tree/v3.6.1 repository branch because we need to create .s2i/bin/assemble file get debugging.
If your openshift environment allows accessing github, add this file (we use delve v1.7.3 because that is the latest version supporting golang 1.15) :

mkdir -p .s2i/bin
cd .s2i/bin
cat > assemble
#!/bin/bash
go get github.com/go-delve/delve/cmd/dlv@1.7.3
mkdir -p /opt/app-root/src/bin
cp $HOME/go/bin/dlv /opt/app-root/src/bin/dlv
/usr/libexec/s2i/assemble
<ctrl-D>

chmod u+x assemble
git add assemble
git commit -m "add assemble file"

And then you could skip the next chapter.
If not, we need to extract the dlv file  first.

dlv v1.7.3 binary file extraction 

docker run -it golang:scl-1.15 bash
go get github.com/go-delve/delve/cmd/dlv@1.7.3
<ctrl-P ctrl-Q>. -- detach from the container
docker ps 

CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS     NAMES

d260382ee77d   golang-container:1.15   "container-entrypoin…"   6 minutes ago   Up 6 minutes             modest_tharp

(take note of the running container id)
mkdir bin
docker cp [container-id]:/opt/app-root/src/go/bin/dlv bin/dlv
docker attach [container-id]
exit

git add bin/dlv

create the .s2i/bin/assemble file :

mkdir -p .s2i/bin
cd .s2i/bin
cat > assemble

#!/bin/bash

mkdir -p /opt/app-root/src/bin

cp /tmp/src/bin/dlv /opt/app-root/src/bin/dlv

/usr/libexec/s2i/assemble

<ctrl-D>


chmod u+x assemble
git add assemble
cd ../..
git commit -m "add assemble and dlv executable"

now we could push it to our (personal) repo.

Building the dockerregistry binary  part 2

Next, create a golang build configuration, in our system the golang template not installed, so we just create php application and replace the builder image with openshift/golang:scl-1.15 tag.
Add these two environment variables :
      env:
        - name: INSTALL_URL
          value: github.com/openshift/origin/cmd/dockerregistry
        - name: IMPORT_URL
          value: github.com/openshift/origin

After add these two environment variable, run the build and hopes that the build runs sucessfully. Actually this build doesn't run the debugger yet.

Activating debug build

To run app using debug build, the syntax is like this :
dlv debug github.com/openshift/origin/cmd/dockerregistry --check-go-version=false --output /tmp/dockerregistry -l 0.0.0.0:2345 --headless=true --log=true --api-version=2

Now, how to we make the openshift deployment run this ? In our case, we add this run file :

cd .s2i/bin
cat > run

#!/bin/bash

if [ "$DEPLOYENTRYPOINT" == "" ] 

then

if [ "$INSTALL_URL"=="github.com/openshift/origin/cmd/dockerregistry" ]

then

 exec /opt/app-root/gobinary $REGISTRY_CONFIGURATION_PATH

else

 echo "Not supported yet"

fi

else

  exec $DEPLOYENTRYPOINT

fi

<ctrl-D>

chmod u+x run

git add run


And then we add DEPLOYENTRYPOINT in the Deployment Environment variables, with this content : 
dlv debug github.com/openshift/origin/cmd/dockerregistry --check-go-version=false --output /tmp/dockerregistry -l 0.0.0.0:2345 --headless=true --log=true --api-version=2

Actually debugging (yes!)

After the deployment runs, we need to do port-forwarding to be able to debug from our laptop/desktop.
oc get pods 
oc port-forward [podname] 2345:2345

Then fire up Visual Studio Code opening the source code folder, and create  .vscode/launch.json file :
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Connect to server",
"type": "go",
"request": "attach",
"mode": "remote",
"remotePath": "/opt/app-root/src/go/src/github.com/openshift/origin",
"port": 2345,
"host": "127.0.0.1",
"useApiV1": false,
"dlvLoadConfig": {
"followPointers": true,
"maxVariableRecurse": 1,
"maxStringLen": 300,
"maxArrayValues": 64,
"maxStructFields": -1
}
}
]
}

Choose menu Run & debug in the left side of Visual Studio Code, and click Connect to server..


Put some breakpoints, and voila .. we now be able to  debug remotely !

debug session


Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more