Displaying running PHP 7.4 stack trace on Red Hat UBI Container

 When running a PHP application in container, sometimes we need to find out what bottleneck is impacting the app's response time. This post documents the approach to do so, but still requiring access to root user in the container. The platform where I was running the PHP app is Red Hat Openshift Community (OKD) version 3.6, which is still using docker to run containers instead of Red Hat's podman.

Preparation - Locating the pod and the worker node

  1. Open openshift's console home
  2. Select your Project (namespace) containing the pod
  3. Select Application => Pods





  4. Choose your pod from the list.



  5. Note the worker node hosting the pod.


Install php debug-info

Normally, the openshift platform doesn't allow containers running as root. So are in this case, the container will be running as normal random-uid-user. But we need to install php debuginfo packages, thus we need to run as root. The trick is to access the node running the application's pods, so we need admin access on the worker node to do so.  I am assuming you already have root access on the worker node.
  1. Access the worker node, locate the pod's container.


  2. Run bash shell inside the container as root user
    docker exec -it -u root <containerid> bash
  3. Install debuginfo for php
    dnf debuginfo-install php
  4. Exit the bash shell.

Install php gdbinit

Next, we need to use gdbinit from php sources.
  1. Open this url in your web browser. Replace PHP version with the one matching the one being used. Url : https://github.com/php/php-src/blob/PHP-7.4.30/.gdbinit
  2. Click on the 'Raw' button. The resulting url is like : https://raw.githubusercontent.com/php/php-src/PHP-7.4.30/.gdbinit
  3. Open pod's terminal from openshift console (or use oc rsh)
  4. Get the file using wget :
    wget https://raw.githubusercontent.com/php/php-src/PHP-7.4.30/.gdbinit

Displaying stack trace

  1. Check pid of running apache process
    ps afxw
  2. Restart apache2 process 
    apachectl -k graceful
  3. Check current pid of running apache processes
    ps afxw
  4. Now, check the pid list from step 1 and 3. The common number between these two list means these pids were not being restarted (still running some process), so debug them (debug only the child, not the parent)
    gdb -p pid
  5. Inside the gdb shell, dump stack trace
    dump_bt executor_globals.current_execute_data
  6. Quit the gdb (otherwise the php script would not continue)
    quit
    y

Closing remarks

This procedure is more generic than the previous tricks I have shared in this blog. This might not work on podman-based containers. Outside openshift ecosystem, php-fpm already has built in capability to dump stack trace for running scripts longer than some time treshold.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more