Detecting rootkit using Linux Kernel Runtime Guard

 You want to determine whether there is an anomaly in your system, that might be caused by some rootkit installed by some hacker. These steps are tested on a Centos 7.9 VM.

Installation steps

First you need to install kernel-headers, kernel-tools, kernel-devel and gcc (gnu c compiler). Remove enablerepo stuffs if you are using latest kernel version in your distribution.

yum install --enablerepo=C7.8.2003-base --enablerepo=C7.8.2003-updates kernel-headers-3.10.0-1127.10.1.el7.x86_64 kernel-tools-3.10.0-1127.10.1.el7.x86_64 kernel-devel-3.10.0-1127.10.1.el7.x86_64 gcc


Alternative for using latest kernel version :

yum install kernel-headers kernel-tools kernel-devel gcc

Download lkrg from the main website : https://www.openwall.com/lkrg/

# wget https://www.openwall.com/lkrg/lkrg-0.9.1.tar.gz

# ls -l lkrg-0.9.1.tar.gz 

-rw-r--r--. 1 root root 121565 Apr 28 02:43 lkrg-0.9.1.tar.gz

Extract and compile the stuffs.

# tar -xvzf lkrg-0.9.1.tar.gz 

# make

# make install

Additional Configuration (important)

Additional configuration steps are neccessary to prevent LKRG killing sshd in my system after being invaded by falc0n (Reptile-based rootkit). This is the configuration needed:

[root@lb ~]# cat > /etc/modprobe.d/lkrg.conf 

options p_lkrg pcfi_enforce=0
<ctrl-D>


Activation and Test

Test LKRG by using 

systemctl start lkrg

Then check your dmesg:

[root@lb lkrg-0.9.1]# dmesg | tail -100

[154746.019372] Freezing user space processes ... (elapsed 0.001 seconds) done.

[154746.722341] Restarting tasks ... done.

[154746.722616] [p_lkrg] LKRG unloaded!

[154750.277176] [p_lkrg] Loading LKRG...

[154750.277718] [p_lkrg] System does NOT support SMEP. LKRG can't enforce SMEP validation :(

[154750.278232] [p_lkrg] System does NOT support SMAP. LKRG can't enforce SMAP validation :(

[154750.282842] Freezing user space processes ... (elapsed 0.001 seconds) done.

[154750.442162] [p_lkrg] [kretprobe] register_kretprobe() for <ovl_create_or_link> failed! [err=-2]

[154750.442630] [p_lkrg] Can't hook 'ovl_create_or_link' function. This is expected if you are not using OverlayFS.

[154750.576751] [p_lkrg] LKRG initialized successfully!

[154750.577213] Restarting tasks ... done.

Messages written by p_lkrg is shown in dmesg and also written in /var/log/messages.

[root@lb lkrg-0.9.1]# cat /var/log/messages | tail 

May 28 07:50:13 lb kernel: [p_lkrg] Loading LKRG...

May 28 07:50:13 lb kernel: [p_lkrg] System does NOT support SMEP. LKRG can't enforce SMEP validation :(

May 28 07:50:13 lb kernel: [p_lkrg] System does NOT support SMAP. LKRG can't enforce SMAP validation :(

May 28 07:50:14 lb kernel: Freezing user space processes ... (elapsed 0.001 seconds) done.

May 28 07:50:14 lb kernel: [p_lkrg] [kretprobe] register_kretprobe() for <ovl_create_or_link> failed! [err=-2]

May 28 07:50:14 lb kernel: [p_lkrg] Can't hook 'ovl_create_or_link' function. This is expected if you are not using OverlayFS.

May 28 07:50:14 lb kernel: [p_lkrg] LKRG initialized successfully!

May 28 07:50:14 lb kernel: Restarting tasks ... done.

May 28 07:50:14 lb modprobe: insmod /lib/modules/3.10.0-1127.10.1.el7.x86_64/extra/p_lkrg.ko pcfi_enforce=0

May 28 07:50:14 lb systemd: Started Linux Kernel Runtime Guard.

Install the falc0n rootkit (this also can be done the other way around, rootkit first and lkrg last)

[root@lb lkrg-0.9.1]# modinfo falc0n.ko 

filename:       /root/lkrg-0.9.1/falc0n.ko

intree:         Y

license:        GPL

retpoline:      Y

rhelversion:    7.8

srcversion:     81F508029A53F7490CCDB44

depends:        

vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversion


[root@lb lkrg-0.9.1]# mkdir -p /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n

[root@lb lkrg-0.9.1]# cp falc0n.ko /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/

[root@lb lkrg-0.9.1]# depmod


Depmod is neccessary to update kernel modules dependency so the falc0n could be loaded by the kernel.

Running the rootkit : (DISCLAIMER: be prepared to restart your machine / VM if something goes wrong, for example if you are unable to remove the kernel module you might want to restart the VM)

[root@lb lkrg-0.9.1]# modprobe falc0n

modprobe: ERROR: could not insert 'falc0n': No locks available


Checking the dmesg :

[root@lb lkrg-0.9.1]# dmesg | tail

[155297.826824]  [<ffffffff85a57065>] pipe_read+0x2e5/0x400

[155297.826825]  [<ffffffff85a4c5e3>] do_sync_read+0x93/0xe0

[155297.826827]  [<ffffffff85a4d01f>] vfs_read+0x9f/0x170

[155297.826828]  [<ffffffffc07a5b4d>] 0xffffffffc07a5b4d

[155297.826831]  [<ffffffff85a4de8f>] SyS_read+0x7f/0xf0

[155297.826833]  [<ffffffff85f92ed2>] system_call_fastpath+0x25/0x2a

[155297.826835] --- END ---

[155297.827260] [p_lkrg] Process[tail | 8748] pCFI verification failed!

[155297.827684] [p_lkrg] <Exploit Detection> Path's inode[705103] mode[0100000] will be isolated!

[155297.828701] [p_lkrg] <Exploit Detection> Not valid call - pCFI violation: process[tail | 8748] !!!

[root@lb lkrg-0.9.1]# dmesg | tail -100

[155302.604811] [p_lkrg] <Exploit Detection> Path's inode[706399] mode[0100000] will be isolated!

[155302.605259] [p_lkrg] <Exploit Detection> Not valid call - pCFI violation: process[tail | 8753] !!!

[155302.606152] [p_lkrg] <Exploit Detection> Frame[5] nr_entries[8]: [0xffffffffc07a5b4d]. Full Stack below:

[155302.607046] --- . ---

[155302.607485]  [<ffffffff85a7e411>] __mark_inode_dirty+0x1/0x270

[155302.607488]  [<ffffffff85a6b35a>] touch_atime+0x10a/0x220

[155302.607490]  [<ffffffff85a57065>] pipe_read+0x2e5/0x400

[155302.607492]  [<ffffffff85a4c5e3>] do_sync_read+0x93/0xe0

[155302.607493]  [<ffffffff85a4d01f>] vfs_read+0x9f/0x170

[155302.607495]  [<ffffffffc07a5b4d>] 0xffffffffc07a5b4d

[155302.607498]  [<ffffffff85a4de8f>] SyS_read+0x7f/0xf0

[155302.607499]  [<ffffffff85f92ed2>] system_call_fastpath+0x25/0x2a

[155302.607502] --- END ---

[155302.607926] [p_lkrg] Process[tail | 8753] pCFI verification failed!

[155302.608350] [p_lkrg] <Exploit Detection> Path's inode[706399] mode[0100000] will be isolated!

[155302.608800] [p_lkrg] <Exploit Detection> Not valid call - pCFI violation: process[tail | 8753] !!!

[155302.609670] [p_lkrg] <Exploit Detection> Frame[5] nr_entries[8]: [0xffffffffc07a5b4d]. Full Stack below:

[155302.610564] --- . ---

[155302.611003]  [<ffffffff85a7e411>] __mark_inode_dirty+0x1/0x270

[155302.611006]  [<ffffffff85a6b35a>] touch_atime+0x10a/0x220

[155302.611008]  [<ffffffff85a57065>] pipe_read+0x2e5/0x400

[155302.611009]  [<ffffffff85a4c5e3>] do_sync_read+0x93/0xe0

[155302.611011]  [<ffffffff85a4d01f>] vfs_read+0x9f/0x170

[155302.611013]  [<ffffffffc07a5b4d>] 0xffffffffc07a5b4d

[155302.611015]  [<ffffffff85a4de8f>] SyS_read+0x7f/0xf0

[155302.611017]  [<ffffffff85f92ed2>] system_call_fastpath+0x25/0x2a

[155302.611019] --- END ---



Even though the name of the offending module not found, LKRG detected pCFI violation and exploit execuition. So now we are able to detect this stuff. And my VM no longer crashed :)

Detection by running LKRG after rootkit


[root@lb lkrg-0.9.1]# systemctl stop lkrg

[root@lb lkrg-0.9.1]# modprobe falc0n

modprobe: ERROR: could not insert 'falc0n': No locks available

[root@lb lkrg-0.9.1]# dmesg | tail

[155849.340591]  [<ffffffff85a56109>] SyS_execve+0x29/0x30

[155849.340592]  [<ffffffff85f93478>] stub_execve+0x48/0x80

[155849.340595] --- END ---

[155849.341062] [p_lkrg] Process[cmd | 9330] pCFI verification failed!

[155849.341530] [p_lkrg] <Exploit Detection> Stack pointer corruption (ROP?) - pCFI violation: process[cmd | 9330] !!!

[155849.342451] [p_lkrg] Process[cmd | 9330] pCFI verification failed!

[155945.170322] [p_lkrg] Unloading LKRG...

[155945.198759] Freezing user space processes ... (elapsed 0.001 seconds) done.

[155945.903737] Restarting tasks ... done.

[155945.904090] [p_lkrg] LKRG unloaded!

[root@lb lkrg-0.9.1]# systemctl start lkrg

[root@lb lkrg-0.9.1]# dmesg | tail -100

[155992.311412] [p_lkrg] <Exploit Detection> Path's inode[710191] mode[0100000] will be isolated!

[155992.311862] [p_lkrg] <Exploit Detection> Not valid call - pCFI violation: process[tail | 9553] !!!

[155992.312733] [p_lkrg] <Exploit Detection> Frame[5] nr_entries[8]: [0xffffffffc07a5b4d]. Full Stack below:

[155992.313626] --- . ---

[155992.314065]  [<ffffffff85a7e411>] __mark_inode_dirty+0x1/0x270

[155992.314068]  [<ffffffff85a6b35a>] touch_atime+0x10a/0x220

[155992.314070]  [<ffffffff85a57065>] pipe_read+0x2e5/0x400

[155992.314071]  [<ffffffff85a4c5e3>] do_sync_read+0x93/0xe0

[155992.314073]  [<ffffffff85a4d01f>] vfs_read+0x9f/0x170

[155992.314075]  [<ffffffffc07a5b4d>] 0xffffffffc07a5b4d

[155992.314078]  [<ffffffff85a4de8f>] SyS_read+0x7f/0xf0

[155992.314079]  [<ffffffff85f92ed2>] system_call_fastpath+0x25/0x2a

[155992.314082] --- END ---

[155992.314507] [p_lkrg] Process[tail | 9553] pCFI verification failed!

[155992.314935] [p_lkrg] <Exploit Detection> Path's inode[710191] mode[0100000] will be isolated!

[155992.315384] [p_lkrg] <Exploit Detection> Not valid call - pCFI violation: process[tail | 9553] !!!

[155992.316273] [p_lkrg] <Exploit Detection> Frame[5] nr_entries[8]: [0xffffffffc07a5b4d]. Full Stack below:

[155992.317166] --- . ---

[155992.317604]  [<ffffffff85a7e411>] __mark_inode_dirty+0x1/0x270

[155992.317607]  [<ffffffff85a6b35a>] touch_atime+0x10a/0x220

[155992.317609]  [<ffffffff85a57065>] pipe_read+0x2e5/0x400

[155992.317611]  [<ffffffff85a4c5e3>] do_sync_read+0x93/0xe0

[155992.317612]  [<ffffffff85a4d01f>] vfs_read+0x9f/0x170

[155992.317614]  [<ffffffffc07a5b4d>] 0xffffffffc07a5b4d

[155992.317617]  [<ffffffff85a4de8f>] SyS_read+0x7f/0xf0

[155992.317618]  [<ffffffff85f92ed2>] system_call_fastpath+0x25/0x2a

[155992.317621] --- END ---

[155992.318045] [p_lkrg] Process[tail | 9553] pCFI verification failed!

[155992.318469] [p_lkrg] <Exploit Detection> Path's inode[710191] mode[0100000] will be isolated!

[155992.318920] [p_lkrg] <Exploit Detection> Not valid call - pCFI violation: process[tail | 9553] !!!

[155992.319791] [p_lkrg] <Exploit Detection> Frame[5] nr_entries[8]: [0xffffffffc07a5b4d]. Full Stack below:

[155992.320684] --- . ---

[155992.321122]  [<ffffffff85a7e411>] __mark_inode_dirty+0x1/0x270

[155992.321125]  [<ffffffff85a6b35a>] touch_atime+0x10a/0x220

[155992.321127]  [<ffffffff85a57065>] pipe_read+0x2e5/0x400

[155992.321129]  [<ffffffff85a4c5e3>] do_sync_read+0x93/0xe0

[155992.321130]  [<ffffffff85a4d01f>] vfs_read+0x9f/0x170

[155992.321132]  [<ffffffffc07a5b4d>] 0xffffffffc07a5b4d

[155992.321135]  [<ffffffff85a4de8f>] SyS_read+0x7f/0xf0

[155992.321136]  [<ffffffff85f92ed2>] system_call_fastpath+0x25/0x2a

[155992.321139] --- END ---


Conclusion

LKRG can be used to detect some exploits executed by falc0n/reptile-based rootkit. But don't expect it to show the location of the rootkit or the name of the rootkit. And also don't forget to set some configuration to prevent LKRG killing your sshd server..

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more