Rants On NFS Lack of File Handle Visibility To Sysadm

 NFS is a not-so-recent solution to share filesystem across linux nodes. It have some capability that are currently indispensable for Linux Clusters : to lock files across nodes and allow either exclusive or non-exclusive access to the same file.

Fault Tolerance / Recovery

I have read some papers on NFS, it should be able to recover a restarting host / server. Unfortunately in several occassion we found this to be not quite true, after a host serving NFS being restarted, we have stale handle errors in the client. The workaround is to restart NFS client, and if that still doesn't fix the situation, restart NFS server. In our cases sometimes we need to restart twice across the cluster (because the client hangs running a program over NFS). Some might said program shouldn't be run over NFS (and only data files should) but we have deployed a SAP documented cluster architecture that requires such use of NFS.

Locks

When a file were locked in the NFS, a lock is being created in the host and the second one is also being created in the client. The problem occured when the lock is exclusive and we don't know which container in which nodes are locking the file. So in that case we have a specific file and we are wondering which process having the file lock. In our openshift cluster, we have 9 worker nodes each with capacity of 160 containers, and each and every container have a possibility to hold such locks. On the other scenario we want to find which files are being locked on the server hosting the NFS share. Why can't there is a kernel API to show the path of NFS locks on ZFS filesystem.. lslocks can't tell us anything about the file path, presumably caused by combination of the filesystem (ZFS) and the lock process is actually in remote host. In previous post I showed zdb -dddd to check file path, but alas in the 7.5.x ZFS this doesn't show any path.

Handles

Sometimes we need to identify which app or process is having a high NFS traffic. This might be a symptom of a runaway process writing endlessly to your shared storage, which need to be identified (so the process could be stopped and avoided disk full incident, or we could decide to move the storage to a faster one for such process/ shared directory) But currently there is no such luck identifying which process or app. If only there is an utility to do that.. When dumping NFS traffic, only file handle are shown, and file contents were sent across the network, but we are unable to get the file name/path (and thus  no clue to detect which app are writing the file).

Disclaimer

If any of this is not true, and there is a solution for the problems above, please contact the author for clarification.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if ...
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, ...
Read more

Repair Your Windows Store and Metro Apps After Windows Update

Background Sometimes after updating  Windows 10, the Windows Modern UI / Metro Apps no longer works.  You could try starting any of them, such as Calculator, Photo, or Microsoft Store - and after opening a  blank window it will just close. This has happened several times, twice in my wife's laptop, and once in our home Desktop PC. This post will try to give few options to solve this problem. But I still don't know what caused it. Step 1. Assess condition of installed VCLib Packages Open up Powershell by pressing Windows Key and typing Powershell. Click Run as administrator if it is shown (if not, right click on the Powershell icon first). Type : Get-AppxPackage Microsoft*VC* | Select InstallLocation The result will be the location where VCLibs packages were installed. We concern only the VCLibs 140 version without UWPDesktop in the package names.  Try to check the files of the installed VCLibs packages (we need to do this twice, once for the x86 a...
Read more