Friday, December 28, 2012

Nginx and Linux Kernel Update Blues


We have a three-server landscape that serves our internal applications. All of them were running nginx on RHEL linux. The nginx were installed from EPEL repository packages. Two days ago I updated the nginx in one server (lets say server A) in order to repackage it after enhancing it with HttpSubsModule, during the compilation process it complains about missing kernel functionality. For the compile to be success, I updated kernel-devel and kernel packages (in server A) to the latest one. After the compilation succeeded, I copied the resulting rpm to server B and installed it.

The problem

After reinstalling the rpm package in server B, I didn't remember to restart the nginx server. After installing new kernel in server A, I havent restarted it either. But the next day, because of one thing and another, server A got restarted, and the application on server A no longer works. Server B's nginx server also got restarted, and afterwards unable to serve any web pages (not even fixed pages). So we got two servers down out of three. Thats not good.

Investigation : Eventfd not implemented

Investigation in nginx error log shows that server B's nginx server complains about failed evenfd() calls (ref) :
eventfd() failed (38: Function not implemented)
I suddenly recall this problem already happens before on server C, that the cause was newer nginx package needs a backported kernel feature. The solution is install latest kernel and reboot.
yum update kernel
The root cause of this is that the nginx EPEL package doesn't correctly specify  the true minimum kernel version required for it to work.

Investigation: SELinux prevents text relocation

After got the kernel update, both server A and B still unable to serve the application pages.
Browsing nginx and php's error log shows that the php-oci8 extension is not loaded (needed to connect to oracle which is being used as database backend of our legacy applications).
The clue why the extension failed to load were found in /var/log/messages, it seems that SELinux complained that the oracle's dynamic library (libclntsh) is trying to do code relocation and failed (see redhat bug report here and article about PHP, SELinux, and Oracle here).
setroubleshoot: SELinux is preventing php from loading /opt/oracle/instantclient/ which requires text relocation. For complete SELinux messages. run sealert -l ac5924e8-3af4-4f26-8d79-e5eedf9d9d7a
The solution is by giving texrel_shlib_t SELinux type to all oracle instant client dynamic libraries.
chcon -t texrel_shlib_t *.so
chcon -t texrel_shlib_t *.so.10.1
You could see the resulting attributes on all files using
ls -lZ
 Why this haven't occured before the kernel update? Maybe SELinux permission is gotten more restricted in recent kernels. But I am not certain about that. I wonder whether SELinux attributes could be specified in rpm packages. If that is so, a better solution for the world is that to build a better rpm package containing instant client, oci8,  and required SELinux permissions. But I wonder would that violate Oracle's EULA..


Thinking in more generic frame of mind, these incidents occured because we update  the kernel and the nginx package. Is it wrong? No. The best practice on server administration is that all security updates is applied as soon as possible. But we better plan to do updates outside application's busy hours, so if problems crop out we have sufficient breathing room :)

Tuesday, December 25, 2012

Workflow Approval Dasar dalam Joget

The case

Dalam pembuatan aplikasi enterprise, banyak aktivitas yang perlu dibuat form elektroniknya. Umumnya form ini perlu menggunakan persetujuan sebelum diakui oleh perusahaan. Aplikasi jenis ini ialah aplikasi berbasis workflow, akan lebih terstruktur dan maintenable jika dibuat memanfaatkan workflow engine yang sudah ada. Salah satu workflow engine yang open source adalah Joget Workflow. Artikel ini mencoba menjelaskan tahap awal pembuatan aplikasi workflow yaitu pembuatan diagram workflow.

Skenario yang didukung

Dokumen request dibuat oleh aktor pembuat, kemudian disubmit ke aktor approver (umumnya atasan). Tiap aktor diperbolehkan melakukan reject dokumen. Approver dapat melakukan pengembalian dokumen (return) atau persetujuan (approve).

Basic Workflow

Pada workflow ini, Pembuat dapat melakukan aktivitas 'BuatDokumen'. Tombol aktivitas yang dipilih (submit , reject/cancel, atau return) diisikan ke variabel 'Keputusan'. Variabel ini menjadi dasar bagi pengaturan aliran. BuatDokumen dibuat ada dua box, sebenarnya cuma satu, tetapi Workflow Engine melarang aliran (panah) ke starting activity selain dari start. Sehingga dibuat BuatDokumen kedua untuk memfasilitasi dikembalikannya dokumen dari atasan.

Variasi - Beda variabel

Variasi ini menggunakan variabel KeputusanPembuat untuk menyimpan aksi pembuat dan variabel KeputusanApprover1 untuk menyimpan aksi approver1. 
Keuntungannya ialah debugging flow lebih mudah, karena dasar dasar keputusan tersimpan di variabel yang berbeda.
Kerugiannya ialah tiap panah harus dicek bahwa telah menggunakan nama variabel keputusan yang sesuai.
Jika variasi ini tidak digunakan, yaitu hanya satu variabel workflow yang digunakan, isi variabel keputusan pembuat akan tertimpa keputusan approver1. Tiap pemilihan keputusan sebenarnya dapat direkam di luar Joget, sehingga menyederhanakan workflow dengan penggunaan hanya satu variabel Keputusan.

Enhance - Otherwise

Ketika tidak ada klausa otherwise, dan workflow engine mendapatkan isi variabel tidak memenuhi semua kondisional, maka workflow akan diterminasi. Satu cara untuk mencegah terminasi yang tidak diinginkan (misalnya karena ada bug di front-end sehingga variabel keputusan tidak terisi) ialah menambahkan klausa otherwise ke semua kondisional.

Monday, December 24, 2012

Nginx Http Subs Module for CentOS - Packaging Howto

The case

I recently involved in a reverse proxy project using nginx as reverse proxy server. Turns out that nginx has a built in HttpSubModule that allow us to replace urls in http stream, which is a very important requirements for us.  But the problem is that the HttpSubModule only allow one replacement per location. 
After a few searches, found that an additional HttpSubsModule (notice the additional s)  will do the task, allowing multiple replacement per location.The nginx wiki is kind enough to provide installation instructions for HttpSubsModule, but provides no rpm package.

DISCLAIMER: This blog post shows step by step tutorial to produce a RPM package file. If you only interested in installing nginx with HttpSubsModule, please jump to the last heading 'Installation'. But if you're not on Centos 6 x86_64, maybe you really should follow all the steps anyway.

Repository Hunt

I prefer repository packages other than compiling manually. We found that EPEL repository carries nginx 0.8.55 for Centos 5 (here) or 1.0.15 for Centos 6 (here), but unfortunately these are compiled without HttpSubsModule. Nginx site points us to a repository at which carries nginx 1.2.6, but this is too without HttpSubsModule.

SRPM install

First to install the source RPM for nginx  (do this as a normal user other than root) :
rpm -i nginx-1.2.6-1.el6.ngx.src.rpm
 Prepare your system for rpmbuild and install devel packages (do these as root):
yum install rpm-build
yum install zlib-devel pcre-devel openssl-devel


Time to bite the bullet and rebuild nginx package.
Rpmbuild is a tool to build rpm. It is a packaging tool with functionality similar to Java's Ant, but albeit a complex one. In the executing rpmbuild we executed these phases:
  1. %prep stage
  2. %build stage
  3. %install stage
The spec file provides similar function such as ant's build.xml, that is specifying build steps and sources.

After wrestling with some rpmbuild online documentation, I found that essentially I need to (do these as a normal user):
  • download the http subs module sources using git, compress it into tar gz format, and host it in a website somewhere
  • add additional source line  in nginx.spec, and also download the tar gzipped file to ~/rpmbuild/SOURCES directory (rpmbuild expects the tar gz file exists with the same name as the URL given below, it won't download it for you)
Source5: nginx.vh.default.conf
Source6: nginx.vh.example_ssl.conf
Source7: nginx.suse.init
  • add additional %setup invocation to extract the httpsubs.tar.gz. The -q means quiet (disable tar's chattiness), -a 8 means extract source 8 after cd to package directory, -T means don't extract source 0 again,  -D means dont delete top level directory before extract. Really complex line for simple behavior.
%setup -q
%setup -q -T -D -a 8
  • add clauses to the configure command (there are two of them):
         --with-http_addition_module \
        --with-http_sub_module \
        --add-module=ngx_http_substitutions_filter_module \        --with-http_dav_module \
        --with-http_flv_module \
And run rpmbuild (still as user), pray that there are no errors :
  • rpmbuild -ba nginx.spec
Voila, we got at rpm files at /home/username/rpmbuild/RPMS/x86_64.


If you don't want to create rpm file, but only want to install one, and it happens that you're running Centos 6 on x86_64 platform, feel free to download my resulting rpm file at
From the resulting rpm file (whether from download or following steps above), install it (as root):

rpm -i nginx-1.2.6-1.el6.ngx.x86_64.rpm
Thats all.. I hope these steps works for you too.

Sunday, December 2, 2012

Useless combination of logical expressions

Recently I stumbled upon this SQL where clause :
status <> 2  OR status <> 0
or in equivalent form for you PHP developers :
(status != 2 )  || (status != 0)

To analyze why such combination are useless,
Lets define two variables to simplify things :

A = (status <> 0)
B = (status <> 2)

And draw it on one table :

 Then I wonder, what good does that do ? It always evaluates to true..
It only make sense if we combine the status comparisons with the and operator :