Adapting Openshift Origin for High load

Openshift Origin is a Platform as a Service software platform which enables us to horizontally scale applications and also manage multiple applications in one cluster. One openshift node could contain many applications, the default settings allows for 100 gears (which could be 100 different applications or may be only 4 applications each with 25 gears). Each gear contains a separate apache instance. This post would describe adjustments that I have done on an Openshift M4 cluster that are deployed using the definitive guide. Maybe I really should upgrade the cluster to newer version, but we are currently running production load in this cluster.

The Node architecture

Load balancing in an Openshift application is done by haproxy. The general application architecture is shown below (replace Java with PHP for cases of PHP-based application) (ref : Openshift Blog : How haproxy scales apps).
The gear shown running code, for PHP applications, each consists of one Apache HTTPD instance. 

What is not shown above is that the user actually connects to another Apache instance running in the node host, working as a Reverse-Proxy Server, this could be seen in below image (taken from Redhat Access : Openshift Node Architecture): 
Using Haproxy as a load balancer means the 'Gear 1' in the picture above is replaced by a haproxy instance which distributes loads into another gears. I tried to draw using yEd with this diagram as a result :

Haproxy limits


First, the haproxy cartridge specified 128 as session limit. What if I planned to have 1000 concurrent users ? If the users are using IE, each could create 3 concurrent connection, totaling 3000 connections. So this need to be changed.
I read somewhere that haproxy is capable proxying to tens of thousands connections, so I changed the maxconn from 256 and 128 into 25600 and 12800, respectively. The file need to be changed is haproxy/conf/haproxy.cfg in the application primary (front) gear, lines 28 and 53. Normal user is allowed to make this change (rhc ssh first to your gear).

It seems that open files limit also need to be changed. The indication is these error messages are shown in app-root/logs/haproxy.log :

[WARNING] 018/142729 (27540) : [/usr/sbin/haproxy.main()] Cannot raise FD limit to 51223.
[WARNING] 018/142729 (27540) : [/usr/sbin/haproxy.main()] FD limit (1024) too low for maxconn=25600/maxsock=51223. Please raise 'ulimit-n' to 51223 or more to avoid any trouble.

To fix this, change configuration file /etc/security/limits.conf as root, add this line:

* hard nofile 60000

And stop-start your gear after the change.

Preventing port number exhaustion

High number of load might cause the openshift node to run out of port numbers. The symptom is this error message in /var/log/httpd/error_log (only accessible to root) :

[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed
[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed
[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed
[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed
[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed
[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed
[Mon Feb 01 10:35:37 2016] [error] (99)Cannot assign requested address: proxy: HTTP: attempt to connect to 127.4.238.2:8080 (*) failed

Such error messages are related to bug https://bugzilla.redhat.com/show_bug.cgi?id=1085115, which give us a clue of what really happened. On my system, I used mod_rewrite as reverse proxy module, which have the shortcoming of not reusing socket connections to backend gears. Socket open and closing if done too often would kept many sockets in TIME_WAIT state, which port number by default will not be used for 2 minutes. So one workaround is to enable net.ipv4.tcp_tw_reuse, add this line to the file  /etc/sysctl.conf  as root:
 
net.ipv4.tcp_tw_reuse = 1

And to force in effect immediately, do this in the shell prompt :

sysctl -w net.ipv4.tcp_tw_reuse=1

Conclusion

These changes, done properly, will adjust software limits in the openshift node, enabling the openshift node to receive high amount of requests and/or concurrent connections. Of course you still need to design your database queries to be quick and keep track of memory usage. But these software limits if left unchanged might became a bottleneck for your Openshift application performance.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more