Inventor's Paradox

Background Open VSwitch or ovs is the software that openshift used to ensure pods in the openshift cluster could talk to each other using cluster-internal IP addresses. In default settings, hosts outside the cluster are not be able to connect directly to pods; and haproxy (running in  'openshift router' pods) sofware listening on host port is doing the work of distributing http request into each and every application's pods.  In order to follow this blog post you will need root-level OS access on the hosts / openshift nodes, so it is geared towards system administrator of the openshift platform, not the casual application developer. Examining OVS bridge [root@node3 ~]# ovs-vsctl list-br br0 So we know the OVS bridge is named br0. [root@node1 ~]# ovs-ofctl -O OpenFlow13 dump-ports-desc br0 OFPST_PORT_DESC reply (OF1.3) (xid=0x2):  1(vxlan0): addr:62:b2:af:23:90:ea      config:     0      state:      0   ...

Background A few months ago we installed openshift origin cluster in a few of our servers. We have tried openshift M3 with good results. Upgrading to M4 and deploying several live apps there, we found gears that hangs each week or two, that cannot be resolved by stopping and starting the gear. After the third time this is getting troublesome. Symptoms - Application URL doesn't respond - haproxy-status URL shows gears in red status before eventually hangs itself - Full processlist done by root in the node will hang when the process in question is being printed Investigation Before we understood the root cause, restarting the VM is sometimes the only solution if the problem occurs. Strace-ing the hanging ps shows that ps is stopped when trying to read some memory. A few instances of the problem 'heals' by itself after one day. In some instances killling the process with the cmdline anomaly will solve the problem. The real eye opening is the blog post...