Sunday, October 30, 2016

How to Run X Windows Server inside Docker Container

Background

Sometimes I need to run X Windows-based applications inside Docker containers, and running the server locally is too unpractical because of latency reasons or the working laptop has no X Windows Server. First I tried to create a VirtualBox-based Vnc Server, and it worked fine albeit a little slow, but Docker containers seem to have better memory and disk footprint. So I tried to create Vnc Server running X Windows inside a Docker container. I already tried suchja/x11server (ref) but it has strange problems ignoring cursor keys of my MacBook on webkit page (such as Pentaho Data Integration's Formula page).

Starting point

Many of my Docker images are based on Debian Jessie. So I start from the instructions from this DigitalOcean article : https://www.digitalocean.com/community/tutorials/how-to-set-up-vnc-server-on-debian-8.  This vnc server is based on XFCE Desktop Environment. The steps are basically is to install :
  • xfce4 
  • xfce4-goodies 
  • gnome-icon-theme 
  • tightvncserver
  • iceweasel
After that, run vncserver :[display-number] with a user-specified password.

The initial dockerfile resulted is as follows :
And the resulting docker-compose.yml is :

Problems and problems

The problem is the vncserver fails looking for some 'default' fonts.

Creating vncserver0
Attaching to vncserver0
vncserver0 | xauth:  file /home/vuser/.Xauthority does not exist
vncserver0 | Couldn't start Xtightvnc; trying default font path.
vncserver0 | Please set correct fontPath in the vncserver script.
vncserver0 | Couldn't start Xtightvnc process.
vncserver0 | 
vncserver0 | 
vncserver0 | 30/10/16 09:20:28 Xvnc version TightVNC-1.3.9
vncserver0 | 30/10/16 09:20:28 Copyright (C) 2000-2007 TightVNC Group
vncserver0 | 30/10/16 09:20:28 Copyright (C) 1999 AT&T Laboratories Cambridge
vncserver0 | 30/10/16 09:20:28 All Rights Reserved.
vncserver0 | 30/10/16 09:20:28 See http://www.tightvnc.com/ for information on TightVNC
vncserver0 | 30/10/16 09:20:28 Desktop name 'X' (8732cbbb4029:3)
vncserver0 | 30/10/16 09:20:28 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
vncserver0 | 30/10/16 09:20:28 Listening for VNC connections on TCP port 5903
vncserver0 | Font directory '/usr/share/fonts/X11/misc/' not found - ignoring
vncserver0 | Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
vncserver0 | Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
vncserver0 | Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring
vncserver0 | 
vncserver0 | Fatal server error:
vncserver0 | could not open default font 'fixed'

After comparing the container condition to my VirtualBox VM (that already worked), the VM has downloaded xfonts-100dpi because xfce4 recommends xorg and it requires xfonts-100dpi. The Dockerfile apt line has standard no-install-recommends clause in order to keep images small. So first step is to change the Dockerfile into :


After doing the change, next problem occurred :
Setting up libfontconfig1:amd64 (2.11.0-6.3+deb8u1) ...
Setting up fontconfig (2.11.0-6.3+deb8u1) ...
Regenerating fonts cache... done.
Setting up keyboard-configuration (1.123) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
Configuring keyboard-configuration
----------------------------------

Please select the layout matching the keyboard for this machine.

  1. English (US)
  2. English (US) - Cherokee
  3. English (US) - English (Colemak)
  4. English (US) - English (Dvorak alternative international no dead keys)
  5. English (US) - English (Dvorak)
  6. English (US) - English (Dvorak, international with dead keys)
  7. English (US) - English (Macintosh)
  8. English (US) - English (US, alternative international)
  9. English (US) - English (US, international with dead keys)
  10. English (US) - English (US, with euro on 5)
  11. English (US) - English (Workman)
  12. English (US) - English (Workman, international with dead keys)
  13. English (US) - English (classic Dvorak)
  14. English (US) - English (international AltGr dead keys)
  15. English (US) - English (left handed Dvorak)
  16. English (US) - English (programmer Dvorak)
  17. English (US) - English (right handed Dvorak)
  18. English (US) - English (the divide/multiply keys toggle the layout)
  19. English (US) - Russian (US, phonetic)
  20. English (US) - Serbo-Croatian (US)
  21. Other

Keyboard layout: 

During image build, there is a prompt asking us about keyboard layout, and typing an answer results in hang process. This error is similar to this stackoverflow question. I tried to do the suggestions there (copying to /etc/default/keyboard), but it still hangs. After struggling with many experiments, finally I use this Dockerfile :

How to use vnc server container

First, you build the image:

  • docker-compose build
Then, create the containers :
  • docker-compose up -d
Check the IP of the container (replace vncserver1 with container_name in your docker-compose.yml)
  • docker inspect vncserver1 | grep 172
Tunnel VNC by performing ssh port forwarding to the local IP (example, 172.17.0.9) and port 5903
  • ssh -L 5903:172.17.0.9:5903 user@servername 
Now to view the screen, you could connect to the VNC server by using VNC Viewer or opening vnc://localhost:5903 url in Safari.
For the X Windows based application, you first must grab the magic cookie :
docker exec -it vncserver1 bash
vuser@5eae70a4a75d:~$ ls                                                                                                                                                          
Desktop
vuser@5eae70a4a75d:~$ xauth list
5eae70a4a75d:3  MIT-MAGIC-COOKIE-1  1469d123c6fcb10e0fe8915e3f44ed71
5eae70a4a75d/unix:3  MIT-MAGIC-COOKIE-1  1469d123c6fcb10e0fe8915e3f44ed71
And then connect the docker container where the X Windows based application need to run :
docker exec -it pentahoserver bash
pentaho@3abd451c9b88:~/data-integration$ xauth add 172.17.0.9:3 MIT-MAGIC-COOKIE-1  1469d123c6fcb10e0fe8915e3f44ed71
pentaho@3abd451c9b88:~/data-integration$ DISPLAY=172.17.0.9:3
pentaho@3abd451c9b88:~/data-integration$ export DISPLAY
pentaho@3abd451c9b88:~/data-integration$ ./spoon.sh


And the resulting X Windows Session is shown thru the VNC channel :

Variations

This Dockerfile uses LXDE  :
This one uses OpenBox Window Manager only :

Conclusion

We are able to run X Window Server running inside a Docker container. The resulting images are about 625 MB, which could be a lot smaller if we remove firefox (iceweasel) and use only Openbox Window Manager.

Friday, October 28, 2016

Docker Basic 101

Background

This post would describe notes that results from my initial exploration using docker. Docker could be described as a thin VM. Essentially docker runs processes in a linux host in a semi-isolated environment. It was a brilliant technical accomplishment that exploits several characteristic of running applications in a linux-based OS. First, that the result of package installation is the distribution of package files in certain directories, and changes to certain files. Second, that executable file from one Linux distribution could be run in another Linux distribution provided that all the required shared library and configuration files are in their places.

Basic characteristic of Docker images

Docker images are essentially similar to zip archives, organized as layer over layers. Each additional layer provide new file or changed files. 
Docker image should be portable, means it could be used in different instances of application in different hosts.
Docker images are built using docker entry script and Dockerfile, which :
a. Dockerfile essentially shows the steps to install the application packages or files. After executing each RUN command in the Dockerfile, Docker creates a layer that stores the files add or updated by the RUN command. 
b. Docker entry script shows what command will be executed if the image is being run. This could be a line running existing program, but it could also point to shell script provided.


The Dockerfile is written using a domain specific language, that shows how to build the layers composing the docker image. Some of the Dockerfile syntax are :
  • ADD : add a file to the docker image filesystem, if the file is tar then Docker would extract it first, and it were allowed to use file from local filesystem or an URL from the internet.
  • FROM : refers to one parent docker image, so the files from the parent image are available in the current docker image
  • RUN : executes some program/command inside the Docker environment, Docker would capture file changes resulting from the execution
  • CMD : shows docker entry command, this could point to a shell script already inside the Docker image filesystem or an existing program

Basic Docker usage

There are a few basic docker commands that you need to know when using docker.

  • docker ps : print running processes (containers in docker lingo). The variant is docker ps -a , shows running and terminated containers.
  • docker inspect : show various information of a running docker container. Join with grep to get shorter output, like : docker inspect | grep 172 -> this will filter to show IP addresses. Container could be referred by its name or id.
  • docker history : shows the layers composing the image, also shows no-operation layers such as MAINTAINER.
  • docker exec -it bash : runs a bash shell inside a container interactively. This functions mostly like ssh without needing ssh daemon running inside the container.
  • docker run -d --name [command] [params] : creates a new container from an image, runs it in the background using the entry point. There are other useful parameters before --name, such as : 
    • -v hostdir:containerdir -> mounts a host directory inside the container, this could also works for single file.
    • --link -> put the IP of othercontainer in /etc/hosts so the container could access the other container by specifying its name
    • -e VAR=value -> sets environment variable
  • docker start : starts the container using the entry point
  • docker stop : terminates the process inside the container
  • docker rm : deletes the container, note that named data volumes would not be deleted
Docker images from the internet could be referred when creating new containers, or also inside a Dockerfile. For example, to create a container using an official mariadb image, with data directory mounted from the host, using TokuDB storage engine, use this command :

docker run -d --name maria -e MYSQL_ROOT_PASSWORD=rootpassword -v /data/docker/mysql:/var/lib/mysql mariadb  --plugin-load=ha_tokudb --plugin-dir=/usr/lib/mysql/plugin --default-storage-engine=tokudb --wait_timeout=2000 --max_connections=500

Docker would try to retrieve mariadb image from the official image source (see https://hub.docker.com/_/mariadb/). The docker image syntax is : 
Specifying no tag in the command implies retrieving mariadb:latest.

Docker compose

Host specific configuration are given as parameters of the docker run command, and they could be very long, complicating the container creation command. To simplify this, use docker-compose :
  • docker-compose up -d : reads docker-compose.yml and starts new containers using the compose specification, assuming the docker images already built
  • docker-compose build : reads the docker-compose.yml and create images from specified Dockerfiles if required
  • docker-compose stop : stops the containers specified in docker-compose.yml
  • docker-compose start : starts the process in the containers (similar to docker start) referred by docker-compose.yml
Docker-compose uses docker-compose.yml as the blueprint of the containers. 
This example dockerfile shows various parts of the file: (note that this is a 1.x docker-compose format)

pdi:
    volumes:
     - ~/pentahorepo:/home/pentaho/pentahorepo
     - /tmp/.X11-unix:/tmp/.X11-unix
     - ~/pdidocker/dotkettle:/home/pentaho/data-integration/.kettle
     - ~/phpdocker/public/isms/comm:/home/pentaho/comm
     - ~/phpdocker/public/isms/mobility:/home/pentaho/mobility
    image: y_widyatama:pdi
    container_name: pdiisms
    environment:
     - DISPLAY=172.17.0.1:10.0
    external_links:
     - maria

The volumes clause specifies volumes that are mounted inside the container, like the -v parameter. Specifying a directory in the volume line before the colon (:) means a directory from the host. Specifying a simple name instead will be interpreted as Docker data volume, which would be created if it doesn't exist yet. The directory after the colon shows the target mount directory inside the container.
The image clause specified the base image of the container. Alternatively, a dockerfile clause could be specified instead of image clause.
The environment clause shows additional environment variables that would be set inside the container. 
External_links clause refers to name of existing running container that will be referred in /etc/hosts of the container.

Another example using two containers :

ismsdash-webserver:
  image: phpdockerio/nginx:latest
  container_name: ismsdash-webserver
  volumes:
      - .:/var/www/ismsdash
      - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf
  ports:
   - "80:80"
  links:
   - ismsdash-php-fpm

ismsdash-php-fpm:
  build: .
  dockerfile: php-fpm/Dockerfile
  container_name: ismsdash-php-fpm
  volumes:
    - .:/var/www/ismsdash
    - /data/isms/CDR:/var/www/ismsdash/public/CDR
    - ./php-fpm/php-ini-overrides.ini:/etc/php5/fpm/conf.d/99-overrides.ini
  external_links:
   - maria

   - pdiisms

This example shows two containers with some dependency links. The first, ismsdash-webserver, links to the second container using links clause. The second, ismsdash-php-fpm refers to two existing containers outside this docker-compose.yml file.
The first container is created from a docker image. The second container requires the image to be built first using a specified Dockerfile.
The ports clause specifies port forwarding from the host port to the container port.

Conclusion

Knowledge of several basic commands are neccessary in order to use docker. This post described some commands, hoping that the reader will be able to start using docker using these commands.

Wednesday, October 19, 2016

Running X11 Apps inside Docker on Remote Server

Background

Docker is fast-growing trend that I could no longer ignore, so I tried Docker running in a Linux server machine. Running server app is a breeze inside docker, but I need to run Pentaho Data Integration in the server, which uses X11 display. There is several references about forwarding X11 connection to a Docker container but none works for my setup, which has Quartz XServer running in  Mac OS X laptop and Docker service running in a remote Linux Server.

The usual way

The steps to run X Windowed Applications in Docker containers can be read from Running GUI Apps with Docker and Alternatives to SSH X11 Forwarding for Docker Containers, which essentially is as follows :
  1. Forwarding DISPLAY environment variable to the container
  2. Forwarding directory /tmp/.X11-unix to the container
I already tried such steps with no results, because I need to add another step before these two, that is forwarding X11 connection thru ssh connection to the server (not container). The step is as follows from an OS X laptop :

ssh -X username@servername.domain

Complications


We have DISPLAY set to something like localhost:10.0, and empty /tmp/.X11-unix directory. The reason is that forwarding thru ssh will result in TCP-based X11 service, and it is the opposite of unix socket based X11 service.



Inside the container, we are also unable to connect the localhost:10.0, because in the container the localhost would refer to the container's loopback interface, which is different from the host's loopback. For X11 display connections, the 10.0 means TCP port 6000+10, which as the result of X11 forwarding  only listens on 127.0.0.1 address.

Solutions

So we need to ensure the container able to connect to the forwarded X11 port. One solutions is to use socat :

nohup socat tcp-listen:6010,bind=172.17.0.1,reuseaddr,fork tcp:127.0.0.1:6010 &

Then the DISPLAY variable should be altered to point host IP in the docker network (172.17.0.1)


Unfortunately as you can see, there are authentication problems. I tried to copy the .Xauthority file from the host into the container, but it still failed. We could only understand why after getting it to work :

It seems that because different IP being used, the Xauthority file need a row with 172.17.0.1:10 as the key.

Having Xclock working, so we could also try something bigger like Pentaho's Spoon UI :

Additional notes

In order to get Pentaho Spoon UI working, I need to add gtk library as one layer in the Dockerfile :

USER root

RUN apt-get update && apt-get install -y libgtk2.0-0 xauth x11-apps

Caveat : for the formula editor to work in Spoon UI, we need to install firefox and libraries bridging firefox and swt, which is a topic for another post.

Conclusion

We have found out how to run X11-based apps from a Docker container with the X Server running in a laptop. What good does this for ? In some cases, we need to run a GUI-based app inside a Docker container, without installing vnc and x server inside the container. However we still have to provide X11 libraries inside the container.