Run your own Ibex Farm using Docker

Nov 5 2018 (updated Nov 13 2018 to use Caddy)

Ibex is a tool for running psycholinguistic experiments online. It was designed to be easy to run on the shared hosting solutions cheaply available in 2007. This was achieved by implementing the backend as a single Python CGI script.

The Ibex Farm, introduced around 2010, lowered the barrier to entry by providing a web interface for creating Ibex experiments. The Ibex Farm is written in Perl using the Catalyst framework.

Hosting an Ibex Farm instance used to be much more complicated than hosting a single Ibex experiment. I've now built a Docker image to make it easier. If you're already hosting Ibex experiments yourself, you might want to consider hosting your own Ibex Farm instance instead.

This guide assumes that you'll run your Ibex Farm instance on a linode running CentOS 7. As Docker does most of the heavy lifting, the instructions below should be easy to adapt to other environments. That being said, you can run a suitable linode for just $5/month, so I'd go with a linode unless you have a good reason not to.

Why set up your own Ibex Farm?

Do I need permission?

This is a question I've had more often than I've expected. The Ibex Farm is open source software. You don't need my permission to run your own instance of it. I request that

You should, however, feel free to post questions to the Google group.

Creating a linode

Linode is one of many cloud hosting providers. It's cheap and easy to use compared to more sophisticated options like AWS.

If you anticipate hosting large number of experiments (more than a few hundred), then check out the ‘Storage Space’ section below.

After creating a linode account, go through the following steps:

If the IP address of your linode is e.g. 192.192.192.192, you can ssh in as follows:

ssh root@192.192.192.192

Setting up the linode

Ssh in as root (as in the example above). Execute the following commands:

adduser ibex
passwd ibex
usermod -aG wheel ibex
yum update -y
shutdown -r now

The linode will now reboot, terminating your ssh session. In a couple of minutes, ssh in again as user ibex (e.g. ssh ibex@192.192.192.192). Install the following packages:

sudo yum install -y git docker
sudo yum install -y epel-release
sudo yum install -y tinyproxy jq

Note that the use of a separate invocation of yum to install epel-release is necessary. We can now start the docker daemon and configure tinyproxy:

sudo systemctl start docker
sudo systemctl enable docker.service
sudo sed -i 's/^Port[ \t]*[0-9][0-9]*$/Port 80/' /etc/tinyproxy/tinyproxy.conf
sudo sed -i 's/^Allow[ \t].*$//g' /etc/tinyproxy/tinyproxy.conf
sudo bash -c 'printf "\nReverseOnly Yes\nReverseMagic Yes\nReversePath \"/\" \"http://localhost:8888/\"\n" >> /etc/tinyproxy/tinyproxy.conf'

The Ibex Farm's data will be stored in a separate container. The data is manipulated by the apache user, since the Ibex Farm runs inside apache using mod_perl. The apache user and apache group both have id 987654 within the container. We can create a dapache (‘docker apache’) user and group in the host CentOS system with the same id:

sudo groupadd dapache -g 987654
sudo useradd -g dapache -u 987654 -s /sbin/nologin dapache

Create a volume called ibexdata to hold the Ibex Farm data:

sudo docker volume create ibexdata

Create /deploy and /ibexexps directories within the volume and chown everything to dapache:dapache:

sudo mkdir `sudo docker volume inspect ibexdata | jq -r .[0].Mountpoint`/ibexexps
sudo mkdir `sudo docker volume inspect ibexdata | jq -r .[0].Mountpoint`/deploy
sudo chown -R dapache:dapache `sudo docker volume inspect ibexdata | jq -r .[0].Mountpoint`

The next steps are to open port 80 and start tinyproxy:

sudo firewall-cmd --zone=public --permanent --add-service=http
sudo firewall-cmd --reload
sudo service tinyproxy start

Pull the Ibex Farm docker container:

sudo docker pull alexdrummond/ibexfarm

Configure the webmaster email address and webmaster name for this instance:

echo 'IBEXFARM_webmaster_email="example@example.com"' >> ~/ibexenv.sh
echo 'IBEXFARM_webmaster_name="Some person"' >> ~/ibexenv.sh

Source the preceding definitions and add them to the system profile:

set -o allexport ; source ~/ibexenv.sh ; set +o allexport
sudo bash -c 'echo "set -o allexport ; source /home/ibex/ibexenv.sh ; set +o allexport" > /etc/profile.d/ibex.sh'

Allow systemd to manage docker containers:

sudo setsebool -P container_manage_cgroup on

Create a systemd service called ibexfarm-server to run the docker container:

printf "[Unit]\nDescription=Ibex Farm server\nWants=docker.service\nAfter=docker.service\n[Service]\nLimitNOFILE=8192\nEnvironmentFile=/home/ibex/ibexenv.sh\nUser=root\nRestart=always\nRestartSec=10\nExecStartPre=-/usr/bin/bash /usr/bin/docker stop ibexfarm_server\nExecStartPre=-/usr/bin/docker rm ibexfarm_server\nExecStartPre=/usr/bin/bash -c 'cat /home/ibex/ibexenv.sh | xargs -n 1 echo > /tmp/ibexenv_docker'\nExecStart=/usr/bin/docker run --name ibexfarm_server --restart no -p 127.0.0.1:8888:80 -e IBEXFARM_url_prefix=/ -e IBEXFARM_port=80 -e IBEXFARM_config_url_envvar=IBEXFARM_config_url -e IBEXFARM_config_url=http://localhost/ajax/config --env-file /tmp/ibexenv_docker -v ibexdata:/ibexdata docker.io/alexdrummond/ibexfarm\nExecStop=/usr/bin/docker stop -t 2 ibexfarm_server\n[Install]\nWantedBy=multi-user.target\n" | sudo bash -c 'tee > /etc/systemd/system/ibexfarm-server.service'
sudo systemctl daemon-reload

Finally, start Ibex Farm using the following commands:

sudo systemctl start ibexfarm-server.service
sudo systemctl enable ibexfarm-server.service

At this point, you should be able to see the Ibex Farm homepage. For example, if the IP address of your linode is 192.192.192.192, go to:

http://192.192.192.192/

You may wish to create an example user with an example experiment, so that the link on the homepage isn't broken.

Storage space

If you anticipate hosting lots of experiments on your Ibex Farm instance, you should store the ibexdata volume on a linode volume rather than on the root filesystem of the linode. Whereas there's no straightforward way to enlarge a linode's root filesystem, it's easy to enlarge a linode volume. See the docker documentation (and in particular the --opt device option to docker volume create) for more info. Be sure to add /ibexexps and /deploy dirs to your volume before running the docker container, and then chown -R everything to dapache:dapache.

Setting up https

The instructions in this section make use of Caddy as a reverse http(s) proxy. If you follow these instructions, you'll be using Caddy via the free personal license. This license is “appropriate for academic research, personal projects, websites that aren't for profit, and development at home.” More details here.

You'll need to get a domain name pointing to the IP of your linode before following these instructions.

Remember that DNS propagation can take a while, so wait for a few hours after you've associated your domain name with your linode's IP address.

This section steps through the process of setting up https using a free letsencrypt certificate. You can either create and manage this certificate automatically via Caddy, or create the certificate manually — in which case you'll be responsible for renewing it when it expires. There's little reason to choose the manual option if you're using letsencrypt, but you might find the relevant instructions useful if you have another SSL certificate that you'd like to use.

Begin by halting tinyproxy and disabling its service:

sudo service tinyproxy stop
sudo systemctl disable tinyproxy

Open port 443:

sudo firewall-cmd --zone=public --permanent --add-service=https
sudo firewall-cmd --reload

Define your hostname:

echo 'IBEXFARM_host="my.domain.name"' >> ~/ibexenv.sh
set -o allexport ; source ~/ibexenv.sh ; set +o allexport

Install caddy:

cd ~
curl -o caddy.tar.gz https://caddyserver.com/download/linux/amd64?license=personal&telemetry=off 
sudo mkdir /caddy
sudo useradd -r -d /caddy -M -s /sbin/nologin caddy
sudo chown caddy:caddy /caddy
sudo tar -xzf caddy.tar.gz -C /caddy
sudo chown -R caddy:caddy /caddy
rm ~/caddy.tar.gz
sudo -u caddy mkdir /caddy/ssl
sudo setcap CAP_NET_BIND_SERVICE=+eip /caddy/caddy

Create a systemd service for Caddy:

printf "[Unit]\nDescription=Caddy HTTP/2 web server\nDocumentation=https://caddyserver.com/docs\nAfter=network-online.target\nWants=network-online.target systemd-networkd-wait-online.service\n[Service]\nRestart=on-abnormal\nUser=caddy\nGroup=caddy\nEnvironment=CADDYPATH=/caddy/ssl\nEnvironmentFile=/home/ibex/ibexenv.sh\nExecStart=/caddy/caddy -log stdout -agree=true -conf=/caddy/caddy.conf\nExecReload=/bin/kill -USR1 \$MAINPID\nKillMode=mixed\nKillSignal=SIGQUIT\nTimeoutStopSec=5s\nLimitNOFILE=1048576\nLimitNPROC=512\nPrivateTmp=true\nPrivateDevices=true\nReadWriteDirectories=/caddy/ssl\nCapabilityBoundingSet=CAP_NET_BIND_SERVICE\nAmbientCapabilities=CAP_NET_BIND_SERVICE\nNoNewPrivileges=true\n[Install]\nWantedBy=multi-user.target\n" | sudo bash -c 'tee > /etc/systemd/system/caddy.service'
sudo systemctl daemon-reload

You can now choose either ‘Option 1’ or ‘Option 2’ below, then follow the instructions in the ‘Start Caddy’ section.

Option 1: Automatic SSL cert management via Caddy

Make sure that you've set IBEXFARM_webmaster_email to a real email address in ~/ibexenv.sh. This email address will be associated with your SSL cert.

Configure Caddy as follows:

sudo -u caddy bash -c 'printf "{\$IBEXFARM_host} {\n  log syslog\n  proxy / http://127.0.0.1:8888\n  tls {\$IBEXFARM_webmaster_email}\n}\n" > /caddy/caddy.conf'

Option 2: Manual SSL cert management

Obtain a letsencrypt certificate:

sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
cd /opt/letsencrypt
sudo -H ./letsencrypt-auto certonly --standalone -d "$IBEXFARM_host"

After answering some questions at the prompt, you should see output like the following:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/my.domain.name/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/my.domain.name/privkey.pem

   ...
   ...

Configure Caddy as follows:

sudo -u caddy bash -c 'printf "http://{\$IBEXFARM_host} {\n  log syslog\n  redir https://{\$IBEXFARM_host}{uri}\n}\n\nhttps://{\$IBEXFARM_host} {\n  log syslog\n  proxy / http://127.0.0.1:8888\n  tls /caddy/ssl/fullchain.pem /caddy/ssl/privkey.pem\n}\n" > /caddy/caddy.conf'

Copy the cert files to the location where Caddy expects them:

sudo find "/etc/letsencrypt/live/$IBEXFARM_host" -name '*.pem' -exec cp {} /caddy/ssl \;
sudo chown -R caddy:caddy /caddy/ssl
sudo chmod 600 /caddy/ssl/*.pem

Start Caddy

Finally, start and enable the Caddy systemd service:

sudo systemctl start caddy.service
sudo systemctl enable caddy.service

You should now have access to your Ibex Farm instance over https. Caddy has been configured to redirect any http requests to https.