Geneos - Gateway Hub installation and cloud-init based system issues

You may encounter problems when installing Gateway Hub 2.4 where the installation fails waiting for orchestrationd to reach a stable state. Further checking in the logs may reveal zookeeper having problems with its election process and unable to contact the other nodes. Going down another level you may then find that zookeeper has only bound itself to listening on 127.0.0.1 and/or ::1 which are the IPv4 and IPv6 values, respectively, for localhost.

Note: If you had the problems described in this article and apply the changes suggested then you MUST start the installation of hub again, after uninstalling what you already have. Trying to start the incomplete installation after these changes will result in only a partially working Hub.

If your nodes are VMs in an environment that uses cloud-init - which is most commercial cloud providers as well as some on-prem set-ups do - you may find that this is because your /etc/hosts file links your node hostname and FQDN to localhost. The quick fix is to edit /etc/hosts and comment out any lines that link your node hostname and FQDN to localhost addresses. This may only last until the next reboot.

cloud-init uses a template file, like the one below, to build the /etc/hosts file so it's important to ensure you also update that file. In the example below you need to comment out the two lines that mention {{fqdn}} :

$ cat /etc/cloud/templates/hosts.redhat.tmpl
## template:jinja
{#
This file /etc/cloud/templates/hosts.redhat.tmpl is only utilized
if enabled in cloud-config. Specifically, in order to enable it
you need to add the following to config:
manage_etc_hosts: True
-#}
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
# /etc/cloud/cloud.cfg or cloud-config from user-data
#
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 {{fqdn}} {{hostname}}                         # <--- THIS LINE
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4

# The following lines are desirable for IPv6 capable hosts
::1 {{fqdn}} {{hostname}}                               # <--- THIS LINE
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

Your system may or may not rebuild your /etc/hosts file on next reboot, so please ensure you update both manually in order to restart your installation.