mongooseimctl ipv6 support

[andywhite37]

MongooseIM version: 6.1.0
Installed from: Kubernetes+MongooseHelm (exported the helm templates to k8s manifests using default values)
Erlang/OTP version: version from helm chart

Background

I'm running MongooseIM 6.1.0 in a Kubernetes cluster in AWS, using the Helm charts from esl/MongooseHelm. (We are not actually using the helm charts as is, but we exported the manifests from the charts with the default values, and have been updating them from there.) I have it running with two replicas, and I believe the server is up and running correctly - it is listening on the desired ports (5222, 5280, etc.).

Our cluster is configured for ipv6 only. I have configured all the listeners to use ip_version = 6, and it all appears to be working.

The hostnames are something like: mongooseim-0.mongooseim.my-identifier.svc.cluster.local and mongooseim-1... for the second replica. I'm able to telnet -6 mongooseim-{n}.mongooseim.my-identifier.svc.cluster.local 5222 from either pod, connecting to itself and the other pod. The telnet connection responds with an XML stream error, which I believe indicates the servers are running. I can also ping6 either host.

Problem

The problem I'm having is mongooseimctl reports Failed RPC connection to the node mongooseim@mongooseim-0.mongooseim.my-identifier.svc.custer.local': nodedown` when I try to run any command.

I've poked around a bit, and my theory is that mongooseimctl is not working correctly with hostnames that resolve to ipv6 addresses. Below are some example commands I've tried running:

# ping fails because host resolves to ipv6 ip address
root@mongooseim-0:/usr/lib/mongooseim/bin# ping mongooseim-0.mongooseim.my-identifier.svc.cluster.local
ping: unknown host

# ping6 works
root@mongooseim-0:/usr/lib/mongooseim/bin# ping6 mongooseim-0.mongooseim.my-identifier.svc.cluster.local
PING mongooseim-0.mongooseim.my-identifier.svc.cluster.local (2600:1f13:525:e70b:f89a::5): 56 data bytes
64 bytes from mongooseim-0.mongooseim.my-identifier.svc.cluster.local: icmp_seq=0 ttl=64 time=0.071 ms
64 bytes from mongooseim-0.mongooseim.my-identifier.svc.cluster.local: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from mongooseim-0.mongooseim.my-identifier.svc.cluster.local: icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from mongooseim-0.mongooseim.my-identifier.svc.cluster.local: icmp_seq=3 ttl=64 time=0.040 ms
^C--- mongooseim-0.mongooseim.my-identifier.svc.cluster.local ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.040/0.049/0.071/0.000 ms

# telnet -6 works
root@mongooseim-0:/usr/lib/mongooseim/bin# telnet -6 mongooseim-0.mongooseim.my-identifier.svc.cluster.local 5222
Trying 2600:1f13:525:e70b:f89a::5...
Connected to mongooseim-0.mongooseim.my-identifier.svc.cluster.local.
Escape character is '^]'.
<stream:error><connection-timeout xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error></stream:stream>Connection closed by foreign host.

# mongooseimctl fails - maybe because RPC connection only works with ipv4?
root@mongooseim-0:/usr/lib/mongooseim/bin# mongooseimctl status
Failed RPC connection to the node 'mongooseim@mongooseim-0.mongooseim.my-identifier.svc.cluster.local': nodedown

Commands to start a MongooseIM node:
  start           Start a MongooseIM node as daemon (detached from terminal)
  debug           Attach an interactive Erlang shell to a running MongooseIM node
  live            Start MongooseIM node in live (interactive) mode
  foreground      Start MongooseIM node in foreground (non-interactive) mode
MongooseIM cluster management commands:
  join_cluster other_node_name                Add current node to cluster
  leave_cluster                               Make the current node leave the cluster
  remove_from_cluster other_node_name         Remove dead node from the cluster
Extra Commands:
  bootstrap           Executes MongooseIM init scripts (used for initial configuration)
  print_install_dir   Prints path to MongooseIM release directory
  escript             Runs escript command using embedded Erlang Runtime System

# mongooseim ping fails too
root@mongooseim-0:/usr/lib/mongooseim/bin# ./mongooseim ping
Node 'mongooseim@mongooseim-0.mongooseim.my-identifier.svc.cluster.local' not responding to pings.

Question

Am I on the right track - is it expected that mongooseimctl wouldn't work with ipv6?

[adamramage]

Hi @andywhite37, reach out if you like, i spent a lot of time getting MongooseIM going in k8's, in particular with Helm.

There are a lot of little gotchas to overcome, the base helm deployments don't really work too well. and i had to tear the charts apart and the start script.

I cant share my helm charts here as they are wrapped up with our deployment and a bit of a PITA to unravel without pulling usefulness out.

[andywhite37]

Hey @adamramage - I'm glad to know you're out there and have worked through similar things, so we're not alone! I think we're still fiddling/debugging to see if we can narrow down the problem, but I will reach out if we get totally stuck.

As a caveat to all this, I'm not an Erlang developer, so I'm having to learn some of this stuff as I go.

The thing we looked at today was related to the Erlang dist communcation on port 4369 and 9100. We had a theory that we didn't have the right ports open in our AWS security group, so we made sure 4369 and 9100 were open, which we think they are.

We noticed on the mongooseim machines, if we run ss -l, there is something listening on 0:0:0:0:9100 (which is inet4 vs inet6). I wondered if that was a problem for us, because our long hostnames map to ipv6 addresses, so if they are trying to communicate with eachother via ipv4, maybe they can't connect on port 9100.

I tried setting -proto_dist inet6_tcp in the vm.args / vm.dist.args, thinking that that might force the communication to happen via inet6, and that changed the error we got from mongooseimctl to this: Protocol 'inet6_tcp': register/listen error: eaddrinuse:

# mongooseimctl fails with eaddrinuse
root@mongooseim-0:/# mongooseimctl
Protocol 'inet6_tcp': register/listen error: eaddrinuse

# Trying to run mongooseim ping gives more error info
root@mongooseim-0:/# ./usr/lib/mongooseim/bin/mongooseim ping
=INFO REPORT==== 20-Sep-2023::21:16:44.606961 ===
Protocol 'inet6_tcp': register/listen error: eaddrinuse

=SUPERVISOR REPORT==== 20-Sep-2023::21:16:44.607088 ===
    supervisor: {local,net_sup}
    errorContext: start_error
    reason: {'EXIT',nodistribution}
    offender: [{pid,undefined},
               {id,net_kernel},
               {mfargs,{net_kernel,start_link,
                                   [#{clean_halt => false,
                                      name =>
                                          'mongooseim_maint_154@mongooseim-0.mongooseim.my-identifier.svc.cluster.local',
                                      name_domain => longnames,
                                      net_tickintensity => 4,
                                      net_ticktime => 60,
                                      supervisor => net_sup_dynamic}]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,2000},
               {child_type,worker}]

=CRASH REPORT==== 20-Sep-2023::21:16:44.607221 ===
  crasher:
    initial call: net_kernel:init/1
    pid: <0.82.0>
    registered_name: []
    exception exit: {error,badarg}
      in function  gen_server:init_it/6 (gen_server.erl, line 835)
    ancestors: [net_sup,kernel_sup,<0.47.0>]
    message_queue_len: 0
    messages: []
    links: [<0.79.0>]
    dictionary: [{longnames,true}]
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 28
    reductions: 3171
  neighbours:

escript: exception error: no match of right hand side value 
                 {error,
                     {{shutdown,
                          {failed_to_start_child,net_kernel,
                              {'EXIT',nodistribution}}},
                      {child,undefined,net_sup_dynamic,
                          {erl_distribution,start_link,
                              [#{clean_halt => false,
                                 name =>
                                     'mongooseim_maint_154@mongooseim-0.mongooseim.my-identifier.svc.cluster.local',
                                 name_domain => longnames,
                                 net_tickintensity => 4,net_ticktime => 60,
                                 supervisor => net_sup_dynamic}]},
                          permanent,false,1000,supervisor,
                          [erl_distribution]}}}
root@mongooseim-0:/# 

The annoyance here is that we are not able to use mongooseimctl, even though the server appears to be working normally (I can can successfully connect to it via an XMPP client, the graphql endpoints are working, etc.)

[adamramage]

its 11pm here so bear with me..

Theres a few problems to address.

1. We had to include a service account

{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ include "mongooseim.serviceAccountName" . }}
  labels:
    {{- include "mongooseim.labels" . | nindent 4 }}
  {{- with .Values.serviceAccount.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
{{- end }}

2. a lot of the charts were incomplete or missing refs which made deploying difficult. I actually just made a new chart from scratch doing a helm create then bringing in what I needed. I'll see if i can sanitize what we use and release it for you. i did and still do question if the example charts actually exist. i can probably attribute some grey hair to trying to understand how they could work with minor tweaks.
3. If you are using a range of ports for your inet-dist, you need to spec these individually (silly k8's doesnt let you do ranges)
4. there are problems with the start script absolutely chewing up configurations. I rewrote this to remove the sed find/replaces in my vm.args as for the life of me i could not get it to work. similarly, i changed things to let me control clustering and debug things a little. We have 12 xmpp instances supporting 1.6m clients and a DMZ set of nodes so we mod this to form the cluster in a different way with particular nodes having specific roles and configurations plus a stack of healthchecks and monitoring
5. MongooseIM / Erlang is very very particular on hostnames. Throw a debug stop in the container startup and check you can:

• get your hostname -s
• get your hostname -f
• resolve your hostname -f to ip (this is likely going to be dependent on you having the service account but may also depend on how your cluster DNS works. We had to mess around a little there i think but good old MS teams is letting me down searching for what change it was.)

6. We are deployed in AWS on EKS managing via helm and ArgoCD
7. We burned our own docker images to use, it is based on the latest and same version which the helm chart uses but we throw in some additional scripts and files plus overwrite the start script as its junk.
8. it does sound a little like there could just be confusion with names etc. Can you do a dns resolve on the fqdn of the pod ?
   eg

nslookup mongooseim-0.mongooseim.my-identifier.svc.cluster.local

The error you posted shows the problem lies with mongoooseim exec trying to connect to the epdm process. It does this with the nodetool exec but will go and look at your vm.args and call your hostname to figure out where to talk to. remember the start script will eat this config file up and rewrite the -name / -sname fields you spec.

Check your vm.args is correctly setup, check you can do something like

$ mongooseim ping mongooseim@mongooseim-0.mongooseim.my-identifier.svc.cluster.local

our naming syntax is like

mongooseim-ctrl-0.ctrl.mongooseim.svc.cluster.local
where:

mongooseim-ctl-0 is the pod
ctrl is the service
mongooseim is the namespace
svc.cluster.local is the service domain

Check this out https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-sethostnameasfqdn-field

9. Silly question, do you have node exporter installed? it typically uses tcp 9100 for its metrics end point. we moved our port range away from 9100 for this reason as we use a service monitor which was targeting 9100. also, if you use a service monitor, it can be configured to scrape targets on that port.

10. you mentioned ports being open and AWS security groups, this should be all in the k8's cluster network so unless your clustering is leaving k8's to talk to nodes elsewhere, it should have at least for the most part in the same namespace all ports open.

11. config files we use are as follows.

  vm.dist.args: |-

    -proto_dist inet_tls
    -ssl_dist_opt server_certfile   /usr/lib/mongooseim/priv/ssl/mongooseim.pem client_certfile   /usr/lib/mongooseim/priv/ssl/mongooseim.pem
    server_keyfile    /usr/lib/mongooseim/priv/ssl/iq3root.pem  client_keyfile   /usr/lib/mongooseim/priv/ssl/iq3root.pem
    server_cacertfile /usr/lib/mongooseim/priv/ssl/ca.pem client_cacertfile /usr/lib/mongooseim/priv/ssl/ca.pem

    client_verify     verify_peer
    server_verify     verify_peer
    server_fail_if_no_peer_cert true

  vm.args: |-
      ## Name of the node.
      -name mongooseim

      ## Cookie for distributed erlang
      -setcookie ejabberd

      ## Enable more processes (10M)
      +P 10000000

      ## Increase number of concurrent ports/sockets
      -env ERL_MAX_PORTS 250000

      ## Tweak GC to run more often
      -env ERL_FULLSWEEP_AFTER 2

      ## With lager sasl reports are redundant so turn them off
      -sasl sasl_error_logger false

      -kernel inet_dist_listen_min 9200
      -kernel inet_dist_listen_max 9209

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ include "mongooseim.serviceName" . }}
  labels:
    {{- include "mongooseim.labels" . | nindent 4 }}
spec:
{{/*  type: {{ .Values.service.type }}*/}}
  ports:
    - name: epmd
      port: 4369
      targetPort: 4369
    - name: c2s
      port: 5222
      targetPort: 5222
    - name: s2s
      port: 5269
      targetPort: 5269
    - name: bosh
      port: 5280
      targetPort: 5280

    - name: erl-dist-9200
      targetPort: 9200
      port: 9200

    - name: erl-dist-9201
      targetPort: 9201
      port: 9201

    - name: erl-dist-9202
      targetPort: 9202
      port: 9202

    - name: erl-dist-9203
      targetPort: 9203
      port: 9203

    - name: erl-dist-9204
      targetPort: 9204
      port: 9204

    - name: erl-dist-9205
      targetPort: 9205
      port: 9205

    - name: erl-dist-9206
      targetPort: 9206
      port: 9206

    - name: erl-dist-9207
      targetPort: 9207
      port: 9207

    - name: erl-dist-9208
      targetPort: 9208
      port: 9208

    - name: erl-dist-9209
      targetPort: 9209
      port: 9209

  clusterIP: None
  selector:
    {{- include "mongooseim.selectorLabels" . | nindent 4 }}

statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ include "mongooseim.fullname" . }}
  labels:
    {{- include "mongooseim.labels" . | nindent 4 }}
spec:
  revisionHistoryLimit: 2
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  updateStrategy:
    type: RollingUpdate
  serviceName: {{ include "mongooseim.serviceName" . }}
  selector:
    matchLabels:
      {{- include "mongooseim.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "mongooseim.selectorLabels" . | nindent 8 }}
    spec:
      {{- if .Values.waitForController }}
      initContainers:
        - name: wait-for-controller-before-starup
          image: busybox
          command: [ "sh", "-c", "until nc -z mongooseim-ctrl-0.ctrl.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local 4369 > /dev/null; do echo Waiting for master.; sleep 2; done;" ]
      {{- end }}
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "mongooseim.serviceAccountName" . }}
      subdomain: stbnet-dev
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: MASTER_ORDINAL
              value: "0"
            - name: NODE_TYPE
              value: "name"
            - name: NODE_NAME
              value: {{ .Values.nodeName }}
          ports:
            - name: epmd
              containerPort: 4369
            - name: c2s
              containerPort: 5222
            - name: s2s
              containerPort: 5269
            - name: bosh
              containerPort: 5280
            - name: erl-dist-9200
              containerPort: 9200
            - name: erl-dist-9201
              containerPort: 9201
            - name: erl-dist-9202
              containerPort: 9202
            - name: erl-dist-9203
              containerPort: 9203
            - name: erl-dist-9204
              containerPort: 9204
            - name: erl-dist-9205
              containerPort: 9205
            - name: erl-dist-9206
              containerPort: 9206
            - name: erl-dist-9207
              containerPort: 9207
            - name: erl-dist-9208
              containerPort: 9208
            - name: erl-dist-9209
              containerPort: 9209
{{/*          livenessProbe:*/}}
{{/*            httpGet:*/}}
{{/*              path: /*/}}
{{/*              port: http*/}}
          readinessProbe:
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 2
            failureThreshold: 1
            exec:
              command:
                - grep
                - -q
                - started
                - /usr/lib/mongooseim/var/status
          volumeMounts:
{{/*            - name: mnesia*/}}
{{/*              mountPath: /var/lib/mongooseim*/}}

            - name: {{ include "mongooseim.fullname" . }}-start
              mountPath: /mongoose-config/start.sh
              subPath: start.sh

            - name: {{ include "mongooseim.fullname" . }}-vmargs
              mountPath: /mongoose-config/vm.args
              subPath: vm.args

            - name: {{ include "mongooseim.fullname" . }}-config
              mountPath: /mongoose-config/mongooseim.toml
              subPath: mongooseim.toml

            - name: {{ include "mongooseim.fullname" . }}-appconfig
              mountPath: /mongoose-config/app.config
              subPath: app.config

          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      volumes:
        - name: {{ include "mongooseim.fullname" . }}-start
          configMap:
            name: {{ include "mongooseim.fullname" . }}-start

        - name: {{ include "mongooseim.fullname" . }}-vmargs
          configMap:
            name: {{ include "mongooseim.fullname" . }}-vmargs

        - name: {{ include "mongooseim.fullname" . }}-config
          configMap:
            name: {{ include "mongooseim.fullname" . }}-config

        - name: {{ include "mongooseim.fullname" . }}-appconfig
          configMap:
            name: {{ include "mongooseim.fullname" . }}-appconfig

      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
  volumeClaimTemplates:
    - metadata:
        name: mnesia
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi

[adamramage]

Give this a whirl.. It was used to get a MVP going for us and it does work, your mileage may vary.

Any issues, let me know. feel free to contribute to it, I'll continue to maintain it as a bit of the templating is incomplete

https://github.com/adamramage/HA-MongooseIM-Helm

[andywhite37]

@adamramage this is amazing info, thanks so much for taking the time to put all this out there. I'll check all of this out and report back!

[adamramage]

@adamramage this is amazing info, thanks so much for taking the time to put all this out there. I'll check all of this out and report back!

no dramas. keep me posted how you go. keen to see if you get it going

[andywhite37]

I'm becoming pretty convinced that my problems are more related to ipv6 support as opposed to anything in Kubernetes/etc. I posted this other issue #4132 to see if anyone can provide any other clues.

[arcusfelis]

@adamramage We appreciate PRs or feedback.
We usually add features to Helm charts when we need them, or some client needs them.

So, if there is a PR with clear explanation and some way to test it, it sounds great.
There are very complex Helm charts for other projects, so adding complexity is "easy". Managing complexity later or adding the right features is hard.

[adamramage]

@adamramage We appreciate PRs or feedback. We usually add features to Helm charts when we need them, or some client needs them.

So, if there is a PR with clear explanation and some way to test it, it sounds great. There are very complex Helm charts for other projects, so adding complexity is "easy". Managing complexity later or adding the right features is hard.

No worries. the repo i forked was adding changes specific to our uses and requirements. merging them back in might break features for other users. we're still investigating / tuning the charts and deployment so a PR might follow