blog.polynom.me/content/blog/2020-02-13-Running-Prosody-traefik.md

TL;DR: This post is about running prosody with HTTPS services both on port 443. If you only care about the how, then jump to Considerations and read from there.

Introduction

As part of my "road to FOSS" I set up my own XMPP server using prosody. While it has been running fine for quite some time, I noticed, while connected to a public Wifi, that my server was unreachable. At that time I was panicing because I thought prosody kept crashing for some reason. After using my mobile data, however, I saw that I could connect to my server. The only possible explanation I came up with is that the provider of the public Wifi is blocking anything that is not port 53, 80 or 443. (Other ports I did not try)

My solution: Move prosody's C2S - Client to Server - port from 5222 to either port 53, 80 or 443. Port 53 did not seem like a good choice as I want to keep myself the possibilty of hosting a DNS server. So the only choice was between 80 and 443.

Considerations

Initially I went with port 80 because it would be the safest bet: You cannot block port 80 while still allowing customers to access the web. This would have probably worked out, but I changed it to port 443 later-on. The reason being that I need port 80 for Let's Encrypt challenges. Since I use nginx as a reverse proxy for most of my services, I thought that I can multiplex port 80 between LE and prosody. This was not possible with nginx.

So I discoverd traefik since it allows such a feat. The only problem is that it can only route TCP connections based on the SNI. This requires the XMPP connection to be encrypted entirely, not after STARTTLS negotiation, which means that I would have to configure prosody to allow such a connection and not offer STARTTLS.

Prosody

Prosody has in its documentation no mentions of direct TLS which made me guess that there is no support for it in prosody. After, however, asking in the support group, I was told that this feature is called legacy_ssl.

As such, one only has to add

-- [...]

legacy_ssl_ports = { 5223 }
legacy_ssl_ssl = {
    [5223] = {
        key = "/path/to/keyfile";
        certificate = "/path/to/certificate";
    }
}

-- [...]

Note: In my testing, prosody would not enable legacy_ssl unless I explicitly set legacy_ssl_ports.

When prosody tells you that it enabled legacy_ssl on the specified ports, then you can test the connection by using OpenSSL to connect to it: openssl s_client -connect your.domain.example:5223. OpenSSL should tell you the data it can get from your certificate.

traefik

In my configuration, I run prosody in an internal Docker network. In order to connect it, in my case port 5223, to the world via port 443, I configured my traefik to distinguish between HTTPS and XMPPS connections based on the set SNI of the connection.

To do so, I firstly configured the static configuration to have port 443 as an entrypoint:

# [...]

entrypoints:
    https:
        address: ":443"

# [...

For the dynamic configuration, I add two routers - one for TCP, one for HTTPS - that both listen on the entrypoint https. As the documentation says, "If both HTTP routers and TCP routers listen to the same entry points, the TCP routers will apply before the HTTP routers.". This means that traefik has to distinguish the two somehow.

We do this by using the Host rule for the HTTP router and HostSNI for the TCP router.

As such, the dynamic configuration looks like this:

tcp:
    routers:
        xmpps:
            entrypoints:
                - "https"
            rule: "HostSNI(`xmpps.your.domain.example`)"
            service: prosody-dtls
            tls:
                passthrough: true
        # [...]
    services:
        prosody-dtls:
            loadBalancer:
                servers:
                    - address: "<IP>:5223"

http:
    routers:
        web-secure:
            entrypoints:
                - "https"
            rule: "Host(`web.your.domain.example`)"
            service: webserver

It is important to note here, that the option passthrough has to be true for the TCP router as otherwise the TLS connection would be terminated by traefik.

Of course, you can instruct prosody to use port 443 directly, but I prefer to keep it like this so I can easily see which connection goes to where.

HTTP Upload

HTTP Upload was a very simple to implement this way. Just add another HTTPS route in the dynamic traefik configuration to either the HTTP port of prosody, which would terminate the TLS connection from traefik onwards, or the HTTPS port, which - if running traefik and prosody on the same host - would lead to a possible unnecessary re-encryption of the data.

This means that prosody's configuration looks like this:

[...]
-- Perhaps just one is enough
http_ports = { 5280 }
https_ports = { 5281 }

Component "your.domain"
    -- Perhaps just one is required, but I prefer to play it safe
    http_external_url = "https://http.xmpp.your.domain"
    http_host = "http.xmpp.your.domain"
[...]

And traefik's like this:

[...]
http:
  routers:
    prosody-https:
      entrypoints:
        - "https"
      rule: "Host(`http.xmpp.your.domain`)"
      service: prosody-http

  services:
    prosody-http:
      loadBalancer:
        servers:
          - "http://prosody-ip:5280"
[...]

DNS

In order for clients to pick this change up, one has to create a DNS SRV record conforming to XEP-0368.

This change takes some time until it reaches the clients, so it would be wise to keep the regular STARTTLS port 5222 open and connected to prosody until the DNS entry has propagated to all DNS servers.

Caveats

Of course, there is nothing without some caveats; some do apply here.

This change does not neccessarilly get applied to all clients automatically. Clients like Conversations and its derivatives, however, do that when they are reconnecting. Note that there may be clients that do not support XEP-0368 which will not apply this change automatically, like - at least in my testing - profanity.

Also there may be some clients that do not support direct TLS and thus cannot connect to the server. In my case, matterbridge was unable to connect as it, without further investigation, can only connect with either no TLS or with STARTTLS.

Conclusion

In my case, I run my prosody server like this:

    <<WORLD>>-------------+
        |                 |
    [traefik]-------------/|/--------------+
        |                 |               |
  {xmpp.your.domain}    [5269]    {other.your.domain}
    [443 -> 5223]         |          [443 -> 80]
{http.xmpp.your.domain}   |               |
    [443 -> 5280]         |               |
        |                 |               |
    [prosody]-------------+            [nginx]

As I had a different port for prosody initially (80), I had to wait until the DNS records are no longer cached by other DNS servers or clients. This meant waiting for the TTL of the record, which in my case were 18000 seconds, or 5 hours.

The port 5222 is, in my case, not reachable from the outside world but via my internal Docker compose network so that my matterbridge bridges still work.