blog.polynom.me/content/2020-02-13-Running-Prosody-traefik.md

+++
title = "Running Prosody on Port 443 Behind traefik"
date = "2020-02-13"
template = "post.html"
aliases = [ "/Running-Prosody-traefik.html" ]
+++
*TL;DR: This post is about running prosody with HTTPS services both on port 443. If you only care about the how, then jump to*
**Considerations** *and read from there.*

<!-- more -->

# Introduction
As part of my [*"road to FOSS"*](https://blog.polynom.me/Road-to-Foss.html) I
set up my own XMPP server using *prosody*. While it has been running fine for
quite some time, I noticed, while connected to a public Wifi, that my
server was unreachable. At that time I was panicing because I thought prosody
kept crashing for some reason. After using my mobile data, however, I saw
that I **could** connect to my server. The only possible explanation I came
up with is that the provider of the public Wifi is blocking anything that
is not port 53, 80 or 443. *(Other ports I did not try)*

My solution: Move *prosody*'s C2S - *Client to Server* - port from 5222 to
either port 53, 80 or 443. Port 53 did not seem like a good choice as I
want to keep myself the possibilty of hosting a DNS server. So the only
choice was between 80 and 443.

# Considerations
Initially I went with port 80 because it would be the safest bet: You cannot
block port 80 while still allowing customers to access the web. This would
have probably worked out, but I changed it to port 443 later-on. The reason
being that I need port 80 for Let's Encrypt challenges. Since I use nginx
as a reverse proxy for most of my services, I thought that I can multiplex
port 80 between LE and *prosody*. This was not possible with nginx.

So I discoverd traefik since it allows such a feat. The only problem is that
it can only route TCP connections based on the
[SNI](https://github.com/containous/traefik/blob/master/docs/content/routing/routers/index.md#rule-1). This requires the
XMPP connection to be encrypted entirely, not after STARTTLS negotiation,
which means that I would have to configure *prosody* to allow such a
connection and not offer STARTTLS.

# Prosody
Prosody has in its documentation no mentions of *direct TLS* which made me
guess that there is no support for it in *prosody*. After, however, asking
in the support group, I was told that this feature is called *legacy_ssl*.

As such, one only has to add

```lua
-- [...]

legacy_ssl_ports = { 5223 }
legacy_ssl_ssl = {
    [5223] = {
        key = "/path/to/keyfile";
        certificate = "/path/to/certificate";
    }
}

-- [...]
```

*Note:* In my testing, *prosody* would not enable *legacy_ssl* unless I
explicitly set `legacy_ssl_ports`.

When *prosody* tells you that it enabled `legacy_ssl` on the specified
ports, then you can test the connection by using OpenSSL to connect to it:
`openssl s_client -connect your.domain.example:5223`. OpenSSL should tell
you the data it can get from your certificate.

# traefik
In my configuration, I run *prosody* in an internal *Docker* network. In
order to connect it, in my case port 5223, to the world via port 443, I
configured my traefik to distinguish between HTTPS and XMPPS connections
based on the set SNI of the connection.

To do so, I firstly configured the static configuration to have
port 443 as an entrypoint:

```yaml
# [...]

entrypoints:
    https:
        address: ":443"

# [...]
```

For the dynamic configuration, I add two routers - one for TCP, one for
HTTPS - that both listen on the entrypoint `https`. As the documentation
[says](https://github.com/containous/traefik/blob/master/docs/content/routing/routers/index.md#general-1),
*"If both HTTP routers and TCP routers listen to the same entry points, the TCP routers will apply before the HTTP routers."*. This means that traefik has
to distinguish the two somehow.

We do this by using the `Host` rule for the HTTP router and `HostSNI` for
the TCP router.

As such, the dynamic configuration looks like this:

```yaml
tcp:
    routers:
        xmpps:
            entrypoints:
                - "https"
            rule: "HostSNI(`xmpps.your.domain.example`)"
            service: prosody-dtls
            tls:
                passthrough: true
        # [...]
    services:
        prosody-dtls:
            loadBalancer:
                servers:
                    - address: "<IP>:5223"

http:
    routers:
        web-secure:
            entrypoints:
                - "https"
            rule: "Host(`web.your.domain.example`)"
            service: webserver
```

It is important to note here, that the option `passthrough` has to be `true`
for the TCP router as otherwise the TLS connection would be terminated by
traefik.

Of course, you can instruct prosody to use port 443 directly, but I prefer
to keep it like this so I can easily see which connection goes to where.

# HTTP Upload
HTTP Upload was a very simple to implement this way. Just add another HTTPS
route in the dynamic traefik configuration to either the HTTP port of
prosody, which would terminate the TLS connection from traefik onwards, or
the HTTPS port, which - if running traefik and prosody on the same host -
would lead to a possible unnecessary re-encryption of the data.

This means that prosody's configuration looks like this:
```lua
[...]
-- Perhaps just one is enough
http_ports = { 5280 }
https_ports = { 5281 }

Component "your.domain"
    -- Perhaps just one is required, but I prefer to play it safe
    http_external_url = "https://http.xmpp.your.domain"
    http_host = "http.xmpp.your.domain"
[...]
```

And traefik's like this:
```yaml
[...]
http:
  routers:
    prosody-https:
      entrypoints:
        - "https"
      rule: "Host(`http.xmpp.your.domain`)"
      service: prosody-http

  services:
    prosody-http:
      loadBalancer:
        servers:
          - "http://prosody-ip:5280"
[...]
```

# DNS
In order for clients to pick this change up, one has to create a DNS SRV
record conforming to [XEP-0368](https://xmpp.org/extensions/xep-0368.html).

This change takes some time until it reaches the clients, so it would be wise
to keep the regular STARTTLS port 5222 open and connected to prosody until
the DNS entry has propagated to all DNS servers.

# Caveats
Of course, there is nothing without some caveats; some do apply here.

This change does not neccessarilly get applied to all clients automatically.
Clients like *Conversations* and its derivatives, however, do that when they
are reconnecting. Note that there may be clients that do not support XEP-0368
which will not apply this change automatically, like - at least in my
testing - *profanity*.

Also there may be some clients that do not support *direct TLS* and thus
cannot connect to the server. In my case, *matterbridge* was unable to
connect as it, without further investigation, can only connect with either
no TLS or with STARTTLS.

# Conclusion
In my case, I run my *prosody* server like this:

```
    <<WORLD>>-------------+
        |                 |
    [traefik]-------------/|/--------------+
        |                 |               |
  {xmpp.your.domain}    [5269]    {other.your.domain}
    [443 -> 5223]         |          [443 -> 80]
{http.xmpp.your.domain}   |               |
    [443 -> 5280]         |               |
        |                 |               |
    [prosody]-------------+            [nginx]
```

As I had a different port for *prosody* initially (80), I had to wait until
the DNS records are no longer cached by other DNS servers or clients. This
meant waiting for the TTL of the record, which in my case were 18000 seconds,
or 5 hours.

The port 5222 is, in my case, not reachable from the outside world but via my
internal *Docker* compose network so that my *matterbridge* bridges still work.