blog.polynom.me/running-prosody-traefik/index.html
2024-01-05 18:32:24 +00:00

240 lines
16 KiB
HTML

<!doctype html>
<html lang="en-gb">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link href="https://blog.polynom.me/css/index.css" rel="stylesheet" integrity="sha384-R7KUcezOBiIXJ95JUBiXFdX0mMReehb8omi2xIGyZ6mbgXtQ3spxTx4c9BfffIA8" />
<link rel="alternate" type="application/rss+xml" title="blog.polynom.me Atom feed" href="https://blog.polynom.me/atom.xml">
<meta property="og:description" content="" />
<meta property="og:title" content="Running Prosody on Port 443 Behind traefik" />
<title>Running Prosody on Port 443 Behind traefik</title>
</head>
<body>
<div class="flex flex-col p-2 md:p-8 items-start md:w-4/5 mx-auto">
<!-- Header -->
<div class="flex flex-row self-center">
<img class="w-12 h-12 md:w-24 md:h-24 rounded-lg" src="https://blog.polynom.me/img/avatar.jpg" integrity="sha384-uiNteVXosQ2+o/izp41L1G9VwuwYDYCOPxzFWks058DMUhW7KfQXcipM7WqgSgEZ" alt="Profile picture"/>
<div class="ml-4 self-center">
<a class="self-center text-2xl font-bold" href="/">PapaTutuWawa's Blog</a>
<ul class="list-none">
<li class="inline mr-8"><a href="/">Posts</a></li>
<li class="inline mr-8"><a href="https://blog.polynom.me/atom.xml">RSS</a></li>
<li class="inline mr-8"><a href="https://polynom.me">About</a></li>
</ul>
</div>
</div>
<!-- Container for posts -->
<div class="mx-auto mt-4 w-full md:max-w-prose">
<h1 class="text-indigo-400 text-3xl">Running Prosody on Port 443 Behind traefik</h1>
<span class="text-md mt-2">Posted on 2020-02-13</span>
<!-- Actual article -->
<article class="prose lg:prose-lg text-white mt-4">
<p><em>TL;DR: This post is about running prosody with HTTPS services both on port 443. If you only care about the how, then jump to</em>
<strong>Considerations</strong> <em>and read from there.</em></p>
<span id="continue-reading"></span><h1 id="introduction">Introduction</h1>
<p>As part of my <a href="https://blog.polynom.me/Road-to-Foss.html"><em>&quot;road to FOSS&quot;</em></a> I
set up my own XMPP server using <em>prosody</em>. While it has been running fine for
quite some time, I noticed, while connected to a public Wifi, that my
server was unreachable. At that time I was panicing because I thought prosody
kept crashing for some reason. After using my mobile data, however, I saw
that I <strong>could</strong> connect to my server. The only possible explanation I came
up with is that the provider of the public Wifi is blocking anything that
is not port 53, 80 or 443. <em>(Other ports I did not try)</em></p>
<p>My solution: Move <em>prosody</em>'s C2S - <em>Client to Server</em> - port from 5222 to
either port 53, 80 or 443. Port 53 did not seem like a good choice as I
want to keep myself the possibilty of hosting a DNS server. So the only
choice was between 80 and 443.</p>
<h1 id="considerations">Considerations</h1>
<p>Initially I went with port 80 because it would be the safest bet: You cannot
block port 80 while still allowing customers to access the web. This would
have probably worked out, but I changed it to port 443 later-on. The reason
being that I need port 80 for Let's Encrypt challenges. Since I use nginx
as a reverse proxy for most of my services, I thought that I can multiplex
port 80 between LE and <em>prosody</em>. This was not possible with nginx.</p>
<p>So I discoverd traefik since it allows such a feat. The only problem is that
it can only route TCP connections based on the
<a href="https://github.com/containous/traefik/blob/master/docs/content/routing/routers/index.md#rule-1">SNI</a>. This requires the
XMPP connection to be encrypted entirely, not after STARTTLS negotiation,
which means that I would have to configure <em>prosody</em> to allow such a
connection and not offer STARTTLS.</p>
<h1 id="prosody">Prosody</h1>
<p>Prosody has in its documentation no mentions of <em>direct TLS</em> which made me
guess that there is no support for it in <em>prosody</em>. After, however, asking
in the support group, I was told that this feature is called <em>legacy_ssl</em>.</p>
<p>As such, one only has to add</p>
<pre data-lang="lua" style="background-color:#2b303b;color:#c0c5ce;" class="language-lua "><code class="language-lua" data-lang="lua"><span style="color:#65737e;">-- [...]
</span><span>
</span><span style="color:#bf616a;">legacy_ssl_ports </span><span>= { </span><span style="color:#d08770;">5223 </span><span>}
</span><span style="color:#bf616a;">legacy_ssl_ssl </span><span>= {
</span><span> [</span><span style="color:#d08770;">5223</span><span>] = {
</span><span> </span><span style="color:#a3be8c;">key </span><span>= &quot;</span><span style="color:#a3be8c;">/path/to/keyfile</span><span>&quot;;
</span><span> </span><span style="color:#a3be8c;">certificate </span><span>= &quot;</span><span style="color:#a3be8c;">/path/to/certificate</span><span>&quot;;
</span><span> }
</span><span>}
</span><span>
</span><span style="color:#65737e;">-- [...]
</span></code></pre>
<p><em>Note:</em> In my testing, <em>prosody</em> would not enable <em>legacy_ssl</em> unless I
explicitly set <code>legacy_ssl_ports</code>.</p>
<p>When <em>prosody</em> tells you that it enabled <code>legacy_ssl</code> on the specified
ports, then you can test the connection by using OpenSSL to connect to it:
<code>openssl s_client -connect your.domain.example:5223</code>. OpenSSL should tell
you the data it can get from your certificate.</p>
<h1 id="traefik">traefik</h1>
<p>In my configuration, I run <em>prosody</em> in an internal <em>Docker</em> network. In
order to connect it, in my case port 5223, to the world via port 443, I
configured my traefik to distinguish between HTTPS and XMPPS connections
based on the set SNI of the connection.</p>
<p>To do so, I firstly configured the static configuration to have
port 443 as an entrypoint:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#65737e;"># [...]
</span><span>
</span><span style="color:#bf616a;">entrypoints</span><span>:
</span><span> </span><span style="color:#bf616a;">https</span><span>:
</span><span> </span><span style="color:#bf616a;">address</span><span>: &quot;</span><span style="color:#a3be8c;">:443</span><span>&quot;
</span><span>
</span><span style="color:#65737e;"># [...]
</span></code></pre>
<p>For the dynamic configuration, I add two routers - one for TCP, one for
HTTPS - that both listen on the entrypoint <code>https</code>. As the documentation
<a href="https://github.com/containous/traefik/blob/master/docs/content/routing/routers/index.md#general-1">says</a>,
<em>&quot;If both HTTP routers and TCP routers listen to the same entry points, the TCP routers will apply before the HTTP routers.&quot;</em>. This means that traefik has
to distinguish the two somehow.</p>
<p>We do this by using the <code>Host</code> rule for the HTTP router and <code>HostSNI</code> for
the TCP router.</p>
<p>As such, the dynamic configuration looks like this:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">tcp</span><span>:
</span><span> </span><span style="color:#bf616a;">routers</span><span>:
</span><span> </span><span style="color:#bf616a;">xmpps</span><span>:
</span><span> </span><span style="color:#bf616a;">entrypoints</span><span>:
</span><span> - &quot;</span><span style="color:#a3be8c;">https</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">rule</span><span>: &quot;</span><span style="color:#a3be8c;">HostSNI(`xmpps.your.domain.example`)</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">service</span><span>: </span><span style="color:#a3be8c;">prosody-dtls
</span><span> </span><span style="color:#bf616a;">tls</span><span>:
</span><span> </span><span style="color:#bf616a;">passthrough</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#65737e;"># [...]
</span><span> </span><span style="color:#bf616a;">services</span><span>:
</span><span> </span><span style="color:#bf616a;">prosody-dtls</span><span>:
</span><span> </span><span style="color:#bf616a;">loadBalancer</span><span>:
</span><span> </span><span style="color:#bf616a;">servers</span><span>:
</span><span> - </span><span style="color:#bf616a;">address</span><span>: &quot;</span><span style="color:#a3be8c;">&lt;IP&gt;:5223</span><span>&quot;
</span><span>
</span><span style="color:#bf616a;">http</span><span>:
</span><span> </span><span style="color:#bf616a;">routers</span><span>:
</span><span> </span><span style="color:#bf616a;">web-secure</span><span>:
</span><span> </span><span style="color:#bf616a;">entrypoints</span><span>:
</span><span> - &quot;</span><span style="color:#a3be8c;">https</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">rule</span><span>: &quot;</span><span style="color:#a3be8c;">Host(`web.your.domain.example`)</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">service</span><span>: </span><span style="color:#a3be8c;">webserver
</span></code></pre>
<p>It is important to note here, that the option <code>passthrough</code> has to be <code>true</code>
for the TCP router as otherwise the TLS connection would be terminated by
traefik.</p>
<p>Of course, you can instruct prosody to use port 443 directly, but I prefer
to keep it like this so I can easily see which connection goes to where.</p>
<h1 id="http-upload">HTTP Upload</h1>
<p>HTTP Upload was a very simple to implement this way. Just add another HTTPS
route in the dynamic traefik configuration to either the HTTP port of
prosody, which would terminate the TLS connection from traefik onwards, or
the HTTPS port, which - if running traefik and prosody on the same host -
would lead to a possible unnecessary re-encryption of the data.</p>
<p>This means that prosody's configuration looks like this:</p>
<pre data-lang="lua" style="background-color:#2b303b;color:#c0c5ce;" class="language-lua "><code class="language-lua" data-lang="lua"><span>[</span><span style="color:#d08770;">...</span><span>]
</span><span style="color:#65737e;">-- Perhaps just one is enough
</span><span style="color:#bf616a;">http_ports </span><span>= { </span><span style="color:#d08770;">5280 </span><span>}
</span><span style="color:#bf616a;">https_ports </span><span>= { </span><span style="color:#d08770;">5281 </span><span>}
</span><span>
</span><span style="color:#bf616a;">Component </span><span>&quot;</span><span style="color:#a3be8c;">your.domain</span><span>&quot;
</span><span> </span><span style="color:#65737e;">-- Perhaps just one is required, but I prefer to play it safe
</span><span> </span><span style="color:#bf616a;">http_external_url </span><span>= &quot;</span><span style="color:#a3be8c;">https://http.xmpp.your.domain</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">http_host </span><span>= &quot;</span><span style="color:#a3be8c;">http.xmpp.your.domain</span><span>&quot;
</span><span>[</span><span style="color:#d08770;">...</span><span>]
</span></code></pre>
<p>And traefik's like this:</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span>[</span><span style="color:#d08770;">...</span><span>]
</span><span style="color:#bf616a;">http</span><span>:
</span><span> </span><span style="color:#bf616a;">routers</span><span>:
</span><span> </span><span style="color:#bf616a;">prosody-https</span><span>:
</span><span> </span><span style="color:#bf616a;">entrypoints</span><span>:
</span><span> - &quot;</span><span style="color:#a3be8c;">https</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">rule</span><span>: &quot;</span><span style="color:#a3be8c;">Host(`http.xmpp.your.domain`)</span><span>&quot;
</span><span> </span><span style="color:#bf616a;">service</span><span>: </span><span style="color:#a3be8c;">prosody-http
</span><span>
</span><span> </span><span style="color:#bf616a;">services</span><span>:
</span><span> </span><span style="color:#bf616a;">prosody-http</span><span>:
</span><span> </span><span style="color:#bf616a;">loadBalancer</span><span>:
</span><span> </span><span style="color:#bf616a;">servers</span><span>:
</span><span> - &quot;</span><span style="color:#a3be8c;">http://prosody-ip:5280</span><span>&quot;
</span><span>[</span><span style="color:#d08770;">...</span><span>]
</span></code></pre>
<h1 id="dns">DNS</h1>
<p>In order for clients to pick this change up, one has to create a DNS SRV
record conforming to <a href="https://xmpp.org/extensions/xep-0368.html">XEP-0368</a>.</p>
<p>This change takes some time until it reaches the clients, so it would be wise
to keep the regular STARTTLS port 5222 open and connected to prosody until
the DNS entry has propagated to all DNS servers.</p>
<h1 id="caveats">Caveats</h1>
<p>Of course, there is nothing without some caveats; some do apply here.</p>
<p>This change does not neccessarilly get applied to all clients automatically.
Clients like <em>Conversations</em> and its derivatives, however, do that when they
are reconnecting. Note that there may be clients that do not support XEP-0368
which will not apply this change automatically, like - at least in my
testing - <em>profanity</em>.</p>
<p>Also there may be some clients that do not support <em>direct TLS</em> and thus
cannot connect to the server. In my case, <em>matterbridge</em> was unable to
connect as it, without further investigation, can only connect with either
no TLS or with STARTTLS.</p>
<h1 id="conclusion">Conclusion</h1>
<p>In my case, I run my <em>prosody</em> server like this:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> &lt;&lt;WORLD&gt;&gt;-------------+
</span><span> | |
</span><span> [traefik]-------------/|/--------------+
</span><span> | | |
</span><span> {xmpp.your.domain} [5269] {other.your.domain}
</span><span> [443 -&gt; 5223] | [443 -&gt; 80]
</span><span>{http.xmpp.your.domain} | |
</span><span> [443 -&gt; 5280] | |
</span><span> | | |
</span><span> [prosody]-------------+ [nginx]
</span></code></pre>
<p>As I had a different port for <em>prosody</em> initially (80), I had to wait until
the DNS records are no longer cached by other DNS servers or clients. This
meant waiting for the TTL of the record, which in my case were 18000 seconds,
or 5 hours.</p>
<p>The port 5222 is, in my case, not reachable from the outside world but via my
internal <em>Docker</em> compose network so that my <em>matterbridge</em> bridges still work.</p>
</article>
<!-- Common post footer -->
<div class="mt-6">
<span class="prose lg:prose-lg text-md text-white">
If you have any questions or comments, then feel free to send me an email (Preferably with GPG encryption)
to papatutuwawa [at] polynom.me or reach out to me on the Fediverse at <a href="https://social.polynom.me&#x2F;papatutuwawa">@papatutuwawa@social.polynom.me</a>.
</span>
</div>
</div>
</div>
</body>
</html>