170 lines
13 KiB
HTML
170 lines
13 KiB
HTML
<!doctype html>
|
|
<html lang="en-gb">
|
|
<head>
|
|
<meta charset="UTF-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
|
<link href="https://blog.polynom.me/css/index.css" rel="stylesheet" integrity="sha384-EJX4ZZbYMJeuoLRp93IbM/RYSzZmxw42TK7sgSRBEMChbBFK4NYUQEfsz3nBJQm8" />
|
|
|
|
|
|
<link rel="alternate" type="application/rss+xml" title="blog.polynom.me Atom feed" href="https://blog.polynom.me/atom.xml">
|
|
|
|
|
|
|
|
<meta property="og:description" content="" />
|
|
<meta property="og:title" content="Running Prosody on Port 443 Behind traefik 2: Electric ALPN" />
|
|
<title>Running Prosody on Port 443 Behind traefik 2: Electric ALPN</title>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</head>
|
|
<body>
|
|
<div class="flex flex-col p-2 md:p-8 items-start md:w-4/5 mx-auto">
|
|
<!-- Header -->
|
|
<div class="flex flex-row self-center">
|
|
<img class="w-12 h-12 md:w-24 md:h-24 rounded-lg" src="https://blog.polynom.me/img/avatar.jpg" integrity="sha384-uiNteVXosQ2+o/izp41L1G9VwuwYDYCOPxzFWks058DMUhW7KfQXcipM7WqgSgEZ" alt="Profile picture"/>
|
|
<div class="ml-4 self-center">
|
|
<a class="self-center text-2xl font-bold" href="/">PapaTutuWawa's Blog</a>
|
|
|
|
<ul class="list-none">
|
|
<li class="inline mr-8"><a href="/">Posts</a></li>
|
|
<li class="inline mr-8"><a href="https://blog.polynom.me/atom.xml">RSS</a></li>
|
|
<li class="inline mr-8"><a href="https://polynom.me">About</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<!-- Container for posts -->
|
|
<div class="mx-auto mt-4 w-full md:max-w-prose">
|
|
<h1 class="text-indigo-400 text-3xl">Running Prosody on Port 443 Behind traefik 2: Electric ALPN</h1>
|
|
|
|
<span class="text-md mt-2">Posted on 2023-07-15</span>
|
|
|
|
|
|
|
|
<!-- Actual article -->
|
|
<article class="prose lg:prose-lg text-white mt-4">
|
|
<p>Hello everyone. Long time, no read.</p>
|
|
<p>In 2020, I published a post titled "<a href="https://blog.polynom.me/Running-Prosody-traefik.html">Running Prosody on Port 443 Behind traefik</a>", where I described how I run my XMPP server
|
|
behind the "application proxy" <a href="https://github.com/traefik/traefik"><em>traefik</em></a>.
|
|
I did this because I wanted to run my XMPP server <em>prosody</em> on port 443, so that the clients connected
|
|
to my server can bypass firewalls that only allow web traffic. While that approach worked,
|
|
over the last three years I changed my setup dramatically.</p>
|
|
<span id="continue-reading"></span>
|
|
<p>While migrating my old server from <em>Debian</em> to <em>NixOS</em>, I decided that I wanted a website
|
|
hosted at the same domain I host my XMPP server at. This, however, was not possible with
|
|
<em>traefik</em> back then because it only allowed the <code>HostSNI</code> rule, which differentiates TLS
|
|
connections using the sent <em>Server Name Indication</em>. This is a problem, because a connection
|
|
to <code>polynom.me</code> the website and <code>polynom.me</code> the XMPP server both result in the same SNI being
|
|
sent by a connecting client.</p>
|
|
<p>Some time later, I stumbled upon <a href="https://github.com/yrutschle/sslh"><em>sslh</em></a>, which is a
|
|
tool similar to <em>traefik</em> in that it allows hosting multiple services on the same port, all
|
|
differentiated by the SNI <strong>and</strong> the ALPN set by the connecting client. ALPN, or <em>Application-Layer Protocol Negotiation</em>, is an extension
|
|
to TLS which allows a connecting client to advertise the protocol(s) it would like to use
|
|
inside the encrypted session <a href="https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation">(source)</a>. As such, I put
|
|
<em>sslh</em> in front of my <em>traefik</em> and told it to route XMPP traffic (identified with an ALPN
|
|
of <code>xmpp-client</code>) to my prosody server and everything else to my <em>traefik</em> server. While this
|
|
worked well, there were two issues:</p>
|
|
<ol>
|
|
<li>I was not running <em>sslh</em> in its <a href="https://github.com/yrutschle/sslh/blob/master/doc/config.md#transparent-proxy-support">"transparent mode"</a>, which uses some fancy iptable rules to allow the services behind it to see a connecting client's real IP address instead of just <code>127.0.0.1</code>. However, this requires more setup to work. This is an issue for services which enforce rate limits, like <em>NextCloud</em> and <em>Akkoma</em>. If one of theses services gets hit by many requests, all the services see are requests from <code>127.0.0.1</code> and may thus rate limit (or ban) <code>127.0.0.1</code>, meaning that all - even legitimate - requests are rate limited. Additionally, I was not sure if I could just use this to route an incoming IPv6 request to <code>127.0.0.1</code>, which is an IPv4 address.</li>
|
|
<li>One day, as I was updating my server, I noticed that all my web services were responding very slowly. After some looking around, it turned out that <em>sslh</em> took about 5 seconds to route IPv6 requests, but not IPv4 requests. As I did not change anything (besides update the server), to this day I am not sure what happened.</li>
|
|
</ol>
|
|
<p>Due to these two issues, I decided to revisit the idea I described in my old post.</p>
|
|
<h2 id="the-prosody-setup">The Prosody Setup</h2>
|
|
<p>On the prosody-side of things, I did not change a lot compared to the old post. I did, however,
|
|
migrate from the <code>legacy_ssl_*</code> options to the newer <code>c2s_direct_tls_*</code> options, which
|
|
<a href="https://hg.prosody.im/trunk/file/tip/doc/doap.xml#l758">replace the former</a>.</p>
|
|
<p>Thus, my prosody configuration regarding direct TLS connections now looks like this:</p>
|
|
<pre data-lang="lua" style="background-color:#2b303b;color:#c0c5ce;" class="language-lua "><code class="language-lua" data-lang="lua"><span style="color:#bf616a;">c2s_direct_tls_ports </span><span>= { </span><span style="color:#d08770;">5223 </span><span>}
|
|
</span><span style="color:#bf616a;">c2s_direct_tls_ssl </span><span>= {
|
|
</span><span> [</span><span style="color:#d08770;">5223</span><span>] = {
|
|
</span><span> </span><span style="color:#a3be8c;">key </span><span>= "</span><span style="color:#a3be8c;">/etc/prosody/certs/polynom.me.key</span><span>";
|
|
</span><span> </span><span style="color:#a3be8c;">certificate </span><span>= "</span><span style="color:#a3be8c;">/etc/prosody/certs/polynom.me.crt</span><span>";
|
|
</span><span> };
|
|
</span><span>}
|
|
</span></code></pre>
|
|
<h2 id="the-traefik-setup">The <em>Traefik</em> Setup</h2>
|
|
<p>On <em>traefik</em>-side of things, only one thing really changed: Instead of just having a rule using
|
|
<code>HostSNI</code>, I now also require that the connection with the XMPP server advertises an ALPN
|
|
of <code>xmpp-client</code>, which is specified in the
|
|
<a href="https://xmpp.org/extensions/xep-0368.html">appropriate XMPP spec</a>. From my deployment
|
|
experience, all clients I tested (<em>Conversations</em>, <em>Blabber</em>, <em>Gajim</em>, <em>Dino</em>, <em>Monal</em>, <a href="https://moxxy.org">Moxxy</a>)
|
|
correctly set the ALPN when connecting via a direct TLS connection.</p>
|
|
<p>So my <em>traefik</em> configuration now looks something like this (Not really, because I let NixOS
|
|
generate the actual config, but it is very similar):</p>
|
|
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span style="color:#bf616a;">tcp</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">routers</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">xmpps</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">entrypoints</span><span>:
|
|
</span><span> - "</span><span style="color:#a3be8c;">https</span><span>"
|
|
</span><span> </span><span style="color:#bf616a;">rule</span><span>: "</span><span style="color:#a3be8c;">HostSNI(`polynom.me`) && ALPN(`xmpp-client`)</span><span>"
|
|
</span><span> </span><span style="color:#bf616a;">service</span><span>: </span><span style="color:#a3be8c;">prosody
|
|
</span><span> </span><span style="color:#bf616a;">tls</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">passthrough</span><span>: </span><span style="color:#d08770;">true
|
|
</span><span> </span><span style="color:#65737e;"># [...]
|
|
</span><span> </span><span style="color:#bf616a;">services</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">prosody</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">loadBalancer</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">servers</span><span>:
|
|
</span><span> - </span><span style="color:#bf616a;">address</span><span>: "</span><span style="color:#a3be8c;">127.0.0.1:5223</span><span>"
|
|
</span><span>
|
|
</span><span style="color:#bf616a;">http</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">routers</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">web-secure</span><span>:
|
|
</span><span> </span><span style="color:#bf616a;">entrypoints</span><span>:
|
|
</span><span> - "</span><span style="color:#a3be8c;">https</span><span>"
|
|
</span><span> </span><span style="color:#bf616a;">rule</span><span>: "</span><span style="color:#a3be8c;">Host(`polynom.me`)</span><span>"
|
|
</span><span> </span><span style="color:#bf616a;">service</span><span>: </span><span style="color:#a3be8c;">webserver
|
|
</span><span> </span><span style="color:#bf616a;">tls</span><span>:
|
|
</span></code></pre>
|
|
<p>The entrypoint <code>https</code> is just set to listen on <code>:443</code>. This way, I can route IPv4 and IPv6
|
|
requests. Also note the <code>passthrough: true</code> in the XMPP router's <code>tls</code> settings. If this is
|
|
not set to <code>true</code>, then <em>traefik</em> would terminate the connection's TLS session before passing
|
|
the data to the XMPP server.</p>
|
|
<p>However, this config has one really big issue: In order
|
|
to have the website hosted at <code>polynom.me</code> be served using TLS, I have to set the
|
|
router's <code>tls</code> attribute. The <em>traefik</em>
|
|
documentation says that "<em>If both HTTP routers and TCP routers listen to the
|
|
same entry points, the TCP routers will apply before the HTTP routers. If no matching route
|
|
is found for the TCP routers, then the HTTP routers will take over.</em>"
|
|
<a href="https://doc.traefik.io/traefik/routing/routers/#general_1">(source)</a>.</p>
|
|
<p>This, however, does not seem to be the case if a HTTP router (in my example with <code>Host(`polynom.me`)</code>) and a TCP router (in my example with <code>HostSNI(`polynom.me`)</code>) respond to the same
|
|
SNI <strong>and</strong> the HTTP router has its <code>tls</code> attribute set. In that case, the HTTP router appears
|
|
to be checked first and will complain, if the sent ALPN is not one of the
|
|
<a href="https://developer.mozilla.org/en-US/docs/Glossary/ALPN">HTTP ALPNs</a>, for example when
|
|
connecting using XMPP. As such we can connect to the HTTP server but not to the
|
|
XMPP server.</p>
|
|
<p>It appears to be an issue that <a href="https://github.com/traefik/traefik/issues/9922">I am not alone with</a>, but also
|
|
one that is not fixed. So I tried digging around in <em>traefik</em>'s code and tried a couple of
|
|
things. So for my setup to work, I have to apply <a href="https://github.com/PapaTutuWawa/traefik/commit/36f0e3c805ca4e645f3313f667a6b3ff5e2fe4a9">this patch</a> to <em>traefik</em>. With that, the issue <em>appears</em>
|
|
to be gone, and I can access both my website and my XMPP server on the same domain and on the
|
|
same port. Do note that this patch is not upstreamed and may break things. For me, it
|
|
works. But I haven't run extensive tests or <em>traefik</em>'s integration and unit tests.</p>
|
|
<h2 id="conclusion">Conclusion</h2>
|
|
<p>This approach solves problem 2 fully and problem 1 partially. <em>Traefik</em> is able to route
|
|
the connections correctly with no delay, compared to <em>sslh</em>. It also provides my web services
|
|
with the connecting clients' IP addresses using HTTP headers. It does not, however, provide
|
|
my XMPP server with a connecting client's IP address. This could be solved with some clever
|
|
trickery, like telling <em>traefik</em> to use the <a href="https://doc.traefik.io/traefik/routing/services/#proxy-protocol"><em>PROXY</em> protocol</a> when connecting to prosody,
|
|
and enabling the <a href="https://modules.prosody.im/mod_net_proxy.html"><code>net_proxy</code></a> module. However,
|
|
I have not yet tried such a setup, though I am very curious and may try that out.</p>
|
|
|
|
</article>
|
|
|
|
<!-- Common post footer -->
|
|
<div class="mt-6">
|
|
<span class="prose lg:prose-lg text-md text-white">
|
|
If you have any questions or comments, then feel free to send me an email (Preferably with GPG encryption)
|
|
to papatutuwawa [at] polynom.me or reach out to me on the Fediverse at <a href="https://social.polynom.me/papatutuwawa">@papatutuwawa@social.polynom.me</a>.
|
|
</span>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
</body>
|
|
</html>
|