145 lines
10 KiB
HTML
145 lines
10 KiB
HTML
<!doctype html>
|
|
<html lang="en-gb">
|
|
<head>
|
|
<meta charset="UTF-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
|
<link href="https://blog.polynom.me/css/index.css" rel="stylesheet" integrity="sha384-R7KUcezOBiIXJ95JUBiXFdX0mMReehb8omi2xIGyZ6mbgXtQ3spxTx4c9BfffIA8" />
|
|
|
|
|
|
<link rel="alternate" type="application/rss+xml" title="blog.polynom.me Atom feed" href="https://blog.polynom.me/atom.xml">
|
|
|
|
|
|
|
|
<meta property="og:description" content="" />
|
|
<meta property="og:title" content="Lessons Learned From Self-Hosting" />
|
|
<title>Lessons Learned From Self-Hosting</title>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
</head>
|
|
<body>
|
|
<div class="flex flex-col p-2 md:p-8 items-start md:w-4/5 mx-auto">
|
|
<!-- Header -->
|
|
<div class="flex flex-row self-center">
|
|
<img class="w-12 h-12 md:w-24 md:h-24 rounded-lg" src="https://blog.polynom.me/img/avatar.jpg" integrity="sha384-uiNteVXosQ2+o/izp41L1G9VwuwYDYCOPxzFWks058DMUhW7KfQXcipM7WqgSgEZ" alt="Profile picture"/>
|
|
<div class="ml-4 self-center">
|
|
<a class="self-center text-2xl font-bold" href="/">PapaTutuWawa's Blog</a>
|
|
|
|
<ul class="list-none">
|
|
<li class="inline mr-8"><a href="/">Posts</a></li>
|
|
<li class="inline mr-8"><a href="https://blog.polynom.me/atom.xml">RSS</a></li>
|
|
<li class="inline mr-8"><a href="https://polynom.me">About</a></li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<!-- Container for posts -->
|
|
<div class="mx-auto mt-4 w-full md:max-w-prose">
|
|
<h1 class="text-indigo-400 text-3xl">Lessons Learned From Self-Hosting</h1>
|
|
|
|
<span class="text-md mt-2">Posted on 2020-01-03</span>
|
|
|
|
|
|
|
|
<!-- Actual article -->
|
|
<article class="prose lg:prose-lg text-white mt-4">
|
|
<p>Roughly eight months ago, according to my hosting provider, I spun up my VM which
|
|
I use to this day to self-host my chat, my mail, my git and so on. At the beginning, I thought that
|
|
it would allow me both to get away from proprietary software and to learn Linux administration. While
|
|
my first goal was met without any problems, the second one I achieved in ways I did not anticipate.</p>
|
|
<span id="continue-reading"></span>
|
|
<p>During these eight months, I learned quite a lot. Not by reading documentation, but by messing up
|
|
deployments. So this post is my telling of how I messed up and what lessons I learned from it.</p>
|
|
<h1 id="lesson-1-document-everything">Lesson 1: Document everything</h1>
|
|
<p>I always tell people that you should document your code. When asked why I answer that you won't
|
|
remember what that line does when you have not looked at your codebase for weeks or months.</p>
|
|
<p>What I did not realise is that this also applies to administration. I only wrote basic documentation
|
|
like a howto for certificate generation or a small troubleshooting guide. This, however, missed the most
|
|
important thing to document: the entire infrastructure.</p>
|
|
<p>Whenever I needed to look up my port mapping, what did I do? I opened up my <em>Docker compose</em> configuration
|
|
and search for the port mappings. What did I do when I wanted to know what services I have? Open my
|
|
<em>nginx</em> configuration and search for <code>server</code> directives.</p>
|
|
<p>This is a very slow process since I have to remember what services I have behind a reverse proxy and which
|
|
ones I have simply exposed. This lead me in the end to creating a folder - called <code>docs</code> - in which
|
|
I document everything. What certificates are used by what and where they are, port mappings, a graph
|
|
showing the dependencies of my services, ... While it may be tedious to create at first, it will really
|
|
help.</p>
|
|
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>[World]
|
|
</span><span>+
|
|
</span><span>|
|
|
</span><span>+-[443]-[nginx]-+-(blog.polynom.me)
|
|
</span><span> +-(git.polynom.me)-[gitea]
|
|
</span></code></pre>
|
|
<p>Above, you can see an excerpt from my <em>"network graph"</em>.</p>
|
|
<h1 id="lesson-2-version-control-everything">Lesson 2: Version Control everything</h1>
|
|
<p>Version Control Systems are a great thing. Want to try something out? Branch, try out and then either
|
|
merge back or roll back. Want to find out what changes broke something? Diff the last revisions and narrow
|
|
down your "search space". Want to know what you did? View the log.</p>
|
|
<p>While it might seem unneccessary, it helps me keep my cool, knowing that if I ever mess up my configuration, I
|
|
can just roll back the configuration from within git.</p>
|
|
<h1 id="lesson-3-have-a-test-environment">Lesson 3: Have a test environment</h1>
|
|
<p>While I was out once, I connected to a public Wifi. There, however, I could not connect to my VPN. It simply
|
|
did not work. A bit later, my Jabber client <em>Conversations</em> told me that it could not find my server. After
|
|
some thinking, I came to the conclusion that the provider of said public Wifi was probably blocking port <code>5222</code>
|
|
<em>(XMPP Client-to-Server)</em> and whatever port the VPN is using. As such, I wanted to change the port my
|
|
Jabber server uses. Since I do not have a failover server I tried testing things out locally, but gave up
|
|
after some time and just went and "tested in production". Needless to say that this was a bad idea. At first,
|
|
<em>Conversations</em> did not do a DNS lookup to see the changed XMPP port, which lead me to removing the DNS entry.
|
|
However, after some time - probably after the DNS change propagated far enough - <em>Conversations</em> said that it
|
|
could not find the server, even though it was listening on port <code>5222</code>. Testing with the new port yieled
|
|
success.</p>
|
|
<p>This experience was terrible for me. Not only was it possible that I broke my Jabber server, but it would
|
|
annoy everyone I got to install a Jabber client to talk to me as it would display <em>"Cannot connect to..."</em>.
|
|
If I had tested this locally, I probably would have been much calmer. In the end, I nervously watched as everyone
|
|
gradually reconnected...</p>
|
|
<h1 id="lesson-4-use-tools-and-write-scripts">Lesson 4: Use tools and write scripts</h1>
|
|
<p>The first server I ever got I provisioned manually. I mean, back then it made sense: It was a one-time provisioning and nothing should
|
|
change after the initial deployment. But now that I have a continually evolving server, I somehow need to document every step in case
|
|
I ever need to provision the same server again.</p>
|
|
<p>In my case it is <em>Ansible</em>. In my playbook I keep all the roles, e.g. <em>nginx</em>, <em>matterbridge</em>, <em>prosody</em>, separate and apply them to my one
|
|
server. In there I also made <strong>heavy</strong> use of templates. The reason for it is that before I started my <a href="https://blog.polynom.me/Road-to-Foss.html"><em>"Road to FOSS"</em></a>
|
|
I used a different domain that I had lying around. Changing the domain name manually would have been a very tedious process, so I decided to use
|
|
templates from the get-go. To make my life easier in case I ever change domains again, I defined all my domain names based on my <code>domain</code> variable.
|
|
The domain for git is defined as {% raw %}<code>git.{{ domain }}</code>{% endraw %}, the blog one as {% raw %}<code>blog.{{ domain }}</code>{% endraw %}.
|
|
Additionally, I make use of <em>Ansible Vaults</em>, allowing me to have encrypted secrets in my playbook.</p>
|
|
<p>During another project, I also set up an <em>Ansible</em> playbook. There, however, I did not use templates. I templated the configuration files using a Makefile
|
|
that was calling <code>sed</code> to replace the patterns. Not only was that a fragile method, it was also unneeded as <em>Ansible</em> was already providing
|
|
this functionality for me. I was just wasting my own time.</p>
|
|
<p>What I also learned was that one <em>Ansible</em> playbook is not enough. While it is nice to automatically provision a server using <em>Ansible</em>, there are other things
|
|
that need to be done. Certificates don't rotate themselves. From that, I derived a rule stating that if a task needs to be done more than once, then it is
|
|
time to write a script for it.</p>
|
|
<h1 id="lesson-4-1-automate">Lesson 4.1: Automate</h1>
|
|
<p>Closely tied to the last point: If a task needs to be performed, then you should consider creating a cronjob, or a systemd timer if that is more your thing,
|
|
to automatically run it. You don't want to enjoy your day, only for it to be ruined by an expired certificate causing issues.</p>
|
|
<p>Since automated cronjobs can cause trouble aswell, I decided to run all automated tasks on days at a time during which I am like to be able to react. As such, it is very
|
|
important to notify yourself of those automated actions. My certificate rotation, for example, sends me an eMail at the end, telling me if the certificates
|
|
were successfully rotated and if not, which ones failed. For those cases, I also keep a log of the rotation process somewhere else so that I can review it.</p>
|
|
<h1 id="lesson-5-unexpected-things-happen">Lesson 5: Unexpected things happen</h1>
|
|
<p>After having my shiny server run for some time, I was happy. It was basically running itself. Until <em>Conversations</em> was unable to contact my server,
|
|
connected to a public Wifi. This is something that I did not anticipate, but happened nevertheless.</p>
|
|
<p>This means that my deployment was not a run-and-forget solution but a constantly evolving system, where small improvements are periodically added.</p>
|
|
<h1 id="conclusion">Conclusion</h1>
|
|
<p>I thought I would just write down my thoughts on all the things that went wrong over the course of my self-hosting adventure. They may not
|
|
be best practices, but things that really helped me a lot.</p>
|
|
<p>Was the entire process difficult? At first. Was the experience an opportunity to learn? Absolutely! Was it fun? Definitely.</p>
|
|
|
|
</article>
|
|
|
|
<!-- Common post footer -->
|
|
<div class="mt-6">
|
|
<span class="prose lg:prose-lg text-md text-white">
|
|
If you have any questions or comments, then feel free to send me an email (Preferably with GPG encryption)
|
|
to papatutuwawa [at] polynom.me or reach out to me on the Fediverse at <a href="https://social.polynom.me/papatutuwawa">@papatutuwawa@social.polynom.me</a>.
|
|
</span>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
</div>
|
|
</body>
|
|
</html>
|