blog.polynom.me/about-logging/index.html

167 lines
9.1 KiB
HTML
Raw Permalink Normal View History

<!doctype html>
<html lang="en-gb">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link href="https://blog.polynom.me/css/index.css" rel="stylesheet" integrity="sha384-EJX4ZZbYMJeuoLRp93IbM/RYSzZmxw42TK7sgSRBEMChbBFK4NYUQEfsz3nBJQm8" crossorigin="anonymous" />
<link rel="alternate" type="application/rss+xml" title="blog.polynom.me Atom feed" href="https://blog.polynom.me/atom.xml">
<meta property="og:description" content="" />
<meta property="og:title" content="About Logging" />
<title>About Logging</title>
</head>
<body>
<div class="flex flex-col p-2 md:p-8 items-start md:w-4/5 mx-auto">
<!-- Header -->
<div class="flex flex-row self-center">
<img class="w-12 h-12 md:w-24 md:h-24 rounded-lg" src="https://blog.polynom.me/img/avatar.jpg" integrity="sha384-uiNteVXosQ2+o/izp41L1G9VwuwYDYCOPxzFWks058DMUhW7KfQXcipM7WqgSgEZ" alt="Profile picture" crossorigin="anonymous"/>
<div class="ml-4 self-center">
<a class="self-center text-2xl font-bold" href="/">PapaTutuWawa's Blog</a>
<ul class="list-none">
<li class="inline mr-8"><a href="/">Posts</a></li>
<li class="inline mr-8"><a href="https://blog.polynom.me/atom.xml">RSS</a></li>
<li class="inline mr-8"><a href="https://polynom.me">About</a></li>
</ul>
</div>
</div>
<!-- Container for posts -->
<div class="mx-auto mt-4 w-full md:max-w-prose">
<h1 class="text-indigo-400 text-3xl">About Logging</h1>
<span class="text-md mt-2">Posted on 2021-04-16</span>
<!-- Actual article -->
<article class="prose lg:prose-lg text-white mt-4">
<p><em>TL;DR</em>: This post also talks about the problems I faced while working on my logging. To log to
syslog from within my containers that do not support configuring a remote syslog server, I had
<em>syslog-ng</em> expose a unix domain socket and mounted it into the container to <code>/dev/log</code>.</p>
<span id="continue-reading"></span><h2 id="introduction">Introduction</h2>
<p>I have written a lot of blog posts about the lessons I have learned while setting up and
maintaining my server. But now that I started to rework my infrastructure a bit, I had to
inevitably look at something I may have overlooked in the past: logging!</p>
<p>Previously, I had <em>Docker</em> <em>kind of</em> manage my logs: If I needed something, I would just
call <code>docker-compose logs &lt;service&gt;</code> and it would spit out logs. Then, I started to
configure my services to log to files in various locations: my <em>prosody</em> server would
log to <code>/etc/prosody/logs/info.log</code>, my <em>nginx</em> to <code>/etc/nginx/logs/error.log</code>, etc.
This, however, turned out to be problematic
as, in my case, <em>prosody</em> stopped logging into the file if I rotated it with <em>logrotate</em>. It was
also a bit impractical, as the logs were not all in the same place, but distributed across multiple
directories.
Moreover, <em>prosody</em> was logging things that I did not want in my logs but I could not turn off,
like when a client connected or authenticated itself. For me, this is a problem from two perspectives:
On the one hand, it is metadata that does not help me debug an hypothetical issue I have with my
<em>prosody</em> installation, on the other hand, it is metadata I straight-up do not want to store.</p>
<p>My solution was using a syslog daemon to process the logs, so that I could remove logs that I do not
want or need, and drop them all off at <code>/var/log</code>. However, there was a problem that I faced almost
immediately: Not all software I can configure to log to syslog, I can configure to log to a specific
syslog server. Why is this a problem? Well, syslog does not work inside a <em>Docker</em> container out of the
box, so I would have to have my syslog daemon expose a TCP/UDP (unix domain) socket that logs can be sent to. To
see this issue you can try to run <code>logger -t SomeTag Hello World</code> inside one of your containers
and try to find it, e.g. in your host's journal.</p>
<p>Today, I found my solution to both syslog logging within the containers and filtering out unneeded logs.</p>
<h2 id="syslog-inside-containers">Syslog inside Containers</h2>
<p>The first step was getting the logs out of my containers without using files. To this end, I configured
my syslog daemon - <em>syslog-ng</em> - to expose a unix domain socket to, for example, <code>/var/run/syslog</code> and
mount it into all containers to <code>/dev/log</code>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>source s_src {
</span><span> system();
</span><span> internal();
</span><span> unix-dgram(&quot;/var/run/syslog&quot;);
</span><span>};
</span></code></pre>
<p>If you now try and run <code>logger -t SomeTag Hello World</code> inside the container, you should be able
to find &quot;Hello World&quot; inside the host's logs or journals.</p>
<h2 id="ignoring-certain-logs">Ignoring Certain Logs</h2>
<p>The next step was ignoring logs that I do not need or care about. For this, I set up two logs within
<em>syslog-ng</em>: One that was going into my actual log file and one that was dropped:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>destination d_prosody {
</span><span> file(&quot;/var/log/prosody.log&quot;);
</span><span>};
</span><span>filter f_prosody {
</span><span> program(&quot;prosody&quot;);
</span><span>};
</span><span>filter f_prosody_drop {
</span><span> program(&quot;prosody&quot;)
</span><span> and message(&quot;(Client connected|Client disconnected|Authenticated as .*|Stream encrypted .*)$&quot;);
</span><span>};
</span><span>
</span><span># Drop
</span><span>log {
</span><span> source(s_src);
</span><span> filter(f_prosody_drop);
</span><span> flags(final);
</span><span>};
</span><span># Log
</span><span>log {
</span><span> source(s_src);
</span><span> filter(f_prosody);
</span><span> destination(d_prosody);
</span><span> flags(final);
</span><span>};
</span><span>
</span></code></pre>
<p>This example would log all things that <em>prosody</em> logs to the <em>prosody</em> location <code>d_prosody</code> and drop all
lines that match the given regular expression, which, in my case, matches all lines that relate to a client
connecting, disconnecting or authenticating.</p>
<p>Important is the <code>flags(final);</code> in the drop rule to indicate that a log line that matches the rule should
not be processed any further. That log also defines no destination, which tells <em>syslog-ng</em> in combination with
the <code>final</code> flag that the log
should be dropped.</p>
<p>Additionally, I moved the log rule that matches everything sent to the configured source to the bottom
of the configuration to prevent any of the logs to <em>also</em> land in the &quot;everything&quot; log.</p>
<p>Since I also host a <em>Nextcloud</em> server, I was also interested in getting rid of HTTP access logs. But I would
also like to know when someone is trying to scan my webserver for vulnerable <em>wordpress</em> installations.</p>
<p>So I again defined rules similar to those above, but added a twist:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>filter f_nextcloud_drop {
</span><span> program(&quot;nextcloud&quot;)
</span><span> and match(&quot;200&quot; value(&quot;.nextcloud.response&quot;));
</span><span>};
</span><span>log {
</span><span> source(s_src);
</span><span> parser { apache-accesslog-parser(prefix(&quot;.nextcloud.&quot;)); };
</span><span> filter(f_nextcloud_drop);
</span><span> flags(final);
</span><span>};
</span><span>
</span></code></pre>
<p>As you can see, the rule for my <em>Nextcloud</em> is quite similar, except that I added a parser. With this, I can
make <em>syslog-ng</em> understand the HTTP access log and expose its parts as variables to my filter rule. There,
I say that my drop rule should match all access log lines that indicate a HTTP response code of 200, since
those are locations on my server that I expect to be accessed and thus do not care about.</p>
<h2 id="conclusion">Conclusion</h2>
<p>With this setup, I feel much better about the logs I produce. I also have done other things not mentioned, like
configure <em>logrotate</em> to rotate my logs daily so that my logs don't grow too large and get removed after a day.</p>
<p>Please note that I am not an expert in <em>syslog-ng</em>. It just happend to be what I first got to do what I want. And
the example rules I showed are also the first thing that I wrote and filtered out what I wanted.</p>
</article>
<!-- Common post footer -->
<div class="mt-6">
<span class="prose lg:prose-lg text-md text-white">
If you have any questions or comments, then feel free to send me an email (Preferably with GPG encryption)
to papatutuwawa [at] polynom.me or reach out to me on the Fediverse at <a href="https://social.polynom.me&#x2F;papatutuwawa">@papatutuwawa@social.polynom.me</a>.
</span>
</div>
</div>
</div>
</body>
</html>