167 lines
9.1 KiB
HTML
167 lines
9.1 KiB
HTML
|
<!doctype html>
|
||
|
|
<html lang="en-gb">
|
||
|
|
<head>
|
||
|
|
<meta charset="UTF-8" />
|
||
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||
|
|
<link href="https://blog.polynom.me/css/index.css" rel="stylesheet" integrity="sha384-EJX4ZZbYMJeuoLRp93IbM/RYSzZmxw42TK7sgSRBEMChbBFK4NYUQEfsz3nBJQm8" crossorigin="anonymous" />
|
||
|
|
|
||
|
|
|
||
|
|
<link rel="alternate" type="application/rss+xml" title="blog.polynom.me Atom feed" href="https://blog.polynom.me/atom.xml">
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<meta property="og:description" content="" />
|
||
|
|
<meta property="og:title" content="About Logging" />
|
||
|
|
<title>About Logging</title>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</head>
|
||
|
|
<body>
|
||
|
|
<div class="flex flex-col p-2 md:p-8 items-start md:w-4/5 mx-auto">
|
||
|
|
<!-- Header -->
|
||
|
|
<div class="flex flex-row self-center">
|
||
|
|
<img class="w-12 h-12 md:w-24 md:h-24 rounded-lg" src="https://blog.polynom.me/img/avatar.jpg" integrity="sha384-uiNteVXosQ2+o/izp41L1G9VwuwYDYCOPxzFWks058DMUhW7KfQXcipM7WqgSgEZ" alt="Profile picture" crossorigin="anonymous"/>
|
||
|
|
<div class="ml-4 self-center">
|
||
|
|
<a class="self-center text-2xl font-bold" href="/">PapaTutuWawa's Blog</a>
|
||
|
|
|
||
|
|
<ul class="list-none">
|
||
|
|
<li class="inline mr-8"><a href="/">Posts</a></li>
|
||
|
|
<li class="inline mr-8"><a href="https://blog.polynom.me/atom.xml">RSS</a></li>
|
||
|
|
<li class="inline mr-8"><a href="https://polynom.me">About</a></li>
|
||
|
|
</ul>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
|
||
|
|
|
||
|
|
<!-- Container for posts -->
|
||
|
|
<div class="mx-auto mt-4 w-full md:max-w-prose">
|
||
|
|
<h1 class="text-indigo-400 text-3xl">About Logging</h1>
|
||
|
|
|
||
|
|
<span class="text-md mt-2">Posted on 2021-04-16</span>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<!-- Actual article -->
|
||
|
|
<article class="prose lg:prose-lg text-white mt-4">
|
||
|
|
<p><em>TL;DR</em>: This post also talks about the problems I faced while working on my logging. To log to
|
||
|
|
syslog from within my containers that do not support configuring a remote syslog server, I had
|
||
|
|
<em>syslog-ng</em> expose a unix domain socket and mounted it into the container to <code>/dev/log</code>.</p>
|
||
|
|
<span id="continue-reading"></span><h2 id="introduction">Introduction</h2>
|
||
|
|
<p>I have written a lot of blog posts about the lessons I have learned while setting up and
|
||
|
|
maintaining my server. But now that I started to rework my infrastructure a bit, I had to
|
||
|
|
inevitably look at something I may have overlooked in the past: logging!</p>
|
||
|
|
<p>Previously, I had <em>Docker</em> <em>kind of</em> manage my logs: If I needed something, I would just
|
||
|
|
call <code>docker-compose logs <service></code> and it would spit out logs. Then, I started to
|
||
|
|
configure my services to log to files in various locations: my <em>prosody</em> server would
|
||
|
|
log to <code>/etc/prosody/logs/info.log</code>, my <em>nginx</em> to <code>/etc/nginx/logs/error.log</code>, etc.
|
||
|
|
This, however, turned out to be problematic
|
||
|
|
as, in my case, <em>prosody</em> stopped logging into the file if I rotated it with <em>logrotate</em>. It was
|
||
|
|
also a bit impractical, as the logs were not all in the same place, but distributed across multiple
|
||
|
|
directories.
|
||
|
|
Moreover, <em>prosody</em> was logging things that I did not want in my logs but I could not turn off,
|
||
|
|
like when a client connected or authenticated itself. For me, this is a problem from two perspectives:
|
||
|
|
On the one hand, it is metadata that does not help me debug an hypothetical issue I have with my
|
||
|
|
<em>prosody</em> installation, on the other hand, it is metadata I straight-up do not want to store.</p>
|
||
|
|
<p>My solution was using a syslog daemon to process the logs, so that I could remove logs that I do not
|
||
|
|
want or need, and drop them all off at <code>/var/log</code>. However, there was a problem that I faced almost
|
||
|
|
immediately: Not all software I can configure to log to syslog, I can configure to log to a specific
|
||
|
|
syslog server. Why is this a problem? Well, syslog does not work inside a <em>Docker</em> container out of the
|
||
|
|
box, so I would have to have my syslog daemon expose a TCP/UDP (unix domain) socket that logs can be sent to. To
|
||
|
|
see this issue you can try to run <code>logger -t SomeTag Hello World</code> inside one of your containers
|
||
|
|
and try to find it, e.g. in your host's journal.</p>
|
||
|
|
<p>Today, I found my solution to both syslog logging within the containers and filtering out unneeded logs.</p>
|
||
|
|
<h2 id="syslog-inside-containers">Syslog inside Containers</h2>
|
||
|
|
<p>The first step was getting the logs out of my containers without using files. To this end, I configured
|
||
|
|
my syslog daemon - <em>syslog-ng</em> - to expose a unix domain socket to, for example, <code>/var/run/syslog</code> and
|
||
|
|
mount it into all containers to <code>/dev/log</code>:</p>
|
||
|
|
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>source s_src {
|
||
|
|
</span><span> system();
|
||
|
|
</span><span> internal();
|
||
|
|
</span><span> unix-dgram("/var/run/syslog");
|
||
|
|
</span><span>};
|
||
|
|
</span></code></pre>
|
||
|
|
<p>If you now try and run <code>logger -t SomeTag Hello World</code> inside the container, you should be able
|
||
|
|
to find "Hello World" inside the host's logs or journals.</p>
|
||
|
|
<h2 id="ignoring-certain-logs">Ignoring Certain Logs</h2>
|
||
|
|
<p>The next step was ignoring logs that I do not need or care about. For this, I set up two logs within
|
||
|
|
<em>syslog-ng</em>: One that was going into my actual log file and one that was dropped:</p>
|
||
|
|
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>destination d_prosody {
|
||
|
|
</span><span> file("/var/log/prosody.log");
|
||
|
|
</span><span>};
|
||
|
|
</span><span>filter f_prosody {
|
||
|
|
</span><span> program("prosody");
|
||
|
|
</span><span>};
|
||
|
|
</span><span>filter f_prosody_drop {
|
||
|
|
</span><span> program("prosody")
|
||
|
|
</span><span> and message("(Client connected|Client disconnected|Authenticated as .*|Stream encrypted .*)$");
|
||
|
|
</span><span>};
|
||
|
|
</span><span>
|
||
|
|
</span><span># Drop
|
||
|
|
</span><span>log {
|
||
|
|
</span><span> source(s_src);
|
||
|
|
</span><span> filter(f_prosody_drop);
|
||
|
|
</span><span> flags(final);
|
||
|
|
</span><span>};
|
||
|
|
</span><span># Log
|
||
|
|
</span><span>log {
|
||
|
|
</span><span> source(s_src);
|
||
|
|
</span><span> filter(f_prosody);
|
||
|
|
</span><span> destination(d_prosody);
|
||
|
|
</span><span> flags(final);
|
||
|
|
</span><span>};
|
||
|
|
</span><span>
|
||
|
|
</span></code></pre>
|
||
|
|
<p>This example would log all things that <em>prosody</em> logs to the <em>prosody</em> location <code>d_prosody</code> and drop all
|
||
|
|
lines that match the given regular expression, which, in my case, matches all lines that relate to a client
|
||
|
|
connecting, disconnecting or authenticating.</p>
|
||
|
|
<p>Important is the <code>flags(final);</code> in the drop rule to indicate that a log line that matches the rule should
|
||
|
|
not be processed any further. That log also defines no destination, which tells <em>syslog-ng</em> in combination with
|
||
|
|
the <code>final</code> flag that the log
|
||
|
|
should be dropped.</p>
|
||
|
|
<p>Additionally, I moved the log rule that matches everything sent to the configured source to the bottom
|
||
|
|
of the configuration to prevent any of the logs to <em>also</em> land in the "everything" log.</p>
|
||
|
|
<p>Since I also host a <em>Nextcloud</em> server, I was also interested in getting rid of HTTP access logs. But I would
|
||
|
|
also like to know when someone is trying to scan my webserver for vulnerable <em>wordpress</em> installations.</p>
|
||
|
|
<p>So I again defined rules similar to those above, but added a twist:</p>
|
||
|
|
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>filter f_nextcloud_drop {
|
||
|
|
</span><span> program("nextcloud")
|
||
|
|
</span><span> and match("200" value(".nextcloud.response"));
|
||
|
|
</span><span>};
|
||
|
|
</span><span>log {
|
||
|
|
</span><span> source(s_src);
|
||
|
|
</span><span> parser { apache-accesslog-parser(prefix(".nextcloud.")); };
|
||
|
|
</span><span> filter(f_nextcloud_drop);
|
||
|
|
</span><span> flags(final);
|
||
|
|
</span><span>};
|
||
|
|
</span><span>
|
||
|
|
</span></code></pre>
|
||
|
|
<p>As you can see, the rule for my <em>Nextcloud</em> is quite similar, except that I added a parser. With this, I can
|
||
|
|
make <em>syslog-ng</em> understand the HTTP access log and expose its parts as variables to my filter rule. There,
|
||
|
|
I say that my drop rule should match all access log lines that indicate a HTTP response code of 200, since
|
||
|
|
those are locations on my server that I expect to be accessed and thus do not care about.</p>
|
||
|
|
<h2 id="conclusion">Conclusion</h2>
|
||
|
|
<p>With this setup, I feel much better about the logs I produce. I also have done other things not mentioned, like
|
||
|
|
configure <em>logrotate</em> to rotate my logs daily so that my logs don't grow too large and get removed after a day.</p>
|
||
|
|
<p>Please note that I am not an expert in <em>syslog-ng</em>. It just happend to be what I first got to do what I want. And
|
||
|
|
the example rules I showed are also the first thing that I wrote and filtered out what I wanted.</p>
|
||
|
|
|
||
|
|
</article>
|
||
|
|
|
||
|
|
<!-- Common post footer -->
|
||
|
|
<div class="mt-6">
|
||
|
|
<span class="prose lg:prose-lg text-md text-white">
|
||
|
|
If you have any questions or comments, then feel free to send me an email (Preferably with GPG encryption)
|
||
|
|
to papatutuwawa [at] polynom.me or reach out to me on the Fediverse at <a href="https://social.polynom.me/papatutuwawa">@papatutuwawa@social.polynom.me</a>.
|
||
|
|
</span>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
|
||
|
|
|
||
|
|
</div>
|
||
|
|
</body>
|
||
|
|
</html>
|