Journal

I Need a New NAS

Sat, 27 Jan 2024 10:00:47 +0100

Context and Problem Statement

My current NAS, a QNAP TS-251+ with 4TB of usable space. I use about 2.5TB of the storage, so there is still some space left. When I set it up 6 years ago, I started with 500GB of used space. This is a monthly growth of about 30GB. Our current usage profile is

Automatically sync photos from our smartphones and use them as an archive
Search the photo library from both the PC and the smartphones
Sync services such as Nextcloud
Data storage for locally hosted services (Prometheus, Firefly III)
Central place to create and send offsite backups

There is still plenty of free space, but the NAS is becoming increasingly unusable. The CPU is too slow for the modern QTS operating system. The web interface is so laggy that routine operations such as updating the OS take around 20 minutes, with 18 minutes of waiting. Also, searching the photo archive on the smartphone via the QNAP apps takes ages. The QuMagic server is unusable on the TS-251+ because it clogs up the CPU.

What is more, I currently miss some fast scratch space for bigger data sets. As the CPU is so slow, any access via the network is bound by the CPU speed and not the disk speed.

The quality of the internal docker support is also rather bad. I currently have some applications still running, but the docker compose description used to configure the applications is not accessible in the Container Station UI anymore. So I can’t change them anymore.

Last but not least, I suspect that the hardware has some issues, as I observe random hard reboots of the NAS. This results in the RAID resyncing. This resync is also very slow and takes two days to complete for this 2 disk RAID.

Decision Drivers

It must be able to run Linux docker container to host small applications
It should consume very few watts of idle power
It should support hot swap disks
It should have 8TB of usable disk space
It should be in the price range of 400-600€

Considered Options

Build NAS myself and use General Purpose Linux OS
Build NAS myself and use specialized NAS Distributions
Rent a server/storage box
Buy a NAS

Decision Outcome

Chosen option: “Build NAS myself and use General Purpose Linux OS”, because I use a small set of services, and have the best flexibility

Consequences

Good, because the NAS has enough computing power to run also as small home server, or some compute intensive ad-hoc tasks.
Good, because I can use my Linux knowledge and the CLI to configure the system
Bad, because I have to spend time to set it up initially and keep it up to date

Pros and Cons of the Options

Build NAS myself and use General Purpose Linux OS

There are several guides around in the internet to build NAS with below 10 Watt idle power consumption. Asuming 10 Watt idle power, this is roughly 7,5 kWh per month, and 90 kWh per year. Assuming 35 cent/kWh, this translates to 31,50€ per year or 2,63€ per month in energy costs.

Good, because the NAS has enough computing power to run also as small home server, or some compute intensive ad-hoc tasks.
Good, because I can use my Linux knowledge and the CLI to configure the system
Bad, because I have to spend time to set it up initially and keep it up to date

Build NAS myself and use specialized NAS Distributions

Use homemade NAS hardware as above, but use a NAS distribution.

Good, because the NAS has enough processing power to run as a small home server or for some computationally intensive ad-hoc tasks.
Good, because I don’t have to spend as much time setting it up and keeping it up to date.
Bad, because they often don’t support Docker out of the box.

Rent a server/storage box

Renting a server or storage box has a higher monthly cost. In addition, we currently only have 30mbit/s upload and 100mbit/s download in our house. This makes any interactive work on large amounts of data very uncomfortable.

Good, because I don’t have to run the NAS in my basement.
Bad because the monthly cost is higher
Bad because my current internet connection is not fast enough to support fast viewing of our image library.

Buy a NAS

Buy a newer NAS from either QNAP or Synology and use their software.

Good, because I can buy a pre-built and pre-configured system.
Bad because these NAS are notoriously underpowered in terms of CPU.

TIL: jq exit code

Tue, 24 Oct 2023 06:00:47 +0100

jq is one of the most common tools I use in shell scripts and ad hoc shell commands. It allows you to parse, and query JSON data. Personally, I’m on the same page as Kelly Brazil, modern Unix tools should support both a human-readable text output and a structured text output, so that data can be safely processed in scripts. He wrote a great blog post back in 2019. He also wrote the great jc wrapper, which parses text output and generate structured JSON data from it.

Today, I learned, that jq can set its exit code depending on the last boolean expression in the jq query. This makes it very easy to write checks like: “Are there 10 infiniband interfaces on my machine?” This is because modern iproute2supports JSON based output. The command line for the above question is:

ip -j l | jq -e '.| map(select(.link_type == \"infiniband\")) | length == 10'

This will result in either exit code 1, of there are less or more then 10 infiniband interfaces.

Another powerful query is:

ip -j a | jq '.[] | .addr_info[] | select(.scope=="global" and .family=="inet6") | .local'

This gets you all IPv6 addresses on your machine with a global scope.

A note on Yaml and JQ

Learning jq also helps you with processing structured yaml data. With small scripts like yaml2json, you can convert yaml files to jq back and forth. There is also a tool called yq out there. Sadly, there are ad least three different implementations, with sligthly different command line arguements. If you are not in charge of the concrete dependencies, user of a script or Makefile have locally installed, I always use yaml2json combined with jq. As there is currently only a single jq implementation out there.

Monitoring nginx with mtail

Sun, 05 Feb 2023 10:00:47 +0100

I use prometheus to monitor my infrastructure and until now, the nginx serving this blog wasn’t monitored properly. Prometheus is a pull based monitoring system, scraping the monitored components via HTTP requests. Sadly, nginx doesn’t provide a native /metrics endpoint, serving metrics in the OpenMetrics format. Hence, we need a prometheus exporter. There is a nginx exporter provided by nginx INC nginx-prometheus-exporter. This exporter only provides basic information like the active connections, and total served connections.

To get more insights, you need to run nginx plus. Another option is to analyze the logs of nginx and generate Prometheus metrics from this. I can recommend mtail for this task. It’s reliable and performant.
Matt Probert has written a good introduction as GitHub Gist. Based on this example, I build my mtail configuration. The main difference is, that I use prometheus histograms to analyze the distribution of the request size and timing. Moreover, I use fewer labels to reduce the size of the prometheus datastore. Prometheus generates for every combination of labels a new time series in the database. Especially, the nginx_uri used in the GitHub Gist will explode the number of timeseries, if the URI contains an object ID in e.g. a REST API. The prometheus documentation advices to not use user provided IDs in prometheus labels. Also, the nginx_port and nginx_content_typeis irrelevant for my monitoring.

This is my current nginx.mtail program based in the example provided by Matt Probert:

# we stick with nginx var names here and prefix the exported metric with nginx_ by using "as"
# "by" allows us to specify our labelling/grouping
counter request_count             by nginx_host, nginx_method, nginx_status as "nginx_request_count"
histogram request_length_bytes    by nginx_host, nginx_method, nginx_status buckets 1024, 2048, 4096, 10000, 30000, 60000, 120000, 240000 as "nginx_request_length_bytes"
histogram bytes_sent              by nginx_host, nginx_method, nginx_status buckets 1024, 2048, 4096, 10000, 30000, 60000, 120000, 240000 as "nginx_bytes_sent"
counter body_bytes_sent           by nginx_host, nginx_method, nginx_status as "nginx_body_bytes_sent"
histogram request_time_hist       by nginx_host, nginx_method, nginx_status buckets 1, 10, 50, 100, 200, 400, 800, 1000, 2000 as "nginx_request_time_milliseconds"
histogram upstream_connect_time   by nginx_host, nginx_method, nginx_status buckets 1, 10, 50, 100, 200, 400, 800, 1000, 2000 as "nginx_upstream_connect_time_milliseconds"
histogram upstream_header_time    by nginx_host, nginx_method, nginx_status buckets 1, 10, 50, 100, 200, 400, 800, 1000, 2000 as "nginx_upstream_header_time_milliseconds"
histogram upstream_response_time  by nginx_host, nginx_method, nginx_status buckets 1, 10, 50, 100, 200, 400, 800, 1000, 2000 as "nginx_upstream_response_time_milliseconds"
counter nginx_log_nomatch_count

# the following pattern matches exactly the tab-separated fields in our custom access_log format
# if you want to customize...this pattern should be updated to match changes you make in the nginx log format.
/^/ +
/(?P<msec>\d+)\.\d+\t/ +       # settime() needs just the seconds so we exclude the .milliseconds part
/(?P<host>\S+)\t/ +
/(?P<server_port>\S+)\t/ +
/(?P<request_method>\S+)\t/ +
/(?P<uri>\S+)\t/ +
/(?P<content_type>\S+)\t/ +
/(?P<status>\S+)\t/ +
/(?P<request_length>\d+)\t/ +
/(?P<bytes_sent>\d+)\t/ +
/(?P<body_bytes_sent>\d+)\t/ +
/(?P<request_time>\d+\.\d+)\t/ +
/(?P<upstream_connect_time>\S+)\t/ +
/(?P<upstream_header_time>\S+)\t/ +
/(?P<upstream_response_time>\S+)/ +
/$/ {
    settime($msec)
    request_count[$host][$request_method][$status]++
    request_length_bytes[$host][$request_method][$status] = $request_length
    bytes_sent[$host][$request_method][$status] = $bytes_sent
    body_bytes_sent[$host][$request_method][$status] += $body_bytes_sent

    # Especially, for static files, the request time is often below a millisecond, hence 0. We set them to 1, to
    # count them in the histogram
    $request_time*1000.0 < 1 {
      request_time_hist[$host][$request_method][$status] = 1.0
    } else {
      request_time_hist[$host][$request_method][$status] = $request_time*1000.0
    }
    # mtail is happier doing counters with floats/ints.
    # nginx logs '-' when there isn't a value which
    # we check for - and skip updating these counters if found.
    # otherwise we cast the string to a float and increment the counter.
    $upstream_connect_time != "-" {
        upstream_connect_time[$host][$request_method][$status] = float($upstream_connect_time)*1000
    }
    $upstream_header_time != "-" {
        upstream_header_time[$host][$request_method][$status] = float($upstream_header_time)*1000
    }
    $upstream_response_time != "-" {
        upstream_response_time[$host][$request_method][$status] = float($upstream_response_time)*1000
    }
} else {
    # our pattern doesn't match.
    # in this example since we have a single program with a single log file...it should always match
    # or we have a bug.
    # can use this metric to detect when our parser is failing.
    nginx_log_nomatch_count++
}

I use the same nginx access log format as Matt Probert have shown in his gist:

log_format prometheus_log '$msec\t$host\t$server_port\t$request_method\t$uri\t$content_type\t$status\t$request_length\t$bytes_sent\t$body_bytes_sent\t$request_time\t$upstream_connect_time\t$upstream_header_time\t$upstream_response_time';

The dashboard I currently use looks like this:

The Grafana export can be found here

[Link] Architecture best practices for your small team

Wed, 14 Dec 2022 10:00:47 +0100

Yesterday, I published a longer text in the blog of my employer. I wrote how to structure your software architecture, as a small team targeting a non-hyper-growth market. Mainly, how to not let Kubernetes eat all your engineering time, yet being able to scale your application and shift it between clouds.

If you like the content here, you should definitly check out that text as well.

Things I learned about VMs with two interfaces on OpenStack

Tue, 29 Nov 2022 21:00:47 +0100

Sometimes you want to attach two interfaces to a OpenStack virtual machine (VM). As discussed in my last post, I had to disable OpenStacks port security feature to allow my IPVS realservers to implement direct server return. Disabling port-security has the downside, that my realserver can participate in other OpenStack security groups. For my use case, my realservers have to be part of a security group to reach certain backends. To implement that, I added a second network interface to my VM. It turned out, it was not easy as expected. This picture shows the setup:

The underlying OpenStack network implemented with Linux bridges is very picky about which traffic can be sent over which interface. By default, Linux is not configured for this setup. You have to change certain defaults to make this work. To sum the linked article up, Linux will respond by default to an ARP request received on any interface requesting an IP assigned to any local interface. Thus, in the example above, an ARP request received on ens3 requesting the IP 192.0.2.4 will be answered. This messes up the OpenStack network, since it contradicts the security model and assumes the response is spoofing. OpenStack compares ARP responses with the configured IP addresses in its network model. The second problem is, that for any packets originating on my realserver, only the interface ens3 is used. Even if the source IP is 192.0.2.4. This is due to the fact, that by default the first matching route is used. We use policy-based routing to fix this problem.

ARP

For ARP, we use the same settings as in my last post

1
2
3

sysctl -qw net.ipv4.conf.all.arp_ignore=1
sysctl -qw net.ipv4.conf.ens3.arp_ignore=1
sysctl -qw net.ipv4.conf.ens4.arp_ignore=1

As explained in my last post, the arp_ignore=1 sysctl settings will instruct the linux kernel only to answer ARP requests for IPs attached to the receiving interface. Thus, in our example, the kernel will only respond to ARP requests received via ens3 requesting the IP 192.0.2.3. ARP requests for the IP 192.0.2.4 are ignored on the interface ens3 after this setting is set.

Policy-Based Routing

The next problem we have to solve, is that the Linux kernel will send traffic originating from the IP 192.0.2.4 via the ens3 interface. This traffic will be dropped by OpenStack, since it assumes this is spoofed traffic. Furthermore, we want to use the security groups attached to ens4. To make sure, traffic originating from 192.0.2.4 is forwarded via the ens4 interface, we must use policy-based routing.

First, we create two new routing tables:

1
2

echo 200 table_ens3 > /etc/iproute2/rt_tables
echo 201 table_ens4 > /etc/iproute2/rt_tables

You can choose what ever name you want. I used table_ensX for clarity. You must however, choose a different ID for both tables. I have chosen the ID 200 and 201.

Next, we add the default routes to the two routing tables:

# Adding host route for our gateway
ip route add table table_ens3 192.0.2.1 dev ens3
# Adding default route via the gateway
ip route add table table_ens3 default via 192.0.2.1 dev ens3

# Adding host route for our gateway
ip route add table table_ens4 192.0.2.1 dev ens4
# Adding default route via the gateway
ip route add table table_ens4 default via 192.0.2.1 dev ens4

The ip route add table table_ensX instructs the ip route add command, to add the route to the routing table table_ensX. If you have other routing requirements, e.g. allowing use ens4 only for traffic within the subnet 192.0.2.0/24, you can do this:

1
2

# Adding route for subnet via the gateway
ip route add table table_ens4 192.0.2.0/24 via 192.0.2.1 dev ens4

The last step is to configure the policy-based routing. Therefore, we use the ip rule command:

# Use the table table_ens4 for routing lookups for the ip 192.0.2.4
ip rule add from 192.0.2.3 lookup table_ens3

# Use the table table_ens4 for routing lookups for the ip 192.0.2.4
ip rule add from 192.0.2.4 lookup table_ens4

To show the rules/policies in effect, you can use the command ip rule show. The output should look like this:

0:	from all lookup local
32764:	from 192.0.2.3 lookup table_ens3
32765:	from 192.0.2.4 lookup table_ens4
32766:	from all lookup main
32767:	from all lookup default

The first number is the priority. The rules are check from the lowest priority number (zero) to the highest (here 32767). The first matching rule is used.

To test your routing setup, the command ip route get is very handy. This let you check the where a packet would be routed. Doing this should give you an output like:

ip route get 8.8.8.8 from 192.0.2.3
8.8.8.8 via 192.0.2.1 dev ens3 src 192.0.2.3 table table_ens3 uid 1000 
    cache 
    
ip route get 8.8.8.8 from 192.0.2.4
8.8.8.8 via 192.0.2.1 dev ens4 src 192.0.2.4 table table_ens3 uid 1000
    cache

As you can see, ip route get shows you the device dev ens3 and the used table to retrieve that information. The downside of this set up is, you have to manage the routes in the tables table_ens3 and table_ens4 by yourself. Thus, you should to this only if your routing and the IPs are pretty static.

To sum it up, we have learned how to configure Linux to run with two interfaces in the same subnet. First, we have to reconfigure the default ARP answering behaviour. Secondly, we have to use policy-based routing to steer the egress traffic to the proper interfaces.

Things I learned about IPVS on OpenStack

Sun, 06 Nov 2022 21:00:47 +0100

Currently, I’m working with IPVS and OpenStack. This block post records some quirks I have found while setting up IPVS within a VM on top of OpenStack. None of these are unknown for experienced networking people, but I think they are worth to write down. In my setup I use IPVS with the TUN mode. Both the IPVS director and the server are located within the same OpenStack network. I have assigned the virtual IP (VIP) to the tunl0 interface of the realservers. This causes some ARP problems since, by default most linux servers answer ARP requests on any interface. For example, let’s assume our realserver has an interface eth0 attached to the OpenStack network. The VIP (192.168.48.3) is attached to tunl0. By default our realserver will answer any ARP request for the IP 192.168.48.3 on the interface eth0 as well. To deactivate this, I have configured

sysctl -qw net.ipv4.conf.all.arp_filter=1
sysctl -qw net.ipv4.conf.all.arp_ignore=1
sysctl -qw net.ipv4.conf.eth0.arp_filter=1
sysctl -qw net.ipv4.conf.eth0.arp_ignore=1

arp_ignore=1 instructs the kernel only to answer ARP requests, if the requested IP address is assigned to the network interface, the ARP request was received on. arp_filter does the same but will also respect your source based routing setup and will answer only if the kernel had routed an outgoing packet via the interface the ARP request was received on. I have set it for both all and the eth0 interface, since the maximum value of the value configured for all and an interface is used to determine the effective value for the interface. The IPVS knowledgebase has more details and other solutions for this problem.

The second and harder problem I ran into was that my realservers got the the SYN packet of a new connection via the tunl0 interface, but never replied with a SYN-ACK. Thus, the TCP three-way handshake could never succeed. After a very helpful tip of @awlnx on twitter I disabled rp_filter on the tunl0 interface. rp_filter is a spoofing protection enabled by default. It discards all packets received on an interface where the source IP address is not reachable via that interface. This is a very secure and advised default, but since we do asymmetrical routing it will drop our SYN packets received via the tunl0 interface. I have configured:

1
2

sysctl -qw net.ipv4.conf.tunl0.rp_filter=0
sysctl -qw net.ipv4.conf.all.rp_filter=0

Also, for both all and the tunl0 interface, because the maximum value is chosen. The default for rp_filter is 1.

The last problem I encountered where related to OpenStacks port security feature. I have added the VIP as allowed-address-pair on the eth0 interface of my realservers, yet the OpenStack networking layer drops the outgoing SYN-ACK packets from my realservers. The only solutions is to disable port security for the interface in OpenStack. I have yet to find out if this is related to the used OpenStack network implementation or a general OpenStack problem. Disabling port security entirely will also disable all security groups attached to that interface.

Explore Container Images Without A Shell

Fri, 19 Feb 2021 21:00:47 +0100

A best practice in building container images is to add as little dependencies as possible into the image. Most application don’t need a shell to run, hence dropping the shell from the container image reduces the size of the image and reduces the attack surface. Popular base images for that use case is either not using a base image at all and build the image FROM SCRATCH or use distroless. The downside is that we can’t open a shell inside the running container for debugging.

Kubernetes introduces ephemeral containers to spawn a debugging container into a running pod. To debug build issues or examine a container image before bringing it in production we have to have a look at the content of the filesystem in the image. In a container image with a shell installed we can create a container instance and launch a shell as entrypoint. This is not possible in images based on SCRATCH or distroless.

The container engine podman allows us to mount a container image into the filesystem of the developer machine. This allows us to examine the filesystem content of the container image with the tools installed on the developers machine.

Mounting the image

To demonstrate this feature of podman we want to examine the current distroless base image . Pull that image:

christoph@thinkpad-sport % podman image pull gcr.io/distroless/base-debian10
Trying to pull gcr.io/distroless/base-debian10:latest...
Getting image source signatures
Copying blob 1cfcc1bb8434 done  
Copying blob d94d38b8f0e6 done  
Copying config af31651e48 done  
Writing manifest to image destination
Storing signatures
af31651e48fe5f4a0788ad9812ba41315d3f9ef41fd5665313c619fdd20e0f6e

The next step is to create a new user namespace. This allows us to freely explore the container image as root user.

christoph@thinkpad-sport % podman unshare

root@thinkpad-sport # id    
uid=0(root) gid=0(root) groups=0(root),65534(nobody)

After this we can mount a container image into the user namespace.

1
2

root@thinkpad-sport # podman image mount gcr.io/distroless/base-debian10
/home/christoph/.local/share/containers/storage/vfs/dir/021295fa05fd0d2ac6b0eabbbc525bcfd6e2e1959f6ac5da80f97dabf9dc0fd1

The command returns the mount point where the image is mounted. Let examine the glibc version installed in the distroless image:

root@thinkpad-sport # cd /home/christoph/.local/share/containers/storage/vfs/dir/021295fa05fd0d2ac6b0eabbbc525bcfd6e2e1959f6ac5da80f97dabf9dc0fd1
root@thinkpad-sport # ls
bin  boot  dev  etc  home  lib  lib64  proc  root  run  sbin  sys  tmp  usr  var
root@thinkpad-sport # ls lib/x86_64-linux-gnu/
ld-2.28.so               libanl-2.28.so    libc.so.6        libmvec.so.1    libnss_compat-2.28.so  libnss_files.so.2       libnss_nis-2.28.so  libresolv-2.28.so    libthread_db.so.1
ld-linux-x86-64.so.2     libanl.so.1       libdl-2.28.so    libm-2.28.so    libnss_compat.so.2     libnss_hesiod-2.28.so   libnss_nis.so.2     libresolv.so.2       libutil-2.28.so
libBrokenLocale-2.28.so  libcrypt-2.28.so  libdl.so.2       libm.so.6       libnss_dns-2.28.so     libnss_hesiod.so.2      libpcprofile.so     librt-2.28.so        libutil.so.1
libBrokenLocale.so.1     libcrypt.so.1     libmemusage.so   libnsl-2.28.so  libnss_dns.so.2        libnss_nisplus-2.28.so  libpthread-2.28.so  librt.so.1
libSegFault.so           libc-2.28.so      libmvec-2.28.so  libnsl.so.1     libnss_files-2.28.so   libnss_nisplus.so.2     libpthread.so.0     libthread_db-1.0.so

Based on the filenames we assume a glibc 2.28 installed in the image. Since we know that the image is based on debian we can have a look at the dpkg database.

root@thinkpad-sport # cat -p var/lib/dpkg/status.d/libc6 | head -n 5
Package: libc6
Source: glibc
Version: 2.28-10
Architecture: amd64
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>

So we have the concrete package version 2.28-10. We can do the same to find the installed openssl version:

root@thinkpad-sport # cat var/lib/dpkg/status.d/libssl1 | head -n 5
Package: libssl1.1
Source: openssl
Version: 1.1.1d-0+deb10u4
Architecture: amd64
Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.org>

Cleanup

To clean up, exit the user namespace:

`1`	`root@thinkpad-sport # exit`

Last but not least delete the image:

1
2
3

christoph@thinkpad-sport % podman image rm gcr.io/distroless/base-debian10
Untagged: gcr.io/distroless/base-debian10:latest
Deleted: af31651e48fe5f4a0788ad9812ba41315d3f9ef41fd5665313c619fdd20e0f6e

Alternatives

An alternative to podman is dive. This tool let you explore the image layers in a ncurses UI. It focuses on the filesystem attributes of files and the changes between the image layers. Hence you can see the names and the attributes of a file but not the content.

Set Default Annotations with Kyverno

Fri, 05 Feb 2021 20:51:02 +0100

Kyverno is a policy engine for kubernetes. It allows you to implement ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks by creating (Cluster)Policy custom resources. Kyverno is less powerful than rego powered OpenPolicyAgent, but it is also easier to write.

Today I had to set an annotation to an Ingress resource if the user does not provide any value for this annotation.

This is very easy using the Add if not present anchor in a mutate rule. In our use case we want to set the alb.ingress.kubernetes.io/certificate-arn to the value arn:aws:acm:eu-central-1:XXXXXX:certificate/12345 if the user does not provide a value.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-default-arn
spec:
  rules:
    - name: set-default-arn
      match:
        resources:
          kinds:
            - Ingress
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              +(alb.ingress.kubernetes.io/certificate-arn): "arn:aws:acm:eu-central-1:XXXXXX:certificate/12345"

After applying the above ClusterPolicy, we can create this ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloworld
spec:
  rules:
    - host: helloworld.example.com
      http:
        paths:
          - backend:
              serviceName: helloworld
              servicePort: http
            path: /
            pathType: ImplementationSpecific

The resulting Ingress will have the annotation alb.ingress.kubernetes.io/certificate-arn set to the value arn:aws:acm:eu-central-1:XXXXXX:certificate/12345:

% kubectl apply -f ingress.yaml               
ingress.extensions/helloworld created
% kubectl get ingress helloworld -o json | jq '.metadata.annotations["alb.ingress.kubernetes.io/certificate-arn"]'
"arn:aws:acm:eu-central-1:XXXXXX:certificate/12345"

If we create the following Ingress, the kyverno policy will not change the annotation at all:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: "dont-touch-this"
  name: helloworld-dont-touch
spec:
  rules:
    - host: helloworld.example.com
      http:
        paths:
          - backend:
              serviceName: helloworld
              servicePort: http
            path: /
            pathType: ImplementationSpecific

Create the resource and observe the value of the annotation:

% kubectl apply -f ingress-dont-touch.yaml               
ingress.extensions/helloworld-dont-touch created
% kubectl get ingress helloworld -o json | jq '.metadata.annotations["alb.ingress.kubernetes.io/certificate-arn"]'
"dont-touch-this"

Combining this with the feature of matching rules on annotation, we can apply this annotation only to certain ALB ingress groups. The AWS loadbalancer controller, will create an ALB ingress for every value of alb.ingress.kubernetes.io/group.name. Certain settings must be equal across all ingresses from the same group. The following kyverno policy allows us to set the alb.ingress.kubernetes.io/scheme annotation to the same default value in the ingress group internal:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-default-internal-scheme
spec:
  rules:
    - name: set-default-internal-scheme
      match:
        resources:
           annotations:
             alb.ingress.kubernetes.io/group.name: "internal"
             kubernetes.io/ingress.class: "alb"
          kinds:
            - Ingress
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              +(alb.ingress.kubernetes.io/scheme): "internal"

After applying this ClusterPolicy every Ingress with the ingress class alb annotation, the ALB group internal will have the alb.ingress.kubernetes.io/scheme annotation set to internal if not set. If we want to enforce the value ìnternal for the annotation alb.ingress.kubernetes.io/scheme we can apply the following policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-internal-scheme
spec:
  rules:
    - name: enforce-internal-scheme
      match:
        resources:
           annotations:
             alb.ingress.kubernetes.io/group.name: "internal"
             kubernetes.io/ingress.class: "alb"
          kinds:
            - Ingress
      mutate:
        patchStrategicMerge:
          metadata:
            annotations:
              alb.ingress.kubernetes.io/scheme: "internal"

The difference is the missing +() anchor around the annotation key. This policy will overwrite any value of alb.ingress.kubernetes.io/scheme a created or changed ingress will have.

My Personal Backup Design

Fri, 08 Jan 2021 14:26:23 +0200

Compute devices may fail and lose your data. Hardware failures, software bugs and encrypting trojans threaten your data. And even if your computers don’t lose your data, a human error can erase a lot of data quickly. Without a backup you are either doomed to redo a lot of work or the precious pictures of the last years are lost forever. In this blogpost I want to show you the design goals I use to design my backup strategy. I will start with the goals itself and will outline my reasoning later and describe my implementation of this design.

Design Goals

Have two backups, an onsite and an offsite backup
Encrypt at least your offsite backup
Be able to restore your offsite backup without any of your personal computing devices
Use different backup tools for your onsite and offsite backups
Script your backups
Test a restore regularly
Synchronisation without version history is not a backup

Reasoning

The two kinds of backups are used to protect your data from different threads and enable restores on different time scales. The onsite backup protects your data from hardware failures, software bugs, encrypting trojans or human errors. The local backup target may be an external USB harddrive or a NAS. Since your backup target device is local to the source of the data, the network or USB speeds are quite high. This allows for a high frequency of backups. The frequency of backups are between every couple of minutes up to every day. The restore time is low since we can achieve a couple hundert MBit/s up to GBit/s over local network or USB connections.

In contrast the offsite backup is often connected with way lower speeds to your home. The offsite backup protects your data from a total loss of your home. Common causes of a total loss of your home is a fire, earth quake or if your local computing hardware is in the basement, a flood. As an offsite backup target you can either use a service like Backblaze, Amazon S3 Glacier or rsync.net via you internet connection or you can store a (USB) harddrive at a friends house or at work. An external harddrive costs no recuring fee, but the creation of a backup involves alot of repetitive manual labor. Thus it requires alot of discipline to get a regular backup. Using a service via the internet allows you to automate the complete backup process.

Regardless of your of your offsite backup target, you want to encrypt your backup since you can’t physical prevent any unauthorized third party from reading your data. In case of the harddrive backup target you can either encrypt the drive itself or use a tool encrypting your files in the filesystem. Most modern backup tools like restic or borg backup the data encrypted by default. Moreover they support to upload the backup directly to services like Backblaze, Amazon S3 Glacier or rsync.net via their native APIs.

Encrypting your backup data and using an external service requires you to memorize some credentials. Those credentials are required to access your data at the external service itself, and you need credentials to decrypt your backup. Todays best practice is to use a password manager to use completly random passwords for every service. As discussed earlier, an offsite backup protects your data from a total loss of your home. In this case it is very likely that you not only use your house but also all of your personal computing devices. Therefore you must be able to restore your offsite backup without any of your personal computing devices in your home. If you use a hardware token to access your password manager – as a service or an offline password manager like keepass – you should have a backup hardware token stored somewhere outside of your home. Hint: Storing the hardware token in the garage instead of your home, may not be the best idea. Disasters like fire, earthquake and floods are not limited to your home. Storing the credentials to your backup somewhere else introduce a conflict with your OpSec. This tradeoff can only made by yourself. If you manage the backup for multiple members of your familie or friends, consider to share the secrets required to restore the backup. Natural disasters threatens not only your data but also your life and thus the capability to decrypt your backup. If you don’t want to trust a single person, you can use Shamir's Secret Sharing to distribute parts of the secret to trusted persons. The Shamir’s Secret Sharing allows the reconstruction of the secret only if a certain number of persons combine their part of the secret. It’s like distributing your digital horcruxes.

Classify Backups

You must not apply the same backup policies and mechanisms to all of your data. Some data might not be that crucial to lose. For example the media library of music and videos you found during the last years in the internet. Those libraries often contain several terrabytes of data. Most backup services charge a requiring fee based on the size of the backed up data. Hence it might be too expensive to backup these files to a managed service. Moreover, in case of a total loss of your home, the loss of your media library might be a minor loss. The confidentialility of your data might be different. The pictures of your family should be restorable of every member of your family, but you might have some data personal to yourself. You can split these two classes of data into two different offsite backup archives with different credentials. The credentials to decrypt your family pictures are shared with your family members, but you don't share the credentials to decrypt your personal data with anybody else.

A common proverb around the IT operations community is “Nobody wants a backup, everyone wants a restore”. Since creating a backup is a dull repeating task, it is best automated. This reduces the likelyhood that you forget – intentionally or not – to create a backup. You should also test to restore your backups. There are many stories like these gitlab outtage around the internet. They had multiple backups automated, by the automation failed. Upon requiring a restore due to human error deleting production data, they found there backups empty. To prevent this, regularly test a restore. This ensures you know how to restore the data, and secondly you know your backup process work. A second major takeaway from that gitlab post mortem is to use different backup mechanism for your offsite and onsite backup. Two different backup mechanism ensure that a bug or configuration error in one of the backup tools only one backup corrupts.

Data synchronisation services like Dropbox, Google Drive or Qsync are used to synchronise data between multiple devices. Since they replicate the data onto multiple devices, some use them as their backup tool, too. This works perfectly fine if you want to prevent data loss due to a hardware failure. But synchronisation itself can’t prevent you from data loss due to human error, software bugs or encrypting trojans. In all those cases your data synchronisation service will replicate the unwanted data change - deletion or encryption - in real time onto every device. You can use them as backup if they support a file version history with a restore functionality. This allows you to restore an old version in case of an unwanted data change. If your data synchronisation tool doesn’t support a file version history it is not a proper backup tool.

My Implementation

The my NAS in the basement is my central data sink. The NAS is backed up daily to a connected external USB harddrive. To create this backup I use the build in QNAP backup tool. Once a day I create a snapshot of the NAS with restic into my Backblaze B2 bucket. The backed up dataset on the NAS is roughly 1 TB in size with a couple of hundred Megabytes added per day. Restic uploads only the added and changed chunks, hence the daily backup is finished on average after five minutes. The initial backup took 4 days with my 30 Mbit/s upload. I use resillio sync to sync the pictures taken by our smartphones to my NAS. As argued earlier this synchronisation itself is not a backup, but I backup the NAS. Hence, all my pictures are backed up automatically. My wife and I use free Dropbox accounts to share our daily documents like letters or forms. I synchronize my Dropbox account via the QNAP buildin synchronization tool to the NAS. This synchronized copy is also backed up. To enable me to access my backup without any of my computing devices I placed a printed and sealed copy of the credentials required to access the backup at another place. I did this OpSec tradeoff since the backup contains all the pictures of my family. The data private to myself is stored in encrypted data containers inside the global backup.

Linkdump: CPU alerts and why they are mostly useless

Tue, 15 Dec 2020 01:26:57 +0200

This article from my reading backlogs shows a nice summary why your CPU alerts are mostly useless. https://blog.koehntopp.info/2020/06/08/waffle-house-index-of-tooling.html

I never saw a useful application of CPU alerts. They generate alert noise and increase your alert fatigue since they are mostly unactionable.

Lookup Public IP via Commandline

Wed, 28 Oct 2020 01:26:57 +0200

When working from home, I have a changing public IP. My ISP assigns me a new public IP on every router reboot. For debugging I often have to know my current public IP. There are many websites you can visit and they will show you your public IP and advertisements.

To look it up from command line without advertisements or from a script, there are two main mechanism. First there are REST services answering your request with the IP they see as client IP, secondly there are some DNS resolver answering with the observed client IP if you query certain DNS records. In general the REST based service responses are easier to parse since the answer is machine-readable. It’s very easy to parse the JSON output with jq.

REST

Many REST services show more metadata they know about your IP. Mostly it is location data. This location data is very inaccurate. The most extensive response is from the ipdata.co IP. Using this API in a script may be a bad idea, since the API is rate limited. You can increase your rate limits by buying on of their plans.

curl https://api.ipdata.co?api-key=test

{
    "ip": "88.71.116.156",
    "is_eu": true,
    "city": "Neuburg",
    "region": "Rheinland-Pfalz",
    "region_code": "RP",
    "country_name": "Germany",
    "country_code": "DE",
    "continent_name": "Europe",
    "continent_code": "EU",
    "latitude": 48.9821,
    "longitude": 8.25,
    "postal": "76776",
    "calling_code": "49",
    "flag": "https://ipdata.co/flags/de.png",
    "emoji_flag": "\ud83c\udde9\ud83c\uddea",
    "emoji_unicode": "U+1F1E9 U+1F1EA",
    "asn": {
        "asn": "AS3209",
        "name": "Vodafone GmbH",
        "domain": "vodafone.de",
        "route": "88.64.0.0/12",
        "type": "isp"
    },
    "carrier": {
        "name": "Vodafone",
        "mcc": "262",
        "mnc": "02"
    },
    "languages": [
        {
            "name": "German",
            "native": "Deutsch"
        }
    ],
    "currency": {
        "name": "Euro",
        "code": "EUR",
        "symbol": "\u20ac",
        "native": "\u20ac",
        "plural": "euros"
    },
    "time_zone": {
        "name": "Europe/Berlin",
        "abbr": "CET",
        "offset": "+0100",
        "is_dst": false,
        "current_time": "2020-10-28T20:51:58.377385+01:00"
    },
    "threat": {
        "is_tor": false,
        "is_proxy": false,
        "is_anonymous": false,
        "is_known_attacker": false,
        "is_known_abuser": false,
        "is_threat": false,
        "is_bogon": false
    },
    "count": "1561"
}

The API of ipinfo.io is less extensive but I think they have no rate limit. As you can see from the example responses of ipinfo.io and ipdata.co, the location data is very inaccurate. Both location responses are in a 130km radius around my actual location.

curl http://ipinfo.io                  
{
  "ip": "88.71.116.156",
  "hostname": "dslb-088-071-116-156.088.071.pools.vodafone-ip.de",
  "city": "Schuttertal",
  "region": "Baden-Württemberg",
  "country": "DE",
  "loc": "48.2667,7.9500",
  "org": "AS3209 Vodafone GmbH",
  "postal": "77978",
  "timezone": "Europe/Berlin",
  "readme": "https://ipinfo.io/missingauth"
}%

DNS

The DNS based services are only returning the IP. One service is the google DNS:

1
2

dig TXT +short o-o.myaddr.l.google.com @ns1.google.com | awk -F'"' '{ print $2}'
88.71.116.156

OpenDNS also provide such a DNS name:

1
2

dig +short myip.opendns.com @resolver1.opendns.com
88.71.116.156

The Non-Obvious properties of JSON Encoder.Encode directly to http.ResponseWriter

Sun, 07 Jun 2020 01:26:57 +0200

Today I came across the following code snipped in a golang HTTP handler:

import(
	"encoding/json"
	"net/http"
	"fmt"
)

func handler(w http.ResponseWriter, r *http.Request) {
	// [... handler code...]
	if err := json.NewEncoder(w).Encode(myApplicationResponse); err != nil {
		fmt.Printf("could not marshal the response: %v\n", err)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
}

My first reaction was that this does not work as indented. The documentation of the ResponseWriter states the following:

If WriteHeader is not called explicitly, the first call to Write will trigger an implicit WriteHeader(http.StatusOK). Thus explicit calls to WriteHeader are mainly used to send error codes.

I thought that the Encoder will write to the given io.Writer while encoding the JSON. So this combined with the w.WriteHeader(http.StatusInternalServerError) in case of an error will cause a superfluous WriteHeader call, since the encoding called already w.Write() and thus setting the response code to 200. The superfluous WriterHeader call will be ignored. This is logicall, since the body is send to the client after the HTTP header. The status code is part of the header. Hence the status code must be written before the first body byte. The ReponseWriter streams the body to the client while the application HTTP handler is still running.

After digging into the Encoder code I learned that the JSON is encoded into a temporary buffer and after that copied to the given io.Writer.

I have three tackaways of this. First of all, the given code works as intented. It sets the status code 503 if the encoding fails. Secondly, the current behaviour (encode complete, than write) is not documented. It might change in the future. Hence the above snipped may not work as intended in the future. This post is written as of Golang 1.14.

The third takeaway is that Encoder.Encode can allocate alot of memory. For example, your JSON struct has a encoded size of 1MB and your application serves 1000 Requests/s. The memory consumption of the temporary encoding buffers alone is roughly 1 GB.

Managing Kubernetes Ingress JSON Log Format with Sanity

Sat, 30 May 2020 22:02:37 +0200

The nginx ingress supports a custom log format. The option can be set via the log_format field in the ingress configmap. As you can see, the log format must be a single line. I wanted to implement the Elasticsearch Common Schema for our nginx ingress access log. Writing the ECS JSON for our access log results in:

{
  "timestamp": "$time_iso8601",
  "network": {"forwarded_ip": "$http_x_forwarded_for"},
  "user":{"name":"$remote_user"},
  "user_agent":{"original":"$http_user_agent"},
  "http":{
    "version": "$server_protocol",
    "request":{
      "body":{"bytes":$body_bytes_sent},
      "bytes": $request_length,
      "method":"$request_method",
      "referrer":"$http_referer"
    },
    "response":{
      "body":{"bytes":$body_bytes_sent},
      "bytes": $bytes_sent,
      "status_code":"$status",
      "time":$request_time
    },
    "upstream": {
      "bytes": $upstream_response_length,
      "status_code":"$upstream_status",
      "time":$upstream_response_time,
      "address": "$upstream_addr",
      "name": "$proxy_upstream_name"
    }
  },
  "url":{
    "domain":"$host",
    "path":"$uri",
    "query":"$args",
    "original":"$request_uri"
  }
}

This 34 lines of JSON must be condensed to one line. Doing this manually is error prone. Moreover, maintaining this JSON struct in one line is nearly impossible. Since we manage the deployment of our ingress in a helm chart, I used the nospace sprig function. This function removes every whitespace from the input string. Be aware, this approach works only if your JSON struct contains no whitespace. The whitespaces from your JSON key and values will be stripped as well.

I saved the JSON struct to the file config/logpattern.json in the helm chart. My templates/configmap.yml looks like this:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress
data:
  log-format-escape-json: "true"
  log-format-upstream: '{{ .Files.Get "config/logpattern.json" | nospace }}'

The .Files.Get loads the JSON log pattern from the log pattern file and pipes that into the nospace function. This results in the following rendered configmap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress
data:
  log-format-escape-json: "true"
  log-format-upstream: '{"network":{"forwarded_ip":"$http_x_forwarded_for"},"user":{"name":"$remote_user"},"user_agent":{"original":"$http_user_agent"},"http":{"version":"$server_protocol","request":{"body":{"bytes":$body_bytes_sent},"bytes":$request_length,"method":"$request_method","referrer":"$http_referer"},"response":{"body":{"bytes":$body_bytes_sent},"bytes":$bytes_sent,"status_code":"$status","time":$request_time}},"url":{"domain":"$host","path":"$uri","query":"$args","original":"$request_uri","foo":"bar"}}'

Chart directory layout:

.
├── Chart.yaml
├── config
│   └── logpattern.json
├── templates
│   ├── configmap.yml
│   ├── ....
....
│   └── ....
└── values.yaml

Moving the NAS

Wed, 27 May 2020 22:32:43 +0200

I had to move my NAS into the basement. There is no room in our flat left for noisy computer equipment. My flat is in the first floor. I have no chance to build a wired ethernet connection into the basement. So I had to build up a wireless connection into the basement. The current setup is drawed in the following pictures:

The settings of the OpenWRT Router are:

Hardware: TP-Link Archer AC1750 C7 v5
Software: OpenWrt 19.07.3 r11063-85e04e9f46

My first approach was to just bridge a WiFi interface of the router to the ethernet ports. However, I learned, that bridging between WiFi and ethernet is not that simple. The root cause is a missing source MAC address in the 802.11 standard. The best solution to this problem is using WDS. This switches WiFi to a four MAC address mode. The details are in the linked article. I didn’t get WDS working. Therefore I used relayd as workaround. I followed the setup in the linked article. Relayd has some drawbacks. Relayd does not support IPv6. Furthermore it does Layer 3 things to emulate Layer 2 connectivity.

I have a static route (192.168.88.0/24) installed on my NAS to route traffic for my VPN to my raspberry Pi. After the initial relayd setup I was unable to connect from my VPN to the NAS. Connecting to the NAS from any device in my home network to the NAS worked. Including the raspberry Pi terminating the VPN. I tracerouted a target in the VPN from the NAS. Packets for the VPN host where routed by my easybox into the network of my ISP.

After some debugging (tcpdump yeah!) I found that my NAS did a correct nexthop and ARP lookup. It set the raspberry Pis MAC address as target. However, the relayd on the OpenWRT router redoes the nexthop and ARP lookup on relaying the packets. Installing the static route on the OpenWRTs main routing table solved the problem. To be sure I installed the route on both the relayd interface, and the WiFi uplink interface. After this the NAS was able to connect to hosts in the VPN.

This twitter thread documents my debugging rants in German.

tl;dr

relayd resets the nexthop. If a host behind your relayd has static routes, you must have that route on the OpenWRT in the main routing table.

Set Default Value of Prometheus Label if not set

Wed, 13 May 2020 08:37:21 +0200

I had to implement the following behaviour in a prometheus relabel config:

Given two labels A and B. I want to set A to the value of label B, if and only if A is empty/not-set.

First I tried to find a proper action in the prometheus relabel_config, but I did not find any. Luckily the replace action only replaces the target label if the given regex matches. Therefore, the following relabel config has the wanted property:

- source_labels: [A, B]
  action: replace
  separator: ";"
  regex: ;(.*)
  replacement: $1
  target_label: A

It works because the values of the source_labels are joined by the separator. After concatenating the two values, the regex is matched against the concatenated values. The regex above, only matches if A is empty. The value of B will be matched by the first and only group reference. Hence replace only replaces the value of label A if it is empty.

On Errors in Golang

Mon, 11 May 2020 21:37:21 +0200

In Golang are two ways to signal an error in a function or method. The first is the convention to return an error value as last return parameter of a function or method. The position of the return parameter is only a convention, but I’m not aware of any library not using this convention. The error type is a builtin interface.

The second way to signal an error is raising a panic. A panic have to be catched via the recover method in a defer block in the call stack. If the panic is not catched it will crash the program. This sounds like exceptions in languages like Java, but panics should only be used if there is something unexpected and non-recoverable. In general, panics should only be recovered to post them services like sentry.io. After delivering the panic to your observability stack, the program should be terminated. This blog post goes into the details of panics, defer and recover.

As described in the first paragraph, the convention is to return an error if the function can return an error. Since it is a good practice to check every error go code has a lot of if err != nil statements. There are a lot of people out there hating that pattern. I got used to it. Now it shapes my way of thinking. Before doing anything with the return value of function I think about the error cases. What should the code do in case of an error? Abort? Does the code have to clean up any resources? Rollback any transaction? Can the code recover safely from the error? Yes this distracts from the core business logic in the first place, but I think it leads overall to a more robust code.

The Must Pattern

However, always checking for that possible error can be annoying in cases where the function returns an error, but I know at time of programming there will never be an error. Or, if there is an error, some fundamental assumptions broke. A good example is the Compile function from the regexp package. This function has the following signature:

`1`	`func Compile(expr string) (*Regexp, error)`

The function tries to compile the given string as regular expression. If the string is not a valid regular expression an error is returned. So far, the usual method signature. There is a possible error, hence return error. In the following example, the error check is never true, since I know in advance that the static expression is always a valid expression.

package main

import (
    "log"
    "regexp"
)

func main() {
    exp := "foo.*"
    input := []byte("foobar")
    r, err := regexp.Compile(exp)
    if err != nil {
        log.Fatal("Failed to compile expression")
    }
    log.Printf("%v matched: %v", input, r.Match(input))
}

The regexp package provides us a handy wrapper around Compile. It is called MustCompile and returns only a *Regexp. If Compile returns an error not equals nil, MustCompile panics. This reduces the above code to:

package main

import (
    "log"
    "regexp"
)

func main() {
    exp := "foo.*"
    input := []byte("foobar")
    r := regexp.MustCompile(exp)
    log.Printf("%v matched: %v", input, r.Match(input))
}

Personally I use MustCompile only if I know the regular expression at compile time. If my program fails to parse the regular expression, something fundamental in the regexp library has changed. If it depends on user input you should never use MustCompile since the user may mess up your regular expression.

There are other packages providing a methods and function with a Must prefix. E.g. MustRegister from the prometheus client library, or MustParse from the uuid package. All those functions panic instead of returning an error. All those functions should only be used if you know for sure an error can’t happen. Vice versa, the Must prefix of a function indicates the user of your package that your function will panic on error. I made it a rule of thumb for myself to prefix every function with Must if I implement a panic in that function.

If your package API has many functions returning the same type, and an error you can copy the Must-wrapper pattern from the template package. This wrapper has the following signature:

`1`	`func Must(t Template, err error) Template`

It accepts a Template and an error. In case err!=nil it will panic. The example for this function shows a lesser known feature of golang. Example:

`1`	`var t = template.Must(template.New("name").Parse("text"))`

template.New("name") returns a Template and an error. Since Must accept exactly those two arguments, you can just pass the output of template.New into Must. This saves a lot of Must prefixed functions in the API of the template package. Sadly, the following example will not compile:

package main

import (
	"fmt"
)

func sub(a, b int) int {
	return a - b
}

func add(a, b, c int) int {
	return a + b + c
}

func main() {
	res := add(sub(1, 2), 3)
	fmt.Printf("(1-2)+3=%d\n", res)
}

The error message is:

1
2
3

./prog.go:16:12: not enough arguments in call to add
	have (int, number)
	want (int, int, int)

If you want to pass multiple return values of a function directly into another function, the receiving function must have exactly the same input values.

In conclusion, use the prefix Must if your method or function panics. Be careful if an API provides a Must function. Expect that function to panic.

On Using Fluentd to Parse Elastic Common Scheme from Kubernetes Pod Logs

Fri, 08 May 2020 21:39:47 +0200

Today I had to parse the JSON logs of our containers in Kubernetes. Our application are logging in the Elastic Common Scheme format to STDOUT. This format is a JSON object with well-defined fields per log line. We use a fluentd daemonset to read the container logs from the nodes. The outputs of STDOUT and STDERR are saved in /var/log/containers on the nodes by the docker daemon. This setup is very common in the Kubernetes environment. The official fluend-elasticsearch addon uses the same approach. In this article I will show the problems I encountered while parsing these logs.

The fluentd input to read the container logs is:

  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </source>

As you can see in the above snippet the input parses the container logs as JSON via the fluentd JSON parser. The format of the container log files is the docker default json-file (details). If the container prints the string this is a log line\n to STDOUT, the docker daemon saves the following line to the container log file:

`1`	`{"log":"this is a log line\n","stream":"stdout","time":"2019-01-01T11:11:11.111111111Z"}`

After passing the fluentd input, the fluentd event has exact three fields. log, stream and the timestamp.

Since our container are logging JSON to STDOUT, the lines in the container logs looks like the following:

1
2

{"log":"{\"message\":\"this is a log line\", \"log\":{\"level\":\"info\"}}","stream":"stdout","time":"2019-01-01T11:11:11.111111111Z"}

This is where the problems start. We told the input to parse every line has JSON. The resulting fluentd event looks like this:

{
  log => "{"message":"this is a log line", "log":{"level":"info"}}",
  stream => stdout,
  time => "2019-01-01T11:11:11.111111111Z",
}

The fluentd JSON parser can not parse the output of our application as JSON since the docker daemon escaped the JSON in the log string. To solve this I used a filter with a JSON parser:

<filter **>
  @type parser
  key_name "log"
  reserve_data true
  reserve_time true
  hash_value_field "parsed"
  remove_key_name_field true
  replace_invalid_sequence true
  emit_invalid_record_to_error false
  <parse>
    @type "json"
  </parse>
</filter>

After our fluentd event passed this filter it looks like this:

{
    {
      parsed = > {
        message => "this is a log line",
        log => {
           level => "info"
        },
      },
      stream => stdout,
      time => "2019-01-01T11:11:11.111111111Z",
    }
}

The parser removes the original log line from the event since we set the option remove_key_name_field. We also tell the JSON parser to keep all data, and the already parsed timestamp via the options reserve_time and reserve_data. Some applications in the cluster are not logging in JSON format. Therefore, we set the option emit_invalid_record_to_error to false. Without that setting, every non-JSON log line get the @ERROR label and require special treatment by other filters.

The ECS logging scheme does not have a field parsed. Moreover, the field message is at the top-level of the object. To mutate records, fluentd has the record_transformer filter. This filter can set event field values to values of other fields, but can’t copy the content of the field parsed to the top-level of the event. This was the hardest problem I encountered. After some googling I found the filter record_modifier. I applied the trick shown in the record_modifier README to implement complex logic in ruby.

<filter **>
  @type record_modifier
  remove_keys _dummy_,parsed
  <record>
    _dummy_ ${record.merge!(record["parsed"])}
  </record>
</filter>

We set the field _dummy_ to the result of the ruby expression ${record.merge!(record["parsed"])}. The content of our fluentd event is the hash record. The function merge!merges the event in place with the values of the field record["parsed]. The result of the merge is irrelevant. Therefore we drop the field _dummy_ after applying the filter. The filter deletes also the merged field parsed. After applying that filter our event looks like this:

{
    {
        message => "this is a log line",
        log => {
           level => "info"
        },
        stream => stdout,
        time => "2019-01-01T11:11:11.111111111Z",
    }
}

This event complies with the Elastic Common Scheme. The final fluentd configuration is:

  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </source>

  <filter **>
    @type parser
    key_name "log"
    reserve_data true
    reserve_time true
    hash_value_field "parsed"
    remove_key_name_field true
    replace_invalid_sequence true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
    </parse>
  </filter>

  <filter **>
    @type record_modifier
    remove_keys _dummy_,parsed
    <record>
      _dummy_ ${record.merge!(record["parsed"])}
    </record>
  </filter>

  <match my.logs>
    @type elasticsearch
    host localhost
    port 9200
    logstash_format true
  </match>

Mount Local GOPATH Directory into Container

Fri, 08 May 2020 01:00:00 +0200

Given you have an old golang project depending on the GOPATH. You want to build that project in the golang docker image. This one liner mounts the current directory into the GOPATH of the golang image.

docker run -it \
	-w /go/src/${$(pwd)#$GOPATH/src/} \
	-v $(pwd):/go/src/${$(pwd)#$GOPATH/src/} \
	golang:1.10

Example:

user@host:/home/user/go/src/github.com/hikhvar/zkillbeat (git)-[master] % docker run -it \                                                               :(
        -w /go/src/${$(pwd)#$GOPATH/src/} \
        -v $(pwd):/go/src/${$(pwd)#$GOPATH/src/} \
        golang:1.10
root@5c7f5e394b2a:/go/src/github.com/hikhvar/zkillbeat# go build

Migrate Elasticsearch 5.x to 6.x

Mon, 04 May 2020 20:40:00 +0200

We want to migrate an pre 6.0.0 cluster to 6.x.x. Therefore we have to make sure every index contains only one mapping. The following snippets shows how to inspect the indices with default unix tools.

Required tools:

cat
jq
curl
mkdir
wc
echo

To decrease the load on the cluster we download the mappings of every index to /tmp/indices.

1
2
3

esHost="http://localhost:9200"
mkdir /tmp/indices
for i in $(curl "${esHost}/_cat/indices?h=index"); do curl "${esHost}/${i}/_mapping" > /tmp/indices/${i}.json; done

Show the number of mappings for all indices:

`1`	`for i in $(ls -1 /tmp/indices); do echo -n $i " ";jq '.[].mappings \| to_entries[].key' /tmp/indices/${i} \| wc -l; done`

Show the names of mappings in an index:

1
2

index="index-name"
cat /tmp/indices/${index}.json | jq ".[\"${index}\"].mappings | to_entries[].key"

Initial Commit

Mon, 04 May 2020 20:15:41 +0200

Initial post to test the theme and all the other stuff.