Ingesting logs into Loki

Here is the standard Loki log processing flow that I use for my logs.

The pipeline is comprised of the following stages:

• adding job label (so that I can query all logs ingested from files)
• add directory label (by obtaining the directory name from the filename label)
• packing the filename label into the log entry using the stage.pack stage (reducing cardinality of the labels, querying can be done by the directory label)
• adding hostname and agent_hostname labels to the logs (agent_hostname refers to the machine running the agent, hostname is obtained from the logs. This is not implemented on my Windows Agent configuration, but it is designed for situations where agent might be handing logs from other sources, for example, syslog or event log)
• dropping the computer label
• dropping logs older than 1 hour (aligns with server side configuration to minimize errors)

All file logs enter at the loki.relabel.file.receiver, Windows event logs enter at the loki.relabel.default.receiver.

Ingesting SCCM logs into Loki

System Center Configuration Manager (SCCM) is still in use in my lab, mostly as a means of deployment of applications and updates. Whilst I’m working on moving some of the functionality into Intune, SCCM will remain for forseeable future the update orchestrator for my server environment.

SCCM logs seem to be standardized around two formats. Here are the examples of them:

Service is up and running.~~ $$<SMS_REST_PROVIDER><01-26-2025 17:35:20.122+00><thread=13372 (0x343C)>

<![LOG[Worker M365ADeploymentPlanWorker was triggered by timer.]LOG]!><time="00:02:06.5649051" date="12-20-2024" component="SMS_SERVICE_CONNECTOR_M365ADeploymentPlanWorker" context="" type="1" thread="126" file="">

DNS service discovery for Prometheus

Background story

Back when I ran SCOM, in addition of Windows machine monitoring and Event Log aggregation, it was performing a duty of ping testing all the servers in my environment. This was a very useful feature as it allowed me to quickly identify servers that were down or unreachable. The main quirk was that it was only aware of the servers that I manually added and not able to discover things automatically as I didn’t have the right setup to leverage the SNMP based discovery.

Veeam Exporter for Prometheus

Over the past couple of months I’ve put a considerable amount of time into deployment of a monitoring infrastructure in my home-lab that would replace Splunk and SCOM. In a way, this setup introduced a new level of monitoring which I did not have before, I’ve deeply fallen for metrics and the power of Prometheus and pretty much sunk into the Grafana’s LGTM ecosystem, quickly implementing Tempo and Loki for a full experience.