Ingesting logs into Loki

Here is the standard Loki log processing flow that I use for my logs. The pipeline is comprised of the following stages: adding job label (so that I can query all logs ingested from files) add directory label (by obtaining the directory name from the filename label) packing the filename label into the log entry using the stage.pack stage (reducing cardinality of the labels, querying can be done by the directory label) adding hostname and agent_hostname labels to the logs (agent_hostname refers to the machine running the agent, hostname is obtained from the logs. Continue reading

Ingesting SCCM logs into Loki

System Center Configuration Manager (SCCM) is still in use in my lab, mostly as a means of deployment of applications and updates. Whilst I’m working on moving some of the functionality into Intune, SCCM will remain for forseeable future the update orchestrator for my server environment. SCCM logs seem to be standardized around two formats. Here are the examples of them: Service is up and running.~~ $$<SMS_REST_PROVIDER><01-26-2025 17:35:20.122+00><thread=13372 (0x343C)> <![LOG[Worker M365ADeploymentPlanWorker was triggered by timer. Continue reading

DNS service discovery for Prometheus

Background story Back when I ran SCOM, in addition of Windows machine monitoring and Event Log aggregation, it was performing a duty of ping testing all the servers in my environment. This was a very useful feature as it allowed me to quickly identify servers that were down or unreachable. The main quirk was that it was only aware of the servers that I manually added and not able to discover things automatically as I didn’t have the right setup to leverage the SNMP based discovery. Continue reading

Veeam Exporter for Prometheus

Over the past couple of months I’ve put a considerable amount of time into deployment of a monitoring infrastructure in my home-lab that would replace Splunk and SCOM. In a way, this setup introduced a new level of monitoring which I did not have before, I’ve deeply fallen for metrics and the power of Prometheus and pretty much sunk into the Grafana’s LGTM ecosystem, quickly implementing Tempo and Loki for a full experience. Continue reading