Here is the standard Loki log processing flow that I use for my logs.
The pipeline is comprised of the following stages:
adding job label (so that I can query all logs ingested from files) add directory label (by obtaining the directory name from the filename label) packing the filename label into the log entry using the stage.pack stage (reducing cardinality of the labels, querying can be done by the directory label) adding hostname and agent_hostname labels to the logs (agent_hostname refers to the machine running the agent, hostname is obtained from the logs.
Continue reading