When something goes wrong in a distributed system, the hardest part isn’t fixing it, it’s understanding what happened. This post walks through a technique for tracing a single request across multiple services.

The problem

You have a user-facing API that fans out to several internal services. A request comes in, something fails, and the logs across five different services are a jumble of timestamps and IDs that don’t obviously connect.

The request flow looks like this:

flowchart LR
    Client -->|HTTP| API["API Gateway"]
    API -->|gRPC| Auth["Auth Service"]
    API -->|gRPC| Catalog["Catalog Service"]
    Catalog -->|SQL| DB[(Database)]
    API -->|event| Queue[("Message Queue")]
    Queue --> Worker["Background Worker"]

Propagating a trace ID

The simplest thing that works: generate a UUID at the edge and thread it through every hop.

In Java, a servlet Filter sits at the HTTP boundary and either reads the incoming X-Trace-ID header or mints a fresh UUID. MDC (Mapped Diagnostic Context from SLF4J) carries the trace ID on the current thread so every log statement picks it up automatically, with no extra arguments at the call site:

package com.example.middleware;

import jakarta.servlet.*;
import jakarta.servlet.annotation.WebFilter;
import jakarta.servlet.http.*;
import org.slf4j.MDC;
import java.io.IOException;
import java.util.UUID;

@WebFilter("/*")
public class TraceFilter implements Filter {

    public static final String TRACE_HEADER = "X-Trace-ID";

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {
        var req = (HttpServletRequest) request;
        var res = (HttpServletResponse) response;

        var traceId = req.getHeader(TRACE_HEADER);
        if (traceId == null || traceId.isBlank()) {
            traceId = UUID.randomUUID().toString();
        }

        res.setHeader(TRACE_HEADER, traceId);
        MDC.put("trace_id", traceId);
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.remove("trace_id");
        }
    }
}

Wire the key into your Logback pattern once and every line gets it for free:

<!-- logback.xml -->
<pattern>%d{ISO8601} [%thread] %-5level %logger{36} trace_id=%X{trace_id} - %msg%n</pattern>

A note on in-process propagation: MDC is backed by a ThreadLocal, which works fine for request-scoped virtual threads but does not auto-propagate when you launch child virtual threads inside a request. For that case, ScopedValue (finalized in Java 25) is the right tool: it is immutable within a scope, composable across forks, and avoids the mutable-state hazards of ThreadLocal. In practice, use both: MDC for log-line decoration, ScopedValue for explicit propagation into subtasks running on their own virtual threads.

Every service logs with this ID:

log.info("request received, service=catalog, method=GetProduct");
// 2026-03-27T10:00:00Z [virtual-8] INFO CatalogService trace_id=abc-123 - request received, service=catalog, method=GetProduct

The trace ID appears in the output because it lives in the MDC, not because anyone passed it explicitly. When the API Gateway fans out via HTTP, it forwards the header to downstream services:

var client = HttpClient.newBuilder()
    .executor(Executors.newVirtualThreadPerTaskExecutor())
    .build();

var request = HttpRequest.newBuilder()
    .uri(URI.create("http://catalog/products/" + productId))
    .header(TraceFilter.TRACE_HEADER, MDC.get("trace_id"))
    .GET()
    .build();

var response = client.send(request, HttpResponse.BodyHandlers.ofString());

Write it as blocking code. Virtual threads release the carrier thread while the socket waits, so you get async throughput with synchronous readability: no reactive pipelines, no callback hell.

Querying across services

With structured logs and a consistent trace_id field, you can pull the full story:

# Loki / LogQL
{app=~"api|auth|catalog|worker"} | json | trace_id = "abc-123"

# or with grep if you're old-fashioned
grep "abc-123" /var/log/services/*.log | sort -t'T' -k2

What comes next

Once you have trace IDs flowing, you’re one step away from proper distributed tracing with OpenTelemetry . The manual approach above is a good way to understand what tracing actually does before adding the framework overhead.

The state machine of a typical request looks like this:

stateDiagram-v2
    [*] --> Received
    Received --> Authenticated: auth ok
    Received --> Failed: auth error
    Authenticated --> Processing
    Processing --> Queued: async path
    Processing --> Responded: sync path
    Queued --> Completed
    Responded --> [*]
    Completed --> [*]
    Failed --> [*]

Start simple. Add structure. Then add tooling.