Table of contents

1. How Metrics Work in ColdBrew
2. Application Metrics Package (Cookiecutter)
3. How to Add Custom Metrics
4. OpenTelemetry Metrics (OTLP Export)
   1. Enabling OTEL Metrics
   2. How it relates to Prometheus
5. Circuit Breaker Metrics

How Metrics Work in ColdBrew

ColdBrew uses Prometheus to collect service metrics. By default, ColdBrew will expose a /metrics endpoint that will return the metrics in the Prometheus exposition format on the configured HTTP port.

A collection of metrics are collected by default, including:

• Golang runtime metrics (e.g. memory usage, goroutine count, etc.)
• gRPC Client/Server metrics (e.g. request count, request duration, etc.)
• HTTP request metrics (e.g. request count, request duration, etc.)

Application Metrics Package (Cookiecutter)

Projects generated from the ColdBrew cookiecutter include a starter service/metrics/ package with an interface-based pattern:

service/metrics/
├── types.go       # Metrics interface (mockable via mockery)
├── metrics.go     # Implementation using promauto
├── labels.go      # Label constants (OutcomeSuccess, OutcomeError)
└── metrics_test.go

The interface enables dependency injection and test mocking. Sample metrics included:

Metric	Type	Description
<app>_echo_total	Counter	Echo RPC calls by outcome
<app>_echo_duration_seconds	Histogram	Echo RPC duration in seconds

Usage in handlers follows the defer pattern for automatic timing:

func (s *svc) Echo(ctx context.Context, req *pb.EchoRequest) (resp *pb.EchoResponse, err error) {
    start := time.Now()
    outcome := metrics.OutcomeSuccess
    defer func() {
        if err != nil {
            outcome = metrics.OutcomeError
        }
        s.monitoring.IncEchoTotal(outcome)
        s.monitoring.ObserveEchoDuration(outcome, time.Since(start))
    }()
    // ... business logic ...
}

To add a new metric: add the method to the Metrics interface in types.go, implement it in metrics.go, run make mock to regenerate the mock.

Duration metrics use seconds (not milliseconds) following Prometheus naming conventions. Grafana handles unit display automatically.

How to Add Custom Metrics

You can also add metrics directly using the Prometheus Go client library and registering them with the default Prometheus registry:

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    myCounter = promauto.NewCounter(prometheus.CounterOpts{
        Name: "my_counter",
        Help: "The total number of processed events",
    })
)

func myFunction() {
    ...
    myCounter.Inc()
    ...
}

These metrics will be automatically collected and exposed by ColdBrew on the /metrics endpoint.

To learn more about the Prometheus and the data types it supports, see here

OpenTelemetry Metrics (OTLP Export)

In addition to Prometheus, ColdBrew can export gRPC metrics via OpenTelemetry’s OTLP protocol. This is useful when your observability stack uses an OTLP-compatible backend (Grafana Cloud, Datadog, Honeycomb, etc.) and you want metrics alongside traces in the same pipeline.

OTEL metrics export is opt-in and runs alongside Prometheus — it does not replace the /metrics endpoint. Both can be active at the same time.

Enabling OTEL Metrics

Set the following environment variables:

export ENABLE_OTEL_METRICS=true
export OTEL_METRICS_INTERVAL=60    # export interval in seconds (default: 60)
export OTLP_ENDPOINT=localhost:4317 # same endpoint used for traces

When enabled, ColdBrew exports standard gRPC OpenTelemetry metrics via the native grpc/stats/opentelemetry package:

Metric	Type	Description
grpc.server.call.started	Counter	Server RPCs started
grpc.server.call.duration	Histogram	Server RPC duration
grpc.server.call.sent_total_compressed_message_size	Histogram	Server response size
grpc.server.call.rcvd_total_compressed_message_size	Histogram	Server request size
grpc.client.call.duration	Histogram	Client RPC duration
grpc.client.attempt.started	Counter	Client RPC attempts

Health check, readiness, and server reflection RPCs are bucketed under a generic "other" method label to reduce cardinality — they still generate data points but won’t create high-cardinality method attributes.

How it relates to Prometheus

Aspect	Prometheus (/metrics)	OTEL Metrics (OTLP)
Protocol	Pull (scrape)	Push (OTLP gRPC)
Metric names	grpc_server_handled_total, etc.	grpc.server.call.duration, etc.
Custom app metrics	promauto.NewCounter(...)	Not exported (Prometheus only)
Enabled by default	Yes	No (ENABLE_OTEL_METRICS=true)
Endpoint config	None (built-in)	OTLP_ENDPOINT (shared with traces)

Both export pipelines use independent metric names and registries, so there is no conflict or double-counting.

Circuit Breaker Metrics

Circuit breaker metrics are the responsibility of the resilience library you plug into ColdBrew’s executor hook. For example, with failsafe-go you can attach event listeners to track circuit state:

cb := circuitbreaker.NewBuilder[any]().
    WithFailureThreshold(5).
    WithDelay(5 * time.Second).
    OnStateChanged(func(e circuitbreaker.StateChangedEvent) {
        circuitStateGauge.WithLabelValues(commandName).Set(float64(e.NewState))
    }).
    Build()

ColdBrew’s built-in grpc_client_* Prometheus metrics (request count, duration, status codes) cover request-level observability automatically. See the integrations page for full executor setup examples.

Legacy: The previous hystrixprometheus package and core.SetupHystrixPrometheus() are deprecated. Circuit breaker metrics are now managed by your chosen resilience library, not ColdBrew.

---