Skip to main content

Arrow-Micrometer

A Micrometer metrics library that forwards application metrics to any HTTP server using Apache Arrow format.

Overview​

This library provides a custom Micrometer MeterRegistry implementation that:

  • Collects standard Micrometer metrics (counters, gauges, timers, distribution summaries, etc.)
  • Serializes metrics to Apache Arrow format for efficient transport
  • Forwards metrics to any HTTP server that accepts Arrow data
  • Supports batching, retries, and disk buffering for reliability

Requirements​

  • Java 21+
  • Maven 3.6+

Installation​

Add the dependency to your pom.xml:

<dependency>
    <groupId>io.dazzleduck.sql</groupId>
    <artifactId>dazzleduck-sql-micrometer</artifactId>
    <version>0.0.13-SNAPSHOT</version>
</dependency>

Quick Start​

Create an application.conf file in your resources:

dazzleduck_micrometer = {
  application_id   = "my-app"
  application_name = "My Application"
  application_host = "localhost"
  enabled          = true

  step_interval_ms = 10000

  min_batch_size       = 1048576    # 1 MB
  max_batch_size       = 16777216   # 16 MB
  max_send_interval_ms = 1000
  max_in_memory_bytes  = 10485760   # 10 MB
  max_on_disk_bytes    = 1073741824 # 1 GB
  retry_count          = 3
  retry_interval_ms    = 1000
  transformations      = []
  partition_by         = []

  http {
    base_url               = "http://localhost:8081"
    username               = "admin"
    password               = "admin"
    target_path            = "metrics"
    http_client_timeout_ms = 5000
  }
}

Then use the factory:

import io.dazzleduck.sql.micrometer.metrics.MetricsRegistryFactory;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;

public class MetricsExample {
    public static void main(String[] args) {
        // Create registry from application.conf
        MeterRegistry registry = MetricsRegistryFactory.create();

        // Record metrics
        Counter requestCounter = registry.counter("http.requests", "method", "GET", "path", "/api/users");
        requestCounter.increment();

        // ... your application code ...
    }
}

Option 2: Programmatic Configuration​

import io.dazzleduck.sql.micrometer.MicrometerForwarder;
import io.dazzleduck.sql.micrometer.config.MicrometerForwarderConfig;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;

import java.time.Duration;

public class MetricsExample {
    public static void main(String[] args) {
        // Build configuration
        MicrometerForwarderConfig config = MicrometerForwarderConfig.builder()
                .baseUrl("http://localhost:8081")
                .username("admin")
                .password("admin")
                .targetPath("metrics")
                .applicationId("my-app")
                .applicationName("My Application")
                .stepInterval(Duration.ofSeconds(10))
                .build();

        // Create and start the forwarder
        MicrometerForwarder forwarder = MicrometerForwarder.createAndStart(config);

        // Get the registry and record metrics
        MeterRegistry registry = forwarder.getRegistry();

        Counter requestCounter = registry.counter("http.requests", "method", "GET", "path", "/api/users");
        requestCounter.increment();

        // Cleanup on shutdown
        Runtime.getRuntime().addShutdownHook(new Thread(forwarder::close));
    }
}

Usage Examples​

Counter​

Counter counter = registry.counter("orders.placed", "region", "us-east");
counter.increment();
counter.increment(5);

Gauge​

AtomicInteger activeConnections = new AtomicInteger(0);
registry.gauge("db.connections.active", activeConnections);

activeConnections.set(10);

Timer​

Timer timer = registry.timer("http.request.duration", "endpoint", "/api/orders");

// Record duration manually
timer.record(Duration.ofMillis(250));

// Or wrap a callable
String result = timer.record(() -> {
    return processRequest();
});

// Or use a sample
Timer.Sample sample = Timer.start(registry);
// ... do work ...
sample.stop(timer);

Distribution Summary​

DistributionSummary summary = registry.summary("order.value", "currency", "USD");
summary.record(99.99);
summary.record(149.50);

Tags (Labels)​

// Add common tags to all metrics
registry.config().commonTags("env", "production", "service", "order-service");

// Add tags per metric
Counter counter = Counter.builder("api.calls")
        .tag("method", "GET")
        .tag("path", "/users")
        .tag("status", "200")
        .register(registry);

Configuration Options​

PropertyDescriptionDefault
applicationIdUnique identifier for your applicationdefault-app
applicationNameHuman-readable application nameDefaultApplication
applicationHostHostname of the applicationauto-detected
baseUrlTarget server URLhttp://localhost:8081
usernameAuthentication usernameadmin
passwordAuthentication passwordadmin
targetPathServer endpoint pathmetrics
stepIntervalHow often metrics are published10 seconds
httpClientTimeoutHTTP request timeout3 seconds
minBatchSizeMinimum batch size before sending1 MB
maxBatchSizeMaximum batch size per request10 MB
maxSendIntervalMaximum time between sends2 seconds
maxInMemorySizeMax memory buffer size10 MB
maxOnDiskSizeMax disk buffer size1 GB
retryCountNumber of retry attempts3
retryIntervalMillisDelay between retries1000 ms
transformationsSQL transformations for metrics[]
partitionByPartition columns for storage[]
enabledEnable/disable forwardingtrue

Metrics Schema​

Metrics are stored with the following Arrow schema:

ColumnTypeDescription
s_noLongSequence number
nameStringMetric name
typeStringMetric type (counter, gauge, timer, etc.)
application_idStringApplication identifier
application_nameStringApplication name
application_hostStringHost where metric originated
tagsMap<String, String>Metric tags/labels
valueDoublePrimary metric value
minDoubleMinimum value (for distributions)
maxDoubleMaximum value (for distributions)
meanDoubleMean value (for distributions)

Spring Boot Integration​

import io.dazzleduck.sql.micrometer.MicrometerForwarder;
import io.dazzleduck.sql.micrometer.config.MicrometerForwarderConfig;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.time.Duration;

@Configuration
public class MetricsConfig {

    @Bean
    public MicrometerForwarder micrometerForwarder() {
        MicrometerForwarderConfig config = MicrometerForwarderConfig.builder()
                .baseUrl("http://metrics-server:8081")
                .username("admin")
                .password("admin")
                .targetPath("metrics")
                .applicationId("spring-app")
                .applicationName("Spring Boot Application")
                .stepInterval(Duration.ofSeconds(15))
                .enabled(true)
                .build();

        return MicrometerForwarder.createAndStart(config);
    }

    @Bean
    public MeterRegistry meterRegistry(MicrometerForwarder forwarder) {
        return forwarder.getRegistry();
    }
}

Lifecycle Management​

MicrometerForwarder forwarder = MicrometerForwarder.createAndStart(config);

// Check status
boolean running = forwarder.isRunning();
boolean enabled = forwarder.isEnabled();

// Graceful shutdown - flushes pending metrics
forwarder.close();

Disabling Metrics​

Via code:

MicrometerForwarderConfig config = MicrometerForwarderConfig.builder()
        .enabled(false)
        .build();

Or via application.conf:

dazzleduck_micrometer {
  enabled = false
}

Troubleshooting​

Metrics not appearing on server​

  1. Check that the server URL is correct and accessible
  2. Verify authentication credentials
  3. Ensure enabled = true in application.conf
  4. Verify the forwarder is started before recording metrics

High memory usage​

Reduce buffer sizes:

.maxInMemorySize(5 * 1024 * 1024)  // 5 MB
.maxBatchSize(5 * 1024 * 1024)     // 5 MB

Slow metric publishing​

Adjust timing parameters:

.stepInterval(Duration.ofSeconds(30))
.maxSendInterval(Duration.ofSeconds(5))

License​

Apache License 2.0