Integrating Golang gRPC Application Custom Statsd Metrics and Logs with Splunk Observability Cloud

[rasrivastava]

Hello Community,

I have gone through the repo for the statsdreceiver (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) but I was not able to configure it successfully.

receivers:
  statsd:
    endpoint: "localhost:8125" # default
    aggregation_interval: 60s  # default
    enable_metric_type: false   # default
    is_monotonic_counter: false # default
    timer_histogram_mapping:
      - statsd_type: "histogram"
        observer_type: "histogram"
        histogram:
          max_size: 50
      - statsd_type: "distribution"
        observer_type: "histogram"
        histogram: 
          max_size: 50    
      - statsd_type: "timing"
        observer_type: "summary"

exporters:
  file:
    path: ./test.json

service:
  pipelines:
    metrics:
     receivers: [statsd]
     exporters: [file]

I tried to configure above but it was not working, here I am not sure how Splunk Oberverability Cloud will know to listen to 8127 post how to configure custom gRPC metrics from a golang application running on port UDP:8125 (statsd)

Let me explain my use case in detail:

I have couple of EC2 Linux instance on which statsd server is running and it generating some custom gRPC metrics from a golang application running on port UDP:8125 (statsd).

Now, I want these custom gRPC metrics from a golang application running on port UDP:8125 (statsd) to send to Splunk Oberverability Cloud, so that I can monitor these custom gRPC metrics there, but this we need to make a connection between EC2 Linux instance and Splunk Oberverability Cloud, Splunk Oberverability Cloud should able to recieve these custom gRPC metrics as we don't have any hostname/IP address for Splunk Oberverability Cloud we have to use some agent for doing this, I think we can using "splunk-otel-collector.service"

Currently I am able to capture the predefined metrices such "^aws.ec2.cpu.utilization", system.filesystem.usage etc on my Splunk Oberverability Cloud but now I also want the custom gRPC metrics same like this.

Before this setup I using a setup in which I was having multiple EC2 Linux instance on which statsd server was running and I was a serepate Spunk Enterprise EC2 instance and it was collecting all the metrics there. But Spunk Enterprise provide commands to connect instances to Spunk Enterprise using "./splunk enable listen 9997" and "./splunk add <destination_hostname>:9997" and I was using below configuration to do so.

    "statsd": {
        "statsd_max_packetsize": 1400,
        "statsd_server" : "ip-172-32-1-187.us-west-2.compute.internal",
        "statsd_port"   : "8125"
    },

Same thing I want to achieve using Splunk Oberverability Cloud. Can you please explain in detail how we can connect EC2 instances with Splunk Oberverability Cloud to send custom gRPC metrics from a golang application running on port UDP:8125 (statsd), if using https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver is the only way then what changes I want to make in the configuration files related to the custom metric collection (has to be added any where in this directory), hostname, ports name mentioning in any files etc in details.

Thanks