Reviewing files that changed from the base of the PR and between a57e957 and a8a3864.

• charts/operator-wandb/Chart.yaml (1 hunks)
• charts/operator-wandb/charts/otel/charts/agent/templates/deamonset.yaml (2 hunks)

• charts/operator-wandb/Chart.yaml

charts/operator-wandb/charts/otel/charts/agent/templates/deamonset.yaml (2)

39-42: Security enhancement looks good but needs verification.

The addition of security context with runAsNonRoot and default seccomp profile is a great security improvement. However, please verify that the OpenTelemetry agent can function correctly without root privileges, especially for its monitoring capabilities.

✅ Verification successful

🌐 Web query:

Can OpenTelemetry agent run as non-root user?

💡 Result:

Yes, OpenTelemetry agents can generally run as non-root users, but configuration adjustments may be required depending on the deployment environment and specific components used. Here are key considerations:

Core Capabilities

• Non-root operation is explicitly recommended by OpenTelemetry security guidelines[4][5] and supported by major distributions like Splunk[3][5].
• Linux capabilities like cap_dac_read_search (bypass file/directory permission checks) and cap_sys_ptrace (process tracing) allow operation without full root privileges[3].

Implementation Requirements

Environment	Requirements
Linux/macOS	Use setcap to grant minimal capabilities[3]: setcap CAP_SYS_PTRACE,CAP_DAC_READ_SEARCH=+eip /path/to/otelcol
Docker/K8s	- Avoid --privileged flag - Set user in Dockerfile (USER 1001) - Use securityContext in Kubernetes[6][8]
Host Metrics	Some metrics collection may require elevated privileges[1]

Security Best Practices

1. Never run as root/admin - Explicitly prohibited in production environments[4][5]
2. Use dedicated service accounts with least-privilege access[5][8]
3. For auto-instrumentation requiring elevated privileges (e.g., Go), isolate using:

   securityContext:
     privileged: true
     runAsUser: 0  # Only when absolutely necessary[8]

Configuration Examples

Docker non-root setup[9]:

FROM otel/opentelemetry-collector-contrib:0.118.0
USER 1001:1001

Kubernetes Security Context[8]:

securityContext:
  capabilities:
    add: ["CAP_DAC_READ_SEARCH", "CAP_SYS_PTRACE"]
  runAsUser: 1001
  allowPrivilegeEscalation: false

While most components work without root access, verify receiver/exporter requirements and test functionality under reduced privileges[3][5]. Proper capability management allows secure operation while maintaining observability pipeline integrity.

Citations:

• 1: https://help.sumologic.com/docs/send-data/opentelemetry-collector/faq/
• 2: https://openobserve.ai/resources/automatic-instrumentation-opentelemetry-java/
• 3: https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-linux/install-linux-manual.html
• 4: https://opentelemetry.io/docs/specs/opamp/
• 5: https://docs.splunk.com/observability/en/gdi/opentelemetry/security.html
• 6: https://docs.datadoghq.com/security/default_rules/cis-docker-1.2.0-4.1/
• 7: https://opentelemetry.io/docs/zero-code/python/configuration/
• 8: https://github.com/open-telemetry/opentelemetry-operator/blob/main/README.md?plain=1
• 9: https://opentelemetry.io/docs/security/config-best-practices/
• 10: https://opentelemetry.io/docs/zero-code/java/agent/configuration/

---

Security Context Verified – Non-Root Operation is Supported

The web query confirms that OpenTelemetry agents can operate as non-root users. The security context addition using runAsNonRoot: true and the default seccomp profile is aligned with best practices for safer container deployments. Please ensure that any specific capability requirements are met if your setup collects host metrics or requires additional Linux capabilities.

---

115-117: Verify metrics collection with new volume mounts.

The switch to mounting /var/lib/containers suggests a move to support CRI-O/containerd instead of Docker. Please verify:

1. The OpenTelemetry agent can still collect all required container metrics
2. The read-only mount is sufficient for metrics collection
3. This change supports your container runtime (Docker/containerd/CRI-O)

✅ Verification successful

🌐 Web query:

What container log paths does OpenTelemetry agent support for different container runtimes (Docker, containerd, CRI-O)?

💡 Result:

The OpenTelemetry Collector supports different log paths depending on the container runtime and environment configuration:

Container Runtime Log Paths

Docker (Standalone)

• Default path: /var/lib/docker/containers/<container_id>/<container_id>-json.log
  Supports JSON-formatted Docker logs through Filelog receiver configuration[6][10]

Kubernetes Environments (All Runtimes)

• Unified collection path: /var/log/pods/*/*/*.log
  Captures logs from Docker/containerd/CRI-O through Kubernetes pod logging architecture[1][9]

CRI-O Specific

• Uses CRI logging format stored in Kubernetes pod paths
  Example log line format:
  2024-04-13T07:59:37.505201169-05:00 stdout F This is a cri-o log line![1][4]

Containerd

• Follows CRI format similar to CRI-O
  Example path in non-Kubernetes environments:
  ~/Library/Containers/com.docker.docker/Data/log/vm/containerd.log (macOS)[10]

Configuration Examples

For Kubernetes environments:

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    operators:
      - type: container

For standalone Docker:

receivers:
  filelog:
    include: [/var/lib/docker/containers/*/*.log]

The new container parser operator automatically detects formats (Docker JSON vs CRI) and handles partial log recombination[1][9]. Metadata extraction from file paths includes Kubernetes attributes like pod name, namespace, and container restart count when using Kubernetes paths[1].

For system-level runtime logging (dockerd/containerd daemons), OpenTelemetry can collect from:

• /var/log/dockerd.log (Linux)
• %LOCALAPPDATA%\Docker\log\vm\dockerd.log (Windows)[10]

Citations:

• 1: https://opentelemetry.io/blog/2024/otel-collector-container-log-parser/
• 2: https://coralogix.com/docs/opentelemetry/configuration-options/opentelemetry-using-docker/
• 3: https://coralogix.com/guides/opentelemetry/opentelemetry-logging-practical-guide/
• 4: https://kubernetes.io/blog/2022/12/01/runtime-observability-opentelemetry/
• 5: https://docs.honeycomb.io/send-data/logs/opentelemetry/collector/
• 6: https://lantern.splunk.com/Data_Descriptors/Docker/Getting_Docker_log_data_into_Splunk_Cloud_Platform_with_OpenTelemetry
• 7: https://opentelemetry.io/docs/collector/configuration/
• 8: https://next.redhat.com/2021/10/20/opentelemetry-tracing-in-kubernetes-core/
• 9: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/container.md
• 10: https://signoz.io/guides/docker-logs-location/

---

Ensure OpenTelemetry Agent Collects Logs Correctly with the New Mount

Based on the web query, the OpenTelemetry Collector supports different log paths depending on the container runtime. The new mount (/var/lib/containers) is consistent with setups using containerd or CRI-O—runtimes that typically use different logging locations than Docker. In this configuration:

• The read-only mount is generally sufficient, as the agent only needs to read the log files.
• If your environment relies on containerd or CRI-O, this change should maintain proper metrics collection.
• However, if using Docker, note that Docker’s default log location is usually /var/lib/docker/containers, so you may need additional configuration adjustments for docker-based log collection.

Please verify that your OpenTelemetry agent configuration (e.g., filelog receiver paths) aligns with the intended log location for your specific runtime.

Reviewing files that changed from the base of the PR and between a8a3864 and 069ef0d.

• charts/operator-wandb/Chart.yaml (1 hunks)
• charts/operator-wandb/charts/otel/charts/agent/templates/deamonset.yaml (2 hunks)

• charts/operator-wandb/Chart.yaml

• GitHub Check: Test Chart (v1.30.8, default)
• GitHub Check: Test Chart (v1.31.4, default)
• GitHub Check: Test Chart (v1.32.1, default)

charts/operator-wandb/charts/otel/charts/agent/templates/deamonset.yaml (2)

39-42: Great security enhancement!

Adding pod-level security context with runAsNonRoot and RuntimeDefault seccomp profile is a solid security improvement that aligns with Kubernetes best practices.

---

115-117:

❓ Verification inconclusive

Verify OpenTelemetry collector's access to container logs.

While the change to use /var/lib/containers is appropriate, please verify that the OpenTelemetry collector can still access all required container logs with this path when using emptyDir volumes.

---

🌐 Web query:

What paths does OpenTelemetry collector use to access container logs in Kubernetes?

💡 Result:

The OpenTelemetry Collector primarily accesses Kubernetes container logs through these paths:

Standard Kubernetes Log Paths

• Primary path:
  /var/log/pods/*/*/*.log
  This captures logs from all pods and containers in the cluster[1][4][5][7][8]

• Docker container path:
  /var/lib/docker/containers/*/*-json.log
  Used when working with Docker runtime[8]

Key Configuration Details

• Path mounting requires these hostPath volumes in the Collector DaemonSet:

  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods
    - name: varlibdockercontainers 
      hostPath:
        path: /var/lib/docker/containers

  These are mounted to the Collector container[8]

• Exclusion example (for Collector's own logs):

  exclude:
    - /var/log/pods/*/otel-collector/*.log
  ```[8]
  

Container Runtime Considerations

The Collector handles different log formats through operators that automatically detect:

• CRI-O format
• Containerd format
• Docker JSON format[7][9]

This path structure works across container runtimes (Docker, containerd, CRI-O) due to Kubernetes' standardized logging pattern where:

1. Each pod gets a directory under /var/log/pods
2. Each container restart creates a new log file with incremental numbering[1][4][8]

Citations:

• 1: https://last9.io/blog/opentelemetry-filelog-receiver-kubernetes-log-collection/
• 2: https://opentelemetry.io/docs/kubernetes/getting-started/
• 3: Improve Kubernetes Logs Collection Experience open-telemetry/opentelemetry-collector-contrib#25251
• 4: https://docs.honeycomb.io/send-data/kubernetes/send-logs-filelog-receiver/
• 5: https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/collector/opentelemetry-collector-kubernetes-stdout/
• 6: https://docs.splunk.com/observability/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.html
• 7: https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/otel-collector/
• 8: https://opentelemetry.io/docs/platforms/kubernetes/collector/components/
• 9: https://opentelemetry.io/blog/2024/otel-collector-container-log-parser/
• 10: https://opentelemetry.io/docs/platforms/kubernetes/collector/

---

Confirm OpenTelemetry Collector Log Access

The updated configuration mounts /var/lib/containers as readOnly, replacing the previous hostPath mounts (hostfs and /var/lib/docker/containers). Since the OpenTelemetry Collector typically accesses logs via /var/log/pods and /var/lib/docker/containers, please ensure that the collector’s configuration is updated accordingly or that your environment (e.g., using containerd/CRI-O) supports log access via /var/lib/containers when emptyDir volumes are used.

• Verify the collector configuration matches the new volume mount path.
• Ensure that the new mount does not hinder access to Docker container logs, should they be required.