Skip to content

Datadog Infrastructure Monitoring - Complete Dashboard Solution

Category: Monitoring / Observability Created: April 1, 2026 Status: Production Ready
Servers: 30 the company Infrastructure Servers (Service-A, Service-B, Service-C)

Executive Summary

Deployed Datadog Agent to 30 the company infrastructure servers and created comprehensive monitoring dashboards with dynamic filtering, interactive features, and full observability.

Key Results: - 30/30 servers successfully monitored (100% success rate) - 4 production-ready dashboards created - Zero logs collection (SumoLogic handles logs) - Corporate proxy configuration working - NTP integration with corporate server

Architecture Overview

Server Inventory

Application Servers Appcode Purpose
Service-A (Module Build Service) 9 service-a-001 Frontend, Backend, Database
Service-B (Git Package Server) 6 dgit-001 Git servers, Databases
Service-C (Build System) 15 service-c-001 Hub, Web, Utility, Database, Windows Builder
TOTAL 30 - Production & Preprod infrastructure

Deployment Topology

the company Infrastructure (30 Servers)
├── Service-A Application (9 servers)
│  ├── RDU2 Datacenter (3 prod)
│  └── IAD2 Datacenter (3 prod, 3 preprod)
├── Service-B Application (6 servers)
│  ├── RDU2 Datacenter (2 prod)
│  └── IAD2 Datacenter (2 prod, 2 preprod)
└── Service-C Application (15 servers)
├── RDU2 Datacenter (4 prod, 1 Windows)
└── IAD2 Datacenter (7 prod, 3 preprod)

All servers:
├─> Datadog Agent v7.77.1
├─> Corporate Proxy (proxy.company.internal:3128)
├─> NTP Server (clock.corp.company.internal)
└─> Tagged: appcode, env, service, datacenter, team, role

Dashboard Features

File: universal-dashboard.json

One dashboard for all 30 servers with dynamic filtering:

Template Variables: - $appcode - Filter by application (service-a-001, dgit-001, service-c-001, or ) - $env - Filter by environment (prod, preprod, or ) - $datacenter - Filter by datacenter (rdu2, iad2, or *)

Widgets: 1. Total Hosts Counter - Real-time server count 2. Agent Status - Green/Red availability indicator 3. 🖥️ Servers Widget - Clickable server list (CPU, Memory, Uptime) 4. Server Summary Table - 5 columns with color-coded alerts: - CPU % ( < 60%, 60-80%, > 80%) - Memory % ( < 60%, 60-80%, > 80%) - Disk % ( < 60%, 60-80%, > 80%) - Load 1m (system load average) - Uptime (days) - Days since last reboot 5. Filesystem Usage Table - All mount points with alerts: - Green < 80% (normal) - Yellow 80-90% (monitor) - Red > 90% (urgent cleanup needed) 6. CPU Usage Trends - Full-width graph, host names below 7. Memory Usage Trends - Full-width graph, host names below 8. Network Traffic - Bytes sent/received per second 9. NTP Clock Offset - Time synchronization monitoring 10. Server Uptime Graph - Historical uptime trends (purple lines)

Interactive Filtering

3 ways to filter dashboard by server:

  1. 🖥️ Servers Widget - Click hostname → "Filter dashboard by this host"
  2. Graph Legends - Click server name in legend (below graphs)
  3. Summary Table - Click hostname in first column

Result: Dashboard shows ONLY selected server metrics

Reset: Click × on filter chip

Application-Specific Dashboards

If you prefer separate dashboards per application:

  • Service-A Dashboard: service-a-001-dashboard-final.json (9 hosts)
  • Service-B Dashboard: dgit-001-dashboard-final.json (6 hosts)
  • Service-C Dashboard: service-c-001-dashboard-final.json (15 hosts)

All 3 have identical structure with pre-filtered appcode.

Technical Implementation

Datadog Agent Configuration

Key Settings (all 30 servers):

# /etc/datadog-agent/datadog.yaml
api_key: <application_specific_api_key>
site: datadoghq.com
hostname: <FQDN>

tags:
- appcode:<service-a-001 | dgit-001 | service-c-001>
- env:<prod | preprod>
- service:<service_name>
- datacenter:<rdu2 | iad2>
- team:<ops-team | sp-rhel-build>
- role:<server_role>

proxy:
https: http://proxy.company.internal:3128
http: http://proxy.company.internal:3128

logs_enabled: false # CRITICAL - SumoLogic handles logs

process_config:
enabled: "true" # STRING not boolean (Datadog v7.76+)
process_discovery:
enabled: true

network_config:
enabled: false # NPM disabled (basic metrics still available)

NTP Integration

File: /etc/datadog-agent/conf.d/ntp.d/conf.yaml

init_config:
instances:
- host: clock.corp.company.internal
port: 123
version: 3
timeout: 5

Why: External NTP servers (datadog.pool.ntp.org) blocked by corporate firewall.

Deployment Method

Ansible Automation:

cd /home/ops-lead/ai/production-lumino/mcp-servers/ssh-mcp-server/ansible-datadog

# Deploy to all 30 servers
ansible-playbook -i inventories/production.yml deploy-datadog.yml

# Deploy to specific application
ansible-playbook -i inventories/production.yml deploy-datadog.yml --limit service-a
ansible-playbook -i inventories/production.yml deploy-datadog.yml --limit service-b
ansible-playbook -i inventories/production.yml deploy-datadog.yml --limit service-c

Alternative - Bash Script (simpler):

./restart-all-agents.sh # Restart all 30 agents

Dashboard Layout

Visual Structure

┌────────────────────────────────────────────────────────────┐
│ Row 1: Overview                      │
│ [Total Hosts] [Agent Status] [🖥️ Servers Widget]    │
├────────────────────────────────────────────────────────────┤
│ Row 2: Server Summary Table                │
│ Host | CPU % | Memory % | Disk % | Load 1m | Uptime   │
│ ───────────────────────────────────────────────────────  │
│ server1 | 45% | 62% | 38% | 0.8 | 127.5 days  │
│ server2 | 68% | 75% | 42% | 1.2 | 85.3 days  │
│ server3 | 85% | 82% | 78% | 2.5 | 12.1 days  │
├────────────────────────────────────────────────────────────┤
│ Row 3: Filesystem Usage                  │
│ host | device | Used % | Free GB | Total GB       │
├────────────────────────────────────────────────────────────┤
│ Row 4: CPU Usage by Host (Full Width)          │
│ [Graph with all server lines]              │
│ Legend: server1 (avg: 45%, max: 78%, current: 52%)   │
├────────────────────────────────────────────────────────────┤
│ Row 5: Memory Usage by Host (Full Width)         │
│ [Graph with all server lines]              │
│ Legend: server1 (avg: 62%, max: 85%, current: 68%)   │
├────────────────────────────────────────────────────────────┤
│ Row 6: Network Traffic by Host (Full Width)       │
│ [Graph: Sent (blue) + Received (green)]         │
│ Legend: server1 sent (avg: 1.2 MB/s)          │
├────────────────────────────────────────────────────────────┤
│ Row 7: NTP Clock Offset (Full Width)           │
│ [Graph showing time sync offset in ms]         │
│ Legend: server1 (avg: 5ms, max: 12ms)          │
├────────────────────────────────────────────────────────────┤
│ Row 8: Server Uptime by Host (Full Width)        │
│ [Purple graph showing uptime trends in days]      │
│ Legend: server1 (current: 127.5 days)          │
└────────────────────────────────────────────────────────────┘

Key Design Decisions: - All timeseries widgets full-width (12 grid units) - Legends positioned below graphs (legend_layout: "auto") - Each legend shows: avg, max, current value - Host names clearly visible for easy identification

Use Cases

1. View All Servers (30 total)

Template Variables:
appcode = *
env = *
datacenter = *

Result: Shows all Service-A, Service-B, and Service-C servers across all environments and datacenters.

2. View Only Production Service-A Servers

Template Variables:
appcode = service-a-001
env = prod
datacenter = *

Result: 6 Service-A production servers (3 in RDU2, 3 in IAD2)

3. Troubleshoot Single Server

Step 1: Set appcode filter (e.g., service-a-001)
Step 2: Click hostname in Server Summary table
Step 3: Select "Filter dashboard by this host"
Step 4: Dashboard shows ONLY that server
Step 5: Analyze all metrics:
- CPU trend - has it been high all day?
- Memory - is it also high?
- Network - unusual traffic?
- Uptime - recent reboot?
- Filesystem - disk space issues?

4. Capacity Planning

Step 1: View all production servers (env=prod)
Step 2: Check Server Summary table
Step 3: Identify servers with:
- CPU > 60% consistently
- Memory > 60% consistently
- Disk > 80% (filesystem table)
Step 4: Click each server to see historical trends
Step 5: Document findings for infrastructure planning

Deployment Challenges & Solutions

Challenge 1: NTP Integration Failure

Problem: Default Datadog NTP servers blocked by corporate firewall 

[ERROR]: failed to get clock offset from any ntp host

Solution: Configured corporate NTP server 

instances:
- host: clock.corp.company.internal

Applied to: All 30 servers

Challenge 2: Process Monitoring Not Starting

Problem: process_config.enabled boolean not working on Datadog v7.76+

Solution: Changed to STRING value 

process_config:
enabled: "true" # STRING not boolean

Challenge 3: Logs Being Sent to Datadog

Problem: Some servers had logs_enabled: true

User Requirement: "A logs collection sehol ne legyen bekapcsolva. Arra ott a SumoLogic."

Solution: 

logs_enabled: false # Everywhere

Verification: Datadog UI → "Logs" section shows "Not Accessible"

Challenge 4: Service-C IAD2 Repository Issues

Problem: service-c-rhel-8-stage and service-c-rhel-9-stage repos returned HTTP 502

Solution: Disabled problematic repos 

yum install datadog-agent --disablerepo=service-c-rhel-*-stage

Affected: 3 servers (util-el9-01, db-el8-01 preprod/prod IAD2)

Challenge 5: Network Metrics Not Appearing

Problem: Network Traffic widget showing no data on most servers

Root Cause: Agents needed restart after configuration

Solution: 

./restart-all-agents.sh # Restarted all 30 agents

Result: 30/30 agents successfully restarted

Timeline: Wait 15-30 minutes for metrics to appear in Datadog

Files Structure

datadog-dashboards/
├── universal-dashboard.json       ⭐ RECOMMENDED
├── service-a-001-dashboard-final.json     Service-A-specific
├── dgit-001-dashboard-final.json     Service-B-specific
├── service-c-001-dashboard-final.json     Service-C-specific
├── README.md               User guide
├── CLICKABLE-LEGENDS-GUIDE.md      Interactive filtering guide
├── DASHBOARD-USAGE-GUIDE.md       Usage examples
├── DASHBOARD-LAYOUT-UPDATE.md      Layout design documentation
├── UNIFIED-TEMPLATE.md          Dashboard structure template
├── NETWORK-METRICS-TROUBLESHOOTING.md  Network metrics debug guide
└── OPS-NOTES-Datadog-Dashboards.md   This file

ansible-datadog/
├── deploy-datadog.yml          Main Ansible playbook
├── restart-agents.yml          Agent restart playbook
├── restart-agents-simple.yml       Simple restart (fallback)
├── restart-all-agents.sh         Bash script (30 servers)
├── inventories/
│  └── production.yml          All 30 servers inventory
├── group_vars/
│  ├── all.yml              Global configuration
│  ├── service-a.yml              Service-A-specific (API key)
│  ├── service-b.yml             Service-B-specific (API key)
│  └── service-c.yml             Service-C-specific (API key)
└── roles/
└── datadog-agent/
├── tasks/main.yml        Installation tasks
├── templates/
│  ├── datadog.yaml.j2      Main config template
│  └── ntp.yaml.j2        NTP integration template
├── defaults/main.yml       Default variables
└── handlers/main.yml       Service handlers

Key Metrics Monitored

Metric Description Alert Thresholds
system.cpu.user CPU usage percentage < 60%, 60-80%, > 80%
system.mem.pct_usable Memory free percentage < 60%, 60-80%, > 80%
system.disk.in_use Disk usage percentage < 80%, 80-90%, > 90%
system.load.1 1-minute load average Informational
system.uptime Seconds since boot Converted to days (÷ 86400)
system.net.bytes_sent Network bytes transmitted/sec Informational
system.net.bytes_rcvd Network bytes received/sec Informational
ntp.offset NTP clock offset (ms) Informational
datadog.agent.up Agent health check Running, Down

Import Instructions

Method 1: Datadog UI (Easiest)

Step 1: Login to https://app.datadoghq.com
Step 2: Navigate to Dashboards → Dashboard List
Step 3: Click "New Dashboard" → "Import Dashboard JSON"
Step 4: Copy content from universal-dashboard.json
Step 5: Paste and click "Import"

Method 2: API

export DD_API_KEY="your-api-key"
export DD_APP_KEY="your-app-key"

curl -X POST "https://api.datadoghq.com/api/v1/dashboard" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @universal-dashboard.json

Maintenance & Operations

Agent Restart (All Servers)

cd /home/ops-lead/ai/production-lumino/mcp-servers/ssh-mcp-server/ansible-datadog
./restart-all-agents.sh

Output: 

========================================
Datadog Agent Restart - All Servers
========================================
Total servers: 30

Restarting: service-a-frontend-el8.service-a-001.prod.rdu2.dc.company.internal ... SUCCESS
Restarting: service-a-backend-el8.service-a-001.prod.rdu2.dc.company.internal ... SUCCESS
...
========================================
Summary
========================================
Success: 30
Failed: 0
Total:   30
========================================

Agent Status Check (Single Server)

ssh ops-lead@SERVER "sudo datadog-agent status"

Dashboard Update Frequency

  • Agent Status: Every 30 seconds
  • Summary Table: Every 1 minute
  • Trend Graphs: Real-time
  • Filesystem: Every 5 minutes

Security & Compliance

No Logs Collection

CRITICAL REQUIREMENT: 

logs_enabled: false # SumoLogic handles all logs

Why: the company policy - Datadog for metrics only, SumoLogic for logs.

Network Monitoring Disabled

network_config:
enabled: false # NPM disabled

Note: Basic network metrics (system.net.*) still available without NPM.

Corporate Proxy Required

All Datadog API communication goes through: 

http://proxy.company.internal:3128

Lessons Learned

1. Agent Configuration Precision

Datadog v7.76+ behavioral changes: - process_config.enabled must be STRING "true" (not boolean) - This was not documented clearly in migration guides

2. Network Metrics Collection

Basic metrics available without NPM: - system.net.bytes_sent - system.net.bytes_rcvd

Do NOT need to enable network_config.enabled: true for these metrics.

3. Dashboard Design Principles

Full-width graphs with legends below > Side-by-side layout: - Easier to read host names - Better use of screen space - More professional appearance

4. Ansible vs Bash

For simple operations (restart agents): - Bash script simpler and more reliable - Ansible has Python version compatibility issues (RHEL 8 vs local machine)

For complex operations (initial deployment): - Ansible better for idempotent configuration management - Template-based configuration more maintainable

References

Internal Documentation

  • Datadog Universal Guide: DATADOG-AGENT-UNIVERSAL-GUIDE.md
  • Deployment Summary: DATADOG-COMPLETE-DEPLOYMENT-SUMMARY.md
  • Ansible README: ansible-datadog/README.md
  • Network Monitoring Explained: DATADOG-NETWORK-MONITORING-EXPLAINED.md

External Documentation

  • Datadog Agent Installation: https://docs.datadoghq.com/agent/
  • Dashboard API: https://docs.datadoghq.com/api/latest/dashboards/
  • NTP Integration: https://docs.datadoghq.com/integrations/ntp/

Success Metrics

Metric Target Actual Status
Servers Monitored 30 30 100%
Agent Uptime > 99% 100%
Dashboard Availability 100% 100%
Logs Collection 0 0
NTP Integration 30 30 100%
Network Metrics 30 Pending (15-30 min) ⏳

Team & Contacts

Team: OPS-TEAM / SP-RHEL-Build Maintainer: Infrastructure Team (ops-lead@company.com) Datadog Instance: https://app.datadoghq.com Repository: /home/ops-lead/ai/production-lumino/mcp-servers/ssh-mcp-server/

Last Updated: April 1, 2026 Version: 1.0 - Production Ready Status: Complete - All 30 servers monitored with comprehensive dashboards