Observability

The Observability module (module id collector, v1.0.0) runs three passive asyncio UDP listeners - SNMP traps, Syslog, and NetFlow - inside the api container and persists everything to TimescaleDB (the logdb). Nothing polls your devices. Devices push telemetry to FreeSDN.

The REST API mounts at /api/v1/collector/. The UI lives at Observability and Observability → Log Explorer in the sidebar.

Both receivers and source IPs are off by default

Every receiver starts disabled. An empty allowed_source_ips list means every inbound packet is rejected - no exceptions. You must enable each receiver you want and add at least one CIDR to allowed_source_ips before any data is collected. An empty allow-list is the secure default, not a misconfiguration. If your dashboards are empty, this is the first thing to check.

What each receiver collects

Protocol	Standard	Default port	What gets stored
SNMP traps	SNMPv1 + v2c	UDP 162	Source IP, enterprise OID, trap type (generic / v2c), varbinds (list of OID + value); community string is embedded in the message field only, not stored as a separate column; severity is not populated for SNMP traps (syslog only)
Syslog	RFC 3164 + RFC 5424	UDP 514	Facility, severity, hostname, app name, message, raw line (capped at 2 000 chars)
NetFlow	v5 + v9	UDP 2055	5-tuple (src IP / dst IP / src port / dst port / protocol), bytes in/out, packets, 1-minute bucket time, DPI app name + category

All records carry the organization_id of your instance (multi-tenant safe). Incoming packets whose source IP resolves to a known managed device are linked to that device_id automatically (60-second TTL cache). Packets from unrecognized IPs are silently dropped - they never enter the database. This is tracked by an internal _dropped_unknown_source counter on each receiver that is not exposed in the status API. To see whether packets are being dropped due to unknown sources, add the device to FreeSDN under its correct IP and watch for data to appear.

v9 is a simplified implementation

The NetFlow v9 parser handles per-source template caches and a common subset of field types (source/dest address, ports, protocol, bytes, packets). IPFIX (v10) is not supported. If you export IPFIX, configure your exporters to fall back to v9.

Permissions

Code	Required for
collector.logs.read	Search and view SNMP and syslog log entries; used by the Log Explorer
collector.flows.read	View NetFlow records and traffic analytics (top talkers, protocol breakdown)
collector.config	Enable/disable receivers, set ports, update retention and source allow-list

viewer and above can hold read permissions. Config changes require the collector.config permission. org_admin and super_admin users have it implicitly; lower roles (including site_admin) must be granted it explicitly via module settings.

Enabling a receiver

All configuration is per-organization. Changes are applied live via hot-reload - no container restart required.

1. Add source CIDR ranges

Before you enable any receiver, decide which IP ranges send telemetry to FreeSDN. You will set allowed_source_ips to a list of CIDRs in the config body below. An empty list blocks everything.

2. PUT your config

Requires collector.config permission.

PUT /api/v1/collector/config
Content-Type: application/json

{
  "snmp_enabled": true,
  "snmp_port": 162,
  "snmp_community": "public",
  "syslog_enabled": true,
  "syslog_port": 514,
  "netflow_enabled": true,
  "netflow_port": 2055,
  "log_retention_days": 30,
  "flow_retention_days": 7,
  "allowed_source_ips": ["10.0.0.0/8", "192.168.1.0/24"]
}

All fields are optional in the request body - omit any you do not want to change. Allowed mutable fields and their defaults:

Field	Type	Default	Notes
snmp_enabled	bool	false	Enable SNMP trap listener
snmp_port	int	162	Bind port
snmp_community	string	"public"	Accepted community string for v1/v2c
syslog_enabled	bool	false	Enable syslog listener
syslog_port	int	514	Bind port
netflow_enabled	bool	false	Enable NetFlow listener
netflow_port	int	2055	Bind port
log_retention_days	int	30	Days to retain SNMP + syslog records (not validated server-side; use a positive integer)
flow_retention_days	int	7	Days to retain NetFlow records (not validated server-side; use a positive integer)
allowed_source_ips	list of CIDRs	[]	Empty = block all. List every range that sends telemetry

3. Verify listeners are running

GET /api/v1/collector/status

The response reports each service: running, port, rejected (allowlist drops), and for NetFlow, dropped (queue overflow).

4. (Optional) Read back the current config

GET /api/v1/collector/config

Returns the stored config, or defaults if none has been saved yet.

UDP port forwarding

The UDP listeners bind inside the api container. Caddy handles TLS/TCP only - it does not proxy UDP. You must expose the collector ports directly on the api service in your Compose file.

services:
  api:
    ports:
      - "162:162/udp"
      - "514:514/udp"
      - "2055:2055/udp"

If you run behind a Docker NAT or Docker Desktop, add the same entries to your docker-compose.override.yml.

Ports below 1024 require elevated privilege

UDP port 162 (SNMP traps) and 514 (syslog) are privileged ports on Linux. Binding them fails silently - the listener logs a warning and the other listeners still start. To fix this, either:

• Add CAP_NET_BIND_SERVICE to the api container (cap_add: [NET_BIND_SERVICE]).
• Change snmp_port and/or syslog_port to a value above 1024 (e.g. 10162 / 10514) and reconfigure your devices to send to the new port.
• Configure your host-level firewall to redirect UDP 162/514 to the higher port before it reaches the container.

Port 2055 (NetFlow) does not have this problem.

Host rsyslog conflict

If you run FreeSDN directly on a Linux host (not in a container), the host’s rsyslog may already hold UDP 514. Either change syslog_port in the config or stop the host rsyslog service.

How source-IP filtering and multi-tenancy work

When a UDP packet arrives, the module applies two checks before any parsing:

1. Allowlist check - the source IP must fall within one of the CIDRs in allowed_source_ips. Packets outside the list are counted as rejected and discarded.
2. Tenant resolution - the source IP is matched against Device.ip_address in the database (60-second cache, up to 4 096 entries). If no managed device matches, the packet is counted as _dropped_unknown_source and discarded. This means only packets from devices you have already added to FreeSDN are persisted.

Both checks must pass. A packet from a known CIDR but an unknown device address is still dropped. Add the device to FreeSDN under its correct IP before expecting its telemetry to appear.

Retention

Data type	Default	Setting key	Maximum
SNMP + syslog entries	30 days	log_retention_days	No enforced limit
NetFlow flow records	7 days	flow_retention_days	No enforced limit

Flow volume can be very large

High-traffic networks generate thousands of NetFlow records per minute. The 7-day default is conservative for storage reasons. Increasing flow_retention_days significantly on a busy network requires extra disk capacity in the logdb volume. Monitor logdb disk usage before extending retention.

Pruning runs as part of the Celery scheduler service. TimescaleDB handles chunk compression and automatic retention policies to reduce on-disk size for high-volume deployments.

Log Explorer

Observability → Log Explorer (GET /api/v1/collector/logs) gives you a filterable, paginated view of collected SNMP trap and syslog entries. The detail drawer shows the full raw_data, enterprise_oid, trap_type, and varbinds for SNMP entries.

Query parameters:

Parameter	Values	Description
source_type	snmp_trap, syslog	Filter to one receiver type
severity	emergency alert critical error warning notice informational debug	Syslog severity level
device_id	UUID	Filter to a specific managed device
start_time / end_time	ISO-8601	Time window (defaults to the last 1 hour)
q	string	Full-text search in the message field (SQL ILIKE)
page	int ≥ 1	Pagination page
size	1-500	Results per page

Example - find critical syslog messages in the last hour:

GET /api/v1/collector/logs?source_type=syslog&severity=critical&start_time=2026-06-06T00:00:00Z&end_time=2026-06-06T01:00:00Z

Aggregate statistics

Returns counts broken down by severity, source type, and top sending IPs:

GET /api/v1/collector/logs/stats?hours=24

hours accepts 1-168 (1 week). Default is 24.

Full log detail

GET /api/v1/collector/logs/{log_id}

Returns the complete record including raw_data (the original unparsed line, capped at 2 000 chars) and all decoded fields.

NetFlow analytics

Flow search

GET /api/v1/collector/flows

Accepted parameters: device_id, source_ip, dest_ip, protocol (IP protocol number as integer), start_time, end_time, page, size (1-500).

Top talkers

GET /api/v1/collector/flows/top-talkers?hours=1&limit=10&sort_by=bytes&site_id=<uuid>

Returns the top-N source IPs ranked by bytes or packet count. Parameters:

Parameter	Values	Default
hours	1-168	1
limit	1-50	10
sort_by	bytes, packets	bytes
site_id	UUID (optional)	Aggregates across whole org if omitted

site_id is validated against your site grants - a site you cannot access returns 403.

Protocol breakdown

GET /api/v1/collector/flows/protocol-breakdown?hours=24

Returns traffic volume grouped by IP protocol number with human-readable names for common protocols (TCP, UDP, ICMP, GRE, ESP, OSPF). hours accepts 1-168.

DPI application classification

Every NetFlow record is classified before storage using a built-in O(1) lookup table of 60+ rules (protocol + destination port → application name + category). The classifier covers common infrastructure protocols (DNS, SNMP, NTP, BGP, RADIUS, LDAP, Kerberos), network services (HTTP, HTTPS, FTP, SSH, SMB), databases (MySQL, PostgreSQL, Redis), VPNs (OpenVPN, WireGuard), VoIP (SIP, RTP, RTSP), and popular applications (Steam, Xbox, Zoom, MQTT, CoAP, among others).

Application categories: web, streaming, conferencing, email, file_transfer, vpn_tunnel, dns, database, gaming, social, infrastructure, security, iot, voip, other.

You can add custom classification rules via the database (ApplicationClassificationRule table in the collector schema). Custom rules by priority and can override built-in entries. Port ranges are supported up to 1 000 ports per rule.

Classified app_name and app_category are stored on each FlowRecord row but are not returned by the flow-search (GET /collector/flows), top-talkers, or protocol-breakdown endpoints. DPI analytics are exposed through a dedicated set of endpoints under /api/v1/dpi/ (summary, app-breakdown, app-trends, per-client usage) that group and aggregate the stored fields directly.

AI Assistant integration

The AI Assistant can query Observability data directly when both modules are enabled. Two tools are available in the chat:

Tool	Required permission
search_collector_logs	collector.logs.read
get_top_talkers	collector.flows.read

The AI Assistant is globally off by default and requires a separate opt-in. See AI Assistant for setup.

API endpoint reference

All endpoints require authentication. Org isolation is enforced in the service layer - you only see your organization’s data. Source status and config endpoints require collector.config; log and flow queries require their respective read permission.

Method	Path	Purpose
GET	/api/v1/collector/config	Read current per-org collector config (returns defaults if none saved)
PUT	/api/v1/collector/config	Update config + hot-reload listeners
GET	/api/v1/collector/status	Per-service running status, port, rejected + dropped counts
GET	/api/v1/collector/logs	Search/filter log entries (SNMP + syslog)
GET	/api/v1/collector/logs/stats	Aggregate counts by severity / source type / IP
GET	/api/v1/collector/logs/{log_id}	Full log detail including raw data
GET	/api/v1/collector/flows	Search/filter NetFlow records
GET	/api/v1/collector/flows/top-talkers	Top-N source IPs by bytes or packets
GET	/api/v1/collector/flows/protocol-breakdown	Traffic by IP protocol, named

Receiver technical details

SNMP trap receiver

The receiver is pure-Python BER decoding with no dependency on net-snmp. It handles:

• SNMPv1 trap PDU (0xA4) - skips agent-address, generic-trap, specific-trap, timestamp fields, then reads varbinds.
• SNMPv2c trap PDU (0xA7) - skips request-id, error-status, error-index, then reads varbinds.

BER tag types decoded: INTEGER, OCTET_STRING, NULL, OID, SEQUENCE. Each trap is stored with the enterprise OID, trap type (generic or v2c), and varbinds as JSONB. The community string is embedded in the message field rather than stored as a separate column.

Syslog receiver

The parser tries RFC 5424 first, then RFC 3164. Lines that match neither format are stored verbatim as the message field. Facility and severity are decoded from the PRI byte (facility = pri >> 3, severity = pri & 0x07). The raw line is capped at 2 000 characters.

NetFlow receiver

• v5: fixed 48-byte records per flow, maximum 30 records per packet.
• v9: template-based. Each source IP has its own template cache (up to 256 templates) with LRU eviction. A maximum of 1 024 distinct source IPs are tracked simultaneously, preventing one chatty exporter from evicting another source’s templates. Flows buffer in memory (up to 100 000 pending), then flush to TimescaleDB every 60 seconds. Queue overflow increments the dropped counter visible in GET /api/v1/collector/status. A final flush runs on shutdown.

Common problems

Symptom	Likely cause	Fix
No data in Log Explorer or flow analytics	Receiver not enabled OR allowed_source_ips is empty	Enable the receiver in config; add your network CIDRs to allowed_source_ips
Status shows listener not running on port 162 or 514	Port < 1024 privilege failure	Add CAP_NET_BIND_SERVICE to the api container or move to a port > 1024
Data arrives (source IP passes allowlist - rejected count stays flat) but nothing is stored	Source device not added to FreeSDN under its correct IP - the tenant resolver cannot map the source IP to a device_id	Add the device to FreeSDN under its correct IP address
rejected count rises in status	Source IP not in allowed_source_ips CIDR	Add the CIDR to the allow-list in config
NetFlow records stop after exporter restart	v9 templates not re-learned	Normal - the exporter must re-send templates; can take up to 60 seconds
syslog_port: 514 fails on Linux host (non-container)	Host rsyslog holds the port	Change syslog_port to 10514 (or any port > 1024) and reconfigure senders

Next steps

• Deployment tiers and Compose profiles - understand the logdb (TimescaleDB) service that backs Observability storage and how to size it per tier.
• AI Assistant - enable the chat interface to query logs and top-talkers conversationally.
• Automation connections - wire collector data into automation rules (e.g. trigger an alert when log volume from a host spikes).
• API overview - full OpenAPI spec location and authentication; enable ENABLE_DOCS=true in a non-production environment to browse Swagger.