Skip to main content
View rawEdit

Alerting configuration

No bundled alerting (0.18.0)

PostgresAI monitoring 0.18.0 does not ship any alert rules or an Alertmanager service. The optional vmalert service (Docker Compose profile host-metrics) evaluates only recording rules (config/prometheus/host_rules.yml, the host_* series); it has no alert rules and sends no notifications. There are no pre-configured alerts (such as connection, replication, or bloat alerts), no Alertmanager API on port 9093, and no notification receivers bundled with the stack.

The metrics-storage backend is VictoriaMetrics (the sink-prometheus service), exposed on host port 59090 (container 9090). The bundled config/prometheus/prometheus.yml contains only a commented-out rule_files: placeholder (VictoriaMetrics single-node ignores rule_files) and no alerting: / alertmanager block:

rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

Everything on this page describes how you can add your own alerting on top of the stack. None of it is provided out of the box.

Adding alerting yourself​

Because the stack stores metrics in VictoriaMetrics, you have two common options to add alerting.

Option 1: Grafana alerting​

Grafana (the grafana service, container grafana-with-datasources, on host port 3000, default login monitor / demo) ships with its own alerting engine. You can define alert rules against the PGWatch-Prometheus datasource entirely inside Grafana, with no extra components.

  1. Open a panel and switch to the Alert tab, or use Alerting → Alert rules.
  2. Build a query against the PGWatch-Prometheus datasource using the real metric names exported by pgwatch (all prefixed pgwatch_, for example pgwatch_pg_stat_activity_count, pgwatch_db_stats_xact_commit, pgwatch_db_stats_numbackends, and pgwatch_settings_numeric_value{setting_name="max_connections"}). See the metrics reference for the exact series names.
  3. Configure a threshold, an evaluation interval, and a contact point.

Option 2: vmalert + Alertmanager (not bundled)​

VictoriaMetrics supports Prometheus-style alerting through the separate vmalert component plus a Prometheus Alertmanager. Neither is part of the PostgresAI compose stack, so you would run them yourself and point them at the sink-prometheus datasource (http://sink-prometheus:9090, which uses basic auth — VM_AUTH_USERNAME / VM_AUTH_PASSWORD).

If you go this route, write your own rules against the real pgwatch_* series. For example, a connection-saturation rule:

groups:
- name: postgresql_alerts
rules:
- alert: HighConnectionUsage
expr: |
sum(pgwatch_db_stats_numbackends) by (cluster, node_name)
/
scalar(max(pgwatch_settings_numeric_value{setting_name="max_connections"}))
> 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "Connection usage above 80%"
description: "{{ $labels.cluster }} has {{ $value | humanizePercentage }} connections used"
Verify metric and label names first

Use real series names and labels. pgwatch exports series as pgwatch_<metric-group>_<column>, and the cluster label is cluster (not cluster_name; cluster_name is only a Grafana template variable). Query http://localhost:59090/api/v1/query?query=... against VictoriaMetrics (with VM basic auth) to confirm a series exists before writing a rule against it.

Best practices for any rules you add​

Use the for duration wisely:

  • Too short — false positives from transient spikes.
  • Too long — delayed notification.

Recommended for values:

Alert typeDuration
Critical outages1m
Performance issues5m
Resource usage10m
Trend alerts30m