On-call

Plans

Free has no on-call. Hobby ($4 a month) has 3 sources, alerts from them and from its monitors, acknowledging, and repeats until someone acknowledges, for one person. Pro ($15 a month, 3 people included and $5 a month for each one after) has 50 sources, escalation policies, schedules, each person's own notifications and the PagerDuty and Opsgenie import, for everyone on the team. On-call is under On-call in the dashboard, for every page or one.

Sources

A source is one integration key on a page, under the page's Alerts, Sources. Its alerts show on that page. Anything that sends to PagerDuty's Events API, Opsgenie's Alert API or Sentry's webhooks sends to a source by changing the URL. The key comes in two forms: 32 characters for PagerDuty senders, and the same characters as a UUID for Opsgenie senders. Owners and admins see it, and a new key stops the old one at once. A source takes 120 events a minute. Removing it resolves its open alerts; the alerts stay on the page.

A tool or SDK that asks for a host takes api.statoss.com. Each source's page in the dashboard shows these with its own key filled in.

An app's issue alert rules and log alerts open alerts too, each on a page of the team's, with no key. They do not count toward the plan's sources.

By tool

• Prometheus Alertmanager: a pagerduty_configs receiver with the key as routing_key and https://api.statoss.com/v2/enqueue as url, as above.
• Grafana 11.1 or later: a PagerDuty contact point with the key as Integration Key and https://api.statoss.com/v2/enqueue as URL, or an Opsgenie contact point with the Opsgenie form of the key and https://api.statoss.com/v2/alerts as Alert API URL.
• Uptime Kuma: a PagerDuty notification with https://api.statoss.com/v2/enqueue as Integration URL and the key as Integration Key.
• Zabbix: the Opsgenie media type with https://api.statoss.com/v2/alerts as opsgenie_api and the Opsgenie form of the key as its token.
• Kibana: a PagerDuty connector with https://api.statoss.com/v2/enqueue as API URL, or an Opsgenie connector with https://api.statoss.com as URL.
• PagerDuty's SDKs: base_url https://api.statoss.com in Python, server: "api.statoss.com" in JavaScript, WithV2EventsAPIEndpoint("https://api.statoss.com") in Go. Opsgenie's SDKs: configuration.host https://api.statoss.com in Python, OpsGenieAPIURL api.statoss.com in Go.

curl -X POST https://api.statoss.com/v2/enqueue \
  -H 'content-type: application/json' \
  -d '{"routing_key":"<key>","event_action":"trigger","payload":{"summary":"Test from curl","source":"curl","severity":"warning"}}'

Where alerts go

An alert is open from its first trigger until it is resolved; later triggers with the same key count onto it. A source's alerts go where the page's alerts go, the destinations ticked under its Settings, Notifications, or to the destinations and people ticked on the source itself. A monitor on the page going down opens a critical alert too, and its recovery resolves it. Alerts has the destinations: email, Slack, Discord, Microsoft Teams, Telegram, PagerDuty, Opsgenie, Pushover, ntfy and a webhook. With on-call, the account's list of them is under Settings, Destinations.

Until someone acknowledges it, an alert from a source is sent again every 5 minutes by default; a source can repeat every 1, 2, 5, 10, 15 or 30 minutes, every hour, or never. Email repeats at most every 30 minutes and 24 times an alert. A monitor's alert stops its "still down" notices once acknowledged.

Acknowledging and resolving

Acknowledging stops the repeats and the escalation; resolving ends the alert. Either works from:

• The dashboard: On-call, for every page or one. Each alert has its timeline, and a note can go on it.
• The alert email: Acknowledge and Resolve, signed links that work for 7 days and ask before they act.
• Pushover: acknowledging an emergency-priority notification acknowledges the alert.
• Slack: alerts in a channel added with Connect Slack carry Acknowledge and Resolve buttons. A channel added by webhook URL gets the message without them.
• The API and the MCP endpoint, below.
• The sender: PagerDuty's acknowledge and resolve events, Opsgenie's acknowledge and close, and Sentry resolving the issue.

An acknowledged alert can be unacknowledged: it is triggered again, and its policy pages from the first level.

An acknowledgement lasts 4 hours. If the alert is still open then, it is triggered again: it goes to its destinations at once and its policy pages from the first level. A page sets this under Settings, Notifications, Acknowledgements: 30 minutes to 24 hours, or until resolved.

Snoozing is for an alert nobody is taking yet: on the alert's page, Snooze 1 h, 4 h or 24 h, or any number of minutes up to a week in the API. Nothing pages or repeats meanwhile; then it is triggered again if still open. The alert shows Snoozed and when it ends, and acknowledging takes a snooze over.

In Slack, the channel where Resolve was pressed shows it on the message itself and gets no separate resolved message; other channels do.

Grouping

A source can group alerts that arrive close together: under the source, Group alerts that arrive within 2, 5, 10 or 30 minutes of its first open alert. A grouped alert pages nobody and is listed under the first as +1 grouped. Acknowledging or snoozing the first does the same to the group. Resolving the first leaves the rest open: the oldest leads the group from then on, and pages if nobody has acknowledged it. A trigger with the same key as an open alert still counts onto that alert, as before.

Maintenance

While a maintenance window covers the whole page, new alerts from its sources are held: they are listed, and can be acknowledged or resolved, but page nobody. When the window ends, each one still open and not acknowledged pages as if it had just arrived, under the same bounds as a new alert: a rule's 20 an hour and the team's 60 (see Floods), the rest going out in a summary. A window for some monitors only holds nothing from sources. The page's own monitors are not checked for alerts during maintenance, as before.

Escalation policies

On Pro. A policy pages people level by level until someone acknowledges. Each level names people and schedules; a schedule stands for whoever is on call when the level is paged. Level 1 is paged at once, and each level waits 1 to 60 minutes before the next. After the last level a policy stops, or starts again from level 1 up to 9 times. A level with nobody on call moves on at once, and someone who has left the team is passed over. A page picks its policy under Settings, Alerts, Escalation, and its monitors' outages page through it; a source follows the page's policy, picks another, or pages nobody. Policies are under On-call, Escalation.

Schedules

On Pro, under On-call, Schedules. A schedule is rotations of people taking turns of hours, days or weeks, from a start and to an optional end, on the schedule's time zone: a handoff at 09:00 stays at 09:00 across daylight saving. A rotation can be limited to some days and hours. Where two rotations overlap, the later one wins where it has someone, so a weekend rotation can sit over a weekday one. An override puts someone on call for a stretch, over every rotation. The schedule shows the next two weeks.

Under On-call, Your notifications, each person sees their own shifts for the next two weeks and a calendar feed of them, from a month back to three months ahead, to subscribe to in Google Calendar, Apple Calendar or Outlook.

Cover

Anyone in the team can ask for cover on a schedule's page: a stretch, and a note. The people in its rotations are emailed, and the schedule's Slack channel is told. The first to press Take the shift gets an override for it, and the person who asked is emailed who took it. A request nobody took can be withdrawn.

Slack

A schedule can name one of the team's Slack channels, which is told who is on call at each handoff and hears of cover requests.

A schedule can also keep a Slack user group, such as @oncall-web, to whoever is on call, so mentioning the group reaches them. Connect Slack for user groups once (it asks to read people's email addresses and to change user groups), then give the group's handle on each schedule: a group is made with that handle. A group someone made in Slack is not taken over, since that would replace its members. People are found by the email address of their StatOSS account. The group changes within a minute of a handoff and is put again every 6 hours; with nobody on call it is left as it was. User groups need a paid Slack plan, and a workspace can limit who may change them.

Load

On-call, Load shows the last 7, 30 or 90 days. For each person: the alerts that paged them, as working hours (09:00 to 18:00 on weekdays), evening, or sleeping hours (23:00 to 07:00), in their own time zone, set under Your notifications; and their hours on call. For each source: the alerts it opened, the share acknowledged, and the middle time to the first acknowledgement or snooze and to resolving.

Your notifications

On Pro, each person sets how they are reached once paged, under On-call, Your notifications. The account's email address is always one way; Pushover (with emergency priority, repeated until acknowledged, if ticked), ntfy and Telegram can be added. The ways are theirs in every team they are in. Rules say which way is used and when: at once, or 1 to 30 minutes after the page. Without rules, a page is an email to the account's address at once.

Floods

Past 60 new alerts from sources in an hour across a team, further warnings and infos are held and sent as one summary per source every 15 minutes, to where the source's alerts go and the first level of its policy. Critical alerts and errors from sources always page one by one. Nothing is dropped: held alerts are listed and can be acknowledged like any other, and alerts from the page's own monitors are never held.

Alerts made by issue alert rules and log alert rules are bounded by the rule, whatever their severity. A rule pages 20 alerts an hour one by one; the rest of that hour are sent as one summary per rule every 15 minutes, to where the rule's alerts go and the first level of its policy. So an app whose first pull brings 200 new issues sends 20 alerts and a summary, not 200. They count in the team's hour like any other alert.

Incidents

From an alert, Open an incident or Add to an open incident. A new one is internal unless that box is unticked: off the page, its history, the feeds, status.json and the page's MCP endpoint, and nobody is told. Publishing it puts it on the page with its updates and tells subscribers, unless asked not to. Several alerts can share one incident, and resolving one does not resolve the other. An incident opened by hand can be internal too. The incidents section has the rest.

Importing from PagerDuty or Opsgenie

On Pro, owners and admins import under On-call, Import. Pick PagerDuty or Opsgenie, the US or EU region, a key, and the page the sources go on. For PagerDuty, a read-only REST API key (Integrations, API Access Keys). For Opsgenie, an API key with Read and Configuration access (Settings, API key management). The key is used for the import and not kept. Preview shows what the import makes and what it leaves out or changes; Import makes what was previewed. Anything imported before is left as it is.

• People are matched to the team by email. Invite anyone missing first; until then their turns go to the next person in the rotation.
• PagerDuty: schedule layers become rotations, checked against PagerDuty's own schedule for the next two weeks, with overrides up to 90 days ahead. Escalation policies come with their levels. Each service's Events API v2 and v1 integrations become sources that keep their keys, so a sender changes only the URL; a key another team already uses gets a new one.
• Opsgenie: each rotation becomes a schedule of its own, since Opsgenie pages every rotation in a schedule at once. Escalations become policies, their rule delays turned into waits between levels. API, Prometheus and Grafana integrations become sources with new keys.
• Not imported: phone, SMS and push contact methods (each person adds their own ways), Opsgenie's routing rules (sources page through the page's policy), PagerDuty's maintenance windows, alert history, heartbeats, rotations and rules that name a team, and integrations that send in a format of their own, which get a source by hand.

After the import, the list says what to change in each tool: the URL, and the key where it is new.

API and MCP

The API lists a page's alerts (open, or the last 100 resolved), reads one with its timeline, and acknowledges, unacknowledges, resolves or snoozes it with a write key, with an optional note. It also says who is on call now, by schedule and by escalation level. Over MCP the same key gets list_alerts, get_alert and get_on_call, and with write access acknowledge_alert, resolve_alert and snooze_alert. open_incident takes internal. For a team without on-call these paths answer 404 and the tools are not listed.