What a good alert looks like at 3am
By Vimal Bhaya, Founder and Lead Systems Architect · Last reviewed 3 September 2026
This is a continuation of The alerting problem, a 5-part tutorial. You are on part 5 of 5.
Somebody half awake in the dark has to decide one thing: is this worth getting out of bed for. An alert that cannot answer that question has failed, no matter how accurate the reading is.
Six things do it: which box, how far from normal, which way it is moving and for how long, what is probably wrong, how long you have, and who else has been told.
Most systems send the first two and call it an alert. That is why people stop reading them.
The alert most systems send
Here is a real alert format, next to what the same event has to say if anybody is expected to act on it. Both describe the same freezer at the same moment.
Everything on the left is true. None of it helps.
Is 10.4°F bad? You cannot tell. It might be a defrost, in which case go back to sleep. It might be a door propped open by the overnight cleaner. It might be the third day of a compressor dying. The number is identical in all three cases, and the correct response is different in all three.
So the person receiving it does the only thing available at 3am, which is guess. Most of the time the guess is it will be fine, because most of the time it is. That habit is formed in about two weeks, and after that the system may as well not be installed.
The six things
None of these are exotic. Every one of them is derivable from data the system already holds.
| What it says | What the reader needs it for |
|---|---|
| Which box, which site | Whether it is theirs, and whether anything expensive is in it. A sensor ID is not a location. |
| How far from normal, not just the number | Turning a reading into a judgement. 10°F means nothing on its own and everything against a normal low of -3°F. |
| Which way, and for how long | Separating a defrost from a failure. This is the single most useful line, and almost nobody sends it. |
| What to look at | So the call to the technician is a diagnosis, not a mystery. It also decides whether this is a tonight job. |
| How long you have | The go-back-to-sleep decision. Two days of runway and two hours of runway are entirely different nights. |
| Who else has been told | Stopping two people from driving in, and stopping both from assuming the other one did. |
The third and fifth are the ones that turn an alarm into information. A reading tells you where the freezer is. A trend tells you where it is going, which is the only thing that can justify waking somebody. The mechanics of that are in threshold versus rate-of-change alerting.
The Sunday morning this is drawn from
The alert on the right is not hypothetical in its numbers. Freezer RN1-138 at a Philadelphia distribution centre, holding about $150,000 of frozen inventory, was flagged at 5:30 on the morning of Sunday 20 April 2025.
Notice what did not happen at 5:30 that Sunday. Nobody drove in. Nobody had to. The freezer was at 10.4°F (-12°C), everything inside it was solidly frozen, and it stayed that way for another two days.
The alert was not an emergency. It was a booking instruction. Somebody read it over coffee, understood that the evaporator side was the problem, and had two working days to get a technician scheduled over a holiday weekend.
The technician came on the Tuesday, found a failed defrost timer and an iced coil, and replaced the timer. The freezer peaked at 31.5°F (-0.3°C), which is just under freezing, and no product was lost.
A bare threshold alarm set anywhere sensible would have gone off on the Tuesday, at the peak. Same freezer, same fault, same technician. The difference is that on Tuesday you are not booking a repair, you are counting pallets.
Why the 3am case is the one to design for
Nobody optimises an alert for Tuesday at 2pm, because at 2pm a bad alert is only annoying. Somebody is on site, they walk over, they look at the box, they work it out. The alert can be terrible and the outcome is still fine.
At three in the morning none of that is available. There is no walking over to look. There is one person, in the dark, with a phone, and a decision that has to be made in about fifteen seconds. Design for that and the 2pm case takes care of itself.
It is also when the failures happen, and not by coincidence. Nights, weekends and holidays are exactly when the site is empty and a fault can run for hours without anyone seeing it. Both of the failures on this page started that way: one at 5:30 on Easter Sunday, one at noon on Boxing Day.
A good alert needs somewhere to go
Writing a perfect alert and sending it to one sleeping phone achieves nothing. The two halves only work together.
The reason it is safe to make the 3am alert genuinely urgent, and to escalate it to a ringing phone, is that the alert is worth it. And the reason it is safe to have a rich, wordy alert is that it is only going to a handful of people who have a reason to read it. Those two design decisions hold each other up.
Three people on the escalation list is not redundancy for its own sake. Somebody is asleep, somebody is ill, somebody is on a plane. You are not trying to reach a specific person. You are trying to reach anybody, and the odds of at least one of three answering are very different from the odds of one answering. How that is configured, and why it is set per freezer, is covered in alerting more than one person.
The same structure protects the ops manager from the volume. They are not on level one for thirty boxes. They appear when the person nearest the freezer has already had a chance and did not take it, which is a few times a year rather than a few times a day.
How to test what you have now
You do not need to buy anything to find out whether your current alerts are any good. Open the last one you received and try to answer these, from the alert alone.
- Which physical box is this, and roughly what is in it
- Is this reading unusual for this box, at this hour
- Has it been getting worse, or did it just spike
- Do I need to do something tonight, or is the morning fine
- Who else got this, and is one of them already on it
If you cannot answer four out of five without opening a dashboard, your alerts are not alerts. They are notifications that a dashboard exists, and at three in the morning nobody opens the dashboard.
Fixing this is mostly not a hardware problem. The readings you need are already being recorded. What is missing is the system doing the comparison for you, at the moment it matters, instead of leaving it to somebody who is half asleep.