This page explains how the NGON auto-sleep, auto-wake, gen-disable, Emergency Sleep, and capacity-hold systems work together. It's written for field techs, so you can understand what the system is already doing for you and avoid unnecessary manual toggling.
| Feature | What it does | Type |
|---|---|---|
| AS Auto-Sleep | Puts miners to sleep when a gen goes down or when group kW goes over target | Automatic |
| AW Auto-Wake | Wakes sleeping miners when the group has kW headroom under target | Automatic |
| Disable Gen | Tells the system "pretend this gen isn't there" — it's dropped from the group's capacity math | Manual |
| EMS Emergency Sleep | Holds a pod's miners asleep — either while booting up, or to keep a running pod asleep indefinitely | Manual |
| Capacity Hold | Reserves a shaky gen's headroom before it fails so the group survives losing it (auto on coolant/shutdown warnings) | Automatic |
| Situation | What to do | What NOT to do |
|---|---|---|
| Gen goes down on its own | Nothing — AS handles it | Don't sleep miners manually |
| Taking a gen out for service | Disable the gen in the Edit kW modal | Don't also turn off AW |
| Bringing a serviced gen back | Re-enable the gen in the Edit kW modal | Don't manually wake miners |
| Pod is physically down / being worked on | Turn on EMS for that pod | Don't rely on AS alone |
| Whole site under maintenance | Turn off AS and AW on those pods |
The whole power system runs on one simple idea: match miner load
to available generator capacity. Every generator group has a
max_gen_kw target (how many kW each running gen can safely
carry). Each miner type has a spec_wattage (set in Site
Manager → Miner Types) that tells the system how many kW one miner of
that type draws; each pod's kW-per-miner is derived from its assigned
miner type. The system continuously watches the gap between current
gen load and target, and either sleeps or wakes miners to close the gap.
There is no longer a separate "auto-wake cron." Both AS and AW run
inside generator_monitor.py, which re-evaluates every group
about every ~5 seconds. That means AS and AW are really two sides of the
same equation — one decision engine, not two independent systems.
The engine reads gen load from one of two feeds. Most gens now stream live telemetry straight from the gen (~1 reading per second). When every gen in a group is streaming fresh live data, that group runs on the faster live path. If any gen's live feed goes missing or stale, the whole group falls back to the Mesa path, which uses the provider's ~5-minute averaged data. Same decisions either way — just slightly different timing. That's why some numbers below are given as a range: the live path reacts faster and settles new gens in ~2 minutes, where the Mesa path waits ~15.
AS and AW are not "settings you tune to respond to events." They are the system's normal heartbeat. If you leave them on, the system will correctly react to gen failures, capacity changes, and gen-disables without you touching them.
max_gen_kw. That's the target.When a gen drops, the load it was carrying gets redistributed onto the surviving gens instantly, but our gen telemetry can be minutes old. Trying to compute "how many miners to shed" using stale data has caused cascade trips. Sleeping everything is the only choice that's guaranteed safe regardless of what the data says. AW then ramps the pod back up gradually so the survivors are never overloaded during recovery.
Whenever EMS turns on for a pod — whether you toggled it manually or the system auto-armed it on a gen event — the pod's Auto-Wake and Reboot-Zero toggles automatically flip OFF too. When EMS clears, all three flip back ON together. This is by design: EMS means "nothing automated should be touching this pod." You'll see the toggles move as a group on the status page.
Config field: auto_sleep_enabled, per pod.
auto_sleep_enabled = falseWithout AS, a gen failure at 3am means miners keep drawing load from the remaining gens, overloading them and tripping them offline too — a cascade that can take an entire site down. AS reacts in seconds — the live-telemetry fast-EMS path (see "Newer Automatic Protections" below) catches a gen dropping off load within ~5 seconds, ahead of the slower provider feed — and protects the remaining gens by arming EMS, which the per-pod agent keeps re-asserting until the pod is fully down. That's more robust than the old single-shot sleep call, which could miss miners that were mid-wake or temporarily unreachable.
AS actions show up in the Miner Monitor section with user AS. If you see AS events, the system just protected a gen for you.
Config field: auto_wake_enabled, per pod.
auto_wake_enabled = true.AW does not wake the whole pod in one shot. Each wake batch is at least one gen's worth of miners (about 90–105 for a 330 kW gen, depending on miner type); on larger groups it may wake somewhat more per batch. After each batch AW waits ~5 minutes before the next — and if a gen is already running near full load (≥90%), it stretches that wait to ~15 minutes — giving the surviving gens time to absorb the new load before more miners are added.
For a fully slept 5-gen pod (e.g., right after panic-sleep or after EMS is turned off), full recovery takes about 15–25 minutes of gradual ramp-up.
Manual wake from the NMT or MFT pages bypasses the batch cap. The batched ramp only applies to AW. So if you've personally verified the gens are solid and want everything up now, manual wake is the escape hatch.
auto_wake_enabled = falseWhen a gen comes back online after a failure or service, someone used to have to manually wake miners across the affected pods. AW does this automatically — and proportionally across pods in the same group — the moment the gen starts contributing load.
AW actions show up with user AW in the Miner Monitor log section.
Where: Status page → click the group's kW target to open the kW modal → toggle individual gens off.
Config field: disabled_gens, per generator
group (a list of gen IDs).
The kW modal lets you do two independent things, and it's easy to mix them up:
max_gen_kw — this raises or lowers
the per-gen kW target for the whole group. Use this when the
generators' actual safe load has changed (e.g., derating for hot weather,
or bumping up after a tune). It affects every gen in the group equally.max_gen_kw stays exactly the same; the group just has
one fewer gen contributing to the total target.Most of the time you only want to disable the gen. Leave
max_gen_kw alone unless you actually need to change the
per-gen target.
When you toggle a gen off in the modal, the system:
disabled_gens list.total_target = max_gen_kw × (remaining enabled gens). Note that max_gen_kw itself does not change — there's just one fewer gen multiplying it.Re-enabling the gen is the reverse: it comes back into the math, total target goes up, AW wakes miners to fill the new capacity.
This is the single biggest point of confusion. The system already excludes disabled gens from AW's math. AW will keep doing its job correctly with the remaining gens — it will not try to wake miners onto the disabled one. Leave AW alone.
If you also turn off AW, you've told the system "don't wake any miners in this pod, ever, for any reason." That usually isn't what you want, and it gets forgotten — so when the gen is put back in service the pod sits half-asleep until someone notices.
If you disable every gen in the group (the "shut the whole group down" workflow), the system treats that like a gen-down event and auto-arms EMS on every eligible pod in the group. EMS turning on auto-flips Auto-Wake and Reboot-Zero off as part of the EMS coupling described above. So when you later re-enable gens, you'll need to toggle EMS off manually to bring the pods back — AW won't wake them on its own because AW is also off. (See Scenario 3, Method B.)
A gen being physically offline (failed, tripped) is different from a gen being intentionally offline (servicing, swapping, parts on order). In the failure case, the provider API tells us it's down and AS fires. In the intentional case, the gen may still be reporting Running, or the tech may pull it before the system notices — so we need a way to say "don't count this one, even if it looks alive." That's what disabling does.
Where: Blue "EMS" toggle on the status page gen group header (per-pod).
EMS is the heavy hammer. It lives on the field server, in the per-pod agent. When EMS is on for a pod, the pod agent:
Meanwhile, generator_monitor sees the pod has EMS on
and skips it for both AS and AW (and Reboot-Zero, since R0
also flips off as part of the EMS coupling). While EMS is on, that pod
is effectively invisible to the power manager and stays asleep no
matter what the power math says.
1. Manual toggle — you click the EMS toggle on the status page. AW and R0 toggles flip off automatically.
2. Auto-arm — the system arms EMS itself on a gen event. Specifically: a gen-down alert (single gen tripping) or a "disable all gens in group" action (the maintenance shutdown path). You'll see the EMS toggle light up on its own and AW/R0 flip off — no one did it, the system did. Auto-arm keeps the pod safe until AW is ready to wake it.
1. Manual toggle — you click the EMS toggle off. AW and R0 toggles flip back on automatically.
2. Auto-clear. There are two automatic cases:
1. Cold boot protection. When a pod boots cold with its generator not yet stable, miners come up and start hashing within seconds — often before the gen can handle the load. Turn EMS on before powering the pod, and the agent will catch the miners the moment they come online.
2. Holding a pod asleep indefinitely. EMS also works on a pod that's already up and running. Turn it on and the agent keeps slamming sleep commands at the miners, so nothing — not AW, not a manual wake, not a miner rebooting itself — can wake them back up. This is the right tool when you want to park a pod for a while (maintenance, low gas, a gen swap in the same group) without worrying about AW trying to wake it every time capacity opens up.
3. The system's own gen-down response. AS now uses EMS internally for gen-trip events. You don't need to do anything for this case; it's documented here so you understand why the EMS toggle moves on its own sometimes.
A pod with EMS left on stays asleep — AS and AW can't help it because they skip EMS pods. The auto-clear paths cover most cases (AW resume, pod-dark disarm), but a manually-toggled EMS only clears when you toggle it off, and a Method-B-style "disable all gens" auto-arm also stays on after re-enabling gens. Always glance at EMS state if a pod isn't waking when you'd expect.
Two newer safety layers run entirely on their own. You don't operate them, but you will see them act — so here's what they are, so nothing looks mysterious.
Every gen that streams live telemetry is watched for the instant it drops off load — either its breaker opens or the engine stops. The moment that happens, the system arms EMS on that gen's pods and sheds their miners, usually within ~5 seconds of the trip — roughly 30 seconds to 2 minutes ahead of the older provider (Mesa) feed, which lags. It's the same auto-EMS you already know from AS; it just fires off the fast live signal instead of waiting for Mesa. It works even if the rest of the group isn't fully live, because sleeping miners is always the safe direction.
After a real trip you'll often see pods already asleep before the gen even shows "Down" on the provider data. That's the fast path doing its job — nothing is broken.
When a gen throws a coolant-temp or shutdown-class warning (the warning triangle on the gen row), the system can put that gen on a capacity hold before it actually fails. The gen keeps running and producing, but the power engine stops counting its capacity toward the group target — so the group sheds enough miners to leave one gen's worth of headroom, and AW won't refill it. If that shaky gen then drops, the group already has room to absorb the load and rides through instead of cascading.
A held gen shows a pause-circle icon on its row.
If a group looks like it's carrying fewer miners than its gen count suggests it could, check for a hold or a warning on one of its gens. That headroom is being held back deliberately so losing that gen won't cascade the group. Don't fight it by bumping kW or force-waking — clear the underlying gen issue and mark it repaired.
What you do: Nothing. Check the log in the morning. Expect AS-armed-EMS events followed by 4–6 small AW batches over the next ~25 minutes. The toggles will end up exactly where they started.
What you do NOT do: Turn off AW. Turn off AS. Manually sleep miners. Manually wake miners. The disable-gen action does all of that for you.
You're about to shut all the gens off (gas line work, swap a header, running cabling, etc.) and need every miner asleep before the gens go down so the gens don't trip from a sudden no-load → load → no-load cycle and so you're not racing physical shutdown against a panic-sleep event. Two ways to do this — pick whichever is faster for your situation.
Why this is preferred: EMS holds miners asleep regardless of any other system state. It survives miner reboots, gen restarts, and network blips. Lowest chance of surprises.
The gen-down auto-arm path (Scenario 1, single gen) self-clears because AW eventually wakes the pod, and AW clears EMS as it does. Method B is different — disabling all gens leaves AW off, so AW never fires the recovery clear. You have to toggle EMS off yourself once gens are back. If you want to avoid this manual step, use Method A instead.
Either EMS the pods or disable all gens — picking one is enough. Using both at once is fine but redundant. The trap is doing neither and just yanking the gens, which causes a real gen-down cascade event.
Why it's wrong: The disable-gen action already excludes the gen from AW's math. Turning off AW on top of that just prevents the pod from recovering normally and gets forgotten later.
What to do instead: Just disable the gen. Leave AW on.
Why it's wrong: AS panic-slept every awake miner in the affected pods within 1–2 seconds of the gen-down alert. There are no awake miners left to sleep — you'd be hitting an empty target.
What to do instead: Check the log for an AS
entry. If you see it, the system handled it. Watch for AW batches over
the next 15–25 min as the pod ramps back up.
Why it's wrong: AW sees headroom and starts waking miners right back up within a minute or two. It looks like the sleep didn't take.
What to do instead: For maintenance, use EMS — it holds the pod asleep regardless of what AW thinks. For taking a gen offline, use the disable-gen toggle.
Why it's wrong: The pod stays asleep — AS and AW cannot bring it back, because they skip EMS pods entirely.
What to do instead: Always clear EMS once the gen is stable and the pod is ready to hash. After a Method-B-style "disable all gens" maintenance, you must toggle EMS off manually because AW can't auto-clear it (AW is also off). After a single-gen gen-down event (Scenario 1), EMS auto-clears as AW wakes the pod, so it's normally hands-off.
Why it's confusing: EMS isn't only manual anymore. Gen-down events and "disable all gens" auto-arm EMS; AW resumes and ~90s of pod-dark-during-EMS auto-clear it. AW and R0 toggles flip with EMS as a group. So a row of three toggles changing state at once on a pod is normal — the system did it on purpose.
What to do: If toggles flipped after a gen alert, that's expected — the system armed EMS for the cascade. If toggles flipped without an alert, glance at the gen group: any disabled gens? That's likely the cause. If neither, then someone clicked the toggle.
Why it's wrong: Gen notes and status (oos, parts, swap) are informational only — they do not affect AS, AW, or capacity calculations. Only the disable toggle in the Edit kW modal affects the math.
What to do instead: If the gen shouldn't be counted, disable it in the Edit kW modal. If it's just a note for the team, set the status in Generator Manager.