o CVj`\@sxddlZddlZejddejddddlmZmZddlmZm Z m Z e Z edZ e e \ZZZerXederDeeederOed ned edermed edeed edeZ ddlZddlZddlZddlZdd lmZddlmZmZdZdZdZ dZ!dZ"ddZ#ddZ$ddZ%ddZ&ddZ'ddZ(d d!Z)d"d#Z*d$Z+e ,d%d&Z-z e.d'e/e+e0e-Z1Wn e2ye+Z1Ynwe$Z3e&e1Z4e'e4e3\Z5Z6e(e4e5e6e3Z7e8e4Z9e8e5Z:e;d(d)e5DZd*d+e7?Dd,d-d.dd/Z@ed0d-ZAe7?D]ZBeAeBd1eBd2CeBq!eA?D]ZDeD?D] ZEeEjFd3d-d.q<q6e>e5d4d-d5d6dd7ZGed edgZHe@D].ZBeHCd8eBd9d:eBd1d:eBd2d;eBd<d=eBd>d?eBd@d:e*eBdAdBqcdCIeHpdDZJgZKe>eALD]ZMgZNe>eAeMLD]ZOgZPeAeMeOD]LZBeBd<dkrdEndFZQeBd>dkrdGndFZRePCd8eBd9dHeQdIeBd<dHeRdIeBd>d?eBdJdKeBd@d:e)eBdAdLe*eBdAdBqe;dMd)eAeMeODZSe;dNd)eAeMeODZTeNCdOeOdPeSdQeTdReTd'kr2dSnddTdIePdU qeKCdVeMdWdIeNdXqdIeKpXdYZUgZVeGD]CZWeWdZrfd[nd\ZXeVCd]eWdZrrd^ndd_e)eWd`d:eXd:eWd1d:eWd2daeWdbdKeWdcdddedfIeWdgdBq]dCIeVpdhZYedIgdie djeedke1d'krdlnddme1dnkrdlnddoe1dpkrdlnddqe1drkrdlnddse1d$krdlnddte+due1dve9dwe=dxe<dye!dze d{e"d|d}eJd~eUdeYde ddS)Nz/opt/ngon/appsz/var/www/html/ngon) AuthManagergenerate_login_page_html)generate_dropdown_htmlgenerate_dropdown_cssgenerate_dropdown_js gen_outageszContent-Type: application/jsonz6{"success": false, "error": "Authentication required"}z{"success": true}zContent-Type: text/htmlzGen Outage Report) defaultdict)datetime timedeltaz#/opt/ngon/logs/generator_alerts.logz#/opt/ngon/config/master_config.jsonxiXc CsN|}zt|WSty&z t|dWYSty%YYdSww)zUHandle both ISO (2025-10-23T17:19:15.816333) and space (2026-05-03 01:48:52) formats.z%Y-%m-%d %H:%M:%SN)stripr fromisoformat ValueErrorstrptimesr!/var/www/html/ngon/gen_outages.pyparse_ts?s   rcCsi}zGtt }t|}Wdn1swY|diD]"\}}|diD]\}}|diD]}||f||<qz"load_down_events..key)r nowr r4endswithgziprrsplitlenrfloatr TypeErrorr/FileNotFoundErrorOSErrorsort)r5cutoffeventsr,openerr#linepartstsr) alert_level power_str prev_alertpowerrrrload_down_eventsjsB        rZc sg}g}t}tD]w\}\}}}||vrq ||} | sq | \} } g} t|dtD](} | \}}}||tkrAn||krFq.||}|rV|| krV| | q.t| tkrx|||| | fdd| D| |dd| | q |||| | |fq t t }|D] }||d|q| D]X\} }|j ddd t|D]G\}}|d h}|g}t|dt|D] } || }|d |d tkrn ||d || qt|d kr|D]}d ||d<qqq||fS)aTFor each fresh Down event, decide initiator / follower / lonely. Also detect site-wide cascades (multi-group within SITE_WIDE_WINDOW). Returns: cascades: [{initiator, initiator_ts, group, site, followers, is_site_wide, power_at_trip}, ...] lonely: list of (ts, gen_id, group, site, power) r;csg|]}|dqS)r;r).0jrQrr sz#classify_events..F) initiator initiator_tssitegroup followers follower_idxs power_at_trip is_site_wideracSr@Nr`rcrrrrBrCz!classify_events..rDrbr`r Trf)set enumeraterrangerJ total_secondsWINDOW_SECONDSr/CASCADE_THRESHOLDupdater listrrOSITE_WIDE_WINDOW_SECONDSadd)rQ gen_lookupcascadeslonelyused_as_followerirUr)rYsgrarbrcr\f_tsf_gen_f_sgby_siteri site_cascades window_groupswindow_indicesdkrr]rclassify_eventssl           rcCs,i}|D]>\}}}||}|sq|\} } |||| | dddddddd } | dd7<| ddus:|| dkrB|| d<|| d<q|D]5} || d } | rd| d d7<| d sd| d d7<| d D]} || }|ry|dd7<qhqE|D]\}}} } }||} | r| dd7<q}|S)zLPer-gen rollup: total outages, initiator count, follower count, last outage.rNr?) r)rarbtotal initiatedfollowedrvinitiated_excl_sitewide last_outage last_powerrr;rrr_rrfrrcrrv)r setdefault)rQrurvrtstatsrUr)rYryrarbrrir{fsrrrbuild_gen_statssF    rcCs|durdS|dS)N—z%Y-%m-%d %H:%M)strftime)dtrrrfmt_dts rcCs`|durdSt|}|j}|dkr|dS|jd}|dkr&|dS|jd}|dS)Nrr;zd agoizh ago<zm ago)r rFr5seconds)rdeltar5hoursminsrrrfmt_agos      rr514r;ccs|] }|drdVqdS)rfr;Nr)r[rirrr %srcCsg|] }|ddkr|qS)rrrr[rrrrr^*sr^cC|d |d fSNrrrrrrrrB+rBrD cCsttS)N)r rqrrrrrB/rCrarbcCrrrrrrrrB6rcCr@rgrrhrrrrB9rCT)rEreverse2zr)z zrzrzrrz  zTNo cascade initiators found in window.redgreyorangezrvzzcc|]}|dVqdS)rNrrrrrraccr)rNrrrrrrbrz

z (z outages, z cascadera initiated)

z"
Gen ID Initiated Followed Others Lonely Total Last Outage
z>

z

z
z9
No outages in selected window.
rfz-SITE-WIDEz&GROUPz r`zr_rez.0fz kWz, rczDNo cascades in window.a Gen Outage Report

NGON Mining - Gen Outage Report

ar
Generator outages and cascade-trigger analysis · identifies repeat-offender gens whose trips knock out their group
alert-log retention is zp days
Window:zY days
Total outages:zT
Group cascades:zY
Site-wide events:u
How to read: A group cascade is when one gen trips and ≥z/ other gens in the same group also trip within us. The first gen down is classified as the Initiator; the rest are Followers. A site-wide event is when cascades hit ≥2 different groups at the same site within ruA min — usually a shared external cause (gas, weather, transformer) and excluded from the per-gen Initiator counts on this page so the problem-child stats stay focused on actionable failures. Lonely = a Down with no sibling trips in window (single-gen failure, group survived).

Initiated vs Followed Others: Initiated counts cascades this gen started — its trip dragged the group down. High = root-cause troublemaker. Followed Others counts cascades where a sibling tripped first and this gen fell along with the rest. High = structural weakness; gen survives standalone but loses load-redistribution races. A gen high in both is a top swap candidate; high in only one tells you whether the failure mode is "this gen is the trigger" or "this gen can't take the hit when others fail."

Top Cascade Initiators

Gens whose trips have triggered group cascades. Site-wide events excluded. Repeat offenders here are candidates for swap or focused diagnostics.
z
Gen ID Site Group Cascades Initiated Followed Others Total Outages Last Outage

All Outages by Site & Group

Per-gen breakdown. Sorted within each group by initiator count, then total outages.
ac

Recent Cascade Timeline

Last 50 detected cascades, most recent first. Site-wide events shown dimmed (not actionable at gen level).
zB
Time Type Site Group Initiator Power at Trip Followers
)Zcgisysr,insertmanagers.auth_managerrrlinksrrr FieldStorage_form_auth require_auth_auth_required _should_exit_headersprintexitget_user_access _user_accessrr+r2rH collectionsr r r r.rrnrorrrr*r4rZrrrrRETENTION_DAYSgetvalue days_parammaxminintr5rrtrQrurvrrJ total_eventstotal_cascadessumtotal_site_widetotal_group_cascadesr1valuestop_initiatorsrrr/ site_groups group_listrOrecent_cascades top_rows_htmljointop_rowssite_sections_htmlr r% group_blocksr'rows init_class follow_class group_total group_init site_sectionscascade_rows_htmlribadge cascade_rowsrrrrs      "H(               ~    5>PV