Times shown US/Pacific PST8PDT -0800 unless indicated otherwise:
PG&E power outage(s):
2021-12-13: 8:00 PM -- 5:00 PM PST
Not out the solid the entire time for me, but multiple outages
including multiple long outages, PG&E reported 1,048 customers impacted.
Which also takes systems offline (network equipment dependencies):
Systems back online and operational again by (some requiring manual
restart in this case) approximately:
1:10:36 PM except*
Impacted for the duration:
IPs: 96.86.170.224/29 2001:470:1f04:19e::2 2001:470:1f05:19e::/64
Domains/[L]UGs/SIGs:
DNS/Domains:
balug.org e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa
sf-lug.orgsf-lug.comsflug.orgsflug.comsflug.netsf-lug.netberkeleylug.com
DNS - reduced redundancy, masters off-line, DNS otherwise functional
Excluding DNS, services of the [L]UGs/SIGs associated with the above:
BALUG - offline
SF-LUG - offline except list still online (separately hosted)
[Pi.]BerkeleyLUG - offline except list still online (separately hosted)
*except, haven't determined exactly why, but, at least after 2nd time VM
had been crashed and restarted due to the outages, the 2nd time the VM was
restarted, there was an error with the web server being restarted ... not
sure why - perhaps some systemd unit file flaw(s) with some subtle
timing or sequencing dependency or such bug/flaw. In any case, web server
didn't restart at boot. I didn't notice this until bit before and had it
successfully started by:
2021-12-14 5:27:02 P.M.
As far as I'm aware thus far, nothing else that was supposed to (re)start
at (re)boot failed to (re)start. Sorry I didn't catch that sooner.
(Better) monitoring is in the queue of many LUG tasks to be done.
A simple normal start of the web server was all that was required to
have it fully operational again.
If there are folks that want sudo access to be able to
reload and (re)start the web server
(and possibly also with sufficient justification, access to stop it),
let me know, and that can be arranged. That could'a got the
web server (re)started as much as about 28:16:26 sooner.
Similar earlier this month:
https://lists.balug.org/pipermail/balug-admin/2021-December/001061.html
earlier BALUG, etc. services outages due to PG&E power outage
PG&E power outage:
2021-12-01:
1:36 PM -- 6:26 PM PST
Which also takes systems offline (network equipment dependencies):
Systems back online and operational again by (some requiring manual
restart in this case) approximately:
9:09:02 PM PST
Impacted for the duration:
IPs: 96.86.170.224/29 2001:470:1f04:19e::2 2001:470:1f05:19e::/64
Domains/[L]UGs/SIGs:
DNS/Domains:
balug.org e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa
sf-lug.orgsf-lug.comsflug.orgsflug.comsflug.netsf-lug.netberkeleylug.com
DNS - reduced redundancy, masters off-line, DNS otherwise functional
Excluding DNS, services of the [L]UGs/SIGs associated with the above:
BALUG - offline
SF-LUG - offline except list still online (separately hosted)
[Pi.]BerkeleyLUG - offline except list still online (separately hosted)
> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
> Subject: Re: Notice: ns1.svlug.org downtime, DNS secondary
> Date: Sat, 13 Nov 2021 01:30:53 -0800
> Thanks, noted, also:
> $ dig +noall +answer +nottl ns1.svlug.org. AAAA www.svlug.org. AAAA
> ns1.svlug.org. IN AAAA 2600:3c01::f03c:91ff:fe96:e78e
> www.svlug.org. IN AAAA 2600:3c01::f03c:91ff:fe96:e78e
>
> (trimmed for relevance/brevity):
> https://www.mpaoli.net/~michael/bin/DNS_CK
> $ DNS_CK
> FQDN: balug.org.:
> authority:
> balug.org. 86400 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa.:
> authority:
> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 4900 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: sf-lug.org.:
> authority:
> sf-lug.org. 86400 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: sf-lug.com.:
> authority:
> sf-lug.com. 172800 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: sflug.org.:
> authority:
> sflug.org. 86400 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: sflug.com.:
> authority:
> sflug.com. 172800 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: sflug.net.:
> authority:
> sflug.net. 172800 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: sf-lug.net.:
> authority:
> sf-lug.net. 172800 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> FQDN: berkeleylug.com.:
> authority:
> berkeleylug.com. 172800 IN NS ns1.svlug.org.
> ;; connection timed out; no servers could be reached @64.62.190.98
> (ns1.svlug.org.)
> ;; connection timed out; no servers could be reached
> @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.)
> $
>
> And removed from masters/primaries:
> balug.org. 86400 IN NS ns1.svlug.org.
> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 4900 IN NS ns1.svlug.org.
> sf-lug.org. 86400 IN NS ns1.svlug.org.
> sf-lug.com. 172800 IN NS ns1.svlug.org.
> sflug.org. 86400 IN NS ns1.svlug.org.
> sflug.com. 172800 IN NS ns1.svlug.org.
> sflug.net. 172800 IN NS ns1.svlug.org.
> sf-lug.net. 172800 IN NS ns1.svlug.org.
> berkeleylug.com. 172800 IN NS ns1.svlug.org.
> # (for d in balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa.
> sf-lug.org. sf-lug.com. sflug.org. sflug.com. sflug.net. sf-lug.net.
> berkeleylug.com.; do { echo "update delete $d IN NS ns1.svlug.org.";
> echo send; } | nsupdate -l; done)
> #
>
> And delegating authority records ...
> removed from:
> balug.org.
> sf-lug.org.
> berkeleylug.com.
> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa.
> DNS_CK now comes back clean, notwithstanding pending delegating
> authority record deletion:
>
> And Al, I presume you will likewise handle the needed for:
> sf-lug.com.
> sflug.org.
> sflug.com.
> sflug.net.
> sf-lug.net.
>
> references/excerpts:
>
>> Date: Fri, 12 Nov 2021 17:24:51 -0800
>> From: Rick Moen <rick(a)linuxmafia.com>
>> To: Michael.Paoli(a)cal.berkeley.edu, [TRIMMED]
>> Subject: Notice: ns1.svlug.org downtime, DNS secondary
>>
>> Starting late this morning (2021-11-12), SVLUG was obliged to
>> take down for rearchitecting and rebuild its Linode virthost
>> www.svlug.org AKA ns1.svlug.org, IP address 64.62.190.98 .
>> Downtime will persist for some time. There is no current ETA.
>>
>> _You_ are receiving this (form) e-mail because you are, or have been a
>> DNS admin for one or more of the following domains:
>>
>> balug.org
>> berkeleylug.com
>> sf-lug.org
>> sf-lug.net
>> sf-lug.com
>> sflug.org
>> sflug.net
>> sflug.com
>>
>> (Please forward this notice to any DNS sysadmin I've missed.)
>>
>> Please be advised that ns1.svlug.org is NOT, for the time being,
>> doing secondary (or any) DNS, so should be removed from rotation
>> at the DNS master, effective immediate. (I'd recommend also
>> taking it out of the parent zone records at your registrar.)
>>
>> SVLUG will advise when ns1.svlug.org service resumes. We expect the IP
>> to be unchanged.
>>
>> -- Rick Moen
>> rick(a)linuxmafia.com
>> (650) 283-7902
>> For Silicon Valley Linux User Group