I think it's a great idea, I've been looking at allot of vm work too and
that sounds like fruitful research.
Especially excited to learn more about filesystems
On Sat, May 3, 2025, 10:03 AM Michael Paoli via BALUG-Talk <
balug-talk(a)lists.balug.org> wrote:
>
>
>
> ---------- Forwarded message ----------
> From: Michael Paoli <michael.paoli(a)berkeley.edu>
> To: BALUG-Talk <balug-talk(a)lists.balug.org>
> Cc:
> Bcc:
> Date: Sat, 3 May 2025 10:01:…
[View More]58 -0700
> Subject: [BALUG-Talk] Possible topic(s) for 2025-05-20 BALUG meeting
> (storage, ...)
> So, was thinking, possible topic(s) for 2025-05-20 BALUG meeting.
> I had in mind, potentially something (like/approximating):
> Storage (partitioning, ext{2|3}4}, md, LVM, LUKS, ZFS, migrating,
> modernizing, growing, shrinking, etc.)
>
> Probably most notably since I've been doing some more mucking about
> with most all of that recently - looking at some drive upgrades
> (notably capacity increases), and been doing fair bit of planning and
> testing (mostly on VMs) to test out various scenarios, and also
> determine how I want to get from where I/we are, to where I/we want to
> be (some of the storage layout hasn't been significantly redesigned in
> well over a decade).
>
> Could also talk about Ventoy (also been doing some testing of that
> recently - pretty nifty, but alas, doesn't handle all bootable
> images), or of course other topic(s) folks might be interested in.
>
> So, thoughts/suggestions on topic(s) for 2025-05-20 meeting?
>
>
>
> ---------- Forwarded message ----------
> From: Michael Paoli via BALUG-Talk <balug-talk(a)lists.balug.org>
> To: BALUG-Talk <balug-talk(a)lists.balug.org>
> Cc:
> Bcc:
> Date: Sat, 3 May 2025 10:01:58 -0700
> Subject: [BALUG-Talk] Possible topic(s) for 2025-05-20 BALUG meeting
> (storage, ...)
> _______________________________________________
> list: BALUG-Talk(a)lists.balug.org help: https://lists.balug.org/help/
> unsubscribe email: balug-talk-unsubscribe(a)lists.balug.org
[View Less]
So, was thinking, possible topic(s) for 2025-05-20 BALUG meeting.
I had in mind, potentially something (like/approximating):
Storage (partitioning, ext{2|3}4}, md, LVM, LUKS, ZFS, migrating,
modernizing, growing, shrinking, etc.)
Probably most notably since I've been doing some more mucking about
with most all of that recently - looking at some drive upgrades
(notably capacity increases), and been doing fair bit of planning and
testing (mostly on VMs) to test out various scenarios, and also
…
[View More]determine how I want to get from where I/we are, to where I/we want to
be (some of the storage layout hasn't been significantly redesigned in
well over a decade).
Could also talk about Ventoy (also been doing some testing of that
recently - pretty nifty, but alas, doesn't handle all bootable
images), or of course other topic(s) folks might be interested in.
So, thoughts/suggestions on topic(s) for 2025-05-20 meeting?
[View Less]
Sorry for late reply,
This sounds like just what I was hoping to learn about
To give some context, I am currently trying to learn how to make
reproducible OS builds using mkosi <https://github.com/systemd/mkosi>
One of the big advantages of the tool is that is helps get closer to the
points outlined in Fitting Everything together
<https://0pointer.net/blog/fitting-everything-together.html> by Pottering
in which he outlines some key goals including:
- increasing usage of the TPM …
[View More]to increase security
- self-signing keys so we can have operating systems that are both
immutable AND hackable (no more corporate signed keys!)
- ensuring the validity of the entire stack by using secure boot, encrypted
home drives, and sandboxing for user applications to make a more secure
environment
All of this is with the goal of running particleos
<https://github.com/systemd/particleos> eventually but i would settle for
just a custom-rolled arch distro that I upgrade in A/B fashion. But one of
the fundamental assumptions of all of this is that im self-signing keys and
using them for everything from secure-boot, to my home drive if necessary
but I have held off from learning these security topics until now!
On Thu, Apr 10, 2025 at 10:54 PM Michael Paoli via BALUG-Talk <
balug-talk(a)lists.balug.org> wrote:
>
>
>
> ---------- Forwarded message ----------
> From: Michael Paoli <michael.paoli(a)berkeley.edu>
> To: BALUG-Talk <balug-talk(a)lists.balug.org>
> Cc: Darrion Burgess <dargondab9(a)gmail.com>
> Bcc:
> Date: Thu, 10 Apr 2025 22:52:53 -0700
> Subject: [BALUG-Talk] Discussion topic(s) for Tuesday's meeting? :-)
> Thanks all, yes meeting last month was excellent, thanks for making it
> so!
>
> So, I was thinking, topic(s) (not that we need be limited to such) for
> meeting this month (soon - Tuesday!) ... and following a bit from last
> month's meeting, perhaps something along the lines:
> Linux encryption and security hardware.
> So, perhaps around LUKS, TPM chip, YubiKey, FIDO2, etc.
> More-or-less extension of fair bit that was discussed at last meeting.
> And, not too horribly redundant with presentations/talks/topics that
> have been done at BALUG before (e.g. done LUKS presentation before,
> have at least had security as topic, but don't think we've (much)
> covered TPM chip, YubiKey, FIDO2, etc. before, so was thinking to mix it
> up and expand it a bit more.
> Also, did we have some other question(s)/topic(s) from last meeting that
> we didn't quite get around to covering?
>
> Feel free to let me know your thoughts, and I'll put together at
> least something regarding (at least leading, but by no means limited to)
> topics for the meeting, and will get that then updated on the web site,
> and also use that for items sent to the Announce list before the
> meeting. Anyway, I'm hoping/aiming to get that done and at least
> initial bits out in the next couple days or so
>
>
>
> ---------- Forwarded message ----------
> From: Michael Paoli via BALUG-Talk <balug-talk(a)lists.balug.org>
> To: BALUG-Talk <balug-talk(a)lists.balug.org>
> Cc: Darrion Burgess <dargondab9(a)gmail.com>
> Bcc:
> Date: Thu, 10 Apr 2025 22:52:53 -0700
> Subject: [BALUG-Talk] Discussion topic(s) for Tuesday's meeting? :-)
> _______________________________________________
> list: BALUG-Talk(a)lists.balug.org help: https://lists.balug.org/help/
> unsubscribe email: balug-talk-unsubscribe(a)lists.balug.org
[View Less]
Well, for starting to dip the toe into Linux, from Microsoft Windows or
macOS ...
On non-ancient Microsoft Windows, for ease of entry, I'd probably first
point to Microsoft's WSL - as such is pretty conveniently available, and
tightly integrates with Microsoft Windows, so it makes it quite easy to,
e.g. move data back and forth, and generally access resources of one
from the other. Been a while since I mucked about with WSL, but one
thing I found (quite) annoying with it, though it's quite …
[View More]the
Linux(-like) operating environment, it's not "really" a Linux operating
system - at least not fully so - and not even a VM. So, if one is
rather to quite used to Linux, and using and treating it as a system,
and probably even more so administration thereof ... well, yeah, there's
a lot of stuff one will bump into that just won't work, because it's not
really and fully an installed running Linux operating system, nor even
such in a VM. But "other than that", yeah, sure, mostly seems to work
quite well, and hard to beat for easily available on (non-ancient)
Microsoft Windows and good tight integration.
Next I might suggest Oracle's VirtualBox. And, it has its advantages
and disadvantages. There is of course the whole Oracle is evil thing:
https://www.youtube.com/watch?v=-zRN7XLCRhc&t=1980s
etc., but some of the other disadvantages, over the years, Oracle has
made it increasingly difficult to do more custom installations of Linux.
Still quite possible, but in more recent years, VirtualBox wants to be
"smart" and "help", by recognizing many Linux ISOs, and for those it
recognizes, essentially to large extent taking over and
automating/controlling the installation, making it much more difficult
to have access to the more typical means by which Linux would generally
be installed from ISO, and accessing its menus and options and
installation dialogs and such - and instead VirtualBox just gives one
its choices, and hides all those other details from one. Anyway, is
still (at least last I checked) possible to do the more customary VM
install, quite as if one were simply booting the ISO, and installing
from what it offers, and with its menus and dialogs, etc., but
VirtualBox just makes it increasingly difficult to manage to do such an
install. Anyway, those are the among the most frustrating bits I find
with VirtualBox ... though last I actually did an install from it was
likely year or more ago - and has been about half a year since I've used
VirtualBox - but I've often heavily used it, notably in $work
environments, on macOS and/or Microsoft Windows. And, it does have the
advantages that it is a full VM, unlike, e.g. WSL (or Cygwin, or
Homebrew). It also has good integration with Microsoft Windows / macOS
(notably filesystem access, etc.), but not as tightly integrated as WSL.
What I may quite suggest though (alas, haven't yet done so myself, but
if I were about to do a fresh installation to Microsoft Windows or
macOS, probably where I'd start), although significantly more complex
to set up, would be Homebrew, and with/from that, qemu/kvm, libvirt and
friends, etc. I think that gives a significantly better VM platform,
though it's fair bit more fiddly bits to get that all set up and nicely
working, and also bit more work to do, e.g. filesystem integration
bits. But once all in place, probably well worth it. Yes, one of my
work peers had gone that direction, and the results were most excellent.
So, though I've not yet done that myself, I'd think the results and
capabilities would be very much like where I've done quite similar where
the host OS is Linux ... but not with Homebrew in there, but including
all the other relevant bits, e.g. qemu/kvm, and libvirt and friends.
I've not added tight filesystem integration between VMs and the physical
host OS with that though, with or without Homebrew ... - I've not yet
had/felt compelling need ... but were I doing it on Microsoft Windows or
macOS, or otherwise needed to more directly share lots of files between
VM and physical host, then I'd probably take the additional steps to set
that up.
Other worthy of mention bits ...
Dual boot - once upon a time, that was generally what folks were pointed
at, and most Linux distros make installing that way pretty easy. The
big downsides with that, though, it's more complex than doing, e.g. WSL
or VirtualBox to have installed and up and running, and probably more
importantly, with dual boot - one only runs one OS at at time. And with
that, most folks typically end up spending most all their time just
running one, and not doing much with the other ... and most of the time
that means not doing much with Linux and often effectively giving up on
it relatively early on. But there still are some circumstances where
I'd recommend dual boot. E.g. if one is transitioning, notably to
entirely switch away from whatever the other OS is, or is on that
particular physical host, then dual boot is best way to be sure one
knows exactly how it will behave on that particular hardware, get any
associated kinks worked out, etc. So, e.g., when I fist ran Linux in
1989, yeah, I was doing dual boot ... Linux first on floppies, then dual
boot on the hard drive (and may have still required a boot floppy for
the way I was set up), and finally completing my migration from UNIX to
Linux. And back then, VM was totally infeasible if not for all intents
and purposes impossible, for the hardware and its resources I was using
at the time. So, sometimes dual boot is the answer, but these days,
most of the time it's not the answer / best fit.
There's also Homebrew itself - not Linux, but can provide much of a
*nix-like environment on, e.g. Microsoft Windows or MacOS.
For X11 on MacOS, I found XQuartz to work highly well.
For X11 on Microsoft Windows, been years since I did it, but I found
Cygwin/X worked quite well.
And, somewhat similar to Homebrew, for Microsoft Windows, there's
Cygwin, though been years since I ran Cygwin on Microsoft Windows, so
not sure how it's comparing capability-wise and such to Homebrew these
days (in fact I've not run Homebrew on Microsoft Windows, but have
generally heard quite good things from those that have).
From other platforms, e.g. the BSDs, what options are (feasibly)
available may be bit different. Of course dual boot is a possibility,
some VM technologies may well work, and there may be some other
possibilities too - of the various bits mentioned, not sure which may be
available to the BSDs.
Oh, another worthy of mention - VMware - they may have different
possibilities available for various platforms, but licensing and/or cost
may be an issue, depending upon environment, what they do/don't
currently offer (and notably what for free and under what terms), etc.,
and what they do offer, may be subject to change (with ownership change
in recent years, that's been a bit more bumpy ... but hopefully that's
more settling/settled out at this point). Anyway, I haven't run such in
some fair number of years now - don't know if they even offer software
for running on macOS. Anyway, it has been a good option in past, not
sure about most current.
And, I may be missing some bits, but I think those are at least most
that come up in conversations, etc., regarding various relevant
possibilities.
On Sun, Mar 23, 2025 at 12:26 PM aaronco thirtysix via BALUG-Talk
<balug-talk(a)lists.balug.org> wrote:
>
>
>
>
> ---------- Forwarded message ----------
> From: aaronco36(a)sdf.org
> To: balug-talk(a)lists.balug.org
> Cc:
> Bcc:
> Date: Sun, 23 Mar 2025 19:26:06 -0000
> Subject: [BALUG-Talk] Linux w/in MS-Windows... great chat! extended
> Quoted snippet of Darrion Burgess from BALUG-Talk thread
> https://lists.balug.org/mailman3/hyperkitty/list/balug-talk@lists.balug.org…
> :
> >
> > to start, the easiest way to ease into is using the windows subsystem
> > for linux (WSL) so you can try out different distributions using
> > the windows terminal.
> >
> > Its even sophisticated enough to have a wayland compositor at this
> > point so you can even run GUI apps from the linux containers.
>
> Besides WSL, another good(IMHO) means of more easily running Linux distros
> w/in MS-Windows is to use Oracle VirtualBox
> Quoting its main site https://www.virtualbox.org/ :
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> VirtualBox is a general-purpose full virtualization
> software for x86_64 hardware (with version 7.1
> additionally for macOS/Arm), targeted at laptop,
> desktop, server and embedded use.
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The VirtualBox 7.1.6 platform packages dwonloads for Windows and other OS
> hosts can be found at https://www.virtualbox.org/wiki/Downloads
>
> Am concurrently attending both the in-person and Jit.si Meet virtual
> meetups of the Berkeley Linux Users Group https://berkeleylug.com/ -- and
> am using VirtualBox for Debian GNU/Linux 12.x 'Bookworm' as hosting OS on
> a second laptop.
> VBox works great and is more [ful]filling! :-D
>
> -A
> --
>
>
>
>
>
> ---------- Forwarded message ----------
> From: aaronco thirtysix via BALUG-Talk <balug-talk(a)lists.balug.org>
> To: balug-talk(a)lists.balug.org
> Cc:
> Bcc:
> Date: Sun, 23 Mar 2025 19:26:06 -0000
> Subject: [BALUG-Talk] Linux w/in MS-Windows... great chat! extended
> _______________________________________________
> list: BALUG-Talk(a)lists.balug.org help: https://lists.balug.org/help/
> unsubscribe email: balug-talk-unsubscribe(a)lists.balug.org
[View Less]
Thanks all, yes meeting last month was excellent, thanks for making it
so!
So, I was thinking, topic(s) (not that we need be limited to such) for
meeting this month (soon - Tuesday!) ... and following a bit from last
month's meeting, perhaps something along the lines:
Linux encryption and security hardware.
So, perhaps around LUKS, TPM chip, YubiKey, FIDO2, etc.
More-or-less extension of fair bit that was discussed at last meeting.
And, not too horribly redundant with presentations/talks/…
[View More]topics that
have been done at BALUG before (e.g. done LUKS presentation before,
have at least had security as topic, but don't think we've (much)
covered TPM chip, YubiKey, FIDO2, etc. before, so was thinking to mix it
up and expand it a bit more.
Also, did we have some other question(s)/topic(s) from last meeting that
we didn't quite get around to covering?
Feel free to let me know your thoughts, and I'll put together at
least something regarding (at least leading, but by no means limited to)
topics for the meeting, and will get that then updated on the web site,
and also use that for items sent to the Announce list before the
meeting. Anyway, I'm hoping/aiming to get that done and at least
initial bits out in the next couple days or so
[View Less]
Quoted snippet of Darrion Burgess from BALUG-Talk thread
https://lists.balug.org/mailman3/hyperkitty/list/balug-talk@lists.balug.org…
:
>
> to start, the easiest way to ease into is using the windows subsystem
> for linux (WSL) so you can try out different distributions using
> the windows terminal.
>
> Its even sophisticated enough to have a wayland compositor at this
> point so you can even run GUI apps from the linux containers.
Besides WSL, another good(IMHO) means of …
[View More]more easily running Linux distros
w/in MS-Windows is to use Oracle VirtualBox
Quoting its main site https://www.virtualbox.org/ :
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
VirtualBox is a general-purpose full virtualization
software for x86_64 hardware (with version 7.1
additionally for macOS/Arm), targeted at laptop,
desktop, server and embedded use.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The VirtualBox 7.1.6 platform packages dwonloads for Windows and other OS
hosts can be found at https://www.virtualbox.org/wiki/Downloads
Am concurrently attending both the in-person and Jit.si Meet virtual
meetups of the Berkeley Linux Users Group https://berkeleylug.com/ -- and
am using VirtualBox for Debian GNU/Linux 12.x 'Bookworm' as hosting OS on
a second laptop.
VBox works great and is more [ful]filling! :-D
-A
--
[View Less]
hello Dan and Michael,
Thanks again for putting on the meeting. It was wonderful to meet you both and I look forward to having more conversations with you all about the topics that interest you.
So, 2024-11-19 BALUG meeting, one of our discussion topics:
device mapper - and dmsetup(8) and related.
So, device mapper - it is used to create/manage block device, which in
turn has a specification of how it blocks are mapped to zero or more
other device(s). It operates in units of traditional 512 byte blocks,
and can handle a quite wide variety of possible ways of doing mapping.
It has at least:
cache, clone, dust, crypt, delay, ebs, error, flakey, linear, mirror,
multipath, raid, snapshot,…
[View More] striped, thin, zero
Some of the bits I mentioned. It's relatively lower level, but many
other services and the like leverage the device mapper. E.g. LVM uses
device mapper for its lower level configuration, as does cryptestup e.g.
for LUKS.
Sometimes using device mapper more directly can be quite useful. E.g.
want to test I/O issues, it has dust, error, and flakey, so one can set
up device that gives errors on I/O based upon various criteria as
specified. Can also specify different specific data to be read back -
which may be different than that which was written, etc.
Also has RAID capabilities - which brings me to example. Some while
back, had case to assist someone in coming up with a solution to
something they wanted to do. They had a fair sized hardware RAID array,
RAID-5 with 4x12TB drives. They wanted to migrate to md raid5, with
4x12TB (new) drives. And they wanted to minimize downtime to the extent
feasible. Well, easiest way to do that, would be to effectively
layer RAID-1 - at least temporarily atop that - sync - then split that
mirror. But LVM or md, etc. RAID-1 not so great for that - as they'd
all generally want to write their relevant headers on the devices, etc.,
and at best non-trivial to write that data, and block status tracking
data, etc., somewhere else - if they even support that at all.
So, solution for that? Use device mapper. Can then just directly raid1
mirror the blocks of the two devices - the original RAID-5 hardware RAID
device, and the newer replacement md raid5 software RAID device.
So, first of all, documentation, etc. There's the dmsetup(8) man page.
Pretty good, but it leaves a lot to be desired. Notably doesn't well
cover a lot of details that are or may be necessary. So next stop - and
what it quite refers to - kernel documentation. E.g.:
file:///usr/share/doc/linux-doc-6.1/html/admin-guide/device-mapper/
etc. Pretty good ... but not so great on, e.g. more complete examples,
etc. and even some relevant information turned out to be quite missing
from the kernel documentation (though hey, could probably read the
relevant source ... but that generally wouldn't be easiest way).
So next, some bits of Internet searching ... and found some good
resources, e.g.:
https://wiki.gentoo.org/wiki/Device-mapper#Mirror_and_RAID1
So, at least taken together, there was sufficient information.
So ... I earlier did a test demo run, to show how it could be done ...
but didn't save all my information/notes on that, so let me repeat that.
And a bit better this time - including log devices - that way even if,
e.g. system were to crash while the sync was in progress, could be
cleanly resumed.
So, I don't have 8x14TB in spare drives sitting around, so I'll do much
smaller with some space on /tmp and some files there and using losetup(8)
to create block devices suitable for use. So, below, my comments on
lines starting with //:
// So, don't have 8x14TB to work with, but on /tmp at present, can
// easily spare 64GiB. So will instead do ~24GiB to emulate the old and
// 4x8GiB to emulate the new. Since the "old" source is hardware
// RAID-5, will just do appropriately sized storage for that, as I don't
// have hardware RAID-5 available for that. First let's create the
// backing files and loop devices for our first 4 devices to represent
// our target drives.
# mkdir /tmp/dmr1 && cd /tmp/dmr1
# truncate -s $(expr 8 \* 1024 \* 1024 \* 1024) f{1,2,3,4} && (for n
in 1 2 3 4; do losetup -f --show f"$n"; done)
/dev/loop2
/dev/loop3
/dev/loop4
/dev/loop5
#
// And now create our software md raid5 device:
# mdadm --create --level=raid5 --raid-devices=4 /dev/md24 /dev/loop[2-5]
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md24 started.
#
// And now let's get its exact size:
$ cat /sys/block/md24/size
50276352
$
// That's in 512 byte blocks. As for bytes
$ expr 50276352 \* 512
25741492224
$
// So let's create our source device of exactly that size:
# truncate -s 25741492224 f0 && losetup -f --show f0
/dev/loop6
#
// In our actual case, source would need to be not larger than the
// target. If source were larger, we'd need to shrink the (relevant)
// data before copying, e.g. reduce the size of the filesystem a bit,
// possibly repartition slightly, etc.
// And let's confirm our sizes match:
$ cmp /sys/block/md24/size /sys/block/loop6/size && echo MATCHED
MATCHED
$
// And let's create some data on our source device:
# mkfs -t ext3 -L 24gr5 -m 0 /dev/loop6 && mount -o nosuid,nodev
/dev/loop6 /mnt && { dd if=/dev/urandom of=/mnt/urandom bs=1048576
status=none; < /mnt/urandom sha512sum && umount /mnt; }
// ...
dd: error writing '/mnt/urandom': No space left on device
6e0487bed425a7bb667d169e001415a4a18c6413ee5be56f032ebac7ea827dae9caee9ab0d0801e1d1b537eabc75cee9842de00a1089fd0afd7e7630752128aa
-
#
// Now lets set up our device mapper device, will do this with metadata
// devices to track, so, e.g. if interrupted (e.g. system crash
// or abrupt power down, or drives disconnected, etc.) can still safely
// resume after (we'll essentially ignore that our backing store is on
// the volatile /tmp - this is still just demo after all).
// Unfortunately the kernel documentation doesn't say how large these
// need to be. Probably relatively small % of the total space, I'll
// give it more than ample space on sparse file, then we can look at
// actual block usage after. So, devices 50276352 512 byte blocks,
// let's say 5% of that - that's ought be much more than enough - but
// sparse files, won't much matter.
$ echo '50276352*512/20' | bc -l
1287074611.20000000000000000000
$
// And let's round that up to 4KiB boundary:
$ echo '1287074611.2/4/1024' | bc -l
314227.20000000000000000000
$ expr 314228 \* 4 \* 1024
1287077888
$
# truncate -s 1287077888 m{0,1}
# (for n in 0 1; do losetup -f --show m"$n"; done)
/dev/loop7
/dev/loop8
#
// Will also look at status right after creation and 30 seconds later,
// and will mount right after we create it too:
# dmsetup create dmr1 --table '0 50276352 raid raid1 5 0 region_size
32 rebuild 1 2 /dev/loop7 /dev/loop6 /dev/loop8 /dev/md24' && dmsetup
status dmr1 && mount -o nosuid,nodev /dev/mapper/dmr1 /mnt && sleep 30
&& dmsetup status dmr1
0 50276352 raid raid1 2 Aa 0/50276352 recover 0 0 -
0 50276352 raid raid1 2 Aa 2046848/50276352 recover 0 0 -
#
// And a while later we have:
# dmsetup status dmr1
0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 -
#
// That field of all uppercase "A" characters tells us the RAID-1
// devices are fully synced up.
// Let's deconstruct our RAID-1 device and compare the files on the
// filesystems - which have now been copied via mirroring. Will also
// read and recompute hash of one of the files to also check that still
// matches. Also, so they don't conflict on the filesystems, will
// change the label and UUID on the "old" one, so the original data of
// that remains on the "new" target one.
# umount /mnt
// check again that we're synced before removal:
# dmsetup status dmr1
0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 -
# dmsetup remove dmr1
# tune2fs -L 24gr5.old -U random /dev/loop6
tune2fs 1.47.0 (5-Feb-2023)
# mkdir mnt-old mnt-new
# mount -o ro,nosuid,nodev /dev/loop6 mnt-old
# mount -o ro,nosuid,nodev /dev/md24 mnt-new
# cmp mnt-{old,new}/urandom && echo MATCHED
MATCHED
# < mnt-new/urandom sha512sum
6e0487bed425a7bb667d169e001415a4a18c6413ee5be56f032ebac7ea827dae9caee9ab0d0801e1d1b537eabc75cee9842de00a1089fd0afd7e7630752128aa
-
#
// And we can see that the files match and the hash matches our earlier.
// Let's do it one more time, except this time with the filesystem very
// busy while it's mounted and doing the RAID-1 sync.
// And this time we mirror from the new, to old, as the new now has
// exactly the data we want.
# umount mnt-old && umount mnt-new
# dmsetup create dmr1 --table '0 50276352 raid raid1 5 0 region_size
32 rebuild 0 2 /dev/loop7 /dev/loop6 /dev/loop8 /dev/md24' && mount -o
nosuid,nodev /dev/mapper/dmr1 /mnt && { dd if=/dev/urandom
of=/mnt/urandom bs=1048576 status=none; < /mnt/urandom sha512sum; }
dd: error writing '/mnt/urandom': No space left on device
c818fb535d037b01868252a3f2464cc17fa70b8f4cb21436a0f7d3d9c85b4783ac7e7835f47d55c588287688013b687743886f5640ddfb91ff9e2f8177dd5b38
-
#
// We check until we see it's synced:
# dmsetup status dmr1
0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 -
#
// Then we unmount, and again reconfirm it's synced:
# umount /mnt
# dmsetup status dmr1
0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 -
#
// Now we again deconstruct the RAID-1, update label and UUID on old,
// and mount and compare, and also again compute hash on one of the
// files to see that also matches:
# dmsetup remove dmr1
# tune2fs -L 24gr5.old -U random /dev/loop6
tune2fs 1.47.0 (5-Feb-2023)
# mount -o ro,nosuid,nodev /dev/loop6 mnt-old
# mount -o ro,nosuid,nodev /dev/md24 mnt-new
# cmp mnt-{old,new}/urandom && echo MATCHED
MATCHED
# < mnt-old/urandom sha512sum
c818fb535d037b01868252a3f2464cc17fa70b8f4cb21436a0f7d3d9c85b4783ac7e7835f47d55c588287688013b687743886f5640ddfb91ff9e2f8177dd5b38
-
// So, files again matched, and our hash again matches. Also, checked
// how much space was actually used for the meta devices to track RAID-1
// status:
# stat -c '%b %n' m[01]
400 m0
400 m1
#
// So that's 400 512 byte blocks, 200 KiB.
// That size apparently depends upon size of device and region_size,
// and does apparently have a limit on total maximum size it will use
// for that meta device.
So, our original scenario to replicate the RAID-5 from hardware RAID to
software (md) RAID while minimizing downtime would go about like this:
o create the new target md device, note its precise size
o stop all I/O on the old device (e.g. unmount it).
o note size of old - must not be larger than new, if necessary shrink
and/or repartition, etc. or the like as appropriate.
o create dm device as RAID-1 between the old and new, syncing to new
o return I/O to service (using the dm device in place of the old device)
o after sync has completed (may take hours to days for larger/slower
storage), as before, stop all I/O - except now on the dm device
instead of the old device. Again check/wait until it's fully synced.
o tear down the dm device
o Adjust labels, UUIDs, etc. if/as applicable on old to not conflict
with new
o return I/O to service, using new in place of old
o if/as desired, tear down or decommission old
Key advantage is relative minimization of downtime - across the hours
(or days or more) while the RAID-1 mirror is syncing to duplicate the
storage data, no downtime is needed, and the storage can be used per
usual. A bit of downtime to shuffle about, notably from old, to dm,
then to new, but other than that, things generally remain online and
actively available. Additionally, no headers or encapsulation need
happen on the old or new storage itself - that's all handled
externally, so that keeps it clean and relatively simple.
[View Less]
Ah, lovely web automation! :-)
So, lately had a little mini-project to give myself.
AT&T's "Unified Messaging" (voicemail). Wanted to "cut the cord" -
bye-bye landline - porting ye olde landline # to mobile.
But first, wanted to download all of my content from
AT&T's "Unified Messaging" (voicemail).
AT&T's "Unified Messaging" (UM/um), in addition to ye olde phone DTMF
("Touch Tone") interface to the voicemail, also has web interface.
So, web interface. Essentially works as web …
[View More]GUI interface to email in
"INBOX", messages are stored in email, and within an email item,
voicemail as .wav attachment, text attachment having transcript as body
- which will generally have empty body if it wasn't able to transcribe
it. And generally html attachment, an html version of that text
attachment.
And, "of course", Perl also has the lovely WWW::Mechanize.
So ... I got to programming. mitmproxy was also handy to figure out
some bits going on within the SSL/TLS communications between client
(e.g. web browser) and AT&T server(s).
And got the key bits of that finished up
this past Sunday. And got 'er all nicely downloaded.
$ um.att.comum.att.com: Inbox is empty. Exiting
$
That's what it outputs at the end, when there's nothing left to
download.
It also handles deleting the "email" item (message and related) from the
AT&T "INBOX" once it's successfully downloaded.
$ cd ~/.um.att.com.d/data
$ ls -A1 | sed -e 's/^.*\././' | sort | uniq -c | sort -k 1,1bnr -k 2,2
117 .eml
117 .wav
113 .txt
112 .html
$
Very nicely handles it all.
.eml is the full raw "original" email as AT&T has it in the "INBOX",
.wav files are the raw audio portion thereof,
.txt the text transcript (or no file if that part was empty), and
.html the html equivalent of that text.
Ah, I was wondering about why one less .html than .txt ... peeking
further, the .txt has:
Message too short for transcription
And that original .eml has no html part, and the .wav ... yeah, no words
in that audio.
Alas, I didn't clean out quite all the junk before downloading
everything ... and the slight mismatch makes that bit of
junk pretty easy to spot ... likewise grep on the .txt files is rather
handy.
So, the file names start with ISO date and time, which is derived from
the Date: header which is timestamp of when the end of the message was
received. Likewise that same time data is used to set the mtime on the
files. File names also contain data from Subject: and From: fields,
generally identifying caller name/number, or when not (CNID) identified
otherwise unknown caller / Identity Withheld, e.g.:
... unknown caller ... Identity withheld <unknown_caller...>
https://www.mpaoli.net/~michael/bin/um.att.com
Ah, one of these days I need to tweak Apache configuration so it
"knows", e.g. that file (and that name and location), can be handled
like plain text, not a binary. Yeah, I know there's a "magic" type
option that can read the files and make intelligent guess on that, but
that's excessive overhead for most cases - so really need to just
configure the exceptions ... down to directory or even per-file basis.
(On my to-do list ... with thousands of other items yet to be done ...
at least maybe when I get around to it).
And ... maybe even others might find it handy, or handy starting point.
Though this one was done almost / mostly as a one-off/one-shot.
Though until the number completes being ported over, very handy to still
check if anything has shown up there, and download it if so.
It might need some adjustments to handle some other email messages.
E.g. the ones from AT&T about the INBOX being nearly full. And looks
like I probably won't have need for that (nor example data to match it
to and test it on). And I didn't handle the more general email case
(which I think UM will also accept and have in "INBOX"), as I only ever
used UM for voicemail.
[View Less]
Now +Web interface:
https://www.digitalwitness.org/
On Thu, Oct 31, 2024 at 12:12 AM Michael Paoli via BALUG-Talk
<balug-talk(a)lists.balug.org> wrote:
> ---------- Forwarded message ----------
> From: Michael Paoli <michael.paoli(a)berkeley.edu>
> To: BALUG-Talk <balug-talk(a)lists.balug.org>
> Date: Thu, 31 Oct 2024 00:11:26 -0700
> Subject: [BALUG-Talk] Announcing digitalwitness.org - free public signing (witnessing) service
> Announcing digitalwitness.…
[View More]org - free public signing (witnessing) service
> At present it's in open/public Beta.
>
> At present the ssh interface is open, other(s) (most notably web) will
> likely follow in time.
>
> Let's say you have a (possibly) large digital artifact (or archive file
> of such items). Or even possibly much smaller item. Suggested approach
> is to get secure hash/digest of the item, e.g.:
> $ sha512sum digital_artifact_file
> 3ddd987f33a96b50777d15f7850d80d8e30badf12501289d28d5ee4857d62c25c2c700b6a1313cace8b128fe1e4d1ff4787d70c46e1f633e5e4589bf3f2343ba
> digital_artifact_file
> $
> Then get that hash/digest signed, e.g.:
> $ ssh -nT digitalwitness(a)digitalwitness.org --
> 3ddd987f33a96b50777d15f7850d80d8e30badf12501289d28d5ee4857d62c25c2c700b6a1313cace8b128fe1e4d1ff4787d70c46e1f633e5e4589bf3f2343ba
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAABCgAdFiEEEMoY7NqvKKtE7xkCOAS8z7K+pwUFAmcjEtkACgkQOAS8z7K+
> pwWH3A//cDtGIHokwF+GEvKnFFE+Cw2hiPVTe0PkBPuymGnLPAC9Um7YkVt1vP8u
> ZZhGOVrAivFV75gVASszq6Au4OKY/2GOO0+SMkVaxd9VzprxBH+j8BVixGiDvU3L
> k9JbSzzLIKNoTpPptWbphPoEO6cE4WSm1HubMFeODE7znQeVfk5UlpIEE/7XcT0r
> lmwnUaoSwhZUL5HxWE3Pt6x4QSTOin38DHCS55LRfDwHC5og5fc7eiC9TSqr/kVB
> goyxYg/lCNrKa7HQto03zJ2ZctZnsNj5n81WhPYzBwlpGfb1T/3htqP17PCLJ19W
> amWC+kn+9vR5cwGwuEnBxfOlE//CI7d3gWXSSsvHCDhX49Drh1m7OIw4I9vnYYcV
> h9fgajSR1SAMiQwo7uQjrmByZnUdXCTfRJ6ywBfuKaZdBX1Y+AGD8cUWc5qnCwrR
> rX4bznL2JZhrwrNh6abmDcyqRzMmXr+jM7WqZiJbcxkhoJZ78c+UtCR6ETreVui0
> Ah7a7HSAuZ6E3dek1oBzA5zTV/bpZHEcdz8UcRYWryEF7wtzJ0gi4SXsWLLGq/LA
> LaSOPgcA8KMB3A5NtSYdn0ax/MEao/r/spNrxiQI5jKTYBuoI3qSA7I8Pp1qIkkY
> DS/XHzT2CNVw6T/YxvqrhqvSPfaQkQWy0lse6WVIG3CfW7Bp0Pk=
> =5mCo
> -----END PGP SIGNATURE-----
> $
> For more information, help, public signing key, etc., just run without
> any "command" or arguments thereof, e.g.:
> $ ssh -nT digitalwitness(a)digitalwitness.org
> And that will provide one with much relevant information, and including
> important details (e.g. the signed data is signed without first adding
> any implicit newline on the end, though one can explicitly provide that
> if desired).
[View Less]