Mailman 3

Debian:
  10 Buster
    LTS EOL 2024-06-30
    First stable to support mailman3
    last to support mailman 2
  balug-sf-lug-v2.balug.org (balug.org)
    mailman3:
      run mailman command with env LANG=en_US.UTF-8, as user list (by default on Debian), probably umask 022
      $ expand -t 4 < ~root/bin/Mailman
      #!/bin/sh
      umask 022 &&
      unset LC_ALL &&
      LANG=en_US.UTF-8 export LANG &&
      case "$(id -n -u)" in
          list)
              umask 022 &&
              exec mailman "$@"
          ;;
          *)
              umask 022 &&
              exec sudo -u list \
              mailman "$@"
          ;;
      esac
      $ 
      # /usr/share/mailman3-web/manage.py # use no options/arguments for help
      # /usr/share/mailman3-web/manage.py createsuperuser
      # /usr/share/mailman3-web/manage.py changepassword login_name_of_user
      databases:
      # sqlite3 /var/lib/mailman3/data/mailman.db
      # sqlite3 /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db # or /var/lib/mailman3/web/mailman3web.db
        backup, e.g.:
        sqlite3 /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db ".backup '/var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.BACKUP.db'"
      created list: balug-test3@lists.balug.org
      https://lists.balug.org/mailman3/
      https://lists.balug.org/mailman3/admin/
      Y posting: X-Archive: no if header is set to value no (regardless of case), doesn't archive
      Y posting: X-No-Archive: if is header present, doesn't archive
      Y posting: if X-Archive and X-No-Archive conflict, or multiple conflicting X-Archive present, appears to not archive unless
        only X-Archive header(s) are present and the first does not have value of no (case insensitive)
      see also: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/archives.html
      Y (Almost?) Everything you wanted to know about Mailman 3 imports/archives (but were afraid to ask?):
        Various things happen between import and "export" ("Download" of archives).
        import appends, existing archive items aren't dropped
        Note also that there may be a raw(er) form in the database.
        some analysis (notably mtime, etc.) implies that the imported data is stored in:
          /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db (sqlite3/postorious database: # sqlite3 /var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db)
          somewhere around, e.g. here:
          sqlite> SELECT sql FROM sqlite_master WHERE tbl_name = 'hyperkitty_email' AND type = 'table';
          CREATE TABLE "hyperkitty_email" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "message_id" varchar(255) NOT NULL, "message_id_hash" varchar(255) NOT NULL, "subject" varchar(512) NOT NULL, "content" text NOT NULL, "date" datetime NOT NULL, "timezone" smallint NOT NULL, "in_reply_to" varchar(255) NULL, "archived_date" datetime NOT NULL, "thread_depth" integer NOT NULL, "thread_order" integer NULL, "sender_id" varchar(255) NOT NULL REFERENCES "hyperkitty_sender" ("address"), "thread_id" integer NOT NULL REFERENCES "hyperkitty_thread" ("id"), "sender_name" varchar(255) NULL, "mailinglist_id" integer NOT NULL REFERENCES "hyperkitty_mailinglist" ("id"), "parent_id" integer NULL REFERENCES "hyperkitty_email" ("id"))
          sqlite> select * from hyperkitty_email WHERE id = 54;
          54|1720513097.9ExYtTWvb7bqxG6Y@balug-sf-lug-v2.balug.org|PTLURYJSWOCAWEZ5WDK5PBHHCQ3R66HW|test - ignore - archive import testing|body 1720513097
          postmaster@example.com
          @example.com
          last line of body
          |2024-07-09 08:18:17|0||2024-07-09 08:18:17|0|0|test-trnzipzntiaotvhgjmee@balug.org|49|test-trnzipzntiaotvhgjmee@balug.org|2|
        E.g. some inconsistent email address obfuscation between GUI display and Download (export),
        hints that there's lower level raw(er) in there somewhere - maybe that can also be usefully accessed?
        So, between import and Download, at least some headers and bodies are altered:
          To: is unconditionally set to only the list
          CC: is unconditionally stripped.
          Various headers may be added/dropped/altered, reordered
          plain email with no MIME may get converted to Content-Type: multipart/mixed, even if it only then has the one part
          X-No-Arcive: unconditionally stripped (and then handled as if not present)
          X-Archive: unconditionally stripped (and then handled as if not present)
          Other conversions/changes, etc. may occur.
          In any case does end up imported into archive
      Y Do users have any ability to access (and backup) entire list's archive as (e.g.) .mbox format file (like Mailman 2 can do)?
        On archive page there's a Download drop-down to download entire archive, no auth needed for public archives,
        URL also works (e.g. with curl), e.g.:
        https://lists.balug.org/mailman3/hyperkitty/list/balug-test3@lists.balug.org/export/balug-test3@lists.balug.org.mbox.gz
      N Are such archives in original unobfuscated (e.g. email addresses) format?
        Although headers aren't obfuscated, bodies are subject to some @ --> (a) email obfuscation,
        Web GUI display of archive does similar, but not even same matched set!  E.g. body munging:
          displays on web GUI:
          obfuscated:
          postmaster(a)[127.0.0.1]
          postmaster(a)example.com
          unobfuscated:
          <postmaster@localhost>
          @example.com
          @localhost
          postmaster@localhost
          And in what it allows to be downloaded via web, ugh,
          slightly worse, but not even the same set!:
          obfuscated:
          <postmaster(a)localhost>
          postmaster(a)example.com
          postmaster(a)localhost
          unobfuscated:
          @example.com
          @localhost
          postmaster@[127.0.0.1]
          And same results whether posted via SMTP/mail, or GUI.
      Y imports appear to apply same @ --> (a) email address obfuscation (at least compared to GUI display and Download)
        ? is actual data in database unadulterated?
      Y how do we remove posting or archive?
        Looks like simplest way is from admin web interface, e.g.:
        "Delete this message" link, e.g.: https://lists.balug.org/mailman3/hyperkitty/list/balug-test3@lists.balug.org/message/DIQNAQZNUJSVN7CRA2NZ2JLQ7UCFFSTM/delete
          Note that that's distinct from "Delete this thread" links.
        lock (to avoid race conditions)
        can delete from archive (e.g. in database), e.g.:
        sqlite> DELETE from hyperkitty_email WHERE id = 55;
        that deletes an individual email - may or may not delete associated data (e.g. attachments).
        ? Note that it doesn't immediately disappear all related information on web pages, such as "RECENTLY ACTIVE DISCUSSIONS", though the links to the email message will no longer work, and seems after a while these clean themselves up (likely some periodic scheduled job or the like)
        ? Something like this may also be useful (unverified/untested):
          in a django management shell, you could try something like
          import datetime
          from hyperkitty.models import Email
          for msg in Email.objects.filter(mailinglist_id=X):
            if msg.date &lt; datetime.datetime(Y, M, D, tzinfo=datetime.timezone.utc):
              msg.delete()
          where X is the id of the mailing list you want to prune, or if you
          wanted to do all lists, just use Email.objects.all() instead of
          filtering on mailinglist_id.
          And, Y, M and D are integer Year, Month and Day for the cutoff.
          Note that something like
          from hyperkitty.models import MailingList
          for ml in MailingList.objects.all():
               print('{}: {}'.format(ml.name, ml.id))
          will print all the list names and their ids.
      ? How do we rename a list (including its archive)?
        Doesn't appear to be well covered, see:
        https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/Q3YHKZKUALBWIESNOQLRBFRNJ6F3O77U/
        Let's attempt - rename of balug-test4@lists.balug.org to balug-test5@lists.balug.org
        Y sanity check balug-test4@lists.balug.org still working
      Y allow postings but prevent additional archiving, this, e.g.:
        https://lists.balug.org/mailman3/postorius/lists/balug-test2.lists.balug.org/settings/archiving
          change setting
          from: Archive policy: Public archives Active archivers: (checkbox checked) hyperkitty
          to: Archive policy: Public archives Active archivers: (checkbox unchecked) hyperkitty
          suffices to prevent new postings from being archived (and if reactivated, doesn't cause retroactive archiving)
        ? how does that interact with, e.g.:
          https://lists.balug.org/mailman3/admin/hyperkitty/mailinglist/4/change/
            change settings
            from: Archive policy: public 
            to: Archive policy: never
            ? will that prevent additional archiving while allowing posting.
            ? Are both of those needed, or will one suffice?
            ? What exactly is the difference?
      Y import adds (doesn't clobber existing)
      ? localpart+tag@ email addresses supported?
        N by MTA in current configuration?
          Doesn't look super easy to add at current, will probably defer and reevaluate later
        Y by Mailman 3
          Looks like Mailman 3 itself probably handles it fine and doesn't care (treats it as local part, including part after +)
      Y how do we export roster of members?
        E.g.: # (for list in $(Mailman lists -a -q); do echo "$list:"; Mailman members --role any "$list"; done)
        or via URLs, e.g. https://lists.balug.org/mailman3/postorius/lists/balug-test3.lists.balug.org/csv_view/
      Y don't reject empty body (e.g. mailman commands don't require body to be present)
      DMARC - tested (mostly with @yahoo.com), seems these are best compromise settings:
        DMARC mitigation action: Wrap the message in an outer message From: the list.
        DMARC Mitigate unconditionally: No
        DMARC rejection notice:
        DMARC wrapped message text:
select documentation:
https://docs.mailman3.org/en/latest/install/virtualenv.html
https://docs.mailman3.org/projects/hyperkitty/en/latest/install.html
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/postorius.html
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/docs/archives.html
https://docs.mailman3.org/projects/mailmanclient/en/latest/
https://docs.mailman3.org/projects/postorius/en/latest/
https://wiki.archlinux.org/title/Postorius
https://www.balug.org/doc/
https://www.balug.org/doc/mailman3-doc/
https://www.balug.org/doc/mailman3-full/
https://www.balug.org/doc/mailman3-web/
https://www.balug.org/doc/mailman3/html/README.html#table-of-contents
https://www.balug.org/doc/mailman3/html/src/mailman/model/docs/mlist-addresses.html
https://www.balug.org/doc/python3-django-hyperkitty/html/
https://www.balug.org/doc/python3-django-postorius/html/
https://www.balug.org/doc/python3-django-postorius/html/deployment.html
Packages:
apache2
exim4-daemon-heavy
lynx
mailman3
mailman3-doc
mailman3-full
mailman3-web
python-django-common
python3-django
python3-django-allauth
python3-django-appconf
python3-django-compressor
python3-django-extensions
python3-django-filters
python3-django-gravatar2
python3-django-guardian
python3-django-haystack
python3-django-hyperkitty
python3-django-mailman3
python3-django-picklefield
python3-django-postorius
python3-django-q
python3-djangorestframework
python3-mailman-hyperkitty
python3-mailmanclient

notes/keys:

• n - No / Not done
• i - in progress or incomplete
• ir - added redirects for main list page + each lists's main page
• y - Yes / done
• - N/A Not Applicable
• lock old: comment out of /etc/aliases
• pending: handle pending as warranted (e.g. held pending moderator approval)

notes/keys:

• n - No / Not done
• i - in progress or incomplete
• y - Yes / done
• - N/A Not Applicable

Various notes on migration, etc.
See also the earlier: https://www.wiki.balug.org/wiki/doku.php?id=balug:mail_and_lists
Debian:
  10 Buster
    last to support mailman 2, also (first to?) supports mailman3
    LTS EOL 2024-06-30
  balug-sf-lug-v2.balug.org (balug.org)
    Location of list configuration in Mailman 2
    /var/lib/mailman/lists/*/config.pck
    Location of list archives in Mailman 2
    /var/lib/mailman/archives/private/*.mbox/*.mbox
    Location of the bin/ commands in Mailman 2
    /usr/lib/mailman/bin
    run commands as user list (on Debian):
    Y How do we do "locking" or list suspensions/shutdowns to avoid race conditions?
    removing Mailman 2 aliases from /etc/aliases (and running newaliases) prevents posts, etc. to corresponding Mailman 2 lists, etc.
    Create the target list on Mailman 3, e.g.: foo-list@example.com, via either:
    https://lists.balug.org/mailman3/postorius/lists/new/ # or: Mailman create foo-list@example.com
    Y if same list name exists on mailman2 and mailman3, and post is sent to list, which does it use?: Mailman 2 if still in /etc/aliases, else Mailman 3
    Migrate the list configuration from Mailman 2 to Mailman 3 by running the following command (use the appropriate file for .pck and .mbox in the below)
    # Mailman import21 foo-list@example.com /var/lib/mailman/lists/*/config.pck
      Ugh.  mailman import12 may also fail, e.g.:
          File "/usr/bin/mailman", line 11, in <module>
            load_entry_point('mailman==3.2.1', 'console_scripts', 'mailman')()
          File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
            return self.main(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
            rv = self.invoke(ctx)
          File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 69, in invoke
            return super().invoke(ctx)
          File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
            return _process_result(sub_ctx.command.invoke(sub_ctx))
          File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
            return ctx.invoke(self.callback, **ctx.params)
          File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
            return callback(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
            return f(get_current_context(), *args, **kwargs)
          File "/usr/lib/python3/dist-packages/mailman/commands/cli_import.py", line 64, in import21
            pickle_file, encoding='utf-8', errors='ignore')
        ModuleNotFoundError: No module named 'Mailman'
        see also, e.g.: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/JEPMB3HW4FI57EUMOST4L7BD2ILIIS3P/
        and this seemed sufficien to work around the issue:
        executed as id list:
        echo 'bounce_info = {}' > reset_bounceinfo.py
        config_list -i reset_bounceinfo.py balug-talk
      Y import21 does mostly bring over the following per-member options from Mailman2 to Mailman3:
      Mailman 2 <--> Mailman 3 one-to-one mappings where present:
      Mailman 2                    Mailman 3
      delivery enable/disable      Delivery status Enabled/Disabled (preserved)
      digest no/yes                Regular / Plain Text Digests / Mime Digests (preserved, see plain below)
      mod                          Hold for moderation (preserved)
      user_options (bit mapped)
        2 not metoo                Receive own postings No/Yes (preserved)
        4 ack                      Acknowledge posts Yes/No    (preserved)
        8 plain                    (if digest mode enabled, sets type above - preserved)
       16 hide                     Hide address Yes/No         (preserved)
      256 no dupes                 Receive list copies No/Yes  (preserved) 
      ? note that some attributes/settings may not be imported in some circumstances, these seem to be the cases:
        ? if nomail is set, no options may end up set (e.g. none of enabled/disabled, yes/no set)
        ? if no options are set, may end up with just Delivery status Enabled, Delivery mode Regular, and no other options set at all (similar to above)
        ? if digest is set and no other options set, may end up with just Delivery status Enabled, Delivery mode Mime Diegest, and no other options set at all (similar to above)
    Migrate the list archives from Mailman 2 to Mailman 3 by running the following command:
    Note that the import and (re)indexing need to be run as www-data on Debian
    $ (unset LC_ALL && LANG=en_US.UTF-8 export LANG && /usr/share/mailman3-web/manage.py hyperkitty_import -l foo-list@example.com --since 1968 /var/lib/mailman/archives/private/*.mbox/*.mbox)
      Notes: on mbox import format as processed by hyperkitty_import:
        The leading From (envelope from, not header From:) can be entirely empty - just "From " and it accepts that (apparently doesn't use it other than to determine where that mail item starts)
        So, it apparently just used Date: field for time and timezone information, and From: header (or possibly similar) to identify sender email and name.
        Also note on import and database - though it does save raw unaltered body, only some header data is saved and even some/much of that isn't saved in original raw form.
        Also, database can have line(s) that start with "From " in body (content), but they can't be imported that way (as in mbox format, that would mark start of a mail item)
    rebuild the index for this list:
    $ (unset LC_ALL && LANG=en_US.UTF-8 export LANG && /usr/share/mailman3-web/manage.py update_index_one_list foo-list@example.com)
    may need to do, e.g.:
    $ (unset LC_ALL && LANG=en_US.UTF-8 export LANG && /usr/share/mailman3-web/manage.py update_index -s 1970-01-01T00:00:00)
    Delete Mailman 2 list:
    $ /usr/lib/mailman/bin/rmlist foo-list
    Y Old Mailman 2 archives remain after Mailman 2 list is removed with rmlist, unless the -a option is given (in which archives are also removed).
    ? other bits, e.g. informational page(s) we should update along the way?
    Y prepare changes of redirection/content on http[s]://lists.balug.org/ for migration & legacy, etc.
    ? prepare replacement reporting, e.g. monthly stats.
      Existing Mailman 2 have:
        /etc/cron.d/local-lists
        /var/local/sbin/lists_monthly2
        to be:
        ? /var/local/sbin/lists_monthly3
    Note: import will append, not replace
        At least partially lock out old (remove aliases from /etc/aliases and run newaliases)
      ? backward compatibility with old links, e.g. archives, do we do as noted/suggested
        here: https://docs.mailman3.org/en/latest/migration.html and here: https://mail.python.org/pipermail/security-sig/
        the files to be customized would be the *.html files under /var/lib/mailman/archives/private/list_name/
      ? update links on, e.g. https://www.balug.org/#Lists
          ? Announce
          ? Talk
          ? Admin
          ? Test
    ? Is it at all reasonably feasible to mass export/import user preferences, passwords, etc.?
      There isn't 1-to-1 mapping between Mailman 2 and Mailman 3 on user logins, preferences, etc.
      some key differences:
        Mailman 2 logins are per-list
        Mailman 3 logins aren't required and if present are per user
        Mailman 3 can have multiple distinct emails associated with same user account, Mailman 2 can't do that
      ? What bits do have one-to-one mappings on per-list basis?
        Mailman 2 has (approximately?) this:
      Mailman 2 <--> Mailman 3 one-to-one mappings where present:
      Mailman 2                    Mailman 3
      delivery enable/disable      delivery enable/disable
      digest no/yes                digest no/plain/MIME/summary (see also Mailman 2 plain/MIME)
      user_options (bit mapped)
        2 not metoo                Receive own postings no/yes
        4 ack                      Acknowledge posts yes/no
        8 plain                    No equivalent except if digest, can be set to plain or MIME
       16 hide                     Hide address yes/no
      256 no dupes                 Receive list copies no/yes
      Other bits we'll want to map as/where feasible (not per list):
      usernames
      passwords
      example files:
      Mailman 2:
        /var/lib/mailman/archives/private/balug-test2/pipermail.pck
        /var/lib/mailman/lists/balug-test2/config.pck
        $ /usr/lib/mailman/bin/dumpdb --pickle /var/lib/mailman/lists/balug-test2/config.pck
          provides users (members), their passwords, and options. 
    Before migrating list archive, check, clean, etc., see: https://docs.mailman3.org/en/latest/migration.html#id2
    Mailman 3 gravatar information leakage
      HyperKitty archive pages are (at least by default) served up with links to gravatar.com
      (looks like) they leak information to gravatar.com - looks like each link is (probably) a hash of the user/member's email address
      Earlier versions of Mailman 3 don't have a "fix" for that.
      Later versions allow it to be disabled in configuration:
        https://docs.mailman3.org/projects/mailman-web/en/latest/settings.html#mailman_web.settings.mailman.HYPERKITTY_ENABLE_GRAVATAR
        Looks like that went in with HyperKitty 1.3.4: https://docs.mailman3.org/projects/hyperkitty/en/latest/news.html#news-1-3-4
        Debian 10.13 --> 11.10 gave us upgrade python3-django-hyperkitty:all 1.2.2-1+deb10u1 --> 1.3.4-4
        So, should now be able to fix the gravatar issue (and was earlier fixed and confirmed fixed).
    Mailman 3 information leakage?
      After fixing the gravatar issue and reasonably checking again, seems there's no longer information leakage present (yay!)
    Minor HyperKitty archive bugs on Debian 11 Bullseye
      From archive page, e.g.: https://lists.balug.org/mailman3/hyperkitty/list/balug-test@lists.balug.org/
      clicking the Threads by month dropdown doesn't do anything.
      After clicking All Threads which goes to, e.g. https://lists.balug.org/mailman3/hyperkitty/list/balug-test@lists.balug.org/latest
        once there, clicking on a year shows the months under year, but those immediately collapse before one can click on any of those months,
        this seems to be the case regardless of browswer.
      These issues weren't present on Debian Buster 10, and hopefully are fixed by Debian 12 Bookworm
    balug-announc@lists.balug.org set users with explicit moderation of hold (imported from Mailman 2) to List default (discard)
      as user list:
      (umask 022 && unset LC_ALL && LANG=en_US.UTF-8 export LANG && sqlite3 /var/lib/mailman3/data/mailman.db "UPDATE member SET moderation_action = NULL WHERE list_id = 'balug-announce.lists.balug.org' and moderation_action = 0;")
    templates:
      from mailman3 package: /usr/lib/python3/dist-packages/mailman/templates/en/*.txt
      formatting of templates is not consistent, see also:
        https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/templates.html
      for some templates (e.g. list:user:notice:welcome)
        empty line starts a new paragraph
        Adjacent non-empty lines that don't start with space may be joined
          partial workaround - appears if the line starts with a space it won't be joined to the preceding, however that leading space isn't stripped
        long lines may be folded - mailman documentation says at 72 characters but appears this happens at 70
      Debian:
        see also:
          /usr/lib/python3/dist-packages/mailman/utilities/string.py
          /usr/lib/python3/dist-packages/mailman/app/notifications.py
        it shows folding at 70
        looks like wrap is applied to (at least):
          list:user:notice:welcome (confirmed)
          list:user:notice:goodbye
          list:user:notice:warning
          list:member:digest:masthead (confirmed)
        And does //not// fold:
          list:member:regular:footer (confirmed)
        looks like also for those templates where wrapping is applied,
        source comments imply if the paragraph starts with whitespace, wrapping won't be applied to the paragraph - but seems the comments lie,
        need each line to start with whitespace (which is preserved) to not have the line folded, also prevents it from being joined to preceding line.
      for other templates (e.g. list:member:regular:footer)
        appears templates may be preserved much closer to unchanged, e.g. lines not joined nor folded, just variable substitutions applied
      'da Internet sayeth:
        https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/RWHXDB7D562YSWP67Y5CVQYDOFCLW6TM/
        Some but not all templates are wrapped at column 70 by the function at
        https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/utilities/string.py#L102
        Those that are wrapped are:
        list:admin:action:subscribe
        list:admin:action:unsubscribe
        list:admin:notice:pending
        list:user:notice:goodbye
        list:user:notice:hold
        list:user:notice:no-more-today
        list:user:notice:probe
        list:user:notice:refuse
        list:user:notice:warning
        list:user:notice:welcome
        Others are not. For those that are wrapped, there are techniques to
        avoid wrapping. In particular, lines beginning with whitespace aren't
        wrapped
  VM: debian10mailman23
select documentation:
https://docs.debops.org/en/stable-2.1/ansible/roles/mailman/mailman2-migration.html
https://docs.mailman3.org/en/latest/faq-migration.html
https://docs.mailman3.org/en/latest/migration.html
https://github.com/jmuhlich/mailman-archive-migration
Packages:
mailman