We should be running those tests ideally with packaged versions.
I have a enhancement to "sinksh selftest" in mind that allows testing of:
- IMAP create mail
- IMAP sync
- Send mail and receive in IMAP inbox
- create and sync via CalDAV/CardDAV
The tests don't have to be extensive, just the very basics.
We should then run these tests with a couple of supported services directly in the CI before publishing (this will require test accounts for that purpose).
Another benefit will also be that we then have a standard testbench for people to tests compatibility with whatever their service of choice happens to be,
and that should make is easier for us to figure out whether we have a compatibility issue that we have to fix on our end, or whether the service just happens to be incompatible somehow.