Nextcloud is not a suitable etherpad replacement for shared documents
Closed, ResolvedPublic

Description

Today when I went to edit the Plasma 5.20 release notes which are hosted on share.kde.org at https://share.kde.org/s/yg4FCeHPFemnzbb, I discovered that they had been vandalized. @jriddell was able to revert the vandalism, but the last version available was months old and the extensive edits I had made last month were lost. I spent hours on that, and having to re-do that work from scratch is not really giving me the warm fuzzies today.

This isn't the first time something like this has happened and caused me to lose work. It seems to be a regular thing.

I know that the previous etherpad-based solution was retired for various reasonable-sounding reasons, but its replacement is a downgrade.

Can we brainstorm paths forward here?

ngraham created this task.Sep 3 2020, 3:42 PM
Restricted Application added a subscriber: sysadmin. ยท View Herald TranscriptSep 3 2020, 3:42 PM
ngraham renamed this task from Nextcloud is not a suitable replacement for etherpad for shared documents to Nextcloud is not a suitable etherpad replacement for shared documents.Sep 3 2020, 3:50 PM
ngraham updated the task description. (Show Details)
ognarb added a comment.Sep 3 2020, 3:55 PM

The advantages provided by Nextcloud are:

  • Restrict the access to certain note for only a few peoples (for example the board members have private notes)
  • Already included out of the box in our Nextcloud install so we don't need more maintenance

But the big disadvantage is that I and others got often problems with concurrent writes and change not saved at all losing hours of work.

I don't know how long etherpad stores the history of each edit, so I doubt it can help against vandalism. In terms of authentification and authorization, etherpad-lite has plugins for oauth2 and I could modify them to have some protected notes for the board or we just let the board continue to use Nextcloud. And in terms of maintenance, I don't mind taking care of that.

With regards to Etherpad Lite, we can always make use of OwnPad, which is a plugin to Nextcloud to embed Etherpad Lite.

This puts Nextcloud in charge of authentication and authorisation, and solves our problem of urls leaking confidential information (as was happening with Board pads) that we had before.

Actual storage of documents does remain with Etherpad sadly though, so their content would be inaccessible to WebDAV client users and we would still have the same issues we hit previously with how poor the Etherpad upgrade path was.

Ideally I'd prefer something else but it seems that there isn't much in the way of collaborative online editors, unless you count Collabora Office (which we already have) and Only Office (which is proprietary)

I don't know what the exact feature requirements would be, but https://demo.codimd.org/features might be another option.

I've had a look into CodiMD and alas it doesn't support any concept of groups - which are necessary for sharing documents in a scalable fashion for us.

dfaure added a subscriber: dfaure.Sep 9 2020, 1:00 PM

My experience with editing files on share.kde.org yesterday: constant conflicts, can't rename the file, *lost my changes*, can't give out a link to the file (it lists the directory), can't copy/paste the filename at the top when opening the file. Next time I'm using the wiki.

duffus added a subscriber: duffus.Sep 9 2020, 1:29 PM

My experience with editing files on share.kde.org yesterday: constant conflicts

Were you using the Markdown/text editor or the full blown office one?

I've not see issues with markdown, and found it much better than the web version of libreoffice for collaborative work

can't rename the file,

in the files view you can rename it, use the ... menu to the right to carry out various things like rename, delete

*lost my changes*,

can't give out a link to the file (it lists the directory),

This can depend a bit on the permissions on the file but for instance with markdown if you use the share option while in the document that will link directly to the document (if you are allowed to)

can't copy/paste the filename at the top when opening the file. Next time I'm using the wiki.

yeah you need to use the sharing either via link or to the specific user if you want others to then access it

I just spent a few minutes doing test edits on the KF6 document (which was the one dfaure had problems with) and I hit *lots* of issues. Edit conflicts, changes not saving, browser console log saying there were conflicts but the conflict UI not appearing, the conflict UI making me choose between two versions and then the document not looking like the version I selected... I don't know if it's a general problem or if something got corrupted with this particular file.

In addition, the nextcloud settings says cronjobs couldn't run because "Your data directory is invalid Ensure there is a file called ".ocdata" in the root of the data directory", but that file does exist, and running the cronjob manually says it succeeded. ?!

maspons added a subscriber: maspons.Sep 9 2020, 2:57 PM
dfaure added a comment.Sep 9 2020, 3:55 PM

Were you using the Markdown/text editor or the full blown office one?

Not sure. I think Markdown given the extension.
https://share.kde.org/apps/files/?dir=/Community Notes/Akademy filter for KF6, choose the 2020 file.

in the files view you can rename it

Yes, in theory. But every time I tried (to remove the additional ".md" given the ".md.md" name), the response was an "error renaming file".

can't give out a link

I mean: try giving out a link to the editable file above by copy/pasting the URL in your browser. It just doesn't work.
After going into menus and things, I found https://share.kde.org/f/1917954 which kind of works. Weird stuff.

I am pissed at losing my changes (even though they appeared to be saved yesterday), I will never use this editor again.

David and Nicolas's experience mirror my own--even when using the Markdown editor, which is indeed better, but still not good. I hit a new bug with the markdown editor yesterday: pasting text that includes both a link and non-link content ruins the link by stripping out underscores.

The whole thing feels like pre-alpha quality software. It is not suitable for production use IMO.

karli added a subscriber: karli.Sep 17 2020, 8:58 AM

Hmm. Nextcloud Text is working fine here and at many other instances on the internet. There might be a configuration issue here. Could you create an issue here? https://github.com/nextcloud/text/issues
Thanks.

Cheers
Frank

From what I have heard, this was some sort of config issue and the Nextcloud Markdown editor should work fine now without any conflict problems. Is that correct? If yes, do we still want to keep this task open or can it be closed?

We haven't yet made any changes to the setup on our side, so it remains to be seen what is triggering the problem.

I just tried the KF6 document again (/Community Notes/Akademy/Akademy_2020_KF6_BoF.md.md) and I had the same problem. I opened the file, deleted 4 characters, closed it. Opened it again, it still said the file wasn't saved ("last edit 20 days ago*" with the asterisk). About a minute later it said there was a conflict, "file was modified outside the editor", showed me the two panels to pick what version to keep, dev console printed "Conflict during file save, please resolve" and an HTTP request returning 409 about 2 times a second. Maybe there's something wrong with the internal state of this particular document?

I have discovered that when the editor tries to auto-save, the HTTP request to do so takes about 4 minutes to respond. Turns out it's doing thousands of LDAP queries. This probably only affects files in the global shared folder, which is why everything is fine when editing a document in your home folder.

We hit that issue earlier with the Collabora Office integration, although that only affected public share links to those documents.
I've reported https://github.com/nextcloud/server/issues/23101 as this is really a bug in the Nextcloud LDAP integration.

As a side note, it appears we're not receiving a response on this matter from the Nextcloud Text maintainers, so not hugely impressed at the moment with the level of support we're receiving here from Nextcloud.

On another matter, i'm not fully convinced that the issue lies solely with just this set of LDAP specific bugs as I recall in limited testing we did earlier that multiple people typing on the same line caused issues as well.

Today I tried creating a new document in the shared folder. Without even having any other person editing it, it was just me writing, the file cleared itself two times, loosing the content. I was expecting this and I had copied the content elsewhere, so it has not damaged me this time. But currently it seems like the shared folder is impossible to use reliably.

karli added a comment.Oct 4 2020, 8:08 AM

I don't understand this problems on the KDE instance. Can we arrange a debugging session with a Nextcloud developer and an KDE sysadmin? And if yes, we should coordinate by email. Can the right contact person send me an email? Thanks

Yes we certainly can arrange a debugging session.

Please have them drop me a line via bcooksley@kde.org and we can go from there. Please note that my timezone is UTC+13 however and I'm limited to being around during the European mornings which may complicate things.

Just as an update to this - i've now followed up with the Nextcloud folks regarding the status of investigating this on their side.

Last weekend myself and some other Sysadmins conducted an evaluation of our options in this regard as this situation has dragged on without any resolution.

It appears that a substantial proportion of the issues we have with performance here are related to the LDAP backend. Given we are moving away from that towards OAuth2, we're investigating our options (there is a third party plugin that looks promising) and will be discussing them with the Nextcloud developers.

It should be noted that it appears that GitLab are looking at options in this space as well (https://gitlab.com/groups/gitlab-org/-/epics/6088) - so wiki pages on GitLab in the future may be sufficiently capable to meet our needs here (and even though they're looking at TipTap/Prosemirror as well, it is likely they will use websockets rather than polling which is what Nextcloud does)

As an update to this, over the weekend Sysadmin completed deployment of an initial testbed system that uses ONLYOFFICE in place of the previous Collabora Office.

While we've yet to deploy the necessary software to connect it with MyKDE (via OAuth2) which we expect to complete in the next 24-48 hours all going well, we'd certainly welcome people to test the new environment - especially if they're able to get a number of people to view/edit the same document simultaneously.

Please let us know if you'd like a public document to use for testing.

The switch to MyKDE over Identity should resolve the issues we were having with changes being lost / not saved / etc as this was principally due to a series of limitations in the Nextcloud LDAP support - which fell severely short of Sysadmin expectations. The switch to ONLYOFFICE should deliver a document editor that has very good scalability while still being very responsive locally.

Unfortunately these changes won't resolve one of the principal issues with it being difficult to give out quick links for people to jump in the same document as you (this is a Nextcloud limitation alas) but creating a public share link is a good workaround for this at the moment.

WIth the launch of Collaborate (collaborate.kde.org) can you please evaluate that to see if any of the previously mentioned problems still remain?

ngraham closed this task as Resolved.Jul 27 2021, 5:40 PM
ngraham claimed this task.

It does seem much better now. ๐Ÿ‘

Thanks for confirming Nate, much appreciated.