Use URLs in multiget requests as returned by the Server
Needs RevisionPublic

Authored by rhaferkamp on Nov 16 2017, 1:27 PM.

Details

Summary

CalDav Servers usually return relative URLs to Events in their
multistatus responses. KDAV up to now converted them into absolute URLs
when creating followup "calendar-multiget" request. It seems that some
server implemenations can't cope with that behavior and return and empty
result set for such queries.

Note: As this changes the format of the "remoteid" this requires and
addtional small fix in kdepim-runtime

Partially fixes: https://bugs.kde.org/show_bug.cgi?id=386985

Diff Detail

Repository
R490 KDAV Library
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
rhaferkamp created this revision.Nov 16 2017, 1:27 PM
Restricted Application added a project: KDE PIM. · View Herald TranscriptNov 16 2017, 1:27 PM
Restricted Application added a subscriber: KDE PIM. · View Herald Transcript

Note: in order to fully work https://phabricator.kde.org/D8844 is need in addtion to this.

knauss accepted this revision.Nov 18 2017, 10:54 PM
knauss added a subscriber: knauss.

looks fine

This revision is now accepted and ready to land.Nov 18 2017, 10:54 PM
rhaferkamp updated this revision to Diff 22798.Nov 23 2017, 8:07 AM

Updated commit message

This revision was automatically updated to reflect the committed changes.

This commit breaks caldav on kolab servers.
I have reverted locally and my caldav calendars work again.

This commit breaks caldav on kolab servers.
I have reverted locally and my caldav calendars work again.

Same here.

looks like there should be more tests to make sure, that a change don't break things. This sounds like there is only a small fine line that a caldav is working...

This commit breaks caldav on kolab servers.

Would you mind giving some more details here? Unfortunately I don't have access to a kolab instance. I.e. does it only break for existing kolab resources or does it also break if you recreate the kolab resource from scratch? If it's just the former that might give a hint what the bug is.

I have reverted locally and my caldav calendars work again.

It also failed for a newly created instance for me. As a result I see all my calendars, but they are all empty.
Sorry, I don't have much more details yet, it failed at work here, where I have limited time to debug this.

It also failed for a newly created instance for me. As a result I see all my calendars, but they are all empty.
Sorry, I don't have much more details yet, it failed at work here, where I have limited time to debug this.

Hm, I am not able to reproduce the problem. I just tried this patch against a fresh Kolab 16 deployment and it worked fine (I also tested it against nextcloud and groupwise). Any more hints on how to reproduce it are appreciated.

dfaure added a subscriber: dfaure.Nov 30 2017, 4:10 PM

Same bug here (but that's not surprising, we're all using the same Kolab server).

If there's no time to debug this for 17.12 (RC today!), I would like to request that this is reverted from the branch Applications/17.12 for now, and debugged in master.

Indeed we need to revert it for 17.12
And we keep some week in master for debugging it.

I reverted it before 17.12 RC tag.

I hope that it will debug in master soon.

Same bug here (but that's not surprising, we're all using the same Kolab server).

What Version of Kolab is that? I tried multiple times and it's working just fine for me with Kolab 16.

If there's no time to debug this for 17.12 (RC today!), I would like to request that this is reverted from the branch Applications/17.12 for now, and debugged in master.

Fine with me. Though I am pretty certain that the original code is broken. Especially if your caldav server is behind a reverse proxy or something like that. Because KDAV would insert the hostname/URL of the reverse proxy in that case into the mulitget request. Which might confuse the caldav server.

I don't know much about kolab server versioning and how to find out, but telnet mail.kdab.com imap says:
Cyrus IMAP v2.4.17-Kolab-2.4.17-30.6.el6.kolab_3.1 server ready
Does that help?

I would like to re-start the discussion. I tried rhaferkamps patches and they are working. The original code still fails to sync with mailbox.org.
The difference is that the original code includes the hostname in the request while the patch omits the hostname.
I would like to help to get mailbox.org working with KDAV again, but I don't know how. I can't think of a solution which might work for both cases and I wouldn't like to add a server specific code to to change the URL.

Can any one give me some directions, what might be a working approach?

I've done some Akonadi DAV resource bug triaging and think this, or a fixed patch, would resolve an issue affecting a large number of users.

After reading through the comments here, I was a bit surprised this patch was not taken up again after reverting it before 17.12.
From the comments, especially the following:

@dfaure wrote: Same bug here (but that's not surprising, we're all using the same Kolab server).

I gather that you decided to revert based on the fact that a single server (version?) failed (or were there more instances?), while the patch fixed an issue arising for a multiplicity of user servers.

So to those that reverted it and have access to the problematic server: please revisit this patch soon and gift those users a working DAV Groupware resource for Christmas!

dfaure added a comment.EditedNov 16 2018, 10:56 PM

Here's some debug output to understand what's going on:

KDAV::DavItemsListJob::davJobFinished: D8843 difference: href="/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/dd261076-60eb-40c2-b6fe-a245f69eac66.ics"

old code QUrl("https://caldav.kdab.com/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/dd261076-60eb-40c2-b6fe-a245f69eac66.ics")
    DavUrl: [ CalDav QUrl("https://dfaure%40kdab.com@caldav.kdab.com/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/dc3eb14d-fb62-4ddc-b7c5-9d8b3eb5fc02.ics") ]

new code QUrl("/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/dd261076-60eb-40c2-b6fe-a245f69eac66.ics")
    DavUrl: [ CalDav QUrl("//dfaure%40kdab.com@/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/901926d0-3732-4e0b-87d1-800f6db33c29.ics") ]

And when enabling the new code (i.e. this patch), here's what I get later on:

KDAV::DavItemsFetchJob::start <calendar-multiget xmlns="urn:ietf:params:xml:ns:caldav">
 <prop xmlns="DAV:">
      <getetag xmlns="DAV:"/>
  <calendar-data xmlns="urn:ietf:params:xml:ns:caldav"/>
 </prop>
  <href xmlns="DAV:">/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/dd261076-60eb-40c2-b6fe-a245f69eac66.ics</href>
 [...]

DavGroupwareResource::onMultigetFinished: Empty item returned.  "/calendars/dfaure@kdab.com/02af0002-0181-4d55-88a7-db543be67d83/dd261076-60eb-40c2-b6fe-a245f69eac66.ics"

The QUrl inside the DavUrl in the "new code" case looks very broken to me, with a username and no host and no scheme.

Here's some debug output to understand what's going on

Thanks for having a look at it again. I guess we now need similar output for the servers people are having issues with. Either they can provide this or they can provide test accounts.

Could you add a comment to https://bugs.kde.org/show_bug.cgi?id=386985 with instructions for getting similar debug output and a request for test accounts?

What's wrong with current patch and when will we see caldav working with any non-kolab server? Now calendar is unusable.

Termy added a subscriber: Termy.Feb 11 2019, 2:46 PM

It would be interessting to know the official DAV-specification. As other clients seem to work without the hostname-stuff, maybe kolab is doing something out-of-spec and it would be better to implement this patch here and open a bugreport at kolab?

Download and apply http://www.davidfaure.fr/2019/debug_output.diff to your kdav library to get debug output similar to mine.

Note that the comment "QUrl would treat this as a file URL be default" seems to be incorrect, there's no file: anywhere in my debug output.

dfaure reopened this revision.Feb 27 2019, 12:15 PM
This revision is now accepted and ready to land.Feb 27 2019, 12:15 PM
dfaure requested changes to this revision.Feb 27 2019, 12:15 PM
This revision now requires changes to proceed.Feb 27 2019, 12:15 PM
christianwolf added a subscriber: christianwolf.EditedThu, Oct 24, 3:34 PM

I have a problem with akonadi since the current update (19.08.1 -> 19.08.2). I asked in the kdepim-users mailinglist about it and I was directed to this discussion. I am unsure if the issue I have is related to this discussion.

Can you tell me if this is a newly introduced bug or somehow related to the current one? If related, I was willing to help getting things running smoothly again. So please tell me what you need then.

That's unrelated to this patch. What you're experiencing is a regression in KF 5.63, which is already fixed in KF5 git master, the fix will be in KF 5.64
See https://bugs.kde.org/show_bug.cgi?id=413316 and https://phabricator.kde.org/D24880