Fix detection of .kexi file types after recent change of mime database on Linux for SQLite type (2018-06-17)
Needs ReviewPublic

Authored by staniek on Dec 31 2018, 4:02 PM.

Details

Summary

Add application/vnd.sqlite3 and application/x-vnd.kde.kexi to be sure everything is catch while detecting .kexi file types.
The change is apparently needed after this commit:

https://cgit.freedesktop.org/xdg/shared-mime-info/commit/freedesktop.org.xml.in?id=36a88b01f6ae90da35a2f6a072db159c84cea768

From https://cgit.freedesktop.org/xdg/shared-mime-info/tree/freedesktop.org.xml.in:
it's apparent that .kexi mime type still inherits from application/x-sqlite3 which is now only an alias:

<mime-type type="application/x-kexiproject-sqlite3">

<_comment>Kexi database file-based project</_comment>
<sub-class-of type="application/x-sqlite3"/>
<glob pattern="*.kexi"/>
<alias type="application/x-vnd.kde.kexi"/>

Also update the driver's autotest.

Git branch: 396999-fix-sqlite-mime

BUG:396999
FIXED-IN:3.2.0

Big thanks to therapon@netzero.com for extensive testing!

CCMAIL:therapon@netzero.com
CCMAIL:ville.skytta@iki.fi
CCMAIL:hadess@hadess.net
CCMAIL:teuf@gnome.org

Test Plan
  1. Run autotests.
  2. Test KEXI with this KDb: try to open .kexi file while you miss the /usr/share/mime/application/x-sqlite3.xml file but have

the new /usr/share/mime/application/vnd.sqlite3.xml present.

  1. Test KEXI with this KDb: try to open .kexi file while you miss the /usr/share/mime/application/vnd.sqlite3.xml file but have

the old /usr/share/mime/application/x-sqlite3.xml present.

  1. Verify that opening from the Welcome sceen's recent list still works.

Expected: KEXI opens the file.

Diff Detail

Repository
R15 KDb
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
staniek created this revision.Dec 31 2018, 4:02 PM
Restricted Application added a project: KDb. · View Herald TranscriptDec 31 2018, 4:02 PM
Restricted Application added a subscriber: Kexi-Devel-list. · View Herald Transcript
staniek requested review of this revision.Dec 31 2018, 4:02 PM
staniek edited the summary of this revision. (Show Details)Dec 31 2018, 4:03 PM
pino added a subscriber: pino.Dec 31 2018, 4:47 PM

Another option could be to resolve the mimetypes when loading the metadata of the plugins, in DriverManagerInternal::lookupDriversInternal().

An untested patch can be:

diff --git a/src/KDbDriverManager.cpp b/src/KDbDriverManager.cpp
index 0d35729c..1de8e9fe 100644
--- a/src/KDbDriverManager.cpp
+++ b/src/KDbDriverManager.cpp
@@ -30,6 +30,9 @@
 #include <KPluginFactory>
 
 #include <QApplication>
+#include <QMimeDatabase>
+#include <QMimeType>
+#include <QSet>
 #include <QTime>
 #include <QWidget>
 
@@ -102,6 +105,7 @@ void DriverManagerInternal::lookupDriversInternal()
             = KDbJsonTrader::self()->query(QLatin1String("KDb/Driver"));
     const QString expectedVersion = QString::fromLatin1("%1.%2")
             .arg(KDB_STABLE_VERSION_MAJOR).arg(KDB_STABLE_VERSION_MINOR);
+    QMimeDatabase mimedb;
     foreach(const QPluginLoader *loader, offers) {
         //QJsonObject json = loader->metaData();
         //drivermanagerDebug() << json;
@@ -123,7 +127,18 @@ void DriverManagerInternal::lookupDriversInternal()
             }
             continue;
         }
+        QSet<QString> resolvedMimeTypes;
         foreach (const QString& mimeType, metaData->mimeTypes()) {
+            const QMimeType mime = mimedb.mimeTypeForName(mimeType);
+            if (!mime.isValid()) {
+                kdbWarning() << "Driver with ID" << metaData->id()
+                             << "supports the unknown mimetype"
+                             << mimeType << "-- skipping it";
+                continue;
+            }
+            resolvedMimeTypes.insert(mime.name());
+        }
+        foreach (const QString& mimeType, resolvedMimeTypes) {
             m_metadata_by_mimetype.insertMulti(mimeType, metaData.data());
         }
         m_driversMetaData.insert(metaData->id(), metaData.data());

The QSet is used to avoid that the same metadats is registered twice in m_metadata_by_mimetype for the same mimetype, in case a plugin specifies two entries (mime name and one alias of it).

In D17887#384384, @pino wrote:

The QSet is used to avoid that the same metadats is registered twice in m_metadata_by_mimetype for the same mimetype, in case a plugin specifies two entries (mime name and one alias of it).

Thanks. Interesting though m_metadata_by_mimetype.insertMulti(mimeType, metaData.data()) is used for a purpose, it's valid case to have multiple plugins supporting the same type. Even if for now it's not widely used possibility. KEXI has a special TODO for that - asking the user to pick driver (handler) for given type if there are multiple options. Currently we're picking first on the list.

pino added a comment.Dec 31 2018, 5:26 PM
In D17887#384384, @pino wrote:

The QSet is used to avoid that the same metadats is registered twice in m_metadata_by_mimetype for the same mimetype, in case a plugin specifies two entries (mime name and one alias of it).

Thanks. Interesting though m_metadata_by_mimetype.insertMulti(mimeType, metaData.data()) is used for a purpose, it's valid case to have multiple plugins supporting the same type.

Let me explain this better -- the situation fixed by using the QSet is:

  • a plugin has application/x-foo, and application/x-foo-alias as MimeTypes
  • application/x-foo-alias is an alias of application/x-foo
  • directly inserting in m_metadata_by_mimetype the result of the mime type resolution means that m_metadata_by_mimetype will have the metadata of this plugin twice for the same key (application/x-foo)

My draft of patch does not change at all the fact that different plugins can register for the same mimetype.

staniek added a comment.EditedDec 31 2018, 5:28 PM

Ah your test is for something else than I wrote above and might be useful but I just learned that we never know (like in the case of the linked FD.o change) what happens on user's OS and whether the OS provide type alias or regular type as a result of detection. So I'd be safer if we handle all mime types and aliases declared by the plugins.

m_metadata_by_mimetype's key is string so application/x-foo and application/x-foo-alias are two different keys that here point to the same plugin. User's code is free to ask about either type by name.

pino added a comment.Dec 31 2018, 5:31 PM

Ah your test is for something else than I wrote above and might be useful but I just learned that we never know (like in the case of the linked FD.o change) what happens on user's OS and whether the OS provide type alias or regular type as a result of detection. So I'd be safer if we handle all mime types and aliases declared by the plugins.

This is exactly what my patch does... in case any of the mimetypes specified by the kdb plugins will become an alias, kdb will still handle that.
The second part of the patch is, if needed, that whatever calls KDbDriverManager::driverIdsForMimeType() ought to better pass resolved mimetypes too.

pino added a comment.Dec 31 2018, 5:38 PM

OK, second attempt (still untested!), resolving mimetypes on lookup:

diff --git a/src/KDbDriverManager.cpp b/src/KDbDriverManager.cpp
index 0d35729c..297910b4 100644
--- a/src/KDbDriverManager.cpp
+++ b/src/KDbDriverManager.cpp
@@ -30,6 +30,9 @@
 #include <KPluginFactory>
 
 #include <QApplication>
+#include <QMimeDatabase>
+#include <QMimeType>
+#include <QSet>
 #include <QTime>
 #include <QWidget>
 
@@ -102,6 +105,7 @@ void DriverManagerInternal::lookupDriversInternal()
             = KDbJsonTrader::self()->query(QLatin1String("KDb/Driver"));
     const QString expectedVersion = QString::fromLatin1("%1.%2")
             .arg(KDB_STABLE_VERSION_MAJOR).arg(KDB_STABLE_VERSION_MINOR);
+    QMimeDatabase mimedb;
     foreach(const QPluginLoader *loader, offers) {
         //QJsonObject json = loader->metaData();
         //drivermanagerDebug() << json;
@@ -123,7 +127,18 @@ void DriverManagerInternal::lookupDriversInternal()
             }
             continue;
         }
+        QSet<QString> resolvedMimeTypes;
         foreach (const QString& mimeType, metaData->mimeTypes()) {
+            const QMimeType mime = mimedb.mimeTypeForName(mimeType);
+            if (!mime.isValid()) {
+                kdbWarning() << "Driver with ID" << metaData->id()
+                             << "supports the unknown mimetype"
+                             << mimeType << "-- skipping it";
+                continue;
+            }
+            resolvedMimeTypes.insert(mime.name());
+        }
+        foreach (const QString& mimeType, resolvedMimeTypes) {
             m_metadata_by_mimetype.insertMulti(mimeType, metaData.data());
         }
         m_driversMetaData.insert(metaData->id(), metaData.data());
@@ -162,7 +177,12 @@ QStringList DriverManagerInternal::driverIdsForMimeType(const QString &mimeType)
     if (!lookupDrivers()) {
         return QStringList();
     }
-    const QList<KDbDriverMetaData*> metaDatas(m_metadata_by_mimetype.values(mimeType.toLower()));
+    QMimeDatabase mimedb;
+    const QMimeType mime = mimedb.mimeTypeForName(mimeType);
+    if (!mime.isValid()) {
+        return QStringList();
+    }
+    const QList<KDbDriverMetaData*> metaDatas(m_metadata_by_mimetype.values(mime.name()));
     QStringList result;
     foreach (const KDbDriverMetaData* metaData, metaDatas) {
         result.append(metaData->id());

Thanks Pino. Your patch adds some future-proof, I would adapt it. But I yet have to see how the problem is solved without patching of the json file. If I understand correctly if we do not explicitly list "application/vnd.sqlite3" in the JSON, current FD.o master https://cgit.freedesktop.org/xdg/shared-mime-info/commit/freedesktop.org.xml.in?id=36a88b01f6ae90da35a2f6a072db159c84cea768 resolves "application/vnd.sqlite3" into "application/x-kexiproject-sqlite3" only if we go to parent mime type "application/x-sqlite3" which is alias of real parent "application/vnd.sqlite3". I see that "application/x-kexiproject-sqlite3" has room for improvements. I do not see that we use QMimeType::parentMimeTypes().

So for me the full fix is to work as expected also on such half-configured systems (so all updated Linux systems became problematic). I am going to even go further and fall back to the sqlite3 plugin when mime database gives no hint at all what the file is. At KDb or KEXI level, preferably KEXI level.

Happy new year :)

pino added a comment.Jan 1 2019, 4:26 PM

If I understand correctly if we do not explicitly list "application/vnd.sqlite3" in the JSON, current FD.o master https://cgit.freedesktop.org/xdg/shared-mime-info/commit/freedesktop.org.xml.in?id=36a88b01f6ae90da35a2f6a072db159c84cea768 resolves "application/vnd.sqlite3" into "application/x-kexiproject-sqlite3" only if we go to parent mime type "application/x-sqlite3" which is alias of real parent "application/vnd.sqlite3". I see that "application/x-kexiproject-sqlite3" has room for improvements. I do not see that we use QMimeType::parentMimeTypes().

I don't see what parent mimetypes have anything to do with this. There is no parent lookup neither "is a" checking in the plugin loader, all it does is "supports this mimetype".
Do not overthink the issue, really...

In D17887#384666, @pino wrote:

If I understand correctly if we do not explicitly list "application/vnd.sqlite3" in the JSON, current FD.o master https://cgit.freedesktop.org/xdg/shared-mime-info/commit/freedesktop.org.xml.in?id=36a88b01f6ae90da35a2f6a072db159c84cea768 resolves "application/vnd.sqlite3" into "application/x-kexiproject-sqlite3" only if we go to parent mime type "application/x-sqlite3" which is alias of real parent "application/vnd.sqlite3". I see that "application/x-kexiproject-sqlite3" has room for improvements. I do not see that we use QMimeType::parentMimeTypes().

I don't see what parent mimetypes have anything to do with this. There is no parent lookup neither "is a" checking in the plugin loader, all it does is "supports this mimetype".
Do not overthink the issue, really...

Shorter: what change in kdb_sqlitedriver.json do you propose?

pino added a comment.Jan 1 2019, 5:53 PM

Shorter: what change in kdb_sqlitedriver.json do you propose?

With the draft of patch I proposed, none.

staniek added a comment.EditedJan 1 2019, 6:27 PM
In D17887#384687, @pino wrote:

Shorter: what change in kdb_sqlitedriver.json do you propose?

With the draft of patch I proposed, none.

OK but that means you propose to revert change for the json or keeping the change as is?

I need to double check we understand if (and how) the plugin is properly detected in two cases:

  1. Old case: before the critical FD.o change - it works because .kexi files' mimetype is "application/x-sqlite3"
  2. New case: after the the critical FD.o change when .kexi files' mimetype is application/vnd.sqlite3 - this is the case of https://bugs.kde.org/show_bug.cgi?id=396999 - how would it work?

Both using the same code.

pino added a comment.Jan 1 2019, 7:25 PM
In D17887#384687, @pino wrote:

Shorter: what change in kdb_sqlitedriver.json do you propose?

With the draft of patch I proposed, none.

OK but that means you propose to revert change for the json or keeping the change as is?

My patch instead of this.

I need to double check we understand if (and how) the plugin is properly detected in two cases:

  1. Old case: before the critical FD.o change - it works because .kexi files' mimetype is "application/x-sqlite3"
  2. New case: after the the critical FD.o change when .kexi files' mimetype is application/vnd.sqlite3 - this is the case of https://bugs.kde.org/show_bug.cgi?id=396999 - how would it work?

Using my patch (v2), the mimetypes claimed by every plugin are always registered as their real names (i.e. resolving possible aliases). When loading a plugin via mimetype, the code finds the real name of the requested mimetype (i.e. resolving possible aliases), and uses it to get the plugin for it (if available).

The sqlitedriver plugin claim to support two mimetypes, and this will not change.

staniek updated this revision to Diff 48494.Jan 1 2019, 9:36 PM
  • Resolve plugin mime types before adding metadata - idea from from Pino
  • Add autotests for checking detection of every mime type that should be supported
staniek added a reviewer: pino.Jan 1 2019, 9:36 PM

Tested before and after adding application/vnd.sqlite3 to the system (exact FD.o patch) - it works only thanks to keeping the old x-sqlite3 type in the JSON and because FD.o has an alias. Good that I asked about json, otherwise it would fail for me. Added more extensive tests to verify. Thanks @pino and be welcome to more reviews :)

pino added a comment.Jan 1 2019, 9:52 PM

I don't understand what is tested in KDbTestUtils::testDriver() now, related to the mimetypes. The old logic looked better (and simpler too) to me, I'd just leave that.
The only simple addition IMHO is that manager.driverIdsForMimeType(mimeName) returns a valid driver for each of the mimetypes specified in the plugin metadata.

Also, the commit message needs to be updated to just say that mimetypes are always resolved in the driver manager, so even if a mimetype claimed by a driver becomes an alias the plugin is still loaded.

CMakeLists.txt
6 ↗(On Diff #48494)

unrelated

autotests/KDbTestUtils.cpp
98–112

already done by QTestLib

261

I don't understand why this is "possibly invalid"; this is the new name of the sqlite3 mimetype, so this is pretty much valid

autotests/KDbTestUtils.h
78–81

QTestLib already supports comparing QStringList

src/drivers/mysql/kdb_mysqldriver.json
93 ↗(On Diff #48494)

unrelated

src/drivers/postgresql/kdb_postgresqldriver.json
93 ↗(On Diff #48494)

unrelated

src/drivers/sqlite/kdb_sqlitedriver.json
96 ↗(On Diff #48494)

unrelated

src/drivers/sybase/kdb_sybasedriver.json
93 ↗(On Diff #48494)

unrelated

src/drivers/xbase/kdb_xbasedriver.json
93 ↗(On Diff #48494)

unrelated

src/tools/KDbUtils.cpp
2 ↗(On Diff #48494)

unrelated

207–209 ↗(On Diff #48494)

unrelated

src/tools/KDbUtils.h
2 ↗(On Diff #48494)

unrelated

77–87 ↗(On Diff #48494)

unrelated

staniek marked 9 inline comments as done.Jan 1 2019, 10:41 PM
staniek added inline comments.
CMakeLists.txt
6 ↗(On Diff #48494)

It's not part of this branch. Bug in Phabricator while using branches? We're dropping it anyway for gitlab.

autotests/KDbTestUtils.cpp
98–112

Like above.

261

It is possibly invalid because it is not guaranteed that this type exists at all. If that's the case it's invalid on given OS. That's exactly the case on my opensuse 42.3 and reason for this generic addition of blacklist for all future cases. Otherwise the test would fail sometimes.

autotests/KDbTestUtils.h
78–81

Unfortunately it only supports comparing list sizes. I plan to use this feature more.

src/drivers/mysql/kdb_mysqldriver.json
93 ↗(On Diff #48494)

It happens on Phab, like mentioned before. Even RB was better.

My 'git cherry origin/3.2' shows exactly 3 commits and I'd like to keep them all.

Maybe you know how to clean it up, thanks.

staniek marked an inline comment as done.Jan 1 2019, 10:49 PM
In D17887#384770, @pino wrote:

I don't understand what is tested in KDbTestUtils::testDriver() now, related to the mimetypes. The old logic looked better (and simpler too) to me, I'd just leave that.
The only simple addition IMHO is that manager.driverIdsForMimeType(mimeName) returns a valid driver for each of the mimetypes specified in the plugin metadata.

Two things:

  • the original logic of the patch completely missed checks of driverIdsForMimeType(), it is also not present in 3.2 branch
  • the issue with invalid mimetypes on testing OS

Also, the commit message needs to be updated to just say that mimetypes are always resolved in the driver manager, so even if a mimetype claimed by a driver becomes an alias the plugin is still loaded.

It's good idea for the API docs, I'll add.

staniek updated this revision to Diff 48501.Jan 1 2019, 11:31 PM
  • Update API docs for KDbDriverManager::driverIdsForMimeType()
This revision was not accepted when it landed; it landed in state Needs Review.Jan 4 2019, 11:32 AM
This revision was automatically updated to reflect the committed changes.
staniek reopened this revision.Jan 4 2019, 11:42 AM
staniek updated this revision to Diff 48699.Jan 4 2019, 8:53 PM

Update to 3.2

This revision was not accepted when it landed; it landed in state Needs Review.Jan 15 2019, 9:19 AM
This revision was automatically updated to reflect the committed changes.

@piggz
Pushed for 3.2 Beta, post-commit review would be welcome.

staniek reopened this revision.Jan 15 2019, 9:20 AM