Not yet tested with MSVS but it can be made afterwards.

Instead of updating the embedded copy of mdbtools, why not just switch to an external/system copy of it? This way tasks like T1361: [ie] Update to current mdbtools (July 2018) will not be needed anymore (and your distro packagers will be happier).

In D14486#300823, @pino wrote:

Instead of updating the embedded copy of mdbtools, why not just switch to an external/system copy of it? This way tasks like T1361: [ie] Update to current mdbtools (July 2018) will not be needed anymore (and your distro packagers will be happier).

Hi Pino, we're getting such questions from time to time. mdbtools, as a great project that supports obsolete MS tech (according to MS itself), it is not mature (or widely used) enough to be versioned properly, separated to functional nonGUI/GUI parts, have BC in mind, all what's requirement of proper packaging. We're not eager to see KEXI depending on GTK+... glib is enough issue.

To show you there's no lack of motivation, it was just recently that sqlite got convinced (by me) to version properly), and KEXI switched immediately to packaged version, yet still sqlite is (by design) configurable at build time so the more of it we use the more is probable we would come back to our copy...

In D14486#300844, @staniek wrote:

In D14486#300823, @pino wrote:

Instead of updating the embedded copy of mdbtools, why not just switch to an external/system copy of it? This way tasks like T1361: [ie] Update to current mdbtools (July 2018) will not be needed anymore (and your distro packagers will be happier).

Hi Pino, we're getting such questions from time to time. mdbtools, as a great project that supports obsolete MS tech (according to MS itself), it is not mature (or widely used) enough to be versioned properly, separated to functional nonGUI/GUI parts, have BC in mind, all what's requirement of proper packaging. We're not eager to see KEXI depending on GTK+... glib is enough issue.

Regarding the dependencies: the libmdb2 library package in Debian and derived distros, generated from the mdbtools source, depends only on glib; GTK+ seems to be used only by gmdb2, which is only one of the tools.

Regarding the stability: libmdb.so has the same SONAME since its introduction in mdbtools 0.7, released 6 years ago, so IMHO there are no concerns about the BC/API stability (or they are low).

Regarding the maturity: of course, this is a "dog that chases its own tail" issue: if projects embed mdbtools instead of using a system copy, of course the upstream project is tested less. In Debian I found 3 projects using mdbtools:

• kexi: uses its own embedded copy
• libgda (http://www.gnome-db.org/): uses its own embedded copy
• recutils (http://www.gnu.org/software/recutils/): optionally uses a system version (no embedded copy)

Obviously I cannot scientifically prove it, but I'm fairly sure the quality of upstream mdbtools would improve if both kexi and libgda switched away from their embedded copies.

Also, the Debian packaging has a couple of backports of fixes from upstream: since I package kexi in Debian, I really do not want to start doing this same job (i.e. backport fixes from another upstream) -- using the system copy means that I almost do not need to care, or if I do then fixing mdbtools will do the job.

Thanks for spending time on this Pino. If I can fix something at KEXI side it will cost us less. KEXI release do not trigger mbbtools release, that's #1 problem: nothing to do unless I switch to working on mdbtools, I do not say it's not possible, but KEXI has tons of such, some of them are below the tip of the iceberg visible for Debian users.

I can tell you stories coming from actual user base. Debian flavor is small percent of our specialized OS universe... Ubuntu is also small but way bigger, neither of the OSes provide upgrades for MDB tools outside of the unstable line or custom installation package (that's only for power users). So the real fix is not only at our side. Ubuntu LTS is ultimate issue on top of that. Users are for example not willing to abandon "LTS" (advertised to them as "long term support"), and then loose MDB support in KEXI, and thus loose opportunity to KEXI for their purpose.

Someone at the mdbtools level itself devoted to splitting to orthogonal elements could help. He would make mdbtools.h indpendent of exporting features, backend, odbs and other things.

On KEXI side it's not likely that there are funds for contributing back to mdbtools - there are not many public projects of mdbtools as you already know. In KEXI there are no funds to even contribute to features in the mdbtools copy but still, by having a copy I reserve this possibility - to add a feature or fix _quickly_ in the same KEXI release, not in some distant future. Even if it's "just" building Android port - how could the vanilla mdbtools support our use cases if it's so small project? We could not be so harsh to request anything.

I am sorry but this approach has been proven to be practical even if not right at academic level. There well requests to integrate similar code with Rekall and Knoda. Both projects are inactive to say at least. Furthermore I personally lost time and money even on barcode libraries here within KDE, e.g. on porting to libPrison.

We also have another copy and that's a full fork proved to be right thing: KReport is a fork of OpenRPT. Where it is (a Qt 5 port) in Debian? Non-existing. We did nothing to contribute there but why would we have to? There are no usable contributions to KEXI of this sort - from other projects.

These stories are all about very specialized subprojects or features that we depend on... Most usual users are closed source ones.

First of all Jarosław: thanks for the details -- OTOH, could you please be slightly a bit less verbose and more to-the-point? Thank you in advance.

In D14486#300882, @staniek wrote:

Thanks for spending time on this Pino. If I can fix something at KEXI side it will cost us less. KEXI release do not trigger mbbtools release, that's #1 problem:

Likewise, a KEXI release does not trigger a release of KF5, nor Qt5, nor anything else that KEXI uses.

nothing to do unless I switch to working on mdbtools, I do not say it's not possible, but KEXI has tons of such, some of them are below the tip of the iceberg visible for Debian users.

I can tell you stories coming from actual user base. Debian flavor is small percent of our specialized OS universe... Ubuntu is also small but way bigger, neither of the OSes provide upgrades for MDB tools outside of the unstable line or custom installation package (that's only for power users). So the real fix is not only at our side. Ubuntu LTS is ultimate issue on top of that. Users are for example not willing to abandon "LTS" (advertised to them as "long term support"), and then loose MDB support in KEXI, and thus loose opportunity to KEXI for their purpose.

While this is something I'm aware of, and I can understand, this is totaly unrelated to the issue at hand.
Also, there is a bit of FUD in the paragraph above -- still, unrelated to the issue, so I will ignore it.

Someone at the mdbtools level itself devoted to splitting to orthogonal elements could help. He would make mdbtools.h indpendent of exporting features, backend, odbs and other things.

Were these needs asked to the mdbtools upstream?

On KEXI side it's not likely that there are funds for contributing back to mdbtools - there are not many public projects of mdbtools as you already know. In KEXI there are no funds to even contribute to features in the mdbtools copy but still, by having a copy I reserve this possibility - to add a feature or fix _quickly_ in the same KEXI release, not in some distant future. Even if it's "just" building Android port - how could the vanilla mdbtools support our use cases if it's so small project? We could not be so harsh to request anything.

As above: for the same reasoning, you would then embed KF5, Qt5, etc, just because of the "possibility" to fix quickly anything in them.
Or where do we draw the line here?

I am sorry but this approach has been proven to be practical even if not right at academic level.

This has nothing to do with "academic level".

We also have another copy and that's a full fork proved to be right thing: KReport is a fork of OpenRPT. Where it is (a Qt 5 port) in Debian? Non-existing. We did nothing to contribute there but why would we have to? There are no usable contributions to KEXI of this sort - from other projects.

Honestly, I would not consider one example as the example.

Qt5 and KF5 are regularly and widely released general purpose development toolkits, which is more than enough for the needs of applications developers. If, on the other hand, KEXI had been built on top of GTK+ instead then GTK+ (or its critical parts) would have been forked here by the time already (or abandoned like for GCompris) to still keep cross platform support, especially on Mac/Windows.

I do not know if you perceive "LTS/Debian-stable is not a long term support offering but long term only-release-security-patches offering" as a FUD but I wanted to assure I so not say for any other reason than to show you my perspective, author who releases version 3.1 and deal with users preferring "stable" distribution with KEXI 2.8 or 2.9 long unmaintained.

Thanks for supporting KEXI packages :)

PS: I am not fully sure KEXI packages other than for Windows, are actually tested before the release. To ensure that for every release I would have to employ full time developer who writes tests for distributors to make sure depending packages work as we expect. Also have dozens of OS images around. I do know reports that KEXI does not start or support any database backend due to database drivers not found. All this is not your fault but definitely fault of distributions not being able to integrate. How in this situation application developers have to be urged to integrate codebase to a single when distributions do not share the same mind set?

Workaround for that would be AppImage on which I have spent quite a bit of time and found no contributors who would continue that. So the focus turned to Craft.
Both these approaches have control over the mdbtools distribution so how mdbtools is distributed is irrelevant there, it is packaged by the KEXI project after all and currently only used by KEXI.

If you say Debian barely uses mdbtools there's not much to fight about so your is proposal correct but of a very low priority for me. Much lower than e.g. ODBC support which I know is at least suboptimal of non-Mac/Windows desktops.

[ snip OT talk about distros, and limited POV on them ]

All this is not your fault but definitely fault of distributions not being able to integrate. How in this situation application developers have to be urged to integrate codebase to a single when distributions do not share the same mind set?

You know, this is not a "us vs. them", "do software and they break it", but rather should be a cooperation -- an upstream helps helps downstreams in shipping the best way of upstream softwares.
There is no "same mind set", since we are not an hive mind. We are all humans, we can make mistakes (both sides), and the important is making sure they issues are fixed.

And btw, since you mentioned it: if sometimes distros are "not [being] able to integrate", sometimes it is because upstream does poor/unclear/wrong choices that make the packaging work hard, or even impossible.

If you say Debian barely uses mdbtools there's not much to fight about

And I explained so already: you cannot complain that mdbtools is not much used, if what KEXI currently does is actually part of the reason why mdbtools is not used much.

so your is proposal correct but of a very low priority for me.

Considering somebody else other than you invested their time in updating the embedded copy (i.e. this differential revision), then it is just matter of adding the build system knobs to use a system library. After all, both the update of the embedded copy and the switch to the system library will require careful testing, so there is no difference nor increase in work needed from this POV.

In D14486#300951, @pino wrote:

so your is proposal correct but of a very low priority for me.

Considering somebody else other than you invested their time in updating the embedded copy [...]

Sorry, I have to stand corrected by myself: you were the one doing this update. OTOH, the rest of my message still stands.

you cannot complain that mdbtools is not much used, if what KEXI currently does is actually part of the reason why mdbtools is not used much.

I do not complain :) mdbtools, like many KEXI features or like genome sequencing software, etc. are *all* not much used by the nature of this software. Even many of the KDE contributors asked did not see such type of software as category or part of KDE (sw system that is not an SQL db backend). Not a complaint here, it's natural. It's expected to get more contributions to image viewers, icon grids (desktops) or file managers since there's the sweet point.

Considering somebody else other than you invested their time in updating the embedded copy (i.e. this differential revision), then it is just matter of adding the build system knobs to use a system library. After all, both the update of the embedded copy and the switch to the system library will require careful testing, so there is no difference nor increase in work needed from this POV.

1. Embedded copy only = forced 1 version per release, on M platforms, controlled upgrade
2. Embedded copy + allowing upstream = N+1 versions, where N is number of upstream versions on these M platforms (no control over this at all, can be that some platforms has no mdbtools at all). (mdbtools is very unlike popular dependencies)

From the top perspective: KEXI has D dependencies consisted of rare and barely/slowly evolving software that is out of control; well, it is even able to use unreleased Java dependencies with strict version of Java needed; each single release of KEXI can have many more than D incarnations of dependencies even on single platform. There is no problem with major dependencies (but even Qt is not 100% green - 5.10.x generates issue like printing broken for 1 to 2 months until a fix appears) but for rare dependencies it's practically impossible to reproduce real combination of KEXI+system_dependencies - user base is so fragmented. In this workflow, package authors are disconnected from testers - and I am not complaining but stating this, as something very different to closed+commercial development.

Reduction of permutations is why I've observed that users were happy with monolytic Appimage - they get exactly the software KEXI authors had on their tested at least for some time. I also do not see popular embedded and mobile platforms doing anything else than embedding custom dependencies.

In D14486#300962, @pino wrote:

In D14486#300951, @pino wrote:

so your is proposal correct but of a very low priority for me.

Considering somebody else other than you invested their time in updating the embedded copy [...]

Sorry, I have to stand corrected by myself: you were the one doing this update. OTOH, the rest of my message still stands.

Of course this shall be a member of KEXI project because mdbtools is not a general purpose solution designed to plug to other GUI apps so easily. It's consisted of set of command line tools (they are always hard to integrate with a front end in other than naive way) or has some hard coded GUI, plus C APIs for all these needs exposed in one relevant header file. API stability through the fact that the header rarely changes is accidental.

Disclaimer: mdbtools is one of best reverse engineering projects I know and one of most important to me. Reverse engineering is the core of it, the rest is "just" necessary thing IMHO.

I can assure you that updating mdbtools, unlike sqlite (which is just big enough that it is fine grained and more general purpose), needs to be performed patch-by-patch. Long ago it was the same story with SQLite (version 2 vs 3, breaking compatibility after 3.0, and so on). Since then sqlite is used on every mobile phone and mdbtools is used, you know how frequently.

It's not just a separate situation. I can recall the story of issues that Okular had with (IIRC?) PDF backends. Or Krita with touch input device support in Qt. These projects were not even able to fork the dependencies, I guess mainly because of the size but number of workarounds added was noticeable. Of course we do not argue that forking or embedding is not a really unclean solution, but it's not true that it's not happening, just like it's not true that there are thousands of projects that use years old compilers.

Here in keximdb nothing changes for you as package author, right? When this patch gets integrated into KEXI there is the same situation from the packages' POV as you it was years ago.

If otherwise I just delegated the risk and testing to users (not packagers, since as we know they do not have just one project to support and are rather project-agnostic) it isn't impossible that packagers would have to know that we request version 0.7.1 supported and not >=0.7.2 for some compatibility or "exceptional" reason. I can't imagine how this contract can be assured distro-independently with tool we have not which is just a plain README file and so limited developer workforce. It's possible that users will have KEXI 3.1 running on their desktop 4 years later and mdbtools package completely unknown to us. Part of the issue come from the fact that we reserve right to use mdbtools not the way it was designed, to get all kind of instrumentation we need.

Jokes: I do not know how it is possible that chromium or Libreoffice or Firefox is packaged as larger monolytic packages (they both contain lots of unrelated tech compiled together). Or MySQL. Why the MySQL parser is not separate package so I can use it...

In D14486#300988, @staniek wrote:

In D14486#300962, @pino wrote:

In D14486#300951, @pino wrote:

so your is proposal correct but of a very low priority for me.

Considering somebody else other than you invested their time in updating the embedded copy [...]

Sorry, I have to stand corrected by myself: you were the one doing this update. OTOH, the rest of my message still stands.

Of course this shall be a member of KEXI project because mdbtools is not a general purpose solution designed to plug to other GUI apps so easily. It's consisted of set of command line tools (they are always hard to integrate with a front end in other than naive way) or has some hard coded GUI, plus C APIs for all these needs exposed in one relevant header file. API stability through the fact that the header rarely changes is accidental.

Disclaimer: mdbtools is one of best reverse engineering projects I know and one of most important to me. Reverse engineering is the core of it, the rest is "just" necessary thing IMHO.

I can assure you that updating mdbtools, unlike sqlite (which is just big enough that it is fine grained and more general purpose), needs to be performed patch-by-patch. Long ago it was the same story with SQLite (version 2 vs 3, breaking compatibility after 3.0, and so on). Since then sqlite is used on every mobile phone and mdbtools is used, you know how frequently.

It's not just a separate situation. I can recall the story of issues that Okular had with (IIRC?) PDF backends. Or Krita with touch input device support in Qt. These projects were not even able to fork the dependencies, I guess mainly because of the size but number of workarounds added was noticeable. Of course we do not argue that forking or embedding is not a really unclean solution, but it's not true that it's not happening, just like it's not true that there are thousands of projects that use years old compilers.

Here in keximdb nothing changes for you as package author, right? When this patch gets integrated into KEXI there is the same situation from the packages' POV as you it was years ago.

If otherwise I just delegated the risk and testing to users (not packagers, since as we know they do not have just one project to support and are rather project-agnostic) it isn't impossible that packagers would have to know that we request version 0.7.1 supported and not >=0.7.2 for some compatibility or "exceptional" reason. I can't imagine how this contract can be assured distro-independently with tool we have not which is just a plain README file and so limited developer workforce. It's possible that users will have KEXI 3.1 running on their desktop 4 years later and mdbtools package completely unknown to us. Part of the issue come from the fact that we reserve right to use mdbtools not the way it was designed, to get all kind of instrumentation we need.

Jokes: I do not know how it is possible that chromium or Libreoffice or Firefox is packaged as larger monolytic packages (they both contain lots of unrelated tech compiled together). Or MySQL. Why the MySQL parser is not separate package so I can use it...

A few questions, in the interest of keeping our downastream happy, and potentially reducing our burden

1. do we carry patches ontop of mdb?
2. how much work would be neded in adding a switch to use the system lib?

--I know this would initially be a little work, but could allow a phased approach to using the system supplied lib over time, as packagers choose. If after 1-2 years, all packagers are onboard, we could then make it the default and remove our version of mdb, and the need to maintain it.

In D14486#301205, @piggz wrote:

A few questions, in the interest of keeping our downastream happy, and potentially reducing our burden

1. do we carry patches ontop of mdb?
2. how much work would be neded in adding a switch to use the system lib? --I know this would initially be a little work, but could allow a phased approach to using the system supplied lib over time, as packagers choose. If after 1-2 years, all packagers are onboard, we could then make it the default and remove our version of mdb, and the need to maintain it.

Thanks for stepping up Adam. I am not sure if:

1. mdbtools carries version information .
2. It even does not maintain cmake module.

This work is needed in mdbtools itself. And we have no evidence there's any other consumer apart of KEXI that uses mdbtools via cmake.

If above bits are supplied and maintained, we could allow exact version of "system" mdbtools. Exact because it's the only one I test.

IMHO it's more effort and risk than benefit, from history I remember that distros can:

• skip mdbtools if we set it optional -> KEXI lacks major feature
• skip KEXI if we set mdbtools required -> KEXI disappears from the distro at least for some time, for packagers it's "just" one release less

Because I am all for seeing any contributions please go ahead to discuss[*] and contribute 1. and 2., then we can have opt-in "system" mdbtools support.
We would keep mdbtools mandatory (as long as glib and iconv is present).

Discuss[*]: your challenge would be that mdbtools is consisted of things orthogonal to MDB support KEXI needs: importer tool, exporter tool, SQL support. Whether that project will let you contribute just cmake files for the MDB C API for the read support is not clear. So there may be more work than KEXI needs.

In D14486#301208, @staniek wrote:

In D14486#301205, @piggz wrote:

A few questions, in the interest of keeping our downastream happy, and potentially reducing our burden

1. do we carry patches ontop of mdb?
2. how much work would be neded in adding a switch to use the system lib? --I know this would initially be a little work, but could allow a phased approach to using the system supplied lib over time, as packagers choose. If after 1-2 years, all packagers are onboard, we could then make it the default and remove our version of mdb, and the need to maintain it.

Thanks for stepping up Adam. I am not sure if:

1. mdbtools carries version information .
2. It even does not maintain cmake module.
   
   This work is needed in mdbtools itself. And we have no evidence there's any other consumer apart of KEXI that uses mdbtools via cmake.
   
   If above bits are supplied and maintained, we could allow exact version of "system" mdbtools. Exact because it's the only one I test.
   
   IMHO it's more effort and risk than benefit, from history I remember that distros can:
3. skip mdbtools if we set it optional -> KEXI lacks major feature
4. skip KEXI if we set mdbtools required -> KEXI disappears from the distro at least for some time, for packagers it's "just" one release less
   
   Because I am all for seeing any contributions please go ahead to discuss[*] and contribute 1. and 2., then we can have opt-in "system" mdbtools support. We would keep mdbtools mandatory (as long as glib and iconv is present).
   
   Discuss[*]: your challenge would be that mdbtools is consisted of things orthogonal to MDB support KEXI needs: importer tool, exporter tool, SQL support. Whether that project will let you contribute just cmake files for the MDB C API for the read support is not clear. So there may be more work than KEXI needs.

out of interest, i was looking to see what mdbtools provides, and they have pkgconfig support we could likely use, but then I came across a maintained fork called cyber-mdbtools, which may be more open to contributions and packaging, and they already have fixes ontop of mdbtools 0.7.1.

@piggz ping :")

Update: mdbtools does not support msvc due to the C2057 error so I am reverting two commits that remove #defines.