I am wondering if we can actually do anything about this but build the software ourselves.

So, I talked with @mak about this and he suggested that we might want to use debian's data as it is up-to-date, not using bugged appstream versions and is altogether more awesome.

Given our rapid pickup times I am wondering if that is necessarily true as we might have a new app long before it arrives in debian. That said, we might consider folding debian's data into our data thus getting the more complete coverage of debian (vs ubuntu) AND our uptodateness. But IIRC there is no actual code for this so that'd need doing from scratch.
For the time being, it might actually be best to simply switch our appstream data for debian's, also has the advantage that we don't need to do our asgen hack which is fairly heavy on the disk usage during generation.

https://appstream.debian.org/

@sitter Oh, looks like you misunderstood me there ^^
I think it's fine to fetch the appstream.debian.org output for your Snappy-description hacks - for actually shipping the data with Neon, you should really run asgen on your packages.

It's not resource-intensive if you let it keep its database between runs, so it knows which stuff needs to be extracted again. Iain is working upstream on download-on-demand support, which will also fix your diskspace problem.
If you set a higher data priority, you will override the data Ubuntu already ships, so this shouldn't be an issue either :-)

Is there any problem I missed?

Ah, and for incomplete data: If you upload a fixed-up package into Neon, asgen should pick it up and generate valid data. reprocessing the Ubuntu data with a newer asgen isn't possible though, since the Ubuntu data is frozen in time.
(of course you could reprocess the full Ubuntu archive, but that's meh...) reprocessing would IMHO only be interesting as soon as asgen can handle fonts properly.

Ah yes. So. Multiple issues all at once. This actually is out of scope for the snap stuff.

Let's exclude snaps for now, since we ultimately snap our debs and our debs are asgen'd by us anyway, so we have control over data quality there. So that's mostly a non-issue.

The actual problem is component org.bar.foo not showing up in discover because the package in Ubuntu is too old to have appstream data. Such as with Krita.

Now, for KDE software that is a transient issue because we'll eventually have all KDE software built by us, in our repo, asgen'd by us thus actually having data. But! For not-KDE software this won't happen since we won't be uploading nor maintaining packages, so we won't have data on that unless the version in Ubuntu 16.04 had them.

@mak I guess the question here really is: should we cheat our way to higher coverage by folding Debian's data into ours (accepting that sometimes package mapping could be wrong?) or accept the fact that certain apps won't be discoverable since their versions are simply too old in 16.04?

(albeit all of this sort of goes away if/when flatpak and/or snaps get dragged into discover)

@sitter Depends on the amount of pain you want to have with this ^^
I would really not merge the Debian data, because:

• It might use the wrong packages
• It might be for a much more recent release than what Ubuntu had available

Wrong data wouldn't help anyone.
The honest solution would be to just fix the packages you care about in Neon by uploading corrected versions to your archive (this is actually what a few projects, like Scribus, also did for Ubuntu).