Improve directory structure
Open, NormalPublic

Description

This is a summary of the discussion here: https://mail.kdab.com/pipermail/kphotoalbum/2020-April/006572.html

Status quo

DBDIR/index.xml
DBDIR/CategoryImages/
DBDIR/layout.dat
DBDIR/exif-info.db
DBDIR/.thumbnails/
DBDIR/.videoThumbnails
XDG_DATA_DIR/kxmlgui5/kphotoalbum/kphotoalbumui.rc
XDG_CONFIG_DIR/kphotoalbumrc

This directory layout worked ok, but has some shortcomings:

  • it's unclear to the user which files they can safely delete
  • potential problems with DBDIR on non-local storage -> there are occasional problems with people on NFS -> SQLITE files on CIFS seem to be unsupported
  • different performance requirements: -> the cache files have different access patterns than the images themselves
  • sharing an image directory across different computers can lead to subtle

problems (e.g. different thumbnail size)

Proposed changes to the file layout

DBDIR/index.xml
DBDIR/CategoryImages
XDG_CACHE_DIR/kphotoalbum/<db>/exif-info.db
XDG_CACHE_DIR/kphotoalbum/<db>/thumbnails/
XDG_CACHE_DIR/kphotoalbum/<db>/videoThumbnails/
XDG_DATA_DIR/kxmlgui5/kphotoalbum/kphotoalbumui.rc
XDG_DATA_DIR/kphotoalbum/<db>/layout.dat
XDG_CONFIG_DIR/kphotoalbumrc

There are different proposals for the <db> part in some of the directories. The value is derived from the index.xml path.

  1. MD5 hash of the path (e.g. ~/.cache/kphotoalbum/0ada08745b07902c10efb58eb9f1833b) → Strongly not favored by Robert, not obviously related to original path
  2. Repeated path (e.g. ~/.cache/kphotoalbum/home/johannes/Pictures) → disliked by all, long paths
  3. Mangled path (e.g. ~/.cache/kphotoalbum/home|johannes|Pictures) → disliked by Johannes and Tobias, long paths and yet another escaping mechanism
  4. Web-like path (e.g. ~/.cache/kphotoalbum/12345-Pictures) → disliked by Johannes, yet another escaping mechanism
  5. Shortened path (e.g. ~/.cache/kphotoalbum/Pictures); like proposal 2, but with the $HOME prefix removed. This deteriorates into proposal 2 for paths outside the home directory of the user. → no feedback yet
  6. Introduce user-defined collection name, defaulting to the base name of the database directory (e.g. ~/.cache/kphotoalbum/Pictures) or to a UUID

Usage scenarios - pitfalls

Any solution has to be able to support the following ways to set up the database.

Image directory/database on remote storage (NFS)

This is a major reason for changing the directory structure as outlined above, and the main scenario besides local storage.
Thumbnails are on (usually fast, low-latency) local storage.

Shared remote image directory/database with local non-shared thumbnails

This scenario is a reasonably safe way of sharing the image directory between different machines and/or users.
Thumbnails are on (usually fast, low-latency) local storage in a size that is optimal for the given machine.

Shared remote image directory/database with shared remote thumbnails

Some users currently share the thumbnail data across different machines alongside the image directory.
This scenario helps users that have slower network drives, because the thumbnail data only has to be created once.
This use-case still needs to be supported. Manual setup (creating symlinks) may be required for those users.

Related Objects

StatusAssignedTask
OpenNone
OpenNone
johanneszarl updated the task description. (Show Details)Apr 7 2020, 9:45 PM

I'm not sure that there is a really good way to do this. One other possibility would be a GUID, with XDG_CONFIG_DIR/kphotoalbumrc containing a mapping. That would also allow (if someone really wanted to do it) multiple configurations pointing to the same image collection.

I think we should allow the DBDIR to be at least optionally read-only. That means moving index.xml into XDG_CONFIG_DIR, again at least as an option. That would be useful for teams sharing an image collection and allow concurrent instances of KPA to run against the same images. I also think that a per-database kphotoalbumui.rc files overriding the user's standard one might be useful.

Finally, I'd like to allow multiple directories to constitute one database (that really needs moving index.xml and friends into the config directory rather than the images directory). That would be useful if an image collection spans multiple filesystems (which it does on my laptop -- 4 TB isn't enough). Currently I use bind mounts to handle that, but it's not the nicest solution.

johanneszarl triaged this task as Normal priority.Sep 5 2022, 9:56 PM

Observe, IIRC the videothumbnails directory is not just a cache. If I choose to assign a custom image as the static one in the thumbnailview, then it is also written into that directory.

Besides that I don't have any strong feelings about any of the above, except that a migration path should be automatic.

johanneszarl moved this task from Backlog to For V6 on the KPhotoAlbum board.Oct 8 2023, 6:58 PM