Differential D16302

improve database performance and reliability
ClosedPublic
Actions

Authored by mgallien on Oct 18 2018, 4:36 PM.

Details

Reviewers
astippich
januz
Commits
R255:f58bdd48ec15: add a test for DatabaseInterface::askRestoredTracks
R255:28769a64efc2: fix import of tracks with paths containing particular character like { }
R255:7cbe013afe91: improve management of files not modified since last scan of music files
R255:f7990d3052bb: keep all files discovered from a source and fix a minor related issue
R255:67bce5909c93: remove TracksArtists Table
R255:89bcd208ac7e: DatabaseInterface: does not emit track changed signals for newly added
R255:e84ee95d94a5: reduce overhead from AllTracksModel
R255:1163bb71e1e4: add albums and artists cache and improve cache usage
R255:6022598637ff: album artist name is directly a foreign key in Albums
R255:7c4cfa460920: fix errors when inserting multiple albums with same title
R255:e3cd920c63ca: really cache only artists and not the other people
R255:a45a570450c3: reduce overhead of AllAlbumsModel and make it synchronous
R255:6f3e8125f51e: let fix warnings from QtCreator in database tests
R255:b33cc33224bd: do not list files too early from Baloo
R255:1eb854dcda97: let the numeric unique id starts at 1 instead of 2
R255:9fa5ca30958d: use the unique fields of Albums for foreign key in Tracks
R255:40f275ae3ce5: allow tracks without artist
R255:cd8062fb91c3: let database test addOneTrackWithoutAlbumArtist have a file database
R255:e200b17ab6c5: [WIP] improve database performance and reliability
R255:fde30aa8c9df: improve database performance and reliability
Summary

let database test addOneTrackWithoutAlbumArtist have a file database

let the numeric unique id starts at 1 instead of 2

let fix warnings from QtCreator in database tests

DatabaseInterface: does not emit track changed signals for newly added

album artist name is directly a foreign key in Albums

should help to ensure greater reliability

lists of authors where not working so let's scrap this

use the unique fields of Albums for foreign key in Tracks

remove TracksArtists Table

no longer store tracks count and single disc states in albums table

reduce overhead from AllTracksModel

reduce overhead of AllAlbumsModel and make it synchronous

do not list files too early from Baloo

keep all files discovered from a source and fix a minor related issue

add a test for DatabaseInterface::askRestoredTracks

add albums and artists cache and improve cache usage

improve management of files not modified since last scan of music files

CCBUG: 389136
CCBUG: 396607
CCBUG: 396840

Test Plan

tests are OK and Elisa application is still working

Diff Detail

Repository
R255 Elisa
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
mgallien requested review of this revision.Oct 18 2018, 4:36 PM
mgallien created this revision.
mgallien updated this revision to Diff 45394.Nov 13 2018, 6:38 AM
  • reduce overhead from AllTracksModel
  • reduce overhead of AllAlbumsModel and make it synchronous
  • do not list files too early from Baloo
  • keep all files discovered from a source and fix a minor related issue
  • add a test for DatabaseInterface::askRestoredTracks
  • add albums and artists cache and improve cache usage
  • improve management of files not modified since last scan of music files

improve memory usage (after first scan run) by 40%

startup seems faster

mgallien retitled this revision from [WIP] improve database performance and reliability to improve database performance and reliability.Nov 13 2018, 6:38 AM
mgallien edited the summary of this revision. (Show Details)
mgallien updated this revision to Diff 45668.Nov 17 2018, 2:14 PM
mgallien edited the summary of this revision. (Show Details)
  • reduce overhead from AllTracksModel
  • reduce overhead of AllAlbumsModel and make it synchronous
  • do not list files too early from Baloo
  • keep all files discovered from a source and fix a minor related issue
  • add a test for DatabaseInterface::askRestoredTracks
  • add albums and artists cache and improve cache usage
  • improve management of files not modified since last scan of music files

rebase on master

mgallien updated this revision to Diff 45670.Nov 17 2018, 3:14 PM
  • allow tracks without artist
astippich added a comment.Nov 17 2018, 3:17 PM

Since this revision is quite large, it's probably difficult to review it properly. I'm going to let it run a little and report back in case I find any issues.

mgallien added a comment.Nov 17 2018, 8:48 PM
In D16302#361032, @astippich wrote:

Since this revision is quite large, it's probably difficult to review it properly. I'm going to let it run a little and report back in case I find any issues.

I understand that it is quite big but each commit is small and atomic. I have also taken care to ensure that any intermediate point is compiling and tests (except the newly landed one) are OK.

mgallien updated this revision to Diff 45917.Nov 20 2018, 9:35 PM
  • fix errors when inserting multiple albums with same title
  • really cache only artists and not the other people
  • fix import of tracks with paths containing particular character like { }

fix last error when importing my music collection

mgallien added a comment.Nov 20 2018, 9:43 PM
In D16302#361032, @astippich wrote:

Since this revision is quite large, it's probably difficult to review it properly. I'm going to let it run a little and report back in case I find any issues.

I would really like to land this diff given startup after initial indexing are now much faster and give a better feeling of the application.

astippich accepted this revision.Nov 24 2018, 6:47 PM

Working good so far. Memory usage seems to have dropped a little with my smaller music collection. Tracks with an empty artist tag are also correctly displayed in the all tracks view.

You can add
CCBUG: 389136
CCBUG: 393118
CCBUG: 396607

to the summary, since that is one more step towards allowing incomplete metadata for the tracks.

This revision is now accepted and ready to land.Nov 24 2018, 6:47 PM
mgallien edited the summary of this revision. (Show Details)Nov 25 2018, 7:25 PM
Closed by commit R255:cd8062fb91c3: let database test addOneTrackWithoutAlbumArtist have a file database (authored by mgallien). · Explain WhyNov 25 2018, 7:27 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 46221

View Options

autotests/databaseinterfacetest.cpp

Loading...