Deal with failing transactions
Closed, ResolvedPublic

Description

We sometimes run into failures on write-transactions, and we should be able to deal with them gracefully:

Error:   resource1.fulltextindex   : Exception during Xapian commit_transaction: Modifications failed (DatabaseOpeningError: Couldn't write new rev file: /home/developer/.qttest/share/sink/storage/resource1/data/fulltext/v.tmp (No such file or directory)), and couldn't open at the old revision: Couldn't open /home/developer/.qttest/share/sink/storage/resource1/data/fulltext/termlist.glass read/write
Error:   resource1.default         : "ASSERT: \"false\" in file /src/sink/common/fulltextindex.cpp, line 103"
SIGABRT received

Given that we have a single writer it is not clear why we end up with those transactions failing in the first place, but we should be able to deal with the failure anyways.

Right now fulltext indexing is part of the regular indexes, and is thus expected to always work and to be available immediately.
The problem we currently have is:

  • A transaction covers e.g. 100 revisions.
  • If a commit fails we have no way of retrying the 100 revisions, unless we:
    • Fail the transaction for the entire storage layer (hard to do because we have two transactions that could fail potentially)
    • We have a codepath that rereads the last 100 revisions, and allows us thus to retry the transaction.
  • The latter codepath would also be valuable to be able to rebuild indexes and have eventually consistent indexes.

Related Objects

cmollekopf triaged this task as Normal priority.

The reason for the above crash was specific to the testsetup.
When restarting the test-setup quickly, the still running resource ended up trying to write to the index that was already deleted by the new initial setup (we always delete all data on startup).
Just deleting stuff from the client side should of course never be done in something that goes to production, so this is not a problem we want to deal with.

cmollekopf edited projects, added Kube; removed Kube (0.6).Feb 28 2018, 1:52 PM
cmollekopf closed this task as Resolved.Jul 5 2018, 3:05 PM
cmollekopf claimed this task.