Differential D23008 Diff 63304 src/file/extractor/app.cpp

Changeset View

Standalone View

View Options

src/file/extractor/app.cpp

Show All 38 Lines
39#include <KFileMetaData/PropertyInfo> 39​#include <KFileMetaData/PropertyInfo>
40#include <KFileMetaData/MimeUtils> 40​#include <KFileMetaData/MimeUtils>
41 41​
42#include <unistd.h> //for STDIN_FILENO 42​#include <unistd.h> //for STDIN_FILENO
43#include <iostream> 43​#include <iostream>
44 44​
45using namespace Baloo; 45​using namespace Baloo;
46 46​
47App::App(QObject* parent) 47​App::App(Database* db)
48 : QObject(parent) 48​ : m_db(db)
49 , m_notifyNewData(STDIN_FILENO, QSocketNotifier::Read) 49​ , m_notifyNewData(STDIN_FILENO, QSocketNotifier::Read)
50 , m_input() 50​ , m_input()
51 , m_inputStream(&m_input) 51​ , m_inputStream(&m_input)
52 , m_outputStream(stdout) 52​ , m_outputStream(stdout)
53 , m_tr(nullptr) 53​ , m_tr(nullptr)
54{ 54​{
55 m_input.open(STDIN_FILENO, QIODevice::ReadOnly | QIODevice::Unbuffered ); 55​ m_input.open(STDIN_FILENO, QIODevice::ReadOnly | QIODevice::Unbuffered );
56 m_inputStream.setByteOrder(QDataStream::BigEndian); 56​ m_inputStream.setByteOrder(QDataStream::BigEndian);
Show All 11 Lines66 if (m_inputStream.status() != QDataStream::Ok) {
68 return; 68​ return;
69 } 69​ }
70 70​
71 if (!m_inputStream.commitTransaction()) { 71​ if (!m_inputStream.commitTransaction()) {
72 m_ids.clear(); 72​ m_ids.clear();
73 return; 73​ return;
74 } 74​ }
75 75​
76 Database *db = globalDatabaseInstance(); 76​ if (m_tr) {
77 if (!db->open(Database::ReadWriteDatabase)) { 77​ // Parent proccess yielded us a new batch while previous have not yet finished
78 qCritical() << "Failed to open the database"; 78​ // This should not happen, but better play safe
79 exit(1); 79​ m_tr->commit();
80​ delete m_tr;
80 } 81​ }
anthonyfieroniUnsubmitted
Not Done

processNextFile should be called till m_ids ends, why it does not happen, probably that's why @bruns reject your patch

anthonyfieroni: processNextFile should be called till m_ids ends, why it does not happen, probably that's why…
poboikoAuthorUnsubmitted
Not Done

Well, if that happens, something went wrong, and we have to do something.
As far as I can see, the options are:

  1. Ignore new batch (that would be simply return)
  2. Ignore old batch (that is done now - committing something that was extracted from the old one if there is any)
  3. Try to merge batches and process everything (that would be m_ids.append(new batch) and do not create a new transaction). This might require some additional housekeeping though, as we do not want resulting transaction to be larger than batchSize. And we probably do not want to do our own splitting here - as it would duplicate work done in the parent baloo_file process.

However, since ignored batch do not get removed from ContentIndexingDB, our parent process baloo_file will retry those documents eventually. I believe it should be pretty safe to just drop one of the batches here.

poboiko: Well, if that happens, something went wrong, and we have to do something. As far as I can see…
81 82​
82 Q_ASSERT(m_tr == nullptr); 83​ m_tr = new Transaction(m_db, Transaction::ReadWrite);
83 ​
84 m_tr = new Transaction(db, Transaction::ReadWrite); ​
85 // FIXME: The transaction is open for way too long. We should just open it for when we're 84​ // FIXME: The transaction is open for way too long. We should just open it for when we're
86 // committing the data not during the extraction. 85​ // committing the data not during the extraction.
87 86​
88 QTimer::singleShot(0, this, &App::processNextFile); 87​ QTimer::singleShot(0, this, &App::processNextFile);
89 88​
90 /** 89​ /**
91 * A Single Batch seems to be triggering the SocketNotifier more than once 90​ * A Single Batch seems to be triggering the SocketNotifier more than once
92 * so we disable it till the batch is done. 91​ * so we disable it till the batch is done.
▲ Show 20 Lines • Show All 116 Lines • Show Last 20 Lines