Differential D12746 Diff 68453 kdevplatform/language/duchain/tests/test_topcontextstore.cpp

Changeset View

Standalone View

View Options

kdevplatform/language/duchain/tests/test_topcontextstore.cpp

  • This file was added.
1​/*
2​ * This file is part of KDevelop
3​ *
4​ * Copyright 2018 R.J.V. Bertin <rjvbertin@gmail.com>
5​ *
6​ * This program is free software; you can redistribute it and/or modify
7​ * it under the terms of the GNU Library General Public License as
8​ * published by the Free Software Foundation; either version 2 of the
9​ * License, or (at your option) any later version.
10​ *
11​ * This program is distributed in the hope that it will be useful,
12​ * but WITHOUT ANY WARRANTY; without even the implied warranty of
13​ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14​ * GNU General Public License for more details.
15​ *
16​ * You should have received a copy of the GNU General Public
17​ * License along with this program; if not, write to the
18​ * Free Software Foundation, Inc.,
19​ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
20​ */
21​
22​// with LMDB, the store is about 1.1x larger than the length of the stored values (but
23​// our LZ4 compression can reach as much as a 2x reduction in average value size).
24​// with KyotoCabinet-lzo, the store is about 1.76x smaller than the length of the stored values
25​
26​// timing results on Linux 4.14 running on an 1.6Ghz Intel N3150 and ZFSonLinux 0.7.6
27​
28​// KDEV_TOPCONTEXTS_USE_FILES:
29​// QINFO : TestTopDUContextStore::benchTopContextStore() 6000 items, max length= 38494 from sample of length 38497
30​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 115232709 total key bytes: 24000
31​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 6000 keys (last flushed): 1.583 seconds
32​// PASS : TestTopDUContextStore::benchTopContextStore()
33​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 6000 keys: 0.445 seconds
34​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
35​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 6000 keys: 0.736 seconds
36​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
37​// PASS : TestTopDUContextStore::cleanupTestCase()
38​// Totals: 21 passed, 0 failed, 0 skipped, 0 blacklisted, 3426ms
39​// ## Idem, with only 1500 keys
40​// QINFO : TestTopDUContextStore::benchTopContextStore() 1500 items, max length= 38484 from sample of length 38497
41​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 28350833 total key bytes: 6000
42​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 1500 keys (last flushed): 0.351 seconds
43​// PASS : TestTopDUContextStore::benchTopContextStore()
44​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 0.092 seconds
45​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
46​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 0.232 seconds
47​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
48​// PASS : TestTopDUContextStore::cleanupTestCase()
49​// Totals: 21 passed, 0 failed, 0 skipped, 0 blacklisted, 1171ms
50​// ## idem, on NFS:
51​// QINFO : TestTopDUContextStore::benchTopContextStore() 6000 items, max length= 38488 from sample of length 38497
52​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 115800253 total key bytes: 24000
53​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 6000 keys (last flushed): 108.579 seconds
54​// PASS : TestTopDUContextStore::benchTopContextStore()
55​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 6000 keys: 50.502 seconds
56​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
57​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 6000 keys: 27.651 seconds
58​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
59​// PASS : TestTopDUContextStore::cleanupTestCase()
60​// Totals: 21 passed, 0 failed, 0 skipped, 0 blacklisted, 194162ms
61​// QINFO : TestTopDUContextStore::benchTopContextStore() 1500 items, max length= 38462 from sample of length 38497
62​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 29375929 total key bytes: 6000
63​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 1500 keys (last flushed): 26.459 seconds
64​// PASS : TestTopDUContextStore::benchTopContextStore()
65​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 11.778 seconds
66​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
67​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 6.494 seconds
68​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
69​// PASS : TestTopDUContextStore::cleanupTestCase()
70​// Totals: 21 passed, 0 failed, 0 skipped, 0 blacklisted, 51921ms
71​// KDEV_TOPCONTEXTS_USE_LMDB:
72​// QINFO : TestTopDUContextStore::benchTopContextStore() 1500 items, max length= 38424 from sample of length 38497
73​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 29024029 total key bytes: 6000
74​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 1500 keys (last flushed): 1.585 seconds
75​// PASS : TestTopDUContextStore::benchTopContextStore()
76​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 0.126 seconds
77​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
mwolffUnsubmitted
Done

you are missing the 6k results here

mwolff: you are missing the 6k results here
rjvbbAuthorUnsubmitted
Done

I can add them of course, but from what I've seen they scale in a nicely linear fashion. My initial tests with the file-based backend showed non-linear scaling there which has now all but disappeared so I could just as well remove the 6k results, probably.

rjvbb: I can add them of course, but from what I've seen they scale in a nicely linear fashion. My…
78​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 0.517 seconds
79​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
80​// QINFO : TestTopDUContextStore::benchTopContextStoreFlush() Flushing the database to disk: 0.656 seconds
81​// PASS : TestTopDUContextStore::benchTopContextStoreFlush()
82​// total 14M
83​// 512 -rw-r--r-- 1 bertin bertin 8.0K May 8 14:19 lock.mdb
84​// 14M -rw-r--r-- 1 bertin bertin 27M May 8 14:19 data.mdb
85​// PASS : TestTopDUContextStore::cleanupTestCase()
86​// Totals: 22 passed, 0 failed, 0 skipped, 0 blacklisted, 18941ms
87​// ********* Finished testing of TestTopDUContextStore *********
88​// average LZ4 compression ratio: 1.95162; 1501 compressed of 3000
89​// ## idem, on NFS:
90​// QINFO : TestTopDUContextStore::benchTopContextStore() 1500 items, max length= 38496 from sample of length 38497
91​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 29311234 total key bytes: 6000
92​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 1500 keys (last flushed): 12.362 seconds
93​// PASS : TestTopDUContextStore::benchTopContextStore()
94​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 0.113 seconds
95​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
96​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 0.221 seconds
97​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
98​// QINFO : TestTopDUContextStore::benchTopContextStoreFlush() Flushing the database to disk: 8.763 seconds
99​// PASS : TestTopDUContextStore::benchTopContextStoreFlush()
100​// total 26M
101​// 8.0K -rw-r--r-- 1 bertin bertin 8.0K May 8 2018 lock.mdb
102​// 26M -rw-r--r-- 1 bertin bertin 26M May 8 2018 data.mdb
103​// PASS : TestTopDUContextStore::cleanupTestCase()
104​// Totals: 22 passed, 0 failed, 0 skipped, 0 blacklisted, 91060ms
105​// ********* Finished testing of TestTopDUContextStore *********
106​// average LZ4 compression ratio: 1.95829; 1499 compressed of 3000
107​
108​// ## on a 2.7Ghz i7 running Mac OS X 10.9.5
109​// KDEV_TOPCONTEXTS_USE_FILES:
110​// QINFO : TestTopDUContextStore::benchTopContextStore() 6000 items, max length= 38495 from sample of length 38497
111​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 115547442 total key bytes: 24000
112​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 6000 keys (last flushed): 1.622 seconds
113​// PASS : TestTopDUContextStore::benchTopContextStore()
114​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 6000 keys: 0.13 seconds
115​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
116​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 6000 keys: 0.924 seconds
117​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
118​// PASS : TestTopDUContextStore::cleanupTestCase()
119​// Totals: 21 passed, 0 failed, 0 skipped, 0 blacklisted, 8829ms
120​// QINFO : TestTopDUContextStore::benchTopContextStore() 1500 items, max length= 38463 from sample of length 38497
121​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 28304777 total key bytes: 6000
122​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 1500 keys (last flushed): 0.299 seconds
123​// PASS : TestTopDUContextStore::benchTopContextStore()
124​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 0.032 seconds
125​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
126​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 0.208 seconds
127​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
128​// PASS : TestTopDUContextStore::cleanupTestCase()
129​// Totals: 21 passed, 0 failed, 0 skipped, 0 blacklisted, 3508ms
130​// KDEV_TOPCONTEXTS_USE_LMDB:
131​// QINFO : TestTopDUContextStore::benchTopContextStore() 1500 items, max length= 38474 from sample of length 38497
132​// QINFO : TestTopDUContextStore::benchTopContextStore() total value bytes: 29277850 total key bytes: 6000
133​// QINFO : TestTopDUContextStore::benchTopContextStore() Writing 1500 keys (last flushed): 0.267 seconds
134​// PASS : TestTopDUContextStore::benchTopContextStore()
135​// QINFO : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 0.028 seconds
136​// PASS : TestTopDUContextStore::benchTopContextStoreRead()
137​// QINFO : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 0.035 seconds
138​// PASS : TestTopDUContextStore::benchTopContextStoreRemove()
139​// QINFO : TestTopDUContextStore::benchTopContextStoreFlush() Flushing the database to disk: 0.188 seconds
140​// PASS : TestTopDUContextStore::benchTopContextStoreFlush()
141​// total 52984
142​// 16 -rw-r--r-- 1 bertin bertin 8.0K May 8 14:44 lock.mdb
143​// 52968 -rw-r--r-- 1 bertin bertin 26M May 8 14:44 data.mdb
144​// PASS : TestTopDUContextStore::cleanupTestCase()
145​// Totals: 22 passed, 0 failed, 0 skipped, 0 blacklisted, 4704ms
146​// ********* Finished testing of TestTopDUContextStore *********
mwolffUnsubmitted
Done

that's pretty bad in comparison

note that we currently just rewrite the complete file, which you can't do when you use one big data base

mwolff: that's pretty bad in comparison note that we currently just rewrite the complete file, which…
rjvbbAuthorUnsubmitted
Not Done

That's removing all entries with a flush after each remove (which should be redundant).

There *is* original code that calls QFile::remove(), hopefully only when the index is not going to be reused immediately. Rewriting an entire file becomes overwriting an existing key/value pair, which all databases tested support in much the same way it is done with files.

rjvbb: That's removing all entries with a flush after each remove (which should be redundant). There…
147​// average LZ4 compression ratio: 1.95736; 1500 compressed of 3000
148​
149​#include "test_topcontextstore.h"
150​
151​#include <unistd.h>
152​
153​#include <QTest>
154​#include <QDir>
155​
156​#include <tests/autotestshell.h>
157​#include <tests/testcore.h>
158​
159​#include <language/duchain/duchain.h>
160​#include <language/codegen/coderepresentation.h>
161​
162​#include <language/duchain/topducontextdynamicdata_p.h>
163​#include <language/duchain/topducontextdynamicdata.h>
164​
165​QTEST_MAIN(TestTopDUContextStore)
166​
167​using namespace KDevelop;
168​
169​uint TestTopDUContextStore::testKeys = 16;
170​uint TestTopDUContextStore::benchKeys = 1500;
171​
172​static QString basePath()
173​{
174​ return globalItemRepositoryRegistry().path() + "/topcontexts/";
175​}
176​
177​void TestTopDUContextStore::initTestCase()
178​{
179​ AutoTestShell::init();
180​ TestCore::initialize(Core::NoUi);
181​
182​ DUChain::self()->disablePersistentStorage();
183​ CodeRepresentation::setDiskChangesForbidden(true);
184​ QDir().mkpath(basePath());
185​
186​ // fetch real TopDUContext data from the qresource
187​ // (data obtained from a KDevelop project of the KDevelop source).
188​ QFile qfp(QStringLiteral(":/topcontextdata.base64"));
189​ if (qfp.open(QIODevice::ReadOnly)) {
190​ realData = QByteArray::fromBase64(qfp.readAll());
191​ qfp.close();
192​ }
193​}
194​
195​void TestTopDUContextStore::cleanupTestCase()
196​{
197​#ifdef Q_OS_UNIX
198​ system(QStringLiteral("ls -lshtr %1").arg(basePath()).toLatin1().constData());
199​#endif
200​ TestCore::shutdown();
201​}
202​
203​void TestTopDUContextStore::testTopContextStore_data()
204​{
205​ QTest::addColumn<uint>("key");
206​ QTest::addColumn<QByteArray>("value");
207​ QTest::addColumn<int>("size");
208​
209​ for (uint i = 0 ; i < benchKeys ; ++i) {
210​ TopDUContextStore writer(i);
211​ QByteArray data;
212​ uint size;
213​ if (i == 0 /*|| i >= testKeys*/) {
214​ // 1st key/value pair will use the stored real data so that we can
215​ // test our own LZ4 compression scheme in the LMDB backend (random
216​ // values won't compress).
217​ // The additional values >= testKeys serve to prime the database store
218​ // to a mostly constant size.
219​ data = realData;
220​ } else {
221​ size = qrand() * 32767.0 / RAND_MAX;
222​ data.resize(size);
223​ char *d = data.data();
224​ for (uint j = 0 ; j < size ; ++j) {
225​ d[j] = qrand() * 256.0 / RAND_MAX;
226​ }
227​ }
228​ if (i < testKeys) {
229​ // we store all keys that will be used in the benchmark so the store will be primed
230​ // but only do write/read/exists/remove verifications for the requested number of items.
231​ QTest::newRow("keyval") << i << data << data.size();
232​ }
233​ if (writer.open(QIODevice::WriteOnly)) {
234​ writer.write(data.constData(), data.size());
235​ writer.commit();
236​ const auto errString = writer.errorString();
237​ if (!errString.isEmpty() && errString != "Unknown error") {
238​ qCritical() << "Failed to write entry for key" << i << "to" << writer.fileName() << ":" << writer.errorString();
239​ } else {
240​ QVERIFY(writer.flush());
241​ }
242​ } else {
243​ qCritical() << "Failed to open" << writer.fileName();
244​ }
245​ }
246​ sync();
247​}
248​
249​void TestTopDUContextStore::testTopContextStore()
250​{
251​ QFETCH(uint, key);
252​ QFETCH(QByteArray, value);
253​ QFETCH(int, size);
254​
255​ TopDUContextStore reader(key);
256​ QByteArray testval;
257​ if (reader.open(QIODevice::ReadOnly)) {
258​ testval = reader.readAll();
259​ if (testval.isEmpty() && !reader.errorString().isEmpty()) {
260​ qCritical() << "Failed to read entry for key" << key << "from" << reader.fileName() << ":" << reader.errorString();
261​ }
262​ } else {
263​ qCritical() << "Failed to open" << reader.fileName();
264​ }
265​ QCOMPARE(testval.size(), size);
266​ QCOMPARE(testval, value);
267​ QVERIFY(TopDUContextStore::exists(key));
268​ QVERIFY(TopDUContextStore::remove(key));
269​ QVERIFY(!TopDUContextStore::exists(key));
270​}
271​
272​void TestTopDUContextStore::benchTopContextStore_data()
273​{
274​ uint maxSize = 0, totalSize = 0;
275​ benchValues.reserve(benchKeys);
276​ // subsample random strings from the real topcontext data
277​ // (always from the left, there should be no interest to
278​ // randomising the offset).
279​ for (uint i = 0 ; i < benchKeys ; ++i) {
280​ uint size = qrand() * double(realData.size()) / RAND_MAX;
281​ if (size > maxSize) {
282​ maxSize = size;
283​ }
284​ benchValues.append(realData.left(size));
285​ totalSize += size;
286​ if (i >= testKeys) {
287​ // remove keys that weren't removed in testTopContextStore()
288​ // so we will time writing, not overwriting
289​ TopDUContextStore::remove(i);
290​ }
291​ }
292​ sync();
293​ sleep(1);
294​ qInfo() << benchValues.size() << "items, max length=" << maxSize << "from sample of length" << realData.size();
295​ qInfo() << "\ttotal value bytes:" << totalSize << "total key bytes:" << benchKeys * sizeof(uint);
296​}
297​
298​// benchmarks don't use QBENCHMARK because they're not suitable for being
299​// called multiple times.
300​void TestTopDUContextStore::benchTopContextStore()
301​{
302​ QElapsedTimer timer;
303​ timer.start();
304​ for (uint i = 0 ; i < benchKeys ; ++i) {
305​ TopDUContextStore writer(i);
306​ if (writer.open(QIODevice::WriteOnly)) {
307​ QByteArray data = benchValues.at(i);
308​ writer.write(data.constData(), data.size());
309​ writer.commit();
310​ if (i == benchKeys - 1) {
311​ writer.flush();
312​ }
313​ }
314​ }
315​ qInfo() << "Writing" << benchKeys << "keys (last flushed):" << timer.elapsed() / 1000.0 << "seconds";
316​}
317​
318​#include <unistd.h>
319​void TestTopDUContextStore::benchTopContextStoreRead()
320​{
321​ QElapsedTimer timer;
322​ timer.start();
323​ for (uint i = 0 ; i < benchKeys ; ++i)
324​ {
325​ TopDUContextStore reader(i);
326​ if (reader.open(QIODevice::ReadOnly)) {
327​#if defined(KDEV_TOPCONTEXTS_USE_FILES) && !defined(KDEV_TOPCONTEXTS_DONT_MMAP)
328​ uchar* data = reader.map(0, reader.size());
329​ reader.commit();
330​#else
331​ QByteArray data = reader.readAll();
332​#endif
333​ }
334​ }
335​ qInfo() << "Reading" << benchKeys << "keys:" << timer.elapsed() / 1000.0 << "seconds";
336​}
337​
338​void TestTopDUContextStore::benchTopContextStoreRemove()
339​{
340​ QElapsedTimer timer;
341​ timer.start();
342​ for (uint i = 0 ; i < benchKeys ; ++i)
343​ {
344​ TopDUContextStore::remove(i);
345​ }
346​ qInfo() << "Removing" << benchKeys << "keys:" << timer.elapsed() / 1000.0 << "seconds";
347​}
348​
349​void TestTopDUContextStore::benchTopContextStoreFlush()
350​{
351​#ifndef KDEV_TOPCONTEXTS_USE_FILES
352​ QElapsedTimer timer;
353​ timer.start();
354​ TopDUContextStore writer(0);
355​ if (writer.open(QIODevice::WriteOnly)) {
356​ writer.flush();
mwolffUnsubmitted
Done

O_o make it a separate data file and read it from a resource instead of embedding it manually

mwolff: O_o make it a separate data file and read it from a resource instead of embedding it manually
rjvbbAuthorUnsubmitted
Done

I thought about using a qresource but the extra work seemed a bit pointless as it wouldn't reduce the size of the patch. I'd like to avoid binary files for now (until this actually begins to stand a chance to be committed).

rjvbb: I thought about using a qresource but the extra work seemed a bit pointless as it wouldn't…
357​ }
358​ qInfo() << "Flushing the database to disk:" << timer.elapsed() / 1000.0 << "seconds";
359​#endif
360​}