Quite some KDevelop autotests currently fail on KDE CI with a SEGV in libLLVM. Looking closer it can be seen that currently 2 different libLLVM instances are pulled into the same process and obviously result in symbol clashes:
- /usr/lib/x86_64-linux-gnu/libLLVM-3.6.so.1, as linked to by mesa
- /usr/lib/llvm-3.9/lib/libLLVM-3.9.so.1, as linked to by KDevelop (plugins)
Why are there two different libLLVM versions even installed on the CI docker images? Because someone (perhaps KDevelop developers even, sysadmin does not remember) asked for some more recent version than what is coming with Ubuntu 15.10, and so the Dockerfile explicitely adds some repo with e.g. llvm-toolchain-wily-3.9 (see https://cgit.kde.org/sysadmin/ci-master-config.git/tree/docker/ubuntu-wily-slave/Dockerfile).
This might have worked for NOGUI tests perhaps (?), as mesa then was not triggered to load any rendering plugins using llvmpipe stuff, so no symbol clash happened. But the failing tests all seem to include QWidget/QApplication stuff, so the bad things are to happen.
How can this be solved?
Question to KDevelop maintainers:
Does KDevelop really need llvm 3.9, or is stock 3.6 also okay, so the same lib can be used by mesa and kdevelop libs/plugins?
Who else might have requested newer llvm version? lxr.kde.org showed hits only for "kdevelop/languages/clang" and "clazy"
Any other idea how to solve this? Seems Debian hit this for some reason as well:
[21:04] <tosky> it happened with the debian package of kdevelop, they had to force a specific version (the one used by mesa) because the default -dev (unversioned) was not the one used by mesa
https://bugs.freedesktop.org/show_bug.cgi?id=93103 talks about some RTLD_LOCAL things (no clue myself about symbol visibility and lib loading, so just pointing to this as related).
Example of a backtrace showing mix-up of symbols:
#0 0x7fa88b83b026 in llvm::cl::AddLiteralOption(llvm::cl::Option&, char const*) (/usr/lib/llvm-3.9/lib/../lib/libLLVM-3.9.so.1+0x611026) #1 0x7fa869cfa198 in llvm::PassRegistry::enumerateWith(llvm::PassRegistrationListener*) (/usr/lib/x86_64-linux-gnu/libLLVM-3.6.so.1+0x720198) #2 0x7fa8699ca685 (/usr/lib/x86_64-linux-gnu/libLLVM-3.6.so.1+0x3f0685) #3 0x7fa8a8e965c9 (/lib64/ld-linux-x86-64.so.2+0x105c9) #4 0x7fa8a8e966da (/lib64/ld-linux-x86-64.so.2+0x106da) #5 0x7fa8a8e9b596 (/lib64/ld-linux-x86-64.so.2+0x15596) #6 0x7fa8a8e96473 (/lib64/ld-linux-x86-64.so.2+0x10473) #7 0x7fa8a8e9a9b2 (/lib64/ld-linux-x86-64.so.2+0x149b2) #8 0x7fa89f88bfc8 (/lib/x86_64-linux-gnu/libdl.so.2+0xfc8) #9 0x7fa8a8e96473 (/lib64/ld-linux-x86-64.so.2+0x10473) #10 0x7fa89f88c62c (/lib/x86_64-linux-gnu/libdl.so.2+0x162c) #11 0x7fa89f88c060 in dlopen (/lib/x86_64-linux-gnu/libdl.so.2+0x1060) #12 0x7fa8a7f4d65f in dlopen (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x3765f) #13 0x7fa88f1c1438 (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1+0x42438) #14 0x7fa88f1c0abf (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1+0x41abf) #15 0x7fa88f19c8cf (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1+0x1d8cf) #16 0x7fa88f198696 in glXGetFBConfigs (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1+0x19696) #17 0x7fa88f199951 in glXChooseFBConfigSGIX (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1+0x1a951) #18 0x7fa86ffce679 in qglx_findConfig(_XDisplay*, int, QSurfaceFormat, bool, int) glxconvenience/qglxconvenience.cpp:192 #19 0x7fa86ffcea07 in qglx_findVisualInfo(_XDisplay*, int, QSurfaceFormat*, int) glxconvenience/qglxconvenience.cpp:237 #20 0x7fa86ffc945a in QXcbGlxWindow::createVisual() /home/jenkins/sources/qt5/kf5-qt5/qtbase/src/plugins/platforms/xcb/gl_integrations/xcb_glx/qxcbglxwindow.cpp:61 #21 0x7fa872fb7671 in QXcbWindow::create() /home/jenkins/sources/qt5/kf5-qt5/qtbase/src/plugins/platforms/xcb/qxcbwindow.cpp:426 #22 0x7fa872f9f613 in QXcbIntegration::createPlatformWindow(QWindow*) const /home/jenkins/sources/qt5/kf5-qt5/qtbase/src/plugins/platforms/xcb/qxcbintegration.cpp:210 #23 0x7fa8a0d7f4ec in QWindowPrivate::create(bool) kernel/qwindow.cpp:404 #24 0x7fa8a0d7fb9a in QWindow::create() kernel/qwindow.cpp:582
Other backtraces are ending only in libLLVM-3.9.so.1, but chances are those are also induced by symbol mixup:
#0 0x7fb39af75368 (/usr/lib/llvm-3.9/lib/../lib/libLLVM-3.9.so.1+0x5c5368) #1 0x7fb3f05745c9 (/lib64/ld-linux-x86-64.so.2+0x105c9) #2 0x7fb3f05746da (/lib64/ld-linux-x86-64.so.2+0x106da) #3 0x7fb3f0579596 (/lib64/ld-linux-x86-64.so.2+0x15596) #4 0x7fb3f0574473 (/lib64/ld-linux-x86-64.so.2+0x10473) #5 0x7fb3f05789b2 (/lib64/ld-linux-x86-64.so.2+0x149b2) #6 0x7fb3ea073fc8 (/lib/x86_64-linux-gnu/libdl.so.2+0xfc8) #7 0x7fb3f0574473 (/lib64/ld-linux-x86-64.so.2+0x10473) #8 0x7fb3ea07462c (/lib/x86_64-linux-gnu/libdl.so.2+0x162c) #9 0x7fb3ea074060 in dlopen (/lib/x86_64-linux-gnu/libdl.so.2+0x1060) #10 0x7fb3ef62b65f in dlopen (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x3765f) #11 0x7fb3eb10a028 in QLibraryPrivate::load_sys() plugin/qlibrary_unix.cpp:233 #12 0x7fb3eb104fc7 in QLibraryPrivate::load() plugin/qlibrary.cpp:538 #13 0x7fb3eb1052ec in QLibraryPrivate::loadPlugin() plugin/qlibrary.cpp:586 #14 0x7fb3eb0fa718 in QPluginLoader::load() plugin/qpluginloader.cpp:238 #15 0x7fb3ebbe9d0f in KPluginLoader::load() /home/jenkins/sources/kcoreaddons/kf5-qt5/src/lib/plugin/kpluginloader.cpp:184 #16 0x7fb3ebbe9a6c in KPluginLoader::instance() /home/jenkins/sources/kcoreaddons/kf5-qt5/src/lib/plugin/kpluginloader.cpp:166 #17 0x7fb3ebbe9216 in KPluginLoader::factory() /home/jenkins/sources/kcoreaddons/kf5-qt5/src/lib/plugin/kpluginloader.cpp:111 #18 0x7fb3ee2ee388 in KDevelop::PluginController::loadPluginInternal(QString const&) /home/jenkins/sources/kdevplatform/kf5-qt5/shell/plugincontroller.cpp:544