Use readelf to find project dependencies
ClosedPublic

Authored by apol on Oct 7 2017, 10:59 AM.

Details

Summary

We were using a link.txt file that cmake used to generate, on newer cmake
versions it doesn't anymore.
Instead use readelf, much like androiddeployqt does, to extract the
depenencies.
Catch: It relies on having all the binaries being at the same subdirectory,
which is the default in ECM since not long ago.

Test Plan

Build kirigamigallery with it

Diff Detail

Repository
R240 Extra CMake Modules
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
apol created this revision.Oct 7 2017, 10:59 AM
Restricted Application added projects: Frameworks, Build System. · View Herald TranscriptOct 7 2017, 10:59 AM
apol updated this revision to Diff 20434.Oct 7 2017, 11:24 AM

remove unrelated change I introduced

apol added a reviewer: aacid.Oct 8 2017, 5:49 PM
aacid requested changes to this revision.Oct 19 2017, 11:29 PM

This breaks KTuberling build for me, libKF5ConfigCore.so is not added to the apk anymore.

Two random guesses since i'm not sure i understood the code enough:

  • does readelf need to be "the android sdk one" or any readelf works?
  • if readelf is being executed in cmake time isn't that too early since the binary (i.e. ktuberling_mobile) doesn't exist yet and you can't readelf it?

Please ignore me if i'm saying stupid stuff :D

This revision now requires changes to proceed.Oct 19 2017, 11:29 PM
apol updated this revision to Diff 21777.Nov 2 2017, 2:25 PM

Make sure we get the dependencies from the prefix as well

aacid added a comment.Nov 12 2017, 6:43 PM

I'm still not getting libkconfigcore.so into the apk with this

apol added a comment.Nov 14 2017, 5:19 PM

I've done a build of with and without and I don't see such differences:

$ zipinfo ktuberling-patch.apk > patch
$ less patch 
$ zipinfo ktuberling-nopatch.apk > nopatch
$ diff patch nopatch 
1,2c1,2
< Archive:  ktuberling-patch.apk
< Zip file size: 31726983 bytes, number of entries: 15236
---
> Archive:  ktuberling-nopatch.apk
> Zip file size: 31725709 bytes, number of entries: 15235
5,6c5,6
< -rw----     2.4 fat  1750077 b- defN 80-000-00 00:00 META-INF/CERT.SF
< -rw----     2.4 fat  1750034 b- defN 80-000-00 00:00 META-INF/MANIFEST.MF
---
> -rw----     2.4 fat  1749983 b- defN 80-000-00 00:00 META-INF/CERT.SF
> -rw----     2.4 fat  1749940 b- defN 80-000-00 00:00 META-INF/MANIFEST.MF
8c8
< -rw----     2.4 fat   715326 b- defN 80-000-00 00:00 assets/--Added-by-androiddeployqt--/qt_cache_pregenerated_file_list
---
> -rw----     2.4 fat   715294 b- defN 80-000-00 00:00 assets/--Added-by-androiddeployqt--/qt_cache_pregenerated_file_list
75,76c75
< -rw----     2.4 fat    11733 b- defN 80-000-00 00:00 assets/share/ECM/toolchain/Android.cmake
---
> -rw----     2.4 fat    11723 b- defN 80-000-00 00:00 assets/share/ECM/toolchain/Android.cmake
78c77
< -rw----     2.4 fat     1551 b- defN 80-000-00 00:00 assets/share/ECM/toolchain/specifydependencies.cmake
---
> -rw----     2.4 fat     1618 b- defN 80-000-00 00:00 assets/share/ECM/toolchain/specifydependencies.cmake
15238,15239c15237,15238
< -rw-rw-rw-  2.3 unx  1961236 b- stor 80-000-00 00:00 resources.arsc
< 15236 files, 63480003 bytes uncompressed, 28709886 bytes compressed:  54.8%
---
> -rw-rw-rw-  2.3 unx  1961148 b- stor 80-000-00 00:00 resources.arsc
> 15235 files, 63478118 bytes uncompressed, 28708802 bytes compressed:  54.8%

I either need more information or we go forward with this and we set infrastructure in the CI to make sure this issue doesn't happen.

In D8173#167776, @apol wrote:

I either need more information

I'm happy to provide any information you want, i just don't know what you need other than KConfigCore.so is in the apk without your patch and not with your patch.

apol added a comment.Nov 15 2017, 12:03 AM
In D8173#167835, @aacid wrote:
In D8173#167776, @apol wrote:

I either need more information

I'm happy to provide any information you want, i just don't know what you need other than KConfigCore.so is in the apk without your patch and not with your patch.

Here's a little patch that you can apply and will offer a bit more information:

diff --git a/toolchain/specifydependencies.cmake b/toolchain/specifydependencies.cmake
index 60b7e94..3406d1e 100644
--- a/toolchain/specifydependencies.cmake
+++ b/toolchain/specifydependencies.cmake
@@ -1,3 +1,4 @@
+message(STATUS "executing: readelf --wide --dynamic ${TARGET}")
 execute_process(COMMAND readelf --wide --dynamic ${TARGET} ERROR_VARIABLE readelf_errors OUTPUT_VARIABLE out RESULT_VARIABLE result)
 
 if (NOT result EQUAL 0)
@@ -11,13 +12,16 @@ foreach(line ${lines})
     if(matched)
         if(EXISTS "${OUTPUT_DIR}/${CMAKE_MATCH_1}")
             list(APPEND extralibs "${OUTPUT_DIR}/${CMAKE_MATCH_1}")
-        endif()
-        if(EXISTS "${EXPORT_DIR}/lib/${CMAKE_MATCH_1}")
+        elseif(EXISTS "${EXPORT_DIR}/lib/${CMAKE_MATCH_1}")
             list(APPEND extralibs "${EXPORT_DIR}/lib/${CMAKE_MATCH_1}")
+        else()
+            message(STATUS "could not find ${line} in ${EXPORT_DIR}/lib or ${OUTPUT_DIR}")
         endif()
     endif()
 endforeach()
 
+message(STATUS "extra libs: ${extralibs}")
+
 if(extralibs)
     string(REPLACE ";" "," libs "${extralibs}")
     set(extralibs "\"android-extra-libs\": \"${libs}\",")

You should get something like (this is a valid output, which you aren't having):

Generating ktuberling_mobile with /opt/Qt/5.9.1/android_armv7/bin/androiddeployqt
-- executing: readelf --wide --dynamic /home/user/build/ktuberling/ktuberling_mobile
-- could not find  0x00000001 (NEEDED)                     Shared library: [libgnustl_shared.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [liblog.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libz.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libdl.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Svg.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Multimedia.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Xml.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Widgets.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Gui.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Network.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libQt5Core.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libstdc++.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libm.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- could not find  0x00000001 (NEEDED)                     Shared library: [libc.so] in /opt/kdeandroid-deps/lib or /home/user/build/ktuberling
-- extra libs: /opt/kdeandroid-deps/lib/libKF5ConfigCore.so
Generating Android Package

Then we see how to figure it out.

apol updated this revision to Diff 22496.Nov 17 2017, 12:27 AM

Pass LANG=C to readelf so it doesn't appear in catalan (or anything other than C) and pass ECM_ADDITIONAL_FIND_ROOT_PATH to look for dependencies

apol updated this revision to Diff 22498.Nov 17 2017, 12:51 AM

Added "" around a variable that may contain ;

still not working in my setup
the setenv seems to not be working
and complains with

  • could not find 0x00000001 (NEEDED) Shared library: [libKF5ConfigCore.so] in /home/tsdgeos/devel/kde/ktuberling/export//lib or /home/tsdgeos/devel/kde/ktuberling/build_android

which is true, the library is not in any of those two paths

apol added a comment.Nov 29 2017, 2:29 PM

Okay, I gave it another unsuccesful go. Will have to test together if you have some time.

I tried putting everything in a different prefix like this:
KConfig:

cmake -DCMAKE_INSTALL_PREFIX=/home/user/prefixfw -DCMAKE_PREFIX_PATH="$QT_ANDROID;/opt/kdeandroid-deps" -DCMAKE_BUILD_TYPE=debug -DCMAKE_TOOLCHAIN_FILE=/opt/kdeandroid-deps/share/ECM/toolchain/Android.cmake -DKF5_HOST_TOOLING=/opt/nativetooling/lib/x86_64-linux-gnu/cmake/ -DANDROID_SDK_BUILD_TOOLS_REVISION=21.1.2 /home/user/kconfig/

KTuberling:

cmake -DCMAKE_INSTALL_PREFIX=/home/user/prefixapp -DECM_ADDITIONAL_FIND_ROOT_PATH="$QT_ANDROID;/opt/kdeandroid-deps;/home/user/prefixfw" -DCMAKE_BUILD_TYPE=debug -DCMAKE_TOOLCHAIN_FILE=/opt/kdeandroid-deps/share/ECM/toolchain/Android.cmake -DKF5_HOST_TOOLING=/opt/nativetooling/lib/x86_64-linux-gnu/cmake/ -DANDROID_SDK_BUILD_TOOLS_REVISION=21.1.2 -DQTANDROID_EXPORTED_TARGET=ktuberling_mobile -DANDROID_APK_DIR=/home/user/ktuberling/android_data/ /home/user/ktuberling/

Successfully found kconfig:

~/build/ktuberling$ cat ktuberling_mobile-deployment.json
{
   "qt": "/opt/Qt/5.9.1/android_armv7",
   "sdk": "/opt/android-sdk-linux",
   "ndk": "/opt/android-ndk",
   "toolchain-prefix": "arm-linux-androideabi",
   "tool-prefix": "arm-linux-androideabi",
   "toolchain-version": "4.9",
   "ndk-host": "linux-x86_64",
   "target-architecture": "armeabi-v7a",
   "application-binary": "/home/user/build/ktuberling/ktuberling_mobile_build_apk//libs/armeabi-v7a/libktuberling_mobile.so",
   "qml-root-path": "/home/user/ktuberling",
   "android-extra-libs": "/home/user/prefixfw/lib/libKF5ConfigCore.so",
   "android-extra-plugins": "/home/user/prefixapp/share",
   "android-package-source-directory": "/home/user/ktuberling/android_data/",
   "sdkBuildToolsRevision": "21.1.2"
}
mart added a comment.Dec 20 2017, 11:20 AM

here the change works perfectly..
right now, we have building on android broken for everybody, since months, this is not acceptable :/

apol updated this revision to Diff 24185.Dec 20 2017, 8:24 PM

Went to albert's place and figured it out

aacid accepted this revision.Dec 20 2017, 8:24 PM
This revision is now accepted and ready to land.Dec 20 2017, 8:24 PM
This revision was automatically updated to reflect the committed changes.