Create Task
Maniphest T15211

Linking error in stellarsolver or cfitsio to zlib
Open, Needs TriagePublic
Actions

Description

Hi Hannah and Ben,

Thank you again for your help with the last issue. Now, the build gets much further. Now we are having another linking error. This time it seems that when stellarsolver gets linked, it has undefined zlib symbols with cfitsio. On my computer, this issue does not happen and I checked it out. On my system, cfitsio is created as a dynamics library that is linked to zlib. On the craft server it seems cfitsio is created as a static library, but I don't know if it includes or links to zlib properly. This linking was working properly a year ago, but we haven't been able to test it recently due to the other issues that have now been resolved. I guess there are really two questions here, why is cfitsio a dynamic library in my craft build, but a static library on the server. And second, how to we fix the linking problem, do we need to add a zlib linking step to stellarsolver, or is this a cfitsio issue?

Thanks,

Rob

Here you can see the last time I know it worked well on the binary factory server:

https://binary-factory.kde.org/job/KStars_Nightly_macos/lastStableBuild/consoleText

Here you can see the last build that failed at this step:

https://binary-factory.kde.org/job/KStars_Nightly_macos/1406/console

Here is the error I see in the server build with the undefined zlib symbols:

12:31:07 [103/104] Linking CXX shared library libstellarsolver.1.8.dylib
12:31:07 FAILED: libstellarsolver.1.8.dylib
12:31:07 : && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk -mmacosx-version-min=10.13 -dynamiclib -Wl,-headerpad_max_install_names -Wl,-rpath,/Users/packaging/Craft/BinaryFactory/macos-64-clang/lib -compatibility_version 1.0.0 -current_version 1.8.0 -o libstellarsolver.1.8.dylib -install_name @rpath/libstellarsolver.1.dylib CMakeFiles/stellarsolver.dir/stellarsolver_autogen/mocs_compilation.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/parameters.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sextractorsolver.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/internalsextractorsolver.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/externalsextractorsolver.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/onlinesolver.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/stellarsolver.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/analyse.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/aperture.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/background.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/convolve.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/deblend.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/extract.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/lutz.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/sep/util.cpp.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/engine.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/blindutils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/blind.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/solver.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/quad-utils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/matchfile.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/matchobj.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/solvedclient.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/solvedfile.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/tweak2.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/verify.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/blind/tweak.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/multiindex.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/index.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/codekd.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/starkd.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/rdlist.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/xylist.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/starxy.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/qidxfile.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/quadfile.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/scamp.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/scamp-catalog.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/wcs-xy2rd.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/wcs-rd2xy.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/sip-utils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/fit-wcs.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/sip.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/anwcs.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/wcs-resample.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/gslutils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/wcs-pv2sip.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/fitsioutils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/sip_qfits.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/fitstable.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/fitsbin.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/fitsfile.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/tic.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/starutil.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/mathutil.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/bl-sort.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/bl.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/bt.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/healpix-utils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/healpix.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/permutedsort.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/ioutils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/fileutils.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/os-features.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/an-endian.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/errors.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/log.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/datalog.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/sparsematrix.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/coadd.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/convolve-image.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/resample.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/intmap.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/histogram.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/util/histogram2d.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdint_ddd.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdint_fff.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdint_ddu.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdint_duu.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdint_dds.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdint_dss.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdtree.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdtree_dim.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdtree_mem.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/kdtree_fits_io.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/dualtree.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/dualtree_rangesearch.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/libkd/dualtree_nearestneighbour.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/anqfits.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_card.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_convert.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_error.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_header.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_image.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_table.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_time.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_tools.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_byteswap.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_memory.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_rw.c.o CMakeFiles/stellarsolver.dir/stellarsolver/astrometry/qfits-an/qfits_float.c.o -Wl,-rpath,/Users/packaging/Craft/BinaryFactory/macos-64-clang/lib /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/libcfitsio.a /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/libgsl.dylib /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/libgslcblas.dylib /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/libwcslib.a /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/QtNetwork.framework/QtNetwork /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/QtWidgets.framework/QtWidgets /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/QtConcurrent.framework/QtConcurrent -lpthread /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/QtGui.framework/QtGui /Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/QtCore.framework/QtCore && :
12:31:07 Undefined symbols for architecture x86_64:
12:31:07 "_deflate", referenced from:
12:31:07 _compress2mem_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 _compress2file_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 "_deflateEnd", referenced from:
12:31:07 _compress2mem_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 _compress2file_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 "_deflateInit2_", referenced from:
12:31:07 _compress2mem_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 _compress2file_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 "_inflate", referenced from:
12:31:07 _uncompress2mem in libcfitsio.a(zcompress.c.o)
12:31:07 _uncompress2mem_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 _uncompress2file in libcfitsio.a(zcompress.c.o)
12:31:07 "_inflateEnd", referenced from:
12:31:07 _uncompress2mem in libcfitsio.a(zcompress.c.o)
12:31:07 _uncompress2mem_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 _uncompress2file in libcfitsio.a(zcompress.c.o)
12:31:07 "_inflateInit2_", referenced from:
12:31:07 _uncompress2mem in libcfitsio.a(zcompress.c.o)
12:31:07 _uncompress2mem_from_mem in libcfitsio.a(zcompress.c.o)
12:31:07 _uncompress2file in libcfitsio.a(zcompress.c.o)
12:31:07 ld: symbol(s) not found for architecture x86_64
12:31:07 clang: error: linker command failed with exit code 1 (use -v to see invocation)
12:31:07 ninja: build stopped: subcommand failed.
12:31:07 Command ['/Users/packaging/Craft/BinaryFactory/macos-64-clang/dev-utils/bin/ninja'] failed with exit code 1
12:31:07 Action: compile for libs/stellarsolver:master FAILED
12:31:07 * Craft all failed: libs/stellarsolver after 26 seconds *

bcooksley added a comment.Jan 18 2022, 10:05 AM

Sorry i'm not much of an expert here but I suspect it could have something to do with it linking against a static version of libcfitsio:

/Users/packaging/Craft/BinaryFactory/macos-64-clang/lib/libcfitsio.a

That might be how CFITSIO is distributed of course.

My next suggested place to check, given that it seems to be using pkgconfig to find libcfitsio would be to make sure that stellarsolver is using the appropriate variables to ensure the necessary linker flags are passed to the compiler. (if memory serves right CMake sets something like XXX_CFLAGS for this when you find something via pkgconfig)

lancaster added a comment.Jan 18 2022, 11:47 AM

Yes that was my initial conclusion since on my own computer cfitsio is a dynamic library and it works. But then my question is if the binary server and my computer are using the exact same craft recipe with the same options for cfitsio, why it would install as static on the binary server, but as dynamic on my computer? Unless the server is doing something different or somehow has different settings from craft that is downloaded to a computer?

lancaster added a comment.Jan 18 2022, 4:01 PM

I guess another question I have is if cfitsio is a static library, would it have the zlib functions baked into it, or would there need to be a separate link to zlib if you link to cfitsio? On my system, cfitsio is a dynamic library and it is linked to zlib, so I don't have to link stellarsolver to zlib since cfitsio took care of that. But on the server, it doesn't seem to know the zlib functions.

Would it hurt to just link stellarsolver to zlib regardless of whether cfitsio is a static or dynamic library? So then if cfitsio is a dynamic library, both it and stellarsolver would be linked to zlib and if cfitsio is static, then stellarsolver is still linked to zlib? Is this a problem if cfitsio uses the zlib functions and stellarsolver is just using cfitsio? Would linking stellarsolver to zlib fix it for cfitsio?

Sorry if this includes too many questions. Maybe I just need to try linking stellarsolver to zlib and see what happens on the kde binary server.

vonreth added a comment.Jan 19 2022, 11:10 AM

If a static lib is used all its dependent libs must be linked too. Ideally the libcfitsio.pc would provide the required libs.
Now why do you build libcfitsio static? I tried to build it locally, it tires to build shared and fails due to the curl issue you had before.

lancaster added a comment.Jan 19 2022, 11:45 AM

Hi Hannah,

I see, thank you for that info. I definitely don’t build it statically. I was using the default craft recipe for cfitsio and that is how it installed it on the KDE binary server. On my own computer using the same recipe it installed dynamically and worked great. It failed on the server as I mentioned because of the static linking. My initial question was why does it do this on the server and how do I make it do it dynamically since that works?

vonreth added a comment.Jan 19 2022, 1:36 PM

cfitsio now builds, my craft was outdated.

What I can tell you is that you use "master" in quite some packages.
Craft will only rebuild those blueprints when the target changes so never, that why using versions is quite important.
It also allows reproducible builds.

So if something uses master, the version on the build server might be a different one than you build locally.

lancaster added a comment.Jan 19 2022, 7:43 PM

Hi Hannah, thank you for your help! So then will cfitsio now build on the binary server again as a dylib so that linking will work with stellarsolver, or do we need to do something to make that happen? I think it ran on the binary factory a couple of hours ago and it is still linking to a static cfitsio according to the log.

For the usage of master, that is not a problem with something like Stellarsolver where the code won't change that much, so I can definitely change that in the dependences like that. But for something like INDIServer, and the 3rdParty INDI drivers, the code is changing and improving literally constantly and people are always asking for new changes and fixes and updates to INDI. So the point of KStars on the Binary Factory Server is so people can get the latest nightly release with the latest kstars code. I would argue that these people also really need the latest INDI code as well. I was trying to make that happen, but it seems the only way to do that in craft is to have it do "master". Is there a way to make it always build from the latest code when KStars builds?

bcooksley added a comment.Jan 20 2022, 7:07 AM

I've initiated a Craft Builder Clean run which will eliminate any legacy artifacts laying around on the Windows and Mac builders.

lancaster added a comment.Jan 20 2022, 2:24 PM

Thanks! I will check on it later to see if it works out.

vonreth added a comment.Jan 20 2022, 2:41 PM

Maybe you could try something like https://invent.kde.org/sysadmin/binary-factory-tooling/-/blob/master/craft/enabled-projects.yaml#L289 to force rebuilds of your unreleased software.

lancaster added a comment.Jan 20 2022, 11:03 PM

would that mean editing the KStars entry to something like this?

'KStars':

buildBlueprint: "kstars"
rebuildBlueprint:
  - indiserver
  - indiserver-3rdparty
  - indiserver-3rdparty-libraries
versions:
- name: 'Nightly'
  target: 'master'
- name: 'Release'
  target: ''
  packageAppx: True
platforms:
- 'macos'
- 'win64'
lancaster added a comment.Jan 20 2022, 11:05 PM

or do you mean we would add a separate entry for INDI packages similar to the one shown above for kstars?

lancaster added a comment.Jan 20 2022, 11:13 PM

I suspect you mean my first thought, since what we really would want is nightly builds of KStars with rebuilt indi drivers inside, not nightly builds of indi outside of kstars.