A variety of people have asked how the speed of my Lisp Fourier Transform Library compares to an FFI call to the FFTW Library. I downloaded the FFTW Library, built it, and installed it. Then, I tried to get someone else’s FFI wrappers for the FFTW Library going. I worked on this for two hours several weeks ago. Today, I put almost an hour into it. If it’s going to get done, I’m just going to have to write my own wrapper.
Here’s the problem…. package dependencies. This wouldn’t be so much of a problem if all of the packages were in the Cliki and thus accessible to ASDF-Install. Most of them are not.
To install fftw-ffi, I needed to install ch-image and clem. One of those depended on ch-util. That (and fftw-ffi) depended on ch-asdf. That depended on smarkup. Something there depended on cl-typesetting which depends on cl-pdf. The ch-asdf also depended on cl-bibtex and ch-bib (who knows why? or where to find it?). And, something in the above depended on cl-fad. CL-FAD was the only one in that list that ASDF-Install could find on its own. Several of the typesetting and PDF libraries along with the ch-image library did not build properly. There was much reassuring ASDF-Install that everything was going to be okay even if there were no PGP signatures for half of this or if the PGP signatures didn’t jive with the owner of the website. There was also much reassuring ASDF-Install that I wanted it to try to forge ahead even if it couldn’t build cl-bibtex for me.
I couldn’t find a list of projects supported by clbuild, so I went to build darcs to fetch clbuild. Did you know darcs needs the GNU Multi-precision Library? I’m assuming it uses its own hashing functions that want big integers or something, but I wasn’t really expecting all of that. So, after a half hour getting clbuild, it knows nothing about fftw-ffi.
Alright, maybe I managed to get enough stuff installed that I can just load fftw-ffi now? No dice. The fftw-ffi seems to have been generated with some tool that spits out an XML file and a corresponding C file. Then, it tries to use special ASDF tricks to get the C file compiled. Alas, these blow up on me.
don't recognize component type FFTW-FFI-GCC-XML-C-SOURCE-FILE
Erf. Defeated.
It is very easy to add projects to your clbuild list, just add the line to clbuild/wnpp-projects. See their FAQ.
Yes, but it’s not going to know where to track down cl-bibtex and ch-asdf and all of the other bits. And, it’s probably not going to know any better than I how to make the funky ASDF types in FFTW-FFI work. I was hoping someone else already suffered through that so I wouldn’t have to.
OTOH, I won’t have to do much FFI to just get some speed numbers on FFTW through CFFI.
And you tried this on Linux which is the best supported platform for CL package managers. Try installing these packages on Win32 and see how far you get; because none of the package managers work in Windows.
I maintain lispbuilder-sdl and make sure that it, and all its dependencies (CFFI, TRIVIAL-GARBAGE, BABEL etc.), can be installed using Edi Weitz’s Lisp Starter Pack.
I think there should be a package list of shame; to shame lispers who release packages that cannot be installed easily.
Well, I tried this under MacOSX… but yes, much easier than Win32. This is why I cringe each time I add something to my DEPENDS-ON lists.
Shameless plug: http://libcl.com contains ch-image and clem and all their dependencies.
As for darcs, I generally download one of the statically-linked binaries.
Not sure what the ASDF error means. It looks like the operation is being used before ASDF sees its definition. If you have experience with C, you might have better luck generating the shared library manually.
Well, I will certainly take a look at LibCL. Thank you.
FWIW, my opinion is that one should avoid long library dependency chains in CL. The user community for libraries is pretty small and, relative to other programming languages, the set of deployment platforms is large (the cross-product of lisp implementations by operating systems, roughly speaking, whereas many languages now have only one implementation). My feeling is that this means that one should avoid a chain that’s longer than one or two hops. Many (most?) libraries involving a FFI fail this test.
I think FFI adds a whole new layer of pain to Lisp packages. Installing some libraries on some OSes in a way that FFI can use them is just dreadful. Plus, unless the person who made the FFI interface had the energy and access to test it on all sorts of platforms, there’s always something that needs to be tweaked so it knows the library extension (.so .dylib .dll ???) for your system or the name of the library or the framework (friggin’ NextSTeP wackos) or cetera.
I’ve been trying to keep most of my code dependent on, at most, cl-opengl or cl-png or vecto. Vecto has quite a few dependencies, but they are all ASDF-Install-able and all pure Lisp, so they’re fairly easy to gather up. cl-opengl and cl-png are both FFI. *shrug*
I don’t think the problem is with FFIs or packages per se, just poorly written ones. The fftw-ffi stuff has bitrotted as I haven’t used it in a long time. I’ll take this post as a bug report and see if it can be quickly cleaned up.
Well, if you’re interested in trying to track down what’s what. I’d say the biggest problem is that your packages have a bunch of dependencies. Every step I took, I found one more package I needed first. ASDF-Install couldn’t find them, so I had to track each one down with Google.
I might have gotten away with only a third of the dependencies if things failed gracefully when the docs won’t build.
The big problem with FFTW-FFI itself was getting the gccxml portion of the ASD file going.
I should also say that I’ve run into this dependency chasing with various other people’s packages in the past. And, I’ve run into similar situations with C libraries and Java libraries, too. It’s not just you. And, it’s not just Lisp.
Pat, there is a new version of fftw-ffi that doesn’t require gcc-xml-ffi (or anything else for that matter), other than fftw3. You can find it at: http://git.cyrusharmon.org/cgi-bin/gitweb.cgi?p=fftw-ffi.git;a=summary
If you want to regenerate the ffi glue, you can use a more current gcc-xml-ffi (found on the above git repo) and asdf load fftw-ffi-gen and it should recreate the bindings.
I don’t think that’s my problem. I think it’s that ASDF doesn’t know what to do with the special forms you have in the defsystem that relate to turning the XML into something useable by Lisp.
Right, and the new version has all that stuff built into it. You only need those special forms for loading fftw-ffi-gen. fftw-ffi should asdf-load out of the box now. if it doesn’t that’s a bug.
This is really a problem of some poor design decisions on my part made long ago, coupled with some unneeded dependencies. Fix those design decisions, as I had done in my other gcc-xml-ffi-using libraries, and fix the bogus dependencies and the problem goes away.
I certainly accept and appreciate your criticisms of the fftw-ffi library, but they were fixable problems with my particular implementation. While the problems were many, they hardly justify an indictment of the entire approach to utilizing functionality that isn’t written in lisp.
If I want to use Zach’s PNG library on some platform, I hop into Lisp and ASDF-Install ZPNG.
If I want to use the FFI CL-PNG, then I hop into Lisp and ASDF-Install CL-PNG only to find that I need libpng. Then, I scramble around the net to find libpng. I figure out how to compile libpng only to discover that I need zlib. I track down zlib, compile and install it. Then, I go back into Lisp and try again to ASDF-Install CL-PNG.
I’m not saying it’s particularly FFI’s fault or ASDF-Install’s fault that neither can track down foreign software and build it. I’m just saying that I far prefer when things just work.
The GHC runtime uses GMP (as of the latest released version) for bignum support. There is a GMP-free option in the next (6.12) release. The darcs binaries that I build for OS X (available on darcs.net) link libgmp statically, so you can simply put the binary in your path and away you go.
– Scott
And, I did end up downloading your binary. I forgot that Darcs was Haskell. Interesting.
unfortunately too many lisp programmers “release” broken/undocumented code.
i say “release” because a big percentage of them don’t even bother with that.
‘Get it straight outta cvs’ they say. Lack of docs is not unique to lisp, lots of opensource projects suffer from it too. The rest however is stupidly unique to lisp. People don’t bother making proper releases, they still haven’t implemented a good app for software distribution and installation (clbuild: only in lisp land could such an atrocity that pulls everything from VCS be regarded as a solution) and if this wasn’t enough, there are horrible dependency chains that you have to deal with. i work around that by not using anything that i can’t install with asdf-install. if i have to waste time in order to look for library dependencies, deal with bleeding edge VCS code, try to figure out how something works without docs, then it is not going to be used by me.
Did you know darcs needs the GNU Multi-precision Library? Iâm assuming it uses its own hashing functions that want big integers or something, but I wasnât really expecting all of that.
Darcs requires libgmp through ghc (it is written in Haskell).
*whistles* I’ve fought that sort of thing in the past, too. Of course, the difficulty varies with platform (SBCL, OpenMCL, ECL) (FreeBSD, OpenBSD, OS X). I even once had to manually (normally, I use ASDF-INSTALL) chase down all the dependencies for Hunchentoot. But, I don’t recall even that being quite as nested and crazy as what you’re doing here.
You may feel defeated, but hey, you stuck it out (far) longer than I would have lasted.