Thursday, August 20, 2009

DebConf thoughts, part two: On cross-compilation environments

This post is part two of my DebConf thoughts series, first post here. At DebConf we also attended three talks directly related to something we use quite often in Maemo and Mer: cross-compilation. The first was by Riku Voipio/suihkulokki about Scratchbox 2 (mirrored by me as I can't find the original URL) - which was also a talk like ours, as well as the round-table session on Multiarch and a talk by Wookey on Crossbuilding on Debian for a derived distro.

Scratchbox 2 is the successor of Scratchbox 1 - which most of you know from Maemo SDK. It is different from SB1 in many ways. It operates with the concept of a host tools distribution - tools which are used to accelerate certain parts of a compilation process. Comparing to SB1, with their devkits and such, this approach is much more flexible in terms of maintaining the tools distribution and keeping it up to date with current packages to match your target. This was previously difficult to maintain in SB1 and has been the source of much headache for developers on Maemo to having to deal with old and broken tools which no longer was able to build 'modern' packages.

Multiarch is the term being used to refer to the capability of a system to run applications of multiple different binary targets on the same system. This means that the different architecture files (libraries, binaries, development headers etc) are hosted in the same file system. The proposal also includes ability to have cross-architecture dependancies of packages. This would mean it would be easier in the same system (such as Debian) to build packages for different architectures within one system. It would however often also mean that you would need same package versions in all architectures as the one you're targetting.

This problem leads me to what I intend to discuss: The builder approach of cross-compilation vs a user environment approach for cross-compilation.

A cross-compilation environment has to user-level emulate as much as to make the cross-compilation of the Debian package succeed.

A user environment will have to provide a full environment with interactive shell, editors, etc - and emulate 'being' the target architecture system. This imposes a greater strain on emulation and is more difficult to achieve.

What most Maemo developers encounter, is the Scratchbox SDK. It is a development environment for all your needs and supports different targets both native and foreign architectures. It provides an emulated Maemo environment for developers to test in and should be enough to develop for the Maemo platform.

Marius Vollmer once wrote a post on "The cardinal sin of Scratchbox". It mentions the overreaching redirection that SB2 inherited from SB1 - that almost everything from doctools to python is redirected in the Python environment. This sin extends to the emulated Maemo environment - and that's what harms developers. The emulated Maemo environment is skewed by the redirection of the cross-compilation environment, to the point that Scratchbox Maemo has little to do with how Maemo on device works.

In Mer we began back in the Maemo Reconstructed proposal to ponder how SDK could be different. What we ended up on is that the entire Mer system should be able to run on a given host for development. We currently do this through chroots and virtual machines. This means, when you develop and test your applications - you develop them natively, first for your host architecture and then test it on your host.

The host is a 'real' Mer environment like on every other device and should behave the same no matter device or architecture. Once you're done developing, you can either send the source package to a builder farm (in our case, OpenSUSE Build Service) - which will spit out cleanly built x86, ARM etc. packages for us to put on our devices and test or cross-compile locally with an automated cross-compilation environment. With Mer, it's even possible to build for ARM on your own ARM device - it's just a Debian/Ubuntu laying underneath the surface.

We've had great success with the builder approach of cross-compilation in Mer. Might it be worth considering making - in abstract terms X86 chroot tar.gz's/VMDKs of full Maemo (themes, icons, etc) for developers to use to match the experience without inference by the cross-compilation environment?

One for development - dpkg-buildpackage is possible (coreutils instead of busybox) and one that matches a typical Maemo device. And then when you want to build for a different architecture, abstract it away in a one-liner that takes a source package as input.

No comments:

Post a Comment