Mer - a mobile Linux distribution

Wayland utilizing Android GPU drivers on glibc based systems, Part 2

noreply@blogger.com (Carsten Munk) — Wed, 08 May 2013 18:22:00 +0000

In this blog series, I am presenting a solution that I've developed that enables the use of Wayland on top of Android hardware adaptations, specifically the GPU drivers, but without actually requiring the OS to be Bionic based.

This is part 2 and will cover the actual server side (and a little bit about generic EGL implementation) of the solution. The first part can be read here. The third and last blog post will revolve around the client side solution and how you can use it today, as well as future work. There are a -lot- of links in this blog, please take a look at them to fully understand what is being explained.

This work was and is done as part of my job as Chief Research Engineer in Jolla, which develop Sailfish OS, a mobile-optimized operating system that has the flexibility, ubiquity and stability of the Linux core with a cutting edge user experience built with the renowned Qt platform.

The views and opinions expressed in this blog series are my own and not that of my employer.

The aim is to have documented the proof of concept code and published it under a "LGPLv2.1 only" license, for the benefit of many different communities and projects (Sailfish, OpenWebOS, Qt Project, KDE, GNOME, Hawaii, Nemo Mobile, Mer Core based projects, EFL, etc).

This work is done with the hope that it will attract more contribution and collaboration to bring this solution and Wayland in general into wider use across the open source ecosystem and use a large selection of reference device designs for their OS'es.

Rendering with OpenGL ES 2.0 to a screen with Android APIs

In Android, when SurfaceFlinger wants to render to the screen, it utilizes a class named FramebufferNativeWindow which it passes to eglCreateWindowSurface. As I mentioned in my previous post, on Android, when you use eglCreateWindowSurface you utilize a type/'class' named ANativeWindow. FramebufferNativeWindow implements this type. This then means it gets buffers utilizing FramebufferNativeWindow, renders to them within the OpenGL ES 2.0 implementation and queues them to be shown on the screen utilizing the same FramebufferNativeWindow.

But what happens under the hood? I'll try to explain with libhybris' "fbdev" windowing system as an example.

We're back to ANativeWindow - what libhybris' "fbdev" windowing system does is practically to do an implementation of ANativeWindow.

When a OpenGL ES 2.0 implementation wants to have a buffer to render into, it will call the dequeueBuffer method of an ANativeWindow. This usually happens upon surface creation or when you have done eglSwapBuffers and would like a new fresh buffer to render into.

You may have heard of fancy things like 'vsync' and you know that you have to follow signaling of vsync to avoid things like tearing. On the occasion that you do not have any buffers available (as some might be waiting to be posted to the framebuffer), you will need to block and wait for a non-busy buffer to be available within a dequeueBuffer implementation - don't just return NULL. Use pthread conditions and be CPU-friendly. This also makes sure you will block in eglSwapBuffers()

A quick note for implementors of ANativeWindow: Many OpenGL ES drivers are very temperamental. When it relays the information to you that it wants to set your buffer count to 4 buffers, it means that it wants 4 buffers and only to see those 4 buffers in the lifetime until usage or format changes. Mess up and it will happily crash on you - these drivers do not come with debug symbols.

When you want to allocate graphical buffers you naturally need gralloc to do so - gralloc is a module that is accessible through Android's libhardware API - in practice, gralloc is a shared object that libhardware dlopen()s, see /system/lib/hw/ for examples of these (gps, lights, sensors, etc).

When loading gralloc you will naturally get the interface of the gralloc module itself, but when you initialize gralloc, you get an allocation device interface where you can allocate and free buffers with, by specifying parameters such as width, height, usage, format. Usage is important since we'd like to allocate buffers for use with the framebuffer - so when we allocate a buffer, we allocate with usage 'usage | GRALLOC_USAGE_HW_FB'.

The return value of the alloc() call is a integer value indicating if it was a success, a native handle in the provided memory location (read my previous blog post for an explanation on what this is) and stride of the buffer.

We then wrap the handle and related information in a ANativeWindowBuffer structure and pass it back to the caller. Please note two things in this structure. incRef and decRef - they are very important. You will need to implement reference counting and you will need to increase/decrease your reference counting matching your own references to it. When reference count reaches 0, the buffer should destruct.

Eventually we will then get the buffer back from the caller in queueBuffer -- but how do we now send it to the framebuffer to be displayed?

In the initialization of our framebuffer window, we should have also opened the framebuffer with the libhardware API. It is in the same hw_module_t as gralloc. The framebuffer interface includes handy information such as the width, height, format, dpi and a few methods to actually utilize the framebuffer. The most important one for us is post(). This allows us to flip an actual buffer to the screen - utilizing the buffer handle, provided it has the same width, height/format as framebuffer and is allocated with appropiate usage (framebuffer usage). This call will on occasion block.

We have to be careful not to deliver the current front buffer to the caller in dequeueBuffer until we have replaced it with another at the front of the screen or we may see flickering.

A note to users of libhybris: there may be some Android adaptations that implement a custom framebuffer interface requiring extra implementation to achieve sane posting of frames that blocks. Check your FramebufferNativeWindow.cpp for this. This does not seem to be pervasive but I've encountered it on HP Touchpad with CyanogenMod/ICS.

Server-side Wayland enablement

The wayland protocol has two sides, server - and client. But unlike X, there is no "Wayland server". The implementation of the protocol communication for each side is implemented in respectively libwayland-server and libwayland-client. When implementing a compositor, you then utilize libwayland-server API to create server sockets, do communication, etc.

But how does the EGL stack get to be connected to a Wayland server instance when the associated EGLDisplay the stack is connected to, probably isn't a Wayland display? (note: may be in nested compositors - ie, a Wayland compositor running as a client to another Wayland compositor) That's where the next topic comes in:

EGL extensions - EGL_WL_bind_wayland_display

In order to connect your EGL stack to a Wayland display, you need to bind to one - you do this with eglBindWaylandDisplay(EGLDisplay, struct wl_display *) from the EGL_WL_bind_wayland_display. In libhybris, we provide this extension when it has been configured with --enable-wayland and available in most windowing systems (we provide an environment variable EGL_PLATFORM to select between 'windowing systems). Since the extension is not just part of the Wayland windowing system, it is possible to do nested Wayland compositors.

But what happens in libhybris when you bind to a Wayland display? We call the server_wlegl_create method in server_wlegl.cpp. What this does is to add a global object to the Wayland protocol - with a certain interface, - but where is this interface defined? As it has to be shared between both client and server; it is specified in a xml format file that is then converted by a tool called 'wayland-scanner' into .c files that are then linked into your client or server part. We then implement the actual interfaces for server side in our code.

Creation of buffers

When a client requests to create a buffer, it first creates an Android native handle object on server side and shares the client's native handle through the Wayland protocol (utilizing the support for fd passing) and then actually asks to create the buffer (note: we actually created the buffer on client side, now we're just sharing it with the compositor and letting it know the details).

When that happens on libhybris side, is when a handle is created, we construct an Android native_handle_t on server side - with the correct fd and integer information. We then map this with registerBuffer into the compositor's address space so the buffer is available to EGL and related stacks.

Once we have the handle in place, we can now create an Android native buffer - like we made ANativeBuffer on client side in the framebuffer/generic window scenario, we make one representing the remote buffer with the reconstructed handle. Finally we construct a Wayland object referencing this Android buffer and increase the reference counter of the Android buffer - and pass the buffer back to the Wayland client.

When we want to destroy the buffer again (well, when the reference count reaches 0), we unregister the buffer and close the native handle.

Utilizing a Wayland-Android buffer as part of your scenegraph

When a compositor would like to utilize a Wayland buffer in general, it uses eglCreateImageKHR with the EGL_WAYLAND_BUFFER_WL target and passing the (server-side) wl_buffer. This means that the compositor does not have to worry about the factual implementation of the wl_buffer behind the scene.

In our case, our wl_buffer is the one we indicated above - so we know it actually encapsulates a ANativeWindowBuffer with a handle/width/height etc. The knowledged reader might realize that eglCreateImageKHR in the Android EGL implementation does not support EGL_WAYLAND_BUFFER_WL. It does however support EGL_NATIVE_BUFFER_ANDROID

The way that this is handled is that we wrap eglCreateImageKHR and when we see EGL_WAYLAND_BUFFER_WL, we call the real eglCreateImageKHR with EGL_NATIVE_BUFFER_ANDROID - with the ANativeWindowBuffer. And we get the Wayland client's buffer as part of our OpenGL scenegraph.

This way we can also easily implement method such as eglQueryWaylandBufferWL as we know the attributes of the Android buffer.

An implementor's note: the destructor of a buffer is first called with wl_buffer_destroy coming from client side. You'll have to remember reference counting and not just delete the buffer

Conclusion

Thanks for reading this (rather technical) second blog post, the third one should follow quite soon. The code is already published and continually developed in http://github.com/libhybris/libhybris but it's not easy to approach or use for general users or developers right now.

The final post will describe how you can use this solution together with QtCompositor on top of Mer Core as well as describe how the Wayland client side works to make all this tie together. You can already now study, comment or flame the client side implementation. But for the explanation and for a description of what is missing, you'll have to wait for next one :)

Feel free to join us in #libhybris on irc.freenode.net to discuss and contribute to this work.

Wayland utilizing Android GPU drivers on glibc based systems, Part 1

noreply@blogger.com (Carsten Munk) — Thu, 11 Apr 2013 14:32:00 +0000

In this blog series, I will be presenting a solution that I've developed that enables the use of Wayland on top of Android hardware adaptations, specifically the GPU drivers, but without actually requiring the OS to be Bionic based. This is part 1.

This work was and is done as part of my job as Chief Research Engineer in Jolla, which develop Sailfish OS, a mobile-optimized operating system that has the flexibility, ubiquity and stability of the Linux core with a cutting edge user experience built with the renowned Qt platform.

The views and opinions expressed in this blog series are my own and not that of my employer.

At the end of the series, the aim is to have finished cleaning up the proof of concept code and published it under a "LGPLv2.1 only" license, for the benefit of many different communities and projects (Sailfish, OpenWebOS, Qt Project, KDE, GNOME, Hawaii, Nemo Mobile, Mer Core based projects, EFL, etc).

QML compositor, libhybris, Wayland on top of Qualcomm GPU Android drivers

The blog series seeks to explain and document the solution and many aspects about non-Android systems, Wayland and Android GPU drivers that are not widely known.

(Ignore the tearing, old demo video)

This work is done with the hope that it will attract more contribution and collaboration to bring this solution and Wayland in general into wider use across the open source ecosystem and use a large selection of reference device designs for their OS'es.

Why am I not releasing code today? Because that code alone doesn't foster collaboration. There's more to involving contributors into development - such as explaining reasons why things are the way they are. It's also my own way to make sure I document the code and clean it up, to make it easier for people to get involved.

Now, let's get to it..

The grim situation in mobile hardware adaptation availability

One of the first thing somebody with a traditional Linux background realizes as he tries to make a mobile device today when he meets with an ODM is that 99% of chipset vendors and hence ODMs - will only offer Android hardware adaptations to go along with the device designs.

When you ask about X11 support within the GPU drivers or even Wayland they'll often look blankly at you and wonder why anybody would want to do anything else than Android based systems. And when you go into details they'll either tell you it can't be done - or charge you a massive cost to have it done.

This means that OS'es and devices that are non-Android will be not able to take into usage the huge availability of (often low cost) device designs that are out there, increasing the time to market and R&D cost massively.

Libhybris

In August 2012, I published my initial prototype for 'libhybris'. What is libhybris? Libhybris is a solution that allows non-Android systems such as glibc-based systems (like most non-Android systems are) to utilize shared objects (libraries) built for Android. In practice this means that you can leverage things like OpenGL ES 2.0 and other hardware interfacing provided within Android hardware adaptations.

I had developed libhybris initially in my idle hours at home and the big question you might have is: Why did I open source it instead of keeping it to myself and earn on it as it obviously was the holy grail for non-Android systems?

The simple answer is this: by working together on open source code, it would help accelerate the development of libhybris and testing of the software for everybody's mutual benefit.

I didn't feel good about libhybris initially, it's not the most perfect solution to the problem: many around me in the open source community were and are fighting to have chipset vendors provide Wayland or X11 adaptations for mobile chipsets or even GPU drivers for non-Android systems in the first place.

But I felt that this was the required road that had to be taken before non-Android systems turned completely irrelevant in the bigger picture. When we again have volume of non-Android devices, we can have our own dedicated HW adaptations again

Open sourcing worked quite well - a small group of people got together, tested it, improved it, got it running on a lot of multiple chipsets - thanks to OpenWebOS, Florian Haenel (heeen), Thomas Perl (thp), Simon Busch (morphis) and others. It turned the project from a late night hacking project into a viable solution for building device OS'es on top of. Or even running Android NDK applications using.

Earlier this year however, I discovered that a well-known company had taken the code - disappeared underground with it for several months, improved upon it, utilized the capability in their advertisements and demos and in the end posted the code utilizing their own source control system, detached from any state of that of the upstream project's. Even to the extent some posters around the web thought libhybris was done by that company itself.

That kind of behavior ruined the initial reason I open sourced libhybris in the first place and I was shocked to the point that I contemplated to by default not open source my hobby projects any more. It's not cool for companies to do things like this, no matter your commercial reasons. It ruins it for all of us who want to strengthen the open source ecosystem. We could have really used your improvements and patches earlier on instead of struggling with some of these issues.

But, I will say that their behavior has improved - they are now participating in the project, discussing, upstreaming patches that are useful. And I forgive them because they've changed their ways and are participating sanely now.

Now for a few words on my story with Wayland..

Wayland

My journey with Wayland started in late 2011. It was my belief around that time that Wayland was a bit dry and boring - it was just a protocol. I was not fully appreciating the power and simplicity that it provided for embedded UI. That it was a building block for much more exciting things - like libhybris turned out to be.

Being in embedded Linux and exploring Qt Lighthouse, I had learnt of Qt Scenegraph and a curious new thing called QtCompositor. QtCompositor was what sold me on Wayland - it enabled amazing capabilities on embedded and ease of development of effects, window management and homescreens. Things that previously would take several manyears to develop for embedded devices was made easy to do. And allowed stack builders to have similar graphical stacks on SDK/virtual machines for development as on target devices.

If you don't know QML, it's a declarative UI language to design beautiful and functional UIs with. What QtCompositor did, was to enable that you first off could get events about windows appearing, changing size, etc - but also that each window, - the graphical buffer, became just another item in your UI scenegraph like an image or a label would be. It could even make your graphical buffer be a widget inside your traditional UI.

Screenshot from http://blog.qt.digia.com/blog/2011/03/18/multi-process-lighthouse/

This could naturally be expanded to much more curious things, such as 3D Wayland compositors. If you'd like to hear more about QtCompositor, you can also watch the following talk from Qt Developer Days by Andy Nichols. Capable QtCompositor technology is something that is here today. Not something that has to be developed from scratch or roadmapped.

I was doing the role of maintainer of the Nokia N900 hardware adaptation for MeeGo at the time and I wanted to see if I could get Wayland working on it - it had a PowerVR SGX GPU. I reached out to #wayland on irc.freenode.net and was met with open arms, guidance and a lot of help from krh (Wayland founder), jlind, capisce (QtWayland/QtCompositor), Andrew Baldwin, Mika Bostrom and many other talented people and I was able to get started very quickly with Wayland.

To get things working with Wayland, what I needed to do was figure out:

How to render an image with OpenGL ES 2.0 into GPU buffers that I had under my control
Share that GPU buffer with another process (the compositor)
Include that GPU buffer as part of a OpenGL scenegraph, a texture - and display this to the screen (in the compositor)
And for performance, flip a full screen GPU buffer straight to the screen, bypassing the need to do OpenGL rendering

To be able to render into a specific GPU buffer under your own control, you usually need to get inside the EGL/OpenGL ES implementation. On some chipsets, it's possible to use specific EGL operations that allows shared (across two processes) images to be rendered into - such as on Raspberry Pi.

In the EGL implementation, you should be able to follow the path of the buffer, it's attributes (size, stride, bpp/format) and when the client has requested to do eglSwapBuffers.

On the PowerVR SGX, there was an API provided called WSEGL, for making plugins that were windowing systems (X11, Framebuffer, QWS, Wayland..) that allowed me to do just that.

Sharing that buffer is sometimes a bit more difficult - you effectively need to make the same GPU buffer appear in two processes at once. On SGX, this was simple - you could request a simple integer handle to the buffer and share that value using whatever protocol you wanted. In the target process you then just map in the GPU buffer through a mapping method.

Wanting to standing on the shoulders of giants, I looked at how Mesa had implemented their Wayland protocol for DRM - it too had simple handles and shared these buffers through a custom Wayland protocol.

Even if it was a custom protocol for buffer handling (creation, modification, etc), the same operations for handling buffers in Wayland still applied to it. I didn't need to do anything extra for compositor or client for the buffers in particular - I could piggyback on existing infrastructure available in Wayland protocol.

Wayland made it easy for me to take existing methods for the techniques/needs listed above into use and made it possible to quickly and easily implement Wayland support for the chipset.

Now, to something a little different, but quite related:

Android and it's ANativeWindow

When you use eglCreateWindowSurface, as in, creating a EGL window surface for later GL rendering with Android drivers, you have to provide a reference to the native window you want to do it within. In Android, the native window type is ANativeWindow.

As you know, Android's graphics stack is roughly application -> libEGL that sends GPU buffers to SurfaceFlinger that either flings the buffer to the screen or composites it with OpenGL again with libEGL.

Why not just include all the functionality and code in the EGL stack which communicates with SurfaceFlinger? The answer is that you need to sometimes target multiple types of outputs - be it video/camera streaming to another process, framebuffer rendering, output to a HW composer or communication with SurfaceFlinger.

One of the good things about Android graphics architecture is that through the use of ANativeWindow, they have made it possible to flexibly keep the code that does this work outside the EGL drivers - that is, open source and available for customization for each purpose. That means that EGL/OpenGL drivers are less tied to the Android version itself (sometimes API versions of ANativeWindow changes) and can be reused in binary form easily across upgrades.

ANativeWindow provides handy hooks for a windowing system to be managing GPU buffers (queueBuffer - send a buffer, I'm done rendering, dequeueBuffer - I need a buffer for rendering, cancelBuffer - woops, I didn't need it anyway, etc) - and it gives the methods you need to accomplish things, like I did on PowerVR SGX.

This is the entry point used to implement Wayland on top of Android GPU drivers on glibc based systems. Some fantastic work in this area has already been done by Pekka Paalanen (pq) as part of his work for Collabora Ltd. (Telepathy, GStreamer, WebKit, X11 experts) which proved that this is possible. Parts of the solution I will publish is based on their work - their work was groundbreaking in this field and made all this possible.

A note on gralloc and native handles

The graphical buffer allocation in Android is handled by a libhardware module named 'gralloc'. This is pretty straightforward, allocate buffer - get a buffer handle, dealloc, register (if you got the buffer from a separate process and want to map it in), but most documentation pieces don't talk about buffer_handle_t and what it actually is.
If you do a little bit of detective work, you'll find out that buffer_handle_t is actually defined as a native_handle_t* .. and what are native handles?

The structure is practically this: number of integers and a number of file descriptors plus the actual integers and file descriptors. How do you share a buffer across two processes then?

You have to employ something as obscure as "file descriptor passing". This page describes it as "socket magic" which it truly is. It takes a file descriptor from one process and makes it available in another.

The android GPU buffers are typically consisting of GPU buffer metadata (handle, size, bpp, usage hints, GPU buffer handle) and then file descriptors mapping GPU memory or otherwise shared memory into memory. To make the buffer appear in two processes, you pass the handle information and the related file descriptors.

The good news however is that Wayland already supports file descriptor passing so you don't have to write obscure code handling it yourself for your custom Wayland compositor.

Conclusion

This concludes the first blog post in this series, to give a bit of background about how Wayland, libhybris and GPU drivers for Android can work together. Next blog post will talk more about the actual server side implementation of this work. Last blog post will talk about direction of the future work on it - and what you can do with it today and how as well as explaining the .

If you'd like to use, discuss and participate in the development of this solution, #libhybris on irc.freenode.net is the best place to be. A neutral place for development across different OS efforts.

Tizen conference, wrapping up

noreply@blogger.com (Carsten Munk) — Sun, 13 May 2012 16:23:00 +0000

When attending a conference, or a music festival, or any other event with multiple tracks, there will always be sessions that you for some reason do not end up attending, be it because of meeting somebody suddenly at the coffee table while a session you'd like to see is about to start, or because there's a session that you'd rather see, or simply because you decided to take a break. The solution to this is the conference recording the sessions on video, which is not often performed very well.

I've encountered the usual screw-ups in conferences: session recording that does not include the slides at all, session recording where the A/V people forgot to wire up the microphone to the recording equipment(!), or so bad audio that you couldn't even hear the speaker. It's also more difficult, when viewing the session afterwards, to having to focus on three things - the movements of the speaker, his/her voice and the slide content.

That's why I welcome a better format, which is what the Tizen conference will be utilizing it seems - recording the speaker audio and along with that, the slide content, but not the speaker him/herself. And exporting it in podcast format, allowing you to catch up while on the move on the newest technology. Without having to dedicate your full attention to it.

Which brings me to that I didn't participate in that many sessions during the last day. But I got to attend the ones that mattered to me the most: Open Build Service—Facts, Features and Future, by adrianS and mls from openSUSE and Next Gen OS Initialization Done Right, by Auke Kok. And missed out on Tizen IVI architecture with Mikko Ylinen.

Meeting the OBS guys is always a pleasure - you get to sync on what their ideas and plans are and they listen to what your expectations and sometimes crazy ideas are. Compared to many other distribution build systems, OBS does not only function or serve their own community (openSUSE) - it is entirely usable, deployable and fantastic for building other distributions. OBS has leveled the amount of infrastructure you previously needed in other to run/roll your own distribution. And that's why we love it in Mer. It enables anyone with a minimum of OBS knowledge to maintain their own customized distribution. And for ISVs, to build against your distribution.

As with most technical discussions - the hallway track is the most interesting. The questions and concerns the audience comes up with during the session seeds the ground for the continued discussion in the hallway afterwards. One concern, that was raised by Dominique Le Foll, of Intel OTC was that OBS-to-OBS links are simply too fragile and too often causes build stalls and problems, was actually a matter we approached at first with Mer too, given the very unstable nature of the meego.com API - we needed a way to synchronize and access the OBS projects MeeGo consisted of, offline. Their need was simply needing a way to export MeeGo (well, now Tizen) releases to customers in a reliable manner and allowing them to modify it too.

What we invented there was a piece of software called FakeOBS (now Mer Delivery System) which, to make a long story short, serves up a HTTP/REST interface that is similar enough to the OBS-to-OBS protocol as to make it able to have another OBS connect to it and it thinking it's a remote OBS.

While in fact, it is a cache of sorts - we extracted through the OBS API the entire OBS project history of sources, built binaries and put it into a on-disk format that FakeOBS could then use to provide those over the OBS-to-OBS protocol, giving us effectively offline access - leaving us able to not have to care about external entities. You can view the latest iteration of it here. There's also a file called 'gitmer.py' which is how we deal with the git-based approach that Mer uses, for sources.

When we generate Mer releases, we do not only export the built binaries, we also export this on-disk format for FakeOBS, allowing anyone to re-create Mer and re-build it in their own OBS, along with the additional use case that we have built OBS package repositories for ISVs to build against. Meaning that even if Merproject.org shuts down, anybody can resurrect the project. As well as that vendors do not have to rely on merproject.org being up.

Next session was Auke Kok's systemd session. Auke's one of my personal open source heroes, always working on quite interesting things. As some of you might know, the traditional way that most UIs are launching applications and daemons on boot are through D-Bus and /etc/xdg/autostart .desktop files. In Mer and MeeGo, this was accomplished with uxlaunch (another of Auke's inventions)

But what if you could use the same flexibility that systemd offers you, in order to create a proper dependency tree for proper optimized booting within the user session? Well, guess what Auke invented :)

Instead of starting uxlaunch, you'd instead start systemd --user as a user, which would properly start up X, perhaps services that do not need X before X starts, and ability to indicate session-internal dependencies. Which leads to amazing results. You can check out the systemd discussion on this mailing list thread.

Another thing that happened was the availability to all conference attendees (except for Intel & Samsung, it seems) of a Tizen reference device, the so-called Lunchbox. It's an amazing piece of hardware, 1280x720 AMOLED display, 4.3", with dual-core Samsung Exonys 4210, Cortex-A9, 1gb ram, 16gb eMMC, microSD slot, sim slot (though modem ability is unknown), Mali GFX chipset, u-boot boot loader, 8mp back camera, 1mp front camera, PN544 NFC chip (unsure how to use), GPS chip, WLAN (WiFi Direct possible).. so, a quite nice kit. And a possible replacement for the N800 as well.

And if you're wondering if we've tried to put Mer on it. Now, of course we have. We've found most interesting pieces, including a "Boot to SD card" mode (with no success just yet - press power key, volume up and volume down at same time), kernel source code (2.6.36) and investigated the system which uses currently Xorg 1.9.3 with a Xorg driver we can't find source for yet. But it'd surprise me if it wasn't somehow similar to the Mali Xorg driver. Once we've figured out SD card boot, it should be a breeze to run Mer on there. Even with X11-GLESv2 acceleration.

That's all I have to say about the conference, will look forward to the next one.

Tizen Conference 2012, first days

noreply@blogger.com (Carsten Munk) — Wed, 09 May 2012 14:41:00 +0000

The Linux Foundation sponsored me to travel to and attend the Tizen Conference 2012 in San Francisco and as part of this sponsorship, I'll be blogging about the conference and my insights and thoughts of the talks and keynotes at it. While some parts may seem negative, please keep reading as I have a point with those parts that is illustrated later on.

While some people might think that I see Mer as a competitor to Tizen, or think that I don't like Tizen, in fact, I see the benefit of Tizen for the entire open source community that they're pushing a proper, fast and working, standards compliant web runtime for Linux based devices. And I hope we'll see people putting this on top of Mer. Mer plays a different game than Tizen and hence not a competitor - it's a solid mobile core that everybody can build on and most importantly, rely on despite the moods of corporate giants, to make their devices - where solid delivery, ability to productise and code is king.

There's always a challenge in how to properly launch a commercial open source project. Especially in mobile Linux, there is unnecessary high media and developer expectations to the features and ability of a mobile handset stack that often enough causes mobile OS projects to break their necks and die.

But Rome wasn't built in a day. And so weren't the mobile stacks you see around the today - Android took over 3 years to get to a remotely functioning stack. And this has a tendency to make mobile OS developers develop in 'closed mode' in order to launch properly on day one.

Tizen has drawn a lot of crap from their complete silence and secret cathedral building behaviour up to 1.0 release. But I can say that if I was in their shoes, having to launch a handset device .. and handset stack. I would probably end up doing the same things as they had. In a world where you will see your semi-ugly alpha release screenshots laughed at in news articles about your 1.0 launch, when you have a perfectly working and very shiny final release that nobody seems to bother to even check out, it's hard to argue for transparent development from day zero.

Luckily I'm not in their shoes and I can concentrate on technical core side of things and make it easier for people to make exciting new devices. I'm not a fan of whole-stack projects - for the reasons listed above. And that's why Mer doesn't contain UIs and hardware adaptations - we let people base and develop on a sane core that we can maintain the openness of, while people work on exciting new devices and UIs in seperate projects from the Mer project.

Reputation and history matters a lot as well - There was a lot of disappointment in the demise of MeeGo and the messaging on MeeGo site was that Tizen was the new direction. And people expected there'd be some kind of relation. But there's not - Tizen, at least the handset side (IVI is more related on the Core side), is so very different from anything you knew in MeeGo - your employees would need complete retraining to work with it. The stack is based off Samsung Linux Platform and actually says it's that, when booting up, last I saw. That should lead to interesting discussions with your company lawyers.

And with reputation and history in mind - many from MeeGo remember last years keynote, Monday Morning with MeeGo.. February 11 had happened months before and there was still a fighting spirit in the community, we needed people who were showing passion in their work, the same fighting spirit. And we got something that was closer to Monday Mourning with MeeGo. Which left many people depressed and unimpressed. A talk that spoke more about the fantastic deployments of the platforms that MeeGo was in practical competition with, than about MeeGo itself and it's qualities and achievements.

When a last moment change in the Tizen conference schedule came in, that moved the first keynote which was supposed to be Imad from Intel and JD from Samsung to the morning after and instead, we got a recycled keynote, void of genuine and documented passion for Tizen, with the same recycled material as in the Monday Morning with MeeGo talk and the same speaker as last year - with him even talking about that if people had noticed he would be on schedule, there'd probably be fewer in the room. I was left unimpressed and depressed, again.

I'm ashamed to say it, but I've grown to appreciate the famous "Developers! Developers! Developers!" dance. Somebody showing passion for their work - people who truly believe in their product and their stack. Someone who wants to stirr excitement in developers, CTOs and platform integrators alike. And I harbour a lot of respect for people who truly believe in what they do and that they want to do a good product and good software.

And we got that in the morning after keynote, Imad, the head of Intel OTC and JD, EVP of Samsung, the Tizen technical steering group. They showed their excitement about what they were doing, showing excellent live demos and they understood the challenges in presenting a mobile OS and explained to depth both the challenges and reasoning behind their choices for the Tizen stack - even the controversial ones. I can recommend watching the video when it gets put online.

This is what the first keynote of the conference should have been like. And I was more excited again.

The second keynote, from Tizen Association was interesting, showing the need of operators to provide and control content and the benefit of having to write to one platform (HTML5). Which is funny, when there were also talks about "The web always wins" .. that in practice, the walled garden never wins, the properitary standards will not either. That content from outside the walled garden will always be more interesting and varied than your own content. Doesn't anybody remember AOL? :)

I might be an idealist, but I have difficulty seeing the word 'open' together with the working method of the Tizen Association. When their terms of membership states at http://www.tizenassociation.org/en/tizen-association/board-of-directors
that there is an annual due of membership of Two Hundred and Twenty Thousand USD. And talking about confidentiality clauses.

But I was reminded of a quote from my former boss, Harri Hakulinen, from his MeeGo blog, speaking of Nokia's choice of WP7: "... What I had witnessed from the side, thanks to Sami J. Mäkinen, was of course the birth of Linux. And, in so many ways, I feel that we have now "replayed" the same with MeeGo in last 12 months.
Co-incidentally, at that time I had a summer job where we replaced some ancient Nixdorf and IBM mainframes with another new OS, called Windows NT. It was hard to explain to Sami and others, that from the point of view of those companies I worked for, Windows NT was, in fact, like "open environment", compared to those things that they had been using before."

And perhaps this is how it's like with Tizen. That compared to how it has been before, this is actually more open than the previous situation for the ecosystem. At least the code is open, actively developed, looking to be intended to be developed in the open, not only codedrops, with a proper code submission process. So perhaps that's an improvement on top of what we have had with Symbian and Android.

What I am more encouraged by in terms of openness, is what I learned in the Tizen Architecture talks, where there is a strong push towards that there will actually be as little Tizen specific WebAPIs as possible. That there's an active desire to use W3C standards where possible and to submit APIs to the standards process. Which makes me hope that for the parts that really matters - the developer facing APIs, that there will no vendor lock down. In the end, the winner would be the Web. And even if Tizen did go away one day, that the code would be living on. Just like MeeGo's code has. Because it's open source.

From an entirely architectural point of view, Tizen architecture (SLP-based) has a lot of NIH in it - it's own telephony stack instead of oFono (though I hear that's coming), and a lot of self-made libraries and methods. Which may impact the portability of the web runtime.

I'm hoping that Tizen will realize the importance of separating the web runtime from OS stack. That one of the reasons that the web standards have developed so fast, has been the ability to get many people to upgrade their browsers over the air.

We probably cannot assume we'll see Tizen 1.0 to Tizen 2.0 upgrades on our handsets - aftermarket hardware adaptation is really hard. But we should make sure that at least for a few generations, we can keep on delivering the web runtime to older devices. To foster a standards compliant and secure web runtime for our apps - just like the browsers has fostered it for the web. And I think this is one area Tizen could potentially be very agile in, comparing to other platforms.

The last talk of the day was on Tizen security - Ryan Ware was presenting along with his Samsung counterpart, where compared to many other platforms, the focus has been to ensure end-user privacy, and not on the need to protect content owner content. That's the right direction - especially in a world where devices are turning very promiscuous and interacting in sometimes unsafe manners with other devices.

One of the most important parts of a conference is not so much the presentations, but the people there and the discussions in the hallway - the 'hallway track'. This is a concept that any conferences doesn't always understand - the importance that people can discuss outside the conference rooms. Tizen conference has as usual done a great job on this (thanks to Amy Leeland's fantastic work), providing good space and hacker lounges with whiteboards for people to properly discuss. I've sat down together with many key people in my line of work and had intellectual discussions about the challenges we meet. And I've gotten to know more people that I previously didn't.

And of course - when working on a distributed-across-the-world project, I've been able to go out and have our traditional icecream shop visit (since 2009) with the rest of the participants of my project. So far it has been a good conference, looking forward to be seeing more presentations today.

Mer is back!

noreply@blogger.com (Carsten Munk) — Mon, 03 Oct 2011 16:25:00 +0000

So, we are back. Please take some time to read the following mailing list post:

http://lists.meego.com/pipermail/meego-dev/2011-October/484215.html

See you in #mer on irc.freenode.net and on http://www.merproject.org

Why Meego on N900 is the right and future-proof direction for the community

noreply@blogger.com (Carsten Munk) — Tue, 17 Aug 2010 17:06:00 +0000

So, I think it's about time for me to really weigh in why it is I'm pushing MeeGo on N900 and why this is the right direction. This is probably going to be a bit long. Some people asked me to repost it on my blog as to not have it lost in the maelstrom of talk.maemo.org.

This is also probably my last long post on the matter as I'm now fully in a MeeGo role, no longer maemo.org distmaster.

The fact is that the model of 770 -> N800 -> N810 -> N900 has been that of one device per OS version. There hasn't been resources for keeping a full product team for multiple devices at a time.

This has caused what some describe as Rapid Obsolescence Syndrome, where the device OS rapidly goes into maintenance mode after device release while mostly everyone involved moves on to the next product/device. This obviously leaves users unhappy.

Now, there are multiple suggestions around maemo.org:

* Fremantle community SSU
* Petition for open sourcing most of Fremantle to keep it maintained
* Harmattan HE for N900
(etc)

And my main point in the following is that those directions will be like pissing your pants to keep yourself warm. Quick review:

Community SSU can work on the open source parts (it has happened on N8x0 just fine). Maybe you can find some ability together with Fremantle aficionados inside Nokia who could be interested in contributing with upgrades of some binary packages -
I doubt there's any problem if those exist to slip some packages out.

But the main point is: no upstream work will really happen. I highly doubt you can find enough people that are still around to keep things working. Even with documentation, the code can be quite hairy to add new features to. So that bends down to a resourcing problem.

Open sourcing most of Fremantle - I simply don't think it's going to happen. Too much work for too little benefit. And the same problems as with normal SSU when the code does arrive.

Harmattan HE. This would already be obsolete by the time we were done as a more modern platform would be available in MeeGo. As well as at some point, no more upstream development.

The main point - you'll piss your pants with these projects to keep yourself warm for a while, but you'll just keep on feeding the Rapid Obsolescence Syndrome to some extent - that you'll never be able to play catch-up to new technologies and satisfy the requirements of your users. It is a physical impossibility as there's many more man-hours poured into MeeGo than you can provide here.

MeeGo and Qt has changed the status quo that existed that:
* Any Linux-based OS platform from Nokia is tied to a certain device
* Any applications from Nokia is tied to a certain device by fact that they were released for a certain OS platform.

It's a game-changer because now, for any given 'Nokia' OS for these devices, this will after Harmattan be based off MeeGo (platform from MeeGo.com).

Now, what are we actually doing in MeeGo for N900?

First off - we're making the hardware adaptation for N900 maintainable.

This means that we are actively upstreaming N900 drivers to the mainline Linux kernel as well as getting those few blobs we have into the MeeGo non-oss repositories with redistributable license. This is actively happening.

Second, we're running QA daily to make sure that N900 still works with the current MeeGo platform state. This means we're always at the front of the platform development. MeeGo platform doesn't move forward if a change breaks N900 - it is a reference device

That's not pissing your pants to keep warm. That's making a bonfire and collecting wood to make us sustainably warm. And that's one of the reasons why I hope for more people to contribute to the MeeGo effort.

Now for the users. Look forward a bit - Nokia's obviously going to make a OS release based on 'real' MeeGo.com. The same code that'll power future handsets will be based on the same code that N900 already now supports. It's the same infrastructure that builds images and software that MeeGo for N900 is made using, as any given future Nokia OS. Instead of Nokia having applications tied to Fremantle or to Diablo or whatever, they now tie to MeeGo - the same platform you have on your N900.

If you want to make a real bonfire and eat your cake too, you will want to do the following:

* Contribute to MeeGo and MeeGo on N900 in the short term - this will improve matters for Nokia N900 obviously.
* Petition for Nokia early on, to consider providing the following things:
** Executive summary: access to the binaries making up the Nokia differentiation
** Weekly (or other interval) repository releases (binaries) available to Nokia N900 users of their future MeeGo.com-based OS under the usual 'this is not for end-users, bla bla'. Kinda like the old 'Sardine'
** Kickstart files for these weekly releases so you can edit them into your own images for the N900 using the Nokia bits.
** Community can be possibly be more involved in the QA process this way and contribute in general.
** If branding is a problem, have community themes and icons.
** Suggest new ways to have both a 'closed' vendor OS with all the goodies based on MeeGo.com while at same time having your power user community close to you.

Now, this is a way that can keep you ahead of the curve regarding your device. The N900 hardware isn't going obsolete for many years and is still very capable.

I think this might be possible to pull off, but it requires people to start gaining skills within MeeGo - and help contribute to MeeGo on N900. It's a similar approach to Harmattan HE, but more involved and more future proof. So I wouldn't say it's impossible.

So - it's your choice: Live in the future instead of the past.

I'll be working to make the future happen - see you on http://www.meego.com (and http://wiki.meego.com/ARM/N900 ).

First instance of 'What have I done for the N8x0's lately'

noreply@blogger.com (Carsten Munk) — Thu, 20 May 2010 20:00:00 +0000

So, this is a new series of blog series on what I've done for the N8x0's lately. So people better see what I've actually tried to do for the owners of these devices.

* I've spent a lot of time with the MBX 3d driver to try to get it in good shape and set up gitorious repo for it and forward ported it to 2.6.33. At the current stage,
* Started a page for the MeeGo N8x0 hardware adaptation team and added plans/milestones
* Working on stabilising MeeGo N8x0 startup, building Xomap for it. I have it running with Xfce4 on my device and hope to show it after 1.0 release.
* Worked with tmr to try and get DSP working on 2.6.33 for N8x0. He forward ported the patches himself.

These are fairly technical things, but should hopefully manifest in actual benefits sooner or later.

Maemo 6 and/or MeeGo on N900: Or How I Learned to Stop Worrying and Love the Bomb

noreply@blogger.com (Carsten Munk) — Sat, 20 Feb 2010 19:12:00 +0000

Since some people don't follow talk.maemo.org, this is to let them know I wrote a quite interesting piece to help calm some of the worries people have been having about N900 future with Maemo6 and MeeGo based on my previous experiences both as community member and as maemo.org distmaster.

You can read it at http://talk.maemo.org/showthread.php?t=45213 and it is very well received so far and covers four questions we should be asking instead of panicking.

The Mer Project - just a bunch of redshirts?

noreply@blogger.com (Carsten Munk) — Fri, 19 Feb 2010 11:27:00 +0000

Recently, big news arrived: The MeeGo project. But let me start with some history. In May 2009 some key members of the Mer project met up for the first time 'in real life' at the Mozilla Maemo Danish Weekend. At this event, we got some very characteristic t-shirts - very very red ones, which we since have worn proudly when we were meeting up again at various events and countries, as we could recognise each other.

While we were introduced as Maemo 5 community edition for the Nokia N810/N800/770 Internet Tablets we were far more ambitious - we loved the principle of Maemo so much that we wanted to reconstruct it, in order to make it better, more suitable for the future - because we thought the platform would not survive otherwise.

We wanted to get away from the notion that a tablet was an embedded system - Tablets are not under-powered embedded systems, they are powerful, power-efficient, economical handheld computers. We wanted to make Maemo a general platform for tablet devices. We wanted to Make it more developer-friendly. More hackable. Align with standard Linux distributions. This was to be done by separating the device and platform code - have open development of the Maemo platform - the device-specific and vendor-specific differentiation
development can be closed and many other things.

The goal was for the core Mer system is fully open source and cross-platform, open for anyone to adapt to their devices. We got far, we have something that works, but it isn't ready for users yet - Mer was a research project to try and help Maemo towards our goals.

A comment (as I recall it) from Maemo Summit 2009, where we Mer members met up again, performed several presentations - again, wearing our red shirts - "We're just a bunch of redshirts.", ambitious ensigns in red shirts - did this come true?

Some background, as it might be a bit geeky. Quoting wikipedia:

Redshirt is a slang term for a minor stock character of an adventure drama who dies violently soon after being introduced in order to dramatize the dangerous situation experienced by the main characters. The term originated with fans of the science fiction television series Star Trek, from the red shirts worn by Starfleet security officers and engineers, who frequently meet their demise during the episodes.

Fast-forwarding to this week, when the MeeGo announcement broke. Reading the pages, many things seemed familiar:

From Quim Gil: MeeGo is just like you would expect a free Linux OS to be: based on the standard Linux and Free Desktop technologies and developed publicly in a project open to all contributors

From architecture page of MeeGo:

The MeeGo OS Base layer contains the Linux kernel and core services along with the Hardware Adaptation Software required to adapt MeeGo to support various hardware architectures.

The MeeGo OS Middleware layer provides a hardware and usage model independent API for building both native applications and web run time applications.

From governance:

There is no admission process or membership fee for MeeGo; just your desire to join the project and contribute.

Any individual or organization can join and get involved in software development. We also welcome other ways of contributing to the success of the project, including writing documentation, testing, and marketing.

And so on - it was like someone had listened to our ideas and wishes..

Basically, MeeGo was what we hoped to achieve with Mer, and it's silly to continue separately instead of collaborating. Now, with the announcement - is the Mer project members then just a bunch of redshirts who helped move the plot along but in the end, is out of the game when the main characters continue with the plot?

No. I don't think so. In Mer, we managed to accomplish many interesting things:

Easy development for mobile devices
We showed it was possible to develop for your mobile device just as easily as developing for your desktop PC. That we didn't need Scratchbox: We began every interesting experiments with OBS (Open Build Service), where our build guy - David Greaves (lbt) made some extremely nice experiments, showing that cross-compilation could work without hassle, even for packages normally requiring native compilation. He's BTW, having a day job working with MeeGo and OBS now - helping to bring the Mer development experience into MeeGo.

Power saving and distribution alignment
Showing that it was possible to retain power saving while making porting easy by having easy access to Debian (or any distro's) huge library collection.

That the community was capable of organising OS development too

That multiple device communities could work together on a common base that would benefit them all.

That we were able to help create policy, identify problems

Some of the things coming out of Mer:

The vendor social contract - a social contract for vendors to commit to, in order to have community spending time on them and not waste their time and end up with locked devices with a open system.

Token based access restriction - a way to deal with closed sourced binaries in a open system in a way that is liveable both technically and legally.

Why an upstream basis shouldn't be a goal, but a policy alignment on mobile devices

Immediate involvement of contributors interested, that any contribution is valuable and the list goes on..

So, are we out of the plot now that the main characters have stepped in? No. My experience has been that this is a new house for us to paint and move into. A fresh start - it would first have happened in many years at the pace we were going.

The developers I've met from the Moblin side has been very open, willing to discuss - and even spoken with one of the two top guys in MeeGo. Us redshirts fit in the MeeGo project because we have experience enough now to discuss and argue on equal footing and to help to shape the future - and to create it. Like we always wanted.

For those wanting to pursue the dream (and reality now) of a open mobile Maemo platform - MeeGo is the place and that's where I (and hopefully my fellow redshirts) will go to contribute to the future of mobile Linux.

So, what will happen to Mer? We'll still be a subsection of the MeeGo community arguing for our positions, our ideas, our hopes and helping out where we can. Best way to shape the future is by participating.

Mer as a system will live on in Mer^2. A quick summary is it is a Debian 5.0 system building inside OBS with tricks making it feel like a Scratchbox for the packages, which makes most of the Fremantle platform build on it. It will work on X86 and on N8x0. Mer 0.17 will be imaged up soon and published - those who want to continue with that can. The goal is not to make a fully open source system - it will serve purpose to be a foundation on which to build a backport of Fremantle for N8x0. The hope is to be able to build most of closed things for Fremantle for N8x0 using this.

Thanks for reading. This is a bit of post-mortem for the ambitious part of Mer. Which keeps on living in the MeeGo project. This blog will be mostly devoted to Mer^2 discussions and maemo.org distmaster related issues. A role which I am yet to figure out how fits in the MeeGo community - but I'm hopeful.

MBX/3D drivers out for N8x0 (and other OMAP2)

noreply@blogger.com (Carsten Munk) — Wed, 17 Feb 2010 07:27:00 +0000

In other news, the drivers was uploaded to TI extranet yesterday - this is something people have waited for, for a long time. And you need to create a TI (my.ti.com) account for this. For those who want to have access the SDK need to send a mail to gamingonomap@list.ti.com stating your need for the SDK mentioned below.

You need to then click "Extranets" on my.ti.com and then WTBU-OMAP Gaming SDK and then OpenGL®ES Graphics SDK for OMAP2420 Nokia N800 OpenGL ES v1.1 and OpenVG 1.0 SDK based on Linux kernel 2.6.21

A fair warning: These drivers are not for the weak at heart. You need to flash a new kernel. You may need to fix up some build scripts and run some mysterious commands. They will not work that easily out of box.

While I kinda assume this will be drowned out in Meego announcements and discussion, well, I got the mail last night

Thanks to qwerty12, javispedro, lardman, the TI guys (Girish, Ameet) and the Nokians helping out to make this reality and others.

__________________
maemo.org distmaster & Mer lead developer, Join the MeeGo project!

Mapping openness of Maemo 5.0 PR1.1 and Maemo 4.1.2 - we're moving in the right direction!

noreply@blogger.com (Carsten Munk) — Wed, 10 Feb 2010 08:55:00 +0000

One of the tasks I've been doing in the last couple of months has been to try and refresh the openness data for Fremantle. This has actually been quite a challenge as I've had to retrieve a lot of different sources to make a proper description of what is actually open and what is not.

The openness reports

For Fremantle PR1.1, you can find the openness report here - this is for a PR1.1 image actually installed on a device.

As per request and for comparison, I've also generated one for Maemo 4.1.2 (Diablo)

It has been difficult finding categorisation for Diablo so a lot of it is in 'Other' as I used the package categorisation from Fremantle in this.

This is supposed to be raw data for further discussion and as such there'll be subsequent analysis on top of this, explaining some of the areas and why things are as they are. Please read through this blog post for further explanation of what the data represents and how it is represented.

This openness report has the purpose of:

Giving exact information of what is open, what is not, if it's openly developed
Help prioritise what should be opened and in which order
Help giving arguments towards why things should be opened
Help new developers find out where components are developed
Help making Maemo more open
To aid open sourcing discussion where we go to the technical heart of a problem, not the ideological one and save all of us precious time that could be used to open source things.
Get information about what components may not be so interesting as to open source and what is kept closed source in those component areas and help to give an understanding of why it is so.

The purpose is not:

To be a fight over numbers and percentages of openness
To be a discussion over open sourcing policies (what Nokia chooses to open source and what not)

Classification of packages

So, to start describing what these openness reports contain. I have chosen to classify source packages into seven levels - this will also explain the headers of the table.

L1: Developed openly, no closed source dependancies

This means that the source package is developed either in garage.maemo.org, maemo.gitorious.org or some other place. This means that you can follow development and contribute patches to it, without having to wait for new releases

L2: Source published in SDK, no closed source dependancies

This means the source package is published in SDK releases. This means that you usually have to wait for a new SDK release to get updates to this package and there's no clear way to submit patches beyond bugtracker

L3: Developed openly, closed source dependancies

This means that the source package is developed either in garage.maemo.org, maemo.gitorious.org or some other place. This means that you can follow development and contribute patches to it, without having to wait for new releases

However, this is usually important for system integrators and porters of Maemo, this package depends on a closed source package to build or run. Which means it is difficult to take the package and use it outside Maemo.

L4: Source published in SDK, closed source dependancies

This means the source package is published in SDK releases. This means that you usually have to wait for a new SDK release to get updates to this package and there's no clear way to submit patches beyond bugtracker

However, this is usually important for system integrators and porters of Maemo, this package depends on a closed source package to build or run. Which means it is difficult to take the package and use it outside Maemo.

Total L1-L4:

This is the percentage that is open source in this particular component. It is red if it's below 80%

L5: Binary-only package published in SDK. Source may exist but may be non-free

This means the package is published in SDK, but may be non-free and hence not possible to use outside Maemo.

L6: Package published under EULA in nokia-binaries

This means the package is published as binary-only in nokia-binaries, which you can get under EULA at http://tablets-dev.nokia.com/eula

L7: Not published except on device and SSU repositories

This package is only published in SSU repositories and as such you cannot get to it without a device.

In each component area, for each level, there is a percentage and a number. This is percentage of components in this area that is in this level and how many source packages. If you click it, you will see the packages from that component area in that level.

If you click this, you will see information about the package, it's level, where it is openly developed if so, where you can retrieve it, etc.

Key numbers

Along with the openness report, there's a few key numbers, which are completely unweighed regarding lines of code, importance, etc.

Total L1+L3 compared to L1-L4:

This is how many % of the open source packages of Maemo that is actually developed openly on maemo.gitorious.org, garage, etc.

Total L1-L4:

This is how many % source packages of the image which is open source

Total L5-L7

This is how many % source packages of the image which is closed source

Total L1-L4 compared to L1-L6

This is how many % of the published source packages that are open source.

The future of openness reports

The idea is to publish these openness reports quite often, as to keep a understanding on the direction we're going in. I plan to do one every PR release for Fremantle and when Harmattan SDKs starts appearing, doing the same. I plan to shine up the CSS a bit as well as it's not the prettiest currently.

I will very soon start maintaining a open sourcing queue and the hope is to release as many L3-L4 packages as possible to become L1 and L2 packages. And L4 and L2 packages turning into L1 and L3 packages.

Please comment any changes or ideas for future improvements, or questions. If there's components that are openly developed that are not represented, let me know as well. I'm 'Stskeeps' on #maemo and feel free to come and discuss with me there.

Dual-booting Mer on Nokia N900

noreply@blogger.com (Carsten Munk) — Tue, 19 Jan 2010 12:43:00 +0000

As of Mer 0.17testing10 we have an image that is fairly easy to try out on Nokia N900. This guide is for advanced users and there is NO WARRANTY. It might turn your N900 back into the crazy guy from this video or chew out your microSD, so be careful.

You will need:
* A microSD
* A SD/microSD card reader for your PC
* A Linux installation on your PC.
* A copy of a Mer rootfs for N900 (0.17testing10)
* A copy of the files in your /lib/firmware from your N900. This includes WiFi and Bluetooth firmware.
* PR1.0, PR1.0.1 or PR1.1 on your N900
* That you have installed fanoush's bootmenu (dpkg -i it and run 'Install bootmenu' application icon)

Instructions (must all be done as root):
* Partition your microSD to first partition being Linux and format it as ext3 using mkfs.ext3.
* Mount the ext3 partition on your PC, let's say in /mnt/mer.
* Make sure using 'mount' that your ext3 partition is -not- mounted 'nosuid' or 'nodev'
* cd /mnt/mer
* tar --numeric-owner -pzxf /full/path/to/mer-armel-n900-rootfs-v0.17testing10.tar.gz
* cd lib/firmware
* cp in the files from /lib/firmware from your N900
* umount /mnt/mer
* Make a file "mer.item" in /etc/bootmenu.d on your N900, containing:

ITEM_NAME="Mer (external SD, partition 1)"
ITEM_ID="mer"
ITEM_DEVICE="${EXT_CARD}p1"
ITEM_MODULES="mbcache jbd ext3"
ITEM_FSTYPE="ext3"
ITEM_FSOPTIONS="noatime,ro"

* Reboot your device, have the keyboard slid out. A bootmenu will appear where you can select Mer with the cursor keys.
* Mer booting.. wait for touchscreen calibration step
* And then you can run through the first boot wizard
* And a non-accelerated Mer desktop appears.

What works:
* WiFi
* Touchscreen
* Charging
* Watchdog handling

Of closed source blobs used in Mer (for N900):
* DSME (open source) with libcal (closed source) running from the Maemo5.0 rootfs
* BME running from the Maemo5.0 rootfs
* Firmware files for WiFi and Bluetooth chip amongst others.

Either way, I hope you enjoy playing with an alternative OS on your N900. Feel free to report bugs at http://wiki.maemo.org/Mer/Releases/0.17 and talk to us on #mer on irc.freenode.net if you'd like to contribute to Mer.

To shut down Mer, either sudo reboot or pop the battery.

Making flashable rootfs's for the N900

noreply@blogger.com (Carsten Munk) — Fri, 15 Jan 2010 10:57:00 +0000

For the N8x0 the way to create flashable rootfs's - that is, rootfs's usable in flasher-3.5 -r rootfs.jffs2 -f -R was through these two steps:

mkfs.jffs2 -d $ROOTFS_DIRECTORY -l -n -e 128KiB
-o rootfs.jffs2.raw
sumtool -l -n -e 128KiB -o rootfs.jffs2 -i rootfs.jffs2.raw

Now that the N900 uses ubifs for rootfs instead, how do you do create a flashable rootfs?

You need to make a file, ubinize.cfg:

[ubifs]
mode="ubi"
image="/full/path/to/base.ubi.img"
vol_id="0"
vol_size="200MiB"
vol_type="dynamic"
vol_name="rootfs"
vol_alignment="1"
vol_flags="autoresize"

Then, you run these two commands - you have to grab mtd-utils - Ubuntu Karmic has mtd-utils with ubifs support.

mkfs.ubifs -m 2048 -e 129024 -c 2047 -R 4MiB -r $ROOTFS_DIRECTORY -v /full/path/to/base.ubi.img
ubinize -o /full/path/to/ubi.img ubinize.cfg -m 2048 -p 128KiB -s 512

You can now run flasher-3.5 -r ubi.img -f -R.

What can this information be used for?

* Generate a full snapshot of your NAND rootfs and restore it with flasher after trying out something stupid that failed.

* Flash alternative OS'es onto your N900 NAND.

* Possibilities in rescue menu as in my previous post about bootmenu.sh hook - dump my rootfs to SD and I'll fix it on my PC and reflash it back to my N900.

A small but important update in PR1.1

noreply@blogger.com (Carsten Munk) — Thu, 14 Jan 2010 12:01:00 +0000

This is slightly technical, but PR1.1 included a small but interesting patch to /sbin/preinit (part of getbootstate). /sbin/preinit is the first thing that gets run on Maemo5 for N900. Some history first: In the past we've always had to patch initfs or /sbin/preinit in order to get any sort of bootmenu on 770/N8x0(W)/N900. This has changed with PR1.1 - the patches are now part of Maemo5.0!

Besides some patches to aid MMC booting, what is added is few very important lines:

SLIDE_STATE=`cat /sys/devices/platform/gpio-switch/state`
if [ x"$SLIDE_STATE = "xopen" ]; then
echo_g "slide open, attempting to use bootmenu"
[ -f /bootmenu.sh ] && . /bootmenu.sh
fi

What does this mean? This means, that when you have the keyboard slid out at power on, it will look for /bootmenu.sh and try to include it in the boot process. If it doesn't exist, it goes on with business as usual.

This means, you can add have fanoush's bootmenu or your own rescue menu, or whatever could be interesting to have this early in the process.

Thanks to fanoush for his bootmenu and to Peter De Schrijver from Nokia for applying my patches.

A creative use of MADDE: A new way for theme makers to make themes for Maemo5

noreply@blogger.com (Carsten Munk) — Tue, 12 Jan 2010 20:29:00 +0000

Recently, three source packages that were published in Diablo times was released in their Fremantle versions which I had waited for a long time for. These are:

* A source package which generates a theme, hildon-theme-variant-example
* This is done on the basis of hildon-theme-layout-5 (also published in Maemo5.0 update2 SDK), a package that provides the GTK theming and how to slice and dice a template into the theme elements.
* The slicers and dicers and dithers are provided in hildon-theme-tools (also published in Maemo5.0 update2 SDK).

hildon-theme-variant-example is CC BY SA 2.5, hildon-theme-layout-5 is CC BY SA 3.0 and hildon-theme-tools is GPL 2.0 for those who are interested in that.

Now, on to what this article is actually about.

What I am going to provide you theme makers of Maemo, is a way to make theme packages similar to how the vendor-provided themes on your device is made. We had this ability in Diablo and now we have it in Fremantle.

There was a catch with the Diablo method - it required you installing Maemo SDK, well, under Linux and this isn't always commonly found on graphics designers computers which typically ran Windows or MacOS X.

Thus, cross-platform solutions like Kontorri's Theme Maker was made which emulated what these packages described are doing. It made a binary Debian package, which could only be uploaded to Maemo Extras in the non-free section.

Normally, developers upload a source package to Maemo Extras and the autobuilder builds it for us.

Now, the ballgame has changed with MADDE. MADDE is a cross-platform Maemo SDK (Linux, Windows, MacOS X), which basically implements just about enough to cross-compile a Qt application. But why is this interesting for theme makers?

I've discovered that it is possible to generate the source package for a theme package within MADDE. What does this mean for your workflow as designers? Let me guide you through a workflow of theme design with the help of MADDE and the Maemo Extras repository.

Requirements:

* Install MADDE on your computer - it's a quite big download but it may be useful when you'd like to wander into Qt development as well. In this example I will be referring to paths from Windows MADDE but it should be transferable to your

* Something to open tar.gz files, such as WinRAR or similar.

* The garage.maemo.org and Extras upload invitation from http://wiki.maemo.org/Uploading_to_Extras-devel. Currently you will need to set a SSH key (write 'blahblahblah' if this is nonsense to you)

* Patience and willingness to learn :)

Workflow when beginning a new theme:

Step 1: Downloading the theme source package template

On this page there is a link that is called Download master as tar.gz. What is this? This is a theme template source package which you can customize to your own needs. Unpack the directory inside this into C:\MADDE\0.5\home\your username. Feel free to rename the directory to your intended theme name.

Step 2: Executing MADDE

In your start menu (or whatever your OS calls it's launcher), there should be MADDE -> MADDE Terminal.

You will now end up in your 'home directory' in MADDE, which is C:\MADDE\0.5\home\your username.

In there, you can 'cd directory name' where directory name is the name you gave the folder/directory from before.

Step 3: Customizing the source package template, basics

In the folder you should now 'sh try_it_out.sh'. This will ask you a couple of questions. You only have to do this once.

It will ask you the following questions:

* What should the package name of your theme be? (Example: hildon-theme-example)

* What should be theme name be? (Example: My example theme)

* Who is the maintainer of this theme?

* What should the directory name be (Example: example)

It will then set up the theme template.

Step 4: Customize the graphics - the fun step!

You can see in the directory several interesting places. The primary place for your work is in the template directory. This contains template.svg (SVG version of the basis template) and template.png (PNG version of the template). While it is up to you which one you want to edit, remember to always export template.png from the SVG.

Similar directories, icons, background images and other things to customize can be found in the applications directory.

Step 5: Building the source package

When you're done customizing, you can finally build your source package. In MADDE, you can do this:

username $ mad dpkg-buildpackage -S -us -uc -d

This will generate a .dsc and .tar.gz in the parent directory which you can use in the next step.

Step 6: Uploading to Maemo Extras

You can now upload this .dsc and .tar.gz file to Maemo Extras through the Maemo Extras Assistant. The autobuilder will then build the theme package for you and your theme will show up in Extras-devel.

As a bonus, anyone can base their themes upon yours as they have the 'source code' of your theme (under CC SA BY 2.5 conditions) - something not possible with ThemeMaker.

There are probably more maintaining issues which I hope the real theme makers will explore such as adding new versions to changelogs and such and tell about in their blogs.

If you have problems with any of the steps, prod me (Stskeeps) on #maemo , irc.freenode.net or comment this post.

Enabling RTC (clock), battery charging and watchdogs on alternative OS'es on N900

noreply@blogger.com (Carsten Munk) — Thu, 07 Jan 2010 14:37:00 +0000

Just spent a day getting RTC (clock), battery charging and watchdogs working for Mer on N900 and I thought I'd like to share how it was done for those of you wanting to run other OS'es on N900. The watchdog package in typical Debian does not by default work with two watchdogs, so we are using DSME.

This example assumes that you have made a /mnt/initfs directory on your rootfs and that you're running this from fanoush's bootmenu which will put the Maemo5 root file system in /mnt/initfs

You should as early in your boot process (after udev has been started), run the following script:

# Set up initfs environment

hwclock --hctosys

chroot /mnt/initfs mount -n -t proc proc /proc

chroot /mnt/initfs mount -n -t sysfs sysfs /sys

mount --bind /dev /mnt/initfs/dev

chroot /mnt/initfs mount -n -t tmpfs -o size=1M,noatime tmpfs /tmp

chroot /mnt/initfs mount -n -t tmpfs -o size=256k,mode=0755,nosuid,noatime tmpfs /var/run

# Check current boot state. Could be 'USER'
export BOOTSTATE=`chroot /mnt/initfs getbootstate 2>/dev/null`

touch /mnt/initfs/tmp/$BOOTSTATE

echo $BOOTSTATE > /mnt/initfs/tmp/STATE

chroot /mnt/initfs /sbin/dsme -p /usr/lib/dsme/libstartup.so &

until chroot /mnt/initfs waitfordsme; do

sleep 1

done

LOGGER='/usr/bin/logger -s -tBME'

SYSFS_VBUS_PATH=/sys/class/i2c-adapter/i2c-1/1-0048/twl4030_usb/vbus

# check dead battery pre-charge

if [ $(cat $SYSFS_VBUS_PATH) -eq 1 ]; then

chroot /mnt/initfs /usr/sbin/bme_RX-51 -b

case $? in

0)

$LOGGER -pdaemon.notice 'precharge -> ok'

;;

2)

$LOGGER -pdaemon.crit 'precharge -> power off'

exit 1

;;

*)

$LOGGER -pdaemon.crit 'precharge -> failure'

;;

esac

fi

# Start battery management entity
chroot /mnt/initfs /usr/sbin/bme_RX-51 &

# Bonus: To get hald-addon-bme working (bad bad voodoo). We need to test if toggles_w's hald-addon-bme replacement works on N900.
# dpkg-repack hald-addon-bme from rootfs and dpkg -i it
mount --bind / /mnt/initfs/mnt/new_root
chroot /mnt/initfs ln -s /mnt/new_root/tmp/bme-dbus-socket /tmp/bme-dbus-socket
ln -s /mnt/initfs/tmp/.bmesrv /tmp/.bmesrv
ln -s /mnt/initfs/tmp/dsmesock /tmp/dsmesock

On the perceived openness of Maemo, part one: Your target audience

noreply@blogger.com (Carsten Munk) — Mon, 04 Jan 2010 13:25:00 +0000

Some time ago, I stumbled over a rather aggressive comment in Quim's blog, which made me wonder if we are approaching the topic of open sourcing Maemo components in the right manner.

The comment in question took big offense to the way the page Why_the_closed_packages presents reasons for keeping things closed source and it appeared to me that the meaning of the different reasons in his world view was completely different to how I perceive them. I usually classify myself as a open source lover with a realistic sense of why there sometimes has to exist closed source software but I'll try to remain objective and respectful to both sides of the spectrum in this article.

To further understand this problem, I asked Paul Fertser of #maemo – one of the people I've had most intense debates with about licensing issues in Maemo and why things are the way they are, to read Why_the_closed_packages sections 1 and 3 (Reasons, Requesting the opening of closed components) with his bias towards free software and document his thoughts of it. The quotes are presented with his permission. His background is in the OpenMoko community which is on the more liberal end of the open source software spectrum in the mobile Linux space.

Why did I ask someone with a huge bias to do this? Because he has been showing willingness to try to understand why things are they are in order to make more sense of the open source situation in Maemo – and that is something I respect. In his reaction to WTCP, he writes pretty clearly why it is useful to approach it with this angle:

“I assume that the purpose of the page is to make the process of open-sourcing in particular and development in general efficient and constructive.

That means that neither your time, nor time of your colleagues should be spent on void talks about something you can't change anyway and that people proposing opensourcing a particular component should concentrate on providing the most relevant arguments instead of taking part in religious debates.

To fulfill this purpose the page should be understandable and acceptable by the target audience.“ --Paul Fertser

The people who are going to make the most noise about open source policies are who our target audience of this page should be in order to be able to move away from time consuming religious debates and into constructive debates and discussions. That said, the page should also target developers for whom open sourcing a component would help their development.

The important thing is to spend more time on the actual open sourcing process than in time consuming discussions.

PF suggests some guidelines for the page, which I quote verbatim as see mostly no argument in what he states, emphasis mine:

First of all, you mainly target developers. And you take into account that among those interested in open-sourcing there will be a considerable amount of "free software zealots" who are sceptical and irritated about the proprietary software by default. So to avoid aversion that will be later reflected upon you and your fellows in form of negative emotions and useless quarrels i think it would be beneficial to follow several guidelines:

* Write from a hacker point of view, that way they'll feel you're "with" them and not "against", that makes people feel much more comfortable, take emotional response of your readers into account.

* Avoid buzzwords and formal style, a convincing page should look nothing like a press-release as we all know everything that comes from a PR department smells like(added by editor) lies.

* Add technical details, that makes you look understanding the issues in deep, increasing credibility of other information.

* Do not use really questionable arguments, that's like a red flag to a bull.

* Take into account many developers do not understand business and business reasons, so those should be explained in a more detailed way.

To me it seems it would be nice to have some introductory words on the page, at least that would make me less irritated and more constructive right from the beginning. I would gladly accept something like:

"We all know proprietary software sucks. Indeed it does, many of the headaches could have been avoided if there was less of it. Unfortunately, the reality is working in such a way we have to make compromises, that's unavoidable. On the other hand, there are actual possibilities to free more of Maemo and Maemo-related software, read on to find out what can be done and how to perform that."

That would make me feel comfortable because it looks informal, friendly and understanding. Also it gives hope. A nice way to get a pleased and listening reader.

While I don't fully agree with the wording of the text or opinions of PR departments, there's some parts I do find worth to mention. The best way to deal with an emotional response due to a default irritation of proprietary software is to provide technical details how the process actually work.

Like what happens when someone requests the open sourcing of a component. Does it go to a manager who agrees or disagrees or is there a wider process of checking with lawyers etc?.

These processes are not documented and transparency of these processes will help understanding how things are proceeding and provide examples of how open sourcing is done in practice from within a company. It is important to note that open sourcing does not happen overnight and present the practices.

The next part of this series will be about the exceptions to open sourcing and discussing the different reasons for having some components closed source.

Thanks to Paul Fertser for his input to this article.

My first day as maemo.org distmaster

noreply@blogger.com (Carsten Munk) — Wed, 28 Oct 2009 13:58:00 +0000

Starting from today, I will be officially starting my work as maemo.org distmaster. Nokia has gracefully sponsored this position for the community at the request of the maemo.org community council.

The current maemo.org team cover areas such as bug management, system administration&collaboration spaces, documentation, forums and maintaining software repositories. All these positions facilitate development and collaboration in the community - one based around the Maemo platform and the Nokia devices. But one particular area is missing: the facilitation of tablet/device operating system development.

While maemo.org currently targets developers building on top of the OS, many people within the community is interested in improving the OS, since it is what they have to deal with daily - as Maemo users. In my time within maemo.org I've found there's been a reliance on Nokia to provide fixes for problems - instead of looking within the community for people who were capable enough to fix the problem, deliver the solution and publish it in a manner that non-developers can utilize the fix.

One of the maemo.org 2010 Agenda goals is to provide Maemo variants - which would be the pipeline for delivering community fixes.

Maintaining a Maemo variant is not a easy job - you have to deal with a lot of different stakeholders in a diplomatic manner, mentor and engage new developers, often do a lot of the heavy lifting yourself, faciliate the developing and invent new collaboration spaces for this.

Facilitating and engaging the community in this manner is a day job - which is why we have the maemo.org team.

I will have 20 hours per week (part-time until January of my own choice) where I will be doing mostly community work and some internal Nokia work as needed.

The primary role as a distmaster is defined as facilitating the development of Maemo variants (Mer, Diablo/Fremantle Community Variants, etc):

* Developing collaboration spaces for developers of Maemo variants
* Mentoring and engaging developers in order to move the Maemo platform and variants ahead
* Communicating with stakeholders (Nokia, distributions implementing the Maemo platform, users and developers)

Secondary role is serving as a paid system developer for the maemo.org community:

* Developing fixes and features for the Maemo OS & variants either on his/her own based on the input of maemo.org community council
* or through activating, mentoring for and collaborating with community developer(s) in order to get it completed in collaboration with upstream developers.

As I am also maintaining the ports and base system area for Mer, I will focus myself on the hardware support for Nokia devices and let other team members deal with other device communities/vendors to avoid any potential conflict of interest situations.

As primary facilitator for the Mer project I've successfully helped activate many developers within the community in order to get Mer to where it is today. A lot of these developers came to the Maemo project at the hope of being able to develop the OS but ended up contributing few pieces to move ahead - but there was noone to group all these things and incorporate them in the system, so the brilliant pieces ended up gathering dust - until now. Now there's a official community position to make this work happen.

Looking forward to working even more closely with this community.

Maemo Summit talks of interest to Mer team members

noreply@blogger.com (Carsten Munk) — Sun, 04 Oct 2009 09:31:00 +0000

So, Maemo Summit 2009 is coming up - and many people who are contributing to Mer are going to this event.

From my perspective, there are some interesting talks that apply to Mer contributors and I can recommend seeing:

Friday:

10:15 Maemo 5 and the Nokia N900 by Ari Jaaksi, VP - Maemo Devices @ Nokia

12:15 Harmattan Highlights - Developer Opportunities by Janne Heikkinen, Director - Maemo Product Planning @ Nokia

14:15 Why the Maemo Community matters to Nokia - Alan "qole" Bruce of Maemo Community Council interviewing Ari Jaaksi, VP - Maemo Devices @ Nokia

15:00 N800 room: Cross-platform with Qt - live - Ariya Hidayat, Software Engineer, Nokia
15:00 N900 room: UX panel (Tim Samoff et al)
15:00 N810 room: UI design of Maemo 5 apps - Annu-Maaria Nivala, UX team, Digia

15:30 N810 room: Developing widgets on Maemo 5 - showcase Foreca Weather,Juha Järvi, Software Designer, Foreca
15:30 N800 room: Mer - how the community innovates - Carsten Munk, Lead Developer, Mer project

16:00 N810 room: Developing apps with Qt for Maemo5 - Kate Alhola, Forum Nokia, Chief Guru on Maemo
16:00 N800 room: Your foundation for open-source innovation: TI’s OMAP processor-based Zoom platform. - Ameet Suri, Texas Instruments

16:30 N810 room: Developing apps with Qt on Harmattan - Ville Lavonius, Product Manager Developer Offering, Nokia and Jussi Mäkinen, Maemo Marketing, Nokia

17:00 N800 room: Harmattan Architecture Overview - Juha Tukkinen, Principal Engineer, Architecture and Sys, Nokia

20:00 Maemo Party in FlexBar coordinated by Jussi Mäkinen. It's a public bar and the doors are open to anybody. Our program will go until 23h or so, but the bar is open until really late with good music. Don't forget the Saturday schedule, though. :)

- where I'll have Mer stickers for your devices along for registered Mer contributors :)

Saturday:

11:00 N900 room: Designing UI for Maemo 5 - Mox Soini
11:00 N810 room: PyQt application development on Maemo - Attila Csipa

12:00 N810 room: Introducing the Harmattan UI framework - Tomas Junnonen
12:00 770 room: BOF, Extras/autobuilder/interfaces round-table (for those interested)

14:30 Lightning talks:

Development Nirvana: How Maemo Application Development Should Be - Andrew Flegg

DVCS? git? - How does that work then? - David Greaves (you'll want to know this skill)

15:45 N810 room: Adapting GNOME applications to Maemo Fremantle - Joaquim Rocha
15:45 N800 room: Contributing with Git & Gitorious - Johan Sørensen (we use Gitorious extensively in Mer)

16:15 N900 room: Mer - a year after - Carsten Munk (a talk on what has gone on in the last year regarding Mer)
16:15 770 room: Git hands-on workshop - David Greaves (useful skill)

16:45 N900 room: maemo.org Bug Management - Andre Klapper

17:30 N800 room: The Qt Mobility Project - Alex Luddy

18:00 N810 room: Maemo and oFono - Aki Niemi, Rémi Denis-Courmont
18:00 N900 room: The Maemo 5 Address Book - Mathias Hasselmann, Travis Reiter
18:00 N810 room: Publishing your software through maemo.org
Niels Breet
18:00 770 room: BoF: The future of GTK+/Hildon in Maemo Harmattan, Alberto Garcia, Claudio Saavedra

Sunday:

11:00 N800 room: Building for Mer - David Greaves

11:30 N800 room: Designing QT application for Maemo 5 and Maemo 6 - Sergiy Dubovik, Ian Monroe

12:00 N800 room: Maemo Platform Security: Principles and Concepts - Elena Reshetova
12:00 770 room: Extending the Hildon desktop - Marc Ordinas i Llopis (we are switching to Fremantle hildon-desktop in 0.17)

12.30: N900 room: Towards painless and quality translations - Dimitris Glezos
12.30: N810 room: Telepathy on Maemo - Marco Barisione
12.30: 770 room: What to do about /opt in Fremantle - Marius Vollmer

14.30 lightning talks:

How to speed up your Maemo application development - Raul Herbster
N900 HW architecture overview & power management - Igor Stoppa
From corporations to communities: responsible and effective engagement - Randall "Texrat" Arnold
Mer from a user's perspective - Tomasz Dominikowski

DebConf thoughts, part two: On cross-compilation environments

noreply@blogger.com (Carsten Munk) — Thu, 20 Aug 2009 09:20:00 +0000

This post is part two of my DebConf thoughts series, first post here. At DebConf we also attended three talks directly related to something we use quite often in Maemo and Mer: cross-compilation. The first was by Riku Voipio/suihkulokki about Scratchbox 2 (mirrored by me as I can't find the original URL) - which was also a maemo.org talk like ours, as well as the round-table session on Multiarch and a talk by Wookey on Crossbuilding on Debian for a derived distro.

Scratchbox 2 is the successor of Scratchbox 1 - which most of you know from Maemo SDK. It is different from SB1 in many ways. It operates with the concept of a host tools distribution - tools which are used to accelerate certain parts of a compilation process. Comparing to SB1, with their devkits and such, this approach is much more flexible in terms of maintaining the tools distribution and keeping it up to date with current packages to match your target. This was previously difficult to maintain in SB1 and has been the source of much headache for developers on Maemo to having to deal with old and broken tools which no longer was able to build 'modern' packages.

Multiarch is the term being used to refer to the capability of a system to run applications of multiple different binary targets on the same system. This means that the different architecture files (libraries, binaries, development headers etc) are hosted in the same file system. The proposal also includes ability to have cross-architecture dependancies of packages. This would mean it would be easier in the same system (such as Debian) to build packages for different architectures within one system. It would however often also mean that you would need same package versions in all architectures as the one you're targetting.

This problem leads me to what I intend to discuss: The builder approach of cross-compilation vs a user environment approach for cross-compilation.

A cross-compilation environment has to user-level emulate as much as to make the cross-compilation of the Debian package succeed.

A user environment will have to provide a full environment with interactive shell, editors, etc - and emulate 'being' the target architecture system. This imposes a greater strain on emulation and is more difficult to achieve.

What most Maemo developers encounter, is the Scratchbox SDK. It is a development environment for all your needs and supports different targets both native and foreign architectures. It provides an emulated Maemo environment for developers to test in and should be enough to develop for the Maemo platform.

Marius Vollmer once wrote a post on "The cardinal sin of Scratchbox". It mentions the overreaching redirection that SB2 inherited from SB1 - that almost everything from doctools to python is redirected in the Python environment. This sin extends to the emulated Maemo environment - and that's what harms developers. The emulated Maemo environment is skewed by the redirection of the cross-compilation environment, to the point that Scratchbox Maemo has little to do with how Maemo on device works.

In Mer we began back in the Maemo Reconstructed proposal to ponder how SDK could be different. What we ended up on is that the entire Mer system should be able to run on a given host for development. We currently do this through chroots and virtual machines. This means, when you develop and test your applications - you develop them natively, first for your host architecture and then test it on your host.

The host is a 'real' Mer environment like on every other device and should behave the same no matter device or architecture. Once you're done developing, you can either send the source package to a builder farm (in our case, OpenSUSE Build Service) - which will spit out cleanly built x86, ARM etc. packages for us to put on our devices and test or cross-compile locally with an automated cross-compilation environment. With Mer, it's even possible to build for ARM on your own ARM device - it's just a Debian/Ubuntu laying underneath the surface.

We've had great success with the builder approach of cross-compilation in Mer. Might it be worth considering making - in abstract terms X86 chroot tar.gz's/VMDKs of full Maemo (themes, icons, etc) for developers to use to match the experience without inference by the cross-compilation environment?

One for development - dpkg-buildpackage is possible (coreutils instead of busybox) and one that matches a typical Maemo device. And then when you want to build for a different architecture, abstract it away in a one-liner that takes a source package as input.

DebConf thoughts

noreply@blogger.com (Carsten Munk) — Mon, 03 Aug 2009 10:23:00 +0000

Recently, zenvoid and I recently participated in DebConf9 and presented an one hour BoF session on Mer - made possible by sponsorship by Nokia for our travel and accommodation.

Arriving in Caceres, Spain - after an exhausting and 20 hour journey from Denmark (train, flight, metro, and then train again) and meeting up with my local travel companion zenvoid - we found our hotel, which had free WiFi - with working power saving management, a must for tablet users. :)

At our first day there was a talk on 'HP and Debian' where some abstractions were presented that apply very well to the Mer project. Quote from presentation:

A key attribute of Linux and many Free Software applications is that they are developed and supported “by the community”
What does that mean?
No one company in charge
A range of contributors with varied interests, abilities, and motivations
Innovation often comes from surprising places, thanks to “the long tail of contribution”

The Mer project likewise consists of a lot of different contributors from different device communities. Some are testers, some are developers, some are artists and some are users. Mer is effectively the result of a long tail of contributions from these people. Our innovation is the mix of this long tail of contribution - and the responsibility of the project is to direct future contributions in the right direction that will encourage new contributors to join in, maintain current contributors, mentor new developers and to encourage users to actually use our software.

Another part of this presentation is the angle of HP - what it wants and a comparison to what Debian developers want. Many comparisons between maemo.org and Nokia/Maemo Devices can be drawn in this part. HP needs revenue, growth, differentiation and corporate reputation. Differentiation is a word often heard in a negative context when speaking of closed source parts of Maemo. The reason why differentation is needed - is the fact that those 20% closed source parts together with the hardware is what generates the revenue, growth that makes it possible to have the 80% open source (and free) software such as Hildon and other Maemo APIs exist. And have active developers developing on them.

This is the reason why we, in Mer, realize that differentiation is needed to compete in the mobile device market. However, like Android, we provide a full base system - and understand that device developers needs to differentiate on top with extra bits. What we're trying to do in our vendor social contract is to have both a open system and yet allow the users of their devices to remix and alter their devices, even though closed source bits (codecs, firmware, special applications for vendor, added value) exist and are needed for full functionality. It would make the user able to use their devices even to the fullest even after the vendor has long given up on the device in question.

In the second day, we had our presentation - in the BoF room, no projector and a deadly hot room with noisy aircondition- but 20 people approx showed up to hear the talk. The TuxBrain/Freerunner buzzfix people were friendly and recorded the session on video (TBA). What we did was to put Mer on Freerunner, N810, N800, 770, SmartQ5 and simply line them up - and after the talk invite people to come and play with the devices and Mer.

I'm often very critical of my own talks - but all in all it went well, even though I did forget the one key element to any presentation: telling people where they can find out more about your project :). People seemed to poke curiously at our touchscreen devices and ask questions about them. One thing to notice about DebConf was the large amount of Nokia tablets and Neo Freerunner's represented.

After our session the OpenMoko buzzfix party went on, where we showed Mer on Freerunner to different people - and saw how people used their Freerunners. Something that really caught my eye was Qalee , which is a Qt-based environment for the Freerunner. Seems like the guy making it sure has a talent for both code and UI design. See screenshots here

In between the talks, we spent most of our time in the hacklabs, - where we met people like suihkulokki (Riku Voipio) and p2 (Peter de Schrijver) from Maemo Devices - and we even saw a N810 debug board with JTAG and serial ports.

In order to get to talk to people, we put out all our gadgets running Mer on the table, and began hacking - hoping it'd attract. And it did. - we met and talked to a lot of people and we hope that they think of Mer next time they'd like to have an environment for their tablet-like devices - or if they talk to someone who would.

Debhelper in Maemo is fairly old - we saw a talk 'not your grandpa's debhelper' - where it's shown how simple a debian/rules file can look these days with a modern debhelper. What really impressed me was this example:

#!/usr/bin/make -f
%:
dh $@

Which would autodetect if the package used autotools, setuputils, etc, and then build the package according to this. It obviously supported overriding things as well, and in a fairly simple manner.

This is the first out of a small bunch of posts about experiences from DebConf (this is the first), and thoughts on communities creating technology.