First working Ethereum C++ cross-builds

TL;DR

Ethereum C++ cross-builds were birthed into the world on Saturday 28th November 2015.   This is a really important milestone, because it opens the door to near-universal portability of Ethereum.

PiZero

A cross-built eth running on a Raspberry Pi Zero, courtesy of Martin Brook

As George was kind enough to say recently:

The work that @doublethink_co are doing on Ethereum is important and you should check it out! – George Hallam

George

I have time to read more details …

The Light Client (LES) sub-protocol is still being refined.   When we have working LES implementations the flood-gates will really open on the range of devices which can run Ethereum nodes.

Getting to Light Client on mobile/wearable is the ultimate goal of all the work which doublethinkco is doing, with the Samsung Gear S2 smartwatch as the flagship target device.

Zsolt Felfoldi gave a very interesting presentation on his Light Client progress at the recent devcon1 conference.   The YouTube video of his presentation is unavailable at the moment, but will likely be available again in an edited form soon.

LightClientDevcon

To my knowledge there is no C++ implementation of LES underway yet.  That was an intentional decision.  No doubt the C++ implementation will get underway as soon as the details of the sub-protocol are settled, and the go implementation has been given a decent workout.

Bob Summerwill and Anthony Cros have been working on Ethereum C++ cross builds since September, after the ETHDEV team came to the conclusion that the work-in-progress on Light Client needed to be rebooted as a sub-protocol, rather than as extensions to the existing ETH protocol.

That change of plan left us with no real way to contribute directly to Light Client development. We needed to get ourselves out of the way while that design work was brewing and to do something else which was useful and would help the ultimate end-goal.

So we took a detour, and focused on porting the existing C++ client code to as many mobile and wearable devices as we could, with a primary focus on Linux over Android and iOS.

02 hotel porter_std

Having the existing full client available in as many places as possible means that we can just “turn on” Light Client on numerous devices as-and-when the C++ implementation of the new sub-protocol is available.

Perhaps we can even help with that Light Client work directly ourselves?

That was our original intention, but we were just too early for that to be practical.   Maybe it makes sense now?

Our detour has gone very much as we expected. That earlier article also talks about what cross-builds are and why they are important.

We realized quite early that our needs were massively aligned with John Gerryts (EthEmbedded), who needed to cross-builds for the final phase of his Devgrant for embedded devices.   So we’ve been working closely together most of the time, and especially in the last week or so, as we have had binaries to share.

Martin Brook has just joined the party too.  He’s one of the main Sailfish OS porters, and his input and testing has been very useful.

Martin

So why is the C++ client so important?

We’ve been working with both the go and C++ development teams, but because of Péter Szilágyi‘s excellent work with xgo, there has been close to no need for us to do any work there.

peter

Why not just leave things to the go team?

It’s because C/C++ is the foundation of all systems programming and performance critical applications, and it has been that way since the very first UNIX releases, and will likely be that way for many years to come. C/C++ is always the first language on any new platform so code written in C/C++ has universal reach, all the way down to the very smallest of devices.

We are finally starting to get some innovation and competition in the systems programming space with go and rust, but it is exceedingly unlikely that those languages will ever make a significant dent in the wearable, embedded and IoT space.

With the renaissance of Modern C++, it is possible to write C++ which is maintainable and understandable as long as you are disciplined and stick to best practices.   It isn’t easy, and elements of C++ like the pre-processor, compilation model, lack of cross-platform build system and lack of cross-platform packaging system make C++ “not fun” to work with.  Bob has been doing it for nearly 20 years though.  Not a problem.

Doesn’t Moore’s Law mean that this kind of “close to the metal” performance doesn’t matter anymore?

Nope, because we’re running into Amdahl’s Law, which essentially states that the potential speedups for a given application are ultimately limited by the work which can only happen in a serial manner.   Having 16 or 32 cores doesn’t help if your application isn’t of a type which can benefit from massive parallelism.

In addition, raw performance is going to be critical for the resource-constrained devices which are going to be spawning in their billions in the next few years with the growth of IoT and smart devices.

 

Why the focus on mobile Linux, not Android?

Isn’t Android an open source platform, and shouldn’t that be the primary focus?

Android is really not as open as Google would like you to think (excellent article on that a few years back), and Google are obviously a nefarious data-farming corporation in general.   Replicant are trying to “free” Android, but it’s a losing battle.   All mobile devices have serious privacy and security issues.

Everybody already knows about Apple’s control-freak nature.  Ethereum may or may not ever be allowed into the App Store.   They have got form.  Don’t be surprised if they ban iOS and watchOS applications which use Ethereum.  It is a mortal threat to their monopolistic control of all commerce and applications on THEIR platform.   We’ll port to iOS and watchOS, but it might never be shippable.

Anyway, to get to the point, we’re nearly done. We’ve got our first running C++ code-compiled builds working now, and will extend that across the rest of the test matrix in the coming days and weeks (months? I hope not!)

So what have you built?

Dockerfiles which let you cross-build ARM binaries which are already known to be working on Raspberry Pi 2, Raspberry Pi Zero, Odroid XU3 and Wandboard Quad. We are working towards working binaries for a very significant range of devices.

docker

Everything is there for whoever wants to use it the
doublethinkco/cpp-ethereum-cross repo on Github, which includes the full test matrix in the README.

What is left to do?

“Just work”, namely …

… then Bob will get automated builds and automated testing running on the mobile/wearable home test farm he has been building.

20151120_083926

Following the go team’s lead, we are likely to start publishing cross-build binaries too, as soon as they are in a stable enough state for that to be worthwhile. It would be very useful to have matching cross-build binaries for the forthcoming eth-1.2.0 release, for example.

If you have further mobile/wearable/embedded devices which you think need some Ethereum love, please tweet us and we’ll see what we can do!

Bob Summerwill, 30th November 2015

Github repo README revamped

Check out our webthree-umbrella-cross Github repo.

Yesterday I spent some time improving that front-page so that the project is close to self-describing. We’re getting very close to having working binaries, and are attacking multiple platforms in parallel.   We have (currently broken) releases which anybody is welcome to use to help to with our debugging efforts.

We’re working with EthEmbedded to get this cross-build infrastructure working for single-board-computers as well as for our mobile/wearable target devices.    Join us in the Gitter porting channel if you would like to help out, or just follow along.

Our new-and-improved DOT dependency graph is in a Pull Request for merge back to the main ethereum/webthree-umbrella repo.

Jolla Phone has arrived at doublethinkco!

I know I’m a bit late to the party, and that it is November 2015, not November 2013, but my Jolla Phone has arrived in Vancouver!

 

 

I had to order the phone indirectly, for delivery to my parent’s house in the UK, and then re-mailed from there.   I have heard some whispers of 3G working for people in Canada, and will report back on whether I can get that going with my Rogers provider, or if I’m tied to 2G.

The company could use some good news, and I hope my purchase and continuing intention to bring Ethereum to Sailfish will raise a faint smile on some weary faces.   A familiar name popped up to help with that process yesterday, with the arrival of Martin Brook (vgrade) who is trying a Sailfish native C++ build in parallel with our ongoing C++ cross-build efforts.  Great to have him helping!

Ironically, getting the Sailfish cross-build working on the Jolla Phone is likely my quickest route to a Tizen Gear S2 version.   Because the whole platform is built for developers, not for dumb consumers!   No GDB on the Gear S2, for example.

Best wishes, Jolla and sailors!

Mobile/wearable test farm takes shape

Laura Kalbag from Ind.ie did me a huge favor recently when she pointed me to the superb DeviceLab from Vanamco.   As a background task I have been putting together a home-based mobile Linux test farm, which I will be using for automated testing of our Ethereum ports.

Before:

20151015_092228

After:

20151120_083926

And all of this is going to sit behind the pfSense box I built, when I finish the OS setup and cabling on that.    I might revisit my wireless router setup at the same time.

So what have I got racked up on those two boards right now?

Left hand board (mainly Tizen)

Form factor Device OS Version
Smartwatch Gear 2 Tizen 2.2.1
Smartwatch Gear S Tizen 2.3.0
Smartphone Samsung Z1 Tizen 2.3.0
Smartphone Samsung Z3 Tizen 2.4.0
Developer phone Samsung RD-PQ Tizen 2.3.0
Smartphone Nokia N9 MeeGo 1.2 (Harmattan)
Developer phone Nexus 4 Sailfish 1.??

Right hand board (a very random selection)

Form factor Device OS Version
Smartphone Nokia Lumia 920 Windows Phone 8.0
Smartphone Geeksphone Peak Firefox OS 1.0
Smartphone Blackberry Z10 BB10 10.0
Smartphone iPhone 3GS iOS 5.0
Smartphone iPhone 5 iOS 5.0
Smartwatch Samsung Gear Live Android Wear 4.4
Smartwatch Apple Watch Sport watchOS 1.0

And what else do I have which isn’t hooked up yet?

Form factor Device OS Version
Developer phone Samsung TM1 Tizen 2.4.0 Beta
Smartwatch Samsung Gear S2 Tizen 2.3.1
Smartphone Meizu MX4 Ubuntu Edition Ubuntu Touch ??.??
Smartphone Jolla Phone Sailfish 2.0
Tablet Jolla Tablet Sailfish 2.0
Tablet Nexus 7 Android  ?
Tablet Samsung Tab S Android  ?
Tablet iPad 3 iOS ?
SBC Raspberry Pi Raspbian ?
SBC Odroid-XU4 Tizen 3.0
SBC Intel NUC Tizen 3.0

And completely unrelated to this effort, I also have my Commodore 64 and Atari VCS to get connected!

 

 

We got on screen at #devcon1 on the final day

Many thanks to John Gerryts for crediting Anthony and myself as contributing to EthEmbedded project, as shown on the “DEVgrants Update” slides at the Ethereum devcon1 conference today.

We got on-screen!   Somewhat over-credited as part of the EthEmbedded team, but we’ll take it 🙂

DoubleThinkOnScreen

It was fantastic to hear of the additional $100K funding from Omise.   We will be applying for a grant for our cross-compilation and porting work 🙂

Porting Ethereum – geth running on Samsung Gear S2 smartwatch. eth coming soon

Back in early June of this year I made a build of ethereumj-android on my Mac, ran it on my Samsung Galaxy S6 and made a video which was pretty popular on the Ethereum sub-reddit at the time:

That was just my first step on a journey towards a Light Client for the Samsung Gear S2 smartwatch (and various other mobile/wearable Linux devices).

Samsung-Gear-S2-AH-faces-1-1600x1067

That aim has come closer to fruition this week, with a working (full node) geth on device, and getting an eth executable on device.   We’re still working through runtime issues on the eth executable, but it’s very close now.

Nearly all of the credit for geth porting lies with Péter Szilágyi and his majestic xgo CGO cross-compilation solution.   We really didn’t have any work to do on geth itself.   There is still a sporadic crash to chase, but it’s workable.

peter

The block-chain download is slow on the smartwatch, but the main issue is storage space, with only 4Gb of on-device storage.   The Morden test-network fits, but the Frontier main network is already too big.   There is no big surprise here.   The need for a Light Client is the motivator this whole project, but getting the full client working on device is a good step in that direction, because the infrastructure is now ready and waiting for the Light Client protocol to take shape.   Geth is particularly important to get working, because it is the Ethereum client which will get Light Client support first.

While it is great to see a working client on-device, the larger progress which we’ve made which is on the cusp of completion is adding cross-compilation support for eth, the C++ client.   Having C++ cross-compilation is critical for resource-constrained platforms which cannot support compilation on-device.   The performance and universal reach of C++ will also be very important in bringing Ethereum to the very smallest IoT and embedded platforms.

Anthony Cros is sub-contracting for doublethinkco, and has done 90%+ of the work.  The C++ side has been quite the battle!    I’d just like to publicly thank Anthony for all of the blood, sweat and tears he has poured into this work over the last couple of months.   I know he will be very happy and relieved when we get eth running 🙂

anthony-cros

Tizen has been a test-case for us, because it doesn’t support native compilation even on mobile, let alone wearable.   We tried to get that working, but it was a dead-end.   So then we flipped to cross-builds, first on the TM1 developer phone, and then just in the last days on the Gear S2 itself.

samsung-gear-s2-press-169

The cross-compilation support consists of:

So it looks like this porting tangent has gone much as I expected it might when we started on it just under two months ago.   We will continue to work with EthEmbedded and SlockIt to ensure that the C++ cross-builds are working for all platforms of interest, and to get the cross-build support upstreamed.   We’ve not started on Sailfish OS and Ubuntu Phone ports, but they will be very similar to Tizen, and may even “just work”.

After that, I hope that we can make some more direct contributions towards the Light Client implementations, starting with geth as Zsolt described at devcon1 yesterday:

Screen Shot 2015-11-13 at 12.20.58 AM

When those come to fruition we can all go and build some cool DAPPS for smartwatches and the 50 billion smart-devices which allegedly will be making their way into the world by 2020.

For months, I have been trying to push in the direction which SlockIt teased and delighted us all with a devcon1 today – bringing block-chain technology to small Internet-connected devices:

Wearables, IoT, smart-devices, embedded systems.   We are heading into a very connected future.   I want that future to be decentralized and under our control.

Hold on to your hats!   We’ll be there soon 🙂

Porting Ethereum – Tizen cross-builds. Update by Anthony Cros, 5th November 2015

20151103_165153

We are still working on 2 fronts in parallel, the webthree (C++) implementation (formerly known as cpp-ethereum) and the go-ethereum (Golang) one.

go-ethereum (Golang)

Block synchronization problem

Not much progress on that front beside confirming an issue that seems to occur with ARM builds regarding block synchronization (see previous post). It seems other people have reported the same issue and hopefully the devs will figure it out soon.

LES sub-protocol (light client)

We are also waiting for the LES sub-protocol to be finished, which according to the latest update (see “Light client/mobile” section) is making good progress.

web-three (C++)

Cross-compilation

A lot of work has been put into cross-compiling the C++ implementation, and after a lot of sweat I finally produced a binary to test out! Unfortunately it results in a good o’ segfault for the moment. I have informed the devs of it and am waiting for their feedback. It may be an issue with the CryptoPP library, which is actually due for removal (see Gavin’s comment on the matter).

Code

We are making the process of producing this binary open for anyone to try out (see our Github repo).
There exists a Dockerfile that can be used as basis, although it is not finalized yet. Still, a basic Ubuntu 14.04 image to which the packages listed in the Dockerfile are added should provide the right environment to run the scripts above.

Documentation

I have also spent some time on improving upon the devs’ representation of the various dependencies involved in building eth. I started from the original DOT file:
BwKjPZ6
and added information I gathered from compiling from source myself as well as various comments from the devs:

dot

which results in

rauzXbx

It is still a work-in-progress but we will hopefully have a good overview of all these dependencies soon.

Stay tune for more progress!