libbitcoin

libbitcoin after 6 months of active development is now a 1.0 release. Software becomes a 1.0 release when the developer decides that it has become solid enough to warrant it. It’s not a sudden change, but a nice milestone that’s been reached.

libbitcoin is vital for a bitcoin ecosystem. In the future we will need all kinds of bitcoin-esque applications; things that behave like bitcoin nodes but aren’t or do a specific task. libbitcoin is a piece of software that helps people achieve this goal.

We call these pieces of software libraries. They are a useful collection of code that developers can use in their bitcoin applications. Instead of that developer spending 2 months writing all the core bitcoin code they need, they can instead use the 6 months of work we put into designing a feature complete resilient piece of software. I thought of the issues for you and did the heavy lifting work.

When designing libbitcoin, I conceptualised what the goals of the project were. The 3 point plan is:

• Flexible
• Scalable
• Asynchronous

This feature-rich software will be extremely popular and likely catapult Electrum to the front of the client-race, though you never know what the next guy has in his back pocket.
~ GoWest

Flexible means that it should not constrain the programmer with any particular one style. libbitcoin is a toolkit library as opposed to an application framework. Generally I’m opposed to frameworks as I see them as being too constraining.

A framework is a complete program with a few empty stubs. The user of that framework then goes and fills in the program with the missing functions. It’s a place to hang your hat.

A toolkit library by contrast is a useful set of classes and utilities. You are free to adopt and use it in any form you see necessary.

Both have different downsides. Generally frameworks are easier to use (in the short term) but have a flattening curve of gain over time. Toolkits are fairly linear in their gain over time, but initially require more effort for integration and setup.

Scalability is a big concern for bitcoin. I see bitcoin running on servers and consuming huge resources in the future. For this reason, I designed libbitcoin to be highly scalable. From a design standpoint the code is highly modular and abstracted to enable pieces to be designed with better algorithms.

To achieve scalability, it is a common tactic to divide code into separate threads of execution. Say that we have:

> Read transaction from hard drive
... Wait for hard drive to respond
> Send version message
... Wait for network to respond
> Calculate a hash
... Wait for CPU to finish


Before we can calculate the hash, we need to wait for the send to the network to finish, and before we can do that, the reading from the hard drive must finish. If the hard drive is being slow, then everything slows down because the second task doesn’t begin until the first one does. We call this synchronous (linear execution).

The common way of dividing these tasks up is to have all 3 of those operations run asynchronously (at the same time):


> Read transaction from hard drive
> Send version message
> Calculate a hash
... Wait for hard drive to respond
... Wait for network to respond
... Wait for CPU to finish


Terminology:

libbitcoin – a piece of backend software for creating software to interact with the bitcoin network and blockchain directly

Stratum – an intermediary between front-end clients and back-end servers

Electrum – the original client which implemented this idea, and later spawned Stratum. Now uses Stratum as it’s intermediary layer.

Usually it is the programmer who surgically divides the code up into sections and adds a bunch of primitives to coordinate all these threads of execution. Those setups are usually very inflexible to change and make it more difficult to scale a piece of software up or down. Bitcoin-Qt is the classic example – synchronising threads is a big issue there. A large portion of slowdowns are simply where it waits for 2 seconds (or whatever) just to be sure everything finished, or a large amount of bugs are problems where multiple threads try to access the same piece of data at once and lockup or whatever.

The way I solved this problem is that different units in libbitcoin are divided into ‘services’. You may have a blockchain service and a transaction_pool service (unconfirmed transactions). In libbitcoin when we create those services, we pass them a reference to an async_service object. The async_service is running an internal thread pool.


async = async_service(2)
tx_pool = transaction_pool(async)
chain = bdb_blockchain(async, "database")


We just ran the tx_pool and chain in the same thread-pool which contains 2 threads. If we want to run those services in more threads, we can either pass more threads to async_service or call async.spawn(). And the tx_pool and chain can both have a different async_service thread-pool.

This means that as the user’s hardware grows, or if there is a speed bottleneck being seen somewhere, you can simply throw more threads at the problem. Throwing more threads doesn’t reduce functionality or cause any problems. The services are intelligent and know how to coordinate themselves internally using a bunch of tricks.

This asynchronous nature means that anyone using libbitcoin in their applications is never blocking. If I perform some operation or do a task, libbitcoin will not block and instantly returns control back to the main program. It then does its thing in the background until the operation is complete, it wakes up and notifies the main program of the result.

This means that programs using libbitcoin have an instant startup time and that acting out and interacting with libbitcoin software doesn’t have long pauses. Actions are instantly reactive.

Being written in C++, I took up the design paradigms of that language:

• Atomicity – functionality is provided at the smallest units rather than making large abstractions. This makes it less intuitive but more powerful.
• Consistency – try to be consistent and logical throughout. Mainly this is reflected in my code’s look which always follows strict formatting rules.
• Encapsulation – think how to provide units of code. Build houses out of bricks, cars out of components and programs out of units.

One big disadvantage of libbitcoin is ease of use. All this interaction with threads and fine grained control is advanced concepts that don’t make it run automatically.

libbitcoin has bindings for an easy to use language called Python. However what is classically considered good C++ code, is considered unpythonic and bad. Python opposes C++ good practice on several points – namely its concept of encapsulation and its lack of atomicity. Good Python code should provide a complete interface (“batteries included”) rather than the very basics. And this encapsulation is presented in the most logical manner to the user, rather than the most maintainable ‘atomic’ manner.

I realised then that libbitcoin is the bottom rung of a full bitcoin system. I started to think of what the next layer would be.

Electrum’s world view

Electrum is the future of bitcoin. ThomasV created a client which stores the blockchain on a server, while the user interacts with the bitcoin blockchain through that server.

This avoids the hassle of needing the entire bitcoin blockchain – someone else manages that.

Some might argue that light headers-only clients are the future of bitcoin, but I disagree. I think ThomasV’s Electrum model is the way forwards.

Done properly and it is just as secure as vanilla bitcoin. Instead of trusting one server, you don’t accept any payment unless 6 servers out of the 8 you’re connected to all report the same payment (6 and 8 are example numbers). You are trusting 8 servers here – the risk has been spread acceptably.

I will avoid the logical parallels to email and web servers.

Stratum

The Electrum client interacts with the server using a special language. Previously this was the Electrum Overlay Network. GoWest from TheBitcoinTrader blog listed it as number 5 on his list of the 10 most anticipated bitcoin projects for 2012.

Still looking for an official name to distinguish itself from the Electrum client (I like Overbit, myself), the Electrum Overlay Network is looking to integrate many of the services that are currently found elsewhere, as well as some that do not already exist, and bring them under the umbrella of a single platform.

…

This feature-rich software will be extremely popular and likely catapult Electrum to the front of the client-race, though you never know what the next guy has in his back pocket.

slush took this Electrum server and started improving it. He needed a similar piece of code for his mining pool, and instead of creating something new, he decided to build on something already existing by pooling his efforts with ThomasV. GoWest is referring to slush’s as yet unnamed improvements.

While improving it, he made substantial changes by reforming the API and adding a bunch of different mechanisms for transporting the data over the wire. This new API was born and we now call it Stratum.

Stratum is an intermediary between front-end clients interacting with users and back-end servers that manage the bitcoin blockchain.

This is what Electrum uses, and I found the concept exhilarating. While libbitcoin is built for the expert developer who needs fast serious bitcoin applications, Stratum is for ease of use and building bitcoin user interaction software like merchants or clients.

Satoshi client problem

I think the Satoshi client is a dead-end. We originally believed the deep structural problems to be fixable when we forked the codebase and worked on Freecoin. But the code is poorly written and unscalable.

The worst part is that it tries to be many things at once: core bitcoin code, a poor-man’s Stratum-type interface and a user facing client. Good opensource software is split into many components and layers. By splitting software into components, specialists can work on their favourite parts independent of the rest. It allows developers to better visualise the entire system abstractly. And it generally results in clean concise code. That’s the UNIX way. The Satoshi client is a huge hodge-splodge of random things.

There have been tentative efforts to improve this – my favourite was the new Bitcoin-Qt client designed by Wladimir J. van der Laan. He crafted an intermediate layer between Bitcoin-Qt and the core Satoshi codebase. This makes Bitcoin-Qt abstractable and self contained. It is a project within a project, and it’s conceivable that I can take his code and switch out the Satoshi code with libbitcoin easily. The code written by Wladimir is in my opinion very well designed.

The defining mantra for open source is the Unix philosophy.

Write programs that do one thing and do it well.

Here is a bad Unix program:

It is made from lots of highly specialised parts that cannot be reused in other models. Below is a good Unix program:

It can be used everywhere and anywhere. It fits easily with other lego bricks and has no special design.

Electrum

Electrum is cool. It uses a remote blockchain server so there is no blockchain download and startup times are instant. You keep the keys (wallet) on your computer and the server never touches them. And you can use your keys on multiple computers without having problems with them. That means you can use the same keys on your desktop and android phone.

ThomasV and I collaborated on a refactoring the Electrum server by building a framework for creating Stratum applications. Then we slotted in libbitcoin. For now it’s pre-alpha but we hope to have it replace the aging ABE backend in the future.

This is the way – having an efficient ecosystem of people working together means we can build applications that work well.

Last summer bitcoin got a lot of press and attention, but there was no growth. This is a really exciting time for me, and we are seeing true development happening here. The bitcoin ecosystem is gaining some stability pricewise and development is keeping in steady advancing lockstep. Interesting times afoot.

Linux

Installation from gitorious (Linux):

sudo easy_install ecdsa
sudo easy_install slowaes
git clone git://gitorious.org/electrum/electrum.git
python ./electrum/client/electrum

Tar archive: Electrum-0.43b.tar.gz.

Windows

• Python code:
  • Install PyQt4
  • Install Electrum-0.43b.zip
  • execute ‘python electrum’
• Executable: (created by BTCurious): check in the forum thread

Forgive me for including this social media spam but it helps our dinky small site get exposure D:

•