Linode has cost me more than 3000 BTC (nearly 12k EUR at current rates) due to a security flaw in their platform. Linode is a cloud computing and web service providing company that is a popular hoster of web services throughout the world. Today I woke up to find my hot wallet on the backup server had all its coins stolen. As a security measure, merchant sites keep the majority of funds offline with just enough in a ‘hot wallet’ to keep operations running smoothly.

Rest assured: I am covering Linode’s mistake from my own income. That means months of my work is wasted and I’m crushed.

Especially upsetting is that I went to great pains to keep everything as secure as possible. But that was all rendered worthless when somebody hacker the upper level service provider. All that time of mine has gone down the drain for nothing.

It seems that also the user database has been compromised. Although passwords are stored using SHA1 with a salt, I strongly recommend to change your pool password immediately.

How it went down

This morning I received an emergency SMS notification that my pool’s bitcoin balance was low. I started investigating and the chain of events turned up strange anomalies. I then noticed 3094 BTC moving out of the pool wallet. I could only sit helpless as the money got confirmed by the network.

While watching the logs, it did not look like the server had been compromised at all.

Then I found that two of my Linode machines has been restarted half a hour ago, too, and the root passwords had been changed. I changed the passwords and found that there was malicious activity on the machines. Then I discover that the passwords were changed over Linode Manager (Linode web management), because there was record about the password change in the Host Job queue (last activity done over the Manager).

I reported accident to Linode staff and asked for log of recent logins to Manager. To my surprise, there were only my own log attempts and last login before the attack was on 08/02/2012! I reported to Linode that something is terribly wrong, because I had been using strong password for my Linode Manager (because I know it’s basically backdoor to my machines) and I didn’t use this password on different places.

Full log of support ticket is here.

I’m still waiting what they’ll find.

Linode is a top provider and lot of people trust them with serious business (like me until now). If they’ll see that Linode is trying to hide this issue, maybe people change their mind.

As a respected hosting provider, I hope they do the correct thing and refund me for this liability due to their error. Many people trust Linode, and they have proven themselves as a serious contender for hosting critical sensitive operations on the internet. I would hate to not see them live up to that reputation.

I am not the only person affected by this. A few hours ago another guy contacted me that his Linode machine has been attacked and his coins was moved to the same wallet, asking me if I knew what happened (because he found that the 1Mining2 address is mine). We found that our issues are the same – changed password in Manager, stolen coins & Linode staff is telling they have no security issue on their side.

It looks like attackers found some vulnerability of Linode Manager and used it to infiltrate Linodes with running bitcoind (we both had bitcoind running on the machine), to gain maximum profit for the least exposure; it does not seem many other machines were compromised, and we found no information on Twitter or anywhere else. It looks like the attackers were interested only in Bitcoins, because they left Namecoins untouched, although they had the same chance to steal them.

The Bitcoin faucet (Gavin Andresen) has also had its coins stolen.

From the attacker’s wallet it looks there were more people affected by this Linode hack, maybe they’ll know anything more?

There’s no reason to think that pool itself was hacked. I changed all passwords everywhere (mainly to database), moved coins to new wallet and everything is working fine. Backup machine didn’t contain keys for accessing pool server, so there’s no need to reinstall pool to another machine. I’m covering all financial loss from my own money, to keep pool users out of this stupid issue. It is between me and Linode.

Update:

Linode has confirmed that the error was due to a fault on their side.

Hello Marek-

We were alerted to the suspicious activity and have identified and corrected the issue. Our investigation has revealed a customer support interface was used to access your account. The compromised credentials have been restricted and we are discussing policy changes to prevent this from recurring.

We regret that this incident has occurred, and apologize for the unnecessary work this may have caused you.

We appreciate your business and certainly want to keep you as a happy and satisfied customer. If there is anything we can do to make this up to you, certainly let us know.

Regards,

Thomas Asaro
Vice President

They’ve made a security announcement on their website which also confirms the error, and alerts their customers of the risk. Linode says they are performing an in-depth audit on their Linode Manager software to confirm the risk.

Bitcoinica

Zhoutong of Bitcoinica, confirmed he has lost 43,554 BTC (~200,000 USD) from this recent Linode theft.

We lost 43,554 BTC from this incident and we will reimburse our customers for the full amount.

- Customer funds will not be affected.

Bitcoinica is committed to absorbing any loss. The thief stole from us, not you.

- Customer data is safe.

The compromised server was entirely dedicated to holding our bitcoin “hot wallet” only. Thankfully, this function is the –only- one ever hosted at Linode. No customer data has ever been hosted at Linode. Also, there is no privileged access from the affected server. This means that no passwords, account activity, or any other customer data has been exposed by this incident.

Forgive me for including this social media spam but it helps our dinky small site get exposure D:

•