pitter patter on the keyboard

If you’re tuning in today you have the opportunity to watch a zero-day attack and response in action.

In the last 12 hours I’ve received a message from two different VPS providers explaining that they’ve taken down their SolusVM web-based VM management software due to a severe vulnerability:

Hello guys,

we learned about a nasty security leak in solusVM today and we decided to switch the SolusVM admin-panel off.
We hope that Solus as the company will soon release a patch that will fix most recent leaks as this is not the first one today.

Please check our status-page at status.edis.at for updates.

A couple of providers have already been hacked and their client’s data and passwords have been leaked or entire hosting-platforms have been wiped.That’s why we decided to shutdown the panel as a preventive measure. If you need reinstalls or reboots, just submit a ticket – we will try to help as fast as we can.We’d like to point out to the fact that this is not a technical flaw on our side.

Thanks a lot for your understanding!

[Waveride]

It has come to our attention that SolusVm (the VPS control panel) may have some exploitable vulnerabilities which we are not aware of.
As a precaution we turned off our SolusVM panel untill a fix is released.
This is not the previous central backup vulnerability which we were patched against, but alledgedly newly discovered vulnerabilities that are about to be disclosed soon.
What does this mean for you:
1. No data is lost.
2. The VPSes themselves are up and running (unless unrelated incidents happen)
3. You can connect using SSH or your control panel, however, the console is part of solus and wont be available, so be careful not to get locked out of SSH for the next day or so.
4. Provisioning of new VPSes, while techincally possible, if done outside solus might result in various disfunctionalities, therefore you can opt for a refund or wait until we think it is safe enough to re-enable solus.
5. Billing panels are still available, but we are limited in what we can do. We use solus too for many tasks, but we will try our best to help you, so log a ticket if you need help.
6. There has been no database leak, no other compromise of any data and solus itself does not store those anyway except some basic things like name and IPs.
7. The billing panels take their data from Solus (traffic consumed, VM status) and is doing any action such as reboot, shutdown through solus too, therefore these functions will not be available and your VPS will appear as offline, when in fact it is not, use SSH for any urgent tasks you may have.
8. While your data is intact and VMs have not been touched, please remember we offer free FTP space and do a backup for the data you think is important enough to be saved. We are an unmanaged host and may or may not have back-ups in case of a disaster like a major hack, an earthquake or fire, for example. Biz plans benefit from offsite backups too.
9. We are using third party software (it is impossible not to, even linux kernel is a third party software we have no control of) and we are dependent on the respective vendors to keep their software secure, therefore, in spite of our best oefforts (and this is valid for everyone) we cannot be immune to hacks. Nobody is, so, one more reason to keep recent backups.

We are sorry for these problems, unfortunately, since we cant do anything to fix them, we choose to turn off the vulnerable software until a fix is released.
We will try to keep you updated here:

http://board.prometeus.net/viewforum.php?f=15

This is, as you can see, valid for other problems, as well.

Thank you for our understanding and support !

[Prometeus]

domVPS has also apparently shut down their SolusVM portal but has not yet issued a statement by e-mail.

You can watch Soluslabs’ response on their blog at http://blog.soluslabs.com/. So far a fix has been released for one vulnerability but at least one other has popped up:

We are aware of the current rumours regarding a further security issue with SolusVM as well as some snippets of code. We have been working hard to audit all of the SolusVM code to find any further potential security issues that may pose a threat.

At this moment we have been unable to locate any problems however we are continuing to search for any possible attack vectors. We have received a few blocks of code from some customers that are currently being reviewed. Should any issues be identified a patch will be released immediately along with further announcement.

In the meantime, we do not believe there to by any immediate threat to customers.

Further updates will be provided within the coming hours.

Thank you for your patience and continued support.

I sympathize. These poor buggers are going to endure a lot of ball-breaking and code sifting.

Unfortunately, Soluslabs doesn’t seem to be planning on releasing details of the exploit for another few days.

Fortunately, we don’t have to rely on them. From LEB:

Today has been an unfortunate day for many hosts and indeed a shocking eye-opener for anyone using SolusVM to offer VPS’ to the public. Earlier on today the website localhost.re reported on a shocking SolusVM exploit that effects every SolusVM version – the now defunct/unused file centralbackup.php contained multiple blunders including SQL Injection, direct exec()ution of any command, and access to the SolusVM server-side binary which can execute any command. Unfortunately for hosts this was a surprise to say the least, and one of the first to be targetted seems to be RamNode.

…

An announcement from RamNode was soon released and it was confirmed that Robert Clarke, founder of ServerCrate, was behind the initial breach of security at RamNode via the exploit. “As you are all aware, this has been a nightmare for [us]. Robert Clarke ran the SolusVM exploit on our control panel early this morning. Someone, him or else, then logged into several nodes and wiped the data.”

Members of LowEndTalk did post findings that correlate with the above statement that Robert Clarke was behind the attack/intrusion. Evidence such as IP-matches & even confirmation that the IP was indeed Roberts’ home network (via the welcome page for a HP media server which clearly stated “Robert’s Pictures” with the hostname ‘clarkeone.homeserver.com’) – not especially good news considering Robert’s previously dubious history and not so great reputation in the industry. While Robert has admitted to the initial “testing” of the exploit he still protests his innocence and vehemently denies doing any of the damange. *Update* Robert has admitted to perpetrating it to several different people. It also appears he targeted BuyVM.

Now that’s just a good story.

At present, LowEndTalk (the forum cited above) is down. Whether that means they are being hosted on a VPS by an affected provider or it has earned the ire of its many DoS-keen members for burning Clarke or not is yet unknown.

Before anyone gets out the pitch forks let’s heed this LEB commenter’s wise words:

Now that everyone has all but burnt Robert at the stake, it is worth considering that this exploit appeared on the net to be then immediately broadcast in many locations, not least the home of the child VPS provider and DDoS hive that is Lowendtalk.

If Robert did cause the issues at Ramnode it is likely, or actually definite that he was simply one of a much larger group of people cutting their way through providers trying to get a “hit”, he was lucky as it were and found Ramnode, others are in the same position. I know of at least 4 with varying degrees of repair work required.

Whilst I am not condoning what he did, if he did it, it is easy to focus in an target him, yet from what Nick has said he is can only be sure Robert accessed something rather than did anything. I am sure all you providers can check your logs and see countless others “all of a sudden” waking up and becoming active in the apparent name of “just testing to check everything is ok”.

Innocent until proven guilty in a court of law, I always say!

The much-anticipated The Pirate Bay documentary covers the events surrounding the trial and conviction of TPB founders.

I’ve had a lot more crummy experiences with web hosts, collocation and dedicated server providers than I have glowy, happy ones – as seems to be the case for most everyone in this industry subject to a budget. For your benefit (and my therapy) here are some of the crappier ones I have had the pleasure of leaving, and why:

1&1
One and one is probably best described as the Wal-Mart of Internet Service Providers. If you don’t expect a lot and you’re happy with cheap crap it could be right for you. Tickets generally go unresolved for one or two full days and their shared hosting servers have hidden limitations even where unlimited resources are advertised. In particular if your site starts receiving “too much” traffic they will start tossing 500 errors and their techs may take days and days to apologetically tell you they have no idea why.

Netfirms
One and one of the North – Nerfirms’ service is based in Toronto, Ontario. They are so oversold if you do not notice the hit to your page load time the minute you move your site there it only speaks to what kind of host you were coming from. Netfirms shared hosting also suffers from hidden limitations (CPU and RAM utilization), which are actually much lower on the higher-priced business package than 1&1′s low-end package.

Hostway
Hostway is yet another crappy discount hosting provider that was at one time a decent place to grab certain domain names on the cheap despite their chronically buggy administration interface. Hostway seems to outsource most of their technical support (as with 1&1), switching from some joint in India to some joint in Russia or another ghastly eastern European country some time during my tenure. The price increase on domains alone would have been enough to drive me away if it weren’t for the fact that I have had multiple tickets with them open for literally weeks at a time. I’ve learned that you can often judge a service provider by the quality of their VoIP system – if the hold music sounds fuzzy or cuts out a lot run. run and never look back.

Hosting Check
NoMonthlyFees.Com was doing fine until it was bought out by Hosting Check, their service is now so oversold the lowest traffic sites are now having serious availability issues, nevermind speed issues. Hosting Check charges $20/yr for dot coms – at that price I expect them to be grown organically and harvested on a fair-trade farm.

PaylessDomains.ca
I’ve never used Payless for anything but .ca domain name registrations so they wouldn’t have made it on the list if their (outgoing) domain transfer function hadn’t been suspiciously broken ever since I started using them over three years ago. Somehow, continuing to renew domains at a registrar whose prices I had matched a long time ago for the simple fact that I was too lazy to contact their technical support every time the front end told me an error occurred trying to unlock the domain made me feel like I was being taken advantage of a little. My fears were confirmed when I e-mailed in and was asked to provide a reason for wanting to transfer my domain.

I told them it was because they are crooked. My authorization code came in a template e-mail, further supporting the theory that the web-based function to unlock and obtain the authorization code was never intended to actually work. Dirty dirty dirty. Mass migration ensued.

3z Canada
3z is a discount collocation provider operating out of 151 Front St. Toronto, Ontario. Their business almost entirely consists of Chinese clients who need servers with a Canadian presence. I don’t have a lot of nasty things to say about them because I had to virtually fight with them to take my money for a month before I decided to stop wasting my time and get colo elsewhere. Interestingly, in the same breath the Moxie Communications/Secure Access Colo owner told me 151 residents are a back-stabbing rumour-driven lot, he told me he had once rented rack space to them and found them “taking liberties” with some of his and other tenants’ free space that had not been properly negotiated for. Of course, how much truth there is in that rumour I can’t say because the Moxie owner is himself a greaseball of epic proportions (see below).

Carat Networks
Carat Networks, formerly “Clearance Rack” is based in Hamilton, Ontario. A discount colo and dedicated provider once operating out of a 10×10′, chronically overheating telecommunications “bunker” out in the middle of some guy’s farm. They now occupy Hamilton’s Mountain Cable facility where the customer service and competence are still as low as the prices.

Secure Access Colo
Another member of the had-to-change-its-name club, Secure Access Colo was formerly known as Moxie Communications. Staffed by imbeciles and at the top of my personal shit list Secure Access Colo actually extorted yours truly out of almost 2 grand. Before I get into that let me take the piss out of them a little so you don’t think this is entirely personal:

• Redundant physical loop: no
• Generator: no
• VoIP (support) not affected by DC going down: no
• Secure access: Lock on the rack, RFID at the front door – and they will swear up and down the $0.15 RFID card is actually worth $100; no one else plays that guff as it’s generally recognised that you pay a $100 deposit for access cards because that’s just the industry standard. You bend over or you don’t.
• Rooftop security sensors: True, but not unique and shouldn’t be the crown jewel of a physical security marketing strategy.

I’m not sure what makes Secure Access Colo think it deserves to put Secure Access right in its name. Mediocre at best, except for their one room at 151 Front – where physical security isn’t up to them anyway. Despite the rooftop sensors there’s both a regular door and loading door in the back and if you really want to get in the easiest way is probably through the huge conference room windows or the RFID controlled front doors (one after the other), also almost entirely glass.

Anyway, after sending out a client-base-wide notification that we would have to switch IPs after a certain date their upstream provider (at the time Carrier Connex) cut their pool “early” leaving us totally disconnected before we had even been assigned our new IPs. As it turns out, the date Moxie had given us had only been arranged as an extension on the actual cutoff date with Carrier Connex – unofficially, verbally and with a low level representative. Secure Access/Moxie was not actually paying for IP service up to that date and they were operating on little more than assurances. They offered emergency remote KVM so their clients could log in and assign new IP addresses but I was so incredibly pissed off by this I demanded the termination of my service.

Apparently this was a big mistake, they cited a contract which I had no record of nor recollection of signing and which they consistently failed to produce. They demanded payment for the remaining 6 months of a one year term or they would not release my servers because my decision to terminate was “no fault of theirs” despite the fact that they were clearly negligent in maintaining or extending their contract with Carrier Connex until at least the date set forward in their notification. They held my servers hostage and disconnected for two weeks while I conferred with my lawyer who eventually made it clear that it would be cheaper to pay the bastards than pay his bill – the only way to get your servers back online in a situation like this is to file an injunction first and sue later which is just not in the budget of a young entrepreneur.

Nonetheless, if I didn’t have a greater obligation to my clients I would have been thrilled to have my day in court – these scum truly deserve everything horrible that happens to them.

You can tell we are dealing with some brilliant people by how well their front page photo turned out. You would have to be an asshat to give them your money (in my defence their old site was slightly more convincing).

iWeb
iWeb is based in Montreal, Canada and is rated one of Canada’s top 100 employers by Ernst & Young – I believe it! I was happy to swallow 24 hour+ turnarounds for support tickets in exchange for their very reasonable prices when I was only dealing with them for my personal interests, but when it came time to move a business client in I felt five days with no communication after proving there was a bad stick of RAM in my freshly provisioned server was more than unacceptable. Expect two to three days minimum to get your new servers provisioned. iWeb is notorious for their slow (and patently lazy) support and this is why I am sure they are one of the best companies in the nation to work for. I envision google-esque “crash zones,” fully stocked beer fridges and plush couches where the technical support staff spend their whole days dispensing witty anecdotes about irate anglophone clients amongst one another.

I was unfortunate enough to be a customer of iWeb’s during a particularly long (two weeks?) DDoS against an isolated portion (commercial DNS services) of their network. In their defence a 30gbit/s attack is a significant event, but knowing how networks like theirs are constructed I am disappointed in their apparent inability to protect clients on unrelated segments from collateral damage.

Consider iWeb if you want a sweet server and loads of bandwidth for great prices and can live without reasonably paced technical support (i.e. a mirror, game server, redundant DNS).

Forget about iWeb if you owe any level of care to your clients. You will be made to look a fool and possibly lose your contracts as a result of their slow support – worse you will know the whole time it is happening that things are entirely out of your hands.

On an interesting note, it turns out iWeb (unintentionally) hosts a number of Syrian government sites by way of a third party reselling their services. Their official response:

Several media outlets have recently published articles based on a report from the Citizen Lab organization, which can be found here: http://citizenlab.org/wp-content/uploads/2011/11/canadian_connection.pdf

The report, entitled “The Canadian Connection: An investigation of Syrian government and Hezbullah web hosting in Canada” highlights a very important issue, providing a summary of what has become a complex problem for the web hosting industry.

The Canadian government has enacted regulations that restrict Canadian firms such as iWeb from doing business with certain foreign individuals and entities. iWeb is committed to strict compliance with these laws and continues to monitor its compliance.

In 2008, iWeb inadvertently hosted two websites affiliated with Hezbollah. When iWeb learned of the websites’ affiliation, it cancelled its web hosting services.

Canada has enacted targeted sanctions against certain Syrian government entities and individuals. Canada has not enacted a broad embargo against doing business with Syria.

The Citizen lab report identified a number of Syrian government entities for which the internet address resolves directly or indirectly to iWeb. With one exception, none of the listed entities are subject to Canadian sanctions. The exception is Addunia T.V. which was listed as a sanctioned entity on October 3, 2011. iWeb has not provided any services directly to Addunia T.V. and is investigating whether its facilities have been used by one of its customers for the benefit of Addunia T.V. without its knowledge. iWeb will be taking all appropriate steps in light of its findings.

If you feel you must go with iWeb I at least strongly encourage you to pay entirely by credit card. This way a chargeback can be made against them in case their billing department decides to dick you around.

Please learn from my mistakes and avoid these companies like the plague. In the wonderful world of hosting, dedicated servers and collocation you don’t always get what you pay for.

Just most of the time.