The Telus LG Keybo 2: Hacker Hater (aka VX9200, CX9200, enV3)


I just bought my first cellphone in years. I know that sounds strange coming from an IT guy but my logic is sound: if you don't have one your boss can't call it. One of my colleagues is a big VoIP genius and he pointed out that by using a dial-out gateway and Telus' one-number-unlimited feature it's possible to have the equivalent of unlimited calling for $7 per month on a prepaid phone - plus the one-time cost of provisioning a phone number that forwards incoming calls back through the gateway (about $25). I've been horny for the newer android-based phones that are in circulation in the states but most of the ones I want don't operate on the Canadian bands yet and/or it will take a long time before they are rolled out up here, thus I decided for unlimited calling at $7 a month I may as well settle for a cheap phone in the mean time. I decided to tack on $10/month unlimited web browsing because it makes the e-mail and instant messaging packages moot.

The Keybo 2 is the closest thing to a smart phone in Telus' prepaid lineup, and at the time of writing the most expensive - clocking in at a modest $99. It sports a 160x96px external screen and flips open to reveal a large QWERTY keyboard and 320x240px internal screen. There are stereo speakers mounted on either side of the internal screen and to be perfectly frank they are better sounding than my laptop's. The Keybo 2 has a 3.2MP flash camera that takes decent pictures but crappy closeups.

In the United States (and possibly elsewhere) the Keybo 2 is marketed as the enV3. The Canadian model number is CX9200 and the US model is VX9200; as far as I can tell the difference is cosmetic. Verizon seems to be the main carrier for enVs in the states and Koodo is popular in Canada as well. Interestingly the Koodo and Verizon firmwares of the phone do not include Java support. It would seem that the popular thing to do with your Keybo is flash it to Telus' firmware if you're on a different provider. Unfortunately Telus' firmware is so locked down that one wonders just how bad Verizon's could be.

I've had a hell of a time over the past few days trying to find information on hacking the Keybo 2, most of the results I have found thus far only apply to the original Keybo (aka enV2 or VX9100), for example it no longer seems possible to simply overwrite application slots with other java apps to install them on the phone. Telus' proxy prevents users from downloading apps from the web that don't come from their store. I have tried altering the proxy settings to use a personal proxy on ports 8118, 80 and 110 but the browser fails to connect. I even tried popping the opera mini .jad and .jar files onto an SSL site and altering the .jad to pull from the new URL to no avail - the download begins, posts, then returns this error:

Issue has been reported.
Please try again later.

950 Server Error (-1289)

then the browser bounces back to the Telus apps store. I tried renaming the .jar to .jax (and updating the .jad accordingly) also to no avail.  I am beginning to suspect that the firmware has been modified to enforce some sort of DRM for applications. Custom ringtones are a pain in the ass as they definitely require DRM, fortunately they can be dropped into the phone's filesystem at /brew/shared/ringtone/ with BitPim (1.0.7+ supports the CX9200) and given DRM with the Sony-Ericsson DRM Packager.

I couldn't find the right SPC code for my particular phone anywhere (unlike the old Keybo/enV2 it does not have nvm_XXXX files) so I had to grab it with CDMA Workshop (it's 105495 by the way). With the correct SPC you can access the programming menus for your phone by dialing:


Note that you can get into ##DEBUG on any LG phone with the unlock code 183729.

Overall, this phone pisses me off because it could do so much more but Telus makes it extremely difficult to modify. While there is some community support for the Keybo/enV2 those of us with the new Keybo 2s and enV3s are practically on our own at present time. The fact that the Telus firmware is considered "the good firmware" is extremely discouraging, flashing your old Verizon enV2 to Telus' old Keybo firmware may let you load on some java apps but how to do this successfully on the newer Keybo 2's firmware is as yet a mystery.

Mass Virtual Hosting Part Eight: MySQL-Proxy for Easy Network Topology Changes and Localhost vs. Sockets


Once your hosting clients are all settled in you may find one day that you need to change their MySQL server address or other configuration parameters. Naturally it's not going to look good on you or be a very good use of your time to contact every webmaster and have them update their settings. Worse, juggling two active database servers would be a nightmare. Fortunately Oracle came up with mysql-proxy, a lightweight app that sits between your clients and MySQL server(s) which acts as a drop-in replacement for mysqld. Users connect to the proxy like they would the actual server and it transports data to and fro. You can do all sorts of neat things to the data while it's in motion with lua scripts but that goes beyond the focus of this article.

By default mysql-proxy listens on port tcp/4040 and mysqld listens on tcp/3306. In my experience most users who come from other ISPs are already wired to use localhost as their default SQL host and I don't want to make them have to remember the port number, which is typically defaulted in most webapps. If you're running mysqld on the same host you're serving web from you'll need to change the port it listens on in /etc/mysql/my.cnf.

MySQL has a contentious age-old convention of changing "localhost" to "use the local socket" rather than resolving localhost to and connecting via TCP. That's because at the time the decision was made local sockets were far more efficient than using TCP, now it's not so much an issue. We need to configure mysql-proxy to listen to a socket too or our users will have to use the numeric address, lest they encounter this error:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

This can be specified - not all too intuitively - by replacing the proxy-address value in /etc/mysql/mysql-proxy.cnf with the default path to the local socket, thus:

proxy-address = /var/run/mysqld/mysqld.sock

Configure the proxy-backend-address variable to reflect the actual server's location and port number. Restart mysql-proxy and you now have a working, default-looking configuration that can be redirected anywhere. Thanks to the lua capability of the proxy you can even implement fast and easy load balancing and failover, but that will be the topic of another article!

Thwarting the isc.org DNS DDoS


Yesterday I posted an article regarding the importance of ACLs in BIND because a client's DNS server was under extreme load and, at the time, I believed it was because a network of web form spammers was outsourcing their lookups to the target. Upon further investigation it turned out that this was not the case - the hosts were indeed a part of some sort of spamming ring/botnet but they were actually performing endless repeated ANY lookups on isc.org - the producers of BIND, among other things.

The whole attack doesn't make any sense; if they wanted to involve this server in the attack against isc they would be wasting reams of bandwidth that could otherwise have been applied directly, unless they knew both recursive lookups were allowed and caching was disabled. Even with no recursive ACL at the time the only server suffering was my client's thanks to caching and there is no conceivable reason, in my mind, that a spam network would be targeting this specific dns server among many in its address space - from one source address at a time. Fortunately adding a recursive ACL reduced the bandwidth impact but it did not stop the scans, and bandwidth is a pricey commodity after all.

I whipped out wireshark and took a sample capture, these are the contents of a typical request packet:

0000  00 16 3e bb 00 02 00 16  3e cc 00 02 08 00 45 00   ..>..... >.....E.
0010  00 40 8b 0d 00 00 e9 11  d7 93 d1 0b f2 7b 00 00   .@...... .....{..
0020  00 00 63 01 00 35 00 2c  00 00 2a 39 01 00 00 01   ..c..5., ..*9....
0030  00 00 00 00 00 01 03 69  73 63 03 6f 72 67 00 00   .......i sc.org..
0040  ff 00 01 00 00 29 10 00  00 00 80 00 00 00         .....).. ......

Next it was just a matter of crafting a suitable snort rule for the upstream gateway:

alert udp $EXTERNAL_NET any -> $HOME_NET 53 (msg:"DNS isc.org DDoS"; content:"|03 69 73 63 03 6f 72 67 00|"; reference:foxpa.ws,369; classtype:attempted-dos; sid:4000002; rev:1; fwsam: src, 1 day;)

Now I'm kicking back, watching the spammer network expose itself as its constituent hosts smack against the firewall like bugs against a windshield.

If only I had some beer.

See how this story progressed:

Portage Errors: app-arch/lzma-utils is blocking app-arch/xz-utils


Gentoo recently made the switch from lzma-utils to xz-utils, to quote the project page at http://tukaani.org/lzma/:

LZMA Utils are legacy data compression software with high compression ratio. LZMA Utils are no longer developed, although critical bugs may be fixed as long as fixing them doesn't require huge changes to the code.

Users of LZMA Utils should move to XZ Utils. XZ Utils support the legacy .lzma format used by LZMA Utils, and can also emulate the command line tools of LZMA Utils. This should make transition from LZMA Utils to XZ Utils relatively easy.

As you are probably aware, Portage has largely used lzma for the past year - before which bzip2 was the compression standard of choice. If alarm bells went off in your head when you saw this message:

[blocks B     ] app-arch/lzma-utils ("app-arch/lzma-utils" is blocking app-arch/xz-utils-4.999.9_beta)

You can rest easy uninstalling lzma, the xz-utils package comes in a good ol' fashioned tarball and it works as a drop-in replacement for it so it's safe to unmerge lzma-utils completely then - before emerging anything else - emerge xz-utils:

# emerge --unmerge lzma-utils; emerge xz-utils

 selected: 4.32.7
 protected: none   
 omitted: none   

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging app-arch/lzma-utils-4.32.7...

 * GNU info directory index is up-to-date.
Calculating dependencies... done!

# emerge xz-utils
Calculating dependencies... done!

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) app-arch/xz-utils-4.999.9_beta
>>> Downloading 'http://gentoo.osuosl.org/distfiles/xz-4.999.9beta.tar.gz'

I'll take this opportunity to point out that in this day and age the word beta has become such a gimmick that Gentoo is including so-called beta software as part of its core system. Back in my day beta meant beta and you had to walk 17 miles uphill both ways to get to the nearest phone jack. Mind you we didn't call them modems in those days, we called em clinkerdinkers....

Way to go, Internet.

Making the Case for Access Controlled Recursive Lookups with BIND


To see how I actually solved this particular problem please see Thwarting the isc.org DDoS

It has long been held as best practice that one should use separate servers for authoritative (server) and recursive (client) domain name services. In some cases, this is not practical - for example I inherited one particular network at an ISP I do consulting for that has had its combined authoritative and recursive DNS servers on the same IP addresses for about a decade. Splitting up the roles of the servers would require either hundreds of clients with statically configured DNS to change their settings (imagine the load on tech support) or dozens of domain-holders to update their registrar glue, assuming their registrar even provides it (another load on tech support).

On this network I had - contrary to the gospel I know only too well - allowed recursive lookups from any source on the Internet for several years. I knew it was ideologically incorrect but have found it immeasurably handy to have a set of memorized nameserver IPs that you know will just work any time, anywhere in the world when you are tech supporting or trying to resolve an issue. No one seemed to notice for the years before and after I took on this client, and I couldn't see the harm if a few folks started using them. It didn't - and still doesn't - make sense to me that a spammer would outsource their DNS operations to a vastly remote location (see updates at bottom, they are spammers but they're actually trying to DDoS), at the cost of not only speed and latency but expensive bandwidth at all.

But today they did.

At around 12:40 I was notified of the situation and began my investigation. Handy-dandy iftop told me it was coming from one IP and they were making DNS requests - good news! The server wasn't hacked - it was just being DoSed as a side-effect of over-querying. In fact, named was performing exceptionally well for a ~5-mbit/s load on a tiny virtual machine used to 150kbit/s, it was only the upstream bandwidth throttling that was keeping the flow in check.

I ran a full port scan on the attacker and much to my surprise it didn't respond to pings and all ports were filtered. I was growing suspicious that this wasn't a typical zombie. I blocked the attacker at the gateway and for a moment the network settled down. Then out of nowhere another attacker from a completely different IP showed up - I gave them a full scan and they too were filtered and unresponsive to ICMP echo requests. The new attacker came from but forward lookups didn't work. I didn't have to go to their website to realize this was a VPS provider, clearly I was dealing with someone who owned their own kit - an organized spammer. My suspicions confirmed, I knew I would have to give up my little "public" DNS service and implement an Access Control List for recursive lookups.

By the time the third and most aggressive (getting pissed off about the IPs I was blocking?) wave of queries rolled around I had collected a list of all the networks and public addresses which require recursive lookups. Regardless of whether you have your authoritative services separated from your recursive unless your recursive server is on a private network it is best practice (and even then I would recommend it) to implement an ACL.

Open your named.conf and before the global options section, define an ACL as shown below. Notice that address ranges can be specified with CIDR notation:

acl "trusted" {

Now we can call the trusted ACL with the allow-recursive directive in the global options section. You can define any number of ACLs with different names, we're just going with trusted as an example. Make sure the ACL includes every network and IP address that should have permission to perform recursive lookups - ensure your private subnets, public networks and any stray uplinks or remote locations are included. Those hosts not in the ACL and not performing lookups on domains for which your server is authoritative or already has cached will fail.

options {
 recursion yes;
 allow-recursion { trusted; };
 allow-query-cache { trusted; };

Run rndc reload or restart BIND for the changes to take effect.

As you can tell from the bumps in my graph after the giant spike this guy (or these guys) is taking his time trying to figure out what I did, but the good news is he's using less bandwidth and a whole whack of spam ain't goin nowhere. Eventually they will realize that this server isn't doing them any good and they'll move on to greener pastures.

Finally, I got one of the attackers post-ACL to submit to a scan:

bzp ~ # nmap -P0

Starting Nmap 5.00 ( http://nmap.org ) at 2010-07-20 18:15 EDT
Interesting ports on
Not shown: 990 closed ports
21/tcp   open     ftp
22/tcp   open     ssh
23/tcp   open     telnet
25/tcp   filtered smtp
80/tcp   open     http
139/tcp  filtered netbios-ssn
445/tcp  filtered microsoft-ds
646/tcp  filtered ldp
2000/tcp open     callbook
8291/tcp open     unknown

Nmap done: 1 IP address (1 host up) scanned in 390.66 seconds

Well, that looks like any number of shared hosting servers, I wonder what's on port 80:

An embedded linux routing solution. And their webconfig is wide open to cracking.

MikroTik RouterOS is the operating system of MikroTik RouterBOARD hardware.

It can also be installed on a PC and will turn it into a router with all the necessary features - routing, firewall, bandwidth management, wireless access point, backhaul link, hotspot gateway, VPN server and more.

RouterOS is a stand-alone operating system based on the Linux v2.6 kernel, and our goal here at MikroTik is to provide all these features with a quick and simple installation and an easy to use interface.

You can try RouterOS today, go to www.mikrotik.com and download the installation CD image. The free trial provides all of the features with no limitations.

If I had to guess I'd say this guy went the server route, or maybe he virtualizes his gateways like me. This kind of explains why my port scans are being cut off, a sufficiently advanced firewall like this probably has some sort of Intrusion Prevention mechanism and since I caught them with their web config (and telnet config for that matter) exposed and flapping in the breeze I'd say this is one of the installations my friend the spammer has not completed configuring, hence this one successful scan.

For the interest of Googlers trying to figure out why so and so is blasting the shit out of their name server(s) one IP at a time, here's a list of IPs and PTRs (where available) I've caught so far:   sju13-2-82-225-94-140.fbx.proxad.net   something from schlund.de  RIPE says the block belongs somewhere in Dubai    hosted by dragonara.net, listed as comment spammer by project honeypot  hosted by ev1servers.net MOLECHART.com  hosted by schlund.de  hosted by schlund.de     Rogers (Wave ???)   Italian-show-passion-quality-hostserver.com (lol)     Rogers (Wave ???)  server12.elnastalk.com

Update I found spamming on a strange Iraqi forum that reveals posters' IP addresses to the general public. Strange indeed, but now I know I'm dealing with web form spammers, not e-mail spammers. Here's a sample of the spam found at http://www.iraqal7ob.com/vb/t710.html

The image at the top was hotlinked from a free hosting service, the exact URL was http://hosting.bearzddl.com/uploads/9b0b60d466.jpg. Bearzddl is itself a shady file site and at the time of writing http://hosting.bearzddl.com/uploads/ provides a directory listing of all sorts of spammy looking images that have been uploaded through their free file hosting feature:

Update This morning we were attacked once again, I took the following sample capture:

 11.695249 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.695449 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS ams.sns-pb.isc.org NS ns.isc.afilias-nst.info NS ord.sns-pb.isc.org NS sfba.sns-pb.isc.org
 11.705516 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.705891 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS ns.isc.afilias-nst.info NS ams.sns-pb.isc.org NS sfba.sns-pb.isc.org NS ord.sns-pb.isc.org
 11.723779 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.723787 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.723983 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.723989 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.724154 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS sfba.sns-pb.isc.org NS ams.sns-pb.isc.org NS ns.isc.afilias-nst.info NS ord.sns-pb.isc.org
 11.724333 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS ord.sns-pb.isc.org NS sfba.sns-pb.isc.org NS ns.isc.afilias-nst.info NS ams.sns-pb.isc.org
 11.724495 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS ams.sns-pb.isc.org NS ns.isc.afilias-nst.info NS sfba.sns-pb.isc.org NS ord.sns-pb.isc.org
 11.724648 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS ams.sns-pb.isc.org NS ord.sns-pb.isc.org NS sfba.sns-pb.isc.org NS ns.isc.afilias-nst.info
 11.725244 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.725256 -> xxx.xxx.xxx.xxx DNS Standard query ANY isc.org
 11.725488 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS sfba.sns-pb.isc.org NS ord.sns-pb.isc.org NS ns.isc.afilias-nst.info NS ams.sns-pb.isc.org
 11.725645 xxx.xxx.xxx.xxx -> DNS Standard query response RRSIG RRSIG DS DS NS ns.isc.afilias-nst.info NS ord.sns-pb.isc.org NS ams.sns-pb.isc.org NS sfba.sns-pb.isc.org

See how this story progressed: