CodeFlow

I’ve been experiencing some performance problems lately on pogo.kaffe.org (which hosts kaffe.org plus my personal email). It’s a bit hard to diagnose, as the version of Debian running in the VM is so old that some tools don’t work. Also, there are no longer security updates available — so I had better upgrade.

Expect to see quite a lot of things break. I’ll try to patch things up over the next few days. If my email isn’t working, you can use my backup email address at jim.pick@gmail.com.

Update: I updated Debian, but just about everything is broken now - web site, ftp site, mailing list, mail server, spam filtering, bugzilla, etc, etc. So it will take me a few days to get it all reconfigured.

Update #2: Email is now mostly working, but the spam filtering isn’t in yet. Then I’ll do the mailing lists, and get the website back up. Other things like Bugzilla will take longer.

Update #3: I think spamassassin is working again. If that’s OK, I’ll try to get the mailing list reconfigured, and then I’ll put the website back up.

Update #4: The mailing list and website are back up. I probably need to fix up some CVS related things next, as well as bugzilla.

I rebooted the server - it had crashed sometime in the past day or so. I’m on holidays right now, visiting family, so I haven’t been checking in very often.

Right now, I’m only running with one DNS server operational, and one lame one, which is sort of bad. In the new year, I think one of my resolutions will be to rebuild my external network.

One idea I had, which works nicely with my CodeFlow ideas, is to rent a bunch of really cheap VPS servers (in the ~$5/month price range), and build a bit of a distributed networking testbed.

I still need to repair my server which failed. I’ve had some difficulty finding a place that sells the parts and can ship to Canada. Since I’m not planning on doing any freelance work in the near future, it doesn’t really make financial sense to put it back online. I think I’ll set it up at home, and use it for development tasks and backups.

I fetched the server from the colo in Seattle on Thursday (en route to my vacation in California). Clearly there was something wrong with the RAID, but I was also discovered that I was having memory problems.

Now that I’ve got it back in Canada, I’m now able to take a look to see what’s wrong with it. Even before I fired it up, I discovered little lots of little plastic bits. At first, I thought I broke something when I packed it into my car. I looked around the case to try to figure out where they came from. They were fan blades…

When I fire up the server, it sounds just awful. It looks like two of the four fans are not spinning at all. That means that one of the dual-core Opteron CPUs, and 4GB of memory were not getting cooled at all. I don’t imagine that there was very good airflow past the drives either. I hope there hasn’t been any long-term damage to any of the components.

Everything is coated with an oily film as well. I don’t think the colo was filtering the air very well. Yuck.

I think the next step is to order replacement fans before I do more testing on it.

I appear to be having some problems with the RAID on my server in Seattle. It seems to be stranger than just a single disk failing. Maybe it’s the controller?

So, for the moment, kaffe.org and builder.classpath.org are down.

I won’t have time to look at it until after work today (in about 10 hours). I’m not sure if I can salvage the data. If not, I have backups from the 21st, and I can restore from those on my server in San Jose.

My email server is also affected. You can reach me at jim.pick@gmail.com in the meantime.

Update: 8pm Monday - I updated the RAID firmware, and started a “rebuild” on the controller about 8 hours ago. It’s about 32% complete, so hopefully it will finish sometime tomorrow. If I’m lucky, I’ll be able to save some data. If not, I’ll have to try to restore from backups.

Update: 7pm Tuesday - Bad news. I wasn’t able to successfully rebuild the RAID. So I’m going to have to restore from backups on my server in California. I’m doing that now. It’s going to take a while. Hopefully, things will be back by tomorrow morning. I also have to prune things back a bit, as 2/3 of my capacity was on the failed RAID.

Update: 11pm Wednesday - I’m almost finished restoring the kaffe server disk image. It’s taking a really long time. I’ll try to install it really early Thursday morning.

Update: 7am Thursday - They’re back! I had to revert pogo.kaffe.org to the backups from the 21st. For builder.classpath.org, I used ddrescue - so there might be a bit of corruption to clean up.

I spent today digging around the Internet trying to find a distributed wiki implementation.

Mostly, I found a lot of dead end. But in the end, I finally found one!

Wooki

Another alternative may have been to try to bolt ikiwiki onto git, monotone, or mercurial.

I’d like to see how much work it would be to do an implementation using Javascript, using Google Gears as the storage mechanism.

I think I’ll try to install a wooki on my server next.

I tried out the Android Emulator. Pretty decent. It would definitely be an improvement over my cell phone. Hopefully, Google is true to it’s word, and opens up all the source code.

It will be interesting to see if the carriers are actually going to be willing to open up their networks to the point where they would let their customers have truly open, customizable cell phones. In Canada at least, the data rates are a complete rip-off.

Disclaimer: Android has been on my radar screen for a while, mostly because my former colleagues at Digeo all used to work with Andy Rubin when he was at WebTV. Also, I did interview at Danger (and Andy) at the same time as I applied at Digeo. I wasn’t enough of a VM hotshot though, I guess…

Paying with the demos, and looking at the architecture, I got a serious case of deja vu. I think that’s because it’s nearly an identical architecture to what I spent 2 years working on at Transvirtual with PocketLinux/XOE back in 2000/2001. It’s an open source stack with Linux on the bottom, a custom JVM, and a rich Java-based set of custom application libraries, built around a web browser and a packaging system.

I think PocketLinux/XOE was an awesome technology - the company just failed due to infighting and the dot-com collapse. I’m sure Android will succeed, especially with Google’s backing.

They won’t be the only cell phone stack out there, of course. Ultimately, I imagine most apps are going to be developed around web standards, as they evolve, instead of any one particular cell phone API.

I had to change the IP addresses on my other server, the one hosted in San Jose. Lots of things seem to be broken - mostly DNS related. It may take me a little while to fully recover.

One of the things that is broken is the domain name record for icedtea.classpath.org. I sent an email to the GNU sysadmin to update it. Hopefully that happens quickly. Until then, the server can be accessed at icedtea.classpath.berkeleysignal.com.

Update: All my reverse DNS entries in Seattle were delegated to my DNS server in San Jose. Unfortunately, when I switched IPs, those went dead. Even worse, just updating the DNS records wasn’t good enough to fix it, because ns1.berkeleysignal.com was registered with my domain registrar, so I had to change it there. But the changes don’t seem to have propagated to my ISP in Seattle. I tried to change the delegation to ns2.berkeleysignal.com, but their system for updating that seems broken as well.

Meanwhile, my ISP in San Jose has no clue about how to set up reverse DNS delegation, and has screwed up all the IPs.

I had to give up after a few hours because I had family driving in from out-of-town. So, for now, my reverse DNS is messed up at both of my sites, and it is unlikely that my email will work. For now, it’s probably best to use my jim.pick@gmail.com address.

Final update: I think everything is back to normal.

My server in Seattle disappeared off the ‘net for a few hours today. So kaffe.org, builder.classpath.org, as well as my personal email server and some other things disappeared.

The colo company I’m using in Seattle apparently abandoned the block of IPs that my server was using. I was supposed to migrate off of the old ones, but I never got the email. So today, the IPs stopped routing, and my server went offline.

I got them to reset the IP address on the Xen server over the phone, and I spent the last few hours updating my DNS records, restarting Xen sessions, etc.

If there’s a silver lining to this, it may be that we’ll have less problems with spam blacklists on this new block, since apparently they don’t go through a third party company.

If you see any problems, send me an email. If my jim@kaffe.org or jim@jimpick.com emails aren’t working, you can also try jim.pick@gmail.com.

I’ve been notified that we’re going to have to change the IP addresses on my server in San Jose as well (which hosts developer.classpath.org and icedtea.classpath.org). I’m still waiting for new IP addresses. Hopefully that will go smoother.

I tried setting up the Dojo Offline editor demo a.k.a. Moxie (wiki).

I looked at the code a bit, and it looks useful. I can see that these style of apps are a bit tricky to write — in order for this model to really work well, one needs to store mutations in a log, and replay that. I’m OK with that architecture, as I was thinking along that something along that line is needed anyways to play nice with storing the data into Distributed Hash Tables on the server side. I think the main advantage of this particular Javascript platform is that it includes some sync capability, and a nice Ajax widget.

I’m going to look at TrimPath Junction next. I quickly read over the O’Reilly OnLamp article about it, and it looks a lot like Rails. My understanding is that apps can also be made to run on the server side using Helma (a Javascript web server framework that runs on Rhino). There’s an app called Next Action that is built with it that you can try out. It looks like that can run on the client, or the server. However, I don’t see any support for sync and transferring between running on the client and the server (like Google Reader can do). I wonder if the Dojo Offline sync widget could be easily hooked into it?

At some point, I’m going to have to bite the bullet, pick a framework, and start writing some apps.

In other news:

• Kaffe 1.1.8 is out!
• I bought a cool little projector. So now I really need to come up with some neat demos. I see that they just cut the price by $200. Doh!
• On Tuesday, Workspace got robbed. 6 PCs were taken. I was actually considering coming in that night, but I didn’t. One of the stolen PCs was an Apple iMac that was set up as a “Photo Booth”, where people could upload their photos to Flickr. After the iMac was stolen, somebody was stupid enough to use it to actually upload their picture to the Flickr account using it. Hilarious. It’s even on Boing Boing, Digg, Fark, MeFi, The Globe and Mail, and The Vancouver Sun. Check Bill MacEwen’s blog for the latest. Update: Got him!

I had to reboot “ariadne”, the server in Seattle. It appears that the ancient version of Xen running on there was misbehaving. It’s had quite a lot of uptime however.

This mostly affected kaffe.org, builder.classpath.org, plus some of my personal stuff.

As for this project, I’m still working on it. I haven’t logged many hours lately, but I’m still committed to it, so stay tuned.