Earlier this week we had a service outage. The proper chain of events would be:

• 00:00:00 Server problem
• 00:00:03 Monitor processes notice problem, send page to admin’s phone
• 00:00:10 Phone rings with new message
• 00:00:30 Admin logs in to server, fixes problem
• 00:01:00 Problem resolved

But what happened was:

• 00:00:00 Server problem
• 00:00:03 Monitor processes notices problem, send page to admin’s phone
• 00:00:10 T-Mobile doesn’t deliver the message
• …

This started around 3am California time, which is why none of the PBwiki team noticed it independent of the sms alert mechanism. What should have been an isolated transient, simple to resolve and not user-visible turned into a cascade of unpleasant timeouts which caused the service to slow and eventually halt. We’ve done an extensive internal examination of what happened, and we’re changing some technology, adding some additional automated checks, and doing a few procedural things more intelligently.

The main process change is something that is probably old hat for old-school ops people — the absence of a page alert is not an indication of systemwide health. We’ve deployed a lot of new infrastructure in the last few weeks, and I’d been getting occasional pages for a while, but none for the prior day or two. I’ve set up the daily equivalent of the Tuesday-at-noon air raid siren test — in which the absence of a message every morning will be a problem itself. We’ve also got independent Nextel phones for on-call ops folks so there are now several routes for the alarm pages to take, plus that funny push-to-talk thing so we can annoy one another at all times.

7:30am on Tuesday, March 13th, 2007: It looks like we’re having a hiccup in service right now, and we’re working on fixing it so we can get PBwiki service back up and running right away.

[Update, 7:42am]: This is fixed. We’re looking into why this happened so it never happens again.

Sweet! You may have heard it around, but we’re super happy to welcome Schtuff.com into the PBwiki family. (For you folks who think we’re recklessly splurging, we made the acquisition with customer revenue, not our newfound VC dollars.) This acquisition brings into the fold a million new wiki enthusiasts. We’re looking forward to working with the team from JanRain (who built Schtuff) to add OpenID support to PBwiki. Here’s to the wiki space!

Today is National Peanut Butter Lover’s Day! We hope you enjoy any real and/or virtual Peanut Butter Sandwiches you create today!

Remember the part about how our huge splurge after getting the financing money was to buy a larger, but very basic, fridge?

More than a week before it was scheduled to be delivered, I received several phone messages about someone wanting to deliver a fridge to us the following day. Needless to say, I was a bit baffled, since the vendor and delivery date were completely different than the fridge I had ordered.

Eventually I figured out that a very nice investor at MDV had actually bought a fridge for us as a surprise, to celebrate the close of the funding round, and, therefore, we could cancel our original order. More money for ramen!

What follows is documentation of the inside of the fridge, a group shot of the happy employees gathered ’round the fridge, and a happy employee (me) bear hugging the fridge.

An email we got after announcing our new Point-and-Click editor.

Thank you. I am a 77-year-old woman with only mild computer skills, and you have just made my life significantly easier.
–Elizabeth D.

To make your own free wiki, visit http://www.pbwiki.com

Folks,

You may have heard it through the grapevine, and it’s true! We’ve raised $2m from MDV to invest primarily in engineering to make PBwiki even friendlier and easier than it is today. We’re going to continue our principled frugality (like getting Hello Boss instead of Starbucks at a quarter the price), but hopefully we can bring some more bright minds on board. Say there, are you an elite Javascript hacker looking for a job?

-David Weekly, CEO

We just got these two pieces of feedback on our Point-and-Click editor within 25 minutes of each other:

oh pbwiki. just when i think you couldn’t make it any easier… there you go again. it is a beautiful editor and it couldn’t be simpler. thank you for your great work! i have started pbwikis for several of my jobs as a grad student and this new editor has only helped grow their popularity.

thanks again!

Yeah!!! And then this one:

I can’t stand it.

Hmm. Well, we do let you switch from the Point-and-Click editor to the Classic editor with just one click. Check the top-right corner of your page when you’re editing.

-Ramit

Dear wiki owners,

A few people have been surprised recently when their users accidentally republished information from private wikis to the world at large by way of an RSS / Atom feed. Here is the low-down on how this can happen and how you can avoid it.

Your users must be careful when using a private wiki to not share the private, unguessable feed URL with third parties. If a user uses a third party web-based feedreader like Bloglines or Google Reader and pastes in the private feed URL, they have given that third party access to the feed, which may likely contain private data, such as email addresses, page names, and snippets of page content.

Some of these third parties may decide to publicly republish the feed’s content, or make it indexable by Google and other search engines. This is in spite of our servers very clearly telling third party services (with the Robots exclusion policy, the Google indexing Atom extension, and the Bloglines access control feed extension) that they should not allow any material obtained on a human’s behalf available to other users. But that doesn’t always stop them, either by accident or malice, from republishing data that a user has given to them.

In order to help protect our private wiki communities from accidental disclosure in this way, we’re going to be disabling feeds by default for all private wikis starting March 15, 2007. If you own a private wiki and want to keep feeds enabled, no problem - just surf on over to the “Feeds” tab of your settings page any time before March 15 and pick “enable” and we won’t get in your way. Or you can disable them immediately if you want to prevent the above situation.

Cheers,

David Weekly
PBwiki’s CEO

Lots of people have requested an enhanced PDF export capability — our new team member Igor has been doing some great work adding features to this little gem of a feature. Top priorities for us are:

1) proper unicode support (for all those wacky citizens of the world)
2) proper handling of lists, especially nested lists
3) inlined images

I’ve done a first-pass integration of Igor’s improved code on our development servers, and have a screenshot to share. Expect to see more tweaking of the vertical spacing and formatting, with general availability in the next few weeks.