Xarg blog Archive

Tim Bray has written a nice collection of documents on searching at his blog.

It covers a lot of background and highlights the multitude of hard (and largely unsolved) problems that still exist.

I think Clay Shirky rocks and if you don't read him now you should. His most recent article is about the semantic web and the promise it really holds. It joins Cory Doctorow's article as one of my favorite articles on metadata.

Much of the proposed value of the Semantic Web is coming, but it is not coming because of the Semantic Web. The amount of meta-data we generate is increasingly dramatically, and it is being exposed for consumption by machines as well as, or instead of, people. But it is being designed a bit at a time, out of self-interest and without regard for global ontology. It is also being adopted piecemeal, and it will bring with it with all the incompatibilities and complexities that implies. There are significant disadvantages to this process relative to the shining vision of the Semantic Web, but the big advantage of this bottom-up design and adoption is that it is actually working now.

Cory Doctorow is more succinct:

A world of exhaustive, reliable metadata would be a utopia. It's also a pipe-dream, founded on self-delusion, nerd hubris and hysterically inflated market opportunities.

I clicked on my amazon gold box offer for the first time and found the Nesco American Harvest, BJW-1, Beef Jerky Works Kit was my personal treasure. It looks like a beef jerky caulk gun to me; I am now a true believer in Amazon's ability to see through to our darkest desires.

I bought a little Shuttle XPC box and installed gentoo linux on it. Gentoo is cool, it's sort of like debian but it's all compiled from source so you can compile everything optimized for your particular box. I would not say it's for the faint of heart and the install summary has about 35 steps but it all worked for me in the end. The package install tool is nice and it's pretty easy to install dev packages (like gimp 1.3.21).

It took a few days to compile everything and it's astonishing to consider how complex the full build was but I do feel like I got my money's worth out of the cpu.

As soon as I get dual screens working on the shuttle box I am going to switch to using it as my desktop box and stick my current (very noisy) tower in the garage as my db/dev/fileserver box.

There is a story on CNN about a company selling personal submarines for $845k. I would like one but what I really really want is the Deep Rover which is amazing, it's two hemispheres of acrylic, has pretty much 360 degree visibility, and can dive to 1000m. No price on the website for that one though. (here's another page with some more photos).

I recently did to installs here, Windows XP (since bit rot had afflicted the windows ME install on the kids machine and it would not stay up for more than a few hours at a time), and Debian + Oracle since the hard drive on my db machine died. I expected Windows to install smoothly and I expected oracle + debian to be the usual eyeball stabbing fun (especially since I had not installed either debian or 9i before).

In a shocking turn of events the Debian install was easy, Oracle was the same as it always is, and the XP install was a hellish nightmare with me having to update the BIOS, pull out all the cards but the video card, and fiddle with memory before it would allow itself to install on the box (and I still am not sure what the problem was). I think Windows ultimate downfall will be it's increasingly annoying attempts to control the platform (with registration, DRM, hardware requirements, etc) while open source platforms like Debian will continue to get easier to use and more robust (and will support legacy hardware not stamped with the Windows logo). Anyway, I am pleased with Debian, although I might try out gentoo as well.

I spent the afternoon preparing myself psychically to stick my clumsy fingers into my digital camera. I bought the eclipse solution (pure methanol I think), the pec-pads, the spatula (no Wendy's here in london) and finally worked up the nerve to risk destroying my favorite toy. All in the interest of getting rid of a little dust.

In the process of figuring out how to clean the camera myself I ran across the best article summing up the joys of owning a DSLR camera.

On the plus side, the Boots down the street has a Fuji Frontier 370 now and can make reasonable prints for 15p (about $.22) each, with a second set half price. I am actually finally getting some of the photos I never took the time print and am finding I like having them on paper rather than just in the computer.

I realized while cleaning out my office that as best I can tell it has been 10 years since I first installed linux. I found my slackware 1.0.2 boot floppy (the other 40 floppies got tossed some time ago), which was released in Septemeber 1993. I think it was installed on a box with a 20MHz 386 and a 80MB seagate st296n. I had used SCO Unix before that (Oh, the shame) and still remember being amazed at how much easier it was to install than SCO and that all the things I remember from BSD were there and just worked.

There was a New York Times article about the new web mail form sequence instituted at www.whitehouse.gov. It takes 9 clicks to go from the from page to actually sending the message. You have to state up front if you are writing a "supporting comment" or a "differing opinion" and it forces you to choose a subject from an absurd list. For example, under Legal/Judicial you have "Pledge of Allegiance" and "Pornography" but not "Judicial Appointments" or "Civil Rights" (even more entertaining is what you get if you turn off javascript, in that case your selections for "Legal/Judicial" are limited to "Campaign Finance Reform", "Pledge of Allegiance", "Racial Profiling", and "Tort Reform"). Furthermore, you are required to enter an address but there is no provision for an address outside the US. Poorly thought out and offensively bad web design.

I don't believe the black helicopters will be swooping down and I think most of what is wrong with the webmail form has less to do with intent (other than the intent to limit the mail received by the whitehouse) than with sloppy programming and a poorly thought out UI but that it went live at all in it's current state is ridiculous. I am not so naive as to hope to find inspired and creative uses of the internet by the executive branch but I would hope for a sort of plodding competence (well, EDGAR is cool and I think was innovative at the time but that's about it).

Bush better watch out because pretty soon both Paul Krugman and Jakob Nielsen will be after him and then he will be in trouble.

There was a story on /. about the Relativistic Heavy Ion Collider which caught my eye since I worked at Brookhaven for a year when they initially got funding to build the RHIC. Big physics is an amazing thing to behold - detectors the size of houses, petabytes of data, and giant rings of superconducting magnets.

The only real downer would be if the collider caused the the universe to explode. The "transition to a lower vacuum state" sounds like the "rip in spacetime" that threatens starships the galaxy over...

Although certainly nothing in our existing knowledge of the laws of Nature demands it, several physicists have speculated on the possibility that our contemporary `vacuum' is only metastable, and that a sufficiently violent disturbance might trigger its decay into something quite different. A transition of this kind would propagate outward from its source throughout the universe at the speed of light, and would be catastrophic.

The creation of a black hole or a stable "strangelet" that eats the planet are the other two scenarios they cover, but by comparison they seem almost benign. Of course they argue that interactions like this have happened many times in the past and if it was even vanishingly likely that something like this could happen it would have already happened.

I am back from the OpenACS conference in Copenhagen and my vacation in Mauritius (both of which were great fun).

The highlight of the OpenACS conference was meeting in person quite a few people that I have worked with for years but had never actually met. In particular, Lars and I worked together on the first project either of us worked on at Ars Digita and a lot subsequently but never managed to be in the same place at the same time.

It was also great to see how broad the community is becoming and how many people who are turned up having already used the toolkit to build real sites and who are looking for ways to contribute back to the community. It feels like the growth in the community is accelerating and that we have definitely reached the critical mass we need to achieve the kinds of things I am excited about doing.

In the "i18n can go horribly wrong" box goes the touristonline.dk credit card validation failure handling. When you put in the wrong expiry for your credit card you get switched to danish in the error handling code. Even worse is that since it is cookie maintained you can't do url surgery to switch back to english. Even worse than that, it is carrying around your booking information tied to your session so when you go back to the main screen to get back to english and go through the whole sequence from scratch it remembers the previous booking sans credit card details and treats the new booking as being in addition to the first booking. Oddly enough, the final confirm screen switches back to Danish as well so it's hard to tell exactly what is going on. I only figured out that I was booking a second room since it asked for the name of the second guest, and then I noticed the total (in DKK) was too high. There does not seem to be any way to fix it other than to exit your browser and start again...

I guess its bad session management and poor code, but one thing is for sure, it's hard to imagine something more detrimental to the user expereince than having to navigate forms in a language you do not understand, and being compelled to do so since you are not entirely sure if your credit card has been charged or not.

Of course the happy ending is that I finally managed to book just one room and will be going to the dotLRN seminar/OpenACS Social in Copenhagen. It should be fun.

Doc Searls has a nice article at Linux Journal, Tale of Two Stories, about press coverage of Google and asks "What does coverage of Google's success tell us about what's really going on with Linux?

Google has achieved maximum Linux irony by becoming the only commercial enterprise to leverage enormous quantities of free software (10,000 Linux servers at last count) into de facto web infrastructure: private enterprise as public utility.

The short answer is that using linux isn't news anymore, it just works.

I finished the forums performance tuning writeup I did a couple months ago and fixed the last big problem which was the message delete function.

Actually spending more time working with PostgreSQL made me like it more (which is the opposite of the what I always came away with from my Oracle tuning experiences). I think it's often more pleasant to work with open source software since so much of the development is driven by what bothers individual developers. There is a lot less marketing and features for feature's sake. On the other hand, in the case of PostgreSQL it's also not had to face the demands of terabyte+ data warehouses and really demanding transactional environments since so few of the contributors are confronted by those problems. I think it will get there though -- sooner rather than later is my guess as well (since there are people now trying to use Postgres for such things).

This time someone wants to fix my design:

I have recently founded your website. Design is bad. We are promoting new great resource of website templates www.designgalaxy.net . Use my templates to build good website.

They say they partner with flamingtext.com and I think I definitely should take them up on it. There is a decided lack of flaming text on my site!

I got an email from some firm saying

We are the UK's experts in Open Source and would value the opportunity of understanding how we can help you take full advantage of Linux. Give us the opportunity to discuss your current IT infrastructure with you and we will provide FREE OF CHARGE a full and comprehensive scope showing where Linux could be used to improve your system and massively reduce your licensing costs.

Of course the mail headers included

X-Mailer: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106

I found a bunch of curious 404 errors in the server log like this:

194.130.56.116 - - [17/Jan/2003:11:33:36 -0500] "GET /photos/real.css HTTP/1.1" 404 549 "" \
  "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)"

All gets for real.css from IE and always with an empty referer. The only place I include this css file is in /css/main.css where I do an @import url("real.css"); which I would assume would always look for the file in base directory of /css/. Unfortunately IE has different ideas here. I would guess it probably happens when printing given how often it happens and which pages it shows up on. It happens to people who are actively viewing pages and not coming back to the site after having quit and restarted the browser. I just changed the import to use the full path but once again I am annoyed to be doing something stupid to work around a bug in IE.

Nothing but Troubling News From the World of Publishing by Verlyn Klinkenborg in the NY Times:

If there's such a thing as foolish cynicism, that is what the corporate overseers of the once-independent publishing houses are displaying. As long as their main task is satisfying shareholders instead of book buyers, the industry is going to keep digging its own grave, the way the music business has.

and this from Textism on Pinguin Book's hiring of Jan Tschichold:

If ever there was an object lesson for ye somewhat mighty producers of cultural commodities, it is right there in Allen Lane’s decision to find someone who genuinely and obsessively knew what he was doing (as opposed, say, to someone who talked flash crap) award him total authority, and then keep capable people close at hand to follow the lead, indefinitely.

Two days ago, the typesetting staff of Penguin Books Ltd, London, were informed of the company’s intention to make their positions redundant.

Ultimately the internet will increase the scope of what people read, watch, and listen to. The content industries quest for larger margins and more consistent profitability (in general by aggressively promoting particular artists or authors) runs counter to this and is likely ultimately to fail. At least I hope it does. This blossoming of opportunity for lesser known artists and writers might be bad for the shareholders but its good for the artists and its good for the public and is long overdue.

Peter Van Dijck has written an Introduction to XFML.

XFML is a simple XML format for exchanging metadata in the form of faceted hierarchies, sometimes called taxonomies. Its basic building blocks are topics, also called categories. XFML won't solve all your metadata needs. It's focused on interchanging faceted classification and indexing data.

I like XFML, it seems simple enough to actually use. I also think anyone who would link to Cory Doctorow's Metacrap article when describing their own metadata format has a pretty good handle on building something practical. Mark Pilgrim also wrote about XFML and there is a demo of viewing his "Dive Into Accessibility" web tutorial in a generic XFML browser (as a portal or as a search engine).

With the latest Microsoft exploit I sat down to watch the Death Of The Net. I dug out my trusty copy of tcpdump and took a look at the traffic I was seeing. Unfortunately someone upstream was already blocking port 1434 so I did not get to see all the people who thought patching the hole in their MS SQL server just was not that important (the patch has been out since July last year). What I did see was my Mac going out to Apple's NTP server:

17:57:58.002902 core.xorch.net.ntp > media.euro.apple.com.ntp:  v4 client strat 3 poll 12 prec -17 [tos 0x10]
17:57:58.213866 media.euro.apple.com.ntp > core.xorch.net.ntp:  v4 server strat 2 poll 12 prec -17 [tos 0x10]

It is surprising to me that more computers don't come with ntp already configured. It's nice not to have to worry about it.

I set up generating RSS feeds from the CVS log Browser. the full feed is at

http://xarg.net/tools/cvs/rss/

but you can also get a feed for given user, module, or days back. For example:

http://xarg.net/tools/cvs/rss/?user=jeffd&days=10&module=acs-tcl,forums,acs-kernel

Mark Pilgrim's new article, Parsing RSS At All Costs, is up at xml.com.

On average, at any given time, about 10% of all RSS feeds are not well-formed XML. Some errors are systemic, due to bugs in publishing software. It took Movable Type a year to properly escape ampersands and entities, and most users are still using old versions or new versions with old buggy templates. Other errors are transient, due to rough edges in authored content that the publishing tools are unable or unwilling to fix on the fly.

I think using a real xml parser and using some regexp's to fix up common problems might be a better approach although Mark's ultra-liberal RSS parser is short enough that maybe it's a moot point.

Sam Ruby has a great attitude about code ownership...from Rights vs Responsibilities

If somebody copies something that I have done, then generally I am quite flattered. If they chose to give me attribution, I am OK with that as long the recipient takes responsibility for making the copy.

If somebody makes a change to code that I wrote, my first thoughts aren't "what right does he/she have!", but instead, "tag, you're it!".

I have been reading most of my news in an aggregator this week and I realize the short textified descriptions in the RSS are really annoying. I am going to change the code so I send full posts in the RSS.

I am incredibly disappointed that the Supreme Court did not fix what is manifestly a bad law, one that is clearly not in the best interests of the people: The Sonny Bono Copyright Term Extension Act. It is unrealistic to expect the court to impose rationality and correct for the failure of congress to do the right thing when confronted by a law that is not unambiguously unconstitutional. From the last paragraph of the majority opinion:

Beneath the facade of their inventive constitutional interpretation, petitioners forcefully urge that Congress pursued very bad policy in prescribing the CTEA's long terms. The wisdom of Congress' action, however, is not within our province to second guess. Satisfied that the legislation before us remains inside the domain the Constitution assigns to the First Branch, we affirm the judgment of the Court of Appeals.

I feel for Larry Lessig given his incredibly hard work and well reasoned arguments. The public has lost here and Justice Breyers dissenting opinion states that quite clearly:

It is easy to understand how the statute might benefit the private financial interests of corporations or heirs who own existing copyrights. But I cannot find any constitutionally legitimate, copyright-related way in which the statute will benefit the public. Indeed, in respect to existing works, the serious public harm and the virtually nonexistent public benefit could not be more clear.

I saw this word somewhere and did not know what it meant. It turns out to be an excellent word for describing a lot of what goes on in online communities.

It comes from Parkinson's Law by C. Northcoate Parkinson. He describes how a planning board will approve spending millions of dollars to build an atomic power plant but if you go to them to get approval to build a bike shed they will argue endlessly. The problem being that the atomic power plant is so large, complex, and difficult to understand that no one can really argue about how exactly it is done. On the other hand, everyone knows what goes into a building a bike shed and so everyone feels qualified to argue about the details.

For some reason, technical discussions seem to be particularly susceptible to bikeshedding. There was a great post by Poul-Henning Kamp on the freebsd-hackers list which describes a particularly virulent attack they had on their list and submits a plea to avoid it in the future. I think just writing the code is a good antidote to engaging in the arguments in the first place. If the functionality is straightforward and easy to implement then just do it rather than argue about it. Most of the people spending their time arguing probably have enough inertia that they are not going to write any code and consequently, if you do have the initiative, the code itself is the most suitable response. As Poul-Henning Kamp said:

I wish we could reduce the amount of noise in our lists and I wish we could let people build a bike shed every so often, and I don't really care what colour they paint it.

Who knows, if you write enough code we might even end up with an atomic power plant (or for you greens in the audience, a lovely old growth forest).

I added a stylesheet for printing. Of course my new favourite browser (Safari) ignores it completely. It seems to work well in Mozilla, although IE is printing the PNG's as black blobs. It does now print with a serif font which I find much more pleasant to read. I also really like the css href printing trick I found in CSS: Going to Print at A List Apart:

#content a:link:after, #content a:visited:after {
   content: " (" attr(href) ") ";
   font-size: 90%;
   }

#content a[href^="/"]:after {
   content: " (http://www.alistapart.com" attr(href) ") ";
   }

I looked at Textism hoping to find a good example of how to do it and was surprised that Dean Allen did not have a print stylesheet. I guess for a lot of bloggers who work at getting their CSS right (and it really seems to be people with blogs who provide most of the good CSS) don't really think their work is going to be printed so don't make a huge effort to get things to print nicely.

Now that I have a Mac I am trying out NetNewsWire from Ranchero Software. I like it a lot. The planned features for the Pro version sound good (especially the Metaweblog/blogger integration so you could post to your weblog from NetNewsWire). I do wish it had a way to display the unread entries from all subscriptions in one headline browsing window (currently it does not do anything sensible when you select multiple subscriptions other than let you mark them as read).

I have also been using Safari. It is really, really fast and with the exception of a couple sites like dive into mark seems to work great (and on dive into mark it fails to display the right links which I never really use anyway). It has spell checking built in which in my mind is an absolutely great feature and something it amazes me took so long to show up. One great thing to see is that they are contributing the code back to khtml. Another is that they are really listening to what the community says (every software vendor should have someone like Dave Hyatt out there). If they keep listening and keep working I have no doubt that Safari will end up being insanely great.

I saw this in the CUPS HOWTO by Kurt Pfeifle and it really struck a chord:

Please be very hesitant to write mails to private adresses of people you know answering questions in public forums. Don't expect them to react dutifully on unsolicited personal mails. Start asking in the suggested public forums and only send material to private mail addresses if asked to do so. Asking on a public list will enable everybody to learn from the given answers. It increases your chance to get a quick response, as there might be different people watching the list fit for giving advice. Remember, this is a volunteer service. Sometimes people are tired to answer. Can you imagine that someone might feel molested by receiving 20 - 30 personal mails per day, asking the same questions about printing over and over again? It is OK to ask the questions -- but keep it voluntary for your advisors, as it is the case, if you ask in the following public forums or on mailing lists:

One corrollary to the LazyWeb idea is that there is tremendous value in public dialogue, which personal email definitely is not. Mailing lists plus a search engine are pretty potent way to store the wisdom and hard work of the people actually solving problems. Of course, the other trick is that the people answering questions need to make an effort to get the answers out into a public forum as well. I still go to google first when I have a problem (and to Usenet really, rather than the web).

There are some tricks to making answers more valuable. One is to provide a little context along with the answer. Another is to make sure that if there is an error message that shows up when the problem manifests then quote it along with the answer (since a lot of people will just search with the text of the error message to try to find solutions). Also, if someone asks the same question again try to point them to the canonical answer so that if someone provides a better solution or your answer is wrong it's more likely people will end up with the best solution.

I love the snow. When we were flying back from California we were watching 101 Dalmations which is filmed around where we live. I told the kids that it does not really snow like that very much in London, that it was only a movie...well, I am a big fat liar. I think I have seen Cruella on King's Road a few times. Dalmation owners better watch out, no telling what might happen now.

We are back in London and I am playing with my new powerbook. It is a lot of fun and I am generally impressed with how wonderfully it all fits together. It's not without it's bumps (although many are just me unlearning old habits from other platforms). The iApps are pretty cool. I love iPhoto but I can see it will not scale very far (I have about 1000 pictures in it and it's already dragging), on the other hand, I transfered my whole cd collection to iTunes and it is lightening fast. I do miss having a graphical version of emacs though.

Naturally, my website looks wrong in IE on the Mac so I will have to strugle some more with the <li> based menu bar again. It's funny that the Mac version of IE would have problems that the windows version did not have (at least at this level). It probably springs from the same place as other inconsistencies in Microsoft software as described by Adam Barr in a recent Kuro5hin article:

Furthermore, some Microsoft groups have always had a bit of a bias against having too many dependencies. The idea was get it done and don't worry about other people in the company. That's why different applications wrote their own Print dialogs, and why networking between Windows NT and Windows 9x isn't a seamless as it could be. That attitude helped shipped some products sooner, but it permeated outside the company too.