1) The directories being blocked were not the issue. The robots.txt reference was used solely as a list of user-agents to test against.

2) If del.icio.us is serving these 404s to prevent spoofing, then it is being done to prevent proxy hijacking, not content-scraping. A content scraper could just spoof a normal Mozilla user-agent if it was worried about getting caught.

On a final note: I’m not sure at what point I became a “self-proclaimed pundit”. I simply encountered unusual behaviour from del.icio.us, did a little investigating, and wrote about what I found. People took from that what they did.