Server Admin

Google Not Interpreting robots.txt Consistently

11 Comments

I had an issue where Googlebot was spidering parts of my site that were not allowed in the robots.txt file…

My old robots.txt file…

User-agent: *
Disallow: /tools/suggestion/?
Disallow: /search.php
Disallow: /go.php
Disallow: /scripts/
Disallow: /ads/

User-agent: Googlebot
Disallow: /ebay_

Hmmmm… that’s weird… Googlebot is still spidering stuff it shouldn’t be…

www.digitalpoint.com 66.249.66.138 - - [14/Mar/2006:06:21:07 -0800] "GET /ads/ HTTP/1.1" 302 38 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +//www.google.com/bot.html)"
www.digitalpoint.com 66.249.66.138 - - [14/Mar/2006:10:26:18 -0800] "GET /ads/ HTTP/1.1" 302 38 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +//www.google.com/bot.html)"
www.digitalpoint.com 66.249.66.138 - - [14/Mar/2006:14:29:35 -0800] "GET /ads/ HTTP/1.1" 302 38 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +//www.google.com/bot.html)"
www.digitalpoint.com 66.249.66.138 - - [14/Mar/2006:17:47:21 -0800] "GET /ads/ HTTP/1.1" 302 38 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +//www.google.com/bot.html)"

So I made an inquiry to Google about this, and I actually heard back (nice!)…

While we normally don’t review individual sites,
we did examine your robots.txt file. Please be advised that it appears
your Googlebot entry in your robots.txt file is overriding your generic
User-Agent listing. We suggest you alter your robots.txt file by
duplicating the forbidden paths under your Googlebot entry:

User-agent: *
Disallow: /tools/suggestion/?
Disallow: /search.php
Disallow: /go.php
Disallow: /scripts/
Disallow: /ads/

User-agent: Googlebot
Disallow: /ebay_
Disallow: /tools/suggestion/?
Disallow: /search.php
Disallow: /go.php
Disallow: /scripts/
Disallow: /ads/

Once you’ve altered your robots.txt file, Google will find it
automatically after we next crawl your site.

Okay… I can live with that… easy fix. But check this out… Google’s own robots.txt testing tool within Google Sitemaps show the old robots.txt as being able to block Googlebot as expected.

So how about some consistency here? And more importantly, if anyone at Google is reading this, how about someone tell me why my blog is banned in your index… πŸ™‚

Movies, MPAA

MPAA Suing Me

2 Comments

I’ve been waiting for this lawsuit for a few months now, and yesterday I finally got served the official complaint filed in the US District Court (San Diego).

Now we can finally get this show on the road… I’m sure it will give me all sorts of interesting stuff to blog about in the coming months. πŸ™‚

I’ll meet with my attorney tomorrow to give them the stuff…

Science, Weird Stuff

Family Walks On All Fours

3 Comments

I saw this a couple weeks ago, and just now ran across it again. Dammit, I forgot to post it the first time around… oh well… better late than never.

An extraordinary family who walk on all fours are being hailed as the breakthrough discovery which could shed light on the moment Man first stood upright.

Scientists believe that the five brothers and sisters found in Turkey could hold unique insights into human evolution.

The Kurdish siblings, aged between 18 and 34 and from the rural south, Γ’β‚¬Λœbear crawlÒ€ℒ on their feet and palms.

Study of the five has shown the astonishing behaviour is not a hoax and they are largely unable to walk otherwise.

Researchers have found a genetic condition which accounts for their extraordinary movement.

And it could provide invaluable information on how humans evolved from a four-legged hominid into a creature walking on two feet.

Science, Weird Stuff

Coke and Mentos

4 Comments

I’m not sure why, but the video oddly makes me want to try it myself to see if it really works. πŸ™‚

Update

We confirmed that this does work. πŸ™‚ We didn’t quite have as good of a technique for getting the Mentos into the 2 liter bottle as quickly, but it does in fact work. Maybe Julien or Scott will post the video to YouTube.

Sports

Nobody Likes Anne

Leave a comment

I would just like to throw out a big *F-U*!!!! to Anne. She went golfing with Julien, Scott and myself this morning. Naturally Scott won (he has played a lot longer than anyone else). But it was Anne’s first time ever playing golf and she kicked the crap out of me and Julien.

“Oh… gee, another drive onto the green.” Whatever… no one likes playing with you Anne. Go eat corned beef instead.

I did get my first birdie ever today, so I’m still cool.

Animals, Tech Stuff, Weird Stuff

Insect Cyborgs

1 Comment

I saw this on Boing Boing this afternoon… it looks like DARPA (Defense Advanced Research Projects Agency) is starting to research the feasibility of engineering insect cyborgs…

Quote from their website…

DARPA seeks innovative proposals to develop technology to create insect-cyborgs, possibly enabled by intimately integrating microsystems within insects, during their early stages of metamorphoses. The healing processes from one metamorphic stage to the next stage are expected to yield more reliable bio-electromechanical interface to insects, as compared to adhesively bonded systems to adult insects. Once these platforms are integrated, various microsystem payloads can be mounted on the platforms with the goal of controlling insect locomotion, sense local environment, and scavenge power. Multidisciplinary teams of engineers, physicists, and biologists are expected to work together to develop new technologies utilizing insect biology, while developing foundations for the new field of insect cyborg engineering. The HI-MEMS may also serve as vehicles to conduct research to answer basic questions in biology.
Sports, Travel

No Mexican Jail For Me

3 Comments

Julien, Anne and myself made it back from the Baja 250 without spending a single day in Mexican jail.

You can get pretty close to the race action (as you can see here)…

On the way there… What the hell is this? 50 miles worth of snow at 4:00 am? Lame!!!!

We made it… this is what was going on.

On the way back… This was rather interesting… this was a Mexican jumping the US border to get INTO Mexico. πŸ™‚

Travel

Baja 250

Leave a comment

I *could* be in a Mexican jail right now, but if all goes well I’m in San Felipe watching the start of the Baja 250 race.

Hopefully this whole “write now, publish later” thing in WordPress works… I’m even more hopeful that by the time you read this, that I actually made it to San Felipe, and am not in jail.

Weird Stuff

Get Drunk Remotely

1 Comment

The site calls them “Lover’s Cups”, but I think they are just a way for lazy people to get drunk with each other (like if I’m too lazy to go to a buddies house)…

Lover’s Cups explore the idea of sharing feelings of drinking between two people in different places by using cups as communication interfaces of drinking. Two cups are wireless connected to each other with sip sensors and LED illumination. The Lover’s cups will glow when your lover is drinking. When both of you are drinking at the same time, both of the Lover’s Cups glow and celebrate this virtual kiss.

//web.media.mit.edu/~jackylee/cups.htm

Music, Weird Stuff

Neverland Ranch Shut Down

Leave a comment

Today is a sad, sad day for the entire world… Michael Jackson’s Neverland Ranch was shut down today…

Michael Jackson has been ordered to shut his Neverland ranch by California authorities who have fined the pop star $US169,000 ($230,700) for failing to pay his employees or maintain proper insurance.

Jackson’s sprawling ranch in the central California foothills was closed yesterday, at least temporarily, by an agent of the State Labor Commissioner after the office discovered that his workers’ compensation policy had lapsed in January.

On a brighter note, Michael Jackson is looking pretty good these days, don’t you think?

Uncategorized

Random Email #8

2 Comments

This guy is lonely… πŸ™

I need true, real, good friend.

That’s the entire email… nothing else. Maybe if you would tell me a little about yourself, your desires in life or whatever else, I could be your one true, real, good friend.

Sports

I Can Drive A Golf Ball?

1 Comment

Uhm… something strange happened tonight. Julien and I went to the driving range to hit some golf balls… and much to both of our amazement, we were both able to hit the ball relatively well with our drivers (this pretty much was a first for both of us). What in the hell happened? Must be something in the water.

That’s the good news… the bad news is I lost $34 to him in our “pitching challenge”… probably because he cheats somehow.

Uncategorized

Random Email #6

Leave a comment

What the hell is this person sending me this for?

Have a list of Credit Card numbers, CVV numbers, phone numbers and addresses of American and British citizens. Have got these from BPOs. Let me know if youre interested.

Uhm… do I look like the credit card fraud auction center to you dude? I don’t even know what a BPO is. I don’t think you are allowed to use other people’s credit cards without their permission, so what good would a bunch of other people’s credit cards do for me? Nada.

Server Admin

MySQL 5.1 Out Of Alpha

Leave a comment

With version 5.1.7, MySQL 5.1 (which is something I’m [not] patiently waiting for) has gone from alpha to beta status.

The stuff I’m really looking forward to is it’s improvements to the NDB Cluster engine…

  • Disk Data tables (before NDB Cluster required everything to be memory-resident)
  • Integration of MySQL Cluster and MySQL replication
  • Variable sized records

Now hurry up through the beta phases! πŸ™‚

The only thing (that I can think of) that MySQL Cluster will be lacking for 5.1 is support for FULLTEXT indexes… not a deal breaker though since you will be able to replicate to a MyISAM to get a FULLTEXT indexed table. Should see it in MySQL 5.2 though (hopefully). Oh, and the ability to alter live tables sure would be nice.