Jerry Stuckle scribed:

>Ed Jay wrote:
>> Jerry Stuckle scribed:
>>
>>> Ed Jay wrote:
>>>> Jerry Stuckle scribed:
>>>>
>>>>> Ed Jay wrote:
>>>>>> Brian Cryer scribed:
>>>>>>
>>>>>>> "Jerry Stuckle" <jstucklex@attglobal.net> wrote in message
>>>>>>> news:R72dnTtJFO7zckTbnZ2dnUVZ_r3inZ2d@comcast.com. ..
>>>>>>>> Brian Cryer wrote:
>>>>>>> <snip>
>>>>>>>>> For what its worth, for link exchanges I don't exchange links with sites
>>>>>>>>> which use perl, nofollow or scripts for their links.
>>>>>>>> There is no such thing as a "perl link" on a web page. Perl may generate
>>>>>>>> the link - but it's straight html code, and no one can tell from the
>>>>>>>> client side whether the link was generated statically, with Perl, PHP, ASP
>>>>>>>> or one of the 1,000,000 parrots pecking on keyboards.
>>>>>>> Quite true.
>>>>>>>
>>>>>>> What I meant, and what I think the OP was referring to is that pages that
>>>>>>> are generated using perl typically seem to have a zero PR. Whether a 0 PR
>>>>>>> means that Google isn't following the link I simply don't know. For example
>>>>>>> while example.com (if generated using perl) might have a PR of say 5,
>>>>>>> example.com/foo.pl?i=3 typically has a PR of 0. (This may not be restricted
>>>>>>> to perl.) More than happy to be shown that I'm wrong on this - my feeling is
>>>>>>> that I should be wrong about it.
>>>>>>>
>>>>>>> I suppose in the context of the OP thread, a link generated using a perl
>>>>>>> script if it were simply generating html wouldn't in any way be
>>>>>>> distinguishable from a normal link. So, in the context of the thread you are
>>>>>>> 100% correct. Good point.
>>>>>> My specific issue pertains to a single page that contains all of my external
>>>>>> links, and having that page generated using Perl (or any other SS solution).
>>>>>> My observations indicate that none of the SE spiders follow links to Perl or
>>>>>> other SS scripts. If one is penalized for the number of outgoing links, then
>>>>>> the SE spider would never see those links.
>>>>> Another though, Ed - have you tried validating the page? It could be
>>>>> the html is screwed up just enough to upset the se spider. Or are you
>>>>> sure this page is being spidered at all?
>>>> Two things, Jerry. I've written my robots.txt file to disallow /cgi-bin. My
>>>> pages validate 100% and it gets spidered.
>>> Hmmm, that's quite interesting then, Ed. Are the external scripts in
>>> /cgi-bin directories? (I doubt it, but had to ask rather than assume :-) ).
>>
>> Yes, they are. Aren't your Perl scripts?
>
>Some are in cgi, some in cgi-bin - it depends on how the server was
>originally set up.
>
>>> Also, a maybe too-obvious question - how do you know the spider doesn't
>>> follow the external link?
>>
>> I see the HTML page hits in my stats, but no page hits on the pages
>> generated by the Perl scripts.
>
>So you said you're disallowing /cgi-bin in your robots.txt file, but now
>the spiders aren't hitting those in the /cgi-bin directory?
>
>I must be missing something here, because I know that isn't the case.

If I'm disallowing the spiders to roam cgi-bin, how can they spider the
scripts contained within it? I don't understand your confusion.
--
Ed Jay (remove 'M' to respond by email)