>>I know something weird goes on
>>with sitemaps, but there's not enough evidence for me to form really
>>firm ideas

Yes, something weird indeed. I've found that they've hurt my sites
(several sites) ranking after I put up the sitemaps. This is
speculation, but that is what I saw happen after adding sitemaps to
Webmaster Tools.