Googlebots and Site Duplication as potential spam!

[zarathustra]

Greetings all. Just signed up.    I have a couple of 'quick' queries I really hope someone can answer.
I created a new website - www.artgraphica.net - which I registered with google two or three weeks ago. From my logs, the googlebots are taking a peek at my robots.txt and my index page, but then disappear. I have a page ranking of zero. I have a number of sites pointing to mine with PR of 3 or 4 on average, so will I have to be patient waiting for something to come along and deep crawl? In contrast MSN has been very quick to look through all my pages.

My other query is that my page content is largely duplicated through a flash enhanced version of the website that people can visit via a link in the main menu. I also intend to include downloadable PDF's pretty much duplicating my online tutorials. Is there a danger someone like google could look at this and consider it spam because the information is there twice? If this is the case would adding a 'disallow' for this flash folder in my robots.txt stop the bots going here and therefore solve a potential problem?

Thanks in advance!!

[Hope]

Greetings all. Just signed up.

Welcome to the forums.

I created a new website - www.artgraphica.net - which I registered with google two or three weeks ago. From my logs, the googlebots are taking a peek at my robots.txt and my index page, but then disappear. I have a page ranking of zero. I have a number of sites pointing to mine with PR of 3 or 4 on average, so will I have to be patient waiting for something to come along and deep crawl? In contrast MSN has been very quick to look through all my pages.

Lets start with 2-3 weeks ago. This is not nearly enough time for google to do a deep crawl. You will need to give it time. You have a PR of 0 because google hasn't updated PR or their index in that time. They are not showing any backlinks to your site. On top of that, they just got hit with a virus and this could  be effecting the results still.
 
As for MSN crawling all over your site, it is a good thing but doesn't help you at all. MSN hasn't launched their search index yet. The spider is going all over the net getting as much information as it can. The estimated launch is December. I woudn't worry too much or be too excited over the MSN spider.

Remember, when you start playing with the search engines, you need to sit back and wait for them. You can not make them move faster than they want. Patience is a key here. If you can't wait or dont' have the patience to wait, I would suggest Google Adwords and Overture.

My other query is that my page content is largely duplicated through a flash enhanced version of the website that people can visit via a link in the main menu.

This appears to be in a frame set. This should not be a problem for the search engines in general. They don't follow frames. This might be an issue for people wanting to link to you though. Keep that in mind.

I also intend to include downloadable PDF's pretty much duplicating my online tutorials.

This should not be an issue. This is a common practice and not frowned upon by search engines. They understand that PDF are used to provide a means of downloading and saving or printing files. Don't worry about this.

Is there a danger someone like google could look at this and consider it spam because the information is there twice? If this is the case would adding a 'disallow' for this flash folder in my robots.txt stop the bots going here and therefore solve a potential problem?

I really don't see this as being an issue. The back side code is different. The flash site is framed. These should solve the issue of spiders seeing spam.

There really isn't a need to have duplicate content though. You don't have enough flash on those pages to make it a problem for users or spiders. Why do you have both?

One the flash page, there is a lot of scripting inside the pages, you might want to pull that from the page and call it from another file. Makes it more spider friendly.

[zarathustra]

Hi Heidi - just wrote a nice lengthy reply, and the browser crashed!! 
Basically I just wanted to say a big thank you for sharing your time and knowledge.

The site was original flash, but I realised it wasn't search engine friendly, and so made it an optional version of the site. The main site itself is what I believe to be search engine friendly; I stripped out all the scripts and used an externall CSS. I like a little interactivity, animation and music which is why I add flash animations here and there, plus I can put it to good use when I come to do future art tutorials online.

I will wait patiently for Google and the bots to do their thing; I've also submitted to Dmoz and Altavista, and hope Altavista won't take too long, but I suspect Dmoz will if I hear from them at all.

I was using google yesterday when the problem occured - I figured it had to be serious, but caught more on the news about the mydoom virus.

Thank you once again, you've been a great help.

Gavin.

[Hope]

Gavin,

Only a full flash site is not indexable right now. Your site has flash elements, but most is nonflash, so it should be ok. Any nav in flash should have a second nav in html. Then it should be fine.

You will not hear from Dmoz. Just keep checking back. If your site is not indexed in 2 months send an email to the editor and request them to review it. Explain that you do not want to resubmit because you don't want it to look like you are trying to spam. This usually motivates an editor.

[zarathustra]

Because the main site contains no flash, I don't mind the flash side not being indexed at all - in fact I included a disallow in my robots.txt to stop them going there! I think the navigation and hyperlinks in the non-flash main side of the site should be enough for the bots to crawl through. 

Thanks for the hint about Dmoz - again thank you, the advice is very useful and much appreciated!

[Lampwize]

I am no SEO specialist or anything, but when I took a look at your code, you have an improper !DOCTYPE declaration. I don't know if this might cause a problem or not, but think it would be smart to fix it. Here is a page on choosing a correct !DOCTYPE: http://htmlhelp.com/tools/validator/doctype.html. After you have done that I would also validate my code: http://validator.w3.org/. Good luck on your high future page rank!

[spherica]

Just an update on this. Google started crawling / parsing flash a while ago.

There have been discussions on this for a while now.

I also know in some circles they were doing contest for ranking flash sites, but can't remember where, so I don't know the outcome.

[Andy]

I have noticed MSN being very active in crawling and found Hope's comment on it crawling the net very interesting. In fact it's like they are fighting 1-1 on my new website that isn't yet ranked in Google. If only I new what to do to give it the edge? Maybe have some MS related keywords :-) e.g. pagerank.net LOL!!!

[Hope]

We are all sitting and waiting to see how this whole MSN search will work out. I know it is in the top 3 for traffic to all my sites. I can't wait to find out how things will settle down so I can get to work on the algo and rank better.