Hi all.
I am creating a social tool - I want to allow search engines to pick up "public" user profiles - like twitter and face-book.
I have seen all
the protocol info at http://www.sitemaps.org and i understand this and how to build such a file - along with an index if i exceed
the 50K limit.
Where i am struggling is
the concept of how i make this run.
The site map for my general site pages is simple i can use a tool to create
the file - or a script - host
the file - submit
the file and done.
What i then need is a script that will create
the site-maps of user profiles. I assume this would be something like:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.socialsite.com/profile/spidee</loc>
<lastmod>2010-5-12</lastmod>
<changefreq>???</changefreq>
<priority>???</priority>
</url>
<url>
<loc>http://www.socialsite.com/profile/webbsterisback</loc>
<lastmod>2010-5-12</lastmod>
<changefreq>???</changefreq>
<priority>???</priority>
</url>
</urlset>
Ive added some ??? as i don't know how i should set these settings for my profiles based on
the following:-
When a new profile is created it must be added to a site-map. If
the profile is changed or if "certain" properties are changed - then i don't know if i update
the entry in
the map - or do something else? (updating would be a nightmare!)
Some users may change their profile. In terms of relevance to
the search engine
the only way a google or yahoo search will find
the users (for my requirement) profile would be for example by means of [user name] and [location] so once
the entry for
the profile has been added to
the map file
the only reason to have
the search-bot re-index
the profile would be if
the user changed their user-name - which they cant. or their location - and or set their settings so that their profile would be "hidden" from search engines.
I assume my map creation will need to be dynamic. From what i have said above i would imagine that creating a new profile and possible editing certain properties could mark it as needing adding/updating in
the sitemap.
Assuming i will have millions of profiles added/being edited how can i manage this in a sensible manner.
i know i need a script that can append urls as each profile is created
i know
the script will prob be a TASK - running at a set freq - perhaps
the profiles have a property like "indexed" and
the TASK sets them to "true" when
the profiles are added to
the map.
I
dont see
the best way to store
the map - do i store it in
the datastore i.e;
model=sitemaps
properties
key_name=sitemap_xml_1 (and for my map sitemap_index_xml)
mapxml=blobstore (the raw xml map or ror map)
full=boolean (set true when url count is 50) # might need this as a shard will tell us
To make this work my thoughts are
m cache
the current site map structure as "sitemap_xml"
keep a shard of url count
when my task executes
1. build
the xml structure for say
the first 100 urls marked "index==false" (how many could u run at a time?)
2. test if
the current mcache sitemap is full (shardcounter+10050K)
3.a if
the map is near full create a new map entry in models "sitemap_xml_2" - update
the map_index file (also stored in my model as "sitemap_index" start a new shard - or reset.2
3.b if
the map is not full grab it from mcache
4.append
the 100 url xml structure
5.save / m cache
the map
I can now add a handler using a url map/route like /sitemaps/*
Get my * as map name and serve
the maps from
the blobstore/mache on
the fly.
Now my question is does this work - is this
the right way or a good way to start? Will this handle
the situation of making sure
the search bots update when a user changes their profile - possibly by setting
the change freq correctly? - Do i need a more advance system :( ? or have i re-invented
the wheel!
I hope this is all clear and make some form of sense :-)
