We are pleased to announce our two great new features at ResellerCluster.com control panel. You can now fully customize the robots.txt and Sitemap.xml files which will enhance your experience with our reseller hosting plan. For those of you who are new to web spiders and crawling I will explain the basics. Web spiders and robots are simply pieces of software that have instructions to browse the web, gathering and indexing specific data. They were invented in the early 90s by sociology companies and later popularized by what is known today as “search engines”. Robots exclusion standard (robots.txt) Robots.txt is a protocol which when set will prevent certain bots and web spiders from analyzing your reseller store or just some of its data. If you wish to do that, then log in your reseller hosting Control Panel, go to Store Design Tools tab and then click “Switch to Advanced Mode” at the top of your screen (see Fig below). advanced-mode-reseller-hosting What Advanced Mode actually does is to unlock a few more customization options. And one of them is “Customize Robots.txt” as shown below.
What Advanced Mode actually does is to unlock a few more customization options. And one of them is “Customize Robots.txt” as shown below. 
Clicking on the blue robot will open a new window with a basic text editor and a drop down menu from which you choose which store’s robots.txt you want to edit (if you have multiple). Now this protocol is more like some advice for the web crawlers and even though most of them tend to follow its settings, some might not. There really are too many rules you can set in this protocol, but I will try to cover the most common ones: Allows all robots to visit all files: User-agent: * Disallow: Keeps all robots out: User-agent: * Disallow: / And this is how to forbid to all crawlers to enter just specific directories: User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /tmp/ Disallow: /private/ In fact you can forbid a specific bot to enter specific directories: User-agent: EvilBot # where you replace the “EvilBot” with the actual user-agent of the bot Disallow: /private/ And last but not least, this one forbids all bots to enter a specific file: User-agent: * Disallow: /directory/file.html You get the idea… The Sitemap protocol (Sitemap.xml) Now this is another protocol that is almost opposite in function to robots.txt. When set it will actually inform bots of your content and they will start crawling. A must have for a decent position in the search engines. It is a simple XML file and contains your URLs in it. There are a few ways to create a sitemap.xml file – with some third-party software, with a plug-in on your site or by hand. Just google “create sitemap” and you will get tons of options. Once you have your sitemap you need to switch to Advanced mode in your control panel and click on the orange sitemap picture which looks like this. 
Once you do it, you will see another text editor where you can paste and edit your new sitemap to your likings. Click change and you are all set! In conclusion when these two protocols are tuned well, they will provide a lot clearer instructions for the crawlers and thus will produce much better ranking and indexing in the search engines and your free reseller hosting will benefit greatly. Thank you for reading, and see you next time!
