Our Data Collection Bot
If you're here, we've probably either shared this link with you or you've noticed our bot visiting your online store.
Here's what you're probably interested in understanding about ObcoreBot:
What is ObcoreBot
ObcoreBot is a bot used to collect product information from selected products on retailer websites. This information includes product titles, descriptions, availability, imagery, ratings and reviews as well as positioning in category and search for relevant keywords.
It is sent by Shelf Conscious (trading name of Digital Go-To Ltd) on behalf of our clients.
Who are Shelf Conscious?
Shelf Conscious help brands and retailers to optimise their digital shelf experience. We do this by providing technology and services to find and fix issues with the customer experience on retailer websites. For example, we will collect a product title as displayed on a retailer website and compare it with the product title provided by a brand. If there is critical information missing, we will help to close that gap.
User-Agent Information
We use two default User-Agents:
- Mozilla/5.0 (compatible; Obcorebot/1.0; +http://www.shelfconscious.com/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36
- Obcorebot/1.0
IP Information
We will perform requests from the following two IPs (the default IPs from our staging and live environments):
- 3.11.5.229
- 18.135.227.139
Whitelisting the bots and/or IPs will ensure a more seamless experience for all parties.
About the Data We Collect
For our Shelf Monitoring technology (if in use)
- There are three page types we will collect information from: product page, search engine results page (SERP) and category page.
- Data will be collected a maximum of once per day, outside of local working hours (we will collect data between the time window of 5pm and 9am). The only exception to this is to support maintenance and/or testing.
- We collect information on a needs only basis, we never scrape entire sites.
- All data we collect is open facing, public information. The only exception to this is where specific user credentials are provided by the retailer/client.
For our Buy Online technology (if in use)
- We check to see if the URL active or if it's started re-directing
- We may leverage the above Shelf Monitoring data or collect individual datapoints to enhace the 'Buy Online' experience (for example, we may check if the product is in stock)
Have Questions? Who Can I Speak To?
It is of critical importance that you understand why the data is being collected and the value we bring to you as a retailer. If you have any further questions or require any additional clarity, please contact hello@shelfconscious.com and we'll put you in contact with our leadership to discuss.
Alternative to Scraping
To date, we have never had any retailers ask us to stop collecting data and we believe this is related to the value we bring to all parties and our approach to the data collection.
Should you prefer to provide us with access to an API to collect this data, that is perfectly fine, please get in touch with API documentation and we'll be happy to switch to this method.
If you would like us to stop collecting this data altogether, please get in touch and we'll take the necessary steps.