EIITbot information page

Dear visitor,

the purpose of this site is to provide information about EIITbot. EIITbot is a web crawler that gathers price data for an academic research project conducted by Tobias Renkin and Prof. Josef Zweimüller at the Department of Economics of the University of Zürich. We follow generally agreed upon best-practice for web-crawling. For further information, please feel free to contact us per email.

Data collection

It is important to us to run EIITbot in a way that will not affect the performance of the webservers we collect data from. To do so we limit the rate at which our crawler contacts webservers to a rate that should be similar to human users.

In addition, we follow the robots.txt standard. Robots.txt is a simple way for content providers to control the behavior of web crawlers. Among other things, it can be used to keep web crawlers from accessing websites or specific parts of websites.

Specifically:

• We will always identify ourselves in the 'User-agent' string of any HTTP requests we send. The 'User-agent' string we use is EIITbot (http://www.econ.uzh.ch/eiit/eiitbot.html).
• We will comply with the robots.txt standard at all times. We follow the 1996 Robots Exclusion Protocol standard, as well as newer extensions to the standard which were implemented by Google, Microsoft and Yahoo.
• We will always request data in compressed format from your webserver to keep transfer size to a minimum.
• We limit our requests to at most one every 3 seconds, or the value of crawl-delay specified in robots.txt (whichever is higher).

Research project

Our research is part of a larger research agenda titled “Income Inequality and International Trade” (in short EIIT). The EIIT project is funded by a generous grant through the Sinergia programme of the Swiss National Science Foundation (Grant no. CRSI11_132272). It involves researchers at the University of Zürich (Prof. Josef Zweimüller, Prof. David Dorn), at ETH Zürich (Prof. Peter Egger) and at the University of St. Gallen (Prof. Reto Foellmi). The EIIT project was initially funded from 2011 to 2014 and has recently been extended until 2018.

Within the EIIT project, our research aims at understanding how countries' per-capita income affects the local price level of tradeable consumer goods. For this purpose, we collect prices of standardized retail products that are sold in many different countries. Our final dataset will cover prices of many different products and firms in a large set of countries. In a sense, our data gathering is an extension to the well-known Big-Mac index to many different tradeable products.

The data we collect will be used for scientific research only. Specifically, we will not make the data openly available to others, we will not quote specific prices in any published work and we will not compare the level of prices of any firm with prices of competitors.

Contact

For further inquiries, please feel free to contact us per Email to tobias.renkin@econ.uzh.ch.