Domain Specific Language for Web Scraper Development

The process of web scraping is done by using an application called scraper. Various tools have been built to facilitate the process of scraper development, but these tools can only be used by programmers, while users who need web scraping not only programmers but also non-programmers. Non programmer referred to in this research is someone who is able to run the computer through the shell and less experienced in programming. It takes a suitable approach in the construction of scrapers by non-programmers. Therefore, in this research will be built scrapingDSL, a domain specific languages that can be used by non-programmers to build scraper. Scrapers built using scrapingDSL must be able to do web scraping against all types of websites with different structures and should be easy to understand, write and modify. To build a scrapingDSL that can meet those needs, the researcher analyzed the tools used in web scraping to determine the right tool for the scrapingDSL engine, analyzed the website structure to determine the need for the scrapingDSL and the translator application, and analyzed the the website content retrieval flow to determine the required language scrapingDSL elements. At the end of the study a test was conducted to determine whether scrapingDSL could be used by non-programmers to build scrapers. Testing is done by requesting 12 non-programmer respondents to build scraper using scrapingDSL.