Volition

About the project

Volition is an app that wants to be the top selling place for a certain kind of product. In order to achieve that, we had to develop a series of crawlers for different vendors, in order to get all the data so the storefront could be created .

Tech Stack

• JavaScript
• TypeScript
• NodeJS
• PuppeteerJS
• Docker/docker-compose
• PostgreSQL
• Google Cloud
• Kubernetes
• Bash
• ELK (ElasticSearch, LogStash, Kibana)

What did I work on?

• Team lead
• Moved the entire project to docker and docker-compose. Before it, the development environment has pretty tricky to setup.
• Improved the old code, introducing standards with esLint and smoke tests.
• Configured a VPN and an Ubuntu VNC session in docker to help with the proxy and the non-headless browser.
• Created new crawlers for the new vendors.
• Configured the new Kibana dashboard.
• Created a gatekeeper to check the crawlers status before going out to the internet.
• Monitored and ran many crawlers.