Volition

Posted on Jul 1, 2020

About the project

Volition is an app that wants to be the top selling place for a certain kind of product. In order to achieve that, we had to develop a series of crawlers for different vendors, in order to get all the data so the storefront could be created .

Tech Stack

  • JavaScript
  • TypeScript
  • NodeJS
  • PuppeteerJS
  • Docker/docker-compose
  • PostgreSQL
  • Google Cloud
  • Kubernetes
  • Bash
  • ELK (ElasticSearch, LogStash, Kibana)

What did I work on?

  • Team lead
  • Moved the entire project to docker and docker-compose. Before it, the development environment has pretty tricky to setup.
  • Improved the old code, introducing standards with esLint and smoke tests.
  • Configured a VPN and an Ubuntu VNC session in docker to help with the proxy and the non-headless browser.
  • Created new crawlers for the new vendors.
  • Configured the new Kibana dashboard.
  • Created a gatekeeper to check the crawlers status before going out to the internet.
  • Monitored and ran many crawlers.