About the project
Volition is an app that wants to be the top selling place for a certain kind of product. In order to achieve that, we had to develop a series of crawlers for different vendors, in order to get all the data so the storefront could be created .
- Google Cloud
- ELK (ElasticSearch, LogStash, Kibana)
What did I work on?
- Team lead
- Moved the entire project to docker and docker-compose. Before it, the development environment has pretty tricky to setup.
- Improved the old code, introducing standards with esLint and smoke tests.
- Configured a VPN and an Ubuntu VNC session in docker to help with the proxy and the non-headless browser.
- Created new crawlers for the new vendors.
- Configured the new Kibana dashboard.
- Created a gatekeeper to check the crawlers status before going out to the internet.
- Monitored and ran many crawlers.