Shivam Dixit
Data Engineer, experienced in scraping websites, building API's and designing CI/CD pipelines.
👉 shivam.r.dixit@gmail.com
Skills & Qualifications
- ✔️ 3+ Years experience with Web scraping, Automation
- ✔️ Experienced in CI/CD Pipeline & basic knowledge in API
- ✔️ 2+ Years experience in designing ETL pipeline
- ✔️ Built LLM-powered applications using OpenAI, Anthropic API
Tech stack
Work History
10/2022 - Present
Worked on exciting projects including collecting data from various sources using the Scrapy framework, building LLM-based automation tool to generate entities. Last but not the least, developing APIs using our own in-house Flask framework.
- Designed a scalable solution using scrapyd to run & control the spiders hosted on AWS EC2.
- Building ETL pipeline to extract the data from the source, transform it and load into database.
- Designing robust CI pipeline from scratch which has been running without failure in the last 1 year.
- Built an LLM-powered application using OpenAI (GPT-4/GPT-3.5) and Claude API to extract domain-relevant entities from web pages and generate industry-agnostic schemas.
10/2021 - 10/2022
collecting data from different travel websites to provide better deals to customers.
- Developed custom web scraping scripts using tools like Scrapy, or Selenium to meet specific project requirements.
- Conducted data cleaning and preprocessing to ensure accuracy and consistency of scraped data
- Implemented proxy and user-agent rotation to prevent IP blocking and improve data collection quality
Accomplishments & Awards
🏆 Reduced CI pipeline infrastructure cost from $300 to $0, achieving significant cost savings for the organization.
🏆 Transformed the organization's ticket system by automating Notion, doubling efficiency and streamlining workflows.
