Lead AI Automation Engineer

Digitain is seeking a highly skilled and strategic Lead AI Automation Engineer to drive the next generation of intelligent process automation across our enterprise ecosystem. In this role, you will lead the architecture, development, and scaling of end-to-end AI workflows—combining advanced web scraping, entity matching, Natural Language Processing (NLP), and Retrieval-Augmented Generation (RAG) systems to transform unstructured data into actionable business intelligence. As a technical lead, you will bridge domain-specific operational needs with cutting-edge AI architectures. You will set engineering standards for data extraction and automated decision-making pipelines, mentor team members, and ensure high reliability for mission-critical automations. Responsibilities Architect Intelligent Automations: Design and deploy resilient, end-to-end automated pipelines that integrate Large Language Models (LLMs), RAG architecture, and custom AI tools with core enterprise systemsData Scraping & Ingestion at Scale: Lead the strategy for large-scale web scraping, API extraction, and document processing, ensuring robust handling of anti-bot protections, changing schemas, and dynamic Web environmentsIntelligent Data Matching & Entity Resolution: Build sophisticated algorithms and semantic search solutions (vector databases, embedding models) to perform high-accuracy data matching, deduplication, and record linkage across disparate datasetsRAG & Knowledge Retrieval: Design, evaluate, and optimize Retrieval-Augmented Generation (RAG) frameworks to ground LLM responses in proprietary databases, knowledge bases, and live web dataUnstructured Text & Document Processing: Implement custom NLP pipelines (classification, named entity recognition, sentiment analysis, document parsing) to automate complex decision-making and report generationSystem Reliability & Monitoring: Establish best practices for exception handling, error recovery, LLM output validation (guardrails), and latency optimization across all automated workflowsMentorship & Team Standards: Guide and mentor AI automation specialists, conduct code reviews, and establish standard operating procedures for workflow documentation, API management, and prompt engineeringStakeholder Alignment: Collaborate with cross-functional leadership to identify high-value automation opportunities, map technical feasibility, and demonstrate ROI Requirements 5+ years of total experience in software engineering, data engineering, or process automation, with at least 2–3 years hands-on experience building AI/LLM-powered automation systemsProven track record of leading complex automation projects involving unstructured data, web scraping, and NLPAdvanced proficiency in Python (Pandas, Asyncio, Playwright, Selenium, BeautifulSoup, Scrapy, FastHTML/FastAPI)Deep expertise with LangChain, LlamaIndex, OpenAI API, Anthropic, Hugging Face, vector stores (e.g., Qdrant, Chroma, Pinecone, FAISS), and fine-tuning/prompting techniquesSolid understanding of fuzzy matching, BM25, semantic search, vector embeddings, and probabilistic record linkage strategiesHands-on experience scaling headless browsers, managing proxy networks, solving CAPTCHAs, and working with complex web scraping frameworksMastery of workflow automation tools and orchestrators (e.g., Apache Airflow, Prefect, n8n, Make, or temporal workflows)Strong architectural mindset with an obsession for error handling, data accuracy, and pipeline resilienceExcellent communication skills to explain AI limitations, confidence scores, and strategic value to non-technical stakeholdersResourceful problem-solver capable of building workarounds for unstable APIs or challenging scraping targetsExperience in high-volume, dynamic industries such as iGaming, FinTech, or E-commerceFamiliarity with database technologies (SQL, ClickHouse, Redis, document databases) and message brokers (RabbitMQ, Kafka)Exposure to MLOps, LLM monitoring tools (e.g., LangSmith, Phoenix), and CI/CD for automation workflows