API Integration and Data Ingestion Platform
Our client is a media analytics startup with a suite of product offering which helps growth-focused brands inform the true incremental contribution of paid media tactics. It’s approach to determining incrementality is rooted in innovative experimental design, now proven to be the most effective and accurate methodology for media mix investment decisions.
What we do
- For making informed media attribution decisions, a data ingestion platform was needed which could reliably bring in accurate incremental data across various media spend channels.
- We built an orchestration framework and an API integration platform responsible for extracting client specific data for various media spend channels such as Facebook, Google AdWords, Bing, Shopify, Magento, Snapchat etc. and a lot more
The solution was completely automated, backend-only API Integration platform, executed in an orchestrated way to cleanly bring in the required data. Building the solution required a thorough understanding of how APIs are exposed by the media channels, which APIs make most sense, how to securely consume the APIs, and finally developing the adapters to extract relevant and accurate data of our interest. Below are few other technical considerations –
- Authentication – Clearunderstanding of the API offerings and authentication security considerations(e.g. OAuth2/OAuth/password-only protocols)
- API Throttling – ImplementingAPI call metering to avoid any potential API throttling issues on the endpoint (Too many requests)
- Auth Tokens – Basedon the endpoint authentication scheme, implement a token strategy in the adapters
- Transport Protocol – Endpointspecific protocol considerations – REST, SOAP/XML, FTP, e-mail
- Data processor – Cleansing the data to bring it to a pristine level and dumping it into a DWH
- Other considerations – Endpointspecific API considerations such as permissible concurrent executions to avoid API throttling, maximum payload data to avoid API timeout issues and response pagination considerations
- Deployed on AWS via an automated CI/CD pipeline with complete build management and deploymenttriggered using Jenkins
- AWS Lambda functions and AWS ECS handles the execution of the adapter jobs at multiple scheduled times through the day