Amplifying Podcast Reach through Real-Time Stream Processing and Analytics

About Client

Our client is a startup that enables podcast creators to grow their audience through a platform where like-minded podcasters can connect.
We partnered with them to develop:

An analytics service to provide podcast analytics and attribution insights to the 12k+ podcasters using the platform.
An internal service used to maintain up-to-date metadata for a million+ podcasts.

Business Need

Challenges

Analytics & Attribution Service

Speed: For a good user experience, the listener should be routed to the actual episode with minimal latency (in milliseconds).
Scale: The real-time analytics pipeline should be able to handle millions of visits per day. During peak traffic, we need to expect a huge burst of concurrent visits.
Data authenticity: The metrics delivered by our service must be accurate for podcasters to interpret the insights
Information privacy: Any personally identifiable information must be anonymized while collecting analytics data.

Update Service

Voluminous data: Beyond a million podcasts are present on the platform and it’s expected to grow. The service should handle a surge in podcast registrations.
Precisely sizing workload: To work around cloud provider quotas and limits.
Throttling & rate-limited APIs: The I/O operations volume puts pressure on infrastructure resulting in throttling. The podcast APIs are also rate-limited.
Ingesting podcast feeds: Feed data can have outliers in the wild that have an inconsistent or invalid format etc.
Instant notifications: Any update in a registered podcast's metadata should result in a notification generated in minutes.

Challenges

Analytics & Attribution Service

Speed: For a good user experience, the listener should be routed to the actual episode with minimal latency (in milliseconds).
Scale: The real-time analytics pipeline should be able to handle millions of visits per day. When there is peak traffic, for example when a new episode of a popular podcast is released, we need to expect a huge burst of concurrent visits.
Data authenticity: The metrics delivered by our service must be accurate for podcasters to interpret the insights and understand the effectiveness of their collaborations. The listens stored must be genuine and spurious visits (e.g. bots) must be ignored.
Information privacy: It is necessary to protect user privacy by anonymizing any personally identifiable information while collecting analytics data.
Monitoring the service: An interruption of the service or pipeline could mean disruption of listens and data loss hence monitoring and alerts are required.

Update Service

Voluminous data: The number of podcasts present on the platform, inclusive of leads, is more than a million and is expected to grow further. The service should scale to handle a surge in podcast registrations.
Precisely sizing the workload: To work around cloud provider quotas and limits we need to precisely size batches to distribute between workers.
Throttling and rate-limited APIs: The service involves a lot of reads and writes which puts pressure on cloud provider infrastructure resulting in throttling. The podcast APIs used to extract metadata also have rate-limited API calls.
Ingesting podcast feeds: The feed data could have outliers in the wild that have an inconsistent format, invalid format, unknown protocols for feed URLs, etc. These cases need to be gracefully handled without interrupting the pipeline.
Instant notifications on metadata change: In case of any update in a registered podcast's metadata a notification must be generated within minutes.

Our Solution

Analytics & Attribution Services

Capturing listener visits: Our service is seamlessly integrated with podcast feeds by setting up a tracking prefix on any hosting platform. Once set up, listener visits to the podcast are captured.
Real-time streaming data ETL pipeline: Our ETL pipeline extracts, pre-processes, transforms and loads the listener visits data into a relational database. This data is now ready to be queried downstream.
Delivering analytics & insights through APIs: The metrics are delivered via our Analytics Reporting APIs.
Service monitoring and alerts: There is monitoring in place which triggers an email alert in case of infrastructure issues.
Attribution: Based on our understanding of the domain and how podcasters expand their audience through collaborations, we developed a custom attribution algorithm.

Podcast Analytics & Attribution Service Architecture

Podcast Update Service Architecture

Update service

Maintaining up-to-date data is necessary for services that maintain a list of users signed up on the platform and potential leads. Our update service does the following:

Registers and deregisters podcasts to be tracked for updates.
Periodically polls for the latest data from different data sources.
Worker scaling to handle the huge data volume.
Relevant attributes are extracted from the feed and the data is standardized for its use downstream.
In case a change is detected, a notification is published to multiple downstream subscribers (i.e. pub-sub pattern).
Exposes latest metadata to other services via APIs.

Business Impact

Reduced cost of lead generation and subsequently customer conversion.
Exceeded expected update frequency performance limits for update service.
Powering the platform with podcast and episode-level metrics which podcasters can use to improve their growth hacking strategy. These consumption analytics can guide editorial content and targeted advertising.
Data signals from the attribution algorithm enable podcasters to measure the effectiveness of collaborations in terms of newly obtained listeners.
Exceeded expected analytics service performance cap. We delivered a real-time, event-driven solution. This would mean podcast creators can view up-to-date metrics about their podcast in near real-time.