YouTube.ai
Video-intelligence backend: map-reduce summarization, distributed Whisper transcription, HLS streaming.
Highlights
- ▸ Outbox Pattern for exactly-once Kafka publishing
- ▸ Map-reduce summarization of long transcripts
- ▸ Bloom Filters for transcript dedup
- ▸ HLS streaming with S3 multipart
Event-driven backend for video intelligence — distributed architecture across NestJS and FastAPI.
Summarization pipeline: Long transcripts fed into a map-reduce layer that chunks and summarizes with GPT-3.5-turbo, then merges partial summaries into a coherent final output — handles transcripts of arbitrary length without hitting context limits.
Reliability: Kafka event publishing uses the Outbox Pattern for exactly-once delivery guarantees. Bloom filters deduplicate transcript processing so distributed workers never double-process the same input.
Transcription: Distributed Whisper inference via Celery workers — horizontal scaling by adding workers, Redis as the broker.
Streaming: Video delivery via HLS with S3 multipart uploads, enabling progressive playback from the first segment.