YouTube.ai
Video-intelligence backend: map-reduce summarization, distributed Whisper transcription, HLS streaming.
Highlights
- ▸ Outbox Pattern for exactly-once Kafka publishing
- ▸ Map-reduce summarization of long transcripts
- ▸ Bloom Filters for transcript dedup
- ▸ HLS streaming with S3 multipart
Event-driven backend for video intelligence, distributed across NestJS and FastAPI.
Summarization pipeline: Long transcripts feed a map-reduce layer that chunks and summarizes with GPT-3.5-turbo, then merges the partial summaries into one final output. It handles transcripts of any length without hitting context limits.
Reliability: Kafka event publishing uses the Outbox Pattern for exactly-once delivery guarantees. Bloom filters deduplicate transcript processing so distributed workers never double-process the same input.
Transcription: Distributed Whisper inference on Celery workers, scaled horizontally by adding workers, with Redis as the broker.
Streaming: Video delivery via HLS with S3 multipart uploads, enabling progressive playback from the first segment.