How Netflix built its in-house LLM serving infrastructure

Netflix engineers detail their custom AI infrastructure for serving Large Language Models. By building a proprietary serving stack, they optimize extreme throughput requirements, strict latency constraints, and inference costs without relying on third-party APIs....