PLATFORMS AND MANAGED SERVICES

GATEWAYS THAT ROUTE, CACHE, AND METER MODEL CALLS A gateway is a single entry point that sits between your applications and the outside services they use. A model gateway sits between your application and one or more model providers. Think of a receptionist in a company building. Every visitor and every phone call goes through the front desk, which routes each one to the right person, keeps a log, and turns away unwanted ones. A model gateway does several useful jobs. Routing: it sends each request to a chosen model or provider, perhaps a small cheap model for easy requests and a larger one for hard requests, as discussed in Chapter 54. Caching: if the same request arrives again, it returns the stored answer instead of paying for another call. Metering: it counts usage by team, project, or user, which supports the internal billing and budgeting of Chapter 53. Failover: if the primary provider has an outage, it sends traffic to a backup, a reliability technique from Chapter 56. Rate limiting: it caps how fast any one user or team can send requests, to avoid surprise bills and to share capacity fairly. Central control: one place to apply security checks, key management, and logging. Because a gateway sees every request, it is also an attractive target and a sensitive point. Secure it carefully, restrict who can change its settings, and consider what it logs. Caching deserves a note of caution. An exact-match cache returns a stored answer only when the new request is identical, which is safe but helps only when requests repeat exactly. A semantic cache returns a stored answer when the new request merely means something similar, which can save more money but can occasionally return an answer that does not quite fit the question. Never cache answers that contain one user's private data and then serve them to another user.