OpenAI explains a 120ms-to-10ms storage-read improvement

A database holds information and retrieves matching items. Having room for more data and handling many requests promptly are separate concerns. A library can have empty shelves yet a long queue at its only help desk.

Azure Cosmos DB expresses the processing, input/output and memory burden in request units, RU. RU/s describes capacity per second. Operations have different costs, so an RU count is not simply a request count.Request units

Shared capacity can put otherwise spare resources to work. A “noisy neighbor,” however, consumes enough resources to slow other workloads. Logical groups of data need not have separate physical processing resources.Partitioning

This fictional example uses imaginary processing units per second, not RU or real prices. Assume A, B and C consume 80, 10 and 10 units respectively.

120ms is 0.12 seconds; 10ms is 0.01 seconds. p50 is the middle value when request latencies are ordered: the median. OpenAI’s comparison across its performance work this year concerns storage reads, not the worst wait for every user or the time spent generating an AI answer.

Dividing 120 by 10 gives 12. Dividing the 110ms reduction by the original 120ms gives approximately 91.7%. These describe that metric; complete responses also include work such as authentication, data retrieval, model computation and transmission.

A cache keeps copies of frequently read information nearby, reducing trips to the underlying store. If the original changes while the copy remains, the application may show an old value. Azure’s general cache documentation describes repeated reads, staleness windows and update conditions.General cache documentation

That document’s Azure dedicated-gateway cache is not the same implementation as Habitat’s Valkey-based cache. OpenAI describes using source-data change signals to invalidate cached copies so the next read fills them with current data.

For a fictional shop renaming an item, reading a nearby copy must be paired with discarding the old copy after a change. Moving data closer also requires considering data-residency requirements and actual regional capacity.