You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
EmbeddingSimilarity class-level embedding cache ignores model and prefix — cross-model cache poisoning #211
EmbeddingSimilarity._CACHE is a class attribute keyed only by the normalized input text (py/autoevals/string.py):
_CACHE= {} # class-level, shared across all instances
...
def_embed(self, value):
value=normalize_value(value, maybe_object=False)
withself._CACHE_LOCK:
ifvalueinself._CACHE:
returnself._CACHE[value]
result=run_cached_request(..., input=f"{self.prefix}{value}", **self.extra_args)
Two consequences:
Cross-model poisoning: two scorers configured with different embedding models share cache entries — the second scorer silently reuses vectors produced by the first scorer's model, producing wrong similarity scores with no error.
Prefix collisions: the request embeds f"{self.prefix}{value}" but the cache key is the un-prefixed value, so instances with different prefix values collide the same way.
Suggested fix: key the cache on (model, prefix, value) (or the full request), e.g. (self.extra_args.get("model"), self.prefix, value).
Found while writing the regression tests for #166 (PR #209) — the tests had to use unique input strings to avoid the embeddings API call being skipped entirely.
Activity
fix: isolate embedding cache by model and prefix (braintrustdata#213)