classifiers.redis_vector_index module

RediSearch KNN helpers for tool/skill/dangerous-command centroid embeddings.

Stores one Redis HASH per item under tool_emb:{name} / skill_emb:{id} / dangerous_cmd_emb:{category_id} / benign_tech_emb:{category_id} and queries via FT.SEARCH (see init_redis_indexes). Legacy monolithic hashes (TOOL_EMBEDDINGS_HASH_KEY, etc.) remain supported as a fallback until fully migrated.

classifiers.redis_vector_index.embedding_to_blob(vec)[source]

Serialize an embedding vector to a FLOAT32 blob for RediSearch.

Converts a numpy array or float list into the little-endian FLOAT32 byte string that RediSearch expects for both stored embedding HASH fields and the $query_vec query parameter. The dimension is validated against VECTOR_DIM up front so a misshapen vector fails loudly here rather than producing a silently corrupt index entry or query.

This is a pure transform with no side effects. It is called by every store helper in this module (store_tool_embedding_hash(), store_skill_embedding_hash(), store_dangerous_cmd_embedding_hash(), store_benign_tech_embedding_hash()) and by every KNN search helper to encode the query vector, and is exercised directly by tests/test_vector_redisearch_knn.py.

Parameters:

vec (ndarray | list[float]) – Embedding as a numpy array or a list of floats.

Returns:

The vector encoded as a contiguous little-endian FLOAT32 blob.

Return type:

bytes

Raises:

ValueError – If the vector’s element count does not equal VECTOR_DIM.

async classifiers.redis_vector_index.store_tool_embedding_hash(redis, tool_name, centroid, metadata)[source]

Upsert the per-tool RediSearch HASH holding a tool’s centroid embedding.

Writes (or overwrites) the tool_emb:{tool_name} HASH with the FLOAT32 embedding blob, the plain name field, and a JSON-encoded meta_json blob, which registers the tool as a document in the RediSearch tool index so it becomes reachable via knn_search_tools(). This is the inverse of delete_tool_embedding_hash() and the per-key successor to the legacy monolithic tool hash.

The embedding is encoded via embedding_to_blob() (validating its dimension) and the function issues a single HSET against Redis. It is called by the tool-embedding build and refresh scripts (classifiers/update_tool_embeddings.py, classifiers/refresh_tool_embeddings.py, classifiers/update_changed_tool_embeddings.py), by migrate_legacy_tool_hashes_to_redisearch() when porting legacy hashes, by the in-process rebuild path in classifiers/vector_classifier.py, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to issue the write.

  • tool_name (str) – Tool name, used both as the key suffix and the name field.

  • centroid (ndarray) – Centroid embedding for the tool’s synthetic queries.

  • metadata (dict[str, Any]) – Arbitrary tool metadata serialized into meta_json.

Return type:

None

Returns:

None.

async classifiers.redis_vector_index.delete_tool_embedding_hash(redis, tool_name)[source]

Delete the per-tool RediSearch HASH for tool_name.

Removes the tool_emb:{tool_name} key (the inverse of store_tool_embedding_hash()), which also drops the document from the RediSearch tool index so it is no longer returned by KNN queries.

This issues a single DEL against tool_emb:{tool_name} and has no other side effects. It is called by classifiers/update_tool_embeddings.py when pruning orphaned tool entries during an embedding rebuild (alongside HDEL on the legacy monolithic hashes), and by tests/test_vector_redisearch_knn.py for fixture cleanup.

Parameters:

  • redis (Redis) – Async Redis client used to issue the deletion.

  • tool_name (str) – Tool name whose tool_emb: HASH should be removed.

Return type:

None

async classifiers.redis_vector_index.store_skill_embedding_hash(redis, skill_id, vec, meta)[source]

Upsert the per-skill RediSearch HASH holding a skill’s embedding.

Writes (or overwrites) the skill_emb:{skill_id} HASH with the FLOAT32 embedding blob plus searchable skill_id, name, description, and JSON meta_json fields (the name and description falling back to the skill id and empty string when absent from meta). This registers the skill as a document in the RediSearch skill index so it is returned by knn_search_skills(), and is the inverse of delete_skill_embedding_hash().

The embedding is encoded via embedding_to_blob() and the function issues a single HSET against Redis. It is called by classifiers/update_skill_embeddings.py during a skill-embedding rebuild, by migrate_legacy_skill_hashes_to_redisearch() when porting legacy hashes, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to issue the write.

  • skill_id (str) – Skill identifier, used as the key suffix and skill_id field.

  • vec (ndarray) – Embedding vector for the skill.

  • meta (dict[str, Any]) – Skill metadata; its name and description entries populate the corresponding fields and the whole dict is serialized into meta_json.

Return type:

None

Returns:

None.

async classifiers.redis_vector_index.delete_skill_embedding_hash(redis, skill_id)[source]

Delete the per-skill RediSearch HASH for skill_id.

Removes the skill_emb:{skill_id} key (the inverse of store_skill_embedding_hash()), dropping the document from the RediSearch skill index so it is excluded from subsequent KNN queries.

This issues a single DEL against skill_emb:{skill_id} with no other side effects. It is called by classifiers/update_skill_embeddings.py when pruning orphaned skill embeddings during a rebuild (alongside HDEL on the legacy monolithic hashes), and by tests/test_vector_redisearch_knn.py for fixture cleanup.

Parameters:

  • redis (Redis) – Async Redis client used to issue the deletion.

  • skill_id (str) – Skill identifier whose skill_emb: HASH should be removed.

Return type:

None

async classifiers.redis_vector_index.store_dangerous_cmd_embedding_hash(redis, category_id, centroid, metadata)[source]

Upsert the per-category dangerous-command RediSearch HASH.

Writes (or overwrites) the dangerous_cmd_emb:{category_id} HASH with the FLOAT32 embedding blob, the category_id field, and a JSON meta_json blob, registering the category as a document in the dangerous-command index so the guard can match against it via knn_search_dangerous_cmds(). This is the inverse of delete_dangerous_cmd_embedding_hash().

The centroid is encoded via embedding_to_blob() and the function issues a single HSET against Redis. It is called by classifiers/update_dangerous_command_embeddings.py when (re)building the dangerous-command corpus, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to issue the write.

  • category_id (str) – Dangerous-command category id, used as key suffix and field.

  • centroid (ndarray) – Centroid embedding for the category’s example commands.

  • metadata (dict[str, Any]) – Category metadata serialized into meta_json.

Return type:

None

Returns:

None.

async classifiers.redis_vector_index.delete_dangerous_cmd_embedding_hash(redis, category_id)[source]

Delete the per-category dangerous-command RediSearch HASH.

Removes the dangerous_cmd_emb:{category_id} key (the inverse of store_dangerous_cmd_embedding_hash()), dropping the document from the dangerous-command index so it no longer participates in the guard’s KNN matching.

This issues a single DEL against dangerous_cmd_emb:{category_id} with no other side effects. It is called by classifiers/update_dangerous_command_embeddings.py when pruning orphaned categories during a corpus rebuild, and by tests/test_vector_redisearch_knn.py for fixture cleanup.

Parameters:

  • redis (Redis) – Async Redis client used to issue the deletion.

  • category_id (str) – Dangerous-command category id whose dangerous_cmd_emb: HASH should be removed.

Return type:

None

async classifiers.redis_vector_index.store_benign_tech_embedding_hash(redis, category_id, centroid, metadata)[source]

Upsert the per-category benign-technical RediSearch HASH.

Writes (or overwrites) the benign_tech_emb:{category_id} HASH with the FLOAT32 embedding blob, the category_id field, and a JSON meta_json blob, registering the category as a document in the benign-technical index so the guard can match against it via knn_search_benign_tech() (the “looks dangerous but is actually benign” allow-list counterpart to the dangerous-command index). This is the inverse of delete_benign_tech_embedding_hash().

The centroid is encoded via embedding_to_blob() and the function issues a single HSET against Redis. It is called by classifiers/update_benign_technical_embeddings.py when (re)building the benign-technical corpus, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to issue the write.

  • category_id (str) – Benign-technical category id, used as key suffix and field.

  • centroid (ndarray) – Centroid embedding for the category’s example phrases.

  • metadata (dict[str, Any]) – Category metadata serialized into meta_json.

Return type:

None

Returns:

None.

async classifiers.redis_vector_index.delete_benign_tech_embedding_hash(redis, category_id)[source]

Delete the per-category benign-technical RediSearch HASH.

Removes the benign_tech_emb:{category_id} key (the inverse of store_benign_tech_embedding_hash()), dropping the document from the benign-technical index so it no longer participates in KNN matching.

This issues a single DEL against benign_tech_emb:{category_id} with no other side effects. It is called by classifiers/update_benign_technical_embeddings.py when pruning orphaned categories during a rebuild, and by tests/test_vector_redisearch_knn.py for fixture cleanup.

Parameters:

  • redis (Redis) – Async Redis client used to issue the deletion.

  • category_id (str) – Benign-technical category id whose benign_tech_emb: HASH should be removed.

Return type:

None

async classifiers.redis_vector_index.knn_search_tools(redis, query_embedding, *, knn_k, ef_runtime=200)[source]

Find the tools whose centroid embeddings are nearest a query vector.

Runs an approximate FT.SEARCH KNN query over the RediSearch tool index for the knn_k nearest neighbours of query_embedding, converting each match’s cosine distance into a cosine similarity (1 - distance) and parsing the stored meta_json back into a dict. Results come out sorted by ascending distance, i.e. descending similarity. Any RediSearch error is swallowed (logged at debug) and yields an empty list so the classifier degrades gracefully when the index is missing or unhealthy.

Internally it encodes the query via embedding_to_blob(), builds the clause with _knn_clause(), and reads result fields through _doc_str(). It is called by tools/search_tools.py and by the tool vector classifier in classifiers/vector_classifier.py to route requests to candidate tools, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to run the search.

  • query_embedding (ndarray) – Query embedding to match against tool centroids.

  • knn_k (int) – Number of nearest neighbours to request and page size.

  • ef_runtime (int) – HNSW query-time exploration factor; defaults to DEFAULT_KNN_EF_RUNTIME.

Returns:

Per-match dicts with name, score (cosine similarity), and metadata (parsed meta_json), ordered most-similar first; empty on any search failure.

Return type:

list[dict[str, Any]]

async classifiers.redis_vector_index.knn_search_skills(redis, query_embedding, *, knn_k, ef_runtime=200)[source]

Find the skills whose embeddings are nearest a query vector.

Runs an approximate FT.SEARCH KNN query over the RediSearch skill index for the knn_k nearest neighbours of query_embedding, converting each match’s cosine distance into a cosine similarity (1 - distance) and parsing the stored meta_json. Each result’s name and description come from the indexed fields, falling back to meta_json (then the skill id / empty string) when those fields are blank. Results are ordered descending by similarity, and any RediSearch error is swallowed (logged at debug) and returns an empty list.

Internally it encodes the query via embedding_to_blob(), builds the clause with _knn_clause(), and reads fields through _doc_str(). It is called by the skill vector classifier in classifiers/vector_classifier.py to surface candidate skills, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to run the search.

  • query_embedding (ndarray) – Query embedding to match against skill embeddings.

  • knn_k (int) – Number of nearest neighbours to request and page size.

  • ef_runtime (int) – HNSW query-time exploration factor; defaults to DEFAULT_KNN_EF_RUNTIME.

Returns:

Per-match dicts with skill_id, name, description, score (cosine similarity), and metadata, ordered most-similar first; empty on any search failure.

Return type:

list[dict[str, Any]]

async classifiers.redis_vector_index.knn_search_dangerous_cmds(redis, query_embedding, *, knn_k, ef_runtime=200)[source]

Find the dangerous-command categories nearest a query vector.

Runs an approximate FT.SEARCH KNN query over the dangerous-command index for the knn_k nearest neighbours of query_embedding, converting each match’s cosine distance into a cosine similarity (1 - distance) and parsing the stored meta_json. Results are ordered descending by similarity, and any RediSearch error is swallowed (logged at debug) and returns an empty list so a missing index fails open rather than crashing the guard.

Internally it encodes the query via embedding_to_blob(), builds the clause with _knn_clause(), and reads fields through _doc_str(). It is called by classifiers/dangerous_command_guard.py to decide whether a candidate command resembles a known dangerous category, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to run the search.

  • query_embedding (ndarray) – Query embedding for the command being screened.

  • knn_k (int) – Number of nearest neighbours to request and page size.

  • ef_runtime (int) – HNSW query-time exploration factor; defaults to DEFAULT_KNN_EF_RUNTIME.

Returns:

Per-match dicts with category_id, score (cosine similarity), and metadata, ordered most-similar first; empty on any search failure.

Return type:

list[dict[str, Any]]

async classifiers.redis_vector_index.knn_search_benign_tech(redis, query_embedding, *, knn_k, ef_runtime=200)[source]

Find the benign-technical categories nearest a query vector.

Runs an approximate FT.SEARCH KNN query over the benign-technical index for the knn_k nearest neighbours of query_embedding, converting each match’s cosine distance into a cosine similarity (1 - distance) and parsing the stored meta_json. This is the allow-list counterpart to knn_search_dangerous_cmds(): a strong benign match lets the guard clear a command that merely looks dangerous. Results are ordered descending by similarity, and any RediSearch error is swallowed (logged at debug) and returns an empty list.

Internally it encodes the query via embedding_to_blob(), builds the clause with _knn_clause(), and reads fields through _doc_str(). It is called by classifiers/dangerous_command_guard.py (typically with knn_k=1 for the nearest benign category) and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to run the search.

  • query_embedding (ndarray) – Query embedding for the command being screened.

  • knn_k (int) – Number of nearest neighbours to request and page size.

  • ef_runtime (int) – HNSW query-time exploration factor; defaults to DEFAULT_KNN_EF_RUNTIME.

Returns:

Per-match dicts with category_id, score (cosine similarity), and metadata, ordered most-similar first; empty on any search failure.

Return type:

list[dict[str, Any]]

async classifiers.redis_vector_index.redisearch_index_doc_count(redis, index_name)[source]

Report how many documents a RediSearch index currently holds.

Calls FT.INFO on index_name and reads back the num_docs field, coercing whatever Redis returns (bytes or str keys, string counts) into an int. It is the cheap liveness/population check the classifiers and guard run before issuing a KNN query, so they can skip the search entirely when an index is empty or absent. Every failure path – a missing index, a non-dict reply, a missing or unparseable num_docs – returns -1 rather than raising.

The only side effect is the single FT.INFO round trip. It is called by the tool and skill vector classifiers in classifiers/vector_classifier.py, by classifiers/dangerous_command_guard.py (for both the dangerous-command and benign-technical indexes), by tools/search_tools.py, and by tests/test_vector_redisearch_knn.py.

Parameters:

  • redis (Redis) – Async Redis client used to run FT.INFO.

  • index_name (str) – Name of the RediSearch index to inspect.

Returns:

The approximate indexed document count, or -1 if the index is missing or the count cannot be determined.

Return type:

int

async classifiers.redis_vector_index.scan_tool_names(redis)[source]

Enumerate every tool name that currently has a stored embedding HASH.

Iterates the keyspace with a cursor-based SCAN over the tool_emb:* pattern (in batches of 500), strips the TOOL_EMB_PREFIX off each matching key, and returns the deduplicated, sorted list of tool names. SCAN is used instead of KEYS so the walk stays non-blocking on a large keyspace. The vector classifier uses this list to know which tools exist so it can expand tool-name prefixes when routing.

The only side effect is the sequence of SCAN calls against Redis. It is called by the tool vector classifier in classifiers/vector_classifier.py to populate its cached tool-name list.

Parameters:

redis (Redis) – Async Redis client used to scan the keyspace.

Returns:

Sorted, de-duplicated tool names derived from the tool_emb: HASH keys.

Return type:

list[str]

async classifiers.redis_vector_index.scan_dangerous_cmd_category_ids(redis)[source]

Enumerate every dangerous-command category id with a stored embedding.

Iterates the keyspace with a cursor-based SCAN over the dangerous_cmd_emb:* pattern (in batches of 500), strips the DANGEROUS_CMD_EMB_PREFIX off each matching key, and returns the deduplicated, sorted category ids. SCAN keeps the walk non-blocking on a large keyspace.

The only side effect is the sequence of SCAN calls against Redis. It is called by classifiers/update_dangerous_command_embeddings.py to learn which categories already exist so it can prune ones that have been removed from the source corpus.

Parameters:

redis (Redis) – Async Redis client used to scan the keyspace.

Returns:

Sorted, de-duplicated dangerous-command category ids derived from the dangerous_cmd_emb: HASH keys.

Return type:

list[str]

async classifiers.redis_vector_index.scan_benign_tech_category_ids(redis)[source]

Enumerate every benign-technical category id with a stored embedding.

Iterates the keyspace with a cursor-based SCAN over the benign_tech_emb:* pattern (in batches of 500), strips the BENIGN_TECH_EMB_PREFIX off each matching key, and returns the deduplicated, sorted category ids. SCAN keeps the walk non-blocking on a large keyspace.

The only side effect is the sequence of SCAN calls against Redis. It is called by classifiers/update_benign_technical_embeddings.py to learn which categories already exist so it can prune ones that have been removed from the source corpus.

Parameters:

redis (Redis) – Async Redis client used to scan the keyspace.

Returns:

Sorted, de-duplicated benign-technical category ids derived from the benign_tech_emb: HASH keys.

Return type:

list[str]

async classifiers.redis_vector_index.migrate_legacy_tool_hashes_to_redisearch(redis, *, embeddings_key, metadata_key)[source]

Backfill per-tool RediSearch HASHs from the legacy monolithic tool hashes.

Reads the two legacy monolithic hashes – embeddings_key (field per tool -> JSON vector) and metadata_key (field per tool -> JSON metadata) – and, for each tool, decodes its vector into a FLOAT32 numpy array and writes a per-key tool_emb: HASH via store_tool_embedding_hash(), which is what migrates the data into the new RediSearch-indexed layout. Tools whose vector JSON fails to parse are skipped; metadata lookups tolerate both bytes and str field keys and fall back to {"name": name} when absent or unparseable.

Side effects are the two HGETALL reads plus one HSET per migrated tool, and an info-level log of the count. It is called by the one-shot migration script classifiers/migrate_embeddings_redisearch.py.

Parameters:

  • redis (Redis) – Async Redis client used for the reads and writes.

  • embeddings_key (str) – Legacy monolithic hash mapping tool name to JSON vector.

  • metadata_key (str) – Legacy monolithic hash mapping tool name to JSON metadata.

Returns:

The number of per-tool HASHs written (0 if the legacy embeddings hash is empty or missing).

Return type:

int

async classifiers.redis_vector_index.migrate_legacy_skill_hashes_to_redisearch(redis, *, embeddings_key, metadata_key)[source]

Backfill per-skill RediSearch HASHs from the legacy monolithic skill hashes.

Reads the two legacy monolithic hashes – embeddings_key (field per skill -> JSON vector) and metadata_key (field per skill -> JSON metadata) – and, for each skill, decodes its vector into a FLOAT32 numpy array and writes a per-key skill_emb: HASH via store_skill_embedding_hash(), migrating the data into the RediSearch-indexed layout. Skills whose vector JSON fails to parse are skipped; metadata lookups tolerate both bytes and str field keys and fall back to {"skill_id": sid} when absent or unparseable. This is the skill counterpart to migrate_legacy_tool_hashes_to_redisearch().

Side effects are the two HGETALL reads plus one HSET per migrated skill, and an info-level log of the count. It is called by the one-shot migration script classifiers/migrate_embeddings_redisearch.py.

Parameters:

  • redis (Redis) – Async Redis client used for the reads and writes.

  • embeddings_key (str) – Legacy monolithic hash mapping skill id to JSON vector.

  • metadata_key (str) – Legacy monolithic hash mapping skill id to JSON metadata.

Returns:

The number of per-skill HASHs written (0 if the legacy embeddings hash is empty or missing).

Return type:

int