Clinical Text Search (ELK / OpenSearch)

Overview

A pile of notes nobody can search has no value. This skill stands up a production-style search index for clinical text using the ELK/OpenSearch stack, tuned for medical language (synonyms, abbreviations, negation-aware filtering) and offering both keyword (BM25) and semantic (vector kNN) retrieval. It doubles as the retrieval layer for RAG and downstream NLP.

When to Use This Skill

• Making a corpus of notes, pathology/radiology reports, or literature searchable.
• Powering retrieval for a clinical QA / RAG assistant.
• Building a faceted evidence-search UI (filter by date, specialty, code).
• Operational log/observability search (the ELK stack's home turf) for a data platform.

Design

1. Index mapping text fields with a custom analyzer; keyword fields for

facets (specialty, encounter_type, ICD-10); dense_vector for embeddings.

1. Medical analyzer lowercasing, a synonym filter (MI ↔ myocardial infarction,

HTN ↔ hypertension), abbreviation expansion, and clinical stop-words.

1. Relevance BM25 with field boosting (impression > body); optional rescore.
2. Hybrid search combine BM25 with kNN vector similarity (embed with a clinical/

sentence-transformer model) and fuse scores (RRF) for better recall on paraphrases.

1. Filtering structured facets and date ranges; respect access controls for PHI.
2. Kibana dashboards for corpus stats, top queries, and result quality monitoring.

Example

from elasticsearch import Elasticsearch
es = Elasticsearch("http://localhost:9200")

es.indices.create(index="clinical_notes", body={
  "settings": {"analysis": {
    "filter": {"med_syn": {"type": "synonym", "synonyms": [
        "mi, myocardial infarction", "htn, hypertension", "t2dm, type 2 diabetes"]}},
    "analyzer": {"clinical": {"tokenizer": "standard",
        "filter": ["lowercase", "med_syn"]}}}},
  "mappings": {"properties": {
    "note_text": {"type": "text", "analyzer": "clinical"},
    "specialty": {"type": "keyword"},
    "service_date": {"type": "date"},
    "embedding": {"type": "dense_vector", "dims": 384, "index": True, "similarity": "cosine"}}}})

# Hybrid query: BM25 + kNN
es.search(index="clinical_notes", body={
  "query": {"bool": {"must": {"match": {"note_text": "chest pain rule out MI"}},
                     "filter": {"term": {"specialty": "cardiology"}}}},
  "knn": {"field": "embedding", "query_vector": qvec, "k": 10, "num_candidates": 100}})

Evaluation

Measure retrieval with precision@k, recall@k, MRR, and nDCG against a labeled query set. A/B the synonym pack and the BM25-vs-hybrid configuration (use ab-testing-healthcare for the rollout). Monitor zero-result and long-tail queries in Kibana to grow the synonym list.

Outputs

• A configured clinical_notes index + reusable mapping/analyzer JSON.
• synonyms.txt curated medical synonym/abbreviation pack.
• retrieval_eval.md P@k / nDCG by configuration.
• Kibana dashboard export for corpus and query monitoring.

Healthcare Context

Tuned for clinical vocabulary and the operational reality of PHI access control and audit logging. Retrieval feeds clinical-text-summarization and RAG assistants; the same stack covers data-platform log search. Use OpenSearch as the Apache-2.0 drop-in where ELK licensing matters.

References

• Elasticsearch / OpenSearch docs https://www.elastic.co/guide / https://opensearch.org/docs
• Robertson & Zaragoza (2009), BM25 and Beyond.
• Reciprocal Rank Fusion (Cormack et al. 2009) for hybrid search.