Fix explorer/ingestion UI and 3D endpoints

.gitignore

@@ -145,6 +145,8 @@ cython_debug/
 /misc/*
 /tmp/*
 /third_party/*
+/qdrant_data/
+/stress_test_report.json
 !/third_party/.placeholder
 !/third_party/p2rank_2.5
 !/checkpoints/.placeholder

ROADMAP.md

@@ -196,6 +196,12 @@ Each phase builds on the previous one, with clear deliverables and success crite
 - [ ] Search latency < 500ms for 10k vectors
 - [ ] System handles 10 concurrent users
 
+### Implementation Notes (added 2026-01-27)
+
+- `bioflow/evaluation/metrics.py` provides Recall@k, MRR@k, nDCG@k, and a cosine intra-list diversity metric.
+- `scripts/evaluate_retrieval.py` evaluates `/api/search` against a user-provided benchmark JSON.
+- `scripts/benchmark_search_api.py` benchmarks `/api/search` latency with configurable concurrency.
+
 ---
 
 ## Phase 6: Advanced Features (Future)

SCOPE_LOCK.md

@@ -0,0 +1,38 @@
+# BioFlow Scope Lock (Phase 0)
+
+**Status:** Confirmed by user  
+**Date:** 2026-01-27
+
+## Single Source of Truth Runtime
+- **Backend:** FastAPI on port `8000`
+- **Frontend:** Next.js on port `3000`
+- **Vector DB:** Qdrant on port `6333`
+- **Canonical pipeline:** Agents + Enhanced Search
+- **Legacy:** Streamlit + legacy pipeline are deprecated and must not be on the runtime path
+
+## Open‑Source Only Constraint
+- **No proprietary dependencies** (OpenAI / Azure OpenAI / InstaDeep / closed models)
+- **All referenced models must be open‑source**
+
+## OBM Role
+- OBM is the **multimodal embedding backbone only** (text / SMILES / protein)
+- OBM is **not** the generator, validator, or orchestrator
+
+## Qdrant Requirement
+Qdrant must be implemented **to the fullest extent needed** for the project, including:
+- **HNSW indexing** for scalable similarity search
+- **Payload metadata + filtering** for evidence, modality, source, organism, dates
+- **Collections** and **collection discovery** for multi‑source datasets
+- **Efficient paging/scrolling** for UI listings and explorer views
+- **Multi‑vector or named vectors** where required by modality
+- **Top‑K filtered retrieval** and **context injection** into agents
+
+## Audit Deliverable (Phase 0 Definition of Done)
+The “Full Audit” report must include:
+1. **Checklist vs requirements**
+2. **Gaps & risks**
+3. **Performance bottlenecks**
+4. **Security/compliance review**
+5. **Prioritized remediation plan**
+6. **Appendix: API + data schemas**
+

UC4_GAP_ANALYSIS.md

@@ -0,0 +1,341 @@
+# UC4 BioFlow Gap Analysis & Action Plan
+
+## Executive Summary
+
+**Stress Test Results:** 21/21 tests PASSED ✅  
+**Warnings Identified:** 6  
+**Test Duration:** 130.8 seconds
+
+All core functionality is operational. This document identifies gaps against the UC4 vision and proposes enhancements.
+
+---
+
+## 1. Current State Assessment
+
+### ✅ Working Features (Phase 1-4 Complete)
+
+| Feature | Status | Notes |
+|---------|--------|-------|
+| Text Ingestion (PubMed) | ✅ Working | 2.2s per document |
+| Molecule Ingestion (ChEMBL) | ✅ Working | 12.9s for 5 molecules |
+| Protein Ingestion (UniProt) | ✅ Working | 5.2s for 2 sequences |
+| Semantic Search | ✅ Working | 8.4s for 4 queries |
+| MMR Diversification | ✅ Working | Diversity score in response |
+| Filtered Search | ✅ Working | 5/5 filters functional |
+| Evidence Linking | ✅ Working | With warnings |
+| Molecule Generation | ✅ Working | 8.1s for 4 prompts |
+| Molecule Mutation | ✅ Working | 2.1s per batch |
+| ADMET Validation | ✅ Working | Lipinski, QED, alerts |
+| Multi-Criteria Ranking | ✅ Working | Configurable weights |
+| Full Workflow Pipeline | ✅ Working | Generate→Validate→Rank |
+| 3D Visualization Page | ✅ Working | CSS 3D transforms |
+| Workflow Builder Page | ✅ Working | Visual step cards |
+| Discovery Page | ✅ Working | Search interface |
+| Concurrent Searches | ✅ Working | 10/10 parallel |
+
+---
+
+## 2. Gaps Identified from UC4 Vision
+
+### 2.1 Critical Gaps (High Priority)
+
+#### 🔴 GAP-1: Source Metadata Consistency
+**Warning:** "No results have source metadata"  
+**Root Cause:** Ingested data doesn't always include `source` field in payload  
+**Impact:** Evidence traceability compromised
+
+**Fix Required:**
+```python
+# In enhanced_search.py - normalize source extraction
+def _extract_source(self, result):
+    payload = result.payload
+    # Try multiple source fields
+    return payload.get('source') or payload.get('database') or payload.get('origin') or 'unknown'
+```
+
+#### 🔴 GAP-2: Cross-Modal Search Returns Single Modality
+**Warning:** "single modality results" for cross-modal queries  
+**Root Cause:** Embedding space not aligned across modalities  
+**Impact:** Can't discover molecules from text queries
+
+**Fix Required:**
+1. Implement multimodal embedding alignment layer
+2. Create cross-modal projection matrix
+3. Or use unified encoder that maps all modalities to same space
+
+#### 🔴 GAP-3: Slow Batch Ingestion (0.4 items/sec)
+**Warning:** "Slow ingestion: 0.4 items/sec"  
+**Root Cause:** Sequential encoding + no batch vectorization  
+**Impact:** Cannot scale to large datasets
+
+**Fix Required:**
+```python
+# In ingest endpoint - batch processing
+async def batch_ingest(items: List[IngestRequest]):
+    # Vectorize all at once
+    embeddings = encoder.encode_batch([i.content for i in items])
+    # Batch upsert to Qdrant
+    qdrant.upsert(collection, points=points, batch_size=100)
+```
+
+#### 🔴 GAP-4: Scientific Traceability Incomplete
+**Warning:** "Only 2/5 results are traceable"  
+**Root Cause:** Evidence links not generated for all sources  
+**Impact:** Scientists can't verify claims
+
+**Fix Required:**
+1. Mandatory source field during ingestion
+2. Auto-generate evidence links for all known sources
+3. Add citation formatter
+
+---
+
+### 2.2 Moderate Gaps (Medium Priority)
+
+#### 🟡 GAP-5: Missing "Navigate Neighbors" Feature
+**UC4 Requirement:** "Guided exploration—navigate neighbors"  
+**Status:** Not implemented  
+**Description:** Ability to explore similar items from any result
+
+**Implementation Plan:**
+```
+POST /api/search/neighbors
+{
+  "point_id": "abc123",
+  "top_k": 10,
+  "exclude_self": true
+}
+```
+
+#### 🟡 GAP-6: No Faceted Search
+**UC4 Requirement:** "Facets and filtering"  
+**Status:** Basic filters only  
+**Missing:** Dynamic facet counts, aggregations
+
+**Implementation Plan:**
+```
+GET /api/search/facets?query=kinase
+Response: {
+  "modality": {"text": 45, "molecule": 23, "protein": 12},
+  "source": {"pubmed": 40, "chembl": 30, "uniprot": 10},
+  "organism": {"human": 60, "mouse": 20}
+}
+```
+
+#### 🟡 GAP-7: No Image Modality Support
+**UC4 Requirement:** "Multimodal: text, sequences, structures, images, measurements"  
+**Status:** Missing images and measurements  
+**Impact:** Can't process microscopy, gel images
+
+**Implementation Plan:**
+1. Add CLIP/BiomedCLIP encoder for images
+2. Create image ingestion endpoint
+3. Implement image-to-molecule similarity
+
+#### 🟡 GAP-8: No Structure Similarity (3D)
+**UC4 Requirement:** "Structure similarity"  
+**Status:** SMILES/fingerprint only  
+**Impact:** Can't find 3D conformer matches
+
+**Implementation Plan:**
+1. Integrate Open Babel for 3D generation
+2. Add 3D fingerprints (USRCAT, E3FP)
+3. Implement structure alignment scoring
+
+---
+
+### 2.3 Enhancement Opportunities (Low Priority)
+
+#### 🟢 ENH-1: Result Diversity Metrics
+Add quantitative diversity score to all search results.
+
+#### 🟢 ENH-2: Feedback Learning Loop
+Implement user feedback collection for ranking refinement.
+
+#### 🟢 ENH-3: Export to Common Formats
+- SDF for molecules
+- FASTA for proteins
+- RIS for citations
+
+#### 🟢 ENH-4: Workflow Templates
+Pre-built workflows for common discovery patterns.
+
+#### 🟢 ENH-5: Batch Validation API
+Validate 100s of molecules in single request.
+
+#### 🟢 ENH-6: Protein Structure Prediction
+Integrate ESMFold for structure predictions.
+
+#### 🟢 ENH-7: Real-Time Notifications
+WebSocket updates for long-running workflows.
+
+#### 🟢 ENH-8: Collaboration Features
+Shared workspaces, annotations, discussions.
+
+---
+
+## 3. Technical Bottlenecks
+
+### ⚡ BOTTLENECK-1: Encoding Latency
+**Current:** ~2s per encoding operation  
+**Target:** <100ms  
+**Cause:** Loading models on each request
+
+**Solution:**
+- Pre-load models at startup
+- Use model caching
+- Consider ONNX optimization
+
+### ⚡ BOTTLENECK-2: Sequential Pipeline Steps
+**Current:** Generate→Validate→Rank runs sequentially  
+**Target:** Parallel where possible
+
+**Solution:**
+```python
+# Parallel validation
+async def validate_batch(smiles_list):
+    tasks = [validate_single(s) for s in smiles_list]
+    return await asyncio.gather(*tasks)
+```
+
+### ⚡ BOTTLENECK-3: Memory Usage with Large Collections
+**Current:** Full PCA on all points  
+**Risk:** OOM with 1M+ vectors
+
+**Solution:**
+- Incremental PCA
+- Sample-based visualization
+- Pagination for large results
+
+### ⚡ BOTTLENECK-4: No GPU Utilization Check
+**Current:** Assumes CPU  
+**Impact:** Slow encoding
+
+**Solution:**
+```python
+import torch
+device = "cuda" if torch.cuda.is_available() else "cpu"
+model = model.to(device)
+```
+
+---
+
+## 4. Action Plan
+
+### Phase 5A: Quick Wins (1-2 days)
+
+| Task | Priority | Effort | Impact |
+|------|----------|--------|--------|
+| Fix source metadata extraction | 🔴 High | 2h | Traceability |
+| Add batch ingestion endpoint | 🔴 High | 4h | Performance |
+| Implement neighbors endpoint | 🟡 Medium | 3h | Exploration |
+| Pre-load encoders at startup | 🟡 Medium | 2h | Latency |
+| Add faceted search | 🟡 Medium | 4h | UX |
+
+### Phase 5B: Cross-Modal Alignment (3-5 days)
+
+| Task | Priority | Effort | Impact |
+|------|----------|--------|--------|
+| Research alignment methods | 🔴 High | 4h | Architecture |
+| Implement projection layer | 🔴 High | 8h | Core feature |
+| Test cross-modal retrieval | 🔴 High | 4h | Validation |
+| Add unified embedding space | 🔴 High | 8h | UC4 compliance |
+
+### Phase 5C: New Modalities (5-7 days)
+
+| Task | Priority | Effort | Impact |
+|------|----------|--------|--------|
+| Add BiomedCLIP encoder | 🟡 Medium | 8h | Images |
+| Image ingestion API | 🟡 Medium | 4h | API |
+| 3D structure support | 🟡 Medium | 8h | Molecules |
+| Measurement data support | 🟢 Low | 6h | Assays |
+
+### Phase 5D: Production Hardening (3-5 days)
+
+| Task | Priority | Effort | Impact |
+|------|----------|--------|--------|
+| Add GPU detection | 🟡 Medium | 2h | Performance |
+| Implement caching layer | 🟡 Medium | 6h | Latency |
+| Add rate limiting | 🟡 Medium | 3h | Stability |
+| Monitoring & alerts | 🟢 Low | 4h | Ops |
+| Load testing | 🟢 Low | 4h | Validation |
+
+---
+
+## 5. Recommended Next Steps
+
+### Immediate (Today)
+
+1. **Fix source metadata** - 2h
+   - Update `enhanced_search.py` to normalize source extraction
+   - Add fallback chain for source field
+
+2. **Add batch ingestion** - 4h
+   - Create `POST /api/ingest/batch` endpoint
+   - Implement parallel encoding
+
+3. **Pre-load models** - 2h
+   - Move encoder initialization to app startup
+   - Add warmup request
+
+### This Week
+
+4. **Implement faceted search** - 4h
+5. **Add neighbors endpoint** - 3h
+6. **Research cross-modal alignment** - 4h
+
+### Next Week
+
+7. **Implement unified embedding space** - 16h
+8. **Add image modality** - 12h
+9. **Performance optimization** - 8h
+
+---
+
+## 6. Success Metrics
+
+| Metric | Current | Target |
+|--------|---------|--------|
+| Test Pass Rate | 100% | 100% |
+| Warning Count | 6 | 0 |
+| Ingestion Speed | 0.4/sec | 10/sec |
+| Search Latency | 2s | <500ms |
+| Cross-Modal Recall | ~0% | >50% |
+| Traceable Results | 40% | 100% |
+| Supported Modalities | 3 | 5+ |
+
+---
+
+## 7. Risk Assessment
+
+| Risk | Probability | Impact | Mitigation |
+|------|-------------|--------|------------|
+| Cross-modal alignment fails | Medium | High | Use separate collections per modality |
+| Memory issues at scale | Medium | Medium | Implement streaming/pagination |
+| Model loading too slow | Low | Medium | Use model registry with lazy loading |
+| Qdrant performance | Low | Medium | Consider sharding for large datasets |
+
+---
+
+## Appendix: Test Report Summary
+
+```
+📊 STRESS TEST REPORT
+======================================================================
+By Category:
+  ✅ ingestion: 3/3 passed, 0 warnings
+  ✅ search: 4/4 passed, 2 warnings
+  ✅ agents: 5/5 passed, 0 warnings
+  ✅ ui: 3/3 passed, 0 warnings
+  ✅ stress: 2/2 passed, 1 warnings
+  ✅ uc4: 4/4 passed, 3 warnings
+
+Total: 21/21 tests passed
+Warnings: 6
+Duration: 130.8s
+```
+
+---
+
+*Document generated: 2025-01-XX*  
+*BioFlow UC4 Evaluation v1.0*

bioflow/agents/workflow.py

@@ -482,12 +482,65 @@ class DiscoveryWorkflow:
         enriched = []
         for r in ranked[:self.top_k]:
             smiles = r.get("smiles")
+            candidate_ctx = next(
+                (c for c in result.context.candidates if c.get("smiles") == smiles),
+                {}
+            )
+
+            validation_ui = self._validation_to_ui(candidate_ctx.get("validation", {}))
+            name = (
+                candidate_ctx.get("name")
+                or candidate_ctx.get("title")
+                or candidate_ctx.get("label")
+                or r.get("name")
+                or f"Candidate {r.get('rank') or 0}"
+            )
+            score = r.get("final_score", r.get("score", 0.0))
             enriched.append({
                 **r,
-                "generation": next(
-                    (c for c in result.context.candidates if c.get("smiles") == smiles),
-                    {}
-                ),
+                # UI-friendly fields
+                "name": name,
+                "score": score,
+                "validation": validation_ui,
+                # Keep full context for debugging / extended UI panels
+                "generation": candidate_ctx,
             })
         
         return enriched
+
+    def _validation_to_ui(self, validation: Any) -> Dict[str, Any]:
+        """
+        Normalize validator output to the UI-friendly shape:
+        { is_valid: bool, checks: {name: bool}, properties: {name: number} }.
+        """
+        if not isinstance(validation, dict):
+            return {"is_valid": True, "checks": {}, "properties": {}}
+
+        # Determine validity
+        if "is_valid" in validation:
+            is_valid = bool(validation.get("is_valid"))
+        else:
+            status = str(validation.get("status", "passed")).lower()
+            is_valid = status in ("passed", "ok", "success", "true")
+
+        checks: Dict[str, bool] = {}
+        properties: Dict[str, float] = {}
+
+        props = validation.get("properties", [])
+        if isinstance(props, list):
+            for p in props:
+                if not isinstance(p, dict):
+                    continue
+                name = str(p.get("name") or "").strip()
+                if not name:
+                    continue
+                checks[name] = bool(p.get("passed", True))
+                value = p.get("value")
+                if isinstance(value, (int, float)):
+                    properties[name] = float(value)
+
+        alerts = validation.get("alerts", [])
+        if isinstance(alerts, list):
+            checks["no_alerts"] = len(alerts) == 0
+
+        return {"is_valid": is_valid, "checks": checks, "properties": properties}

bioflow/api/qdrant_service.py

@@ -38,6 +38,7 @@ class SearchResult:
     content: str
     modality: str
     metadata: Dict[str, Any] = field(default_factory=dict)
+    vector: Optional[List[float]] = None
 
 
 @dataclass
@@ -106,6 +107,13 @@ class QdrantService:
         self.url = url or os.getenv("QDRANT_URL")
         self.path = path or os.getenv("QDRANT_PATH", "./qdrant_data")
         self.vector_dim = vector_dim
+        self.hnsw_m = int(os.getenv("QDRANT_HNSW_M", "16"))
+        self.hnsw_ef_construct = int(os.getenv("QDRANT_HNSW_EF_CONSTRUCT", "128"))
+        self.hnsw_ef = int(os.getenv("QDRANT_HNSW_EF", "128"))
+        self.optimizers_memmap_threshold = int(os.getenv("QDRANT_OPTIMIZERS_MEMMAP_THRESHOLD", "20000"))
+        self.optimizers_indexing_threshold = int(os.getenv("QDRANT_OPTIMIZERS_INDEXING_THRESHOLD", "20000"))
+        self.optimizers_flush_interval_sec = int(os.getenv("QDRANT_OPTIMIZERS_FLUSH_INTERVAL", "5"))
+        self.on_disk_payload = os.getenv("QDRANT_ON_DISK_PAYLOAD", "false").lower() in ("1", "true", "yes")
         
         self._client = None
         self._initialized_collections: set = set()
@@ -140,6 +148,13 @@ class QdrantService:
             else:
                 self._client = QdrantClient(path=self.path)
                 logger.info(f"Using local Qdrant at {self.path}")
+
+            # Populate known collections so searches work after process restart.
+            try:
+                collections = self._client.get_collections().collections
+                self._initialized_collections = {c.name for c in collections}
+            except Exception as e:
+                logger.warning(f"Failed to pre-load collections list: {e}")
             
             return self._client
         except Exception as e:
@@ -148,6 +163,7 @@ class QdrantService:
     def _ensure_collection(self, collection: str):
         """Ensure collection exists."""
         if collection in self._initialized_collections:
+            self._ensure_payload_indexes(collection)
             return
         
         client = self._get_client()
@@ -159,18 +175,83 @@ class QdrantService:
             exists = any(c.name == collection for c in collections)
             
             if not exists:
-                client.create_collection(
-                    collection_name=collection,
-                    vectors_config=VectorParams(
+                hnsw_config = None
+                optimizers_config = None
+                try:
+                    from qdrant_client.models import HnswConfigDiff, OptimizersConfigDiff
+                    hnsw_config = HnswConfigDiff(
+                        m=self.hnsw_m,
+                        ef_construct=self.hnsw_ef_construct,
+                    )
+                    optimizers_config = OptimizersConfigDiff(
+                        memmap_threshold=self.optimizers_memmap_threshold,
+                        indexing_threshold=self.optimizers_indexing_threshold,
+                        flush_interval_sec=self.optimizers_flush_interval_sec,
+                    )
+                except Exception as e:
+                    logger.warning(f"Qdrant advanced configs unavailable: {e}")
+
+                create_kwargs = {
+                    "collection_name": collection,
+                    "vectors_config": VectorParams(
                         size=self.vector_dim,
-                        distance=Distance.COSINE
+                        distance=Distance.COSINE,
+                    ),
+                }
+                if hnsw_config is not None:
+                    create_kwargs["hnsw_config"] = hnsw_config
+                if optimizers_config is not None:
+                    create_kwargs["optimizers_config"] = optimizers_config
+                if self.on_disk_payload:
+                    create_kwargs["on_disk_payload"] = True
+
+                try:
+                    client.create_collection(**create_kwargs)
+                except TypeError:
+                    client.create_collection(
+                        collection_name=collection,
+                        vectors_config=VectorParams(
+                            size=self.vector_dim,
+                            distance=Distance.COSINE,
+                        ),
                     )
-                )
                 logger.info(f"Created collection: {collection}")
+            self._ensure_payload_indexes(collection)
         except Exception as e:
             raise QdrantServiceError(f"Failed to ensure collection {collection}: {e}")
         
         self._initialized_collections.add(collection)
+
+    def _ensure_payload_indexes(self, collection: str) -> None:
+        """Ensure payload indexes exist for common filter fields."""
+        client = self._get_client()
+
+        try:
+            from qdrant_client.models import PayloadSchemaType
+        except Exception as e:
+            logger.warning(f"Payload indexes unavailable: {e}")
+            return
+
+        index_specs = [
+            ("source", PayloadSchemaType.KEYWORD),
+            ("modality", PayloadSchemaType.KEYWORD),
+            ("organism", PayloadSchemaType.KEYWORD),
+            ("organism_id", PayloadSchemaType.KEYWORD),
+            ("year", PayloadSchemaType.INTEGER),
+            ("pmid", PayloadSchemaType.KEYWORD),
+            ("chembl_id", PayloadSchemaType.KEYWORD),
+            ("accession", PayloadSchemaType.KEYWORD),
+        ]
+
+        for field_name, field_schema in index_specs:
+            try:
+                client.create_payload_index(
+                    collection_name=collection,
+                    field_name=field_name,
+                    field_schema=field_schema,
+                )
+            except Exception as e:
+                logger.debug(f"Payload index {field_name} not created: {e}")
     
     def _get_embedding(self, content: str, modality: str) -> List[float]:
         """Get embedding for content."""
@@ -228,14 +309,17 @@ class QdrantService:
         
         # Generate ID
         point_id = id or str(uuid.uuid4())
-        
+
         # Get embedding - will raise if fails
         vector = self._get_embedding(content, modality)
-        
+
+        # Normalize modality for downstream UI consistency
+        stored_modality = "molecule" if modality in ("smiles", "molecule") else modality
+
         # Prepare payload
         payload = {
             "content": content,
-            "modality": modality,
+            "modality": stored_modality,
             **(metadata or {})
         }
         
@@ -299,7 +383,8 @@ class QdrantService:
         modality: str = "text",
         collection: str = None,
         limit: int = 10,
-        filter_modality: str = None
+        filter_modality: str = None,
+        with_vectors: bool = False,
     ) -> List[SearchResult]:
         """
         Semantic search across vectors.
@@ -324,14 +409,26 @@ class QdrantService:
         if collection:
             collections = [collection]
         else:
-            collections = [c.value for c in CollectionType]
+            # Search across all existing collections (not just the enum defaults).
+            try:
+                collections = self.list_collections()
+            except Exception:
+                collections = [c.value for c in CollectionType]
         
         client = self._get_client()
         all_results = []
+        search_params = None
+        try:
+            from qdrant_client.models import SearchParams
+            search_params = SearchParams(hnsw_ef=self.hnsw_ef)
+        except Exception:
+            search_params = None
         
         for coll in collections:
             try:
+                # Skip collections that don't exist (or are not accessible)
                 if coll not in self._initialized_collections:
+                    # If we populated from list_collections() this won't happen, but keep safe.
                     continue
                 
                 filter_conditions = None
@@ -345,20 +442,41 @@ class QdrantService:
                     )
                 
                 # Use query_points for newer qdrant-client versions
-                results = client.query_points(
-                    collection_name=coll,
-                    query=query_vector,
-                    limit=limit,
-                    query_filter=filter_conditions
-                ).points
+                try:
+                    results = client.query_points(
+                        collection_name=coll,
+                        query=query_vector,
+                        limit=limit,
+                        query_filter=filter_conditions,
+                        search_params=search_params,
+                        with_payload=True,
+                        with_vectors=with_vectors,
+                    ).points
+                except TypeError:
+                    results = client.query_points(
+                        collection_name=coll,
+                        query=query_vector,
+                        limit=limit,
+                        query_filter=filter_conditions,
+                        with_payload=True,
+                        with_vectors=with_vectors,
+                    ).points
                 
                 for r in results:
+                    raw_modality = r.payload.get("modality", "unknown")
+                    normalized_modality = "molecule" if raw_modality == "smiles" else raw_modality
+                    vec = None
+                    if with_vectors:
+                        vec = r.vector
+                        if isinstance(vec, dict):
+                            vec = list(vec.values())[0] if vec else None
                     all_results.append(SearchResult(
                         id=str(r.id),
                         score=r.score,
                         content=r.payload.get("content", ""),
-                        modality=r.payload.get("modality", "unknown"),
-                        metadata=r.payload
+                        modality=normalized_modality,
+                        metadata=r.payload,
+                        vector=vec,
                     ))
             except Exception as e:
                 raise QdrantServiceError(f"Search in {coll} failed: {e}")
@@ -377,9 +495,86 @@ class QdrantService:
         
         try:
             collections = client.get_collections().collections
-            return [c.name for c in collections]
+            names = [c.name for c in collections]
+            self._initialized_collections = set(names)
+            return names
         except Exception as e:
             raise QdrantServiceError(f"Failed to list collections: {e}")
+
+    def list_items(
+        self,
+        collection: str,
+        limit: int = 20,
+        offset: int = 0,
+        filter_modality: Optional[str] = None,
+    ) -> List[SearchResult]:
+        """
+        List items from a collection.
+
+        Note: Qdrant pagination uses a point-id offset rather than numeric offsets.
+        For UI compatibility, we over-fetch `offset + limit` and slice.
+        """
+        client = self._get_client()
+
+        try:
+            existing = set(self.list_collections())
+            if collection not in existing:
+                return []
+
+            scroll_limit = max(1, offset + limit)
+
+            scroll_filter = None
+            if filter_modality:
+                from qdrant_client.models import Filter, FieldCondition, MatchValue
+                if filter_modality == "molecule":
+                    # Support legacy stored value "smiles"
+                    scroll_filter = Filter(
+                        should=[
+                            FieldCondition(key="modality", match=MatchValue(value="molecule")),
+                            FieldCondition(key="modality", match=MatchValue(value="smiles")),
+                        ]
+                    )
+                else:
+                    scroll_filter = Filter(
+                        must=[FieldCondition(key="modality", match=MatchValue(value=filter_modality))]
+                    )
+
+            # qdrant-client API has used both `scroll_filter=` and `filter=` across versions.
+            try:
+                points, _next = client.scroll(
+                    collection_name=collection,
+                    scroll_filter=scroll_filter,
+                    limit=scroll_limit,
+                    with_payload=True,
+                    with_vectors=False,
+                )
+            except TypeError:
+                points, _next = client.scroll(
+                    collection_name=collection,
+                    filter=scroll_filter,
+                    limit=scroll_limit,
+                    with_payload=True,
+                    with_vectors=False,
+                )
+
+            sliced = points[offset:offset + limit]
+            results: List[SearchResult] = []
+            for p in sliced:
+                payload = p.payload or {}
+                raw_modality = payload.get("modality", "unknown")
+                normalized_modality = "molecule" if raw_modality == "smiles" else raw_modality
+                results.append(
+                    SearchResult(
+                        id=str(p.id),
+                        score=0.0,
+                        content=payload.get("content", ""),
+                        modality=normalized_modality,
+                        metadata=payload,
+                    )
+                )
+            return results
+        except Exception as e:
+            raise QdrantServiceError(f"Failed to list items in {collection}: {e}")
     
     def get_collection_stats(self, collection: str) -> Dict[str, Any]:
         """Get statistics for a collection."""

bioflow/api/server.py

@@ -9,11 +9,17 @@ import os
 import sys
 import uuid
 import logging
+import json
+import time
+import json
+import time
+import requests
 from datetime import datetime
 from typing import Any, Dict, List, Optional
 from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, HTTPException, BackgroundTasks
+from fastapi.responses import PlainTextResponse
 from fastapi.middleware.cors import CORSMiddleware
 from pydantic import BaseModel, Field
 
@@ -74,6 +80,48 @@ class IngestRequest(BaseModel):
     metadata: Optional[Dict[str, Any]] = None
 
 
+class IngestSourceRequest(BaseModel):
+    """Request to ingest from a specific source."""
+    query: str
+    limit: int = Field(default=100, ge=1, le=10000)
+    batch_size: Optional[int] = Field(default=None, ge=1, le=1000)
+    rate_limit: Optional[float] = Field(default=None, ge=0.0)
+    collection: Optional[str] = Field(default="bioflow_memory")
+    sync: bool = Field(default=False, description="Run synchronously (may block)")
+    # PubMed-specific
+    email: Optional[str] = None
+    api_key: Optional[str] = None
+    # ChEMBL-specific
+    search_mode: Optional[str] = Field(default=None, description="target | molecule")
+
+
+class BatchIngestRequest(BaseModel):
+    """Request to ingest multiple items at once."""
+    items: List[IngestRequest] = Field(..., description="List of items to ingest")
+    parallel: bool = Field(default=True, description="Process items in parallel")
+    batch_size: int = Field(default=10, ge=1, le=100, description="Batch size for parallel processing")
+
+
+class IngestAllRequest(BaseModel):
+    """Request to ingest from all sources."""
+    query: str
+    pubmed_limit: int = Field(default=100, ge=0, le=10000)
+    uniprot_limit: int = Field(default=50, ge=0, le=10000)
+    chembl_limit: int = Field(default=30, ge=0, le=10000)
+    batch_size: Optional[int] = Field(default=None, ge=1, le=1000)
+    rate_limit: Optional[float] = Field(default=None, ge=0.0)
+    collection: Optional[str] = Field(default="bioflow_memory")
+    skip_pubmed: bool = False
+    skip_uniprot: bool = False
+    skip_chembl: bool = False
+    sync: bool = Field(default=False, description="Run synchronously (may block)")
+    # PubMed-specific
+    email: Optional[str] = None
+    api_key: Optional[str] = None
+    # ChEMBL-specific
+    search_mode: Optional[str] = Field(default=None, description="target | molecule")
+
+
 class JobStatus(BaseModel):
     """Status of an async job."""
     job_id: str
@@ -95,7 +143,7 @@ class HealthResponse(BaseModel):
 class EnhancedSearchRequest(BaseModel):
     """Request for enhanced search with MMR and filters."""
     query: str = Field(..., description="Search query (text, SMILES, or protein sequence)")
-    modality: str = Field(default="text", description="Query modality: text, molecule, protein")
+    modality: str = Field(default="auto", description="Query modality: auto, text, molecule, protein")
     collection: Optional[str] = Field(default=None, description="Target collection")
     top_k: int = Field(default=20, ge=1, le=100)
     use_mmr: bool = Field(default=True, description="Apply MMR diversification")
@@ -187,6 +235,54 @@ async def health():
     )
 
 
+@app.get("/api/health/metrics")
+async def health_metrics():
+    """Detailed health metrics for Qdrant and model readiness."""
+    metrics = {
+        "status": "ok",
+        "timestamp": datetime.utcnow().isoformat(),
+        "qdrant": {"available": False, "collections": []},
+        "models": {"available": False, "device": None, "obm_loaded": False},
+    }
+
+    if qdrant_service:
+        try:
+            collections = qdrant_service.list_collections()
+            metrics["qdrant"]["available"] = True
+            metrics["qdrant"]["collections"] = collections
+            try:
+                client = qdrant_service._get_client()
+                collection_stats = {}
+                for name in collections:
+                    try:
+                        info = client.get_collection(name)
+                        collection_stats[name] = {
+                            "vectors_count": getattr(info, "vectors_count", None),
+                            "points_count": getattr(info, "points_count", None),
+                        }
+                    except Exception:
+                        collection_stats[name] = {}
+                metrics["qdrant"]["stats"] = collection_stats
+            except Exception:
+                metrics["qdrant"]["stats"] = {}
+        except Exception:
+            metrics["qdrant"]["available"] = False
+
+    if model_service:
+        try:
+            metrics["models"]["available"] = True
+            metrics["models"]["device"] = getattr(model_service, "device", None)
+            try:
+                _ = model_service.get_obm_encoder()
+                metrics["models"]["obm_loaded"] = True
+            except Exception:
+                metrics["models"]["obm_loaded"] = False
+        except Exception:
+            metrics["models"]["available"] = False
+
+    return metrics
+
+
 # ============================================================================
 # Discovery Pipeline
 # ============================================================================
@@ -329,6 +425,20 @@ def get_predictor() -> DeepPurposePredictor:
     return _dti_predictor
 
 
+def _fallback_dti_prediction(drug_smiles: str, target_sequence: str) -> Dict[str, Any]:
+    """Fallback prediction when DeepPurpose is unavailable."""
+    seed = abs(hash(drug_smiles + target_sequence)) % 1000
+    affinity = round(0.1 + (seed % 100) / 100.0, 4)
+    confidence = 0.2
+    return {
+        "drug_smiles": drug_smiles,
+        "target_sequence": target_sequence,
+        "binding_affinity": affinity,
+        "confidence": confidence,
+        "interaction_probability": min(confidence + 0.05, 1.0),
+    }
+
+
 @app.post("/api/predict")
 async def predict_dti(request: PredictRequest):
     """
@@ -354,7 +464,17 @@ async def predict_dti(request: PredictRequest):
                 **result.metadata,
             }
         }
-        
+    except ImportError:
+        fallback = _fallback_dti_prediction(request.drug_smiles, request.target_sequence)
+        return {
+            "success": True,
+            "prediction": fallback,
+            "metadata": {
+                "model": "fallback",
+                "timestamp": datetime.utcnow().isoformat(),
+                "note": "DeepPurpose not installed; using fallback prediction.",
+            },
+        }
     except Exception as e:
         logger.error(f"Prediction failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
@@ -370,6 +490,8 @@ def get_enhanced_search_service():
     global _enhanced_search_service
     if _enhanced_search_service is None:
         from bioflow.search.enhanced_search import EnhancedSearchService
+        if not model_service:
+            raise HTTPException(status_code=503, detail="Model service not available")
         encoder = model_service.get_obm_encoder()
         _enhanced_search_service = EnhancedSearchService(
             qdrant_service=qdrant_service,
@@ -378,6 +500,19 @@ def get_enhanced_search_service():
     return _enhanced_search_service
 
 
+def _log_event(event: str, request_id: str, **fields: Any) -> None:
+    payload = {
+        "event": event,
+        "request_id": request_id,
+        "timestamp": datetime.utcnow().isoformat(),
+        **fields,
+    }
+    try:
+        logger.info(json.dumps(payload, ensure_ascii=False))
+    except Exception:
+        logger.info(f"{event} {payload}")
+
+
 @app.post("/api/search")
 async def enhanced_search(request: EnhancedSearchRequest):
     """
@@ -389,6 +524,8 @@ async def enhanced_search(request: EnhancedSearchRequest):
     - Citations and source tracking
     - Filtered search by modality, source, etc.
     """
+    request_id = uuid.uuid4().hex[:12]
+    start = time.perf_counter()
     try:
         if not qdrant_service:
             raise HTTPException(status_code=503, detail="Qdrant service not available")
@@ -404,9 +541,26 @@ async def enhanced_search(request: EnhancedSearchRequest):
             filters=request.filters,
         )
         
-        return response.to_dict()
+        payload = response.to_dict()
+        _log_event(
+            "search",
+            request_id,
+            query=request.query[:200],
+            top_k=request.top_k,
+            use_mmr=request.use_mmr,
+            returned=payload.get("returned"),
+            total_found=payload.get("total_found"),
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
+        return payload
         
     except Exception as e:
+        _log_event(
+            "search_error",
+            request_id,
+            error=str(e),
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
         logger.error(f"Enhanced search failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
@@ -446,6 +600,8 @@ async def hybrid_search(request: HybridSearchRequest):
 @app.post("/api/ingest")
 async def ingest_data(request: IngestRequest):
     """Ingest data into vector database."""
+    request_id = uuid.uuid4().hex[:12]
+    start = time.perf_counter()
     try:
         if not qdrant_service:
             raise HTTPException(status_code=503, detail="Qdrant service not available")
@@ -455,6 +611,12 @@ async def ingest_data(request: IngestRequest):
             modality=request.modality,
             metadata=request.metadata
         )
+        _log_event(
+            "ingest_single",
+            request_id,
+            modality=request.modality,
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
         return {
             "success": result.success,
             "id": result.id,
@@ -464,10 +626,171 @@ async def ingest_data(request: IngestRequest):
         }
         
     except Exception as e:
+        _log_event(
+            "ingest_single_error",
+            request_id,
+            error=str(e),
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
         logger.error(f"Ingest failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 
 
+@app.post("/api/ingest/batch")
+async def ingest_batch(request: BatchIngestRequest):
+    """
+    Batch ingest multiple items for improved performance.
+    
+    Processes items in parallel (if parallel=True) to significantly
+    improve ingestion speed compared to sequential single-item ingestion.
+    """
+    import asyncio
+    import concurrent.futures
+    
+    request_id = uuid.uuid4().hex[:12]
+    start = time.perf_counter()
+    
+    if not qdrant_service:
+        raise HTTPException(status_code=503, detail="Qdrant service not available")
+    
+    items = request.items
+    if not items:
+        return {"success": True, "ingested": 0, "failed": 0, "rate_per_sec": 0}
+    
+    results = {"ingested": 0, "failed": 0, "ids": [], "errors": []}
+    
+    def ingest_one(item: IngestRequest):
+        """Ingest a single item (for parallel execution)."""
+        try:
+            result = qdrant_service.ingest(
+                content=item.content,
+                modality=item.modality,
+                metadata=item.metadata
+            )
+            return {"success": result.success, "id": result.id, "error": None}
+        except Exception as e:
+            return {"success": False, "id": None, "error": str(e)}
+    
+    try:
+        if request.parallel:
+            # Process in batches using thread pool
+            batch_size = min(request.batch_size, len(items))
+            with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as executor:
+                futures = [executor.submit(ingest_one, item) for item in items]
+                for future in concurrent.futures.as_completed(futures):
+                    res = future.result()
+                    if res["success"]:
+                        results["ingested"] += 1
+                        if res["id"]:
+                            results["ids"].append(res["id"])
+                    else:
+                        results["failed"] += 1
+                        if res["error"]:
+                            results["errors"].append(res["error"])
+        else:
+            # Sequential processing
+            for item in items:
+                res = ingest_one(item)
+                if res["success"]:
+                    results["ingested"] += 1
+                    if res["id"]:
+                        results["ids"].append(res["id"])
+                else:
+                    results["failed"] += 1
+                    if res["error"]:
+                        results["errors"].append(res["error"])
+        
+        duration_s = time.perf_counter() - start
+        rate = results["ingested"] / duration_s if duration_s > 0 else 0
+        
+        _log_event(
+            "ingest_batch",
+            request_id,
+            count=len(items),
+            ingested=results["ingested"],
+            failed=results["failed"],
+            rate_per_sec=round(rate, 2),
+            duration_ms=round(duration_s * 1000, 2),
+        )
+        
+        return {
+            "success": results["failed"] == 0,
+            "ingested": results["ingested"],
+            "failed": results["failed"],
+            "ids": results["ids"][:50],  # Limit response size
+            "rate_per_sec": round(rate, 2),
+            "duration_ms": round(duration_s * 1000, 2),
+            "errors": results["errors"][:10] if results["errors"] else None,
+        }
+        
+    except Exception as e:
+        _log_event(
+            "ingest_batch_error",
+            request_id,
+            error=str(e),
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
+        logger.error(f"Batch ingest failed: {e}")
+        raise HTTPException(status_code=500, detail=str(e))
+
+
+def _list_items_by_modality(modality: str, limit: int, offset: int):
+    """List items across all collections by modality."""
+    if not qdrant_service:
+        return [], 0
+
+    collections = []
+    try:
+        collections = qdrant_service.list_collections()
+    except Exception:
+        collections = ["molecules", "proteins", "bioflow_memory"]
+
+    items: List[Any] = []
+    target = offset + limit
+    for coll in collections:
+        if len(items) >= target:
+            break
+        try:
+            items.extend(
+                qdrant_service.list_items(
+                    collection=coll,
+                    limit=target,
+                    offset=0,
+                    filter_modality=modality,
+                )
+            )
+        except Exception:
+            continue
+
+    total = len(items)
+    return items[offset:offset + limit], total
+
+
+def _find_point_by_id(point_id: str):
+    """Find a point by ID across collections."""
+    if not qdrant_service:
+        return None, None
+    client = qdrant_service._get_client()
+    try:
+        collections = qdrant_service.list_collections()
+    except Exception:
+        collections = ["molecules", "proteins", "bioflow_memory"]
+
+    for coll in collections:
+        try:
+            points = client.retrieve(
+                collection_name=coll,
+                ids=[point_id],
+                with_payload=True,
+                with_vectors=False,
+            )
+            if points:
+                return coll, points[0]
+        except Exception:
+            continue
+    return None, None
+
+
 @app.get("/api/molecules")
 async def list_molecules(limit: int = 20, offset: int = 0):
     """List molecules in the database."""
@@ -475,23 +798,21 @@ async def list_molecules(limit: int = 20, offset: int = 0):
         raise HTTPException(status_code=503, detail="Qdrant service not available")
     
     try:
-        results = qdrant_service.search(
-            query="molecule",
-            modality="text",
-            collection="molecules",
-            limit=limit
-        )
+        results, total = _list_items_by_modality("molecule", limit, offset)
         molecules = []
         for r in results:
+            metadata = r.metadata or {}
             molecules.append({
                 "id": r.id,
                 "smiles": r.content,
-                "name": r.metadata.get("name", "Unknown"),
-                "mw": r.metadata.get("mw", 0),
+                "name": metadata.get("name", metadata.get("title", "Unknown")),
+                "pubchemCid": metadata.get("pubchem_cid", metadata.get("pubchemCid", metadata.get("cid", 0))),
+                "description": metadata.get("description", metadata.get("title", "")),
+                "mw": metadata.get("mw", metadata.get("molecular_weight", 0)),
             })
         return {
             "molecules": molecules,
-            "total": len(molecules),
+            "total": total,
             "limit": limit,
             "offset": offset,
         }
@@ -507,24 +828,27 @@ async def list_proteins(limit: int = 20, offset: int = 0):
         raise HTTPException(status_code=503, detail="Qdrant service not available")
     
     try:
-        results = qdrant_service.search(
-            query="protein kinase receptor",
-            modality="text",
-            collection="proteins",
-            limit=limit
-        )
+        results, total = _list_items_by_modality("protein", limit, offset)
         proteins = []
         for r in results:
+            metadata = r.metadata or {}
+            pdb_ids = metadata.get("pdb_ids", []) or []
+            if isinstance(pdb_ids, str):
+                pdb_ids = [pdb_ids]
+            pdb_id = metadata.get("pdb_id") or (pdb_ids[0] if pdb_ids else "")
+            name = metadata.get("name") or metadata.get("protein_name") or metadata.get("entry_name") or "Unknown"
             proteins.append({
                 "id": r.id,
                 "sequence": r.content[:50] + "..." if len(r.content) > 50 else r.content,
-                "uniprot_id": r.metadata.get("uniprot_id", ""),
-                "name": r.metadata.get("name", "Unknown"),
+                "uniprot_id": metadata.get("uniprot_id", metadata.get("accession", "")),
+                "name": name,
+                "pdbId": pdb_id,
+                "description": metadata.get("function", metadata.get("description", "")),
                 "length": len(r.content),
             })
         return {
             "proteins": proteins,
-            "total": len(proteins),
+            "total": total,
             "limit": limit,
             "offset": offset,
         }
@@ -533,52 +857,482 @@ async def list_proteins(limit: int = 20, offset: int = 0):
         raise HTTPException(status_code=500, detail=str(e))
 
 
+@app.get("/api/molecules/{molecule_id}")
+async def get_molecule(molecule_id: str):
+    """Get molecule details by ID."""
+    if not qdrant_service:
+        raise HTTPException(status_code=503, detail="Qdrant service not available")
+
+    _coll, point = _find_point_by_id(molecule_id)
+    if point is None:
+        raise HTTPException(status_code=404, detail="Molecule not found")
+
+    payload = point.payload or {}
+    smiles = payload.get("smiles") or payload.get("content", "")
+    return {
+        "id": str(point.id),
+        "name": payload.get("name", payload.get("title", "Unknown")),
+        "smiles": smiles,
+        "pubchemCid": payload.get("pubchem_cid", payload.get("pubchemCid", payload.get("cid", 0))),
+        "description": payload.get("description", payload.get("title", "")),
+    }
+
+
+@app.get("/api/molecules/{molecule_id}/sdf")
+async def get_molecule_sdf(molecule_id: str):
+    """Get molecule 3D structure as SDF."""
+    if not qdrant_service:
+        raise HTTPException(status_code=503, detail="Qdrant service not available")
+
+    _coll, point = _find_point_by_id(molecule_id)
+    if point is None:
+        raise HTTPException(status_code=404, detail="Molecule not found")
+
+    payload = point.payload or {}
+    smiles = payload.get("smiles") or payload.get("content", "")
+    if not smiles:
+        raise HTTPException(status_code=404, detail="Molecule SMILES not available")
+
+    try:
+        from rdkit import Chem
+        from rdkit.Chem import AllChem
+    except Exception:
+        raise HTTPException(status_code=503, detail="RDKit is required for 3D structures")
+
+    mol = Chem.MolFromSmiles(smiles)
+    if mol is None:
+        raise HTTPException(status_code=400, detail="Invalid SMILES")
+
+    mol = Chem.AddHs(mol)
+    try:
+        AllChem.EmbedMolecule(mol, AllChem.ETKDG())
+        AllChem.UFFOptimizeMolecule(mol)
+    except Exception:
+        pass
+    sdf = Chem.MolToMolBlock(mol)
+    return PlainTextResponse(sdf, media_type="chemical/x-mdl-sdfile")
+
+
+@app.get("/api/proteins/{protein_id}")
+async def get_protein(protein_id: str):
+    """Get protein details by ID."""
+    if not qdrant_service:
+        raise HTTPException(status_code=503, detail="Qdrant service not available")
+
+    _coll, point = _find_point_by_id(protein_id)
+    if point is None:
+        raise HTTPException(status_code=404, detail="Protein not found")
+
+    payload = point.payload or {}
+    pdb_ids = payload.get("pdb_ids", []) or []
+    if isinstance(pdb_ids, str):
+        pdb_ids = [pdb_ids]
+    pdb_id = payload.get("pdb_id") or (pdb_ids[0] if pdb_ids else "")
+    name = payload.get("name") or payload.get("protein_name") or payload.get("entry_name") or "Unknown"
+
+    return {
+        "id": str(point.id),
+        "pdbId": pdb_id,
+        "name": name,
+        "description": payload.get("function", payload.get("description", "")),
+    }
+
+
+@app.get("/api/proteins/{protein_id}/pdb")
+async def get_protein_pdb(protein_id: str):
+    """Get protein structure as PDB text."""
+    if not qdrant_service:
+        raise HTTPException(status_code=503, detail="Qdrant service not available")
+
+    _coll, point = _find_point_by_id(protein_id)
+    if point is None:
+        raise HTTPException(status_code=404, detail="Protein not found")
+
+    payload = point.payload or {}
+    pdb_ids = payload.get("pdb_ids", []) or []
+    if isinstance(pdb_ids, str):
+        pdb_ids = [pdb_ids]
+    pdb_id = payload.get("pdb_id") or (pdb_ids[0] if pdb_ids else "")
+    if not pdb_id:
+        raise HTTPException(status_code=404, detail="No PDB ID available for this protein")
+
+    try:
+        pdb_url = f"https://files.rcsb.org/download/{pdb_id}.pdb"
+        resp = requests.get(pdb_url, timeout=20)
+        if resp.status_code != 200:
+            raise HTTPException(status_code=404, detail="PDB file not found")
+        return PlainTextResponse(resp.text, media_type="chemical/x-pdb")
+    except HTTPException:
+        raise
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+
 # ============================================================================
 # Explorer (Embeddings)
 # ============================================================================
 @app.get("/api/explorer/embeddings")
-async def get_embeddings(dataset: str = "default", method: str = "umap"):
-    """Get 2D projections of embeddings for visualization."""
+async def get_embeddings(
+    dataset: str = "default",
+    method: str = "pca",
+    query: Optional[str] = None,
+    modality: str = "auto",
+    limit: int = 100,
+):
+    """Get 3D projections of embeddings for visualization."""
     import numpy as np
-    
+
     if not qdrant_service:
         raise HTTPException(status_code=503, detail="Qdrant service not available")
-    
+
+    def _detect_modality(q: str) -> str:
+        if not q:
+            return "text"
+        seq = q.replace("\n", "").replace(" ", "")
+        if len(seq) >= 25 and all(c.upper() in "ACDEFGHIKLMNPQRSTVWYBXZJUO" for c in seq[:25]):
+            return "protein"
+        if any(c in q for c in "[]()=#@") and any(c.isalpha() for c in q):
+            return "molecule"
+        return "text"
+
+    def _cluster_for_modality(m: str) -> int:
+        if m == "molecule":
+            return 0
+        if m == "protein":
+            return 1
+        return 2
+
     points = []
-    
+    vectors = []
+
     try:
-        # Get molecules and proteins from Qdrant
-        mol_results = qdrant_service.search("", modality="text", collection="molecules", limit=50)
-        prot_results = qdrant_service.search("", modality="text", collection="proteins", limit=50)
-        
-        all_results = mol_results + prot_results
-        
-        # Simple 2D projection using content hash for deterministic positions
-        # (In production, use proper UMAP/t-SNE)
-        for i, r in enumerate(all_results):
-            np.random.seed(hash(r.content) % 2**32)
-            cluster = 0 if r.modality == "molecule" else 1
-            cx, cy = [(2, 3), (-2, -1)][cluster]
-            points.append({
-                "id": r.id,
-                "x": float(cx + np.random.randn() * 0.8),
-                "y": float(cy + np.random.randn() * 0.8),
-                "cluster": cluster,
-                "label": r.metadata.get("name", r.content[:20]),
-                "modality": r.modality,
-            })
+        if query:
+            effective_modality = modality if modality != "auto" else _detect_modality(query)
+            search_results = qdrant_service.search(
+                query=query,
+                modality=effective_modality,
+                limit=min(limit, 500),
+                with_vectors=True,
+            )
+            for r in search_results:
+                if r.vector is None:
+                    continue
+                vectors.append(r.vector)
+                points.append({
+                    "id": r.id,
+                    "label": r.metadata.get("name", r.content[:40]),
+                    "content": r.content,
+                    "modality": r.modality,
+                    "source": r.metadata.get("source", "unknown"),
+                    "score": r.score,
+                    "cluster": _cluster_for_modality(r.modality),
+                    "metadata": r.metadata,
+                })
+        else:
+            client = qdrant_service._get_client()
+            collections = []
+            try:
+                collections = qdrant_service.list_collections()
+            except Exception:
+                collections = ["molecules", "proteins", "bioflow_memory"]
+
+            per_collection = max(10, int(limit / max(1, len(collections))))
+            for coll in collections:
+                try:
+                    res, _ = client.scroll(
+                        collection_name=coll,
+                        limit=per_collection,
+                        with_payload=True,
+                        with_vectors=True,
+                    )
+                    for p in res:
+                        vec = p.vector
+                        if isinstance(vec, dict):
+                            vec = list(vec.values())[0] if vec else None
+                        if vec is None:
+                            continue
+                        payload = p.payload or {}
+                        modality_val = payload.get("modality", "text")
+                        if modality_val == "smiles":
+                            modality_val = "molecule"
+                        vectors.append(vec)
+                        points.append({
+                            "id": str(p.id),
+                            "label": payload.get("name", payload.get("content", "")[:40]),
+                            "content": payload.get("content", ""),
+                            "modality": modality_val,
+                            "source": payload.get("source", "unknown"),
+                            "score": payload.get("score", 0),
+                            "cluster": _cluster_for_modality(modality_val),
+                            "metadata": payload,
+                        })
+                except Exception as e:
+                    logger.warning(f"Failed to scroll collection {coll}: {e}")
     except Exception as e:
         logger.error(f"Failed to get embeddings from Qdrant: {e}")
         raise HTTPException(status_code=500, detail=str(e))
-    
+
+    # If we have vectors, compute projection (fallback to deterministic positions on failure)
+    coords = []
+    method_used = method.lower()
+    if vectors and len(vectors) >= 2:
+        try:
+            if method_used not in ("pca", "umap", "tsne"):
+                method_used = "pca"
+            arr = np.array(vectors, dtype=float)
+
+            if method_used == "umap":
+                try:
+                    import umap  # type: ignore
+                    reducer = umap.UMAP(n_components=3, random_state=42)
+                    coords = reducer.fit_transform(arr).tolist()
+                except Exception as e:
+                    logger.warning(f"UMAP unavailable, falling back to PCA: {e}")
+                    method_used = "pca"
+
+            if method_used == "tsne":
+                from sklearn.manifold import TSNE
+                perplexity = min(30, max(5, int(len(arr) / 3)))
+                tsne = TSNE(n_components=3, init="random", learning_rate="auto", perplexity=perplexity)
+                coords = tsne.fit_transform(arr).tolist()
+
+            if method_used == "pca":
+                from sklearn.decomposition import PCA
+                pca = PCA(n_components=3)
+                coords = pca.fit_transform(arr).tolist()
+        except Exception as e:
+            logger.warning(f"Projection failed, using fallback: {e}")
+            method_used = "fallback"
+
+    if not coords:
+        coords = []
+        for p in points:
+            np.random.seed(hash(p["id"]) % 2**32)
+            cx, cy, cz = [(2, 3, 1), (-2, -1, -1), (1, -3, 0)][p["cluster"] % 3]
+            coords.append([
+                float(cx + np.random.randn() * 0.6),
+                float(cy + np.random.randn() * 0.6),
+                float(cz + np.random.randn() * 0.6),
+            ])
+        method_used = "fallback"
+
+    for p, c in zip(points, coords):
+        p["x"], p["y"], p["z"] = float(c[0]), float(c[1]), float(c[2])
+
+    avg_score = float(np.mean([p.get("score", 0) for p in points])) if points else 0.0
+
     return {
         "points": points,
-        "method": method,
+        "method": method_used,
         "dataset": dataset,
         "n_clusters": len(set(p["cluster"] for p in points)) if points else 0,
+        "avg_score": avg_score,
     }
 
 
+# ============================================================================
+# Source Ingestion (Phase 3)
+# ============================================================================
+def _resolve_batch_size(requested: Optional[int]) -> int:
+    if requested is not None:
+        return requested
+    env_val = os.getenv("INGEST_BATCH_SIZE")
+    if env_val:
+        try:
+            return int(env_val)
+        except ValueError:
+            pass
+    return 50
+
+
+def _resolve_rate_limit(source: str, requested: Optional[float]) -> float:
+    if requested is not None:
+        return requested
+    env_key = f"{source.upper()}_RATE_LIMIT"
+    env_val = os.getenv(env_key)
+    if env_val:
+        try:
+            return float(env_val)
+        except ValueError:
+            pass
+    defaults = {"pubmed": 0.4, "uniprot": 0.2, "chembl": 0.3}
+    return defaults.get(source, 0.3)
+
+
+def _execute_source_ingestion(source: str, request: IngestSourceRequest):
+    if not model_service or not qdrant_service:
+        raise HTTPException(status_code=503, detail="Services not available")
+
+    encoder = model_service.get_obm_encoder()
+    batch_size = _resolve_batch_size(request.batch_size)
+    rate_limit = _resolve_rate_limit(source, request.rate_limit)
+    collection = request.collection or "bioflow_memory"
+
+    if source == "pubmed":
+        from bioflow.ingestion.pubmed_ingestor import PubMedIngestor
+        ingestor = PubMedIngestor(
+            qdrant_service=qdrant_service,
+            obm_encoder=encoder,
+            collection=collection,
+            batch_size=batch_size,
+            rate_limit=rate_limit,
+            email=request.email or os.getenv("NCBI_EMAIL", "bioflow@example.com"),
+            api_key=request.api_key or os.getenv("NCBI_API_KEY"),
+        )
+        return ingestor.ingest(request.query, request.limit)
+
+    if source == "uniprot":
+        from bioflow.ingestion.uniprot_ingestor import UniProtIngestor
+        ingestor = UniProtIngestor(
+            qdrant_service=qdrant_service,
+            obm_encoder=encoder,
+            collection=collection,
+            batch_size=batch_size,
+            rate_limit=rate_limit,
+        )
+        return ingestor.ingest(request.query, request.limit)
+
+    if source == "chembl":
+        from bioflow.ingestion.chembl_ingestor import ChEMBLIngestor
+        ingestor = ChEMBLIngestor(
+            qdrant_service=qdrant_service,
+            obm_encoder=encoder,
+            collection=collection,
+            batch_size=batch_size,
+            rate_limit=rate_limit,
+            search_mode=request.search_mode or os.getenv("CHEMBL_SEARCH_MODE", "target"),
+        )
+        return ingestor.ingest(request.query, request.limit)
+
+    raise HTTPException(status_code=400, detail=f"Unknown source: {source}")
+
+
+def _start_ingestion_job(source: str, payload: Dict[str, Any], background_tasks: BackgroundTasks):
+    job_id = f"ing_{uuid.uuid4().hex[:12]}"
+    now = datetime.utcnow().isoformat()
+    JOBS[job_id] = {
+        "job_id": job_id,
+        "type": "ingestion",
+        "source": source,
+        "status": "pending",
+        "progress": 0,
+        "result": None,
+        "error": None,
+        "created_at": now,
+        "updated_at": now,
+        "request": payload,
+    }
+
+    background_tasks.add_task(_run_ingestion_job, job_id, source, payload)
+    return job_id
+
+
+def _execute_all_ingestion(request: IngestAllRequest):
+    results = {}
+    if not request.skip_pubmed:
+        results["pubmed"] = _execute_source_ingestion(
+            "pubmed",
+            IngestSourceRequest(
+                query=request.query,
+                limit=request.pubmed_limit,
+                batch_size=request.batch_size,
+                rate_limit=request.rate_limit,
+                collection=request.collection,
+                email=request.email,
+                api_key=request.api_key,
+            ),
+        )
+    if not request.skip_uniprot:
+        results["uniprot"] = _execute_source_ingestion(
+            "uniprot",
+            IngestSourceRequest(
+                query=request.query,
+                limit=request.uniprot_limit,
+                batch_size=request.batch_size,
+                rate_limit=request.rate_limit,
+                collection=request.collection,
+            ),
+        )
+    if not request.skip_chembl:
+        results["chembl"] = _execute_source_ingestion(
+            "chembl",
+            IngestSourceRequest(
+                query=request.query,
+                limit=request.chembl_limit,
+                batch_size=request.batch_size,
+                rate_limit=request.rate_limit,
+                collection=request.collection,
+                search_mode=request.search_mode,
+            ),
+        )
+    return results
+
+
+def _run_ingestion_job(job_id: str, source: str, payload: Dict[str, Any]) -> None:
+    JOBS[job_id]["status"] = "running"
+    JOBS[job_id]["updated_at"] = datetime.utcnow().isoformat()
+    try:
+        if source == "all":
+            results = _execute_all_ingestion(IngestAllRequest(**payload))
+            JOBS[job_id]["result"] = {k: v.to_dict() for k, v in results.items()}
+        else:
+            req = IngestSourceRequest(**payload)
+            result = _execute_source_ingestion(source, req)
+            JOBS[job_id]["result"] = result.to_dict()
+
+        JOBS[job_id]["status"] = "completed"
+        JOBS[job_id]["progress"] = 100
+        JOBS[job_id]["updated_at"] = datetime.utcnow().isoformat()
+    except Exception as e:
+        JOBS[job_id]["status"] = "failed"
+        JOBS[job_id]["error"] = str(e)
+        JOBS[job_id]["updated_at"] = datetime.utcnow().isoformat()
+        logger.error(f"Ingestion job failed: {e}")
+
+
+@app.post("/api/ingest/pubmed")
+async def ingest_pubmed(request: IngestSourceRequest, background_tasks: BackgroundTasks):
+    if request.sync:
+        result = _execute_source_ingestion("pubmed", request)
+        return {"success": True, "result": result.to_dict()}
+    job_id = _start_ingestion_job("pubmed", request.model_dump(), background_tasks)
+    return {"success": True, "job_id": job_id, "status": "pending"}
+
+
+@app.post("/api/ingest/uniprot")
+async def ingest_uniprot(request: IngestSourceRequest, background_tasks: BackgroundTasks):
+    if request.sync:
+        result = _execute_source_ingestion("uniprot", request)
+        return {"success": True, "result": result.to_dict()}
+    job_id = _start_ingestion_job("uniprot", request.model_dump(), background_tasks)
+    return {"success": True, "job_id": job_id, "status": "pending"}
+
+
+@app.post("/api/ingest/chembl")
+async def ingest_chembl(request: IngestSourceRequest, background_tasks: BackgroundTasks):
+    if request.sync:
+        result = _execute_source_ingestion("chembl", request)
+        return {"success": True, "result": result.to_dict()}
+    job_id = _start_ingestion_job("chembl", request.model_dump(), background_tasks)
+    return {"success": True, "job_id": job_id, "status": "pending"}
+
+
+@app.post("/api/ingest/all")
+async def ingest_all(request: IngestAllRequest, background_tasks: BackgroundTasks):
+    if request.sync:
+        results = _execute_all_ingestion(request)
+        return {"success": True, "result": {k: v.to_dict() for k, v in results.items()}}
+    job_id = _start_ingestion_job("all", request.model_dump(), background_tasks)
+    return {"success": True, "job_id": job_id, "status": "pending"}
+
+
+@app.get("/api/ingest/jobs/{job_id}")
+async def get_ingest_status(job_id: str):
+    if job_id not in JOBS:
+        raise HTTPException(status_code=404, detail="Job not found")
+    return JOBS[job_id]
+
+
 # ============================================================================
 # Additional API Endpoints
 # ============================================================================
@@ -811,6 +1565,8 @@ async def run_discovery_workflow(request: WorkflowRequest):
     
     Returns top candidates with all validation and ranking metadata.
     """
+    request_id = uuid.uuid4().hex[:12]
+    start = time.perf_counter()
     try:
         from bioflow.agents import DiscoveryWorkflow
         
@@ -822,17 +1578,32 @@ async def run_discovery_workflow(request: WorkflowRequest):
         result = workflow.run(request.query)
         top_candidates = workflow.get_top_candidates(result)
         
-        return {
+        payload = {
             "success": result.status.value == "completed",
             "status": result.status.value,
             "steps_completed": result.steps_completed,
             "total_steps": result.total_steps,
             "execution_time_ms": result.execution_time_ms,
             "top_candidates": top_candidates,
+            "candidates": top_candidates,
             "all_outputs": result.outputs,
             "errors": result.errors,
         }
+        _log_event(
+            "workflow",
+            request_id,
+            status=payload.get("status"),
+            steps_completed=payload.get("steps_completed"),
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
+        return payload
     except Exception as e:
+        _log_event(
+            "workflow_error",
+            request_id,
+            error=str(e),
+            duration_ms=round((time.perf_counter() - start) * 1000, 2),
+        )
         logger.error(f"Workflow failed: {e}")
         raise HTTPException(status_code=500, detail=str(e))
 

bioflow/app.py

@@ -1,569 +0,0 @@
-"""
-BioFlow Explorer - Streamlit Interface
-=======================================
-
-Interactive web interface for testing and exploring the BioFlow
-multimodal biological intelligence system.
-
-Run with: streamlit run bioflow/app.py
-"""
-
-import streamlit as st
-import numpy as np
-import pandas as pd
-from typing import List, Dict, Any
-import json
-import os
-import sys
-
-# Add project root to path
-ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-sys.path.insert(0, ROOT_DIR)
-
-# Page config
-st.set_page_config(
-    page_title="BioFlow Explorer",
-    page_icon="🧬",
-    layout="wide",
-    initial_sidebar_state="expanded"
-)
-
-# Custom CSS
-st.markdown("""
-<style>
-    .main-header {
-        font-size: 2.5rem;
-        font-weight: bold;
-        background: linear-gradient(90deg, #667eea 0%, #764ba2 100%);
-        -webkit-background-clip: text;
-        -webkit-text-fill-color: transparent;
-        margin-bottom: 1rem;
-    }
-    .metric-card {
-        background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
-        padding: 1rem;
-        border-radius: 0.5rem;
-        margin: 0.5rem 0;
-    }
-    .result-card {
-        border: 1px solid #ddd;
-        border-radius: 0.5rem;
-        padding: 1rem;
-        margin: 0.5rem 0;
-        background: white;
-    }
-    .modality-text { color: #3b82f6; }
-    .modality-molecule { color: #10b981; }
-    .modality-protein { color: #f59e0b; }
-</style>
-""", unsafe_allow_html=True)
-
-
-@st.cache_resource
-def init_bioflow():
-    """Initialize BioFlow components (cached)."""
-    try:
-        from bioflow.obm_wrapper import OBMWrapper
-        from bioflow.qdrant_manager import QdrantManager
-        from bioflow.pipeline import BioFlowPipeline, MinerAgent, ValidatorAgent
-        
-        obm = OBMWrapper()
-        qdrant = QdrantManager(obm, qdrant_path=None)  # In-memory
-        qdrant.create_collection("bioflow_demo", recreate=True)
-        
-        pipeline = BioFlowPipeline(obm, qdrant)
-        pipeline.register_agent(MinerAgent(obm, qdrant, "bioflow_demo"))
-        pipeline.register_agent(ValidatorAgent(obm, qdrant, "bioflow_demo"))
-        
-        return {
-            "obm": obm,
-            "qdrant": qdrant,
-            "pipeline": pipeline,
-            "ready": True
-        }
-    except Exception as e:
-        st.error(f"Failed to initialize: {e}")
-        return {"ready": False, "error": str(e)}
-
-
-def render_sidebar():
-    """Render the sidebar with controls."""
-    st.sidebar.markdown("## 🧬 BioFlow Explorer")
-    st.sidebar.markdown("---")
-    
-    # Mode selection
-    mode = st.sidebar.selectbox(
-        "Mode",
-        ["🔍 Search & Explore", "📥 Data Ingestion", "🧪 Cross-Modal Analysis", 
-         "📊 Visualization", "🔬 Pipeline Demo", "📚 Documentation"]
-    )
-    
-    st.sidebar.markdown("---")
-    
-    # Settings
-    with st.sidebar.expander("⚙️ Settings"):
-        vector_dim = st.number_input("Vector Dimension", value=768, disabled=True)
-        
-    st.sidebar.markdown("---")
-    st.sidebar.markdown("### Quick Stats")
-    
-    return mode
-
-
-def render_search_page(components):
-    """Render the search and explore page."""
-    st.markdown('<p class="main-header">🔍 Search & Explore</p>', unsafe_allow_html=True)
-    
-    col1, col2 = st.columns([2, 1])
-    
-    with col1:
-        query = st.text_area(
-            "Enter your query",
-            placeholder="e.g., 'KRAS inhibitor for cancer treatment' or a SMILES string like 'CCO'",
-            height=100
-        )
-        
-        query_modality = st.selectbox(
-            "Query Modality",
-            ["text", "smiles", "protein"],
-            help="Select the type of your input"
-        )
-    
-    with col2:
-        target_modality = st.selectbox(
-            "Search for",
-            ["All", "text", "smiles", "protein"],
-            help="Filter results by modality"
-        )
-        
-        top_k = st.slider("Number of results", 1, 20, 5)
-    
-    if st.button("🔍 Search", type="primary"):
-        if not query:
-            st.warning("Please enter a query")
-            return
-        
-        with st.spinner("Encoding and searching..."):
-            obm = components["obm"]
-            qdrant = components["qdrant"]
-            
-            # Encode query
-            embedding = obm.encode(query, query_modality)
-            
-            # Display query embedding info
-            with st.expander("📊 Query Embedding Details"):
-                st.json({
-                    "modality": embedding.modality.value,
-                    "dimension": embedding.dimension,
-                    "content_hash": embedding.content_hash,
-                    "vector_sample": embedding.vector[:5].tolist()
-                })
-            
-            # Search
-            filter_mod = None if target_modality == "All" else target_modality
-            results = qdrant.search(
-                query=query,
-                query_modality=query_modality,
-                limit=top_k,
-                filter_modality=filter_mod
-            )
-            
-            if results:
-                st.markdown("### 📋 Search Results")
-                for i, r in enumerate(results):
-                    with st.container():
-                        col1, col2, col3 = st.columns([1, 4, 1])
-                        with col1:
-                            st.metric("Rank", i + 1)
-                        with col2:
-                            modality_class = f"modality-{r.modality}"
-                            st.markdown(f"**<span class='{modality_class}'>[{r.modality.upper()}]</span>** {r.content[:100]}...", unsafe_allow_html=True)
-                        with col3:
-                            st.metric("Score", f"{r.score:.3f}")
-                        st.divider()
-            else:
-                st.info("No results found. Try ingesting some data first!")
-
-
-def render_ingestion_page(components):
-    """Render the data ingestion page."""
-    st.markdown('<p class="main-header">📥 Data Ingestion</p>', unsafe_allow_html=True)
-    
-    tab1, tab2, tab3 = st.tabs(["📝 Single Entry", "📄 Batch Upload", "🧪 Sample Data"])
-    
-    with tab1:
-        st.markdown("### Add Single Entry")
-        
-        col1, col2 = st.columns(2)
-        with col1:
-            content = st.text_area("Content", placeholder="Enter text, SMILES, or protein sequence")
-            modality = st.selectbox("Type", ["text", "smiles", "protein"])
-        
-        with col2:
-            source = st.text_input("Source", placeholder="e.g., PubMed:12345")
-            tags = st.text_input("Tags (comma-separated)", placeholder="e.g., cancer, kinase")
-        
-        if st.button("➕ Add Entry"):
-            if content:
-                qdrant = components["qdrant"]
-                item = {
-                    "content": content,
-                    "modality": modality,
-                    "source": source,
-                    "tags": [t.strip() for t in tags.split(",") if t.strip()]
-                }
-                stats = qdrant.ingest([item])
-                st.success(f"Added successfully! Stats: {stats}")
-            else:
-                st.warning("Please enter content")
-    
-    with tab2:
-        st.markdown("### Batch Upload")
-        
-        uploaded_file = st.file_uploader("Upload JSON or CSV", type=["json", "csv"])
-        
-        if uploaded_file:
-            try:
-                if uploaded_file.name.endswith('.json'):
-                    data = json.load(uploaded_file)
-                else:
-                    df = pd.read_csv(uploaded_file)
-                    data = df.to_dict('records')
-                
-                st.write(f"Found {len(data)} entries")
-                st.dataframe(pd.DataFrame(data).head())
-                
-                if st.button("📤 Upload All"):
-                    qdrant = components["qdrant"]
-                    stats = qdrant.ingest(data)
-                    st.success(f"Ingestion complete! {stats}")
-            except Exception as e:
-                st.error(f"Error parsing file: {e}")
-    
-    with tab3:
-        st.markdown("### Load Sample Data")
-        st.markdown("Load pre-defined sample data to test the system.")
-        
-        sample_data = [
-            {"content": "Aspirin is used to reduce fever and relieve mild to moderate pain", "modality": "text", "source": "sample", "tags": ["pain", "fever"]},
-            {"content": "CC(=O)OC1=CC=CC=C1C(=O)O", "modality": "smiles", "source": "ChEMBL", "tags": ["aspirin", "nsaid"]},
-            {"content": "Ibuprofen is a nonsteroidal anti-inflammatory drug used for treating pain", "modality": "text", "source": "sample", "tags": ["pain", "nsaid"]},
-            {"content": "CC(C)CC1=CC=C(C=C1)C(C)C(=O)O", "modality": "smiles", "source": "ChEMBL", "tags": ["ibuprofen", "nsaid"]},
-            {"content": "KRAS mutations are found in many cancers and are difficult to target", "modality": "text", "source": "PubMed", "tags": ["cancer", "KRAS"]},
-            {"content": "MTEYKLVVVGAGGVGKSALTIQLIQNHFVDEYDPTIEDSYRKQVVIDGETCLLDILDTAGQEEYSAMRDQYMRTGEGFLCVFAINNTKSFEDIHHYREQIKRVKDSEDVPMVLVGNKCDLPSRTVDTKQAQDLARSYGIPFIETSAKTRQGVDDAFYTLVREIRKHKEKMSKDGKKKKKKSKTKCVIM", "modality": "protein", "source": "UniProt:P01116", "tags": ["KRAS", "GTPase"]},
-            {"content": "Sotorasib is a first-in-class KRAS G12C inhibitor", "modality": "text", "source": "PubMed", "tags": ["KRAS", "inhibitor", "cancer"]},
-            {"content": "C[C@@H]1CC(=O)N(C2=C1C=CC(=C2)NC(=O)C3=CC=C(C=C3)N4CCN(CC4)C)C5=NC=CC(=N5)C6CCCCC6", "modality": "smiles", "source": "ChEMBL", "tags": ["sotorasib", "KRAS", "inhibitor"]},
-        ]
-        
-        if st.button("🧪 Load Sample Data"):
-            qdrant = components["qdrant"]
-            stats = qdrant.ingest(sample_data)
-            st.success(f"Loaded {len(sample_data)} sample entries! {stats}")
-            st.balloons()
-
-
-def render_crossmodal_page(components):
-    """Render cross-modal analysis page."""
-    st.markdown('<p class="main-header">🧪 Cross-Modal Analysis</p>', unsafe_allow_html=True)
-    
-    st.markdown("""
-    Explore how different modalities relate to each other in the shared embedding space.
-    This is the core capability of BioFlow - connecting text, molecules, and proteins.
-    """)
-    
-    col1, col2 = st.columns(2)
-    
-    with col1:
-        st.markdown("### Query")
-        query = st.text_area("Enter query", height=100)
-        query_mod = st.selectbox("Query type", ["text", "smiles", "protein"], key="q_mod")
-    
-    with col2:
-        st.markdown("### Targets")
-        targets = st.text_area("Enter targets (one per line)", height=100)
-        target_mod = st.selectbox("Target type", ["text", "smiles", "protein"], key="t_mod")
-    
-    if st.button("🔄 Compute Cross-Modal Similarity"):
-        if query and targets:
-            obm = components["obm"]
-            target_list = [t.strip() for t in targets.strip().split("\n") if t.strip()]
-            
-            results = obm.cross_modal_similarity(
-                query=query,
-                query_modality=query_mod,
-                targets=target_list,
-                target_modality=target_mod
-            )
-            
-            st.markdown("### Results (sorted by similarity)")
-            
-            df = pd.DataFrame(results, columns=["Content", "Similarity"])
-            df["Rank"] = range(1, len(df) + 1)
-            df = df[["Rank", "Content", "Similarity"]]
-            
-            st.dataframe(df, use_container_width=True)
-            
-            # Visualize
-            import plotly.express as px
-            fig = px.bar(df, x="Content", y="Similarity", title="Cross-Modal Similarities")
-            st.plotly_chart(fig, use_container_width=True)
-
-
-def render_visualization_page(components):
-    """Render visualization page."""
-    st.markdown('<p class="main-header">📊 Visualization</p>', unsafe_allow_html=True)
-    
-    tab1, tab2, tab3 = st.tabs(["🌐 Embedding Space", "📈 Similarity Matrix", "🧬 Molecules"])
-    
-    with tab1:
-        st.markdown("### Embedding Space Visualization")
-        
-        # Get all points from collection
-        qdrant = components["qdrant"]
-        info = qdrant.get_collection_info()
-        
-        if info.get("points_count", 0) == 0:
-            st.warning("No data in collection. Go to Data Ingestion to add some!")
-            return
-        
-        st.metric("Points in collection", info.get("points_count", 0))
-        
-        if st.button("🎨 Generate Embedding Plot"):
-            # This would require fetching all vectors - simplified for demo
-            st.info("Embedding visualization requires fetching all vectors. In production, use sampling.")
-            
-            # Demo with random data
-            n_points = min(info.get("points_count", 20), 50)
-            fake_embeddings = np.random.randn(n_points, 2)
-            
-            import plotly.express as px
-            fig = px.scatter(
-                x=fake_embeddings[:, 0],
-                y=fake_embeddings[:, 1],
-                title="Embedding Space (Demo - PCA projection)"
-            )
-            st.plotly_chart(fig, use_container_width=True)
-    
-    with tab2:
-        st.markdown("### Compute Similarity Matrix")
-        
-        items = st.text_area("Enter items (one per line)", height=150)
-        modality = st.selectbox("Modality", ["text", "smiles", "protein"], key="sim_mod")
-        
-        if st.button("🔢 Compute Matrix"):
-            if items:
-                obm = components["obm"]
-                item_list = [i.strip() for i in items.strip().split("\n") if i.strip()]
-                
-                if modality == "text":
-                    embeddings = obm.encode_text(item_list)
-                elif modality == "smiles":
-                    embeddings = obm.encode_smiles(item_list)
-                else:
-                    embeddings = obm.encode_protein(item_list)
-                
-                vectors = np.array([e.vector for e in embeddings])
-                
-                # Compute similarity
-                norms = np.linalg.norm(vectors, axis=1, keepdims=True)
-                normalized = vectors / np.clip(norms, 1e-9, None)
-                similarity = np.dot(normalized, normalized.T)
-                
-                import plotly.figure_factory as ff
-                labels = [i[:20] for i in item_list]
-                fig = ff.create_annotated_heatmap(
-                    similarity,
-                    x=labels,
-                    y=labels,
-                    colorscale='RdBu'
-                )
-                st.plotly_chart(fig, use_container_width=True)
-    
-    with tab3:
-        st.markdown("### Molecule Visualization")
-        
-        smiles = st.text_input("Enter SMILES", placeholder="CC(=O)OC1=CC=CC=C1C(=O)O")
-        
-        if smiles:
-            try:
-                from rdkit import Chem
-                from rdkit.Chem import Draw
-                
-                mol = Chem.MolFromSmiles(smiles)
-                if mol:
-                    img = Draw.MolToImage(mol, size=(400, 300))
-                    st.image(img, caption=f"Molecule: {smiles}")
-                else:
-                    st.error("Invalid SMILES")
-            except ImportError:
-                st.warning("RDKit not installed. Install with: pip install rdkit")
-
-
-def render_pipeline_page(components):
-    """Render pipeline demo page."""
-    st.markdown('<p class="main-header">🔬 Pipeline Demo</p>', unsafe_allow_html=True)
-    
-    st.markdown("""
-    Run a complete discovery workflow that:
-    1. Searches for related literature
-    2. Finds similar molecules
-    3. Validates candidates
-    4. Analyzes result diversity
-    """)
-    
-    query = st.text_input("Enter discovery query", placeholder="e.g., KRAS inhibitor for lung cancer")
-    
-    col1, col2 = st.columns(2)
-    with col1:
-        query_mod = st.selectbox("Query modality", ["text", "smiles", "protein"])
-    with col2:
-        target_mod = st.selectbox("Target modality", ["smiles", "text", "protein"])
-    
-    if st.button("🚀 Run Discovery Pipeline", type="primary"):
-        if query:
-            pipeline = components["pipeline"]
-            
-            with st.spinner("Running pipeline..."):
-                results = pipeline.run_discovery_workflow(
-                    query=query,
-                    query_modality=query_mod,
-                    target_modality=target_mod
-                )
-            
-            st.markdown("## 📊 Pipeline Results")
-            
-            # Literature
-            with st.expander("📚 Related Literature", expanded=True):
-                lit = results.get("stages", {}).get("literature", [])
-                if lit:
-                    for item in lit:
-                        st.markdown(f"- **Score: {item['score']:.3f}** - {item['content'][:100]}...")
-                else:
-                    st.info("No literature found")
-            
-            # Molecules
-            with st.expander("🧪 Similar Molecules", expanded=True):
-                mols = results.get("stages", {}).get("molecules", [])
-                if mols:
-                    df = pd.DataFrame(mols)
-                    st.dataframe(df)
-                else:
-                    st.info("No molecules found")
-            
-            # Validation
-            with st.expander("✅ Validation Results"):
-                val = results.get("stages", {}).get("validation", [])
-                if val:
-                    st.json(val)
-                else:
-                    st.info("No validation performed")
-            
-            # Diversity
-            with st.expander("📈 Diversity Analysis"):
-                div = results.get("stages", {}).get("diversity", {})
-                if div:
-                    col1, col2, col3 = st.columns(3)
-                    col1.metric("Mean Similarity", f"{div.get('mean_similarity', 0):.3f}")
-                    col2.metric("Diversity Score", f"{div.get('diversity_score', 0):.3f}")
-                    col3.metric("Modalities", len(div.get('modality_distribution', {})))
-                    st.json(div)
-
-
-def render_docs_page():
-    """Render documentation page."""
-    st.markdown('<p class="main-header">📚 Documentation</p>', unsafe_allow_html=True)
-    
-    st.markdown("""
-    ## BioFlow + OpenBioMed Integration
-    
-    ### 🎯 Overview
-    
-    BioFlow is a multimodal biological intelligence framework that leverages OpenBioMed (OBM)
-    for encoding biological data and Qdrant for vector storage and retrieval.
-    
-    ### 🧩 Components
-    
-    | Component | Description |
-    |-----------|-------------|
-    | **OBMWrapper** | Encodes text, molecules (SMILES), and proteins into a shared vector space |
-    | **QdrantManager** | Manages vector storage, indexing, and similarity search |
-    | **BioFlowPipeline** | Orchestrates agents in discovery workflows |
-    | **Visualizer** | Creates plots for embeddings, similarities, and molecules |
-    
-    ### 🔌 API Examples
-    
-    ```python
-    from bioflow import OBMWrapper, QdrantManager, BioFlowPipeline
-    
-    # Initialize
-    obm = OBMWrapper(device="cuda")
-    qdrant = QdrantManager(obm, qdrant_path="./data/qdrant")
-    
-    # Encode different modalities
-    text_vec = obm.encode_text("KRAS inhibitor for cancer")
-    mol_vec = obm.encode_smiles("CCO")
-    prot_vec = obm.encode_protein("MTEYKLVVV...")
-    
-    # Cross-modal search
-    results = qdrant.cross_modal_search(
-        query="anti-inflammatory drug",
-        query_modality="text",
-        target_modality="smiles",
-        limit=10
-    )
-    ```
-    
-    ### 🌟 Key Features
-    
-    1. **Unified Embedding Space**: All modalities map to the same vector dimension
-    2. **Cross-Modal Search**: Find molecules from text queries and vice versa
-    3. **Pipeline Orchestration**: Chain agents for complex discovery workflows
-    4. **Mock Mode**: Test without GPU using deterministic random embeddings
-    
-    ### 📁 File Structure
-    
-    ```
-    bioflow/
-    ├── __init__.py          # Package exports
-    ├── obm_wrapper.py       # OBM encoding interface
-    ├── qdrant_manager.py    # Qdrant operations
-    ├── pipeline.py          # Workflow orchestration
-    ├── visualizer.py        # Visualization utilities
-    └── app.py               # Streamlit interface
-    ```
-    """)
-
-
-def main():
-    """Main application entry point."""
-    mode = render_sidebar()
-    
-    # Initialize components
-    components = init_bioflow()
-    
-    if not components.get("ready"):
-        st.error("System not ready. Check configuration.")
-        return
-    
-    # Display collection stats in sidebar
-    info = components["qdrant"].get_collection_info()
-    st.sidebar.metric("📊 Vectors", info.get("points_count", 0))
-    st.sidebar.metric("📐 Dimension", info.get("vector_size", 768))
-    
-    # Route to appropriate page
-    if "Search" in mode:
-        render_search_page(components)
-    elif "Ingestion" in mode:
-        render_ingestion_page(components)
-    elif "Cross-Modal" in mode:
-        render_crossmodal_page(components)
-    elif "Visualization" in mode:
-        render_visualization_page(components)
-    elif "Pipeline" in mode:
-        render_pipeline_page(components)
-    elif "Documentation" in mode:
-        render_docs_page()
-
-
-if __name__ == "__main__":
-    main()

bioflow/demo.py

@@ -221,7 +221,9 @@ def demo_visualization():
         print("   - plot_embeddings_3d(embeddings, labels)")
         print("   - plot_similarity_matrix(embeddings, labels)")
         print("   - create_dashboard(results, embeddings)")
-        print("\n   Run the Streamlit app to see interactive visualizations!")
+        print("\n   Use the Next.js UI for interactive visualizations:")
+        print("     - Start backend: python -m uvicorn bioflow.api.server:app --host 0.0.0.0 --port 8000")
+        print("     - Start UI:      cd ui && pnpm dev")
         
     except ImportError as e:
         print(f"⚠️ Some visualization dependencies missing: {e}")
@@ -245,8 +247,9 @@ def main():
     
     print_header("✅ Demo Complete!")
     print("Next steps:")
-    print("  1. Run the Streamlit interface:")
-    print("     streamlit run bioflow/app.py")
+    print("  1. Run the full stack:")
+    print("     - Backend: python -m uvicorn bioflow.api.server:app --host 0.0.0.0 --port 8000")
+    print("     - UI:      cd ui && pnpm dev")
     print("")
     print("  2. Ensure OBM model is configured:")
     print("     - BioMedGPT checkpoints are downloaded")

bioflow/evaluation/__init__.py

@@ -0,0 +1,21 @@
+"""
+BioFlow Evaluation
+==================
+
+Offline evaluation utilities for retrieval and diversification quality.
+"""
+
+from .metrics import (
+    recall_at_k,
+    mrr_at_k,
+    ndcg_at_k,
+    intra_list_diversity_cosine,
+)
+
+__all__ = [
+    "recall_at_k",
+    "mrr_at_k",
+    "ndcg_at_k",
+    "intra_list_diversity_cosine",
+]
+

bioflow/evaluation/metrics.py

@@ -0,0 +1,82 @@
+from __future__ import annotations
+
+import math
+from typing import Dict, Iterable, List, Mapping, Sequence, Set
+
+
+def recall_at_k(relevant: Set[str], ranked: Sequence[str], k: int) -> float:
+    if k <= 0:
+        return 0.0
+    if not relevant:
+        return 0.0
+    top = set(ranked[:k])
+    return len(relevant.intersection(top)) / float(len(relevant))
+
+
+def mrr_at_k(relevant: Set[str], ranked: Sequence[str], k: int) -> float:
+    if k <= 0:
+        return 0.0
+    if not relevant:
+        return 0.0
+    for i, doc_id in enumerate(ranked[:k]):
+        if doc_id in relevant:
+            return 1.0 / float(i + 1)
+    return 0.0
+
+
+def _dcg(relevances: Sequence[float], k: int) -> float:
+    total = 0.0
+    for i, rel in enumerate(relevances[:k]):
+        total += (2.0 ** float(rel) - 1.0) / math.log2(float(i + 2))
+    return total
+
+
+def ndcg_at_k(relevance_by_id: Mapping[str, float], ranked: Sequence[str], k: int) -> float:
+    if k <= 0:
+        return 0.0
+    if not relevance_by_id:
+        return 0.0
+
+    rels = [float(relevance_by_id.get(doc_id, 0.0)) for doc_id in ranked]
+    dcg = _dcg(rels, k)
+
+    ideal_rels = sorted(relevance_by_id.values(), reverse=True)
+    idcg = _dcg(ideal_rels, k)
+    if idcg == 0.0:
+        return 0.0
+    return dcg / idcg
+
+
+def cosine_similarity(a: Sequence[float], b: Sequence[float]) -> float:
+    dot = 0.0
+    na = 0.0
+    nb = 0.0
+    for x, y in zip(a, b):
+        dot += float(x) * float(y)
+        na += float(x) * float(x)
+        nb += float(y) * float(y)
+    denom = math.sqrt(na) * math.sqrt(nb)
+    return (dot / denom) if denom else 0.0
+
+
+def cosine_distance(a: Sequence[float], b: Sequence[float]) -> float:
+    return 1.0 - cosine_similarity(a, b)
+
+
+def intra_list_diversity_cosine(embeddings: Sequence[Sequence[float]]) -> float:
+    """
+    Average pairwise cosine distance within a list.
+    Higher = more diverse.
+    """
+    n = len(embeddings)
+    if n < 2:
+        return 1.0
+
+    total = 0.0
+    count = 0
+    for i in range(n):
+        for j in range(i + 1, n):
+            total += cosine_distance(embeddings[i], embeddings[j])
+            count += 1
+    return total / float(count) if count else 1.0
+

bioflow/ingestion/pubmed_ingestor.py

@@ -15,6 +15,7 @@ Usage:
 import logging
 import requests
 import xml.etree.ElementTree as ET
+import re
 from typing import Dict, Any, Optional, Generator
 from datetime import datetime
 
@@ -134,6 +135,7 @@ class PubMedIngestor(BaseIngestor):
                     # Extract publication date
                     pub_date = ""
                     date_elem = article.find(".//PubDate")
+                    year_value = None
                     if date_elem is not None:
                         year = date_elem.find("Year")
                         month = date_elem.find("Month")
@@ -141,6 +143,14 @@ class PubMedIngestor(BaseIngestor):
                             pub_date = year.text
                             if month is not None:
                                 pub_date = f"{year.text}-{month.text}"
+                            try:
+                                year_value = int(year.text)
+                            except (TypeError, ValueError):
+                                year_value = None
+                    if year_value is None and pub_date:
+                        match = re.match(r"(\\d{4})", pub_date)
+                        if match:
+                            year_value = int(match.group(1))
                     
                     # Extract authors
                     authors = []
@@ -170,6 +180,7 @@ class PubMedIngestor(BaseIngestor):
                         "authors": authors,
                         "journal": journal,
                         "pub_date": pub_date,
+                        "year": year_value,
                         "mesh_terms": mesh_terms,
                     }
                     
@@ -246,6 +257,7 @@ class PubMedIngestor(BaseIngestor):
                 "authors": raw_data.get("authors", []),
                 "journal": raw_data.get("journal", ""),
                 "pub_date": raw_data.get("pub_date", ""),
+                "year": raw_data.get("year"),
                 "mesh_terms": raw_data.get("mesh_terms", []),
                 "url": f"https://pubmed.ncbi.nlm.nih.gov/{pmid}/",
             }

bioflow/pipeline.py

@@ -4,6 +4,9 @@ BioFlow Pipeline - Workflow Orchestration
 
 This module provides the pipeline orchestration for BioFlow,
 connecting agents, memory (Qdrant), and OBM encoders.
+
+DEPRECATED (legacy): Prefer the FastAPI backend (`bioflow/api/server.py`) and
+the agent system (`bioflow/agents/*`). This module is kept for older demos/scripts.
 """
 
 import logging

bioflow/qdrant_manager.py

@@ -4,6 +4,9 @@ Qdrant Manager - Vector Database Integration
 
 This module provides high-level management for Qdrant collections,
 including ingestion, search, and retrieval operations for BioFlow.
+
+DEPRECATED (legacy): Prefer `bioflow/api/qdrant_service.py` for the active FastAPI backend.
+This module is kept for backward compatibility with older demos/scripts.
 """
 
 import logging

bioflow/search/enhanced_search.py

@@ -25,8 +25,12 @@ import logging
 from typing import List, Dict, Any, Optional, Union
 from dataclasses import dataclass, field
 from datetime import datetime
+import re
+import threading
+from collections import OrderedDict
 
 from bioflow.search.mmr import MMRReranker, MMRResult
+from bioflow.search.mmr import mmr_rerank
 from bioflow.search.evidence import EvidenceLinker, EnrichedResult, EvidenceLink
 
 logger = logging.getLogger(__name__)
@@ -80,6 +84,8 @@ class EnhancedSearchResult:
                 }
                 for l in self.evidence_links
             ],
+            # UI expects `source`; keep `source_type` for backward compatibility.
+            "source": self.source_type,
             "source_type": self.source_type,
             "citation": self.citation,
             "rank": self.rank,
@@ -123,6 +129,8 @@ class EnhancedSearchService:
         obm_encoder,
         default_lambda: float = 0.7,
         default_top_k: int = 20,
+        query_cache_max_size: int = 256,
+        query_cache_ttl_s: float = 300.0,
     ):
         """
         Initialize enhanced search service.
@@ -138,6 +146,12 @@ class EnhancedSearchService:
         self.mmr_reranker = MMRReranker(lambda_param=default_lambda)
         self.evidence_linker = EvidenceLinker()
         self.default_top_k = default_top_k
+
+        # Simple in-memory cache for query embeddings (reduces repeated encoding latency).
+        self._query_cache_max_size = int(query_cache_max_size)
+        self._query_cache_ttl_s = float(query_cache_ttl_s)
+        self._query_cache: "OrderedDict[tuple[str, str], tuple[float, List[float]]]" = OrderedDict()
+        self._query_cache_lock = threading.Lock()
     
     def search(
         self,
@@ -177,33 +191,54 @@ class EnhancedSearchService:
         elif filters is None:
             filters = SearchFilters()
         
-        # Get query embedding
+        # Normalize / auto-detect modality for encoding.
+        requested_modality = (modality or "text").lower()
+        if requested_modality == "auto":
+            requested_modality = self._detect_modality(query)
+
+        # Get query embedding (cached)
         from bioflow.core.base import Modality
-        modality_enum = self._get_modality_enum(modality)
-        query_result = self.encoder.encode(query, modality_enum)
-        query_embedding = query_result.vector
+        modality_enum = self._get_modality_enum(requested_modality)
+        query_embedding = self._get_query_embedding_cached(query, modality_enum)
+        query_dim = len(query_embedding)
         
         # Execute vector search
         # Fetch more results if using MMR (for better diversity)
         fetch_limit = top_k * 3 if use_mmr else top_k
+        need_vectors = bool(use_mmr or include_embeddings)
         
         raw_results = self._execute_search(
             query_embedding=query_embedding,
             collection=collection,
             limit=fetch_limit,
             filters=filters,
+            with_vectors=need_vectors,
         )
+
+        # Post-filters that are difficult to express robustly in Qdrant filters.
+        raw_results = self._apply_post_filters(raw_results, filters)
         
         total_found = len(raw_results)
         
         # Apply MMR if requested
         if use_mmr and len(raw_results) > 1:
             # Get embeddings for MMR
-            embeddings = self._get_result_embeddings(raw_results) if include_embeddings or use_mmr else None
+            embeddings = (
+                self._get_result_embeddings(raw_results, expected_dim=query_dim)
+                if include_embeddings or use_mmr
+                else None
+            )
+
+            effective_lambda = (
+                float(lambda_param)
+                if lambda_param is not None
+                else float(self.mmr_reranker.lambda_param)
+            )
             
-            mmr_results = self.mmr_reranker.rerank(
+            mmr_results = mmr_rerank(
                 results=raw_results,
                 query_embedding=query_embedding,
+                lambda_param=effective_lambda,
                 embeddings=embeddings,
                 top_k=top_k,
             )
@@ -226,7 +261,7 @@ class EnhancedSearchService:
         return SearchResponse(
             results=enhanced_results,
             query=query,
-            modality=modality,
+            modality=requested_modality,
             total_found=total_found,
             returned=len(enhanced_results),
             diversity_score=diversity_score,
@@ -314,38 +349,56 @@ class EnhancedSearchService:
         collection: str,
         limit: int,
         filters: SearchFilters,
+        with_vectors: bool,
     ) -> List[Dict[str, Any]]:
         """Execute search against Qdrant."""
-        from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
+        from qdrant_client.models import Filter, FieldCondition, MatchValue
         
         # Build filter conditions
-        conditions = []
+        must_conditions = []
         
         if filters.modality:
-            conditions.append(FieldCondition(
-                key="modality",
-                match=MatchValue(value=filters.modality)
-            ))
+            requested = str(filters.modality).lower()
+            if requested in ("molecule", "smiles"):
+                # Historical payloads may use "smiles". Treat both as molecule.
+                must_conditions.append(Filter(should=[
+                    FieldCondition(key="modality", match=MatchValue(value="molecule")),
+                    FieldCondition(key="modality", match=MatchValue(value="smiles")),
+                ]))
+            else:
+                must_conditions.append(FieldCondition(
+                    key="modality",
+                    match=MatchValue(value=requested)
+                ))
         
         if filters.source:
-            conditions.append(FieldCondition(
+            must_conditions.append(FieldCondition(
                 key="source",
                 match=MatchValue(value=filters.source)
             ))
         
         if filters.sources:
-            # Multiple sources - need OR logic
-            # Qdrant supports this via should conditions
-            pass  # TODO: Implement OR logic
+            must_conditions.append(Filter(should=[
+                FieldCondition(key="source", match=MatchValue(value=s))
+                for s in filters.sources
+            ]))
         
         if filters.organism:
-            conditions.append(FieldCondition(
+            must_conditions.append(FieldCondition(
                 key="organism",
                 match=MatchValue(value=filters.organism)
             ))
+
+        if filters.organism_id is not None:
+            must_conditions.append(FieldCondition(
+                key="organism_id",
+                match=MatchValue(value=filters.organism_id)
+            ))
         
         # Build filter
-        query_filter = Filter(must=conditions) if conditions else None
+        query_filter = None
+        if must_conditions:
+            query_filter = Filter(must=must_conditions)
         
         # Get client and search
         client = self.qdrant._get_client()
@@ -366,17 +419,20 @@ class EnhancedSearchService:
                     limit=limit,
                     query_filter=query_filter,
                     with_payload=True,
-                    with_vectors=True,  # Need vectors for MMR
+                    with_vectors=with_vectors,
                 ).points
                 
                 for r in results:
+                    payload_modality = r.payload.get('modality', 'unknown')
+                    # Normalize legacy modality value for UI consistency.
+                    normalized_modality = "molecule" if payload_modality == "smiles" else payload_modality
                     all_results.append({
                         'id': str(r.id),
                         'score': r.score,
                         'content': r.payload.get('content', ''),
-                        'modality': r.payload.get('modality', 'unknown'),
+                        'modality': normalized_modality,
                         'metadata': r.payload,
-                        'vector': r.vector,
+                        'vector': r.vector if with_vectors else None,
                     })
             except Exception as e:
                 logger.warning(f"Search in {coll} failed: {e}")
@@ -385,7 +441,11 @@ class EnhancedSearchService:
         all_results.sort(key=lambda x: x['score'], reverse=True)
         return all_results[:limit]
     
-    def _get_result_embeddings(self, results: List[Dict]) -> List[List[float]]:
+    def _get_result_embeddings(
+        self,
+        results: List[Dict],
+        expected_dim: int,
+    ) -> List[List[float]]:
         """Extract embeddings from results."""
         embeddings = []
         for r in results:
@@ -397,9 +457,54 @@ class EnhancedSearchService:
                 embeddings.append(vec)
             else:
                 # Missing vector - use zeros (will have low similarity)
-                embeddings.append([0.0] * 768)
+                embeddings.append([0.0] * expected_dim)
         return embeddings
     
+    def _extract_source(self, metadata: Dict[str, Any]) -> str:
+        """
+        Extract source from metadata with fallback chain.
+        
+        Tries multiple field names and patterns to ensure traceability.
+        """
+        if not metadata:
+            return "unknown"
+        
+        # Direct source fields (priority order)
+        for field in ["source", "database", "origin", "db", "data_source"]:
+            val = metadata.get(field)
+            if val and isinstance(val, str):
+                return val.lower()
+        
+        # Check for known identifiers that imply source
+        if metadata.get("pmid") or metadata.get("pubmed_id"):
+            return "pubmed"
+        if metadata.get("uniprot_id") or metadata.get("accession"):
+            return "uniprot"
+        if metadata.get("chembl_id"):
+            return "chembl"
+        if metadata.get("drugbank_id"):
+            return "drugbank"
+        if metadata.get("pdb_id"):
+            return "pdb"
+        
+        # Check for URL patterns
+        url = metadata.get("url", "")
+        if "pubmed" in url.lower() or "ncbi.nlm.nih.gov" in url.lower():
+            return "pubmed"
+        if "uniprot" in url.lower():
+            return "uniprot"
+        if "chembl" in url.lower():
+            return "chembl"
+        
+        # Check modality hints
+        modality = metadata.get("modality", "")
+        if modality in ("protein", "sequence"):
+            return "protein_db"
+        if modality in ("molecule", "smiles", "compound"):
+            return "molecule_db"
+        
+        return "unknown"
+    
     def _mmr_to_enhanced(
         self,
         mmr_results: List[MMRResult],
@@ -417,7 +522,7 @@ class EnhancedSearchService:
                 modality=r.modality,
                 metadata=r.metadata,
                 evidence_links=[],  # Added later
-                source_type=r.metadata.get('source', 'unknown'),
+                source_type=self._extract_source(r.metadata),
                 citation=None,  # Added later
                 rank=i + 1,
             ))
@@ -427,6 +532,7 @@ class EnhancedSearchService:
         """Convert raw results to enhanced results."""
         enhanced = []
         for i, r in enumerate(results):
+            metadata = r.get('metadata', {})
             enhanced.append(EnhancedSearchResult(
                 id=r.get('id', ''),
                 score=r.get('score', 0),
@@ -434,9 +540,9 @@ class EnhancedSearchService:
                 diversity_penalty=None,
                 content=r.get('content', ''),
                 modality=r.get('modality', 'unknown'),
-                metadata=r.get('metadata', {}),
+                metadata=metadata,
                 evidence_links=[],
-                source_type=r.get('metadata', {}).get('source', 'unknown'),
+                source_type=self._extract_source(metadata),
                 citation=None,
                 rank=i + 1,
             ))
@@ -469,6 +575,107 @@ class EnhancedSearchService:
             "protein": Modality.PROTEIN,
         }
         return mapping.get(modality.lower(), Modality.TEXT)
+
+    def _get_query_embedding_cached(self, query: str, modality_enum) -> List[float]:
+        import time
+
+        key = (getattr(modality_enum, "value", str(modality_enum)), str(query))
+        now = time.time()
+
+        with self._query_cache_lock:
+            cached = self._query_cache.get(key)
+            if cached is not None:
+                ts, vec = cached
+                if (now - ts) <= self._query_cache_ttl_s:
+                    self._query_cache.move_to_end(key)
+                    return vec
+                self._query_cache.pop(key, None)
+
+        query_result = self.encoder.encode(query, modality_enum)
+        vec = query_result.vector.tolist() if hasattr(query_result.vector, "tolist") else list(query_result.vector)
+
+        with self._query_cache_lock:
+            self._query_cache[key] = (now, vec)
+            self._query_cache.move_to_end(key)
+            while len(self._query_cache) > self._query_cache_max_size:
+                self._query_cache.popitem(last=False)
+
+        return vec
+
+    def _detect_modality(self, query: str) -> str:
+        """
+        Heuristically detect modality from raw query.
+
+        Notes:
+        - "protein": long sequences of amino-acid letters
+        - "molecule": SMILES-like strings (bond symbols, brackets, digits, etc.)
+        - otherwise: "text"
+        """
+        q = (query or "").strip()
+        if not q:
+            return "text"
+
+        # Protein FASTA / sequence heuristic (AAs + length)
+        seq = q.replace("\n", "").replace(" ", "")
+        if len(seq) >= 25 and re.fullmatch(r"[ACDEFGHIKLMNPQRSTVWYBXZJUO]+", seq, flags=re.IGNORECASE):
+            return "protein"
+
+        # SMILES heuristic: contains typical SMILES tokens and at least one letter/digit.
+        if re.search(r"[\[\]=#@\\/()%0-9]", q) and re.search(r"[A-Za-z]", q):
+            return "molecule"
+
+        return "text"
+
+    def _apply_post_filters(
+        self,
+        results: List[Dict[str, Any]],
+        filters: SearchFilters,
+    ) -> List[Dict[str, Any]]:
+        """Apply filters that are not reliably expressed in Qdrant payload filters."""
+        if not results:
+            return results
+
+        year_min = filters.year_min
+        year_max = filters.year_max
+        keywords = filters.keywords or []
+
+        def extract_year(payload: Dict[str, Any]) -> Optional[int]:
+            for key in ("year", "publication_year", "pub_year", "date"):
+                v = payload.get(key)
+                if v is None:
+                    continue
+                if isinstance(v, int):
+                    return v
+                if isinstance(v, str):
+                    m = re.search(r"(19\d{2}|20\d{2})", v)
+                    if m:
+                        try:
+                            return int(m.group(1))
+                        except Exception:
+                            return None
+            return None
+
+        filtered: List[Dict[str, Any]] = []
+        for r in results:
+            payload = r.get("metadata", {}) or {}
+
+            if year_min is not None or year_max is not None:
+                y = extract_year(payload)
+                if y is None:
+                    continue
+                if year_min is not None and y < year_min:
+                    continue
+                if year_max is not None and y > year_max:
+                    continue
+
+            if keywords:
+                hay = (r.get("content", "") or "").lower() + " " + str(payload).lower()
+                if not all(str(kw).lower() in hay for kw in keywords):
+                    continue
+
+            filtered.append(r)
+
+        return filtered
     
     def _filters_to_dict(self, filters: SearchFilters) -> Dict[str, Any]:
         """Convert filters to dictionary."""

bioflow/ui/__init__.py

@@ -1,15 +0,0 @@
-"""
-BioFlow UI Package
-===================
-
-Modern Streamlit-based interface for the BioFlow platform.
-
-Pages:
-- Home: Dashboard with key metrics and quick actions
-- Discovery: Drug discovery pipeline interface
-- Explorer: Vector space visualization
-- Data: Data ingestion and management
-- Settings: Configuration and preferences
-"""
-
-__version__ = "2.0.0"

bioflow/ui/app.py

@@ -1,61 +0,0 @@
-"""
-BioFlow - AI-Powered Drug Discovery Platform
-==============================================
-Main application entry point.
-"""
-
-import streamlit as st
-import sys
-import os
-
-# Setup path for imports
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
-
-from bioflow.ui.config import get_css
-from bioflow.ui.components import side_nav
-from bioflow.ui.pages import home, discovery, explorer, data, settings
-
-
-def main():
-    """Main application."""
-    
-    # Page config
-    st.set_page_config(
-        page_title="BioFlow",
-        page_icon="🧬",
-        layout="wide",
-        initial_sidebar_state="collapsed",
-    )
-    
-    # Inject custom CSS
-    st.markdown(get_css(), unsafe_allow_html=True)
-    
-    # Initialize session state
-    if "current_page" not in st.session_state:
-        st.session_state.current_page = "home"
-    
-    # Layout with left navigation
-    nav_col, content_col = st.columns([1, 3.6], gap="large")
-    
-    with nav_col:
-        selected = side_nav(active_page=st.session_state.current_page)
-    
-    if selected != st.session_state.current_page:
-        st.session_state.current_page = selected
-        st.rerun()
-    
-    with content_col:
-        page_map = {
-            "home": home.render,
-            "discovery": discovery.render,
-            "explorer": explorer.render,
-            "data": data.render,
-            "settings": settings.render,
-        }
-        
-        render_fn = page_map.get(st.session_state.current_page, home.render)
-        render_fn()
-
-
-if __name__ == "__main__":
-    main()

bioflow/ui/components.py

@@ -1,481 +0,0 @@
-"""
-BioFlow UI - Components Library
-================================
-Reusable, modern UI components for Streamlit.
-"""
-
-import streamlit as st
-from typing import List, Dict, Any, Optional, Callable
-import plotly.express as px
-import plotly.graph_objects as go
-
-# Import colors
-import sys
-import os
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
-from bioflow.ui.config import COLORS
-
-
-# === Navigation ===
-
-def side_nav(active_page: str = "home") -> str:
-    """Left vertical navigation list. Returns the selected page key."""
-
-    nav_items = [
-        ("home", "🏠", "Home"),
-        ("discovery", "🔬", "Discovery"),
-        ("explorer", "🧬", "Explorer"),
-        ("data", "📊", "Data"),
-        ("settings", "⚙️", "Settings"),
-    ]
-
-    st.markdown(
-        f"""
-        <div class="nav-rail">
-            <div class="nav-brand">
-                <div class="nav-logo">🧬</div>
-                <div class="nav-title">Bio<span>Flow</span></div>
-            </div>
-            <div class="nav-section">Navigation</div>
-        </div>
-        """,
-        unsafe_allow_html=True,
-    )
-
-    label_map = {key: f"{icon} {label}" for key, icon, label in nav_items}
-    options = [item[0] for item in nav_items]
-
-    selected = st.radio(
-        "Navigation",
-        options=options,
-        index=options.index(active_page),
-        format_func=lambda x: label_map.get(x, x),
-        key="nav_radio",
-        label_visibility="collapsed",
-    )
-
-    return selected
-
-
-# === Page Structure ===
-
-def page_header(title: str, subtitle: str = "", icon: str = ""):
-    """Page header with title and optional subtitle."""
-    header_html = f"""
-    <div style="margin-bottom: 2rem;">
-        <h1 style="display: flex; align-items: center; gap: 0.75rem; margin: 0;">
-            {f'<span style="font-size: 2rem;">{icon}</span>' if icon else ''}
-            {title}
-        </h1>
-        {f'<p style="margin-top: 0.5rem; font-size: 1rem; color: {COLORS.text_muted};">{subtitle}</p>' if subtitle else ''}
-    </div>
-    """
-    st.markdown(header_html, unsafe_allow_html=True)
-
-
-def section_header(title: str, icon: str = "", link_text: str = "", link_action: Optional[Callable] = None):
-    """Section header with optional action link."""
-    col1, col2 = st.columns([4, 1])
-    
-    with col1:
-        st.markdown(f"""
-        <div class="section-title">
-            {f'<span>{icon}</span>' if icon else ''}
-            {title}
-        </div>
-        """, unsafe_allow_html=True)
-    
-    with col2:
-        if link_text:
-            if st.button(link_text, key=f"section_{title}", use_container_width=True):
-                if link_action:
-                    link_action()
-
-
-def divider():
-    """Visual divider."""
-    st.markdown('<div class="divider"></div>', unsafe_allow_html=True)
-
-
-def spacer(height: str = "1rem"):
-    """Vertical spacer."""
-    st.markdown(f'<div style="height: {height};"></div>', unsafe_allow_html=True)
-
-
-# === Metrics ===
-
-def metric_card(
-    value: str,
-    label: str,
-    icon: str = "📊",
-    change: Optional[str] = None,
-    change_type: str = "up",
-    color: str = COLORS.primary
-):
-    """Single metric card with icon and optional trend."""
-    bg_color = color.replace(")", ", 0.15)").replace("rgb", "rgba") if "rgb" in color else f"{color}22"
-    change_html = ""
-    if change:
-        arrow = "↑" if change_type == "up" else "↓"
-        change_html = f'<div class="metric-change {change_type}">{arrow} {change}</div>'
-    
-    st.markdown(f"""
-    <div class="metric">
-        <div class="metric-icon" style="background: {bg_color}; color: {color};">
-            {icon}
-        </div>
-        <div class="metric-value">{value}</div>
-        <div class="metric-label">{label}</div>
-        {change_html}
-    </div>
-    """, unsafe_allow_html=True)
-
-
-def metric_row(metrics: List[Dict[str, Any]]):
-    """Row of metric cards."""
-    cols = st.columns(len(metrics))
-    for col, metric in zip(cols, metrics):
-        with col:
-            metric_card(**metric)
-
-
-# === Quick Actions ===
-
-def quick_action(icon: str, title: str, description: str, key: str) -> bool:
-    """Single quick action card. Returns True if clicked."""
-    clicked = st.button(
-        f"{icon}  {title}",
-        key=key,
-        use_container_width=True,
-        help=description
-    )
-    return clicked
-
-
-def quick_actions_grid(actions: List[Dict[str, Any]], columns: int = 4) -> Optional[str]:
-    """Grid of quick action cards. Returns clicked action key or None."""
-    cols = st.columns(columns)
-    clicked_key = None
-    
-    for i, action in enumerate(actions):
-        with cols[i % columns]:
-            st.markdown(f"""
-            <div class="quick-action">
-                <span class="quick-action-icon">{action['icon']}</span>
-                <div class="quick-action-title">{action['title']}</div>
-                <div class="quick-action-desc">{action.get('description', '')}</div>
-            </div>
-            """, unsafe_allow_html=True)
-            
-            if st.button("Select", key=action['key'], use_container_width=True):
-                clicked_key = action['key']
-    
-    return clicked_key
-
-
-# === Pipeline Progress ===
-
-def pipeline_progress(steps: List[Dict[str, Any]]):
-    """Visual pipeline with steps showing progress."""
-    html = '<div class="pipeline">'
-    
-    for i, step in enumerate(steps):
-        status = step.get('status', 'pending')
-        icon = step.get('icon', str(i + 1))
-        name = step.get('name', f'Step {i + 1}')
-        
-        # Display icon for completed steps
-        if status == 'done':
-            display = '✓'
-        elif status == 'active':
-            display = icon
-        else:
-            display = str(i + 1)
-        
-        html += f'''
-        <div class="step">
-            <div class="step-dot {status}">{display}</div>
-            <span class="step-name">{name}</span>
-        </div>
-        '''
-        
-        # Add connecting line (except after last step)
-        if i < len(steps) - 1:
-            line_status = 'done' if status == 'done' else ''
-            html += f'<div class="step-line {line_status}"></div>'
-    
-    html += '</div>'
-    st.markdown(html, unsafe_allow_html=True)
-
-
-# === Results ===
-
-def result_card(
-    title: str,
-    score: float,
-    properties: Dict[str, str] = None,
-    badges: List[str] = None,
-    key: str = ""
-) -> bool:
-    """Result card with score and properties. Returns True if clicked."""
-    
-    # Score color
-    if score >= 0.8:
-        score_class = "score-high"
-    elif score >= 0.5:
-        score_class = "score-med"
-    else:
-        score_class = "score-low"
-    
-    # Properties HTML
-    props_html = ""
-    if properties:
-        props_html = '<div style="display: flex; gap: 1rem; margin-top: 0.75rem; flex-wrap: wrap;">'
-        for k, v in properties.items():
-            props_html += f'''
-            <div style="font-size: 0.8125rem;">
-                <span style="color: {COLORS.text_muted};">{k}:</span>
-                <span style="color: {COLORS.text_secondary}; margin-left: 0.25rem;">{v}</span>
-            </div>
-            '''
-        props_html += '</div>'
-    
-    # Badges HTML
-    badges_html = ""
-    if badges:
-        badges_html = '<div style="display: flex; gap: 0.5rem; margin-top: 0.75rem;">'
-        for b in badges:
-            badges_html += f'<span class="badge badge-primary">{b}</span>'
-        badges_html += '</div>'
-    
-    st.markdown(f"""
-    <div class="result">
-        <div style="display: flex; justify-content: space-between; align-items: flex-start;">
-            <div style="font-weight: 600; color: {COLORS.text_primary};">{title}</div>
-            <div class="{score_class}" style="font-size: 1.25rem; font-weight: 700;">{score:.1%}</div>
-        </div>
-        {props_html}
-        {badges_html}
-    </div>
-    """, unsafe_allow_html=True)
-    
-    return st.button("View Details", key=key, use_container_width=True) if key else False
-
-
-def results_list(results: List[Dict[str, Any]], empty_message: str = "No results found"):
-    """List of result cards."""
-    if not results:
-        empty_state(icon="🔍", title="No Results", description=empty_message)
-        return
-    
-    for i, result in enumerate(results):
-        result_card(
-            title=result.get('title', f'Result {i + 1}'),
-            score=result.get('score', 0),
-            properties=result.get('properties'),
-            badges=result.get('badges'),
-            key=f"result_{i}"
-        )
-        spacer("0.75rem")
-
-
-# === Charts ===
-
-def bar_chart(data: Dict[str, float], title: str = "", height: int = 300):
-    """Styled bar chart."""
-    fig = go.Figure(data=[
-        go.Bar(
-            x=list(data.keys()),
-            y=list(data.values()),
-            marker_color=COLORS.primary,
-            marker_line_width=0,
-        )
-    ])
-    
-    fig.update_layout(
-        title=title,
-        paper_bgcolor='rgba(0,0,0,0)',
-        plot_bgcolor='rgba(0,0,0,0)',
-        font=dict(family="Inter", color=COLORS.text_secondary),
-        height=height,
-        margin=dict(l=40, r=20, t=40, b=40),
-        xaxis=dict(
-            showgrid=False,
-            showline=True,
-            linecolor=COLORS.border,
-        ),
-        yaxis=dict(
-            showgrid=True,
-            gridcolor=COLORS.border,
-            showline=False,
-        ),
-    )
-    
-    st.plotly_chart(fig, use_container_width=True)
-
-
-def scatter_chart(x: List, y: List, labels: List = None, title: str = "", height: int = 400):
-    """Styled scatter plot."""
-    fig = go.Figure(data=[
-        go.Scatter(
-            x=x,
-            y=y,
-            mode='markers',
-            marker=dict(
-                size=10,
-                color=COLORS.primary,
-                opacity=0.7,
-            ),
-            text=labels,
-            hovertemplate='<b>%{text}</b><br>X: %{x}<br>Y: %{y}<extra></extra>' if labels else None,
-        )
-    ])
-    
-    fig.update_layout(
-        title=title,
-        paper_bgcolor='rgba(0,0,0,0)',
-        plot_bgcolor='rgba(0,0,0,0)',
-        font=dict(family="Inter", color=COLORS.text_secondary),
-        height=height,
-        margin=dict(l=40, r=20, t=40, b=40),
-        xaxis=dict(
-            showgrid=True,
-            gridcolor=COLORS.border,
-            showline=True,
-            linecolor=COLORS.border,
-        ),
-        yaxis=dict(
-            showgrid=True,
-            gridcolor=COLORS.border,
-            showline=True,
-            linecolor=COLORS.border,
-        ),
-    )
-    
-    st.plotly_chart(fig, use_container_width=True)
-
-
-def heatmap(data: List[List[float]], x_labels: List[str], y_labels: List[str], title: str = "", height: int = 400):
-    """Styled heatmap."""
-    fig = go.Figure(data=[
-        go.Heatmap(
-            z=data,
-            x=x_labels,
-            y=y_labels,
-            colorscale=[
-                [0, COLORS.bg_hover],
-                [0.5, COLORS.primary],
-                [1, COLORS.cyan],
-            ],
-        )
-    ])
-    
-    fig.update_layout(
-        title=title,
-        paper_bgcolor='rgba(0,0,0,0)',
-        plot_bgcolor='rgba(0,0,0,0)',
-        font=dict(family="Inter", color=COLORS.text_secondary),
-        height=height,
-        margin=dict(l=80, r=20, t=40, b=60),
-    )
-    
-    st.plotly_chart(fig, use_container_width=True)
-
-
-# === Data Display ===
-
-def data_table(data: List[Dict], columns: List[str] = None):
-    """Styled data table."""
-    import pandas as pd
-    df = pd.DataFrame(data)
-    if columns:
-        df = df[columns]
-    st.dataframe(df, use_container_width=True, hide_index=True)
-
-
-# === States ===
-
-def empty_state(icon: str = "📭", title: str = "No Data", description: str = ""):
-    """Empty state placeholder."""
-    st.markdown(f"""
-    <div class="empty">
-        <div class="empty-icon">{icon}</div>
-        <div class="empty-title">{title}</div>
-        <div class="empty-desc">{description}</div>
-    </div>
-    """, unsafe_allow_html=True)
-
-
-def loading_state(message: str = "Loading..."):
-    """Loading state with spinner."""
-    st.markdown(f"""
-    <div class="loading">
-        <div class="spinner"></div>
-        <div class="loading-text">{message}</div>
-    </div>
-    """, unsafe_allow_html=True)
-
-
-# === Molecule Display ===
-
-def molecule_2d(smiles: str, size: int = 200):
-    """Display 2D molecule structure from SMILES."""
-    try:
-        from rdkit import Chem
-        from rdkit.Chem import Draw
-        import base64
-        from io import BytesIO
-        
-        mol = Chem.MolFromSmiles(smiles)
-        if mol:
-            img = Draw.MolToImage(mol, size=(size, size))
-            buffered = BytesIO()
-            img.save(buffered, format="PNG")
-            img_str = base64.b64encode(buffered.getvalue()).decode()
-            
-            st.markdown(f"""
-            <div class="mol-container">
-                <img src="data:image/png;base64,{img_str}" alt="Molecule" style="max-width: 100%; height: auto;">
-            </div>
-            """, unsafe_allow_html=True)
-        else:
-            st.warning("Invalid SMILES")
-    except ImportError:
-        st.info(f"SMILES: `{smiles}`")
-
-
-# === Evidence & Links ===
-
-def evidence_row(items: List[Dict[str, str]]):
-    """Row of evidence/source links."""
-    html = '<div style="display: flex; gap: 0.5rem; flex-wrap: wrap; margin-top: 0.75rem;">'
-    for item in items:
-        icon = item.get('icon', '📄')
-        label = item.get('label', 'Source')
-        url = item.get('url', '#')
-        html += f'''
-        <a href="{url}" target="_blank" class="evidence">
-            <span>{icon}</span>
-            <span>{label}</span>
-        </a>
-        '''
-    html += '</div>'
-    st.markdown(html, unsafe_allow_html=True)
-
-
-# === Badges ===
-
-def badge(text: str, variant: str = "primary"):
-    """Inline badge component."""
-    st.markdown(f'<span class="badge badge-{variant}">{text}</span>', unsafe_allow_html=True)
-
-
-def badge_row(badges: List[Dict[str, str]]):
-    """Row of badges."""
-    html = '<div style="display: flex; gap: 0.5rem; flex-wrap: wrap;">'
-    for b in badges:
-        text = b.get('text', '')
-        variant = b.get('variant', 'primary')
-        html += f'<span class="badge badge-{variant}">{text}</span>'
-    html += '</div>'
-    st.markdown(html, unsafe_allow_html=True)

bioflow/ui/config.py

@@ -1,583 +0,0 @@
-"""
-BioFlow UI - Modern Design System
-==================================
-Clean, minimal, and highly usable interface.
-"""
-
-from dataclasses import dataclass
-
-
-@dataclass
-class Colors:
-    """Color palette - Modern dark theme."""
-    # Primary
-    primary: str = "#8B5CF6"
-    primary_hover: str = "#A78BFA"
-    primary_muted: str = "rgba(139, 92, 246, 0.15)"
-    
-    # Accents
-    cyan: str = "#22D3EE"
-    emerald: str = "#34D399"
-    amber: str = "#FBBF24"
-    rose: str = "#FB7185"
-    
-    # Backgrounds
-    bg_app: str = "#0C0E14"
-    bg_surface: str = "#14161E"
-    bg_elevated: str = "#1C1F2B"
-    bg_hover: str = "#252836"
-    
-    # Text
-    text_primary: str = "#F8FAFC"
-    text_secondary: str = "#A1A7BB"
-    text_muted: str = "#6B7280"
-    
-    # Borders
-    border: str = "#2A2D3A"
-    border_hover: str = "#3F4354"
-    
-    # Status
-    success: str = "#10B981"
-    warning: str = "#F59E0B"
-    error: str = "#EF4444"
-    info: str = "#3B82F6"
-
-
-COLORS = Colors()
-
-
-def get_css() -> str:
-    """Minimalist, professional CSS using string concatenation to avoid f-string issues."""
-    
-    css = """
-    <style>
-    @import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap');
-    
-    :root {
-        --primary: """ + COLORS.primary + """;
-        --bg-app: """ + COLORS.bg_app + """;
-        --bg-surface: """ + COLORS.bg_surface + """;
-        --text: """ + COLORS.text_primary + """;
-        --text-muted: """ + COLORS.text_muted + """;
-        --border: """ + COLORS.border + """;
-        --radius: 12px;
-        --transition: 150ms ease;
-    }
-    
-    .stApp {
-        background: """ + COLORS.bg_app + """;
-        font-family: 'Inter', sans-serif;
-    }
-    
-    #MainMenu, footer, header { visibility: hidden; }
-    .stDeployButton { display: none; }
-    
-    ::-webkit-scrollbar { width: 6px; height: 6px; }
-    ::-webkit-scrollbar-track { background: transparent; }
-    ::-webkit-scrollbar-thumb { background: """ + COLORS.border + """; border-radius: 3px; }
-    ::-webkit-scrollbar-thumb:hover { background: """ + COLORS.border_hover + """; }
-    
-    section[data-testid="stSidebar"] { display: none !important; }
-    
-    h1, h2, h3 {
-        font-weight: 600;
-        color: """ + COLORS.text_primary + """;
-        letter-spacing: -0.025em;
-    }
-    
-    h1 { font-size: 1.875rem; margin-bottom: 0.5rem; }
-    h2 { font-size: 1.5rem; }
-    h3 { font-size: 1.125rem; }
-    
-    p { color: """ + COLORS.text_secondary + """; line-height: 1.6; }
-    
-    .card {
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: var(--radius);
-        padding: 1.25rem;
-    }
-    
-    .metric {
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: var(--radius);
-        padding: 1.25rem;
-        transition: border-color var(--transition);
-    }
-    
-    .metric:hover { border-color: """ + COLORS.primary + """; }
-    
-    .metric-icon {
-        width: 44px;
-        height: 44px;
-        border-radius: 10px;
-        display: flex;
-        align-items: center;
-        justify-content: center;
-        font-size: 1.375rem;
-        margin-bottom: 1rem;
-    }
-    
-    .metric-value {
-        font-size: 2rem;
-        font-weight: 700;
-        color: """ + COLORS.text_primary + """;
-        line-height: 1;
-    }
-    
-    .metric-label {
-        font-size: 0.875rem;
-        color: """ + COLORS.text_muted + """;
-        margin-top: 0.375rem;
-    }
-    
-    .metric-change {
-        display: inline-flex;
-        align-items: center;
-        font-size: 0.75rem;
-        font-weight: 500;
-        padding: 0.25rem 0.5rem;
-        border-radius: 6px;
-        margin-top: 0.5rem;
-    }
-    
-    .metric-change.up { background: rgba(16, 185, 129, 0.15); color: """ + COLORS.success + """; }
-    .metric-change.down { background: rgba(239, 68, 68, 0.15); color: """ + COLORS.error + """; }
-    
-    .stButton > button {
-        font-family: 'Inter', sans-serif;
-        font-weight: 500;
-        font-size: 0.875rem;
-        border-radius: 8px;
-        padding: 0.625rem 1.25rem;
-        transition: all var(--transition);
-        border: none;
-    }
-    
-    .stTextInput input,
-    .stTextArea textarea,
-    .stSelectbox > div > div {
-        background: """ + COLORS.bg_app + """ !important;
-        border: 1px solid """ + COLORS.border + """ !important;
-        border-radius: 10px !important;
-        color: """ + COLORS.text_primary + """ !important;
-        font-family: 'Inter', sans-serif !important;
-    }
-    
-    .stTextInput input:focus,
-    .stTextArea textarea:focus {
-        border-color: """ + COLORS.primary + """ !important;
-        box-shadow: 0 0 0 3px """ + COLORS.primary_muted + """ !important;
-    }
-    
-    .stTabs [data-baseweb="tab-list"] {
-        gap: 0;
-        background: """ + COLORS.bg_surface + """;
-        border-radius: 10px;
-        padding: 4px;
-        border: 1px solid """ + COLORS.border + """;
-    }
-    
-    .stTabs [data-baseweb="tab"] {
-        height: auto;
-        padding: 0.625rem 1.25rem;
-        border-radius: 8px;
-        font-weight: 500;
-        font-size: 0.875rem;
-        color: """ + COLORS.text_muted + """;
-        background: transparent;
-    }
-    
-    .stTabs [aria-selected="true"] {
-        background: """ + COLORS.primary + """ !important;
-        color: white !important;
-    }
-    
-    .stTabs [data-baseweb="tab-highlight"],
-    .stTabs [data-baseweb="tab-border"] { display: none; }
-    
-    .pipeline {
-        display: flex;
-        align-items: center;
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: var(--radius);
-        padding: 1.5rem;
-        gap: 0;
-    }
-    
-    .step {
-        display: flex;
-        flex-direction: column;
-        align-items: center;
-        gap: 0.5rem;
-        flex: 1;
-    }
-    
-    .step-dot {
-        width: 44px;
-        height: 44px;
-        border-radius: 50%;
-        display: flex;
-        align-items: center;
-        justify-content: center;
-        font-size: 1.125rem;
-        font-weight: 600;
-        transition: all var(--transition);
-    }
-    
-    .step-dot.pending {
-        background: """ + COLORS.bg_hover + """;
-        color: """ + COLORS.text_muted + """;
-        border: 2px dashed """ + COLORS.border_hover + """;
-    }
-    
-    .step-dot.active {
-        background: """ + COLORS.primary + """;
-        color: white;
-        box-shadow: 0 0 24px rgba(139, 92, 246, 0.5);
-    }
-    
-    .step-dot.done {
-        background: """ + COLORS.emerald + """;
-        color: white;
-    }
-    
-    .step-name {
-        font-size: 0.75rem;
-        font-weight: 500;
-        color: """ + COLORS.text_muted + """;
-    }
-    
-    .step-line {
-        flex: 0.6;
-        height: 2px;
-        background: """ + COLORS.border + """;
-    }
-    
-    .step-line.done { background: """ + COLORS.emerald + """; }
-    
-    .result {
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: var(--radius);
-        padding: 1.25rem;
-        transition: all var(--transition);
-        cursor: pointer;
-    }
-    
-    .result:hover {
-        border-color: """ + COLORS.primary + """;
-        transform: translateY(-2px);
-        box-shadow: 0 8px 24px rgba(0, 0, 0, 0.2);
-    }
-    
-    .score-high { color: """ + COLORS.emerald + """; }
-    .score-med { color: """ + COLORS.amber + """; }
-    .score-low { color: """ + COLORS.rose + """; }
-    
-    .badge {
-        display: inline-flex;
-        align-items: center;
-        padding: 0.25rem 0.625rem;
-        border-radius: 6px;
-        font-size: 0.6875rem;
-        font-weight: 600;
-        text-transform: uppercase;
-    }
-    
-    .badge-primary { background: """ + COLORS.primary_muted + """; color: """ + COLORS.primary + """; }
-    .badge-success { background: rgba(16, 185, 129, 0.15); color: """ + COLORS.success + """; }
-    .badge-warning { background: rgba(245, 158, 11, 0.15); color: """ + COLORS.warning + """; }
-    .badge-error { background: rgba(239, 68, 68, 0.15); color: """ + COLORS.error + """; }
-    
-    .quick-action {
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: var(--radius);
-        padding: 1.5rem;
-        text-align: center;
-        cursor: pointer;
-        transition: all var(--transition);
-    }
-    
-    .quick-action:hover {
-        border-color: """ + COLORS.primary + """;
-        transform: translateY(-4px);
-        box-shadow: 0 12px 32px rgba(0, 0, 0, 0.25);
-    }
-    
-    .quick-action-icon {
-        font-size: 2.5rem;
-        margin-bottom: 0.75rem;
-        display: block;
-    }
-    
-    .quick-action-title {
-        font-size: 0.9375rem;
-        font-weight: 600;
-        color: """ + COLORS.text_primary + """;
-    }
-    
-    .quick-action-desc {
-        font-size: 0.8125rem;
-        color: """ + COLORS.text_muted + """;
-        margin-top: 0.25rem;
-    }
-    
-    .section-header {
-        display: flex;
-        align-items: center;
-        justify-content: space-between;
-        margin-bottom: 1rem;
-    }
-    
-    .section-title {
-        font-size: 1rem;
-        font-weight: 600;
-        color: """ + COLORS.text_primary + """;
-        display: flex;
-        align-items: center;
-        gap: 0.5rem;
-    }
-    
-    .section-link {
-        font-size: 0.8125rem;
-        color: """ + COLORS.primary + """;
-        cursor: pointer;
-    }
-    
-    .section-link:hover { text-decoration: underline; }
-    
-    .empty {
-        display: flex;
-        flex-direction: column;
-        align-items: center;
-        justify-content: center;
-        padding: 4rem 2rem;
-        text-align: center;
-    }
-    
-    .empty-icon { font-size: 3.5rem; margin-bottom: 1rem; opacity: 0.4; }
-    .empty-title { font-size: 1.125rem; font-weight: 600; color: """ + COLORS.text_primary + """; }
-    .empty-desc { font-size: 0.9375rem; color: """ + COLORS.text_muted + """; max-width: 320px; margin-top: 0.5rem; }
-    
-    .loading {
-        display: flex;
-        flex-direction: column;
-        align-items: center;
-        padding: 3rem;
-    }
-    
-    .spinner {
-        width: 40px;
-        height: 40px;
-        border: 3px solid """ + COLORS.border + """;
-        border-top-color: """ + COLORS.primary + """;
-        border-radius: 50%;
-        animation: spin 0.8s linear infinite;
-    }
-    
-    @keyframes spin { to { transform: rotate(360deg); } }
-    
-    .loading-text {
-        margin-top: 1rem;
-        color: """ + COLORS.text_muted + """;
-        font-size: 0.875rem;
-    }
-    
-    .stProgress > div > div > div {
-        background: linear-gradient(90deg, """ + COLORS.primary + """ 0%, """ + COLORS.cyan + """ 100%);
-        border-radius: 4px;
-    }
-    
-    .stProgress > div > div {
-        background: """ + COLORS.bg_hover + """;
-        border-radius: 4px;
-    }
-    
-    .divider {
-        height: 1px;
-        background: """ + COLORS.border + """;
-        margin: 1.5rem 0;
-    }
-    
-    .mol-container {
-        background: white;
-        border-radius: 10px;
-        padding: 0.75rem;
-        display: flex;
-        align-items: center;
-        justify-content: center;
-    }
-    
-    .evidence {
-        display: inline-flex;
-        align-items: center;
-        gap: 0.375rem;
-        padding: 0.5rem 0.75rem;
-        background: """ + COLORS.bg_app + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: 8px;
-        font-size: 0.8125rem;
-        color: """ + COLORS.text_secondary + """;
-        transition: all var(--transition);
-        text-decoration: none;
-    }
-    
-    .evidence:hover {
-        border-color: """ + COLORS.primary + """;
-        color: """ + COLORS.primary + """;
-    }
-    
-    .stAlert { border-radius: 10px; border: none; }
-    
-    .stDataFrame {
-        border-radius: var(--radius);
-        overflow: hidden;
-        border: 1px solid """ + COLORS.border + """;
-    }
-
-    .block-container {
-        padding-top: 1.25rem;
-    }
-
-    .nav-rail {
-        position: sticky;
-        top: 1rem;
-        display: flex;
-        flex-direction: column;
-        gap: 0.75rem;
-        padding: 1rem;
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: 16px;
-        margin-bottom: 1rem;
-    }
-
-    .nav-brand {
-        display: flex;
-        align-items: center;
-        gap: 0.75rem;
-        padding-bottom: 0.5rem;
-        border-bottom: 1px solid """ + COLORS.border + """;
-    }
-
-    .nav-logo { font-size: 1.5rem; }
-
-    .nav-title {
-        font-size: 1.1rem;
-        font-weight: 700;
-        color: """ + COLORS.text_primary + """;
-    }
-
-    .nav-title span {
-        background: linear-gradient(135deg, """ + COLORS.primary + """ 0%, """ + COLORS.cyan + """ 100%);
-        -webkit-background-clip: text;
-        -webkit-text-fill-color: transparent;
-    }
-
-    .nav-section {
-        font-size: 0.75rem;
-        text-transform: uppercase;
-        letter-spacing: 0.08em;
-        color: """ + COLORS.text_muted + """;
-    }
-
-    div[data-testid="stRadio"] {
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: 16px;
-        padding: 0.75rem;
-    }
-
-    div[data-testid="stRadio"] div[role="radiogroup"] {
-        display: flex;
-        flex-direction: column;
-        gap: 0.5rem;
-        margin-top: 0.25rem;
-    }
-
-    div[data-testid="stRadio"] input {
-        display: none !important;
-    }
-
-    div[data-testid="stRadio"] label {
-        background: """ + COLORS.bg_app + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: 12px;
-        padding: 0.65rem 0.9rem;
-        font-weight: 500;
-        color: """ + COLORS.text_secondary + """;
-        transition: all var(--transition);
-        margin: 0 !important;
-    }
-
-    div[data-testid="stRadio"] label:hover {
-        border-color: """ + COLORS.primary + """;
-        color: """ + COLORS.text_primary + """;
-    }
-
-    div[data-testid="stRadio"] label:has(input:checked) {
-        background: """ + COLORS.primary + """;
-        border-color: """ + COLORS.primary + """;
-        color: white;
-        box-shadow: 0 8px 20px rgba(139, 92, 246, 0.25);
-    }
-
-    .hero {
-        position: relative;
-        background: linear-gradient(135deg, rgba(139, 92, 246, 0.12) 0%, rgba(34, 211, 238, 0.08) 100%);
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: 20px;
-        padding: 2.75rem;
-        overflow: hidden;
-    }
-
-    .hero-badge {
-        display: inline-flex;
-        align-items: center;
-        gap: 0.5rem;
-        padding: 0.35rem 0.75rem;
-        border-radius: 999px;
-        background: """ + COLORS.primary_muted + """;
-        color: """ + COLORS.primary + """;
-        font-size: 0.75rem;
-        font-weight: 600;
-        text-transform: uppercase;
-        letter-spacing: 0.08em;
-    }
-
-    .hero-title {
-        font-size: 2.25rem;
-        font-weight: 700;
-        color: """ + COLORS.text_primary + """;
-        margin-top: 1rem;
-        line-height: 1.1;
-    }
-
-    .hero-subtitle {
-        font-size: 1rem;
-        color: """ + COLORS.text_muted + """;
-        margin-top: 0.75rem;
-        max-width: 560px;
-    }
-
-    .hero-actions {
-        display: flex;
-        gap: 0.75rem;
-        margin-top: 1.5rem;
-        flex-wrap: wrap;
-    }
-
-    .hero-card {
-        background: """ + COLORS.bg_surface + """;
-        border: 1px solid """ + COLORS.border + """;
-        border-radius: 16px;
-        padding: 1.5rem;
-    }
-    </style>
-    """
-    
-    return css

bioflow/ui/pages/__init__.py

@@ -1,5 +0,0 @@
-"""Page exports."""
-
-from bioflow.ui.pages import home, discovery, explorer, data, settings
-
-__all__ = ["home", "discovery", "explorer", "data", "settings"]

bioflow/ui/pages/data.py

@@ -1,163 +0,0 @@
-"""
-BioFlow - Data Page
-===================
-Data management and upload.
-"""
-
-import streamlit as st
-import sys
-import os
-import pandas as pd
-import numpy as np
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
-
-from bioflow.ui.components import (
-    page_header, section_header, divider, spacer,
-    metric_card, data_table, empty_state
-)
-from bioflow.ui.config import COLORS
-
-
-def render():
-    """Render data page."""
-    
-    page_header("Data Management", "Upload, manage, and organize your datasets", "📊")
-    
-    # Stats Row
-    cols = st.columns(4)
-    
-    with cols[0]:
-        metric_card("5", "Datasets", "📁", color=COLORS.primary)
-    with cols[1]:
-        metric_card("24.5K", "Molecules", "🧪", color=COLORS.cyan)
-    with cols[2]:
-        metric_card("1.2K", "Proteins", "🧬", color=COLORS.emerald)
-    with cols[3]:
-        metric_card("156 MB", "Storage Used", "💾", color=COLORS.amber)
-    
-    spacer("2rem")
-    
-    # Tabs
-    tabs = st.tabs(["📁 Datasets", "📤 Upload", "🔧 Processing"])
-    
-    with tabs[0]:
-        section_header("Your Datasets", "📁")
-        
-        # Dataset list
-        datasets = [
-            {"name": "DrugBank Compounds", "type": "Molecules", "count": "12,450", "size": "45.2 MB", "updated": "2024-01-15"},
-            {"name": "ChEMBL Kinase Inhibitors", "type": "Molecules", "count": "8,234", "size": "32.1 MB", "updated": "2024-01-10"},
-            {"name": "Custom Protein Targets", "type": "Proteins", "count": "1,245", "size": "78.5 MB", "updated": "2024-01-08"},
-        ]
-        
-        for ds in datasets:
-            st.markdown(f"""
-            <div class="card" style="margin-bottom: 0.75rem;">
-                <div style="display: flex; justify-content: space-between; align-items: center;">
-                    <div>
-                        <div style="font-weight: 600; color: {COLORS.text_primary};">{ds["name"]}</div>
-                        <div style="display: flex; gap: 1.5rem; margin-top: 0.5rem;">
-                            <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">
-                                <span style="color: {COLORS.primary};">●</span> {ds["type"]}
-                            </span>
-                            <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">{ds["count"]} items</span>
-                            <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">{ds["size"]}</span>
-                            <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">Updated: {ds["updated"]}</span>
-                        </div>
-                    </div>
-                    <div style="display: flex; gap: 0.5rem;">
-                        <span class="badge badge-primary">{ds["type"]}</span>
-                    </div>
-                </div>
-            </div>
-            """, unsafe_allow_html=True)
-            
-            # Action buttons
-            btn_cols = st.columns([1, 1, 1, 4])
-            with btn_cols[0]:
-                st.button("View", key=f"view_{ds['name']}", use_container_width=True)
-            with btn_cols[1]:
-                st.button("Export", key=f"export_{ds['name']}", use_container_width=True)
-            with btn_cols[2]:
-                st.button("Delete", key=f"delete_{ds['name']}", use_container_width=True)
-            
-            spacer("0.5rem")
-    
-    with tabs[1]:
-        section_header("Upload New Data", "📤")
-        
-        # Upload area
-        st.markdown(f"""
-        <div style="
-            border: 2px dashed {COLORS.border};
-            border-radius: 16px;
-            padding: 3rem;
-            text-align: center;
-            background: {COLORS.bg_surface};
-        ">
-            <div style="font-size: 3rem; margin-bottom: 1rem;">📁</div>
-            <div style="font-size: 1.125rem; font-weight: 600; color: {COLORS.text_primary};">
-                Drag & drop files here
-            </div>
-            <div style="font-size: 0.875rem; color: {COLORS.text_muted}; margin-top: 0.5rem;">
-                or click to browse
-            </div>
-            <div style="font-size: 0.75rem; color: {COLORS.text_muted}; margin-top: 1rem;">
-                Supports: CSV, SDF, FASTA, PDB, JSON
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        uploaded_file = st.file_uploader(
-            "Upload file",
-            type=["csv", "sdf", "fasta", "pdb", "json"],
-            label_visibility="collapsed"
-        )
-        
-        if uploaded_file:
-            st.success(f"✓ File uploaded: {uploaded_file.name}")
-            
-            col1, col2 = st.columns(2)
-            with col1:
-                dataset_name = st.text_input("Dataset Name", value=uploaded_file.name.split('.')[0])
-            with col2:
-                data_type = st.selectbox("Data Type", ["Molecules", "Proteins", "Text"])
-            
-            if st.button("Process & Import", type="primary", use_container_width=True):
-                with st.spinner("Processing..."):
-                    import time
-                    time.sleep(2)
-                st.success("✓ Dataset imported successfully!")
-    
-    with tabs[2]:
-        section_header("Data Processing", "🔧")
-        
-        st.markdown(f"""
-        <div class="card">
-            <div style="font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 0.75rem;">
-                Available Operations
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        operations = [
-            {"icon": "🧹", "name": "Clean & Validate", "desc": "Remove duplicates, fix invalid structures"},
-            {"icon": "🔢", "name": "Compute Descriptors", "desc": "Calculate molecular properties and fingerprints"},
-            {"icon": "🧠", "name": "Generate Embeddings", "desc": "Create vector representations using AI models"},
-            {"icon": "🔗", "name": "Merge Datasets", "desc": "Combine multiple datasets with deduplication"},
-        ]
-        
-        for op in operations:
-            st.markdown(f"""
-            <div class="quick-action" style="margin-bottom: 0.75rem; text-align: left;">
-                <div style="display: flex; align-items: center; gap: 1rem;">
-                    <span style="font-size: 1.5rem;">{op["icon"]}</span>
-                    <div>
-                        <div style="font-weight: 600; color: {COLORS.text_primary};">{op["name"]}</div>
-                        <div style="font-size: 0.8125rem; color: {COLORS.text_muted};">{op["desc"]}</div>
-                    </div>
-                </div>
-            </div>
-            """, unsafe_allow_html=True)
-            st.button(f"Run {op['name']}", key=f"op_{op['name']}", use_container_width=True)

bioflow/ui/pages/discovery.py

@@ -1,165 +0,0 @@
-"""
-BioFlow - Discovery Page
-========================
-Drug discovery pipeline interface.
-"""
-
-import streamlit as st
-import sys
-import os
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
-
-from bioflow.ui.components import (
-    page_header, section_header, divider, spacer,
-    pipeline_progress, bar_chart, empty_state, loading_state
-)
-from bioflow.ui.config import COLORS
-
-
-def render():
-    """Render discovery page."""
-    
-    page_header("Drug Discovery", "Search for drug candidates with AI-powered analysis", "🔬")
-    
-    # Query Input Section
-    st.markdown(f"""
-    <div class="card" style="margin-bottom: 1.5rem;">
-        <div style="font-size: 0.875rem; font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 0.75rem;">
-            Search Query
-        </div>
-    </div>
-    """, unsafe_allow_html=True)
-    
-    col1, col2 = st.columns([3, 1])
-    
-    with col1:
-        query = st.text_area(
-            "Query",
-            placeholder="Enter a natural language query, SMILES string, or FASTA sequence...",
-            height=100,
-            label_visibility="collapsed"
-        )
-    
-    with col2:
-        st.selectbox("Search Type", ["Similarity", "Binding Affinity", "Properties"], label_visibility="collapsed")
-        st.selectbox("Database", ["All", "DrugBank", "ChEMBL", "ZINC"], label_visibility="collapsed")
-        search_clicked = st.button("🔍 Search", type="primary", use_container_width=True)
-    
-    spacer("1.5rem")
-    
-    # Pipeline Progress
-    section_header("Pipeline Status", "🔄")
-    
-    if "discovery_step" not in st.session_state:
-        st.session_state.discovery_step = 0
-    
-    steps = [
-        {"name": "Input", "status": "done" if st.session_state.discovery_step > 0 else "active"},
-        {"name": "Encode", "status": "done" if st.session_state.discovery_step > 1 else ("active" if st.session_state.discovery_step == 1 else "pending")},
-        {"name": "Search", "status": "done" if st.session_state.discovery_step > 2 else ("active" if st.session_state.discovery_step == 2 else "pending")},
-        {"name": "Predict", "status": "done" if st.session_state.discovery_step > 3 else ("active" if st.session_state.discovery_step == 3 else "pending")},
-        {"name": "Results", "status": "active" if st.session_state.discovery_step == 4 else "pending"},
-    ]
-    
-    pipeline_progress(steps)
-    
-    spacer("2rem")
-    divider()
-    spacer("2rem")
-    
-    # Results Section
-    section_header("Results", "🎯")
-    
-    if search_clicked and query:
-        st.session_state.discovery_step = 4
-        st.session_state.discovery_query = query
-    
-    if st.session_state.discovery_step >= 4:
-        # Show results
-        tabs = st.tabs(["Top Candidates", "Property Analysis", "Evidence"])
-        
-        with tabs[0]:
-            # Results list
-            results = [
-                {"name": "Candidate A", "score": 0.95, "mw": "342.4", "logp": "2.1", "hbd": "2"},
-                {"name": "Candidate B", "score": 0.89, "mw": "298.3", "logp": "1.8", "hbd": "3"},
-                {"name": "Candidate C", "score": 0.82, "mw": "415.5", "logp": "3.2", "hbd": "1"},
-                {"name": "Candidate D", "score": 0.76, "mw": "267.3", "logp": "1.5", "hbd": "2"},
-                {"name": "Candidate E", "score": 0.71, "mw": "389.4", "logp": "2.8", "hbd": "2"},
-            ]
-            
-            for r in results:
-                score_color = COLORS.emerald if r["score"] >= 0.8 else (COLORS.amber if r["score"] >= 0.5 else COLORS.rose)
-                st.markdown(f"""
-                <div class="result">
-                    <div style="display: flex; justify-content: space-between; align-items: flex-start;">
-                        <div>
-                            <div style="font-weight: 600; color: {COLORS.text_primary};">{r["name"]}</div>
-                            <div style="display: flex; gap: 1rem; margin-top: 0.5rem;">
-                                <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">MW: {r["mw"]}</span>
-                                <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">LogP: {r["logp"]}</span>
-                                <span style="font-size: 0.8125rem; color: {COLORS.text_muted};">HBD: {r["hbd"]}</span>
-                            </div>
-                        </div>
-                        <div style="font-size: 1.5rem; font-weight: 700; color: {score_color};">{r["score"]:.0%}</div>
-                    </div>
-                </div>
-                """, unsafe_allow_html=True)
-                spacer("0.75rem")
-        
-        with tabs[1]:
-            # Property distribution
-            col1, col2 = st.columns(2)
-            
-            with col1:
-                bar_chart(
-                    {"<200": 5, "200-300": 12, "300-400": 8, "400-500": 3, ">500": 2},
-                    title="Molecular Weight Distribution",
-                    height=250
-                )
-            
-            with col2:
-                bar_chart(
-                    {"<1": 4, "1-2": 10, "2-3": 8, "3-4": 5, ">4": 3},
-                    title="LogP Distribution",
-                    height=250
-                )
-        
-        with tabs[2]:
-            # Evidence
-            st.markdown(f"""
-            <div class="card">
-                <div style="font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 1rem;">
-                    Related Literature
-                </div>
-            </div>
-            """, unsafe_allow_html=True)
-            
-            papers = [
-                {"title": "Novel therapeutic targets for cancer treatment", "year": "2024", "journal": "Nature Medicine"},
-                {"title": "Molecular docking studies of kinase inhibitors", "year": "2023", "journal": "J. Med. Chem."},
-                {"title": "AI-driven drug discovery approaches", "year": "2024", "journal": "Drug Discovery Today"},
-            ]
-            
-            for p in papers:
-                st.markdown(f"""
-                <div style="
-                    padding: 1rem;
-                    border: 1px solid {COLORS.border};
-                    border-radius: 8px;
-                    margin-bottom: 0.75rem;
-                ">
-                    <div style="font-weight: 500; color: {COLORS.text_primary};">{p["title"]}</div>
-                    <div style="font-size: 0.8125rem; color: {COLORS.text_muted}; margin-top: 0.25rem;">
-                        {p["journal"]} • {p["year"]}
-                    </div>
-                </div>
-                """, unsafe_allow_html=True)
-    
-    else:
-        empty_state(
-            "🔍",
-            "No Results Yet",
-            "Enter a query and click Search to find drug candidates"
-        )

bioflow/ui/pages/explorer.py

@@ -1,127 +0,0 @@
-"""
-BioFlow - Explorer Page
-=======================
-Data exploration and visualization.
-"""
-
-import streamlit as st
-import sys
-import os
-import numpy as np
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
-
-from bioflow.ui.components import (
-    page_header, section_header, divider, spacer,
-    scatter_chart, heatmap, metric_card, empty_state
-)
-from bioflow.ui.config import COLORS
-
-
-def render():
-    """Render explorer page."""
-    
-    page_header("Data Explorer", "Visualize molecular embeddings and relationships", "🧬")
-    
-    # Controls
-    col1, col2, col3, col4 = st.columns(4)
-    
-    with col1:
-        dataset = st.selectbox("Dataset", ["DrugBank", "ChEMBL", "ZINC", "Custom"])
-    with col2:
-        viz_type = st.selectbox("Visualization", ["UMAP", "t-SNE", "PCA"])
-    with col3:
-        color_by = st.selectbox("Color by", ["Activity", "MW", "LogP", "Cluster"])
-    with col4:
-        st.write("")  # Spacing
-        st.write("")
-        if st.button("🔄 Refresh", use_container_width=True):
-            st.rerun()
-    
-    spacer("1.5rem")
-    
-    # Main visualization area
-    col_viz, col_details = st.columns([2, 1])
-    
-    with col_viz:
-        section_header("Embedding Space", "🗺️")
-        
-        # Generate sample data
-        np.random.seed(42)
-        n_points = 200
-        
-        # Create clusters
-        cluster1_x = np.random.normal(2, 0.8, n_points // 4)
-        cluster1_y = np.random.normal(3, 0.8, n_points // 4)
-        
-        cluster2_x = np.random.normal(-2, 1, n_points // 4)
-        cluster2_y = np.random.normal(-1, 1, n_points // 4)
-        
-        cluster3_x = np.random.normal(4, 0.6, n_points // 4)
-        cluster3_y = np.random.normal(-2, 0.6, n_points // 4)
-        
-        cluster4_x = np.random.normal(-1, 0.9, n_points // 4)
-        cluster4_y = np.random.normal(4, 0.9, n_points // 4)
-        
-        x = list(cluster1_x) + list(cluster2_x) + list(cluster3_x) + list(cluster4_x)
-        y = list(cluster1_y) + list(cluster2_y) + list(cluster3_y) + list(cluster4_y)
-        labels = [f"Mol_{i}" for i in range(n_points)]
-        
-        scatter_chart(x, y, labels, title=f"{viz_type} Projection - {dataset}", height=450)
-    
-    with col_details:
-        section_header("Statistics", "📊")
-        
-        metric_card("12,450", "Total Molecules", "🧪", color=COLORS.primary)
-        spacer("0.75rem")
-        metric_card("4", "Clusters Found", "🎯", color=COLORS.cyan)
-        spacer("0.75rem")
-        metric_card("0.89", "Silhouette Score", "📈", color=COLORS.emerald)
-        spacer("0.75rem")
-        metric_card("85%", "Coverage", "✓", color=COLORS.amber)
-    
-    spacer("2rem")
-    divider()
-    spacer("2rem")
-    
-    # Similarity Heatmap
-    section_header("Similarity Matrix", "🔥")
-    
-    # Sample similarity matrix
-    np.random.seed(123)
-    labels_short = ["Cluster A", "Cluster B", "Cluster C", "Cluster D", "Cluster E"]
-    similarity = np.random.uniform(0.3, 1.0, (5, 5))
-    similarity = (similarity + similarity.T) / 2  # Make symmetric
-    np.fill_diagonal(similarity, 1.0)
-    
-    heatmap(
-        similarity.tolist(),
-        labels_short,
-        labels_short,
-        title="Inter-cluster Similarity",
-        height=350
-    )
-    
-    spacer("2rem")
-    
-    # Export options
-    st.markdown(f"""
-    <div class="card">
-        <div style="display: flex; justify-content: space-between; align-items: center;">
-            <div>
-                <div style="font-weight: 600; color: {COLORS.text_primary};">Export Data</div>
-                <div style="font-size: 0.8125rem; color: {COLORS.text_muted}; margin-top: 0.25rem;">
-                    Download embeddings, clusters, or full dataset
-                </div>
-            </div>
-        </div>
-    </div>
-    """, unsafe_allow_html=True)
-    
-    exp_cols = st.columns(3)
-    with exp_cols[0]:
-        st.button("📥 Embeddings (CSV)", use_container_width=True)
-    with exp_cols[1]:
-        st.button("📥 Clusters (JSON)", use_container_width=True)
-    with exp_cols[2]:
-        st.button("📥 Full Dataset", use_container_width=True)

bioflow/ui/pages/home.py

@@ -1,213 +0,0 @@
-"""
-BioFlow - Home Page
-====================
-Clean dashboard with key metrics and quick actions.
-"""
-
-import streamlit as st
-import sys
-import os
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
-
-from bioflow.ui.components import (
-    section_header, divider, spacer,
-    metric_card, pipeline_progress, bar_chart
-)
-from bioflow.ui.config import COLORS
-
-
-def render():
-    """Render home page."""
-    
-    # Hero Section (Tailark-inspired)
-    hero_col, hero_side = st.columns([3, 1.4])
-    
-    with hero_col:
-        st.markdown(f"""
-        <div class="hero">
-            <div class="hero-badge">New • BioFlow 2.0</div>
-            <div class="hero-title">AI-Powered Drug Discovery</div>
-            <div class="hero-subtitle">
-                Run discovery pipelines, predict binding, and surface evidence in one streamlined workspace.
-            </div>
-            <div class="hero-actions">
-                <span class="badge badge-primary">Model-aware search</span>
-                <span class="badge badge-success">Evidence-linked</span>
-                <span class="badge badge-warning">Fast iteration</span>
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        spacer("0.75rem")
-        btn1, btn2 = st.columns(2)
-        with btn1:
-            if st.button("Start Discovery", type="primary", use_container_width=True):
-                st.session_state.current_page = "discovery"
-                st.rerun()
-        with btn2:
-            if st.button("Explore Data", use_container_width=True):
-                st.session_state.current_page = "explorer"
-                st.rerun()
-    
-    with hero_side:
-        st.markdown(f"""
-        <div class="hero-card">
-            <div style="font-size: 0.75rem; text-transform: uppercase; letter-spacing: 0.08em; color: {COLORS.text_muted};">
-                Today
-            </div>
-            <div style="font-size: 1.75rem; font-weight: 700; color: {COLORS.text_primary}; margin-top: 0.5rem;">
-                156 Discoveries
-            </div>
-            <div style="font-size: 0.875rem; color: {COLORS.text_muted}; margin-top: 0.5rem;">
-                +12% vs last week
-            </div>
-            <div class="divider" style="margin: 1rem 0;"></div>
-            <div style="display: flex; flex-direction: column; gap: 0.5rem;">
-                <span class="badge badge-primary">Discovery</span>
-                <span class="badge badge-success">Prediction</span>
-                <span class="badge badge-warning">Evidence</span>
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-    
-    # Metrics Row
-    cols = st.columns(4)
-    
-    with cols[0]:
-        metric_card("12.5M", "Molecules", "🧪", "+2.3%", "up", COLORS.primary)
-    with cols[1]:
-        metric_card("847K", "Proteins", "🧬", "+1.8%", "up", COLORS.cyan)
-    with cols[2]:
-        metric_card("1.2M", "Papers", "📚", "+5.2%", "up", COLORS.emerald)
-    with cols[3]:
-        metric_card("156", "Discoveries", "✨", "+12%", "up", COLORS.amber)
-    
-    spacer("2rem")
-    
-    # Quick Actions
-    section_header("Quick Actions", "⚡")
-    
-    action_cols = st.columns(4)
-    
-    with action_cols[0]:
-        st.markdown(f"""
-        <div class="quick-action">
-            <span class="quick-action-icon">🔍</span>
-            <div class="quick-action-title">New Discovery</div>
-            <div class="quick-action-desc">Start a pipeline</div>
-        </div>
-        """, unsafe_allow_html=True)
-        if st.button("Start", key="qa_discovery", use_container_width=True):
-            st.session_state.current_page = "discovery"
-            st.rerun()
-    
-    with action_cols[1]:
-        st.markdown(f"""
-        <div class="quick-action">
-            <span class="quick-action-icon">📊</span>
-            <div class="quick-action-title">Explore Data</div>
-            <div class="quick-action-desc">Visualize embeddings</div>
-        </div>
-        """, unsafe_allow_html=True)
-        if st.button("Explore", key="qa_explorer", use_container_width=True):
-            st.session_state.current_page = "explorer"
-            st.rerun()
-    
-    with action_cols[2]:
-        st.markdown(f"""
-        <div class="quick-action">
-            <span class="quick-action-icon">📁</span>
-            <div class="quick-action-title">Upload Data</div>
-            <div class="quick-action-desc">Add molecules</div>
-        </div>
-        """, unsafe_allow_html=True)
-        if st.button("Upload", key="qa_data", use_container_width=True):
-            st.session_state.current_page = "data"
-            st.rerun()
-    
-    with action_cols[3]:
-        st.markdown(f"""
-        <div class="quick-action">
-            <span class="quick-action-icon">⚙️</span>
-            <div class="quick-action-title">Settings</div>
-            <div class="quick-action-desc">Configure models</div>
-        </div>
-        """, unsafe_allow_html=True)
-        if st.button("Configure", key="qa_settings", use_container_width=True):
-            st.session_state.current_page = "settings"
-            st.rerun()
-    
-    spacer("2rem")
-    divider()
-    spacer("2rem")
-    
-    # Two Column Layout
-    col1, col2 = st.columns([3, 2])
-    
-    with col1:
-        section_header("Recent Discoveries", "🎯")
-        
-        # Sample results
-        results = [
-            {"name": "Aspirin analog", "score": 0.94, "mw": "180.16"},
-            {"name": "Novel kinase inhibitor", "score": 0.87, "mw": "331.39"},
-            {"name": "EGFR binder candidate", "score": 0.72, "mw": "311.38"},
-        ]
-        
-        for r in results:
-            score_color = COLORS.emerald if r["score"] >= 0.8 else (COLORS.amber if r["score"] >= 0.5 else COLORS.rose)
-            st.markdown(f"""
-            <div class="result">
-                <div style="display: flex; justify-content: space-between; align-items: center;">
-                    <div style="font-weight: 600; color: {COLORS.text_primary};">{r["name"]}</div>
-                    <div style="font-size: 1.25rem; font-weight: 700; color: {score_color};">{r["score"]:.0%}</div>
-                </div>
-                <div style="font-size: 0.8125rem; color: {COLORS.text_muted}; margin-top: 0.5rem;">
-                    MW: {r["mw"]}
-                </div>
-            </div>
-            """, unsafe_allow_html=True)
-            spacer("0.75rem")
-    
-    with col2:
-        section_header("Pipeline Activity", "📈")
-        
-        bar_chart(
-            {"Mon": 23, "Tue": 31, "Wed": 28, "Thu": 45, "Fri": 38, "Sat": 12, "Sun": 8},
-            title="",
-            height=250
-        )
-        
-        spacer("1rem")
-        section_header("Active Pipeline", "🔄")
-        
-        pipeline_progress([
-            {"name": "Encode", "status": "done"},
-            {"name": "Search", "status": "active"},
-            {"name": "Predict", "status": "pending"},
-            {"name": "Verify", "status": "pending"},
-        ])
-    
-    spacer("2rem")
-    
-    # Tip
-    st.markdown(f"""
-    <div style="
-        background: {COLORS.bg_surface};
-        border: 1px solid {COLORS.border};
-        border-radius: 12px;
-        padding: 1.25rem;
-        display: flex;
-        align-items: center;
-        gap: 1rem;
-    ">
-        <span style="font-size: 1.5rem;">💡</span>
-        <div>
-            <div style="font-size: 0.9375rem; color: {COLORS.text_primary}; font-weight: 500;">Pro Tip</div>
-            <div style="font-size: 0.8125rem; color: {COLORS.text_muted};">
-                Use natural language like "Find molecules similar to aspirin that can cross the blood-brain barrier"
-            </div>
-        </div>
-    </div>
-    """, unsafe_allow_html=True)

bioflow/ui/pages/settings.py

@@ -1,192 +0,0 @@
-"""
-BioFlow - Settings Page
-========================
-Configuration and preferences.
-"""
-
-import streamlit as st
-import sys
-import os
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))))
-
-from bioflow.ui.components import page_header, section_header, divider, spacer
-from bioflow.ui.config import COLORS
-
-
-def render():
-    """Render settings page."""
-    
-    page_header("Settings", "Configure models, databases, and preferences", "⚙️")
-    
-    # Tabs for different settings sections
-    tabs = st.tabs(["🧠 Models", "🗄️ Database", "🔌 API Keys", "🎨 Appearance"])
-    
-    with tabs[0]:
-        section_header("Model Configuration", "🧠")
-        
-        st.markdown(f"""
-        <div class="card" style="margin-bottom: 1rem;">
-            <div style="font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 0.5rem;">
-                Embedding Models
-            </div>
-            <div style="font-size: 0.8125rem; color: {COLORS.text_muted};">
-                Configure models used for molecular and protein embeddings
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        col1, col2 = st.columns(2)
-        
-        with col1:
-            st.selectbox(
-                "Molecule Encoder",
-                ["MolCLR (Recommended)", "ChemBERTa", "GraphMVP", "MolBERT"],
-                help="Model for generating molecular embeddings"
-            )
-            
-            st.selectbox(
-                "Protein Encoder", 
-                ["ESM-2 (Recommended)", "ProtTrans", "UniRep", "SeqVec"],
-                help="Model for generating protein embeddings"
-            )
-        
-        with col2:
-            st.selectbox(
-                "Binding Predictor",
-                ["DrugBAN (Recommended)", "DeepDTA", "GraphDTA", "Custom"],
-                help="Model for predicting drug-target binding"
-            )
-            
-            st.selectbox(
-                "Property Predictor",
-                ["ADMET-AI (Recommended)", "ChemProp", "Custom"],
-                help="Model for ADMET property prediction"
-            )
-        
-        spacer("1rem")
-        
-        st.markdown(f"""
-        <div class="card" style="margin-bottom: 1rem;">
-            <div style="font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 0.5rem;">
-                LLM Settings
-            </div>
-            <div style="font-size: 0.8125rem; color: {COLORS.text_muted};">
-                Configure language models for evidence retrieval and reasoning
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        col1, col2 = st.columns(2)
-        
-        with col1:
-            st.selectbox(
-                "LLM Provider",
-                ["OpenAI", "Anthropic", "Local (Ollama)", "Azure OpenAI"]
-            )
-        
-        with col2:
-            st.selectbox(
-                "Model",
-                ["GPT-4o", "GPT-4-turbo", "Claude 3.5 Sonnet", "Llama 3.1 70B"]
-            )
-        
-        st.slider("Temperature", 0.0, 1.0, 0.7, 0.1)
-        st.number_input("Max Tokens", 100, 4096, 2048, 100)
-    
-    with tabs[1]:
-        section_header("Database Configuration", "🗄️")
-        
-        st.markdown(f"""
-        <div class="card" style="margin-bottom: 1rem;">
-            <div style="font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 0.5rem;">
-                Vector Database
-            </div>
-            <div style="font-size: 0.8125rem; color: {COLORS.text_muted};">
-                Configure the vector store for similarity search
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        col1, col2 = st.columns(2)
-        
-        with col1:
-            st.selectbox("Vector Store", ["Qdrant (Recommended)", "Milvus", "Pinecone", "Weaviate", "ChromaDB"])
-            st.text_input("Host", value="localhost")
-        
-        with col2:
-            st.number_input("Port", 1, 65535, 6333)
-            st.text_input("Collection Name", value="bioflow_embeddings")
-        
-        spacer("1rem")
-        
-        st.markdown(f"""
-        <div class="card" style="margin-bottom: 1rem;">
-            <div style="font-weight: 600; color: {COLORS.text_primary}; margin-bottom: 0.5rem;">
-                Knowledge Sources
-            </div>
-            <div style="font-size: 0.8125rem; color: {COLORS.text_muted};">
-                External databases for evidence retrieval
-            </div>
-        </div>
-        """, unsafe_allow_html=True)
-        
-        col1, col2 = st.columns(2)
-        
-        with col1:
-            st.checkbox("PubMed", value=True)
-            st.checkbox("DrugBank", value=True)
-            st.checkbox("ChEMBL", value=True)
-        
-        with col2:
-            st.checkbox("UniProt", value=True)
-            st.checkbox("KEGG", value=False)
-            st.checkbox("Reactome", value=False)
-    
-    with tabs[2]:
-        section_header("API Keys", "🔌")
-        
-        st.warning("⚠️ API keys are stored locally and never sent to external servers.")
-        
-        st.text_input("OpenAI API Key", type="password", placeholder="sk-...")
-        st.text_input("Anthropic API Key", type="password", placeholder="sk-ant-...")
-        st.text_input("PubMed API Key", type="password", placeholder="Optional - for higher rate limits")
-        st.text_input("ChEMBL API Key", type="password", placeholder="Optional")
-        
-        spacer("1rem")
-        
-        if st.button("💾 Save API Keys", type="primary"):
-            st.success("✓ API keys saved securely")
-    
-    with tabs[3]:
-        section_header("Appearance", "🎨")
-        
-        st.selectbox("Theme", ["Dark (Default)", "Light", "System"])
-        st.selectbox("Accent Color", ["Purple", "Blue", "Green", "Cyan", "Pink"])
-        st.checkbox("Enable animations", value=True)
-        st.checkbox("Compact mode", value=False)
-        st.slider("Font size", 12, 18, 14)
-    
-    spacer("2rem")
-    divider()
-    spacer("1rem")
-    
-    # Save buttons
-    col1, col2, col3 = st.columns([1, 1, 2])
-    
-    with col1:
-        if st.button("💾 Save Settings", type="primary", use_container_width=True):
-            st.success("✓ Settings saved successfully!")
-    
-    with col2:
-        if st.button("🔄 Reset to Defaults", use_container_width=True):
-            st.info("Settings reset to defaults")
-    
-    spacer("2rem")
-    
-    # Version info
-    st.markdown(f"""
-    <div style="text-align: center; padding: 1rem; color: {COLORS.text_muted}; font-size: 0.75rem;">
-        BioFlow v0.1.0 • Built with OpenBioMed
-    </div>
-    """, unsafe_allow_html=True)

bioflow/ui/requirements.txt

@@ -1,31 +0,0 @@
-# BioFlow UI Dependencies
-# =======================
-
-# Core Streamlit
-streamlit>=1.29.0
-
-# Visualization
-plotly>=5.18.0
-altair>=5.2.0
-
-# Data handling
-pandas>=2.0.0
-numpy>=1.24.0
-
-# Molecular visualization (optional)
-rdkit>=2023.9.1
-py3Dmol>=2.0.0
-
-# Machine Learning (optional, for real encoders)
-# torch>=2.0.0
-# transformers>=4.35.0
-
-# Vector database
-qdrant-client>=1.7.0
-
-# Image processing
-Pillow>=10.0.0
-
-# Utilities
-python-dotenv>=1.0.0
-pyyaml>=6.0.1

docs/BIOFLOW_OBM_REPORT.md

@@ -4,6 +4,9 @@
 
 Ce document présente l'intégration complète de **BioFlow** avec **OpenBioMed (OBM)** et **Qdrant** pour créer un système d'intelligence biologique multimodale. L'architecture permet d'unifier textes scientifiques, molécules (SMILES) et protéines dans un espace vectoriel commun, facilitant la découverte cross-modale et la conception de médicaments assistée par IA.
 
+> **Note (27/01/2026)**: L'interface Streamlit historique a été retirée du runtime.  
+> L'UI officielle est **Next.js** (dossier `ui/`) et le backend est **FastAPI** (port 8000).
+
 ---
 
 ## 📋 Table des matières
@@ -24,7 +27,7 @@ Ce document présente l'intégration complète de **BioFlow** avec **OpenBioMed
 ```
 ┌─────────────────────────────────────────────────────────────────┐
 │                        BioFlow Explorer                         │
-│                     (Interface Streamlit)                        │
+│                     (Interface Next.js)                          │
 └─────────────────────────────────┬───────────────────────────────┘
                                   │
 ┌─────────────────────────────────▼───────────────────────────────┐
@@ -140,16 +143,9 @@ Outils de visualisation pour exploration.
 - **MoleculeVisualizer** : SVG, grilles de molécules (via RDKit)
 - **ResultsVisualizer** : Dashboard, graphiques de scores
 
-### 5. Application Streamlit (`bioflow/app.py`)
-
-Interface interactive complète avec :
+### 5. Application Web (Next.js)
 
-- 🔍 **Search & Explore** : Recherche multimodale
-- 📥 **Data Ingestion** : Upload simple/batch
-- 🧪 **Cross-Modal Analysis** : Comparaison inter-modalités
-- 📊 **Visualization** : Plots embeddings et molécules
-- 🔬 **Pipeline Demo** : Workflow de découverte complet
-- 📚 **Documentation** : Guide intégré
+L'interface officielle est la **Next.js UI** dans `ui/` (aucun runtime Streamlit).
 
 ---
 
@@ -192,7 +188,7 @@ protein → protein: Protéines homologues
 ```bash
 # Dépendances principales
 pip install -r requirements.txt
-pip install qdrant-client streamlit plotly scikit-learn
+pip install qdrant-client plotly scikit-learn
 
 # Optionnel pour visualisation moléculaire
 pip install rdkit
@@ -202,7 +198,9 @@ pip install rdkit
 
 ```bash
 cd OpenBioMed
-streamlit run bioflow/app.py
+# UI (Next.js)
+cd ui
+pnpm dev
 ```
 
 ### Utilisation Programmatique
@@ -350,7 +348,7 @@ if not validation.content["passed"]:
 | Mémoire vectorielle centrale | `QdrantManager` avec collection partagée |
 | Encodeur multimodal | `OBMWrapper` (BioMedGPT) |
 | Nœuds-agents | Classes `*Agent` dans `pipeline.py` |
-| Workflow visuel | `BioFlowPipeline` + Streamlit UI |
+| Workflow visuel | **Next.js UI** (`ui/`) + API FastAPI |
 | Evidence linking | Payload avec `source`, `tags`, scores |
 
 ### Points d'extension
@@ -385,7 +383,7 @@ pipeline:
 - [x] OBM Wrapper avec encodage multimodal
 - [x] Intégration Qdrant
 - [x] Agents de base (Miner, Validator, Ranker)
-- [x] Interface Streamlit
+- [x] Interface Next.js (UI officielle)
 - [x] Mode Mock pour développement
 
 ### Phase 2 (Court terme)

docs/COMPLIANCE_REPORT.md

@@ -0,0 +1,36 @@
+# BioFlow Compliance Report (Phase 1)
+
+**Date:** 2026-01-27  
+**Scope:** Open-source compliance + forbidden runtime dependencies
+
+## Summary
+- **Streamlit UI removed** from runtime and repository path.
+- **OpenAI / Azure OpenAI / Anthropic UI references removed**.
+- **Runtime stack** confirmed: FastAPI + Next.js + Qdrant only.
+
+## Removed / Deprecated
+- `bioflow/app.py` (legacy Streamlit app) **deleted**
+- `bioflow/ui/*` (Streamlit UI package) **deleted**
+- Streamlit dependency **removed** from runtime requirements
+- UI settings no longer expose proprietary LLM providers
+
+## Allowed / Kept
+- **OBM (OpenBioMed)** for embeddings only
+- **DeepPurpose** for DTI (open-source)
+- **Qdrant** as primary vector database
+
+## Dependencies (Runtime)
+From `requirements.txt` (open-source only):
+- `fastapi`, `uvicorn`
+- `qdrant-client`
+- `torch`, `transformers`, `rdkit`, `numpy`, `scikit-learn`
+- `requests`, `pandas`, `dotenv`
+
+## Remaining Risks / Follow-ups
+- **Legacy references in docs** should avoid implying Streamlit runtime.
+- Ensure **no proprietary endpoints** are configured in deployment.
+
+## Evidence
+- Streamlit files removed: `bioflow/app.py`, `bioflow/ui/*`
+- UI settings updated: `ui/app/dashboard/settings/page.tsx`
+

docs/FRONTEND_FALLBACKS.md

@@ -0,0 +1,40 @@
+# Frontend Fallback Behavior (Phase 5)
+
+## Overview
+The UI uses Next.js `/api/*` route handlers as a proxy layer to the FastAPI backend.  
+If the backend is unavailable, these routes return **safe defaults** so the UI remains usable.
+
+## Proxy Routes (Next.js)
+All routes forward to `API_CONFIG.baseUrl` (`NEXT_PUBLIC_API_URL`, default `http://localhost:8000`).
+
+### Search
+- `POST /api/search`
+- `POST /api/search/hybrid`
+- Fallback: empty results with metadata stubs
+
+### Agents
+- `POST /api/agents/generate`
+- `POST /api/agents/validate`
+- `POST /api/agents/rank`
+- `POST /api/agents/workflow`
+- Fallback: empty payloads with `mock: true` flags (where applicable)
+
+### Explorer
+- `GET /api/explorer/embeddings`
+- Fallback: `503` + empty points
+
+### Ingestion
+- `POST /api/ingest/pubmed`
+- `POST /api/ingest/uniprot`
+- `POST /api/ingest/chembl`
+- `POST /api/ingest/all`
+- `GET /api/ingest/jobs/{job_id}`
+- Fallback: `503` + error message
+
+## UI Empty‑State Handling
+- Visualization page shows a **“No points to display”** message until a search runs.
+- Workflow page shows **“Run workflow to see results”** until execution completes.
+
+## Recommendation
+For demos, keep the FastAPI backend running to ensure real data/embeddings are shown.
+

docs/INGESTION_GUIDE.md

@@ -0,0 +1,95 @@
+# BioFlow Ingestion Guide (Phase 3)
+
+This guide explains how to ingest data from **PubMed**, **UniProt**, and **ChEMBL** into Qdrant.
+
+## 1) FastAPI Endpoints (Recommended)
+
+### PubMed
+`POST /api/ingest/pubmed`
+```json
+{
+  "query": "EGFR lung cancer",
+  "limit": 100,
+  "batch_size": 50,
+  "rate_limit": 0.4,
+  "collection": "bioflow_memory",
+  "email": "you@example.com",
+  "api_key": "NCBI_API_KEY",
+  "sync": false
+}
+```
+
+### UniProt
+`POST /api/ingest/uniprot`
+```json
+{
+  "query": "EGFR AND organism_id:9606",
+  "limit": 50,
+  "batch_size": 50,
+  "rate_limit": 0.2,
+  "collection": "bioflow_memory",
+  "sync": false
+}
+```
+
+### ChEMBL
+`POST /api/ingest/chembl`
+```json
+{
+  "query": "EGFR",
+  "limit": 30,
+  "batch_size": 50,
+  "rate_limit": 0.3,
+  "collection": "bioflow_memory",
+  "search_mode": "target",
+  "sync": false
+}
+```
+
+### All Sources
+`POST /api/ingest/all`
+```json
+{
+  "query": "EGFR lung cancer",
+  "pubmed_limit": 100,
+  "uniprot_limit": 50,
+  "chembl_limit": 30,
+  "batch_size": 50,
+  "rate_limit": 0.3,
+  "collection": "bioflow_memory",
+  "sync": false
+}
+```
+
+### Job Status
+`GET /api/ingest/jobs/{job_id}`
+
+## 2) Next.js Proxy Routes (Optional)
+If you want to call the backend through Next.js:
+```
+/api/ingest/pubmed
+/api/ingest/uniprot
+/api/ingest/chembl
+/api/ingest/all
+/api/ingest/jobs/{job_id}
+```
+
+## 3) CLI Ingestion
+```
+python -m bioflow.ingestion.ingest_all --query "EGFR lung cancer" --limit 100
+```
+
+## 4) Environment Variables
+- `INGEST_BATCH_SIZE`
+- `PUBMED_RATE_LIMIT`
+- `UNIPROT_RATE_LIMIT`
+- `CHEMBL_RATE_LIMIT`
+- `NCBI_EMAIL`
+- `NCBI_API_KEY`
+- `CHEMBL_SEARCH_MODE`
+
+## 5) Recommended Minimums
+- PubMed: 100 records
+- UniProt: 50 records
+- ChEMBL: 30 records
+

docs/METADATA_SCHEMA.md

@@ -0,0 +1,62 @@
+# BioFlow Metadata Schema (Phase 3)
+
+All ingested items are stored in Qdrant with a **payload** that includes core provenance fields plus source‑specific metadata.
+
+## Core Fields (all modalities)
+
+| Field | Type | Description |
+|------|------|-------------|
+| `source` | string | Source name (`pubmed`, `uniprot`, `chembl`) |
+| `source_id` | string | Source identifier (e.g., `pubmed:12345`) |
+| `indexed_at` | string | ISO timestamp when ingested |
+| `content` | string | Stored raw content (text, SMILES, or sequence) |
+| `modality` | string | `text`, `molecule`, or `protein` |
+
+## PubMed (text)
+
+| Field | Type | Description |
+|------|------|-------------|
+| `pmid` | string | PubMed ID |
+| `title` | string | Article title |
+| `authors` | list[string] | Authors |
+| `journal` | string | Journal name |
+| `pub_date` | string | Publication date |
+| `year` | number | Publication year |
+| `mesh_terms` | list[string] | MeSH terms |
+| `url` | string | PubMed URL |
+
+## UniProt (protein)
+
+| Field | Type | Description |
+|------|------|-------------|
+| `accession` | string | UniProt accession |
+| `entry_name` | string | UniProt entry name |
+| `protein_name` | string | Protein name |
+| `gene_names` | list[string] | Gene names |
+| `organism` | string | Scientific name |
+| `organism_id` | string | Taxon ID |
+| `function` | string | Function text (truncated) |
+| `sequence_length` | number | Sequence length |
+| `pdb_ids` | list[string] | PDB references |
+| `url` | string | UniProt URL |
+
+## ChEMBL (molecule)
+
+| Field | Type | Description |
+|------|------|-------------|
+| `chembl_id` | string | ChEMBL molecule ID |
+| `name` | string | Preferred name |
+| `synonyms` | list[string] | Synonyms (limited) |
+| `smiles` | string | Canonical SMILES |
+| `inchi_key` | string | InChIKey |
+| `molecular_weight` | number | Full molecular weight |
+| `alogp` | number | ALogP |
+| `hba` | number | H‑bond acceptors |
+| `hbd` | number | H‑bond donors |
+| `psa` | number | Polar surface area |
+| `ro5_violations` | number | Rule‑of‑5 violations |
+| `target_chembl_id` | string | Target ID (if available) |
+| `activity_type` | string | Activity type (e.g., IC50) |
+| `activity_value` | number | Activity value |
+| `activity_units` | string | Activity units |
+| `url` | string | ChEMBL URL |

docs/OBSERVABILITY.md

@@ -0,0 +1,45 @@
+# Observability (Phase 6)
+
+## Structured Logs
+FastAPI emits JSON logs for key actions:
+- `search` / `search_error`
+- `ingest_single` / `ingest_single_error`
+- `workflow` / `workflow_error`
+
+Each log includes:
+- `event`
+- `request_id`
+- `timestamp`
+- relevant fields (query, top_k, duration_ms, etc.)
+
+## Health Metrics Endpoint
+`GET /api/health/metrics`
+
+Returns:
+```json
+{
+  "status": "ok",
+  "timestamp": "...",
+  "qdrant": {
+    "available": true,
+    "collections": ["molecules", "proteins"],
+    "stats": { "molecules": { "points_count": 1234 } }
+  },
+  "models": {
+    "available": true,
+    "device": "cuda",
+    "obm_loaded": true
+  }
+}
+```
+
+## CI-Style Test Runner
+`python scripts/run_tests.py`
+
+Runs:
+- `test_search_api.py`
+- `test_agent_api.py`
+- `test_search_filters.py`
+- `test_phase4_ui.py`
+- `test_ingestion_api.py`
+

docs/ROADMAP.md

@@ -98,75 +98,21 @@ The team works on their respective modules using the core interfaces.
 ## 📊 Phase 4: UI/UX & Deployment ✅ COMPLETE
 **Goal:** Build an intuitive, modern interface for the BioFlow platform.
 
-- [x] **Theme & Styling** (`bioflow/ui/config.py`):
-  - Glassmorphism design system with dark theme
-  - Custom CSS with animations, cards, badges
-  - Responsive layout with Inter font family
-  - Color palette: Indigo primary, Emerald success, Cyan accent
-
-- [x] **Reusable Components** (`bioflow/ui/components.py`):
-  - `hero_section`: Landing hero with stats
-  - `metric_card`: Animated metric displays
-  - `glass_card`, `feature_cards`: Content cards
-  - `pipeline_flow`: Visual pipeline status
-  - `binding_affinity_chart`, `scatter_embedding`, `similarity_heatmap`: Charts
-  - `molecule_viewer_2d`: RDKit molecule rendering
-  - `evidence_card`: Traceability links
-  - `chat_message`, `chat_container`: AI chat interface
-  - `step_progress`, `notification`: UX helpers
-
-- [x] **Dashboard Home** (`bioflow/ui/pages/home.py`):
-  - Hero section with platform branding
-  - Key metrics (molecules, proteins, literature, predictions)
-  - Quick action cards (Discovery, Explorer, Upload)
-  - Feature highlights grid
-  - Recent discoveries chart
-  - Activity timeline
-  - Active pipeline visualization
-
-- [x] **Discovery Page** (`bioflow/ui/pages/discovery.py`):
-  - Query input (text, SMILES, FASTA)
-  - Target protein selection with common targets
-  - Real-time pipeline progress visualization
-  - Results with binding affinity chart
-  - Top hits with molecule viewer
-  - Evidence linking to PubMed/ChEMBL/PubChem
-  - Export options (CSV, SMILES, Report)
-
-- [x] **Explorer Page** (`bioflow/ui/pages/explorer.py`):
-  - 2D/3D embedding visualization
-  - Dimensionality reduction (t-SNE, PCA, UMAP)
-  - Modality filtering
-  - Cross-modal similarity heatmap
-  - Nearest neighbor search
-  - Cluster analysis with K-Means/DBSCAN
-
-- [x] **Data Management Page** (`bioflow/ui/pages/data.py`):
-  - Collection overview with metrics
-  - File upload (CSV, JSON, SMILES, FASTA)
-  - Data preview with molecule rendering
-  - Batch processing with progress
-  - Collection browsing and search
-  - Scheduled task management
-
-- [x] **Settings Page** (`bioflow/ui/pages/settings.py`):
-  - Qdrant connection configuration
-  - Model selection (PubMedBERT, ESM-2, ChemBERTa)
-  - Predictor configuration (DeepPurpose)
-  - Theme and appearance settings
-  - System status monitoring
-  - Resource usage (Memory, GPU, Storage)
-
-- [x] **Main App** (`bioflow/ui/app.py`):
-  - Navigation sidebar with routing
-  - User profile display
-  - Quick stats in sidebar
-
-**Launch:** `python launch_ui.py` or `streamlit run bioflow/ui/app.py`
+- [x] **Next.js Frontend** (`ui/`):
+  - Next.js 16 app router + Tailwind + shadcn/ui
+  - Dashboard pages: Discovery, 3D Visualization, Workflow Builder
+  - `/app/api/*` proxy routes to the FastAPI backend
+  - Optional mock fallbacks for molecules/proteins list routes
+
+**Launch:**
+- Full stack (Windows): `launch_bioflow_full.bat`
+- Manual:
+  - Backend: `python -m uvicorn bioflow.api.server:app --host 0.0.0.0 --port 8000`
+  - UI: `cd ui && pnpm dev`
 
 ---
 
 ## 🚀 Phase 5: Open-Source Alignment
-- **Laila Connector**: Allow Laila to query the Qdrant memory.
-- **InstaNovo+ Specs**: Add support for peptide sequencing integration.
-- **Controlled Generation**: Pilot generation via ProtBFN/AbBFN2.
+- **Strict Open-Source Compliance**: remove proprietary integrations and keep only OSS models/tools.
+- **Open Protein/Peptide Options**: integrate open models (e.g., ESM-2 / ProGen2) behind `BioGenerator`.
+- **Open Retrieval + Evidence**: improve evidence traceability (PubMed/UniProt/ChEMBL) and evaluation.

launch_bioflow.bat

@@ -6,10 +6,18 @@ echo  ║   🧬 BioFlow - AI-Powered Drug Discovery Platform        ║
 echo  ║                                                          ║
 echo  ╚══════════════════════════════════════════════════════════╝
 echo.
-echo Starting BioFlow UI...
+echo Starting BioFlow UI (Next.js)...
 echo.
 
 cd /d "%~dp0"
-python -m streamlit run bioflow/ui/app.py --server.port 8501
+if not exist "ui\package.json" (
+  echo ❌ Error: Next.js UI not found at .\ui
+  echo Run `launch_bioflow_full.bat` from the repo root.
+  pause
+  exit /b 1
+)
+
+cd /d "%~dp0\ui"
+pnpm dev
 
 pause

launch_bioflow_full.bat

@@ -22,7 +22,7 @@ start "BioFlow API" cmd /k "cd /d %~dp0 && python -m uvicorn bioflow.api.server:
 
 echo [2/2] Starting Next.js Frontend on port 3000...
 timeout /t 3 /nobreak > nul
-start "BioFlow UI" cmd /k "cd /d %~dp0\lacoste001\ui && pnpm dev"
+start "BioFlow UI" cmd /k "cd /d %~dp0\ui && pnpm dev"
 
 echo.
 echo ============================================

launch_ui.py

@@ -2,11 +2,11 @@
 BioFlow UI Launch Script
 =========================
 
-Quick launcher for the BioFlow Streamlit application.
+Quick launcher for the BioFlow Next.js application.
 
 Usage:
     python launch_ui.py
-    python launch_ui.py --port 8502
+    python launch_ui.py --port 3001
     python launch_ui.py --debug
 """
 
@@ -18,16 +18,15 @@ from pathlib import Path
 
 def main():
     """Launch the BioFlow UI."""
-    # Get the app path
     script_dir = Path(__file__).parent
-    app_path = script_dir / "bioflow" / "ui" / "app.py"
-    
-    if not app_path.exists():
-        print(f"❌ Error: App not found at {app_path}")
+    ui_dir = script_dir / "ui"
+
+    if not (ui_dir / "package.json").exists():
+        print(f"❌ Error: Next.js UI not found at {ui_dir}")
         sys.exit(1)
     
     # Parse arguments
-    port = 8501
+    port = 3000
     debug = False
     
     for i, arg in enumerate(sys.argv[1:]):
@@ -36,17 +35,10 @@ def main():
         elif arg == "--debug":
             debug = True
     
-    # Build command
-    cmd = [
-        sys.executable, "-m", "streamlit", "run",
-        str(app_path),
-        "--server.port", str(port),
-        "--server.headless", "false",
-        "--browser.gatherUsageStats", "false",
-    ]
-    
+    env = os.environ.copy()
+    env["PORT"] = str(port)
     if debug:
-        cmd.extend(["--logger.level", "debug"])
+        env["NODE_OPTIONS"] = env.get("NODE_OPTIONS", "") + " --trace-warnings"
     
     print(f"""
     ╔══════════════════════════════════════════════════════════╗
@@ -59,7 +51,7 @@ def main():
     """)
     
     try:
-        subprocess.run(cmd, cwd=str(script_dir))
+        subprocess.run(["pnpm", "dev"], cwd=str(ui_dir), env=env, check=False)
     except KeyboardInterrupt:
         print("\n\n👋 BioFlow server stopped.")
     except Exception as e:

open_biomed/__init__.py

@@ -1,6 +1,17 @@
-from open_biomed.utils import *
-from open_biomed.models import *
-from open_biomed.datasets import *
-from open_biomed.tasks import *
-from open_biomed.core import *
-from open_biomed.scripts import *
+"""
+OpenBioMed package (vendored)
+=============================
+
+This repository vendors an upstream OpenBioMed codebase. For BioFlow, OpenBioMed
+is treated as an optional dependency: BioFlow must remain importable even when
+some heavy optional dependencies (e.g., `scanpy`) are not installed.
+
+To avoid import-time failures, this package intentionally does not `import *`
+from all submodules at import time. Import the specific subpackages you need:
+
+  - `open_biomed.core.*`
+  - `open_biomed.data.*`
+  - `open_biomed.models.*`
+"""
+
+__all__ = []

open_biomed/core/llm_request.py

@@ -9,8 +9,7 @@ import shutil
 from datetime import datetime
 from typing import List, Optional, TypedDict, AsyncGenerator, Tuple
 from typing_extensions import Self
-from openai import OpenAI, Stream
-from openai.types.chat.chat_completion_chunk import ChatCompletionChunk, ChoiceDelta
+import requests
 
 from open_biomed.data import Text
 from open_biomed.utils.config import Config
@@ -77,7 +76,7 @@ class LLM_Local(LLM):
         except:
             raise ValueError("Only support BioMedGPTR1 and BioMedGPT for now.")
 
-        self._init_client(api_infos)
+        self._init_client(model_name_or_path=model_name_or_path, device=device)
     
     def _init_client(self, model_name_or_path: str, device: Optional[str]=None) -> Self:
         self.client = self.client_model.from_pretrained(model_name_or_path=model_name_or_path, device=device)
@@ -111,11 +110,15 @@ class LLM_API(LLM):
         self.think_start, self.think_end = "<think>", "</think>"
     
     def _init_client(self, api_infos: dict) -> Self:
-
-        self.client = OpenAI(
-            api_key=api_infos['api_key'],
-            base_url=api_infos['api_url']
-            )
+        # NOTE: OpenAI SDK usage is intentionally avoided to keep the project fully open-source.
+        # This client speaks a minimal OpenAI-compatible HTTP interface (e.g., local vLLM/Ollama proxy).
+        self.api_key = api_infos.get("api_key")
+        self.api_url = (api_infos.get("api_url") or "").rstrip("/")
+        if not self.api_url:
+            raise ValueError("API_URL is required for LLM_API (set in .env)")
+        if not api_infos.get("model_name"):
+            raise ValueError("MODEL_NAME is required for LLM_API (set in .env)")
+        return self
 
     
     def _update_query(self, query: str, context: ContextDict = {"ref_text": "", "others": dict()}) -> str:
@@ -131,46 +134,54 @@ class LLM_API(LLM):
 
         return messages
 
-    async def generate_stream(self, query: str, context: ContextDict = {"ref_text": "", "others": dict()}, is_debug=False):
-
-        messages = self._get_input(query=query, context=context, is_debug=is_debug)
-
-        stream: Stream[ChatCompletionChunk] = self.client.chat.completions.create(
-            model=self.model_name,
-            messages=messages,
-            stream=True,
-        )
-
-        is_think = False
-        for chunk in stream:
-            delta: ChoiceDelta = chunk.choices[0].delta
-            delta_json = delta.model_dump()
-            content = delta_json.get("content", "")
-            if content != "" or content!=None:
-                if content == self.think_start:
-                    is_think = True
-                    continue
-                elif content == self.think_end:
-                    is_think = False
-                    continue
-                elif is_think:
-                    stream_chunk: StreamChunk = {"final_resp": "", "reasoning": content}
-                else:
-                    stream_chunk: StreamChunk = {"final_resp": content, "reasoning": ""}
-            yield stream_chunk
+    async def generate_stream(
+        self,
+        query: str,
+        context: ContextDict = {"ref_text": "", "others": dict()},
+        is_debug: bool = False,
+    ) -> AsyncGenerator[StreamChunk, None]:
+        """
+        Best-effort streaming adapter.
+
+        This implementation does not rely on proprietary SDKs. If you need true token streaming,
+        run an OpenAI-compatible server that supports SSE streaming and implement parsing here.
+        """
+        resp = self.generate(query=query, context=context, is_debug=is_debug)
+        yield {"final_resp": resp.get("final_resp", ""), "reasoning": resp.get("reasoning", "")}
 
     def generate(self, query: str, context: ContextDict = {"ref_text": "", "others": dict()}, is_debug=False):
 
         messages = self._get_input(query=query, context=context, is_debug=is_debug)
+        # Build OpenAI-compatible endpoint
+        base = self.api_url
+        if base.endswith("/v1"):
+            endpoint = f"{base}/chat/completions"
+        elif "/v1/" in base:
+            # If user provided a deeper URL, try to append the standard path safely.
+            endpoint = f"{base.rstrip('/')}/chat/completions"
+        else:
+            endpoint = f"{base}/v1/chat/completions"
+
+        headers = {"Content-Type": "application/json"}
+        if self.api_key:
+            headers["Authorization"] = f"Bearer {self.api_key}"
+
+        payload = {
+            "model": self.model_name,
+            "messages": messages,
+            "stream": False,
+            "temperature": self.temperature,
+        }
 
-        response = self.client.chat.completions.create(
-            model=self.model_name,
-            messages=messages,
-            stream=False,
-            temperature=self.temperature
-        )
+        r = requests.post(endpoint, json=payload, headers=headers, timeout=120)
+        r.raise_for_status()
+        data = r.json()
 
-        text=response.choices[0].message.content
+        text = (
+            data.get("choices", [{}])[0]
+            .get("message", {})
+            .get("content", "")
+        )
         start_index = text.find(self.think_start) + len(self.think_start)
         end_index = text.find(self.think_end)
         resp_thinking = text[start_index:end_index].strip()

open_biomed/data/__init__.py

@@ -1,5 +1,10 @@
 from open_biomed.data.molecule import *
 from open_biomed.data.protein import *
 from open_biomed.data.pocket import *
-from open_biomed.data.cell import *
-from open_biomed.data.text import *
+from open_biomed.data.text import *
+
+# Optional modality (heavy dependency).
+try:
+    from open_biomed.data.cell import *  # type: ignore
+except ImportError:
+    pass

open_biomed/data/molecule.py

@@ -12,8 +12,17 @@ from rdkit import Chem, DataStructs, RDLogger
 RDLogger.DisableLog("rdApp.*")
 from rdkit.Chem import AllChem, MACCSkeys, rdMolDescriptors, Descriptors, Lipinski
 from rdkit.Chem.AllChem import RWMol
-from rdkit.six import iteritems
-from rdkit.six.moves import cPickle
+try:
+    # Older RDKit vendored `six` utilities here.
+    from rdkit.six import iteritems  # type: ignore
+except Exception:
+    def iteritems(d):
+        return d.items()
+
+try:
+    from rdkit.six.moves import cPickle  # type: ignore
+except Exception:
+    import pickle as cPickle
 import re
 
 from open_biomed.core.tool import Tool
@@ -458,4 +467,4 @@ class MoleculeSimilarityTool(Tool):
 
     def run(self, molecule_1: Molecule, molecule_2: Molecule) -> float:
         return molecule_fingerprint_similarity(molecule_1, molecule_2, fingerprint_type="morgan")
-        
+        

qdrant_data/.lock

@@ -1 +0,0 @@
-tmp lock file

qdrant_data/collection/bioflow_memory/storage.sqlite

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1c3e8b65a92a7fd78e4c3d3cb9dac1ded03b80544437fb331469ba98046d6e8b
-size 409600

qdrant_data/collection/molecules/storage.sqlite

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:83a947b90cba756851eb0b4cc10fa98f98c58936296716e81bf17a053d2679ed
-size 45056

qdrant_data/meta.json

@@ -1 +0,0 @@
-{"collections": {"molecules": {"vectors": {"size": 768, "distance": "Cosine", "hnsw_config": null, "quantization_config": null, "on_disk": null, "datatype": null, "multivector_config": null}, "shard_number": null, "sharding_method": null, "replication_factor": null, "write_consistency_factor": null, "on_disk_payload": null, "hnsw_config": null, "wal_config": null, "optimizers_config": null, "quantization_config": null, "sparse_vectors": null, "strict_mode_config": null, "metadata": null}, "bioflow_memory": {"vectors": {"size": 768, "distance": "Cosine", "hnsw_config": null, "quantization_config": null, "on_disk": null, "datatype": null, "multivector_config": null}, "shard_number": null, "sharding_method": null, "replication_factor": null, "write_consistency_factor": null, "on_disk_payload": null, "hnsw_config": null, "wal_config": null, "optimizers_config": null, "quantization_config": null, "sparse_vectors": null, "strict_mode_config": null, "metadata": null}}, "aliases": {}}

requirements.txt

@@ -10,14 +10,13 @@ numpy==1.26.4
 absl-py==2.1.0
 easydict==1.13
 ratelimiter==1.2.0.post0
-openai==1.64.0
+requests>=2.31.0
 uvicorn===0.32.1
 fastapi===0.115.5
 oss2==2.19.1
-dotenv==1.0.1
+python-dotenv>=1.0.1
 protobuf==5.28.3
 scanpy==1.10.3
 qdrant-client>=1.7.0
-streamlit>=1.28.0
 plotly>=5.18.0
 scikit-learn>=1.3.0

scripts/benchmark_mmr.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+"""
+Compare /api/search latency + diversity with and without MMR.
+"""
+import os
+import statistics
+import time
+import requests
+
+BASE_URL = os.getenv("BIOFLOW_API_URL", "http://localhost:8000")
+
+QUERIES = [
+    "EGFR inhibitor",
+    "BRCA1 breast cancer",
+    "kinase inhibitor therapy",
+    "TP53 mutation cancer",
+    "lung cancer EGFR signaling",
+]
+
+
+def run_batch(use_mmr: bool, runs: int = 10):
+    latencies = []
+    diversities = []
+    for i in range(runs):
+        q = QUERIES[i % len(QUERIES)]
+        t0 = time.perf_counter()
+        r = requests.post(
+            f"{BASE_URL}/api/search",
+            json={"query": q, "top_k": 20, "use_mmr": use_mmr, "modality": "auto"},
+            timeout=30,
+        )
+        dt = (time.perf_counter() - t0) * 1000.0
+        if r.status_code != 200:
+            continue
+        data = r.json()
+        latencies.append(dt)
+        if data.get("diversity_score") is not None:
+            diversities.append(float(data.get("diversity_score") or 0))
+    return latencies, diversities
+
+
+if __name__ == "__main__":
+    print("MMR Benchmark")
+    lat_no, div_no = run_batch(False, runs=10)
+    lat_yes, div_yes = run_batch(True, runs=10)
+
+    def _stats(xs):
+        if not xs:
+            return "n/a"
+        return f"p50={statistics.median(xs):.1f}ms avg={statistics.mean(xs):.1f}ms"
+
+    print(f"MMR OFF: {_stats(lat_no)} | diversity avg={statistics.mean(div_no) if div_no else 'n/a'}")
+    print(f"MMR ON : {_stats(lat_yes)} | diversity avg={statistics.mean(div_yes) if div_yes else 'n/a'}")

scripts/benchmark_search_api.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""
+Benchmark /api/search latency and error rate.
+
+Usage:
+  python scripts/benchmark_search_api.py --runs 50 --concurrency 5
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures as cf
+import statistics
+import time
+from typing import Any, Dict, List, Tuple
+
+import requests
+
+
+DEFAULT_QUERIES = [
+    "EGFR inhibitor",
+    "BRCA1 breast cancer",
+    "kinase inhibitor therapy",
+    "TP53 mutation cancer",
+    "lung cancer EGFR signaling",
+]
+
+
+def _one(base_url: str, query: str, top_k: int, use_mmr: bool) -> Tuple[bool, float, str]:
+    t0 = time.perf_counter()
+    try:
+        r = requests.post(
+            f"{base_url}/api/search",
+            json={"query": query, "top_k": top_k, "use_mmr": use_mmr, "modality": "auto"},
+            timeout=30,
+        )
+        dt_ms = (time.perf_counter() - t0) * 1000.0
+        if r.status_code != 200:
+            return False, dt_ms, f"HTTP {r.status_code}"
+        return True, dt_ms, ""
+    except Exception as e:
+        dt_ms = (time.perf_counter() - t0) * 1000.0
+        return False, dt_ms, str(e)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--base-url", default="http://localhost:8000")
+    ap.add_argument("--runs", type=int, default=50)
+    ap.add_argument("--concurrency", type=int, default=5)
+    ap.add_argument("--top-k", type=int, default=20)
+    ap.add_argument("--mmr", action="store_true", help="Enable MMR")
+    args = ap.parse_args()
+
+    queries = (DEFAULT_QUERIES * ((args.runs // len(DEFAULT_QUERIES)) + 1))[: args.runs]
+
+    latencies: List[float] = []
+    errors: List[str] = []
+
+    with cf.ThreadPoolExecutor(max_workers=args.concurrency) as ex:
+        futures = [
+            ex.submit(_one, args.base_url, q, args.top_k, bool(args.mmr))
+            for q in queries
+        ]
+        for f in cf.as_completed(futures):
+            ok, dt_ms, err = f.result()
+            latencies.append(dt_ms)
+            if not ok:
+                errors.append(err)
+
+    latencies.sort()
+    p50 = latencies[int(0.50 * (len(latencies) - 1))]
+    p95 = latencies[int(0.95 * (len(latencies) - 1))]
+    p99 = latencies[int(0.99 * (len(latencies) - 1))]
+
+    print("=" * 60)
+    print("BioFlow /api/search Benchmark")
+    print("=" * 60)
+    print(f"Runs: {args.runs} | Concurrency: {args.concurrency} | top_k: {args.top_k} | mmr: {bool(args.mmr)}")
+    print(f"OK: {args.runs - len(errors)} | Errors: {len(errors)}")
+    print(f"p50: {p50:.1f}ms | p95: {p95:.1f}ms | p99: {p99:.1f}ms | mean: {statistics.mean(latencies):.1f}ms")
+    if errors:
+        print("Sample errors:")
+        for e in errors[:5]:
+            print(f"  - {e}")
+
+    # Non-zero exit on errors to allow CI usage.
+    return 1 if errors else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+

scripts/evaluate_retrieval.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+"""
+Evaluate retrieval quality against a small benchmark file.
+
+Benchmark format (JSON):
+{
+  "queries": [
+    {
+      "name": "egfr_lung",
+      "query": "EGFR lung cancer",
+      "modality": "auto",
+      "top_k": 20,
+      "relevant_ids": ["..."],
+      "relevance_by_id": {"...": 1.0, "...": 2.0}
+    }
+  ],
+  "k": 10
+}
+
+Notes:
+- `relevant_ids` is used for Recall@k and MRR@k.
+- `relevance_by_id` is used for nDCG@k.
+- You can provide either or both.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any, Dict, List, Set
+
+import requests
+
+from bioflow.evaluation.metrics import mrr_at_k, ndcg_at_k, recall_at_k
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--benchmark", required=True, help="Path to benchmark JSON")
+    ap.add_argument("--base-url", default="http://localhost:8000")
+    args = ap.parse_args()
+
+    bench = json.loads(Path(args.benchmark).read_text(encoding="utf-8"))
+    k = int(bench.get("k", 10))
+    queries = bench.get("queries", [])
+    if not queries:
+        raise SystemExit("Benchmark has no queries")
+
+    recalls: List[float] = []
+    mrrs: List[float] = []
+    ndcgs: List[float] = []
+
+    for q in queries:
+        query = q["query"]
+        modality = q.get("modality", "auto")
+        top_k = int(q.get("top_k", max(k, 20)))
+
+        r = requests.post(
+            f"{args.base_url}/api/search",
+            json={"query": query, "modality": modality, "top_k": top_k, "use_mmr": False},
+            timeout=60,
+        )
+        r.raise_for_status()
+        data = r.json()
+
+        ranked_ids = [str(item.get("id")) for item in data.get("results", []) if item.get("id") is not None]
+
+        relevant_ids = set(map(str, q.get("relevant_ids", [])))
+        relevance_by_id = {str(k): float(v) for k, v in (q.get("relevance_by_id", {}) or {}).items()}
+
+        if relevant_ids:
+            recalls.append(recall_at_k(relevant_ids, ranked_ids, k))
+            mrrs.append(mrr_at_k(relevant_ids, ranked_ids, k))
+
+        if relevance_by_id:
+            ndcgs.append(ndcg_at_k(relevance_by_id, ranked_ids, k))
+
+        print(f"- {q.get('name', query[:30])}: got={len(ranked_ids)} recall@{k}={recalls[-1] if relevant_ids else 'n/a'} mrr@{k}={mrrs[-1] if relevant_ids else 'n/a'} ndcg@{k}={ndcgs[-1] if relevance_by_id else 'n/a'}")
+
+    def _avg(xs: List[float]) -> float:
+        return sum(xs) / float(len(xs)) if xs else 0.0
+
+    print("=" * 60)
+    print(f"Aggregate (@{k})")
+    if recalls:
+        print(f"Recall: {_avg(recalls):.4f}")
+        print(f"MRR:    {_avg(mrrs):.4f}")
+    if ndcgs:
+        print(f"nDCG:   {_avg(ndcgs):.4f}")
+    if not (recalls or ndcgs):
+        print("No relevance labels provided; nothing to score.")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+

scripts/evidence_audit.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+"""
+Audit evidence-link coverage in /api/search results.
+"""
+import os
+import requests
+
+BASE_URL = os.getenv("BIOFLOW_API_URL", "http://localhost:8000")
+
+QUERIES = [
+    "EGFR inhibitor",
+    "BRCA1 breast cancer",
+    "kinase inhibitor therapy",
+    "TP53 mutation cancer",
+]
+
+
+def main():
+    total = 0
+    with_evidence = 0
+
+    for q in QUERIES:
+        r = requests.post(
+            f"{BASE_URL}/api/search",
+            json={"query": q, "top_k": 10, "use_mmr": True},
+            timeout=30,
+        )
+        if r.status_code != 200:
+            print(f"[SKIP] {q} -> {r.status_code}")
+            continue
+        data = r.json()
+        for item in data.get("results", []):
+            total += 1
+            if item.get("evidence_links"):
+                with_evidence += 1
+
+    if total == 0:
+        print("No results returned; evidence audit skipped.")
+        return 0
+
+    coverage = (with_evidence / total) * 100.0
+    print(f"Evidence coverage: {with_evidence}/{total} ({coverage:.1f}%)")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+

scripts/run_tests.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""
+CI-style test runner for core BioFlow checks.
+"""
+import os
+import subprocess
+import sys
+
+
+TESTS = [
+    ["python", "test_search_api.py"],
+    ["python", "test_agent_api.py"],
+    ["python", "test_search_filters.py"],
+    ["python", "test_phase4_ui.py"],
+    ["python", "test_ingestion_api.py"],
+]
+
+
+def main() -> int:
+    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    os.chdir(root)
+    failed = False
+
+    for cmd in TESTS:
+        print("=" * 80)
+        print("Running:", " ".join(cmd))
+        result = subprocess.run(cmd, check=False)
+        if result.returncode != 0:
+            failed = True
+
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+