62 appify the demo (#122)
Browse filesCo-authored-by: Andres Marafioti <[email protected]>
- .gitignore +3 -0
- README.md +21 -0
- index.html +125 -0
- pyproject.toml +8 -1
- src/reachy_mini_conversation_app/camera_worker.py +9 -3
- src/reachy_mini_conversation_app/config.py +28 -3
- src/reachy_mini_conversation_app/console.py +339 -6
- src/reachy_mini_conversation_app/gradio_personality.py +301 -0
- src/reachy_mini_conversation_app/headless_personality.py +102 -0
- src/reachy_mini_conversation_app/headless_personality_ui.py +276 -0
- src/reachy_mini_conversation_app/main.py +117 -34
- src/reachy_mini_conversation_app/openai_realtime.py +296 -32
- src/reachy_mini_conversation_app/prompts.py +20 -0
- src/reachy_mini_conversation_app/static/index.html +116 -0
- src/reachy_mini_conversation_app/static/main.js +496 -0
- src/reachy_mini_conversation_app/static/style.css +317 -0
- src/reachy_mini_conversation_app/utils.py +2 -2
- src/reachy_mini_conversation_app/vision/processors.py +4 -3
- src/reachy_mini_conversation_app/vision/yolo_head_tracker.py +1 -1
- style.css +386 -0
.gitignore
CHANGED
|
@@ -56,3 +56,6 @@ cache/
|
|
| 56 |
.directory
|
| 57 |
.Trash-*
|
| 58 |
.nfs*
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
.directory
|
| 57 |
.Trash-*
|
| 58 |
.nfs*
|
| 59 |
+
|
| 60 |
+
# User-created personalities (managed by UI)
|
| 61 |
+
src/reachy_mini_conversation_app/profiles/user_personalities/
|
README.md
CHANGED
|
@@ -1,3 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Reachy Mini conversation app
|
| 2 |
|
| 3 |
Conversational app for the Reachy Mini robot combining OpenAI's realtime APIs, vision pipelines, and choreographed motion libraries.
|
|
@@ -200,6 +213,14 @@ Tools are resolved first from Python files in the profile folder (custom tools),
|
|
| 200 |
On top of built-in tools found in the shared library, you can implement custom tools specific to your profile by adding Python files in the profile folder.
|
| 201 |
Custom tools must subclass `reachy_mini_conversation_app.tools.core_tools.Tool` (see `profiles/example/sweep_look.py`).
|
| 202 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 203 |
|
| 204 |
|
| 205 |
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Reachy Mini Conversation App
|
| 3 |
+
emoji: 🎤
|
| 4 |
+
colorFrom: red
|
| 5 |
+
colorTo: blue
|
| 6 |
+
sdk: static
|
| 7 |
+
pinned: false
|
| 8 |
+
short_description: Talk with Reachy Mini !
|
| 9 |
+
tags:
|
| 10 |
+
- reachy_mini
|
| 11 |
+
- reachy_mini_python_app
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
# Reachy Mini conversation app
|
| 15 |
|
| 16 |
Conversational app for the Reachy Mini robot combining OpenAI's realtime APIs, vision pipelines, and choreographed motion libraries.
|
|
|
|
| 213 |
On top of built-in tools found in the shared library, you can implement custom tools specific to your profile by adding Python files in the profile folder.
|
| 214 |
Custom tools must subclass `reachy_mini_conversation_app.tools.core_tools.Tool` (see `profiles/example/sweep_look.py`).
|
| 215 |
|
| 216 |
+
### Edit personalities from the UI
|
| 217 |
+
When running with `--gradio`, open the “Personality” accordion:
|
| 218 |
+
- Select among available profiles (folders under `src/reachy_mini_conversation_app/profiles/`) or the built‑in default.
|
| 219 |
+
- Click “Apply” to update the current session instructions live.
|
| 220 |
+
- Create a new personality by entering a name and instructions text; it stores files under `profiles/<name>/` and copies `tools.txt` from the `default` profile.
|
| 221 |
+
|
| 222 |
+
Note: The “Personality” panel updates the conversation instructions. Tool sets are loaded at startup from `tools.txt` and are not hot‑reloaded.
|
| 223 |
+
|
| 224 |
|
| 225 |
|
| 226 |
|
index.html
ADDED
|
@@ -0,0 +1,125 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html>
|
| 3 |
+
|
| 4 |
+
<head>
|
| 5 |
+
<meta charset="utf-8" />
|
| 6 |
+
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
| 7 |
+
<title>Reachy Mini Conversation App</title>
|
| 8 |
+
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 9 |
+
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
| 10 |
+
<link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&family=Manrope:wght@400;500;600&display=swap" rel="stylesheet">
|
| 11 |
+
<link rel="stylesheet" href="style.css" />
|
| 12 |
+
</head>
|
| 13 |
+
|
| 14 |
+
<body>
|
| 15 |
+
<header class="hero">
|
| 16 |
+
<div class="topline">
|
| 17 |
+
<div class="brand">
|
| 18 |
+
<span class="logo">🤖</span>
|
| 19 |
+
<span class="brand-name">Reachy Mini</span>
|
| 20 |
+
</div>
|
| 21 |
+
<div class="pill">Realtime voice · Vision aware · Expressive motion</div>
|
| 22 |
+
</div>
|
| 23 |
+
<div class="hero-grid">
|
| 24 |
+
<div class="hero-copy">
|
| 25 |
+
<p class="eyebrow">Conversation App</p>
|
| 26 |
+
<h1>Talk, see, and move together.</h1>
|
| 27 |
+
<p class="lede">
|
| 28 |
+
A friendly, camera-aware companion for Reachy Mini. Chat out loud, watch it follow faces, dance, or react with recorded emotions—all while streaming transcripts in a clean web UI.
|
| 29 |
+
</p>
|
| 30 |
+
<div class="hero-actions">
|
| 31 |
+
<a class="btn primary" href="#highlights">Explore features</a>
|
| 32 |
+
<a class="btn ghost" href="#story">See how it feels</a>
|
| 33 |
+
</div>
|
| 34 |
+
<div class="hero-badges">
|
| 35 |
+
<span>Low-latency voice loop</span>
|
| 36 |
+
<span>Camera insights on demand</span>
|
| 37 |
+
<span>Choreographed dances & emotions</span>
|
| 38 |
+
<span>Personality profiles via web UI</span>
|
| 39 |
+
</div>
|
| 40 |
+
</div>
|
| 41 |
+
<div class="hero-visual">
|
| 42 |
+
<div class="glass-card">
|
| 43 |
+
<img src="docs/assets/reachy_mini_dance.gif" alt="Reachy Mini dancing" class="hero-gif">
|
| 44 |
+
<p class="caption">Reachy Mini can move, dance, and emote while holding a natural conversation.</p>
|
| 45 |
+
</div>
|
| 46 |
+
</div>
|
| 47 |
+
</div>
|
| 48 |
+
</header>
|
| 49 |
+
|
| 50 |
+
<section id="highlights" class="section features">
|
| 51 |
+
<div class="section-header">
|
| 52 |
+
<p class="eyebrow">What’s inside</p>
|
| 53 |
+
<h2>All-in-one conversational layer for your robot</h2>
|
| 54 |
+
<p class="intro">
|
| 55 |
+
The app blends realtime speech, vision, and motion so Reachy Mini feels present..
|
| 56 |
+
</p>
|
| 57 |
+
</div>
|
| 58 |
+
<div class="feature-grid">
|
| 59 |
+
<div class="feature-card">
|
| 60 |
+
<span class="icon">🎤</span>
|
| 61 |
+
<h3>Natural voice chat</h3>
|
| 62 |
+
<p>Talk freely and get fast, high-quality replies powered by realtime models.</p>
|
| 63 |
+
</div>
|
| 64 |
+
<div class="feature-card">
|
| 65 |
+
<span class="icon">🎥</span>
|
| 66 |
+
<h3>Vision-aware replies</h3>
|
| 67 |
+
<p>Ask the camera tool to see what’s in front, track a face, or keep attention on whoever is speaking.</p>
|
| 68 |
+
</div>
|
| 69 |
+
<div class="feature-card">
|
| 70 |
+
<span class="icon">💃</span>
|
| 71 |
+
<h3>Expressive motion</h3>
|
| 72 |
+
<p>Queue dances, play recorded emotions while Reachy listens and talks.</p>
|
| 73 |
+
</div>
|
| 74 |
+
<div class="feature-card">
|
| 75 |
+
<span class="icon">🧠</span>
|
| 76 |
+
<h3>Personalities on demand</h3>
|
| 77 |
+
<p>Switch conversation styles through profiles and decide which tools (dance, camera, tracking) each persona can use.</p>
|
| 78 |
+
</div>
|
| 79 |
+
<div class="feature-card">
|
| 80 |
+
<span class="icon">🌐</span>
|
| 81 |
+
<h3>Ready for your setup</h3>
|
| 82 |
+
<p>Works with wired or wireless Reachy Mini, and can run vision locally or through the default cloud model.</p>
|
| 83 |
+
</div>
|
| 84 |
+
</div>
|
| 85 |
+
</section>
|
| 86 |
+
|
| 87 |
+
<section id="story" class="section story">
|
| 88 |
+
<div class="story-grid">
|
| 89 |
+
<div class="story-card">
|
| 90 |
+
<p class="eyebrow">How it feels</p>
|
| 91 |
+
<h3>From hello to helpful in seconds</h3>
|
| 92 |
+
<ul class="story-list">
|
| 93 |
+
<li><span>👋</span> Say “Hey Reachy” and start chatting—no extra setup in the moment.</li>
|
| 94 |
+
<li><span>👀</span> Ask what it sees; it can peek through the camera or keep focus on your face.</li>
|
| 95 |
+
<li><span>🎭</span> Trigger emotions or dance breaks to keep the conversation lively.</li>
|
| 96 |
+
<li><span>📝</span> Follow along with live transcripts in the web UI or run audio-only from the console.</li>
|
| 97 |
+
</ul>
|
| 98 |
+
</div>
|
| 99 |
+
<div class="story-card secondary">
|
| 100 |
+
<p class="eyebrow">Where it shines</p>
|
| 101 |
+
<h3>Great for demos, teaching, and playful exploration</h3>
|
| 102 |
+
<p class="story-text">
|
| 103 |
+
Show off how Reachy Mini listens, responds, and moves in sync. Whether you’re guiding a class, hosting a booth, or experimenting at home, the app keeps the robot expressive without juggling scripts or joystick controls.
|
| 104 |
+
</p>
|
| 105 |
+
<div class="chips">
|
| 106 |
+
<span class="chip">Live conversation</span>
|
| 107 |
+
<span class="chip">Face tracking</span>
|
| 108 |
+
<span class="chip">Camera tool</span>
|
| 109 |
+
<span class="chip">Dance library</span>
|
| 110 |
+
<span class="chip">Profiles & tools</span>
|
| 111 |
+
</div>
|
| 112 |
+
</div>
|
| 113 |
+
</div>
|
| 114 |
+
</section>
|
| 115 |
+
|
| 116 |
+
<footer class="footer">
|
| 117 |
+
<p>
|
| 118 |
+
Reachy Mini Conversation App by <a href="https://github.com/pollen-robotics" target="_blank" rel="noopener">Pollen Robotics</a>.
|
| 119 |
+
Explore more apps on <a href="https://huggingface.co/spaces/pollen-robotics/Reachy_Mini_Apps" target="_blank" rel="noopener">Hugging Face Spaces</a>.
|
| 120 |
+
</p>
|
| 121 |
+
</footer>
|
| 122 |
+
|
| 123 |
+
</body>
|
| 124 |
+
|
| 125 |
+
</html>
|
pyproject.toml
CHANGED
|
@@ -13,7 +13,7 @@ dependencies = [
|
|
| 13 |
#Media
|
| 14 |
"aiortc>=1.13.0",
|
| 15 |
"fastrtc>=0.0.34",
|
| 16 |
-
"gradio
|
| 17 |
"huggingface_hub>=0.34.4",
|
| 18 |
"opencv-python>=4.12.0.88",
|
| 19 |
|
|
@@ -57,6 +57,9 @@ dev = [
|
|
| 57 |
[project.scripts]
|
| 58 |
reachy-mini-conversation-app = "reachy_mini_conversation_app.main:main"
|
| 59 |
|
|
|
|
|
|
|
|
|
|
| 60 |
[tool.setuptools]
|
| 61 |
package-dir = { "" = "src" }
|
| 62 |
include-package-data = true
|
|
@@ -67,8 +70,12 @@ where = ["src"]
|
|
| 67 |
[tool.setuptools.package-data]
|
| 68 |
reachy_mini_conversation_app = [
|
| 69 |
"images/*",
|
|
|
|
|
|
|
| 70 |
"demos/**/*.txt",
|
| 71 |
"prompts_library/*.txt",
|
|
|
|
|
|
|
| 72 |
]
|
| 73 |
|
| 74 |
[tool.ruff]
|
|
|
|
| 13 |
#Media
|
| 14 |
"aiortc>=1.13.0",
|
| 15 |
"fastrtc>=0.0.34",
|
| 16 |
+
"gradio==5.50.1.dev1",
|
| 17 |
"huggingface_hub>=0.34.4",
|
| 18 |
"opencv-python>=4.12.0.88",
|
| 19 |
|
|
|
|
| 57 |
[project.scripts]
|
| 58 |
reachy-mini-conversation-app = "reachy_mini_conversation_app.main:main"
|
| 59 |
|
| 60 |
+
[project.entry-points."reachy_mini_apps"]
|
| 61 |
+
reachy_mini_conversation_app = "reachy_mini_conversation_app.main:ReachyMiniConversationApp"
|
| 62 |
+
|
| 63 |
[tool.setuptools]
|
| 64 |
package-dir = { "" = "src" }
|
| 65 |
include-package-data = true
|
|
|
|
| 70 |
[tool.setuptools.package-data]
|
| 71 |
reachy_mini_conversation_app = [
|
| 72 |
"images/*",
|
| 73 |
+
"static/*",
|
| 74 |
+
".env.example",
|
| 75 |
"demos/**/*.txt",
|
| 76 |
"prompts_library/*.txt",
|
| 77 |
+
"profiles/**/*.txt",
|
| 78 |
+
"prompts/**/*.txt",
|
| 79 |
]
|
| 80 |
|
| 81 |
[tool.ruff]
|
src/reachy_mini_conversation_app/camera_worker.py
CHANGED
|
@@ -156,6 +156,9 @@ class CameraWorker:
|
|
| 156 |
translation = target_pose[:3, 3]
|
| 157 |
rotation = R.from_matrix(target_pose[:3, :3]).as_euler("xyz", degrees=False)
|
| 158 |
|
|
|
|
|
|
|
|
|
|
| 159 |
# Thread-safe update of face tracking offsets (use pose as-is)
|
| 160 |
with self.face_tracking_lock:
|
| 161 |
self.face_tracking_offsets = [
|
|
@@ -189,7 +192,8 @@ class CameraWorker:
|
|
| 189 |
pose_matrix = np.eye(4, dtype=np.float32)
|
| 190 |
pose_matrix[:3, 3] = current_translation
|
| 191 |
pose_matrix[:3, :3] = R.from_euler(
|
| 192 |
-
"xyz",
|
|
|
|
| 193 |
).as_matrix()
|
| 194 |
self.interpolation_start_pose = pose_matrix
|
| 195 |
|
|
@@ -199,7 +203,9 @@ class CameraWorker:
|
|
| 199 |
|
| 200 |
# Interpolate between current pose and neutral pose
|
| 201 |
interpolated_pose = linear_pose_interpolation(
|
| 202 |
-
self.interpolation_start_pose,
|
|
|
|
|
|
|
| 203 |
)
|
| 204 |
|
| 205 |
# Extract translation and rotation from interpolated pose
|
|
@@ -225,7 +231,7 @@ class CameraWorker:
|
|
| 225 |
# else: Keep current offsets (within 2s delay period)
|
| 226 |
|
| 227 |
# Small sleep to prevent excessive CPU usage (same as main_works.py)
|
| 228 |
-
time.sleep(0.
|
| 229 |
|
| 230 |
except Exception as e:
|
| 231 |
logger.error(f"Camera worker error: {e}")
|
|
|
|
| 156 |
translation = target_pose[:3, 3]
|
| 157 |
rotation = R.from_matrix(target_pose[:3, :3]).as_euler("xyz", degrees=False)
|
| 158 |
|
| 159 |
+
translation *= 0.5 # Scale down translation effect
|
| 160 |
+
rotation *= 0.5 # Scale down rotation effect
|
| 161 |
+
|
| 162 |
# Thread-safe update of face tracking offsets (use pose as-is)
|
| 163 |
with self.face_tracking_lock:
|
| 164 |
self.face_tracking_offsets = [
|
|
|
|
| 192 |
pose_matrix = np.eye(4, dtype=np.float32)
|
| 193 |
pose_matrix[:3, 3] = current_translation
|
| 194 |
pose_matrix[:3, :3] = R.from_euler(
|
| 195 |
+
"xyz",
|
| 196 |
+
current_rotation_euler,
|
| 197 |
).as_matrix()
|
| 198 |
self.interpolation_start_pose = pose_matrix
|
| 199 |
|
|
|
|
| 203 |
|
| 204 |
# Interpolate between current pose and neutral pose
|
| 205 |
interpolated_pose = linear_pose_interpolation(
|
| 206 |
+
self.interpolation_start_pose,
|
| 207 |
+
neutral_pose,
|
| 208 |
+
t,
|
| 209 |
)
|
| 210 |
|
| 211 |
# Extract translation and rotation from interpolated pose
|
|
|
|
| 231 |
# else: Keep current offsets (within 2s delay period)
|
| 232 |
|
| 233 |
# Small sleep to prevent excessive CPU usage (same as main_works.py)
|
| 234 |
+
time.sleep(0.04)
|
| 235 |
|
| 236 |
except Exception as e:
|
| 237 |
logger.error(f"Camera worker error: {e}")
|
src/reachy_mini_conversation_app/config.py
CHANGED
|
@@ -23,11 +23,13 @@ class Config:
|
|
| 23 |
# Required
|
| 24 |
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
|
| 25 |
if not OPENAI_API_KEY or not OPENAI_API_KEY.strip():
|
| 26 |
-
raise RuntimeError
|
| 27 |
-
"
|
| 28 |
"Either:\n"
|
| 29 |
" 1. Create a .env file with: OPENAI_API_KEY=your_api_key_here (recomended)\n"
|
| 30 |
-
" 2. Set environment variable: export OPENAI_API_KEY=your_api_key_here"
|
|
|
|
|
|
|
| 31 |
)
|
| 32 |
|
| 33 |
# Optional
|
|
@@ -41,4 +43,27 @@ class Config:
|
|
| 41 |
REACHY_MINI_CUSTOM_PROFILE = os.getenv("REACHY_MINI_CUSTOM_PROFILE")
|
| 42 |
logger.debug(f"Custom Profile: {REACHY_MINI_CUSTOM_PROFILE}")
|
| 43 |
|
|
|
|
| 44 |
config = Config()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
# Required
|
| 24 |
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
|
| 25 |
if not OPENAI_API_KEY or not OPENAI_API_KEY.strip():
|
| 26 |
+
logger.warning( # was raise RuntimeError
|
| 27 |
+
"\nOPENAI_API_KEY is missing or empty.\n"
|
| 28 |
"Either:\n"
|
| 29 |
" 1. Create a .env file with: OPENAI_API_KEY=your_api_key_here (recomended)\n"
|
| 30 |
+
" 2. Set environment variable: export OPENAI_API_KEY=your_api_key_here\n"
|
| 31 |
+
" 3. If using Gradio, you can enter it in the API Key textbox.\n\n"
|
| 32 |
+
""
|
| 33 |
)
|
| 34 |
|
| 35 |
# Optional
|
|
|
|
| 43 |
REACHY_MINI_CUSTOM_PROFILE = os.getenv("REACHY_MINI_CUSTOM_PROFILE")
|
| 44 |
logger.debug(f"Custom Profile: {REACHY_MINI_CUSTOM_PROFILE}")
|
| 45 |
|
| 46 |
+
|
| 47 |
config = Config()
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def set_custom_profile(profile: str | None) -> None:
|
| 51 |
+
"""Update the selected custom profile at runtime and expose it via env.
|
| 52 |
+
|
| 53 |
+
This ensures modules that read `config` and code that inspects the
|
| 54 |
+
environment see a consistent value.
|
| 55 |
+
"""
|
| 56 |
+
try:
|
| 57 |
+
config.REACHY_MINI_CUSTOM_PROFILE = profile
|
| 58 |
+
except Exception:
|
| 59 |
+
pass
|
| 60 |
+
try:
|
| 61 |
+
import os as _os
|
| 62 |
+
|
| 63 |
+
if profile:
|
| 64 |
+
_os.environ["REACHY_MINI_CUSTOM_PROFILE"] = profile
|
| 65 |
+
else:
|
| 66 |
+
# Remove to reflect default
|
| 67 |
+
_os.environ.pop("REACHY_MINI_CUSTOM_PROFILE", None)
|
| 68 |
+
except Exception:
|
| 69 |
+
pass
|
src/reachy_mini_conversation_app/console.py
CHANGED
|
@@ -1,19 +1,44 @@
|
|
| 1 |
-
"""Bidirectional local audio stream.
|
| 2 |
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
"""
|
| 5 |
|
|
|
|
|
|
|
| 6 |
import time
|
| 7 |
import asyncio
|
| 8 |
import logging
|
| 9 |
-
from typing import List
|
|
|
|
| 10 |
|
| 11 |
from fastrtc import AdditionalOutputs, audio_to_float32
|
| 12 |
from scipy.signal import resample
|
| 13 |
|
| 14 |
from reachy_mini import ReachyMini
|
| 15 |
from reachy_mini.media.media_manager import MediaBackend
|
|
|
|
| 16 |
from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
|
| 19 |
logger = logging.getLogger(__name__)
|
|
@@ -22,23 +47,331 @@ logger = logging.getLogger(__name__)
|
|
| 22 |
class LocalStream:
|
| 23 |
"""LocalStream using Reachy Mini's recorder/player."""
|
| 24 |
|
| 25 |
-
def __init__(
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
self.handler = handler
|
| 28 |
self._robot = robot
|
| 29 |
self._stop_event = asyncio.Event()
|
| 30 |
self._tasks: List[asyncio.Task[None]] = []
|
| 31 |
# Allow the handler to flush the player queue when appropriate.
|
| 32 |
self.handler._clear_queue = self.clear_audio_queue
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
def launch(self) -> None:
|
| 35 |
-
"""Start the recorder/player and run the async processing loops.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
self._stop_event.clear()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
self._robot.media.start_recording()
|
| 38 |
self._robot.media.start_playing()
|
| 39 |
time.sleep(1) # give some time to the pipelines to start
|
| 40 |
|
| 41 |
async def runner() -> None:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
self._tasks = [
|
| 43 |
asyncio.create_task(self.handler.start_up(), name="openai-handler"),
|
| 44 |
asyncio.create_task(self.record_loop(), name="stream-record-loop"),
|
|
|
|
| 1 |
+
"""Bidirectional local audio stream with optional settings UI.
|
| 2 |
|
| 3 |
+
In headless mode, there is no Gradio UI. If the OpenAI API key is not
|
| 4 |
+
available via environment/.env, we expose a minimal settings page via the
|
| 5 |
+
Reachy Mini Apps settings server to let non-technical users enter it.
|
| 6 |
+
|
| 7 |
+
The settings UI is served from this package's ``static/`` folder and offers a
|
| 8 |
+
single password field to set ``OPENAI_API_KEY``. Once set, we persist it to the
|
| 9 |
+
app instance's ``.env`` file (if available) and proceed to start streaming.
|
| 10 |
"""
|
| 11 |
|
| 12 |
+
import os
|
| 13 |
+
import sys
|
| 14 |
import time
|
| 15 |
import asyncio
|
| 16 |
import logging
|
| 17 |
+
from typing import List, Optional
|
| 18 |
+
from pathlib import Path
|
| 19 |
|
| 20 |
from fastrtc import AdditionalOutputs, audio_to_float32
|
| 21 |
from scipy.signal import resample
|
| 22 |
|
| 23 |
from reachy_mini import ReachyMini
|
| 24 |
from reachy_mini.media.media_manager import MediaBackend
|
| 25 |
+
from reachy_mini_conversation_app.config import config
|
| 26 |
from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
|
| 27 |
+
from reachy_mini_conversation_app.headless_personality_ui import mount_personality_routes
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
try:
|
| 31 |
+
# FastAPI is provided by the Reachy Mini Apps runtime
|
| 32 |
+
from fastapi import FastAPI, Response
|
| 33 |
+
from pydantic import BaseModel
|
| 34 |
+
from fastapi.responses import FileResponse, JSONResponse
|
| 35 |
+
from starlette.staticfiles import StaticFiles
|
| 36 |
+
except Exception: # pragma: no cover - only loaded when settings_app is used
|
| 37 |
+
FastAPI = object # type: ignore
|
| 38 |
+
FileResponse = object # type: ignore
|
| 39 |
+
JSONResponse = object # type: ignore
|
| 40 |
+
StaticFiles = object # type: ignore
|
| 41 |
+
BaseModel = object # type: ignore
|
| 42 |
|
| 43 |
|
| 44 |
logger = logging.getLogger(__name__)
|
|
|
|
| 47 |
class LocalStream:
|
| 48 |
"""LocalStream using Reachy Mini's recorder/player."""
|
| 49 |
|
| 50 |
+
def __init__(
|
| 51 |
+
self,
|
| 52 |
+
handler: OpenaiRealtimeHandler,
|
| 53 |
+
robot: ReachyMini,
|
| 54 |
+
*,
|
| 55 |
+
settings_app: Optional[FastAPI] = None,
|
| 56 |
+
instance_path: Optional[str] = None,
|
| 57 |
+
):
|
| 58 |
+
"""Initialize the stream with an OpenAI realtime handler and pipelines.
|
| 59 |
+
|
| 60 |
+
- ``settings_app``: the Reachy Mini Apps FastAPI to attach settings endpoints.
|
| 61 |
+
- ``instance_path``: directory where per-instance ``.env`` should be stored.
|
| 62 |
+
"""
|
| 63 |
self.handler = handler
|
| 64 |
self._robot = robot
|
| 65 |
self._stop_event = asyncio.Event()
|
| 66 |
self._tasks: List[asyncio.Task[None]] = []
|
| 67 |
# Allow the handler to flush the player queue when appropriate.
|
| 68 |
self.handler._clear_queue = self.clear_audio_queue
|
| 69 |
+
self._settings_app: Optional[FastAPI] = settings_app
|
| 70 |
+
self._instance_path: Optional[str] = instance_path
|
| 71 |
+
self._settings_initialized = False
|
| 72 |
+
self._asyncio_loop = None
|
| 73 |
+
|
| 74 |
+
# ---- Settings UI (only when API key is missing) ----
|
| 75 |
+
def _read_env_lines(self, env_path: Path) -> list[str]:
|
| 76 |
+
"""Load env file contents or a template as a list of lines."""
|
| 77 |
+
inst = env_path.parent
|
| 78 |
+
try:
|
| 79 |
+
if env_path.exists():
|
| 80 |
+
try:
|
| 81 |
+
return env_path.read_text(encoding="utf-8").splitlines()
|
| 82 |
+
except Exception:
|
| 83 |
+
return []
|
| 84 |
+
template_text = None
|
| 85 |
+
ex = inst / ".env.example"
|
| 86 |
+
if ex.exists():
|
| 87 |
+
try:
|
| 88 |
+
template_text = ex.read_text(encoding="utf-8")
|
| 89 |
+
except Exception:
|
| 90 |
+
template_text = None
|
| 91 |
+
if template_text is None:
|
| 92 |
+
try:
|
| 93 |
+
cwd_example = Path.cwd() / ".env.example"
|
| 94 |
+
if cwd_example.exists():
|
| 95 |
+
template_text = cwd_example.read_text(encoding="utf-8")
|
| 96 |
+
except Exception:
|
| 97 |
+
template_text = None
|
| 98 |
+
if template_text is None:
|
| 99 |
+
packaged = Path(__file__).parent / ".env.example"
|
| 100 |
+
if packaged.exists():
|
| 101 |
+
try:
|
| 102 |
+
template_text = packaged.read_text(encoding="utf-8")
|
| 103 |
+
except Exception:
|
| 104 |
+
template_text = None
|
| 105 |
+
return template_text.splitlines() if template_text else []
|
| 106 |
+
except Exception:
|
| 107 |
+
return []
|
| 108 |
+
|
| 109 |
+
def _persist_api_key(self, key: str) -> None:
|
| 110 |
+
"""Persist API key to environment and instance ``.env`` if possible.
|
| 111 |
+
|
| 112 |
+
Behavior:
|
| 113 |
+
- Always sets ``OPENAI_API_KEY`` in process env and in-memory config.
|
| 114 |
+
- Writes/updates ``<instance_path>/.env``:
|
| 115 |
+
* If ``.env`` exists, replaces/append OPENAI_API_KEY line.
|
| 116 |
+
* Else, copies template from ``<instance_path>/.env.example`` when present,
|
| 117 |
+
otherwise falls back to the packaged template
|
| 118 |
+
``reachy_mini_conversation_app/.env.example``.
|
| 119 |
+
* Ensures the resulting file contains the full template plus the key.
|
| 120 |
+
- Loads the written ``.env`` into the current process environment.
|
| 121 |
+
"""
|
| 122 |
+
k = (key or "").strip()
|
| 123 |
+
if not k:
|
| 124 |
+
return
|
| 125 |
+
# Update live process env and config so consumers see it immediately
|
| 126 |
+
try:
|
| 127 |
+
os.environ["OPENAI_API_KEY"] = k
|
| 128 |
+
except Exception: # best-effort
|
| 129 |
+
pass
|
| 130 |
+
try:
|
| 131 |
+
config.OPENAI_API_KEY = k
|
| 132 |
+
except Exception:
|
| 133 |
+
pass
|
| 134 |
+
|
| 135 |
+
if not self._instance_path:
|
| 136 |
+
return
|
| 137 |
+
try:
|
| 138 |
+
inst = Path(self._instance_path)
|
| 139 |
+
env_path = inst / ".env"
|
| 140 |
+
lines = self._read_env_lines(env_path)
|
| 141 |
+
replaced = False
|
| 142 |
+
for i, ln in enumerate(lines):
|
| 143 |
+
if ln.strip().startswith("OPENAI_API_KEY="):
|
| 144 |
+
lines[i] = f"OPENAI_API_KEY={k}"
|
| 145 |
+
replaced = True
|
| 146 |
+
break
|
| 147 |
+
if not replaced:
|
| 148 |
+
lines.append(f"OPENAI_API_KEY={k}")
|
| 149 |
+
final_text = "\n".join(lines) + "\n"
|
| 150 |
+
env_path.write_text(final_text, encoding="utf-8")
|
| 151 |
+
logger.info("Persisted OPENAI_API_KEY to %s", env_path)
|
| 152 |
+
|
| 153 |
+
# Load the newly written .env into this process to ensure downstream imports see it
|
| 154 |
+
try:
|
| 155 |
+
from dotenv import load_dotenv
|
| 156 |
+
|
| 157 |
+
load_dotenv(dotenv_path=str(env_path), override=True)
|
| 158 |
+
except Exception:
|
| 159 |
+
pass
|
| 160 |
+
except Exception as e:
|
| 161 |
+
logger.warning("Failed to persist OPENAI_API_KEY: %s", e)
|
| 162 |
+
|
| 163 |
+
def _persist_personality(self, profile: Optional[str]) -> None:
|
| 164 |
+
"""Persist the startup personality to the instance .env and config."""
|
| 165 |
+
selection = (profile or "").strip() or None
|
| 166 |
+
try:
|
| 167 |
+
from reachy_mini_conversation_app.config import set_custom_profile
|
| 168 |
+
|
| 169 |
+
set_custom_profile(selection)
|
| 170 |
+
except Exception:
|
| 171 |
+
pass
|
| 172 |
+
|
| 173 |
+
if not self._instance_path:
|
| 174 |
+
return
|
| 175 |
+
try:
|
| 176 |
+
env_path = Path(self._instance_path) / ".env"
|
| 177 |
+
lines = self._read_env_lines(env_path)
|
| 178 |
+
replaced = False
|
| 179 |
+
for i, ln in enumerate(list(lines)):
|
| 180 |
+
if ln.strip().startswith("REACHY_MINI_CUSTOM_PROFILE="):
|
| 181 |
+
if selection:
|
| 182 |
+
lines[i] = f"REACHY_MINI_CUSTOM_PROFILE={selection}"
|
| 183 |
+
else:
|
| 184 |
+
lines.pop(i)
|
| 185 |
+
replaced = True
|
| 186 |
+
break
|
| 187 |
+
if selection and not replaced:
|
| 188 |
+
lines.append(f"REACHY_MINI_CUSTOM_PROFILE={selection}")
|
| 189 |
+
if selection is None and not env_path.exists():
|
| 190 |
+
return
|
| 191 |
+
final_text = "\n".join(lines) + "\n"
|
| 192 |
+
env_path.write_text(final_text, encoding="utf-8")
|
| 193 |
+
logger.info("Persisted startup personality to %s", env_path)
|
| 194 |
+
try:
|
| 195 |
+
from dotenv import load_dotenv
|
| 196 |
+
|
| 197 |
+
load_dotenv(dotenv_path=str(env_path), override=True)
|
| 198 |
+
except Exception:
|
| 199 |
+
pass
|
| 200 |
+
except Exception as e:
|
| 201 |
+
logger.warning("Failed to persist REACHY_MINI_CUSTOM_PROFILE: %s", e)
|
| 202 |
+
|
| 203 |
+
def _read_persisted_personality(self) -> Optional[str]:
|
| 204 |
+
"""Read persisted startup personality from instance .env (if any)."""
|
| 205 |
+
if not self._instance_path:
|
| 206 |
+
return None
|
| 207 |
+
env_path = Path(self._instance_path) / ".env"
|
| 208 |
+
try:
|
| 209 |
+
if env_path.exists():
|
| 210 |
+
for ln in env_path.read_text(encoding="utf-8").splitlines():
|
| 211 |
+
if ln.strip().startswith("REACHY_MINI_CUSTOM_PROFILE="):
|
| 212 |
+
_, _, val = ln.partition("=")
|
| 213 |
+
v = val.strip()
|
| 214 |
+
return v or None
|
| 215 |
+
except Exception:
|
| 216 |
+
pass
|
| 217 |
+
return None
|
| 218 |
+
|
| 219 |
+
def _init_settings_ui_if_needed(self) -> None:
|
| 220 |
+
"""Attach minimal settings UI to the settings app.
|
| 221 |
+
|
| 222 |
+
Always mounts the UI when a settings_app is provided so that users
|
| 223 |
+
see a confirmation message even if the API key is already configured.
|
| 224 |
+
"""
|
| 225 |
+
if self._settings_initialized:
|
| 226 |
+
return
|
| 227 |
+
if self._settings_app is None:
|
| 228 |
+
return
|
| 229 |
+
|
| 230 |
+
static_dir = Path(__file__).parent / "static"
|
| 231 |
+
index_file = static_dir / "index.html"
|
| 232 |
+
|
| 233 |
+
if hasattr(self._settings_app, "mount"):
|
| 234 |
+
try:
|
| 235 |
+
# Serve /static/* assets
|
| 236 |
+
self._settings_app.mount("/static", StaticFiles(directory=str(static_dir)), name="static")
|
| 237 |
+
except Exception:
|
| 238 |
+
pass
|
| 239 |
+
|
| 240 |
+
class ApiKeyPayload(BaseModel):
|
| 241 |
+
openai_api_key: str
|
| 242 |
+
|
| 243 |
+
# GET / -> index.html
|
| 244 |
+
@self._settings_app.get("/")
|
| 245 |
+
def _root() -> FileResponse:
|
| 246 |
+
return FileResponse(str(index_file))
|
| 247 |
+
|
| 248 |
+
# GET /favicon.ico -> optional, avoid noisy 404s on some browsers
|
| 249 |
+
@self._settings_app.get("/favicon.ico")
|
| 250 |
+
def _favicon() -> Response:
|
| 251 |
+
return Response(status_code=204)
|
| 252 |
+
|
| 253 |
+
# GET /status -> whether key is set
|
| 254 |
+
@self._settings_app.get("/status")
|
| 255 |
+
def _status() -> JSONResponse:
|
| 256 |
+
has_key = bool(config.OPENAI_API_KEY and str(config.OPENAI_API_KEY).strip())
|
| 257 |
+
return JSONResponse({"has_key": has_key})
|
| 258 |
+
|
| 259 |
+
# GET /ready -> whether backend finished loading tools
|
| 260 |
+
@self._settings_app.get("/ready")
|
| 261 |
+
def _ready() -> JSONResponse:
|
| 262 |
+
try:
|
| 263 |
+
mod = sys.modules.get("reachy_mini_conversation_app.tools.core_tools")
|
| 264 |
+
ready = bool(getattr(mod, "_TOOLS_INITIALIZED", False)) if mod else False
|
| 265 |
+
except Exception:
|
| 266 |
+
ready = False
|
| 267 |
+
return JSONResponse({"ready": ready})
|
| 268 |
+
|
| 269 |
+
# POST /openai_api_key -> set/persist key
|
| 270 |
+
@self._settings_app.post("/openai_api_key")
|
| 271 |
+
def _set_key(payload: ApiKeyPayload) -> JSONResponse:
|
| 272 |
+
key = (payload.openai_api_key or "").strip()
|
| 273 |
+
if not key:
|
| 274 |
+
return JSONResponse({"ok": False, "error": "empty_key"}, status_code=400)
|
| 275 |
+
self._persist_api_key(key)
|
| 276 |
+
return JSONResponse({"ok": True})
|
| 277 |
+
|
| 278 |
+
# POST /validate_api_key -> validate key without persisting it
|
| 279 |
+
@self._settings_app.post("/validate_api_key")
|
| 280 |
+
async def _validate_key(payload: ApiKeyPayload) -> JSONResponse:
|
| 281 |
+
key = (payload.openai_api_key or "").strip()
|
| 282 |
+
if not key:
|
| 283 |
+
return JSONResponse({"valid": False, "error": "empty_key"}, status_code=400)
|
| 284 |
+
|
| 285 |
+
# Try to validate by checking if we can fetch the models
|
| 286 |
+
try:
|
| 287 |
+
import httpx
|
| 288 |
+
|
| 289 |
+
headers = {"Authorization": f"Bearer {key}", "Content-Type": "application/json"}
|
| 290 |
+
async with httpx.AsyncClient(timeout=10.0) as client:
|
| 291 |
+
response = await client.get("https://api.openai.com/v1/models", headers=headers)
|
| 292 |
+
if response.status_code == 200:
|
| 293 |
+
return JSONResponse({"valid": True})
|
| 294 |
+
elif response.status_code == 401:
|
| 295 |
+
return JSONResponse({"valid": False, "error": "invalid_api_key"}, status_code=401)
|
| 296 |
+
else:
|
| 297 |
+
return JSONResponse(
|
| 298 |
+
{"valid": False, "error": "validation_failed"}, status_code=response.status_code
|
| 299 |
+
)
|
| 300 |
+
except Exception as e:
|
| 301 |
+
logger.warning(f"API key validation failed: {e}")
|
| 302 |
+
return JSONResponse({"valid": False, "error": "validation_error"}, status_code=500)
|
| 303 |
+
|
| 304 |
+
self._settings_initialized = True
|
| 305 |
|
| 306 |
def launch(self) -> None:
|
| 307 |
+
"""Start the recorder/player and run the async processing loops.
|
| 308 |
+
|
| 309 |
+
If the OpenAI key is missing, expose a tiny settings UI via the
|
| 310 |
+
Reachy Mini settings server to collect it before starting streams.
|
| 311 |
+
"""
|
| 312 |
self._stop_event.clear()
|
| 313 |
+
|
| 314 |
+
# Always expose settings UI if a settings app is available
|
| 315 |
+
self._init_settings_ui_if_needed()
|
| 316 |
+
|
| 317 |
+
# Try to load an existing instance .env first (covers subsequent runs)
|
| 318 |
+
if self._instance_path:
|
| 319 |
+
try:
|
| 320 |
+
from dotenv import load_dotenv
|
| 321 |
+
|
| 322 |
+
from reachy_mini_conversation_app.config import set_custom_profile
|
| 323 |
+
|
| 324 |
+
env_path = Path(self._instance_path) / ".env"
|
| 325 |
+
if env_path.exists():
|
| 326 |
+
load_dotenv(dotenv_path=str(env_path), override=True)
|
| 327 |
+
# Update config with newly loaded values
|
| 328 |
+
new_key = os.getenv("OPENAI_API_KEY", "").strip()
|
| 329 |
+
if new_key:
|
| 330 |
+
try:
|
| 331 |
+
config.OPENAI_API_KEY = new_key
|
| 332 |
+
except Exception:
|
| 333 |
+
pass
|
| 334 |
+
new_profile = os.getenv("REACHY_MINI_CUSTOM_PROFILE")
|
| 335 |
+
if new_profile is not None:
|
| 336 |
+
try:
|
| 337 |
+
set_custom_profile(new_profile.strip() or None)
|
| 338 |
+
except Exception:
|
| 339 |
+
pass
|
| 340 |
+
except Exception:
|
| 341 |
+
pass
|
| 342 |
+
|
| 343 |
+
# If key is still missing -> wait until provided via the settings UI
|
| 344 |
+
if not (config.OPENAI_API_KEY and str(config.OPENAI_API_KEY).strip()):
|
| 345 |
+
logger.warning("OPENAI_API_KEY not found. Open the app settings page to enter it.")
|
| 346 |
+
# Poll until the key becomes available (set via the settings UI)
|
| 347 |
+
try:
|
| 348 |
+
while not (config.OPENAI_API_KEY and str(config.OPENAI_API_KEY).strip()):
|
| 349 |
+
time.sleep(0.2)
|
| 350 |
+
except KeyboardInterrupt:
|
| 351 |
+
logger.info("Interrupted while waiting for API key.")
|
| 352 |
+
return
|
| 353 |
+
|
| 354 |
+
# Start media after key is set/available
|
| 355 |
self._robot.media.start_recording()
|
| 356 |
self._robot.media.start_playing()
|
| 357 |
time.sleep(1) # give some time to the pipelines to start
|
| 358 |
|
| 359 |
async def runner() -> None:
|
| 360 |
+
# Capture loop for cross-thread personality actions
|
| 361 |
+
loop = asyncio.get_running_loop()
|
| 362 |
+
self._asyncio_loop = loop # type: ignore[assignment]
|
| 363 |
+
# Mount personality routes now that loop and handler are available
|
| 364 |
+
try:
|
| 365 |
+
if self._settings_app is not None:
|
| 366 |
+
mount_personality_routes(
|
| 367 |
+
self._settings_app,
|
| 368 |
+
self.handler,
|
| 369 |
+
lambda: self._asyncio_loop,
|
| 370 |
+
persist_personality=self._persist_personality,
|
| 371 |
+
get_persisted_personality=self._read_persisted_personality,
|
| 372 |
+
)
|
| 373 |
+
except Exception:
|
| 374 |
+
pass
|
| 375 |
self._tasks = [
|
| 376 |
asyncio.create_task(self.handler.start_up(), name="openai-handler"),
|
| 377 |
asyncio.create_task(self.record_loop(), name="stream-record-loop"),
|
src/reachy_mini_conversation_app/gradio_personality.py
ADDED
|
@@ -0,0 +1,301 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Gradio personality UI components and wiring.
|
| 2 |
+
|
| 3 |
+
This module encapsulates the UI elements and logic related to managing
|
| 4 |
+
conversation "personalities" (profiles) so that `main.py` stays lean.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
from typing import Any
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
|
| 11 |
+
import gradio as gr
|
| 12 |
+
|
| 13 |
+
from .config import config
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
class PersonalityUI:
|
| 17 |
+
"""Container for personality-related Gradio components."""
|
| 18 |
+
|
| 19 |
+
def __init__(self) -> None:
|
| 20 |
+
"""Initialize the PersonalityUI instance."""
|
| 21 |
+
# Constants and paths
|
| 22 |
+
self.DEFAULT_OPTION = "(built-in default)"
|
| 23 |
+
self._profiles_root = Path(__file__).parent / "profiles"
|
| 24 |
+
self._tools_dir = Path(__file__).parent / "tools"
|
| 25 |
+
self._prompts_dir = Path(__file__).parent / "prompts"
|
| 26 |
+
|
| 27 |
+
# Components (initialized in create_components)
|
| 28 |
+
self.personalities_dropdown: gr.Dropdown
|
| 29 |
+
self.apply_btn: gr.Button
|
| 30 |
+
self.status_md: gr.Markdown
|
| 31 |
+
self.preview_md: gr.Markdown
|
| 32 |
+
self.person_name_tb: gr.Textbox
|
| 33 |
+
self.person_instr_ta: gr.TextArea
|
| 34 |
+
self.tools_txt_ta: gr.TextArea
|
| 35 |
+
self.voice_dropdown: gr.Dropdown
|
| 36 |
+
self.new_personality_btn: gr.Button
|
| 37 |
+
self.available_tools_cg: gr.CheckboxGroup
|
| 38 |
+
self.save_btn: gr.Button
|
| 39 |
+
|
| 40 |
+
# ---------- Filesystem helpers ----------
|
| 41 |
+
def _list_personalities(self) -> list[str]:
|
| 42 |
+
names: list[str] = []
|
| 43 |
+
try:
|
| 44 |
+
if self._profiles_root.exists():
|
| 45 |
+
for p in sorted(self._profiles_root.iterdir()):
|
| 46 |
+
if p.name == "user_personalities":
|
| 47 |
+
continue
|
| 48 |
+
if p.is_dir() and (p / "instructions.txt").exists():
|
| 49 |
+
names.append(p.name)
|
| 50 |
+
user_dir = self._profiles_root / "user_personalities"
|
| 51 |
+
if user_dir.exists():
|
| 52 |
+
for p in sorted(user_dir.iterdir()):
|
| 53 |
+
if p.is_dir() and (p / "instructions.txt").exists():
|
| 54 |
+
names.append(f"user_personalities/{p.name}")
|
| 55 |
+
except Exception:
|
| 56 |
+
pass
|
| 57 |
+
return names
|
| 58 |
+
|
| 59 |
+
def _resolve_profile_dir(self, selection: str) -> Path:
|
| 60 |
+
return self._profiles_root / selection
|
| 61 |
+
|
| 62 |
+
def _read_instructions_for(self, name: str) -> str:
|
| 63 |
+
try:
|
| 64 |
+
if name == self.DEFAULT_OPTION:
|
| 65 |
+
default_file = self._prompts_dir / "default_prompt.txt"
|
| 66 |
+
if default_file.exists():
|
| 67 |
+
return default_file.read_text(encoding="utf-8").strip()
|
| 68 |
+
return ""
|
| 69 |
+
target = self._resolve_profile_dir(name) / "instructions.txt"
|
| 70 |
+
if target.exists():
|
| 71 |
+
return target.read_text(encoding="utf-8").strip()
|
| 72 |
+
return ""
|
| 73 |
+
except Exception as e:
|
| 74 |
+
return f"Could not load instructions: {e}"
|
| 75 |
+
|
| 76 |
+
@staticmethod
|
| 77 |
+
def _sanitize_name(name: str) -> str:
|
| 78 |
+
import re
|
| 79 |
+
|
| 80 |
+
s = name.strip()
|
| 81 |
+
s = re.sub(r"\s+", "_", s)
|
| 82 |
+
s = re.sub(r"[^a-zA-Z0-9_-]", "", s)
|
| 83 |
+
return s
|
| 84 |
+
|
| 85 |
+
# ---------- Public API ----------
|
| 86 |
+
def create_components(self) -> None:
|
| 87 |
+
"""Instantiate Gradio components for the personality UI."""
|
| 88 |
+
current_value = config.REACHY_MINI_CUSTOM_PROFILE or self.DEFAULT_OPTION
|
| 89 |
+
|
| 90 |
+
self.personalities_dropdown = gr.Dropdown(
|
| 91 |
+
label="Select personality",
|
| 92 |
+
choices=[self.DEFAULT_OPTION, *(self._list_personalities())],
|
| 93 |
+
value=current_value,
|
| 94 |
+
)
|
| 95 |
+
self.apply_btn = gr.Button("Apply personality")
|
| 96 |
+
self.status_md = gr.Markdown(visible=True)
|
| 97 |
+
self.preview_md = gr.Markdown(value=self._read_instructions_for(current_value))
|
| 98 |
+
self.person_name_tb = gr.Textbox(label="Personality name")
|
| 99 |
+
self.person_instr_ta = gr.TextArea(label="Personality instructions", lines=10)
|
| 100 |
+
self.tools_txt_ta = gr.TextArea(label="tools.txt", lines=10)
|
| 101 |
+
self.voice_dropdown = gr.Dropdown(label="Voice", choices=["cedar"], value="cedar")
|
| 102 |
+
self.new_personality_btn = gr.Button("New personality")
|
| 103 |
+
self.available_tools_cg = gr.CheckboxGroup(label="Available tools (helper)", choices=[], value=[])
|
| 104 |
+
self.save_btn = gr.Button("Save personality (instructions + tools)")
|
| 105 |
+
|
| 106 |
+
def additional_inputs_ordered(self) -> list[Any]:
|
| 107 |
+
"""Return the additional inputs in the expected order for Stream."""
|
| 108 |
+
return [
|
| 109 |
+
self.personalities_dropdown,
|
| 110 |
+
self.apply_btn,
|
| 111 |
+
self.new_personality_btn,
|
| 112 |
+
self.status_md,
|
| 113 |
+
self.preview_md,
|
| 114 |
+
self.person_name_tb,
|
| 115 |
+
self.person_instr_ta,
|
| 116 |
+
self.tools_txt_ta,
|
| 117 |
+
self.voice_dropdown,
|
| 118 |
+
self.available_tools_cg,
|
| 119 |
+
self.save_btn,
|
| 120 |
+
]
|
| 121 |
+
|
| 122 |
+
# ---------- Event wiring ----------
|
| 123 |
+
def wire_events(self, handler: Any, blocks: gr.Blocks) -> None:
|
| 124 |
+
"""Attach event handlers to components within a Blocks context."""
|
| 125 |
+
|
| 126 |
+
async def _apply_personality(selected: str) -> tuple[str, str]:
|
| 127 |
+
profile = None if selected == self.DEFAULT_OPTION else selected
|
| 128 |
+
status = await handler.apply_personality(profile)
|
| 129 |
+
preview = self._read_instructions_for(selected)
|
| 130 |
+
return status, preview
|
| 131 |
+
|
| 132 |
+
def _read_voice_for(name: str) -> str:
|
| 133 |
+
try:
|
| 134 |
+
if name == self.DEFAULT_OPTION:
|
| 135 |
+
return "cedar"
|
| 136 |
+
vf = self._resolve_profile_dir(name) / "voice.txt"
|
| 137 |
+
if vf.exists():
|
| 138 |
+
v = vf.read_text(encoding="utf-8").strip()
|
| 139 |
+
return v or "cedar"
|
| 140 |
+
except Exception:
|
| 141 |
+
pass
|
| 142 |
+
return "cedar"
|
| 143 |
+
|
| 144 |
+
async def _fetch_voices(selected: str) -> dict[str, Any]:
|
| 145 |
+
try:
|
| 146 |
+
voices = await handler.get_available_voices()
|
| 147 |
+
current = _read_voice_for(selected)
|
| 148 |
+
if current not in voices:
|
| 149 |
+
current = "cedar"
|
| 150 |
+
return gr.update(choices=voices, value=current)
|
| 151 |
+
except Exception:
|
| 152 |
+
return gr.update(choices=["cedar"], value="cedar")
|
| 153 |
+
|
| 154 |
+
def _available_tools_for(selected: str) -> tuple[list[str], list[str]]:
|
| 155 |
+
shared: list[str] = []
|
| 156 |
+
try:
|
| 157 |
+
for py in self._tools_dir.glob("*.py"):
|
| 158 |
+
if py.stem in {"__init__", "core_tools"}:
|
| 159 |
+
continue
|
| 160 |
+
shared.append(py.stem)
|
| 161 |
+
except Exception:
|
| 162 |
+
pass
|
| 163 |
+
local: list[str] = []
|
| 164 |
+
try:
|
| 165 |
+
if selected != self.DEFAULT_OPTION:
|
| 166 |
+
for py in (self._profiles_root / selected).glob("*.py"):
|
| 167 |
+
local.append(py.stem)
|
| 168 |
+
except Exception:
|
| 169 |
+
pass
|
| 170 |
+
return sorted(shared), sorted(local)
|
| 171 |
+
|
| 172 |
+
def _parse_enabled_tools(text: str) -> list[str]:
|
| 173 |
+
enabled: list[str] = []
|
| 174 |
+
for line in text.splitlines():
|
| 175 |
+
s = line.strip()
|
| 176 |
+
if not s or s.startswith("#"):
|
| 177 |
+
continue
|
| 178 |
+
enabled.append(s)
|
| 179 |
+
return enabled
|
| 180 |
+
|
| 181 |
+
def _load_profile_for_edit(selected: str) -> tuple[dict[str, Any], dict[str, Any], dict[str, Any], str]:
|
| 182 |
+
instr = self._read_instructions_for(selected)
|
| 183 |
+
tools_txt = ""
|
| 184 |
+
if selected != self.DEFAULT_OPTION:
|
| 185 |
+
tp = self._resolve_profile_dir(selected) / "tools.txt"
|
| 186 |
+
if tp.exists():
|
| 187 |
+
tools_txt = tp.read_text(encoding="utf-8")
|
| 188 |
+
shared, local = _available_tools_for(selected)
|
| 189 |
+
all_tools = sorted(set(shared + local))
|
| 190 |
+
enabled = _parse_enabled_tools(tools_txt)
|
| 191 |
+
status_text = f"Loaded profile '{selected}'."
|
| 192 |
+
return (
|
| 193 |
+
gr.update(value=instr),
|
| 194 |
+
gr.update(value=tools_txt),
|
| 195 |
+
gr.update(choices=all_tools, value=enabled),
|
| 196 |
+
status_text,
|
| 197 |
+
)
|
| 198 |
+
|
| 199 |
+
def _new_personality() -> tuple[
|
| 200 |
+
dict[str, Any], dict[str, Any], dict[str, Any], dict[str, Any], str, dict[str, Any]
|
| 201 |
+
]:
|
| 202 |
+
try:
|
| 203 |
+
# Prefill with hints
|
| 204 |
+
instr_val = """# Write your instructions here\n# e.g., Keep responses concise and friendly."""
|
| 205 |
+
tools_txt_val = "# tools enabled for this profile\n"
|
| 206 |
+
return (
|
| 207 |
+
gr.update(value=""),
|
| 208 |
+
gr.update(value=instr_val),
|
| 209 |
+
gr.update(value=tools_txt_val),
|
| 210 |
+
gr.update(choices=sorted(_available_tools_for(self.DEFAULT_OPTION)[0]), value=[]),
|
| 211 |
+
"Fill in a name, instructions and (optional) tools, then Save.",
|
| 212 |
+
gr.update(value="cedar"),
|
| 213 |
+
)
|
| 214 |
+
except Exception:
|
| 215 |
+
return (
|
| 216 |
+
gr.update(),
|
| 217 |
+
gr.update(),
|
| 218 |
+
gr.update(),
|
| 219 |
+
gr.update(),
|
| 220 |
+
"Failed to initialize new personality.",
|
| 221 |
+
gr.update(),
|
| 222 |
+
)
|
| 223 |
+
|
| 224 |
+
def _save_personality(
|
| 225 |
+
name: str, instructions: str, tools_text: str, voice: str
|
| 226 |
+
) -> tuple[dict[str, Any], dict[str, Any], str]:
|
| 227 |
+
name_s = self._sanitize_name(name)
|
| 228 |
+
if not name_s:
|
| 229 |
+
return gr.update(), gr.update(), "Please enter a valid name."
|
| 230 |
+
try:
|
| 231 |
+
target_dir = self._profiles_root / "user_personalities" / name_s
|
| 232 |
+
target_dir.mkdir(parents=True, exist_ok=True)
|
| 233 |
+
(target_dir / "instructions.txt").write_text(instructions.strip() + "\n", encoding="utf-8")
|
| 234 |
+
(target_dir / "tools.txt").write_text(tools_text.strip() + "\n", encoding="utf-8")
|
| 235 |
+
(target_dir / "voice.txt").write_text((voice or "cedar").strip() + "\n", encoding="utf-8")
|
| 236 |
+
|
| 237 |
+
choices = self._list_personalities()
|
| 238 |
+
value = f"user_personalities/{name_s}"
|
| 239 |
+
if value not in choices:
|
| 240 |
+
choices.append(value)
|
| 241 |
+
return (
|
| 242 |
+
gr.update(choices=[self.DEFAULT_OPTION, *sorted(choices)], value=value),
|
| 243 |
+
gr.update(value=instructions),
|
| 244 |
+
f"Saved personality '{name_s}'.",
|
| 245 |
+
)
|
| 246 |
+
except Exception as e:
|
| 247 |
+
return gr.update(), gr.update(), f"Failed to save personality: {e}"
|
| 248 |
+
|
| 249 |
+
def _sync_tools_from_checks(selected: list[str], current_text: str) -> dict[str, Any]:
|
| 250 |
+
comments = [ln for ln in current_text.splitlines() if ln.strip().startswith("#")]
|
| 251 |
+
body = "\n".join(selected)
|
| 252 |
+
out = ("\n".join(comments) + ("\n" if comments else "") + body).strip() + "\n"
|
| 253 |
+
return gr.update(value=out)
|
| 254 |
+
|
| 255 |
+
with blocks:
|
| 256 |
+
self.apply_btn.click(
|
| 257 |
+
fn=_apply_personality,
|
| 258 |
+
inputs=[self.personalities_dropdown],
|
| 259 |
+
outputs=[self.status_md, self.preview_md],
|
| 260 |
+
)
|
| 261 |
+
|
| 262 |
+
self.personalities_dropdown.change(
|
| 263 |
+
fn=_load_profile_for_edit,
|
| 264 |
+
inputs=[self.personalities_dropdown],
|
| 265 |
+
outputs=[self.person_instr_ta, self.tools_txt_ta, self.available_tools_cg, self.status_md],
|
| 266 |
+
)
|
| 267 |
+
|
| 268 |
+
blocks.load(
|
| 269 |
+
fn=_fetch_voices,
|
| 270 |
+
inputs=[self.personalities_dropdown],
|
| 271 |
+
outputs=[self.voice_dropdown],
|
| 272 |
+
)
|
| 273 |
+
|
| 274 |
+
self.available_tools_cg.change(
|
| 275 |
+
fn=_sync_tools_from_checks,
|
| 276 |
+
inputs=[self.available_tools_cg, self.tools_txt_ta],
|
| 277 |
+
outputs=[self.tools_txt_ta],
|
| 278 |
+
)
|
| 279 |
+
|
| 280 |
+
self.new_personality_btn.click(
|
| 281 |
+
fn=_new_personality,
|
| 282 |
+
inputs=[],
|
| 283 |
+
outputs=[
|
| 284 |
+
self.person_name_tb,
|
| 285 |
+
self.person_instr_ta,
|
| 286 |
+
self.tools_txt_ta,
|
| 287 |
+
self.available_tools_cg,
|
| 288 |
+
self.status_md,
|
| 289 |
+
self.voice_dropdown,
|
| 290 |
+
],
|
| 291 |
+
)
|
| 292 |
+
|
| 293 |
+
self.save_btn.click(
|
| 294 |
+
fn=_save_personality,
|
| 295 |
+
inputs=[self.person_name_tb, self.person_instr_ta, self.tools_txt_ta, self.voice_dropdown],
|
| 296 |
+
outputs=[self.personalities_dropdown, self.person_instr_ta, self.status_md],
|
| 297 |
+
).then(
|
| 298 |
+
fn=_apply_personality,
|
| 299 |
+
inputs=[self.personalities_dropdown],
|
| 300 |
+
outputs=[self.status_md, self.preview_md],
|
| 301 |
+
)
|
src/reachy_mini_conversation_app/headless_personality.py
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Headless personality management (console-based).
|
| 2 |
+
|
| 3 |
+
Provides an interactive CLI to browse, preview, apply, create and edit
|
| 4 |
+
"personalities" (profiles) when running without Gradio.
|
| 5 |
+
|
| 6 |
+
This module is intentionally not shared with the Gradio implementation to
|
| 7 |
+
avoid coupling and keep responsibilities clear for headless mode.
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
from typing import List
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
DEFAULT_OPTION = "(built-in default)"
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def _profiles_root() -> Path:
|
| 19 |
+
return Path(__file__).parent / "profiles"
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _prompts_dir() -> Path:
|
| 23 |
+
return Path(__file__).parent / "prompts"
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def _tools_dir() -> Path:
|
| 27 |
+
return Path(__file__).parent / "tools"
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def _sanitize_name(name: str) -> str:
|
| 31 |
+
import re
|
| 32 |
+
|
| 33 |
+
s = name.strip()
|
| 34 |
+
s = re.sub(r"\s+", "_", s)
|
| 35 |
+
s = re.sub(r"[^a-zA-Z0-9_-]", "", s)
|
| 36 |
+
return s
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def list_personalities() -> List[str]:
|
| 40 |
+
"""List available personality profile names."""
|
| 41 |
+
names: List[str] = []
|
| 42 |
+
root = _profiles_root()
|
| 43 |
+
try:
|
| 44 |
+
if root.exists():
|
| 45 |
+
for p in sorted(root.iterdir()):
|
| 46 |
+
if p.name == "user_personalities":
|
| 47 |
+
continue
|
| 48 |
+
if p.is_dir() and (p / "instructions.txt").exists():
|
| 49 |
+
names.append(p.name)
|
| 50 |
+
udir = root / "user_personalities"
|
| 51 |
+
if udir.exists():
|
| 52 |
+
for p in sorted(udir.iterdir()):
|
| 53 |
+
if p.is_dir() and (p / "instructions.txt").exists():
|
| 54 |
+
names.append(f"user_personalities/{p.name}")
|
| 55 |
+
except Exception:
|
| 56 |
+
pass
|
| 57 |
+
return names
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def resolve_profile_dir(selection: str) -> Path:
|
| 61 |
+
"""Resolve the directory path for the given profile selection."""
|
| 62 |
+
return _profiles_root() / selection
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def read_instructions_for(name: str) -> str:
|
| 66 |
+
"""Read the instructions.txt content for the given profile name."""
|
| 67 |
+
try:
|
| 68 |
+
if name == DEFAULT_OPTION:
|
| 69 |
+
df = _prompts_dir() / "default_prompt.txt"
|
| 70 |
+
return df.read_text(encoding="utf-8").strip() if df.exists() else ""
|
| 71 |
+
target = resolve_profile_dir(name) / "instructions.txt"
|
| 72 |
+
return target.read_text(encoding="utf-8").strip() if target.exists() else ""
|
| 73 |
+
except Exception as e:
|
| 74 |
+
return f"Could not load instructions: {e}"
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def available_tools_for(selected: str) -> List[str]:
|
| 78 |
+
"""List available tool modules for the given profile selection."""
|
| 79 |
+
shared: List[str] = []
|
| 80 |
+
try:
|
| 81 |
+
for py in _tools_dir().glob("*.py"):
|
| 82 |
+
if py.stem in {"__init__", "core_tools"}:
|
| 83 |
+
continue
|
| 84 |
+
shared.append(py.stem)
|
| 85 |
+
except Exception:
|
| 86 |
+
pass
|
| 87 |
+
local: List[str] = []
|
| 88 |
+
try:
|
| 89 |
+
if selected != DEFAULT_OPTION:
|
| 90 |
+
for py in resolve_profile_dir(selected).glob("*.py"):
|
| 91 |
+
local.append(py.stem)
|
| 92 |
+
except Exception:
|
| 93 |
+
pass
|
| 94 |
+
return sorted(set(shared + local))
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def _write_profile(name_s: str, instructions: str, tools_text: str, voice: str = "cedar") -> None:
|
| 98 |
+
target_dir = _profiles_root() / "user_personalities" / name_s
|
| 99 |
+
target_dir.mkdir(parents=True, exist_ok=True)
|
| 100 |
+
(target_dir / "instructions.txt").write_text(instructions.strip() + "\n", encoding="utf-8")
|
| 101 |
+
(target_dir / "tools.txt").write_text((tools_text or "").strip() + "\n", encoding="utf-8")
|
| 102 |
+
(target_dir / "voice.txt").write_text((voice or "cedar").strip() + "\n", encoding="utf-8")
|
src/reachy_mini_conversation_app/headless_personality_ui.py
ADDED
|
@@ -0,0 +1,276 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Settings UI routes for headless personality management.
|
| 2 |
+
|
| 3 |
+
Exposes REST endpoints on the provided FastAPI settings app. The
|
| 4 |
+
implementation schedules backend actions (apply personality, fetch voices)
|
| 5 |
+
onto the running LocalStream asyncio loop using the supplied get_loop
|
| 6 |
+
callable to avoid cross-thread issues.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
import asyncio
|
| 11 |
+
import logging
|
| 12 |
+
from typing import Any, Callable, Optional
|
| 13 |
+
|
| 14 |
+
from fastapi import FastAPI
|
| 15 |
+
|
| 16 |
+
from .config import config
|
| 17 |
+
from .openai_realtime import OpenaiRealtimeHandler
|
| 18 |
+
from .headless_personality import (
|
| 19 |
+
DEFAULT_OPTION,
|
| 20 |
+
_sanitize_name,
|
| 21 |
+
_write_profile,
|
| 22 |
+
list_personalities,
|
| 23 |
+
available_tools_for,
|
| 24 |
+
resolve_profile_dir,
|
| 25 |
+
read_instructions_for,
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def mount_personality_routes(
|
| 30 |
+
app: FastAPI,
|
| 31 |
+
handler: OpenaiRealtimeHandler,
|
| 32 |
+
get_loop: Callable[[], asyncio.AbstractEventLoop | None],
|
| 33 |
+
*,
|
| 34 |
+
persist_personality: Callable[[Optional[str]], None] | None = None,
|
| 35 |
+
get_persisted_personality: Callable[[], Optional[str]] | None = None,
|
| 36 |
+
) -> None:
|
| 37 |
+
"""Register personality management endpoints on a FastAPI app."""
|
| 38 |
+
try:
|
| 39 |
+
from fastapi import Request
|
| 40 |
+
from pydantic import BaseModel
|
| 41 |
+
from fastapi.responses import JSONResponse
|
| 42 |
+
except Exception: # pragma: no cover - only when settings app not available
|
| 43 |
+
return
|
| 44 |
+
|
| 45 |
+
class SavePayload(BaseModel):
|
| 46 |
+
name: str
|
| 47 |
+
instructions: str
|
| 48 |
+
tools_text: str
|
| 49 |
+
voice: Optional[str] = "cedar"
|
| 50 |
+
|
| 51 |
+
class ApplyPayload(BaseModel):
|
| 52 |
+
name: str
|
| 53 |
+
persist: Optional[bool] = False
|
| 54 |
+
|
| 55 |
+
def _startup_choice() -> Any:
|
| 56 |
+
"""Return the persisted startup personality or default."""
|
| 57 |
+
try:
|
| 58 |
+
if get_persisted_personality is not None:
|
| 59 |
+
stored = get_persisted_personality()
|
| 60 |
+
if stored:
|
| 61 |
+
return stored
|
| 62 |
+
env_val = getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None)
|
| 63 |
+
if env_val:
|
| 64 |
+
return env_val
|
| 65 |
+
except Exception:
|
| 66 |
+
pass
|
| 67 |
+
return DEFAULT_OPTION
|
| 68 |
+
|
| 69 |
+
def _current_choice() -> str:
|
| 70 |
+
try:
|
| 71 |
+
cur = getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None)
|
| 72 |
+
return cur or DEFAULT_OPTION
|
| 73 |
+
except Exception:
|
| 74 |
+
return DEFAULT_OPTION
|
| 75 |
+
|
| 76 |
+
@app.get("/personalities")
|
| 77 |
+
def _list() -> dict: # type: ignore
|
| 78 |
+
choices = [DEFAULT_OPTION, *list_personalities()]
|
| 79 |
+
return {"choices": choices, "current": _current_choice(), "startup": _startup_choice()}
|
| 80 |
+
|
| 81 |
+
@app.get("/personalities/load")
|
| 82 |
+
def _load(name: str) -> dict: # type: ignore
|
| 83 |
+
instr = read_instructions_for(name)
|
| 84 |
+
tools_txt = ""
|
| 85 |
+
voice = "cedar"
|
| 86 |
+
if name != DEFAULT_OPTION:
|
| 87 |
+
pdir = resolve_profile_dir(name)
|
| 88 |
+
tp = pdir / "tools.txt"
|
| 89 |
+
if tp.exists():
|
| 90 |
+
tools_txt = tp.read_text(encoding="utf-8")
|
| 91 |
+
vf = pdir / "voice.txt"
|
| 92 |
+
if vf.exists():
|
| 93 |
+
v = vf.read_text(encoding="utf-8").strip()
|
| 94 |
+
voice = v or "cedar"
|
| 95 |
+
avail = available_tools_for(name)
|
| 96 |
+
enabled = [ln.strip() for ln in tools_txt.splitlines() if ln.strip() and not ln.strip().startswith("#")]
|
| 97 |
+
return {
|
| 98 |
+
"instructions": instr,
|
| 99 |
+
"tools_text": tools_txt,
|
| 100 |
+
"voice": voice,
|
| 101 |
+
"available_tools": avail,
|
| 102 |
+
"enabled_tools": enabled,
|
| 103 |
+
}
|
| 104 |
+
|
| 105 |
+
@app.post("/personalities/save")
|
| 106 |
+
async def _save(request: Request) -> dict: # type: ignore
|
| 107 |
+
# Accept raw JSON only to avoid validation-related 422s
|
| 108 |
+
try:
|
| 109 |
+
raw = await request.json()
|
| 110 |
+
except Exception:
|
| 111 |
+
raw = {}
|
| 112 |
+
name = str(raw.get("name", ""))
|
| 113 |
+
instructions = str(raw.get("instructions", ""))
|
| 114 |
+
tools_text = str(raw.get("tools_text", ""))
|
| 115 |
+
voice = str(raw.get("voice", "cedar")) if raw.get("voice") is not None else "cedar"
|
| 116 |
+
|
| 117 |
+
name_s = _sanitize_name(name)
|
| 118 |
+
if not name_s:
|
| 119 |
+
return JSONResponse({"ok": False, "error": "invalid_name"}, status_code=400) # type: ignore
|
| 120 |
+
try:
|
| 121 |
+
logger.info(
|
| 122 |
+
"Headless save: name=%r voice=%r instr_len=%d tools_len=%d",
|
| 123 |
+
name_s,
|
| 124 |
+
voice,
|
| 125 |
+
len(instructions),
|
| 126 |
+
len(tools_text),
|
| 127 |
+
)
|
| 128 |
+
_write_profile(name_s, instructions, tools_text, voice or "cedar")
|
| 129 |
+
value = f"user_personalities/{name_s}"
|
| 130 |
+
choices = [DEFAULT_OPTION, *list_personalities()]
|
| 131 |
+
return {"ok": True, "value": value, "choices": choices}
|
| 132 |
+
except Exception as e:
|
| 133 |
+
return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
|
| 134 |
+
|
| 135 |
+
@app.post("/personalities/save_raw")
|
| 136 |
+
async def _save_raw(
|
| 137 |
+
request: Request,
|
| 138 |
+
name: Optional[str] = None,
|
| 139 |
+
instructions: Optional[str] = None,
|
| 140 |
+
tools_text: Optional[str] = None,
|
| 141 |
+
voice: Optional[str] = None,
|
| 142 |
+
) -> dict: # type: ignore
|
| 143 |
+
# Accept query params, form-encoded, or raw JSON
|
| 144 |
+
data = {"name": name, "instructions": instructions, "tools_text": tools_text, "voice": voice}
|
| 145 |
+
# Prefer form if present
|
| 146 |
+
try:
|
| 147 |
+
form = await request.form()
|
| 148 |
+
for k in ("name", "instructions", "tools_text", "voice"):
|
| 149 |
+
if k in form and form[k] is not None:
|
| 150 |
+
data[k] = str(form[k])
|
| 151 |
+
except Exception:
|
| 152 |
+
pass
|
| 153 |
+
# Try JSON
|
| 154 |
+
try:
|
| 155 |
+
raw = await request.json()
|
| 156 |
+
if isinstance(raw, dict):
|
| 157 |
+
for k in ("name", "instructions", "tools_text", "voice"):
|
| 158 |
+
if raw.get(k) is not None:
|
| 159 |
+
data[k] = str(raw.get(k))
|
| 160 |
+
except Exception:
|
| 161 |
+
pass
|
| 162 |
+
|
| 163 |
+
name_s = _sanitize_name(str(data.get("name") or ""))
|
| 164 |
+
if not name_s:
|
| 165 |
+
return JSONResponse({"ok": False, "error": "invalid_name"}, status_code=400) # type: ignore
|
| 166 |
+
instr = str(data.get("instructions") or "")
|
| 167 |
+
tools = str(data.get("tools_text") or "")
|
| 168 |
+
v = str(data.get("voice") or "cedar")
|
| 169 |
+
try:
|
| 170 |
+
logger.info(
|
| 171 |
+
"Headless save_raw: name=%r voice=%r instr_len=%d tools_len=%d", name_s, v, len(instr), len(tools)
|
| 172 |
+
)
|
| 173 |
+
_write_profile(name_s, instr, tools, v)
|
| 174 |
+
value = f"user_personalities/{name_s}"
|
| 175 |
+
choices = [DEFAULT_OPTION, *list_personalities()]
|
| 176 |
+
return {"ok": True, "value": value, "choices": choices}
|
| 177 |
+
except Exception as e:
|
| 178 |
+
return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
|
| 179 |
+
|
| 180 |
+
@app.get("/personalities/save_raw")
|
| 181 |
+
async def _save_raw_get(name: str, instructions: str = "", tools_text: str = "", voice: str = "cedar") -> dict: # type: ignore
|
| 182 |
+
name_s = _sanitize_name(name)
|
| 183 |
+
if not name_s:
|
| 184 |
+
return JSONResponse({"ok": False, "error": "invalid_name"}, status_code=400) # type: ignore
|
| 185 |
+
try:
|
| 186 |
+
logger.info(
|
| 187 |
+
"Headless save_raw(GET): name=%r voice=%r instr_len=%d tools_len=%d",
|
| 188 |
+
name_s,
|
| 189 |
+
voice,
|
| 190 |
+
len(instructions),
|
| 191 |
+
len(tools_text),
|
| 192 |
+
)
|
| 193 |
+
_write_profile(name_s, instructions, tools_text, voice or "cedar")
|
| 194 |
+
value = f"user_personalities/{name_s}"
|
| 195 |
+
choices = [DEFAULT_OPTION, *list_personalities()]
|
| 196 |
+
return {"ok": True, "value": value, "choices": choices}
|
| 197 |
+
except Exception as e:
|
| 198 |
+
return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
|
| 199 |
+
|
| 200 |
+
logger = logging.getLogger(__name__)
|
| 201 |
+
|
| 202 |
+
@app.post("/personalities/apply")
|
| 203 |
+
async def _apply(
|
| 204 |
+
payload: ApplyPayload | None = None,
|
| 205 |
+
name: str | None = None,
|
| 206 |
+
persist: Optional[bool] = None,
|
| 207 |
+
request: Optional[Request] = None,
|
| 208 |
+
) -> dict: # type: ignore
|
| 209 |
+
loop = get_loop()
|
| 210 |
+
if loop is None:
|
| 211 |
+
return JSONResponse({"ok": False, "error": "loop_unavailable"}, status_code=503) # type: ignore
|
| 212 |
+
|
| 213 |
+
# Accept both JSON payload and query param for convenience
|
| 214 |
+
sel_name: Optional[str] = None
|
| 215 |
+
persist_flag = bool(persist) if persist is not None else False
|
| 216 |
+
if payload and getattr(payload, "name", None):
|
| 217 |
+
sel_name = payload.name
|
| 218 |
+
persist_flag = bool(getattr(payload, "persist", False))
|
| 219 |
+
elif name:
|
| 220 |
+
sel_name = name
|
| 221 |
+
elif request is not None:
|
| 222 |
+
try:
|
| 223 |
+
body = await request.json()
|
| 224 |
+
if isinstance(body, dict) and body.get("name"):
|
| 225 |
+
sel_name = str(body.get("name"))
|
| 226 |
+
if isinstance(body, dict) and "persist" in body:
|
| 227 |
+
persist_flag = bool(body.get("persist"))
|
| 228 |
+
except Exception:
|
| 229 |
+
sel_name = None
|
| 230 |
+
if request is not None:
|
| 231 |
+
try:
|
| 232 |
+
q_persist = request.query_params.get("persist")
|
| 233 |
+
if q_persist is not None:
|
| 234 |
+
persist_flag = str(q_persist).lower() in {"1", "true", "yes", "on"}
|
| 235 |
+
except Exception:
|
| 236 |
+
pass
|
| 237 |
+
if not sel_name:
|
| 238 |
+
sel_name = DEFAULT_OPTION
|
| 239 |
+
|
| 240 |
+
async def _do_apply() -> str:
|
| 241 |
+
sel = None if sel_name == DEFAULT_OPTION else sel_name
|
| 242 |
+
status = await handler.apply_personality(sel)
|
| 243 |
+
return status
|
| 244 |
+
|
| 245 |
+
try:
|
| 246 |
+
logger.info("Headless apply: requested name=%r", sel_name)
|
| 247 |
+
fut = asyncio.run_coroutine_threadsafe(_do_apply(), loop)
|
| 248 |
+
status = fut.result(timeout=10)
|
| 249 |
+
persisted_choice = _startup_choice()
|
| 250 |
+
if persist_flag and persist_personality is not None:
|
| 251 |
+
try:
|
| 252 |
+
persist_personality(None if sel_name == DEFAULT_OPTION else sel_name)
|
| 253 |
+
persisted_choice = _startup_choice()
|
| 254 |
+
except Exception as e:
|
| 255 |
+
logger.warning("Failed to persist startup personality: %s", e)
|
| 256 |
+
return {"ok": True, "status": status, "startup": persisted_choice}
|
| 257 |
+
except Exception as e:
|
| 258 |
+
return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
|
| 259 |
+
|
| 260 |
+
@app.get("/voices")
|
| 261 |
+
async def _voices() -> list[str]:
|
| 262 |
+
loop = get_loop()
|
| 263 |
+
if loop is None:
|
| 264 |
+
return ["cedar"]
|
| 265 |
+
|
| 266 |
+
async def _get_v() -> list[str]:
|
| 267 |
+
try:
|
| 268 |
+
return await handler.get_available_voices()
|
| 269 |
+
except Exception:
|
| 270 |
+
return ["cedar"]
|
| 271 |
+
|
| 272 |
+
try:
|
| 273 |
+
fut = asyncio.run_coroutine_threadsafe(_get_v(), loop)
|
| 274 |
+
return fut.result(timeout=10)
|
| 275 |
+
except Exception:
|
| 276 |
+
return ["cedar"]
|
src/reachy_mini_conversation_app/main.py
CHANGED
|
@@ -2,23 +2,23 @@
|
|
| 2 |
|
| 3 |
import os
|
| 4 |
import sys
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
import gradio as gr
|
| 8 |
from fastapi import FastAPI
|
| 9 |
from fastrtc import Stream
|
|
|
|
| 10 |
|
| 11 |
-
from reachy_mini import ReachyMini
|
| 12 |
-
from reachy_mini_conversation_app.moves import MovementManager
|
| 13 |
from reachy_mini_conversation_app.utils import (
|
| 14 |
parse_args,
|
| 15 |
setup_logger,
|
| 16 |
handle_vision_stuff,
|
| 17 |
)
|
| 18 |
-
from reachy_mini_conversation_app.console import LocalStream
|
| 19 |
-
from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
|
| 20 |
-
from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
|
| 21 |
-
from reachy_mini_conversation_app.audio.head_wobbler import HeadWobbler
|
| 22 |
|
| 23 |
|
| 24 |
def update_chatbot(chatbot: List[Dict[str, Any]], response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
|
@@ -29,7 +29,24 @@ def update_chatbot(chatbot: List[Dict[str, Any]], response: Dict[str, Any]) -> L
|
|
| 29 |
|
| 30 |
def main() -> None:
|
| 31 |
"""Entrypoint for the Reachy Mini conversation app."""
|
| 32 |
-
args = parse_args()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
logger = setup_logger(args.debug)
|
| 35 |
logger.info("Starting Reachy Mini Conversation App")
|
|
@@ -37,23 +54,24 @@ def main() -> None:
|
|
| 37 |
if args.no_camera and args.head_tracker is not None:
|
| 38 |
logger.warning("Head tracking is not activated due to --no-camera.")
|
| 39 |
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
|
|
|
| 57 |
|
| 58 |
# Check if running in simulation mode without --gradio
|
| 59 |
if robot.client.get_status()["simulation_enabled"] and not args.gradio:
|
|
@@ -91,25 +109,52 @@ def main() -> None:
|
|
| 91 |
)
|
| 92 |
logger.debug(f"Chatbot avatar images: {chatbot.avatar_images}")
|
| 93 |
|
| 94 |
-
handler = OpenaiRealtimeHandler(deps)
|
| 95 |
|
| 96 |
stream_manager: gr.Blocks | LocalStream | None = None
|
| 97 |
|
| 98 |
if args.gradio:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
stream = Stream(
|
| 100 |
handler=handler,
|
| 101 |
mode="send-receive",
|
| 102 |
modality="audio",
|
| 103 |
-
additional_inputs=[
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
additional_outputs=[chatbot],
|
| 105 |
additional_outputs_handler=update_chatbot,
|
| 106 |
ui_args={"title": "Talk with Reachy Mini"},
|
| 107 |
)
|
| 108 |
stream_manager = stream.ui
|
| 109 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
app = gr.mount_gradio_app(app, stream.ui, path="/")
|
| 111 |
else:
|
| 112 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
|
| 114 |
# Each async service → its own thread/loop
|
| 115 |
movement_manager.start()
|
|
@@ -119,15 +164,25 @@ def main() -> None:
|
|
| 119 |
if vision_manager:
|
| 120 |
vision_manager.start()
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
try:
|
| 123 |
stream_manager.launch()
|
| 124 |
except KeyboardInterrupt:
|
| 125 |
logger.info("Keyboard interruption in main thread... closing server.")
|
| 126 |
finally:
|
| 127 |
-
# Stop the stream manager and its pipelines
|
| 128 |
-
stream_manager.close()
|
| 129 |
-
|
| 130 |
-
# Stop other services
|
| 131 |
movement_manager.stop()
|
| 132 |
head_wobbler.stop()
|
| 133 |
if camera_worker:
|
|
@@ -137,8 +192,36 @@ def main() -> None:
|
|
| 137 |
|
| 138 |
# prevent connection to keep alive some threads
|
| 139 |
robot.client.disconnect()
|
|
|
|
| 140 |
logger.info("Shutdown complete.")
|
| 141 |
|
| 142 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
if __name__ == "__main__":
|
| 144 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
import os
|
| 4 |
import sys
|
| 5 |
+
import time
|
| 6 |
+
import asyncio
|
| 7 |
+
import argparse
|
| 8 |
+
import threading
|
| 9 |
+
from typing import Any, Dict, List, Optional
|
| 10 |
|
| 11 |
import gradio as gr
|
| 12 |
from fastapi import FastAPI
|
| 13 |
from fastrtc import Stream
|
| 14 |
+
from gradio.utils import get_space
|
| 15 |
|
| 16 |
+
from reachy_mini import ReachyMini, ReachyMiniApp
|
|
|
|
| 17 |
from reachy_mini_conversation_app.utils import (
|
| 18 |
parse_args,
|
| 19 |
setup_logger,
|
| 20 |
handle_vision_stuff,
|
| 21 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
|
| 24 |
def update_chatbot(chatbot: List[Dict[str, Any]], response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
|
|
|
| 29 |
|
| 30 |
def main() -> None:
|
| 31 |
"""Entrypoint for the Reachy Mini conversation app."""
|
| 32 |
+
args, _ = parse_args()
|
| 33 |
+
run(args)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def run(
|
| 37 |
+
args: argparse.Namespace,
|
| 38 |
+
robot: ReachyMini = None,
|
| 39 |
+
app_stop_event: Optional[threading.Event] = None,
|
| 40 |
+
settings_app: Optional[FastAPI] = None,
|
| 41 |
+
instance_path: Optional[str] = None,
|
| 42 |
+
) -> None:
|
| 43 |
+
"""Run the Reachy Mini conversation app."""
|
| 44 |
+
# Putting these dependencies here makes the dashboard faster to load when the conversation app is installed
|
| 45 |
+
from reachy_mini_conversation_app.moves import MovementManager
|
| 46 |
+
from reachy_mini_conversation_app.console import LocalStream
|
| 47 |
+
from reachy_mini_conversation_app.openai_realtime import OpenaiRealtimeHandler
|
| 48 |
+
from reachy_mini_conversation_app.tools.core_tools import ToolDependencies
|
| 49 |
+
from reachy_mini_conversation_app.audio.head_wobbler import HeadWobbler
|
| 50 |
|
| 51 |
logger = setup_logger(args.debug)
|
| 52 |
logger.info("Starting Reachy Mini Conversation App")
|
|
|
|
| 54 |
if args.no_camera and args.head_tracker is not None:
|
| 55 |
logger.warning("Head tracking is not activated due to --no-camera.")
|
| 56 |
|
| 57 |
+
if robot is None:
|
| 58 |
+
# Initialize robot with appropriate backend
|
| 59 |
+
# TODO: Implement dynamic robot connection detection
|
| 60 |
+
# Automatically detect and connect to available Reachy Mini robot(s!)
|
| 61 |
+
# Priority checks (in order):
|
| 62 |
+
# 1. Reachy Lite connected directly to the host
|
| 63 |
+
# 2. Reachy Mini daemon running on localhost (same device)
|
| 64 |
+
# 3. Reachy Mini daemon on local network (same subnet)
|
| 65 |
+
|
| 66 |
+
if args.wireless_version and not args.on_device:
|
| 67 |
+
logger.info("Using WebRTC backend for fully remote wireless version")
|
| 68 |
+
robot = ReachyMini(media_backend="webrtc", localhost_only=False)
|
| 69 |
+
elif args.wireless_version and args.on_device:
|
| 70 |
+
logger.info("Using GStreamer backend for on-device wireless version")
|
| 71 |
+
robot = ReachyMini(media_backend="gstreamer")
|
| 72 |
+
else:
|
| 73 |
+
logger.info("Using default backend for lite version")
|
| 74 |
+
robot = ReachyMini(media_backend="default")
|
| 75 |
|
| 76 |
# Check if running in simulation mode without --gradio
|
| 77 |
if robot.client.get_status()["simulation_enabled"] and not args.gradio:
|
|
|
|
| 109 |
)
|
| 110 |
logger.debug(f"Chatbot avatar images: {chatbot.avatar_images}")
|
| 111 |
|
| 112 |
+
handler = OpenaiRealtimeHandler(deps, gradio_mode=args.gradio, instance_path=instance_path)
|
| 113 |
|
| 114 |
stream_manager: gr.Blocks | LocalStream | None = None
|
| 115 |
|
| 116 |
if args.gradio:
|
| 117 |
+
api_key_textbox = gr.Textbox(
|
| 118 |
+
label="OPENAI API Key",
|
| 119 |
+
type="password",
|
| 120 |
+
value=os.getenv("OPENAI_API_KEY") if not get_space() else "",
|
| 121 |
+
)
|
| 122 |
+
|
| 123 |
+
from reachy_mini_conversation_app.gradio_personality import PersonalityUI
|
| 124 |
+
|
| 125 |
+
personality_ui = PersonalityUI()
|
| 126 |
+
personality_ui.create_components()
|
| 127 |
+
|
| 128 |
stream = Stream(
|
| 129 |
handler=handler,
|
| 130 |
mode="send-receive",
|
| 131 |
modality="audio",
|
| 132 |
+
additional_inputs=[
|
| 133 |
+
chatbot,
|
| 134 |
+
api_key_textbox,
|
| 135 |
+
*personality_ui.additional_inputs_ordered(),
|
| 136 |
+
],
|
| 137 |
additional_outputs=[chatbot],
|
| 138 |
additional_outputs_handler=update_chatbot,
|
| 139 |
ui_args={"title": "Talk with Reachy Mini"},
|
| 140 |
)
|
| 141 |
stream_manager = stream.ui
|
| 142 |
+
if not settings_app:
|
| 143 |
+
app = FastAPI()
|
| 144 |
+
else:
|
| 145 |
+
app = settings_app
|
| 146 |
+
|
| 147 |
+
personality_ui.wire_events(handler, stream_manager)
|
| 148 |
+
|
| 149 |
app = gr.mount_gradio_app(app, stream.ui, path="/")
|
| 150 |
else:
|
| 151 |
+
# In headless mode, wire settings_app + instance_path to console LocalStream
|
| 152 |
+
stream_manager = LocalStream(
|
| 153 |
+
handler,
|
| 154 |
+
robot,
|
| 155 |
+
settings_app=settings_app,
|
| 156 |
+
instance_path=instance_path,
|
| 157 |
+
)
|
| 158 |
|
| 159 |
# Each async service → its own thread/loop
|
| 160 |
movement_manager.start()
|
|
|
|
| 164 |
if vision_manager:
|
| 165 |
vision_manager.start()
|
| 166 |
|
| 167 |
+
def poll_stop_event() -> None:
|
| 168 |
+
"""Poll the stop event to allow graceful shutdown."""
|
| 169 |
+
if app_stop_event is not None:
|
| 170 |
+
app_stop_event.wait()
|
| 171 |
+
|
| 172 |
+
logger.info("App stop event detected, shutting down...")
|
| 173 |
+
try:
|
| 174 |
+
stream_manager.close()
|
| 175 |
+
except Exception as e:
|
| 176 |
+
logger.error(f"Error while closing stream manager: {e}")
|
| 177 |
+
|
| 178 |
+
if app_stop_event:
|
| 179 |
+
threading.Thread(target=poll_stop_event, daemon=True).start()
|
| 180 |
+
|
| 181 |
try:
|
| 182 |
stream_manager.launch()
|
| 183 |
except KeyboardInterrupt:
|
| 184 |
logger.info("Keyboard interruption in main thread... closing server.")
|
| 185 |
finally:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 186 |
movement_manager.stop()
|
| 187 |
head_wobbler.stop()
|
| 188 |
if camera_worker:
|
|
|
|
| 192 |
|
| 193 |
# prevent connection to keep alive some threads
|
| 194 |
robot.client.disconnect()
|
| 195 |
+
time.sleep(1)
|
| 196 |
logger.info("Shutdown complete.")
|
| 197 |
|
| 198 |
|
| 199 |
+
class ReachyMiniConversationApp(ReachyMiniApp): # type: ignore[misc]
|
| 200 |
+
"""Reachy Mini Apps entry point for the conversation app."""
|
| 201 |
+
|
| 202 |
+
custom_app_url = "http://0.0.0.0:7860/"
|
| 203 |
+
dont_start_webserver = False
|
| 204 |
+
|
| 205 |
+
def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None:
|
| 206 |
+
"""Run the Reachy Mini conversation app."""
|
| 207 |
+
loop = asyncio.new_event_loop()
|
| 208 |
+
asyncio.set_event_loop(loop)
|
| 209 |
+
|
| 210 |
+
args, _ = parse_args()
|
| 211 |
+
# args.head_tracker = "mediapipe"
|
| 212 |
+
instance_path = self._get_instance_path().parent
|
| 213 |
+
run(
|
| 214 |
+
args,
|
| 215 |
+
robot=reachy_mini,
|
| 216 |
+
app_stop_event=stop_event,
|
| 217 |
+
settings_app=self.settings_app,
|
| 218 |
+
instance_path=instance_path,
|
| 219 |
+
)
|
| 220 |
+
|
| 221 |
+
|
| 222 |
if __name__ == "__main__":
|
| 223 |
+
app = ReachyMiniConversationApp()
|
| 224 |
+
try:
|
| 225 |
+
app.wrapped_run()
|
| 226 |
+
except KeyboardInterrupt:
|
| 227 |
+
app.stop()
|
src/reachy_mini_conversation_app/openai_realtime.py
CHANGED
|
@@ -3,7 +3,8 @@ import base64
|
|
| 3 |
import random
|
| 4 |
import asyncio
|
| 5 |
import logging
|
| 6 |
-
from typing import Any, Final, Tuple, Literal
|
|
|
|
| 7 |
from datetime import datetime
|
| 8 |
|
| 9 |
import cv2
|
|
@@ -16,7 +17,7 @@ from scipy.signal import resample
|
|
| 16 |
from websockets.exceptions import ConnectionClosedError
|
| 17 |
|
| 18 |
from reachy_mini_conversation_app.config import config
|
| 19 |
-
from reachy_mini_conversation_app.prompts import get_session_instructions
|
| 20 |
from reachy_mini_conversation_app.tools.core_tools import (
|
| 21 |
ToolDependencies,
|
| 22 |
get_tool_specs,
|
|
@@ -33,7 +34,7 @@ OPEN_AI_OUTPUT_SAMPLE_RATE: Final[Literal[24000]] = 24000
|
|
| 33 |
class OpenaiRealtimeHandler(AsyncStreamHandler):
|
| 34 |
"""An OpenAI realtime handler for fastrtc Stream."""
|
| 35 |
|
| 36 |
-
def __init__(self, deps: ToolDependencies):
|
| 37 |
"""Initialize the handler."""
|
| 38 |
super().__init__(
|
| 39 |
expected_layout="mono",
|
|
@@ -57,15 +58,81 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 57 |
self.last_activity_time = asyncio.get_event_loop().time()
|
| 58 |
self.start_time = asyncio.get_event_loop().time()
|
| 59 |
self.is_idle_tool_call = False
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
# Debouncing for partial transcripts
|
| 62 |
self.partial_transcript_task: asyncio.Task[None] | None = None
|
| 63 |
-
self.partial_transcript_sequence: int = 0
|
| 64 |
self.partial_debounce_delay = 0.5 # seconds
|
| 65 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
def copy(self) -> "OpenaiRealtimeHandler":
|
| 67 |
"""Create a copy of the handler."""
|
| 68 |
-
return OpenaiRealtimeHandler(self.deps)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
async def _emit_debounced_partial(self, transcript: str, sequence: int) -> None:
|
| 71 |
"""Emit partial transcript after debounce delay."""
|
|
@@ -73,9 +140,7 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 73 |
await asyncio.sleep(self.partial_debounce_delay)
|
| 74 |
# Only emit if this is still the latest partial (by sequence number)
|
| 75 |
if self.partial_transcript_sequence == sequence:
|
| 76 |
-
await self.output_queue.put(
|
| 77 |
-
AdditionalOutputs({"role": "user_partial", "content": transcript})
|
| 78 |
-
)
|
| 79 |
logger.debug(f"Debounced partial emitted: {transcript}")
|
| 80 |
except asyncio.CancelledError:
|
| 81 |
logger.debug("Debounced partial cancelled")
|
|
@@ -83,7 +148,27 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 83 |
|
| 84 |
async def start_up(self) -> None:
|
| 85 |
"""Start the handler with minimal retries on unexpected websocket closure."""
|
| 86 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
max_attempts = 3
|
| 89 |
for attempt in range(1, max_attempts + 1):
|
|
@@ -93,10 +178,7 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 93 |
return
|
| 94 |
except ConnectionClosedError as e:
|
| 95 |
# Abrupt close (e.g., "no close frame received or sent") → retry
|
| 96 |
-
logger.warning(
|
| 97 |
-
"Realtime websocket closed unexpectedly (attempt %d/%d): %s",
|
| 98 |
-
attempt, max_attempts, e
|
| 99 |
-
)
|
| 100 |
if attempt < max_attempts:
|
| 101 |
# exponential backoff with jitter
|
| 102 |
base_delay = 2 ** (attempt - 1) # 1s, 2s, 4s, 8s, etc.
|
|
@@ -109,6 +191,43 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 109 |
finally:
|
| 110 |
# never keep a stale reference
|
| 111 |
self.connection = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
async def _run_realtime_session(self) -> None:
|
| 114 |
"""Establish and manage a single realtime session."""
|
|
@@ -124,10 +243,7 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 124 |
"type": "audio/pcm",
|
| 125 |
"rate": self.input_sample_rate,
|
| 126 |
},
|
| 127 |
-
"transcription": {
|
| 128 |
-
"model": "gpt-4o-transcribe",
|
| 129 |
-
"language": "en"
|
| 130 |
-
},
|
| 131 |
"turn_detection": {
|
| 132 |
"type": "server_vad",
|
| 133 |
"interrupt_response": True,
|
|
@@ -138,13 +254,21 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 138 |
"type": "audio/pcm",
|
| 139 |
"rate": self.output_sample_rate,
|
| 140 |
},
|
| 141 |
-
"voice":
|
| 142 |
},
|
| 143 |
},
|
| 144 |
-
"tools":
|
| 145 |
"tool_choice": "auto",
|
| 146 |
},
|
| 147 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
except Exception:
|
| 149 |
logger.exception("Realtime session.update failed; aborting startup")
|
| 150 |
return
|
|
@@ -153,6 +277,10 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 153 |
|
| 154 |
# Manage event received from the openai server
|
| 155 |
self.connection = conn
|
|
|
|
|
|
|
|
|
|
|
|
|
| 156 |
async for event in self.connection:
|
| 157 |
logger.debug(f"OpenAI event: {event.type}")
|
| 158 |
if event.type == "input_audio_buffer.speech_started":
|
|
@@ -168,10 +296,10 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 168 |
logger.debug("User speech stopped - server will auto-commit with VAD")
|
| 169 |
|
| 170 |
if event.type in (
|
| 171 |
-
"response.audio.done",
|
| 172 |
-
"response.output_audio.done",
|
| 173 |
-
"response.audio.completed",
|
| 174 |
-
"response.completed",
|
| 175 |
):
|
| 176 |
logger.debug("response completed")
|
| 177 |
|
|
@@ -336,7 +464,9 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 336 |
|
| 337 |
# Only show user-facing errors, not internal state errors
|
| 338 |
if code not in ("input_audio_buffer_commit_empty", "conversation_already_has_active_response"):
|
| 339 |
-
await self.output_queue.put(
|
|
|
|
|
|
|
| 340 |
|
| 341 |
# Microphone receive
|
| 342 |
async def receive(self, frame: Tuple[int, NDArray[np.int16]]) -> None:
|
|
@@ -355,7 +485,7 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 355 |
|
| 356 |
input_sample_rate, audio_frame = frame
|
| 357 |
|
| 358 |
-
#Reshape if needed
|
| 359 |
if audio_frame.ndim == 2:
|
| 360 |
# Scipy channels last convention
|
| 361 |
if audio_frame.shape[1] > audio_frame.shape[0]:
|
|
@@ -366,17 +496,18 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 366 |
|
| 367 |
# Resample if needed
|
| 368 |
if self.input_sample_rate != input_sample_rate:
|
| 369 |
-
audio_frame = resample(
|
| 370 |
-
audio_frame,
|
| 371 |
-
int(len(audio_frame) * self.input_sample_rate / input_sample_rate)
|
| 372 |
-
)
|
| 373 |
|
| 374 |
# Cast if needed
|
| 375 |
audio_frame = audio_to_int16(audio_frame)
|
| 376 |
|
| 377 |
-
# Send to OpenAI
|
| 378 |
-
|
| 379 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 380 |
|
| 381 |
async def emit(self) -> Tuple[int, NDArray[np.int16]] | AdditionalOutputs | None:
|
| 382 |
"""Emit audio frame to be played by the speaker."""
|
|
@@ -398,6 +529,7 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 398 |
|
| 399 |
async def shutdown(self) -> None:
|
| 400 |
"""Shutdown the handler."""
|
|
|
|
| 401 |
# Cancel any pending debounce task
|
| 402 |
if self.partial_transcript_task and not self.partial_transcript_task.done():
|
| 403 |
self.partial_transcript_task.cancel()
|
|
@@ -430,6 +562,73 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 430 |
dt = datetime.now() # wall-clock
|
| 431 |
return f"[{dt.strftime('%Y-%m-%d %H:%M:%S')} | +{elapsed_seconds:.1f}s]"
|
| 432 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 433 |
async def send_idle_signal(self, idle_duration: float) -> None:
|
| 434 |
"""Send an idle signal to the openai server."""
|
| 435 |
logger.debug("Sending idle signal")
|
|
@@ -451,3 +650,68 @@ class OpenaiRealtimeHandler(AsyncStreamHandler):
|
|
| 451 |
"tool_choice": "required",
|
| 452 |
},
|
| 453 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
import random
|
| 4 |
import asyncio
|
| 5 |
import logging
|
| 6 |
+
from typing import Any, Final, Tuple, Literal, Optional
|
| 7 |
+
from pathlib import Path
|
| 8 |
from datetime import datetime
|
| 9 |
|
| 10 |
import cv2
|
|
|
|
| 17 |
from websockets.exceptions import ConnectionClosedError
|
| 18 |
|
| 19 |
from reachy_mini_conversation_app.config import config
|
| 20 |
+
from reachy_mini_conversation_app.prompts import get_session_voice, get_session_instructions
|
| 21 |
from reachy_mini_conversation_app.tools.core_tools import (
|
| 22 |
ToolDependencies,
|
| 23 |
get_tool_specs,
|
|
|
|
| 34 |
class OpenaiRealtimeHandler(AsyncStreamHandler):
|
| 35 |
"""An OpenAI realtime handler for fastrtc Stream."""
|
| 36 |
|
| 37 |
+
def __init__(self, deps: ToolDependencies, gradio_mode: bool = False, instance_path: Optional[str] = None):
|
| 38 |
"""Initialize the handler."""
|
| 39 |
super().__init__(
|
| 40 |
expected_layout="mono",
|
|
|
|
| 58 |
self.last_activity_time = asyncio.get_event_loop().time()
|
| 59 |
self.start_time = asyncio.get_event_loop().time()
|
| 60 |
self.is_idle_tool_call = False
|
| 61 |
+
self.gradio_mode = gradio_mode
|
| 62 |
+
self.instance_path = instance_path
|
| 63 |
+
# Track how the API key was provided (env vs textbox) and its value
|
| 64 |
+
self._key_source: Literal["env", "textbox"] = "env"
|
| 65 |
+
self._provided_api_key: str | None = None
|
| 66 |
|
| 67 |
# Debouncing for partial transcripts
|
| 68 |
self.partial_transcript_task: asyncio.Task[None] | None = None
|
| 69 |
+
self.partial_transcript_sequence: int = 0 # sequence counter to prevent stale emissions
|
| 70 |
self.partial_debounce_delay = 0.5 # seconds
|
| 71 |
|
| 72 |
+
# Internal lifecycle flags
|
| 73 |
+
self._shutdown_requested: bool = False
|
| 74 |
+
self._connected_event: asyncio.Event = asyncio.Event()
|
| 75 |
+
|
| 76 |
def copy(self) -> "OpenaiRealtimeHandler":
|
| 77 |
"""Create a copy of the handler."""
|
| 78 |
+
return OpenaiRealtimeHandler(self.deps, self.gradio_mode, self.instance_path)
|
| 79 |
+
|
| 80 |
+
async def apply_personality(self, profile: str | None) -> str:
|
| 81 |
+
"""Apply a new personality (profile) at runtime if possible.
|
| 82 |
+
|
| 83 |
+
- Updates the global config's selected profile for subsequent calls.
|
| 84 |
+
- If a realtime connection is active, sends a session.update with the
|
| 85 |
+
freshly resolved instructions so the change takes effect immediately.
|
| 86 |
+
|
| 87 |
+
Returns a short status message for UI feedback.
|
| 88 |
+
"""
|
| 89 |
+
try:
|
| 90 |
+
# Update the in-process config value and env
|
| 91 |
+
from reachy_mini_conversation_app.config import config as _config
|
| 92 |
+
from reachy_mini_conversation_app.config import set_custom_profile
|
| 93 |
+
|
| 94 |
+
set_custom_profile(profile)
|
| 95 |
+
logger.info(
|
| 96 |
+
"Set custom profile to %r (config=%r)", profile, getattr(_config, "REACHY_MINI_CUSTOM_PROFILE", None)
|
| 97 |
+
)
|
| 98 |
+
|
| 99 |
+
try:
|
| 100 |
+
instructions = get_session_instructions()
|
| 101 |
+
voice = get_session_voice()
|
| 102 |
+
except BaseException as e: # catch SystemExit from prompt loader without crashing
|
| 103 |
+
logger.error("Failed to resolve personality content: %s", e)
|
| 104 |
+
return f"Failed to apply personality: {e}"
|
| 105 |
+
|
| 106 |
+
# Attempt a live update first, then force a full restart to ensure it sticks
|
| 107 |
+
if self.connection is not None:
|
| 108 |
+
try:
|
| 109 |
+
await self.connection.session.update(
|
| 110 |
+
session={
|
| 111 |
+
"type": "realtime",
|
| 112 |
+
"instructions": instructions,
|
| 113 |
+
"audio": {"output": {"voice": voice}},
|
| 114 |
+
},
|
| 115 |
+
)
|
| 116 |
+
logger.info("Applied personality via live update: %s", profile or "built-in default")
|
| 117 |
+
except Exception as e:
|
| 118 |
+
logger.warning("Live update failed; will restart session: %s", e)
|
| 119 |
+
|
| 120 |
+
# Force a real restart to guarantee the new instructions/voice
|
| 121 |
+
try:
|
| 122 |
+
await self._restart_session()
|
| 123 |
+
return "Applied personality and restarted realtime session."
|
| 124 |
+
except Exception as e:
|
| 125 |
+
logger.warning("Failed to restart session after apply: %s", e)
|
| 126 |
+
return "Applied personality. Will take effect on next connection."
|
| 127 |
+
else:
|
| 128 |
+
logger.info(
|
| 129 |
+
"Applied personality recorded: %s (no live connection; will apply on next session)",
|
| 130 |
+
profile or "built-in default",
|
| 131 |
+
)
|
| 132 |
+
return "Applied personality. Will take effect on next connection."
|
| 133 |
+
except Exception as e:
|
| 134 |
+
logger.error("Error applying personality '%s': %s", profile, e)
|
| 135 |
+
return f"Failed to apply personality: {e}"
|
| 136 |
|
| 137 |
async def _emit_debounced_partial(self, transcript: str, sequence: int) -> None:
|
| 138 |
"""Emit partial transcript after debounce delay."""
|
|
|
|
| 140 |
await asyncio.sleep(self.partial_debounce_delay)
|
| 141 |
# Only emit if this is still the latest partial (by sequence number)
|
| 142 |
if self.partial_transcript_sequence == sequence:
|
| 143 |
+
await self.output_queue.put(AdditionalOutputs({"role": "user_partial", "content": transcript}))
|
|
|
|
|
|
|
| 144 |
logger.debug(f"Debounced partial emitted: {transcript}")
|
| 145 |
except asyncio.CancelledError:
|
| 146 |
logger.debug("Debounced partial cancelled")
|
|
|
|
| 148 |
|
| 149 |
async def start_up(self) -> None:
|
| 150 |
"""Start the handler with minimal retries on unexpected websocket closure."""
|
| 151 |
+
openai_api_key = config.OPENAI_API_KEY
|
| 152 |
+
if self.gradio_mode and not openai_api_key:
|
| 153 |
+
# api key was not found in .env or in the environment variables
|
| 154 |
+
await self.wait_for_args() # type: ignore[no-untyped-call]
|
| 155 |
+
args = list(self.latest_args)
|
| 156 |
+
textbox_api_key = args[3] if len(args[3]) > 0 else None
|
| 157 |
+
if textbox_api_key is not None:
|
| 158 |
+
openai_api_key = textbox_api_key
|
| 159 |
+
self._key_source = "textbox"
|
| 160 |
+
self._provided_api_key = textbox_api_key
|
| 161 |
+
else:
|
| 162 |
+
openai_api_key = config.OPENAI_API_KEY
|
| 163 |
+
else:
|
| 164 |
+
if not openai_api_key or not openai_api_key.strip():
|
| 165 |
+
# In headless console mode, LocalStream now blocks startup until the key is provided.
|
| 166 |
+
# However, unit tests may invoke this handler directly with a stubbed client.
|
| 167 |
+
# To keep tests hermetic without requiring a real key, fall back to a placeholder.
|
| 168 |
+
logger.warning("OPENAI_API_KEY missing. Proceeding with a placeholder (tests/offline).")
|
| 169 |
+
openai_api_key = "DUMMY"
|
| 170 |
+
|
| 171 |
+
self.client = AsyncOpenAI(api_key=openai_api_key)
|
| 172 |
|
| 173 |
max_attempts = 3
|
| 174 |
for attempt in range(1, max_attempts + 1):
|
|
|
|
| 178 |
return
|
| 179 |
except ConnectionClosedError as e:
|
| 180 |
# Abrupt close (e.g., "no close frame received or sent") → retry
|
| 181 |
+
logger.warning("Realtime websocket closed unexpectedly (attempt %d/%d): %s", attempt, max_attempts, e)
|
|
|
|
|
|
|
|
|
|
| 182 |
if attempt < max_attempts:
|
| 183 |
# exponential backoff with jitter
|
| 184 |
base_delay = 2 ** (attempt - 1) # 1s, 2s, 4s, 8s, etc.
|
|
|
|
| 191 |
finally:
|
| 192 |
# never keep a stale reference
|
| 193 |
self.connection = None
|
| 194 |
+
try:
|
| 195 |
+
self._connected_event.clear()
|
| 196 |
+
except Exception:
|
| 197 |
+
pass
|
| 198 |
+
|
| 199 |
+
async def _restart_session(self) -> None:
|
| 200 |
+
"""Force-close the current session and start a fresh one in background.
|
| 201 |
+
|
| 202 |
+
Does not block the caller while the new session is establishing.
|
| 203 |
+
"""
|
| 204 |
+
try:
|
| 205 |
+
if self.connection is not None:
|
| 206 |
+
try:
|
| 207 |
+
await self.connection.close()
|
| 208 |
+
except Exception:
|
| 209 |
+
pass
|
| 210 |
+
finally:
|
| 211 |
+
self.connection = None
|
| 212 |
+
|
| 213 |
+
# Ensure we have a client (start_up must have run once)
|
| 214 |
+
if getattr(self, "client", None) is None:
|
| 215 |
+
logger.warning("Cannot restart: OpenAI client not initialized yet.")
|
| 216 |
+
return
|
| 217 |
+
|
| 218 |
+
# Fire-and-forget new session and wait briefly for connection
|
| 219 |
+
try:
|
| 220 |
+
self._connected_event.clear()
|
| 221 |
+
except Exception:
|
| 222 |
+
pass
|
| 223 |
+
asyncio.create_task(self._run_realtime_session(), name="openai-realtime-restart")
|
| 224 |
+
try:
|
| 225 |
+
await asyncio.wait_for(self._connected_event.wait(), timeout=5.0)
|
| 226 |
+
logger.info("Realtime session restarted and connected.")
|
| 227 |
+
except asyncio.TimeoutError:
|
| 228 |
+
logger.warning("Realtime session restart timed out; continuing in background.")
|
| 229 |
+
except Exception as e:
|
| 230 |
+
logger.warning("_restart_session failed: %s", e)
|
| 231 |
|
| 232 |
async def _run_realtime_session(self) -> None:
|
| 233 |
"""Establish and manage a single realtime session."""
|
|
|
|
| 243 |
"type": "audio/pcm",
|
| 244 |
"rate": self.input_sample_rate,
|
| 245 |
},
|
| 246 |
+
"transcription": {"model": "gpt-4o-transcribe", "language": "en"},
|
|
|
|
|
|
|
|
|
|
| 247 |
"turn_detection": {
|
| 248 |
"type": "server_vad",
|
| 249 |
"interrupt_response": True,
|
|
|
|
| 254 |
"type": "audio/pcm",
|
| 255 |
"rate": self.output_sample_rate,
|
| 256 |
},
|
| 257 |
+
"voice": get_session_voice(),
|
| 258 |
},
|
| 259 |
},
|
| 260 |
+
"tools": get_tool_specs(), # type: ignore[typeddict-item]
|
| 261 |
"tool_choice": "auto",
|
| 262 |
},
|
| 263 |
)
|
| 264 |
+
logger.info(
|
| 265 |
+
"Realtime session initialized with profile=%r voice=%r",
|
| 266 |
+
getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None),
|
| 267 |
+
get_session_voice(),
|
| 268 |
+
)
|
| 269 |
+
# If we reached here, the session update succeeded which implies the API key worked.
|
| 270 |
+
# Persist the key to a newly created .env (copied from .env.example) if needed.
|
| 271 |
+
self._persist_api_key_if_needed()
|
| 272 |
except Exception:
|
| 273 |
logger.exception("Realtime session.update failed; aborting startup")
|
| 274 |
return
|
|
|
|
| 277 |
|
| 278 |
# Manage event received from the openai server
|
| 279 |
self.connection = conn
|
| 280 |
+
try:
|
| 281 |
+
self._connected_event.set()
|
| 282 |
+
except Exception:
|
| 283 |
+
pass
|
| 284 |
async for event in self.connection:
|
| 285 |
logger.debug(f"OpenAI event: {event.type}")
|
| 286 |
if event.type == "input_audio_buffer.speech_started":
|
|
|
|
| 296 |
logger.debug("User speech stopped - server will auto-commit with VAD")
|
| 297 |
|
| 298 |
if event.type in (
|
| 299 |
+
"response.audio.done", # GA
|
| 300 |
+
"response.output_audio.done", # GA alias
|
| 301 |
+
"response.audio.completed", # legacy (for safety)
|
| 302 |
+
"response.completed", # text-only completion
|
| 303 |
):
|
| 304 |
logger.debug("response completed")
|
| 305 |
|
|
|
|
| 464 |
|
| 465 |
# Only show user-facing errors, not internal state errors
|
| 466 |
if code not in ("input_audio_buffer_commit_empty", "conversation_already_has_active_response"):
|
| 467 |
+
await self.output_queue.put(
|
| 468 |
+
AdditionalOutputs({"role": "assistant", "content": f"[error] {msg}"})
|
| 469 |
+
)
|
| 470 |
|
| 471 |
# Microphone receive
|
| 472 |
async def receive(self, frame: Tuple[int, NDArray[np.int16]]) -> None:
|
|
|
|
| 485 |
|
| 486 |
input_sample_rate, audio_frame = frame
|
| 487 |
|
| 488 |
+
# Reshape if needed
|
| 489 |
if audio_frame.ndim == 2:
|
| 490 |
# Scipy channels last convention
|
| 491 |
if audio_frame.shape[1] > audio_frame.shape[0]:
|
|
|
|
| 496 |
|
| 497 |
# Resample if needed
|
| 498 |
if self.input_sample_rate != input_sample_rate:
|
| 499 |
+
audio_frame = resample(audio_frame, int(len(audio_frame) * self.input_sample_rate / input_sample_rate))
|
|
|
|
|
|
|
|
|
|
| 500 |
|
| 501 |
# Cast if needed
|
| 502 |
audio_frame = audio_to_int16(audio_frame)
|
| 503 |
|
| 504 |
+
# Send to OpenAI (guard against races during reconnect)
|
| 505 |
+
try:
|
| 506 |
+
audio_message = base64.b64encode(audio_frame.tobytes()).decode("utf-8")
|
| 507 |
+
await self.connection.input_audio_buffer.append(audio=audio_message)
|
| 508 |
+
except Exception as e:
|
| 509 |
+
logger.debug("Dropping audio frame: connection not ready (%s)", e)
|
| 510 |
+
return
|
| 511 |
|
| 512 |
async def emit(self) -> Tuple[int, NDArray[np.int16]] | AdditionalOutputs | None:
|
| 513 |
"""Emit audio frame to be played by the speaker."""
|
|
|
|
| 529 |
|
| 530 |
async def shutdown(self) -> None:
|
| 531 |
"""Shutdown the handler."""
|
| 532 |
+
self._shutdown_requested = True
|
| 533 |
# Cancel any pending debounce task
|
| 534 |
if self.partial_transcript_task and not self.partial_transcript_task.done():
|
| 535 |
self.partial_transcript_task.cancel()
|
|
|
|
| 562 |
dt = datetime.now() # wall-clock
|
| 563 |
return f"[{dt.strftime('%Y-%m-%d %H:%M:%S')} | +{elapsed_seconds:.1f}s]"
|
| 564 |
|
| 565 |
+
async def get_available_voices(self) -> list[str]:
|
| 566 |
+
"""Try to discover available voices for the configured realtime model.
|
| 567 |
+
|
| 568 |
+
Attempts to retrieve model metadata from the OpenAI Models API and look
|
| 569 |
+
for any keys that might contain voice names. Falls back to a curated
|
| 570 |
+
list known to work with realtime if discovery fails.
|
| 571 |
+
"""
|
| 572 |
+
# Conservative fallback list with default first
|
| 573 |
+
fallback = [
|
| 574 |
+
"cedar",
|
| 575 |
+
"alloy",
|
| 576 |
+
"aria",
|
| 577 |
+
"ballad",
|
| 578 |
+
"verse",
|
| 579 |
+
"sage",
|
| 580 |
+
"coral",
|
| 581 |
+
]
|
| 582 |
+
try:
|
| 583 |
+
# Best effort discovery; safe-guarded for unexpected shapes
|
| 584 |
+
model = await self.client.models.retrieve(config.MODEL_NAME)
|
| 585 |
+
# Try common serialization paths
|
| 586 |
+
raw = None
|
| 587 |
+
for attr in ("model_dump", "to_dict"):
|
| 588 |
+
fn = getattr(model, attr, None)
|
| 589 |
+
if callable(fn):
|
| 590 |
+
try:
|
| 591 |
+
raw = fn()
|
| 592 |
+
break
|
| 593 |
+
except Exception:
|
| 594 |
+
pass
|
| 595 |
+
if raw is None:
|
| 596 |
+
try:
|
| 597 |
+
raw = dict(model)
|
| 598 |
+
except Exception:
|
| 599 |
+
raw = None
|
| 600 |
+
# Scan for voice candidates
|
| 601 |
+
candidates: set[str] = set()
|
| 602 |
+
|
| 603 |
+
def _collect(obj: object) -> None:
|
| 604 |
+
try:
|
| 605 |
+
if isinstance(obj, dict):
|
| 606 |
+
for k, v in obj.items():
|
| 607 |
+
kl = str(k).lower()
|
| 608 |
+
if "voice" in kl and isinstance(v, (list, tuple)):
|
| 609 |
+
for item in v:
|
| 610 |
+
if isinstance(item, str):
|
| 611 |
+
candidates.add(item)
|
| 612 |
+
elif isinstance(item, dict) and "name" in item and isinstance(item["name"], str):
|
| 613 |
+
candidates.add(item["name"])
|
| 614 |
+
else:
|
| 615 |
+
_collect(v)
|
| 616 |
+
elif isinstance(obj, (list, tuple)):
|
| 617 |
+
for it in obj:
|
| 618 |
+
_collect(it)
|
| 619 |
+
except Exception:
|
| 620 |
+
pass
|
| 621 |
+
|
| 622 |
+
if isinstance(raw, dict):
|
| 623 |
+
_collect(raw)
|
| 624 |
+
# Ensure default present and stable order
|
| 625 |
+
voices = sorted(candidates) if candidates else fallback
|
| 626 |
+
if "cedar" not in voices:
|
| 627 |
+
voices = ["cedar", *[v for v in voices if v != "cedar"]]
|
| 628 |
+
return voices
|
| 629 |
+
except Exception:
|
| 630 |
+
return fallback
|
| 631 |
+
|
| 632 |
async def send_idle_signal(self, idle_duration: float) -> None:
|
| 633 |
"""Send an idle signal to the openai server."""
|
| 634 |
logger.debug("Sending idle signal")
|
|
|
|
| 650 |
"tool_choice": "required",
|
| 651 |
},
|
| 652 |
)
|
| 653 |
+
|
| 654 |
+
def _persist_api_key_if_needed(self) -> None:
|
| 655 |
+
"""Persist the API key into `.env` inside `instance_path/` when appropriate.
|
| 656 |
+
|
| 657 |
+
- Only runs in Gradio mode when key came from the textbox and is non-empty.
|
| 658 |
+
- Only saves if `self.instance_path` is not None.
|
| 659 |
+
- Writes `.env` to `instance_path/.env` (does not overwrite if it already exists).
|
| 660 |
+
- If `instance_path/.env.example` exists, copies its contents while overriding OPENAI_API_KEY.
|
| 661 |
+
"""
|
| 662 |
+
try:
|
| 663 |
+
if not self.gradio_mode:
|
| 664 |
+
logger.warning("Not in Gradio mode; skipping API key persistence.")
|
| 665 |
+
return
|
| 666 |
+
if self._key_source != "textbox":
|
| 667 |
+
logger.info("API key not provided via textbox; skipping persistence.")
|
| 668 |
+
return
|
| 669 |
+
key = (self._provided_api_key or "").strip()
|
| 670 |
+
if not key:
|
| 671 |
+
logger.warning("No API key provided via textbox; skipping persistence.")
|
| 672 |
+
return
|
| 673 |
+
if self.instance_path is None:
|
| 674 |
+
logger.warning("Instance path is None; cannot persist API key.")
|
| 675 |
+
return
|
| 676 |
+
|
| 677 |
+
# Update the current process environment for downstream consumers
|
| 678 |
+
try:
|
| 679 |
+
import os
|
| 680 |
+
|
| 681 |
+
os.environ["OPENAI_API_KEY"] = key
|
| 682 |
+
except Exception: # best-effort
|
| 683 |
+
pass
|
| 684 |
+
|
| 685 |
+
target_dir = Path(self.instance_path)
|
| 686 |
+
env_path = target_dir / ".env"
|
| 687 |
+
if env_path.exists():
|
| 688 |
+
# Respect existing user configuration
|
| 689 |
+
logger.info(".env already exists at %s; not overwriting.", env_path)
|
| 690 |
+
return
|
| 691 |
+
|
| 692 |
+
example_path = target_dir / ".env.example"
|
| 693 |
+
content_lines: list[str] = []
|
| 694 |
+
if example_path.exists():
|
| 695 |
+
try:
|
| 696 |
+
content = example_path.read_text(encoding="utf-8")
|
| 697 |
+
content_lines = content.splitlines()
|
| 698 |
+
except Exception as e:
|
| 699 |
+
logger.warning("Failed to read .env.example at %s: %s", example_path, e)
|
| 700 |
+
|
| 701 |
+
# Replace or append the OPENAI_API_KEY line
|
| 702 |
+
replaced = False
|
| 703 |
+
for i, line in enumerate(content_lines):
|
| 704 |
+
if line.strip().startswith("OPENAI_API_KEY="):
|
| 705 |
+
content_lines[i] = f"OPENAI_API_KEY={key}"
|
| 706 |
+
replaced = True
|
| 707 |
+
break
|
| 708 |
+
if not replaced:
|
| 709 |
+
content_lines.append(f"OPENAI_API_KEY={key}")
|
| 710 |
+
|
| 711 |
+
# Ensure file ends with newline
|
| 712 |
+
final_text = "\n".join(content_lines) + "\n"
|
| 713 |
+
env_path.write_text(final_text, encoding="utf-8")
|
| 714 |
+
logger.info("Created %s and stored OPENAI_API_KEY for future runs.", env_path)
|
| 715 |
+
except Exception as e:
|
| 716 |
+
# Never crash the app for QoL persistence; just log.
|
| 717 |
+
logger.warning("Could not persist OPENAI_API_KEY to .env: %s", e)
|
src/reachy_mini_conversation_app/prompts.py
CHANGED
|
@@ -12,6 +12,7 @@ logger = logging.getLogger(__name__)
|
|
| 12 |
PROFILES_DIRECTORY = Path(__file__).parent / "profiles"
|
| 13 |
PROMPTS_LIBRARY_DIRECTORY = Path(__file__).parent / "prompts"
|
| 14 |
INSTRUCTIONS_FILENAME = "instructions.txt"
|
|
|
|
| 15 |
|
| 16 |
|
| 17 |
def _expand_prompt_includes(content: str) -> str:
|
|
@@ -82,3 +83,22 @@ def get_session_instructions() -> str:
|
|
| 82 |
except Exception as e:
|
| 83 |
logger.error(f"Failed to load instructions from profile '{profile}': {e}")
|
| 84 |
sys.exit(1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
PROFILES_DIRECTORY = Path(__file__).parent / "profiles"
|
| 13 |
PROMPTS_LIBRARY_DIRECTORY = Path(__file__).parent / "prompts"
|
| 14 |
INSTRUCTIONS_FILENAME = "instructions.txt"
|
| 15 |
+
VOICE_FILENAME = "voice.txt"
|
| 16 |
|
| 17 |
|
| 18 |
def _expand_prompt_includes(content: str) -> str:
|
|
|
|
| 83 |
except Exception as e:
|
| 84 |
logger.error(f"Failed to load instructions from profile '{profile}': {e}")
|
| 85 |
sys.exit(1)
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def get_session_voice(default: str = "cedar") -> str:
|
| 89 |
+
"""Resolve the voice to use for the session.
|
| 90 |
+
|
| 91 |
+
If a custom profile is selected and contains a voice.txt, return its
|
| 92 |
+
trimmed content; otherwise return the provided default ("cedar").
|
| 93 |
+
"""
|
| 94 |
+
profile = config.REACHY_MINI_CUSTOM_PROFILE
|
| 95 |
+
if not profile:
|
| 96 |
+
return default
|
| 97 |
+
try:
|
| 98 |
+
voice_file = PROFILES_DIRECTORY / profile / VOICE_FILENAME
|
| 99 |
+
if voice_file.exists():
|
| 100 |
+
voice = voice_file.read_text(encoding="utf-8").strip()
|
| 101 |
+
return voice or default
|
| 102 |
+
except Exception:
|
| 103 |
+
pass
|
| 104 |
+
return default
|
src/reachy_mini_conversation_app/static/index.html
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!doctype html>
|
| 2 |
+
<html lang="en">
|
| 3 |
+
<head>
|
| 4 |
+
<meta charset="utf-8" />
|
| 5 |
+
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
| 6 |
+
<title>Reachy Mini Conversation – Settings</title>
|
| 7 |
+
<link rel="stylesheet" href="/static/style.css" />
|
| 8 |
+
</head>
|
| 9 |
+
<body>
|
| 10 |
+
<div class="ambient"></div>
|
| 11 |
+
<div id="loading" class="loading">
|
| 12 |
+
<div class="spinner"></div>
|
| 13 |
+
<p>Loading…</p>
|
| 14 |
+
</div>
|
| 15 |
+
<div class="container">
|
| 16 |
+
<header class="hero">
|
| 17 |
+
<div class="pill">Headless control</div>
|
| 18 |
+
<h1>Reachy Mini Conversation</h1>
|
| 19 |
+
<p class="subtitle">Configure your OpenAI key and tweak personalities without the full UI.</p>
|
| 20 |
+
</header>
|
| 21 |
+
|
| 22 |
+
<div id="configured" class="panel hidden">
|
| 23 |
+
<div class="panel-heading">
|
| 24 |
+
<div>
|
| 25 |
+
<p class="eyebrow">Credentials</p>
|
| 26 |
+
<h2>API key ready</h2>
|
| 27 |
+
</div>
|
| 28 |
+
<span class="chip chip-ok">Connected</span>
|
| 29 |
+
</div>
|
| 30 |
+
<p class="muted">OpenAI API key is already configured. You can jump straight to personalities.</p>
|
| 31 |
+
<button id="change-key-btn" class="ghost">Change API key</button>
|
| 32 |
+
</div>
|
| 33 |
+
|
| 34 |
+
<div id="form-panel" class="panel hidden">
|
| 35 |
+
<div class="panel-heading">
|
| 36 |
+
<div>
|
| 37 |
+
<p class="eyebrow">Credentials</p>
|
| 38 |
+
<h2>Connect OpenAI</h2>
|
| 39 |
+
</div>
|
| 40 |
+
<span class="chip">Required</span>
|
| 41 |
+
</div>
|
| 42 |
+
<p class="muted">Paste your API key once and we will store it locally for the headless conversation loop.</p>
|
| 43 |
+
<label for="api-key">OpenAI API Key</label>
|
| 44 |
+
<input id="api-key" type="password" placeholder="sk-..." autocomplete="off" />
|
| 45 |
+
<div class="actions">
|
| 46 |
+
<button id="save-btn">Save key</button>
|
| 47 |
+
<p id="status" class="status"></p>
|
| 48 |
+
</div>
|
| 49 |
+
</div>
|
| 50 |
+
|
| 51 |
+
<div id="personality-panel" class="panel hidden">
|
| 52 |
+
<div class="panel-heading">
|
| 53 |
+
<div>
|
| 54 |
+
<p class="eyebrow">Profiles</p>
|
| 55 |
+
<h2>Personality studio</h2>
|
| 56 |
+
</div>
|
| 57 |
+
<span class="chip">Live</span>
|
| 58 |
+
</div>
|
| 59 |
+
<p class="muted">Create lean instruction sets, toggle tools, and apply a voice for your Reachy Mini.</p>
|
| 60 |
+
<div class="section">
|
| 61 |
+
<div class="section-heading">
|
| 62 |
+
<h3>Select & launch</h3>
|
| 63 |
+
<p class="muted small">Pick a profile and choose what should launch on startup.</p>
|
| 64 |
+
</div>
|
| 65 |
+
<div class="row row-top">
|
| 66 |
+
<label for="personality-select">Select</label>
|
| 67 |
+
<select id="personality-select"></select>
|
| 68 |
+
<button id="persist-personality" class="ghost">Use on startup</button>
|
| 69 |
+
<button id="apply-personality" class="ghost">Apply</button>
|
| 70 |
+
<button id="new-personality" class="ghost">New</button>
|
| 71 |
+
</div>
|
| 72 |
+
<div class="row">
|
| 73 |
+
<label>Startup personality</label>
|
| 74 |
+
<div class="startup-row">
|
| 75 |
+
<span id="startup-label" class="chip">Built-in default</span>
|
| 76 |
+
</div>
|
| 77 |
+
</div>
|
| 78 |
+
</div>
|
| 79 |
+
|
| 80 |
+
<div class="section">
|
| 81 |
+
<div class="section-heading">
|
| 82 |
+
<h3>Create / edit</h3>
|
| 83 |
+
<p class="muted small">Adjust instructions, tools, and voice, then save your profile.</p>
|
| 84 |
+
</div>
|
| 85 |
+
<div class="row">
|
| 86 |
+
<label for="personality-name">Name</label>
|
| 87 |
+
<input id="personality-name" type="text" class="input-field" placeholder="my_profile" />
|
| 88 |
+
<label for="voice-select">Voice</label>
|
| 89 |
+
<select id="voice-select"></select>
|
| 90 |
+
</div>
|
| 91 |
+
<div class="row">
|
| 92 |
+
<label for="instructions-ta">Instructions</label>
|
| 93 |
+
<textarea id="instructions-ta" rows="8" placeholder="# Write your instructions here"></textarea>
|
| 94 |
+
</div>
|
| 95 |
+
<div class="row">
|
| 96 |
+
<label for="tools-ta">tools.txt</label>
|
| 97 |
+
<textarea id="tools-ta" rows="6" placeholder="# tools enabled for this profile"></textarea>
|
| 98 |
+
</div>
|
| 99 |
+
<div class="row">
|
| 100 |
+
<label for="tools-available">Available tools</label>
|
| 101 |
+
<div id="tools-available" class="checkbox-grid"></div>
|
| 102 |
+
</div>
|
| 103 |
+
<div class="row row-save">
|
| 104 |
+
<label></label>
|
| 105 |
+
<div class="actions">
|
| 106 |
+
<button id="save-personality">Save</button>
|
| 107 |
+
</div>
|
| 108 |
+
</div>
|
| 109 |
+
</div>
|
| 110 |
+
<p id="personality-status" class="status"></p>
|
| 111 |
+
</div>
|
| 112 |
+
</div>
|
| 113 |
+
|
| 114 |
+
<script src="/static/main.js"></script>
|
| 115 |
+
</body>
|
| 116 |
+
</html>
|
src/reachy_mini_conversation_app/static/main.js
ADDED
|
@@ -0,0 +1,496 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
async function fetchStatus() {
|
| 2 |
+
try {
|
| 3 |
+
const url = new URL("/status", window.location.origin);
|
| 4 |
+
url.searchParams.set("_", Date.now().toString());
|
| 5 |
+
const resp = await fetchWithTimeout(url, {}, 2000);
|
| 6 |
+
if (!resp.ok) throw new Error("status error");
|
| 7 |
+
return await resp.json();
|
| 8 |
+
} catch (e) {
|
| 9 |
+
return { has_key: false, error: true };
|
| 10 |
+
}
|
| 11 |
+
}
|
| 12 |
+
|
| 13 |
+
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
|
| 14 |
+
|
| 15 |
+
async function fetchWithTimeout(url, options = {}, timeoutMs = 2000) {
|
| 16 |
+
const controller = new AbortController();
|
| 17 |
+
const id = setTimeout(() => controller.abort(), timeoutMs);
|
| 18 |
+
try {
|
| 19 |
+
return await fetch(url, { ...options, signal: controller.signal });
|
| 20 |
+
} finally {
|
| 21 |
+
clearTimeout(id);
|
| 22 |
+
}
|
| 23 |
+
}
|
| 24 |
+
|
| 25 |
+
async function waitForStatus(timeoutMs = 15000) {
|
| 26 |
+
const deadline = Date.now() + timeoutMs;
|
| 27 |
+
while (true) {
|
| 28 |
+
try {
|
| 29 |
+
const url = new URL("/status", window.location.origin);
|
| 30 |
+
url.searchParams.set("_", Date.now().toString());
|
| 31 |
+
const resp = await fetchWithTimeout(url, {}, 2000);
|
| 32 |
+
if (resp.ok) return await resp.json();
|
| 33 |
+
} catch (e) {}
|
| 34 |
+
if (Date.now() >= deadline) return null;
|
| 35 |
+
await sleep(500);
|
| 36 |
+
}
|
| 37 |
+
}
|
| 38 |
+
|
| 39 |
+
async function waitForPersonalityData(timeoutMs = 15000) {
|
| 40 |
+
const loadingText = document.querySelector("#loading p");
|
| 41 |
+
let attempts = 0;
|
| 42 |
+
const deadline = Date.now() + timeoutMs;
|
| 43 |
+
while (true) {
|
| 44 |
+
attempts += 1;
|
| 45 |
+
try {
|
| 46 |
+
const url = new URL("/personalities", window.location.origin);
|
| 47 |
+
url.searchParams.set("_", Date.now().toString());
|
| 48 |
+
const resp = await fetchWithTimeout(url, {}, 2000);
|
| 49 |
+
if (resp.ok) return await resp.json();
|
| 50 |
+
} catch (e) {}
|
| 51 |
+
|
| 52 |
+
if (loadingText) {
|
| 53 |
+
loadingText.textContent = attempts > 8 ? "Starting backend…" : "Loading…";
|
| 54 |
+
}
|
| 55 |
+
if (Date.now() >= deadline) return null;
|
| 56 |
+
await sleep(500);
|
| 57 |
+
}
|
| 58 |
+
}
|
| 59 |
+
|
| 60 |
+
async function validateKey(key) {
|
| 61 |
+
const body = { openai_api_key: key };
|
| 62 |
+
const resp = await fetch("/validate_api_key", {
|
| 63 |
+
method: "POST",
|
| 64 |
+
headers: { "Content-Type": "application/json" },
|
| 65 |
+
body: JSON.stringify(body),
|
| 66 |
+
});
|
| 67 |
+
const data = await resp.json().catch(() => ({}));
|
| 68 |
+
if (!resp.ok) {
|
| 69 |
+
throw new Error(data.error || "validation_failed");
|
| 70 |
+
}
|
| 71 |
+
return data;
|
| 72 |
+
}
|
| 73 |
+
|
| 74 |
+
async function saveKey(key) {
|
| 75 |
+
const body = { openai_api_key: key };
|
| 76 |
+
const resp = await fetch("/openai_api_key", {
|
| 77 |
+
method: "POST",
|
| 78 |
+
headers: { "Content-Type": "application/json" },
|
| 79 |
+
body: JSON.stringify(body),
|
| 80 |
+
});
|
| 81 |
+
if (!resp.ok) {
|
| 82 |
+
const data = await resp.json().catch(() => ({}));
|
| 83 |
+
throw new Error(data.error || "save_failed");
|
| 84 |
+
}
|
| 85 |
+
return await resp.json();
|
| 86 |
+
}
|
| 87 |
+
|
| 88 |
+
// ---------- Personalities API ----------
|
| 89 |
+
async function getPersonalities() {
|
| 90 |
+
const url = new URL("/personalities", window.location.origin);
|
| 91 |
+
url.searchParams.set("_", Date.now().toString());
|
| 92 |
+
const resp = await fetchWithTimeout(url, {}, 2000);
|
| 93 |
+
if (!resp.ok) throw new Error("list_failed");
|
| 94 |
+
return await resp.json();
|
| 95 |
+
}
|
| 96 |
+
|
| 97 |
+
async function loadPersonality(name) {
|
| 98 |
+
const url = new URL("/personalities/load", window.location.origin);
|
| 99 |
+
url.searchParams.set("name", name);
|
| 100 |
+
url.searchParams.set("_", Date.now().toString());
|
| 101 |
+
const resp = await fetchWithTimeout(url, {}, 3000);
|
| 102 |
+
if (!resp.ok) throw new Error("load_failed");
|
| 103 |
+
return await resp.json();
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
async function savePersonality(payload) {
|
| 107 |
+
// Try JSON POST first
|
| 108 |
+
const saveUrl = new URL("/personalities/save", window.location.origin);
|
| 109 |
+
saveUrl.searchParams.set("_", Date.now().toString());
|
| 110 |
+
let resp = await fetchWithTimeout(saveUrl, {
|
| 111 |
+
method: "POST",
|
| 112 |
+
headers: { "Content-Type": "application/json" },
|
| 113 |
+
body: JSON.stringify(payload),
|
| 114 |
+
}, 5000);
|
| 115 |
+
if (resp.ok) return await resp.json();
|
| 116 |
+
|
| 117 |
+
// Fallback to form-encoded POST
|
| 118 |
+
try {
|
| 119 |
+
const form = new URLSearchParams();
|
| 120 |
+
form.set("name", payload.name || "");
|
| 121 |
+
form.set("instructions", payload.instructions || "");
|
| 122 |
+
form.set("tools_text", payload.tools_text || "");
|
| 123 |
+
form.set("voice", payload.voice || "cedar");
|
| 124 |
+
const url = new URL("/personalities/save_raw", window.location.origin);
|
| 125 |
+
url.searchParams.set("_", Date.now().toString());
|
| 126 |
+
resp = await fetchWithTimeout(url, {
|
| 127 |
+
method: "POST",
|
| 128 |
+
headers: { "Content-Type": "application/x-www-form-urlencoded" },
|
| 129 |
+
body: form.toString(),
|
| 130 |
+
}, 5000);
|
| 131 |
+
if (resp.ok) return await resp.json();
|
| 132 |
+
} catch {}
|
| 133 |
+
|
| 134 |
+
// Fallback to GET (query params)
|
| 135 |
+
try {
|
| 136 |
+
const url = new URL("/personalities/save_raw", window.location.origin);
|
| 137 |
+
url.searchParams.set("name", payload.name || "");
|
| 138 |
+
url.searchParams.set("instructions", payload.instructions || "");
|
| 139 |
+
url.searchParams.set("tools_text", payload.tools_text || "");
|
| 140 |
+
url.searchParams.set("voice", payload.voice || "cedar");
|
| 141 |
+
url.searchParams.set("_", Date.now().toString());
|
| 142 |
+
resp = await fetchWithTimeout(url, { method: "GET" }, 5000);
|
| 143 |
+
if (resp.ok) return await resp.json();
|
| 144 |
+
} catch {}
|
| 145 |
+
|
| 146 |
+
const data = await resp.json().catch(() => ({}));
|
| 147 |
+
throw new Error(data.error || "save_failed");
|
| 148 |
+
}
|
| 149 |
+
|
| 150 |
+
async function applyPersonality(name, { persist = false } = {}) {
|
| 151 |
+
// Send as query param to avoid any body parsing issues on the server
|
| 152 |
+
const url = new URL("/personalities/apply", window.location.origin);
|
| 153 |
+
url.searchParams.set("name", name || "");
|
| 154 |
+
if (persist) {
|
| 155 |
+
url.searchParams.set("persist", "1");
|
| 156 |
+
}
|
| 157 |
+
url.searchParams.set("_", Date.now().toString());
|
| 158 |
+
const resp = await fetchWithTimeout(url, { method: "POST" }, 5000);
|
| 159 |
+
if (!resp.ok) {
|
| 160 |
+
const data = await resp.json().catch(() => ({}));
|
| 161 |
+
throw new Error(data.error || "apply_failed");
|
| 162 |
+
}
|
| 163 |
+
return await resp.json();
|
| 164 |
+
}
|
| 165 |
+
|
| 166 |
+
async function getVoices() {
|
| 167 |
+
try {
|
| 168 |
+
const url = new URL("/voices", window.location.origin);
|
| 169 |
+
url.searchParams.set("_", Date.now().toString());
|
| 170 |
+
const resp = await fetchWithTimeout(url, {}, 3000);
|
| 171 |
+
if (!resp.ok) throw new Error("voices_failed");
|
| 172 |
+
return await resp.json();
|
| 173 |
+
} catch (e) {
|
| 174 |
+
return ["cedar"];
|
| 175 |
+
}
|
| 176 |
+
}
|
| 177 |
+
|
| 178 |
+
function show(el, flag) {
|
| 179 |
+
el.classList.toggle("hidden", !flag);
|
| 180 |
+
}
|
| 181 |
+
|
| 182 |
+
async function init() {
|
| 183 |
+
const loading = document.getElementById("loading");
|
| 184 |
+
show(loading, true);
|
| 185 |
+
const statusEl = document.getElementById("status");
|
| 186 |
+
const formPanel = document.getElementById("form-panel");
|
| 187 |
+
const configuredPanel = document.getElementById("configured");
|
| 188 |
+
const personalityPanel = document.getElementById("personality-panel");
|
| 189 |
+
const saveBtn = document.getElementById("save-btn");
|
| 190 |
+
const changeKeyBtn = document.getElementById("change-key-btn");
|
| 191 |
+
const input = document.getElementById("api-key");
|
| 192 |
+
|
| 193 |
+
// Personality elements
|
| 194 |
+
const pSelect = document.getElementById("personality-select");
|
| 195 |
+
const pApply = document.getElementById("apply-personality");
|
| 196 |
+
const pPersist = document.getElementById("persist-personality");
|
| 197 |
+
const pNew = document.getElementById("new-personality");
|
| 198 |
+
const pSave = document.getElementById("save-personality");
|
| 199 |
+
const pStartupLabel = document.getElementById("startup-label");
|
| 200 |
+
const pName = document.getElementById("personality-name");
|
| 201 |
+
const pInstr = document.getElementById("instructions-ta");
|
| 202 |
+
const pTools = document.getElementById("tools-ta");
|
| 203 |
+
const pStatus = document.getElementById("personality-status");
|
| 204 |
+
const pVoice = document.getElementById("voice-select");
|
| 205 |
+
const pAvail = document.getElementById("tools-available");
|
| 206 |
+
|
| 207 |
+
const AUTO_WITH = {
|
| 208 |
+
dance: ["stop_dance"],
|
| 209 |
+
play_emotion: ["stop_emotion"],
|
| 210 |
+
};
|
| 211 |
+
|
| 212 |
+
statusEl.textContent = "Checking configuration...";
|
| 213 |
+
show(formPanel, false);
|
| 214 |
+
show(configuredPanel, false);
|
| 215 |
+
show(personalityPanel, false);
|
| 216 |
+
|
| 217 |
+
const st = (await waitForStatus()) || { has_key: false };
|
| 218 |
+
if (st.has_key) {
|
| 219 |
+
statusEl.textContent = "";
|
| 220 |
+
show(configuredPanel, true);
|
| 221 |
+
}
|
| 222 |
+
|
| 223 |
+
// Handler for "Change API key" button
|
| 224 |
+
changeKeyBtn.addEventListener("click", () => {
|
| 225 |
+
show(configuredPanel, false);
|
| 226 |
+
show(formPanel, true);
|
| 227 |
+
input.value = "";
|
| 228 |
+
statusEl.textContent = "";
|
| 229 |
+
statusEl.className = "status";
|
| 230 |
+
});
|
| 231 |
+
|
| 232 |
+
// Remove error styling when user starts typing
|
| 233 |
+
input.addEventListener("input", () => {
|
| 234 |
+
input.classList.remove("error");
|
| 235 |
+
});
|
| 236 |
+
|
| 237 |
+
saveBtn.addEventListener("click", async () => {
|
| 238 |
+
const key = input.value.trim();
|
| 239 |
+
if (!key) {
|
| 240 |
+
statusEl.textContent = "Please enter a valid key.";
|
| 241 |
+
statusEl.className = "status warn";
|
| 242 |
+
input.classList.add("error");
|
| 243 |
+
return;
|
| 244 |
+
}
|
| 245 |
+
statusEl.textContent = "Validating API key...";
|
| 246 |
+
statusEl.className = "status";
|
| 247 |
+
input.classList.remove("error");
|
| 248 |
+
try {
|
| 249 |
+
// First validate the key
|
| 250 |
+
const validation = await validateKey(key);
|
| 251 |
+
if (!validation.valid) {
|
| 252 |
+
statusEl.textContent = "Invalid API key. Please check your key and try again.";
|
| 253 |
+
statusEl.className = "status error";
|
| 254 |
+
input.classList.add("error");
|
| 255 |
+
return;
|
| 256 |
+
}
|
| 257 |
+
|
| 258 |
+
// If valid, save it
|
| 259 |
+
statusEl.textContent = "Key valid! Saving...";
|
| 260 |
+
statusEl.className = "status ok";
|
| 261 |
+
await saveKey(key);
|
| 262 |
+
statusEl.textContent = "Saved. Reloading…";
|
| 263 |
+
statusEl.className = "status ok";
|
| 264 |
+
window.location.reload();
|
| 265 |
+
} catch (e) {
|
| 266 |
+
input.classList.add("error");
|
| 267 |
+
if (e.message === "invalid_api_key") {
|
| 268 |
+
statusEl.textContent = "Invalid API key. Please check your key and try again.";
|
| 269 |
+
} else {
|
| 270 |
+
statusEl.textContent = "Failed to validate/save key. Please try again.";
|
| 271 |
+
}
|
| 272 |
+
statusEl.className = "status error";
|
| 273 |
+
}
|
| 274 |
+
});
|
| 275 |
+
|
| 276 |
+
if (!st.has_key) {
|
| 277 |
+
statusEl.textContent = "";
|
| 278 |
+
show(formPanel, true);
|
| 279 |
+
show(loading, false);
|
| 280 |
+
return;
|
| 281 |
+
}
|
| 282 |
+
|
| 283 |
+
// Wait until backend routes are ready before rendering personalities UI
|
| 284 |
+
const list = (await waitForPersonalityData()) || { choices: [] };
|
| 285 |
+
statusEl.textContent = "";
|
| 286 |
+
show(formPanel, false);
|
| 287 |
+
if (!list.choices.length) {
|
| 288 |
+
statusEl.textContent = "Personality endpoints not ready yet. Retry shortly.";
|
| 289 |
+
statusEl.className = "status warn";
|
| 290 |
+
show(loading, false);
|
| 291 |
+
return;
|
| 292 |
+
}
|
| 293 |
+
|
| 294 |
+
// Initialize personalities UI
|
| 295 |
+
try {
|
| 296 |
+
const choices = Array.isArray(list.choices) ? list.choices : [];
|
| 297 |
+
const DEFAULT_OPTION = choices[0] || "(built-in default)";
|
| 298 |
+
const startupChoice = choices.includes(list.startup) ? list.startup : DEFAULT_OPTION;
|
| 299 |
+
const currentChoice = choices.includes(list.current) ? list.current : startupChoice;
|
| 300 |
+
|
| 301 |
+
function setStartupLabel(name) {
|
| 302 |
+
const display = name && name !== DEFAULT_OPTION ? name : "Built-in default";
|
| 303 |
+
pStartupLabel.textContent = `Launch on start: ${display}`;
|
| 304 |
+
}
|
| 305 |
+
|
| 306 |
+
// Populate select
|
| 307 |
+
pSelect.innerHTML = "";
|
| 308 |
+
for (const n of choices) {
|
| 309 |
+
const opt = document.createElement("option");
|
| 310 |
+
opt.value = n;
|
| 311 |
+
opt.textContent = n;
|
| 312 |
+
pSelect.appendChild(opt);
|
| 313 |
+
}
|
| 314 |
+
if (choices.length) {
|
| 315 |
+
const preferred = choices.includes(startupChoice) ? startupChoice : currentChoice;
|
| 316 |
+
pSelect.value = preferred;
|
| 317 |
+
}
|
| 318 |
+
const voices = await getVoices();
|
| 319 |
+
pVoice.innerHTML = "";
|
| 320 |
+
for (const v of voices) {
|
| 321 |
+
const opt = document.createElement("option");
|
| 322 |
+
opt.value = v;
|
| 323 |
+
opt.textContent = v;
|
| 324 |
+
pVoice.appendChild(opt);
|
| 325 |
+
}
|
| 326 |
+
setStartupLabel(startupChoice);
|
| 327 |
+
|
| 328 |
+
function renderToolCheckboxes(available, enabled) {
|
| 329 |
+
pAvail.innerHTML = "";
|
| 330 |
+
const enabledSet = new Set(enabled);
|
| 331 |
+
for (const t of available) {
|
| 332 |
+
const wrap = document.createElement("div");
|
| 333 |
+
wrap.className = "chk";
|
| 334 |
+
const id = `tool-${t}`;
|
| 335 |
+
const cb = document.createElement("input");
|
| 336 |
+
cb.type = "checkbox";
|
| 337 |
+
cb.id = id;
|
| 338 |
+
cb.value = t;
|
| 339 |
+
cb.checked = enabledSet.has(t);
|
| 340 |
+
const lab = document.createElement("label");
|
| 341 |
+
lab.htmlFor = id;
|
| 342 |
+
lab.textContent = t;
|
| 343 |
+
wrap.appendChild(cb);
|
| 344 |
+
wrap.appendChild(lab);
|
| 345 |
+
pAvail.appendChild(wrap);
|
| 346 |
+
}
|
| 347 |
+
}
|
| 348 |
+
|
| 349 |
+
function getSelectedTools() {
|
| 350 |
+
const selected = new Set();
|
| 351 |
+
pAvail.querySelectorAll('input[type="checkbox"]').forEach((el) => {
|
| 352 |
+
if (el.checked) selected.add(el.value);
|
| 353 |
+
});
|
| 354 |
+
// Auto-include dependencies
|
| 355 |
+
for (const [main, deps] of Object.entries(AUTO_WITH)) {
|
| 356 |
+
if (selected.has(main)) {
|
| 357 |
+
for (const d of deps) selected.add(d);
|
| 358 |
+
}
|
| 359 |
+
}
|
| 360 |
+
return Array.from(selected);
|
| 361 |
+
}
|
| 362 |
+
|
| 363 |
+
function syncToolsTextarea() {
|
| 364 |
+
const selected = getSelectedTools();
|
| 365 |
+
const comments = pTools.value
|
| 366 |
+
.split("\n")
|
| 367 |
+
.filter((ln) => ln.trim().startsWith("#"));
|
| 368 |
+
const body = selected.join("\n");
|
| 369 |
+
pTools.value = (comments.join("\n") + (comments.length ? "\n" : "") + body).trim() + "\n";
|
| 370 |
+
}
|
| 371 |
+
|
| 372 |
+
function attachToolHandlers() {
|
| 373 |
+
pAvail.addEventListener("change", (ev) => {
|
| 374 |
+
const target = ev.target;
|
| 375 |
+
if (!(target instanceof HTMLInputElement) || target.type !== "checkbox") return;
|
| 376 |
+
const name = target.value;
|
| 377 |
+
// If a main tool toggled, propagate to deps
|
| 378 |
+
if (AUTO_WITH[name]) {
|
| 379 |
+
for (const dep of AUTO_WITH[name]) {
|
| 380 |
+
const depEl = pAvail.querySelector(`input[value="${dep}"]`);
|
| 381 |
+
if (depEl) depEl.checked = target.checked || depEl.checked;
|
| 382 |
+
}
|
| 383 |
+
}
|
| 384 |
+
syncToolsTextarea();
|
| 385 |
+
});
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
async function loadSelected() {
|
| 389 |
+
const selected = pSelect.value;
|
| 390 |
+
const data = await loadPersonality(selected);
|
| 391 |
+
pInstr.value = data.instructions || "";
|
| 392 |
+
pTools.value = data.tools_text || "";
|
| 393 |
+
pVoice.value = data.voice || "cedar";
|
| 394 |
+
// Available tools as checkboxes
|
| 395 |
+
renderToolCheckboxes(data.available_tools, data.enabled_tools);
|
| 396 |
+
attachToolHandlers();
|
| 397 |
+
// Default name field to last segment of selection
|
| 398 |
+
const idx = selected.lastIndexOf("/");
|
| 399 |
+
pName.value = idx >= 0 ? selected.slice(idx + 1) : "";
|
| 400 |
+
pStatus.textContent = `Loaded ${selected}`;
|
| 401 |
+
pStatus.className = "status";
|
| 402 |
+
}
|
| 403 |
+
|
| 404 |
+
pSelect.addEventListener("change", loadSelected);
|
| 405 |
+
await loadSelected();
|
| 406 |
+
show(personalityPanel, true);
|
| 407 |
+
|
| 408 |
+
// pAvail change handler registered in attachToolHandlers()
|
| 409 |
+
|
| 410 |
+
pApply.addEventListener("click", async () => {
|
| 411 |
+
pStatus.textContent = "Applying...";
|
| 412 |
+
pStatus.className = "status";
|
| 413 |
+
try {
|
| 414 |
+
const res = await applyPersonality(pSelect.value);
|
| 415 |
+
if (res.startup) setStartupLabel(res.startup);
|
| 416 |
+
pStatus.textContent = res.status || "Applied.";
|
| 417 |
+
pStatus.className = "status ok";
|
| 418 |
+
} catch (e) {
|
| 419 |
+
pStatus.textContent = `Failed to apply${e.message ? ": " + e.message : ""}`;
|
| 420 |
+
pStatus.className = "status error";
|
| 421 |
+
}
|
| 422 |
+
});
|
| 423 |
+
|
| 424 |
+
pPersist.addEventListener("click", async () => {
|
| 425 |
+
pStatus.textContent = "Saving for startup...";
|
| 426 |
+
pStatus.className = "status";
|
| 427 |
+
try {
|
| 428 |
+
const res = await applyPersonality(pSelect.value, { persist: true });
|
| 429 |
+
if (res.startup) setStartupLabel(res.startup);
|
| 430 |
+
pStatus.textContent = res.status || "Saved for startup.";
|
| 431 |
+
pStatus.className = "status ok";
|
| 432 |
+
} catch (e) {
|
| 433 |
+
pStatus.textContent = `Failed to persist${e.message ? ": " + e.message : ""}`;
|
| 434 |
+
pStatus.className = "status error";
|
| 435 |
+
}
|
| 436 |
+
});
|
| 437 |
+
|
| 438 |
+
pNew.addEventListener("click", () => {
|
| 439 |
+
pName.value = "";
|
| 440 |
+
pInstr.value = "# Write your instructions here\n# e.g., Keep responses concise and friendly.";
|
| 441 |
+
pTools.value = "# tools enabled for this profile\n";
|
| 442 |
+
// Keep available tools list, clear selection
|
| 443 |
+
pAvail.querySelectorAll('input[type="checkbox"]').forEach((el) => {
|
| 444 |
+
el.checked = false;
|
| 445 |
+
});
|
| 446 |
+
pVoice.value = "cedar";
|
| 447 |
+
pStatus.textContent = "Fill fields and click Save.";
|
| 448 |
+
pStatus.className = "status";
|
| 449 |
+
});
|
| 450 |
+
|
| 451 |
+
pSave.addEventListener("click", async () => {
|
| 452 |
+
const name = (pName.value || "").trim();
|
| 453 |
+
if (!name) {
|
| 454 |
+
pStatus.textContent = "Enter a valid name.";
|
| 455 |
+
pStatus.className = "status warn";
|
| 456 |
+
return;
|
| 457 |
+
}
|
| 458 |
+
pStatus.textContent = "Saving...";
|
| 459 |
+
pStatus.className = "status";
|
| 460 |
+
try {
|
| 461 |
+
// Ensure tools.txt reflects checkbox selection and auto-includes
|
| 462 |
+
syncToolsTextarea();
|
| 463 |
+
const res = await savePersonality({
|
| 464 |
+
name,
|
| 465 |
+
instructions: pInstr.value || "",
|
| 466 |
+
tools_text: pTools.value || "",
|
| 467 |
+
voice: pVoice.value || "cedar",
|
| 468 |
+
});
|
| 469 |
+
// Refresh select choices
|
| 470 |
+
pSelect.innerHTML = "";
|
| 471 |
+
for (const n of res.choices) {
|
| 472 |
+
const opt = document.createElement("option");
|
| 473 |
+
opt.value = n;
|
| 474 |
+
opt.textContent = n;
|
| 475 |
+
if (n === res.value) opt.selected = true;
|
| 476 |
+
pSelect.appendChild(opt);
|
| 477 |
+
}
|
| 478 |
+
pStatus.textContent = "Saved.";
|
| 479 |
+
pStatus.className = "status ok";
|
| 480 |
+
// Auto-apply
|
| 481 |
+
try { await applyPersonality(pSelect.value); } catch {}
|
| 482 |
+
} catch (e) {
|
| 483 |
+
pStatus.textContent = "Failed to save.";
|
| 484 |
+
pStatus.className = "status error";
|
| 485 |
+
}
|
| 486 |
+
});
|
| 487 |
+
} catch (e) {
|
| 488 |
+
statusEl.textContent = "UI failed to load. Please refresh.";
|
| 489 |
+
statusEl.className = "status warn";
|
| 490 |
+
} finally {
|
| 491 |
+
// Hide loading when initial setup is done (regardless of key presence)
|
| 492 |
+
show(loading, false);
|
| 493 |
+
}
|
| 494 |
+
}
|
| 495 |
+
|
| 496 |
+
window.addEventListener("DOMContentLoaded", init);
|
src/reachy_mini_conversation_app/static/style.css
ADDED
|
@@ -0,0 +1,317 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
:root {
|
| 2 |
+
--bg: #060b1a;
|
| 3 |
+
--bg-2: #071023;
|
| 4 |
+
--panel: rgba(11, 18, 36, 0.8);
|
| 5 |
+
--border: rgba(255, 255, 255, 0.08);
|
| 6 |
+
--text: #eaf2ff;
|
| 7 |
+
--muted: #9fb6d7;
|
| 8 |
+
--ok: #4ce0b3;
|
| 9 |
+
--warn: #ffb547;
|
| 10 |
+
--error: #ff5c70;
|
| 11 |
+
--accent: #45c4ff;
|
| 12 |
+
--accent-2: #5ef0c1;
|
| 13 |
+
--shadow: 0 20px 70px rgba(0, 0, 0, 0.45);
|
| 14 |
+
}
|
| 15 |
+
|
| 16 |
+
* { box-sizing: border-box; }
|
| 17 |
+
body {
|
| 18 |
+
margin: 0;
|
| 19 |
+
min-height: 100vh;
|
| 20 |
+
font-family: "Space Grotesk", "Inter", "Segoe UI", sans-serif;
|
| 21 |
+
background: radial-gradient(circle at 20% 20%, rgba(69, 196, 255, 0.16), transparent 35%),
|
| 22 |
+
radial-gradient(circle at 80% 0%, rgba(94, 240, 193, 0.16), transparent 32%),
|
| 23 |
+
linear-gradient(135deg, var(--bg), var(--bg-2));
|
| 24 |
+
color: var(--text);
|
| 25 |
+
}
|
| 26 |
+
|
| 27 |
+
.ambient {
|
| 28 |
+
position: fixed;
|
| 29 |
+
inset: 0;
|
| 30 |
+
background: radial-gradient(circle at 30% 60%, rgba(255, 255, 255, 0.05), transparent 35%),
|
| 31 |
+
radial-gradient(circle at 75% 30%, rgba(69, 196, 255, 0.08), transparent 32%);
|
| 32 |
+
filter: blur(60px);
|
| 33 |
+
z-index: 0;
|
| 34 |
+
pointer-events: none;
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
/* Loading overlay */
|
| 38 |
+
.loading {
|
| 39 |
+
position: fixed;
|
| 40 |
+
inset: 0;
|
| 41 |
+
background: rgba(5, 10, 24, 0.92);
|
| 42 |
+
backdrop-filter: blur(4px);
|
| 43 |
+
display: flex;
|
| 44 |
+
flex-direction: column;
|
| 45 |
+
align-items: center;
|
| 46 |
+
justify-content: center;
|
| 47 |
+
z-index: 9999;
|
| 48 |
+
}
|
| 49 |
+
.loading .spinner {
|
| 50 |
+
width: 46px;
|
| 51 |
+
height: 46px;
|
| 52 |
+
border: 4px solid rgba(255,255,255,0.15);
|
| 53 |
+
border-top-color: var(--accent);
|
| 54 |
+
border-radius: 50%;
|
| 55 |
+
animation: spin 1s linear infinite;
|
| 56 |
+
margin-bottom: 12px;
|
| 57 |
+
}
|
| 58 |
+
.loading p { color: var(--muted); margin: 0; letter-spacing: 0.4px; }
|
| 59 |
+
@keyframes spin { to { transform: rotate(360deg); } }
|
| 60 |
+
|
| 61 |
+
.container {
|
| 62 |
+
position: relative;
|
| 63 |
+
max-width: 960px;
|
| 64 |
+
margin: 7vh auto;
|
| 65 |
+
padding: 0 24px 40px;
|
| 66 |
+
z-index: 1;
|
| 67 |
+
}
|
| 68 |
+
|
| 69 |
+
.hero {
|
| 70 |
+
margin-bottom: 24px;
|
| 71 |
+
}
|
| 72 |
+
.hero h1 {
|
| 73 |
+
margin: 6px 0 6px;
|
| 74 |
+
font-size: 32px;
|
| 75 |
+
letter-spacing: -0.4px;
|
| 76 |
+
}
|
| 77 |
+
.subtitle {
|
| 78 |
+
margin: 0;
|
| 79 |
+
color: var(--muted);
|
| 80 |
+
line-height: 1.5;
|
| 81 |
+
}
|
| 82 |
+
.pill {
|
| 83 |
+
display: inline-flex;
|
| 84 |
+
align-items: center;
|
| 85 |
+
gap: 6px;
|
| 86 |
+
padding: 6px 12px;
|
| 87 |
+
border-radius: 999px;
|
| 88 |
+
background: rgba(94, 240, 193, 0.1);
|
| 89 |
+
color: var(--accent-2);
|
| 90 |
+
font-size: 12px;
|
| 91 |
+
letter-spacing: 0.3px;
|
| 92 |
+
border: 1px solid rgba(94, 240, 193, 0.25);
|
| 93 |
+
}
|
| 94 |
+
|
| 95 |
+
.panel {
|
| 96 |
+
background: var(--panel);
|
| 97 |
+
border: 1px solid var(--border);
|
| 98 |
+
border-radius: 14px;
|
| 99 |
+
padding: 18px 18px 16px;
|
| 100 |
+
box-shadow: var(--shadow);
|
| 101 |
+
backdrop-filter: blur(10px);
|
| 102 |
+
margin-top: 16px;
|
| 103 |
+
}
|
| 104 |
+
.panel-heading {
|
| 105 |
+
display: flex;
|
| 106 |
+
align-items: center;
|
| 107 |
+
justify-content: space-between;
|
| 108 |
+
gap: 12px;
|
| 109 |
+
margin-bottom: 8px;
|
| 110 |
+
}
|
| 111 |
+
.panel-heading h2 {
|
| 112 |
+
margin: 2px 0;
|
| 113 |
+
font-size: 22px;
|
| 114 |
+
}
|
| 115 |
+
.eyebrow {
|
| 116 |
+
margin: 0;
|
| 117 |
+
text-transform: uppercase;
|
| 118 |
+
font-size: 11px;
|
| 119 |
+
letter-spacing: 0.5px;
|
| 120 |
+
color: var(--muted);
|
| 121 |
+
}
|
| 122 |
+
.muted { color: var(--muted); }
|
| 123 |
+
.chip {
|
| 124 |
+
display: inline-flex;
|
| 125 |
+
align-items: center;
|
| 126 |
+
padding: 6px 10px;
|
| 127 |
+
border-radius: 999px;
|
| 128 |
+
font-size: 12px;
|
| 129 |
+
color: var(--text);
|
| 130 |
+
background: rgba(255, 255, 255, 0.08);
|
| 131 |
+
border: 1px solid var(--border);
|
| 132 |
+
}
|
| 133 |
+
.chip-ok {
|
| 134 |
+
background: rgba(76, 224, 179, 0.15);
|
| 135 |
+
color: var(--ok);
|
| 136 |
+
border-color: rgba(76, 224, 179, 0.4);
|
| 137 |
+
}
|
| 138 |
+
|
| 139 |
+
.hidden { display: none; }
|
| 140 |
+
label {
|
| 141 |
+
display: block;
|
| 142 |
+
margin: 8px 0 6px;
|
| 143 |
+
font-size: 13px;
|
| 144 |
+
color: var(--muted);
|
| 145 |
+
letter-spacing: 0.2px;
|
| 146 |
+
}
|
| 147 |
+
input[type="password"],
|
| 148 |
+
input[type="text"],
|
| 149 |
+
select,
|
| 150 |
+
textarea {
|
| 151 |
+
width: 100%;
|
| 152 |
+
padding: 12px 14px;
|
| 153 |
+
border: 1px solid var(--border);
|
| 154 |
+
border-radius: 10px;
|
| 155 |
+
background: rgba(255, 255, 255, 0.04);
|
| 156 |
+
color: var(--text);
|
| 157 |
+
transition: border 0.15s ease, box-shadow 0.15s ease;
|
| 158 |
+
}
|
| 159 |
+
input:focus,
|
| 160 |
+
select:focus,
|
| 161 |
+
textarea:focus {
|
| 162 |
+
border-color: rgba(94, 240, 193, 0.7);
|
| 163 |
+
outline: none;
|
| 164 |
+
box-shadow: 0 0 0 3px rgba(94, 240, 193, 0.15);
|
| 165 |
+
}
|
| 166 |
+
input.error {
|
| 167 |
+
border-color: var(--error);
|
| 168 |
+
box-shadow: 0 0 0 3px rgba(255, 92, 112, 0.15);
|
| 169 |
+
}
|
| 170 |
+
select option {
|
| 171 |
+
background: #0b152a;
|
| 172 |
+
color: var(--text);
|
| 173 |
+
}
|
| 174 |
+
textarea { resize: vertical; }
|
| 175 |
+
|
| 176 |
+
button {
|
| 177 |
+
display: inline-flex;
|
| 178 |
+
align-items: center;
|
| 179 |
+
justify-content: center;
|
| 180 |
+
margin-top: 12px;
|
| 181 |
+
padding: 11px 16px;
|
| 182 |
+
border: none;
|
| 183 |
+
border-radius: 10px;
|
| 184 |
+
background: linear-gradient(120deg, var(--accent), var(--accent-2));
|
| 185 |
+
color: #031022;
|
| 186 |
+
cursor: pointer;
|
| 187 |
+
font-weight: 600;
|
| 188 |
+
letter-spacing: 0.2px;
|
| 189 |
+
box-shadow: 0 14px 40px rgba(69, 196, 255, 0.25);
|
| 190 |
+
transition: transform 0.12s ease, filter 0.12s ease, box-shadow 0.12s ease;
|
| 191 |
+
}
|
| 192 |
+
button:hover { filter: brightness(1.06); transform: translateY(-1px); }
|
| 193 |
+
button:active { transform: translateY(0); }
|
| 194 |
+
button.ghost {
|
| 195 |
+
background: rgba(255, 255, 255, 0.05);
|
| 196 |
+
color: var(--text);
|
| 197 |
+
box-shadow: none;
|
| 198 |
+
border: 1px solid var(--border);
|
| 199 |
+
}
|
| 200 |
+
button.ghost:hover { border-color: rgba(94, 240, 193, 0.4); }
|
| 201 |
+
.actions {
|
| 202 |
+
display: flex;
|
| 203 |
+
align-items: center;
|
| 204 |
+
gap: 12px;
|
| 205 |
+
flex-wrap: wrap;
|
| 206 |
+
}
|
| 207 |
+
.status {
|
| 208 |
+
margin: 0;
|
| 209 |
+
color: var(--muted);
|
| 210 |
+
font-size: 13px;
|
| 211 |
+
}
|
| 212 |
+
.status.ok { color: var(--ok); }
|
| 213 |
+
.status.warn { color: var(--warn); }
|
| 214 |
+
.status.error { color: var(--error); }
|
| 215 |
+
|
| 216 |
+
/* Personality layout */
|
| 217 |
+
.row {
|
| 218 |
+
display: grid;
|
| 219 |
+
grid-template-columns: 160px 1fr;
|
| 220 |
+
gap: 12px 18px;
|
| 221 |
+
align-items: center;
|
| 222 |
+
margin-top: 12px;
|
| 223 |
+
}
|
| 224 |
+
.row > label { margin: 0; }
|
| 225 |
+
.row > button { margin: 0; }
|
| 226 |
+
|
| 227 |
+
/* First row: controls inline */
|
| 228 |
+
#personality-panel .row-top {
|
| 229 |
+
grid-template-columns: 160px 1fr auto auto auto;
|
| 230 |
+
}
|
| 231 |
+
|
| 232 |
+
#tools-available {
|
| 233 |
+
max-height: 240px;
|
| 234 |
+
overflow: auto;
|
| 235 |
+
padding: 10px;
|
| 236 |
+
border: 1px solid var(--border);
|
| 237 |
+
border-radius: 10px;
|
| 238 |
+
background: rgba(255, 255, 255, 0.03);
|
| 239 |
+
}
|
| 240 |
+
|
| 241 |
+
/* Checkbox grid for tools */
|
| 242 |
+
.checkbox-grid {
|
| 243 |
+
display: grid;
|
| 244 |
+
grid-template-columns: repeat(auto-fill, minmax(170px, 1fr));
|
| 245 |
+
gap: 10px 14px;
|
| 246 |
+
}
|
| 247 |
+
.startup-row {
|
| 248 |
+
display: flex;
|
| 249 |
+
align-items: center;
|
| 250 |
+
gap: 10px;
|
| 251 |
+
flex-wrap: wrap;
|
| 252 |
+
}
|
| 253 |
+
.row-save .actions {
|
| 254 |
+
justify-content: flex-start;
|
| 255 |
+
}
|
| 256 |
+
.input-field {
|
| 257 |
+
width: 100%;
|
| 258 |
+
padding: 12px 14px;
|
| 259 |
+
border: 1px solid var(--border);
|
| 260 |
+
border-radius: 10px;
|
| 261 |
+
background: rgba(255, 255, 255, 0.05);
|
| 262 |
+
color: var(--text);
|
| 263 |
+
transition: border 0.15s ease, box-shadow 0.15s ease;
|
| 264 |
+
}
|
| 265 |
+
.input-field:focus {
|
| 266 |
+
border-color: rgba(94, 240, 193, 0.7);
|
| 267 |
+
outline: none;
|
| 268 |
+
box-shadow: 0 0 0 3px rgba(94, 240, 193, 0.15);
|
| 269 |
+
}
|
| 270 |
+
.section {
|
| 271 |
+
border: 1px solid var(--border);
|
| 272 |
+
border-radius: 12px;
|
| 273 |
+
padding: 12px 14px;
|
| 274 |
+
margin-top: 14px;
|
| 275 |
+
background: rgba(255, 255, 255, 0.02);
|
| 276 |
+
}
|
| 277 |
+
.section-heading {
|
| 278 |
+
display: flex;
|
| 279 |
+
align-items: baseline;
|
| 280 |
+
gap: 10px;
|
| 281 |
+
justify-content: space-between;
|
| 282 |
+
}
|
| 283 |
+
.section-heading h3 {
|
| 284 |
+
margin: 6px 0;
|
| 285 |
+
font-size: 16px;
|
| 286 |
+
letter-spacing: -0.1px;
|
| 287 |
+
}
|
| 288 |
+
.section-heading .small {
|
| 289 |
+
margin: 0;
|
| 290 |
+
font-size: 12px;
|
| 291 |
+
}
|
| 292 |
+
.checkbox-grid .chk {
|
| 293 |
+
display: flex;
|
| 294 |
+
align-items: center;
|
| 295 |
+
gap: 8px;
|
| 296 |
+
padding: 8px 10px;
|
| 297 |
+
border-radius: 10px;
|
| 298 |
+
background: rgba(255, 255, 255, 0.02);
|
| 299 |
+
border: 1px solid transparent;
|
| 300 |
+
transition: border 0.12s ease, background 0.12s ease;
|
| 301 |
+
}
|
| 302 |
+
.checkbox-grid .chk:hover { border-color: rgba(94, 240, 193, 0.3); background: rgba(255, 255, 255, 0.04); }
|
| 303 |
+
.checkbox-grid input[type="checkbox"] {
|
| 304 |
+
width: 16px; height: 16px;
|
| 305 |
+
accent-color: var(--accent);
|
| 306 |
+
}
|
| 307 |
+
.checkbox-grid label {
|
| 308 |
+
margin: 0; font-size: 13px; color: var(--text);
|
| 309 |
+
}
|
| 310 |
+
|
| 311 |
+
@media (max-width: 760px) {
|
| 312 |
+
.hero h1 { font-size: 26px; }
|
| 313 |
+
.row { grid-template-columns: 1fr; }
|
| 314 |
+
#personality-panel .row:first-of-type { grid-template-columns: 1fr; }
|
| 315 |
+
button { width: 100%; justify-content: center; }
|
| 316 |
+
.actions { flex-direction: column; align-items: flex-start; }
|
| 317 |
+
}
|
src/reachy_mini_conversation_app/utils.py
CHANGED
|
@@ -7,7 +7,7 @@ from reachy_mini import ReachyMini
|
|
| 7 |
from reachy_mini_conversation_app.camera_worker import CameraWorker
|
| 8 |
|
| 9 |
|
| 10 |
-
def parse_args() -> argparse.Namespace:
|
| 11 |
"""Parse command line arguments."""
|
| 12 |
parser = argparse.ArgumentParser("Reachy Mini Conversation App")
|
| 13 |
parser.add_argument(
|
|
@@ -37,7 +37,7 @@ def parse_args() -> argparse.Namespace:
|
|
| 37 |
action="store_true",
|
| 38 |
help="Use when conversation app is running on the same device as Reachy Mini daemon",
|
| 39 |
)
|
| 40 |
-
return parser.
|
| 41 |
|
| 42 |
|
| 43 |
def handle_vision_stuff(args: argparse.Namespace, current_robot: ReachyMini) -> Tuple[CameraWorker | None, Any, Any]:
|
|
|
|
| 7 |
from reachy_mini_conversation_app.camera_worker import CameraWorker
|
| 8 |
|
| 9 |
|
| 10 |
+
def parse_args() -> Tuple[argparse.Namespace, list]: # type: ignore
|
| 11 |
"""Parse command line arguments."""
|
| 12 |
parser = argparse.ArgumentParser("Reachy Mini Conversation App")
|
| 13 |
parser.add_argument(
|
|
|
|
| 37 |
action="store_true",
|
| 38 |
help="Use when conversation app is running on the same device as Reachy Mini daemon",
|
| 39 |
)
|
| 40 |
+
return parser.parse_known_args()
|
| 41 |
|
| 42 |
|
| 43 |
def handle_vision_stuff(args: argparse.Namespace, current_robot: ReachyMini) -> Tuple[CameraWorker | None, Any, Any]:
|
src/reachy_mini_conversation_app/vision/processors.py
CHANGED
|
@@ -61,7 +61,7 @@ class VisionProcessor:
|
|
| 61 |
"""Load model and processor onto the selected device."""
|
| 62 |
try:
|
| 63 |
logger.info(f"Loading SmolVLM2 model on {self.device} (HF_HOME={config.HF_HOME})")
|
| 64 |
-
self.processor = AutoProcessor.from_pretrained(self.model_path) # type: ignore
|
| 65 |
|
| 66 |
# Select dtype depending on device
|
| 67 |
if self.device == "cuda":
|
|
@@ -78,7 +78,7 @@ class VisionProcessor:
|
|
| 78 |
model_kwargs["_attn_implementation"] = "flash_attention_2"
|
| 79 |
|
| 80 |
# Load model weights
|
| 81 |
-
self.model = AutoModelForImageTextToText.from_pretrained(self.model_path, **model_kwargs).to(self.device) # type: ignore
|
| 82 |
|
| 83 |
if self.model is not None:
|
| 84 |
self.model.eval()
|
|
@@ -247,7 +247,8 @@ class VisionManager:
|
|
| 247 |
frame = self.camera.get_latest_frame()
|
| 248 |
if frame is not None:
|
| 249 |
description = self.processor.process_image(
|
| 250 |
-
frame,
|
|
|
|
| 251 |
)
|
| 252 |
|
| 253 |
# Only update if we got a valid response
|
|
|
|
| 61 |
"""Load model and processor onto the selected device."""
|
| 62 |
try:
|
| 63 |
logger.info(f"Loading SmolVLM2 model on {self.device} (HF_HOME={config.HF_HOME})")
|
| 64 |
+
self.processor = AutoProcessor.from_pretrained(self.model_path) # type: ignore
|
| 65 |
|
| 66 |
# Select dtype depending on device
|
| 67 |
if self.device == "cuda":
|
|
|
|
| 78 |
model_kwargs["_attn_implementation"] = "flash_attention_2"
|
| 79 |
|
| 80 |
# Load model weights
|
| 81 |
+
self.model = AutoModelForImageTextToText.from_pretrained(self.model_path, **model_kwargs).to(self.device) # type: ignore
|
| 82 |
|
| 83 |
if self.model is not None:
|
| 84 |
self.model.eval()
|
|
|
|
| 247 |
frame = self.camera.get_latest_frame()
|
| 248 |
if frame is not None:
|
| 249 |
description = self.processor.process_image(
|
| 250 |
+
frame,
|
| 251 |
+
"Briefly describe what you see in one sentence.",
|
| 252 |
)
|
| 253 |
|
| 254 |
# Only update if we got a valid response
|
src/reachy_mini_conversation_app/vision/yolo_head_tracker.py
CHANGED
|
@@ -8,7 +8,7 @@ from numpy.typing import NDArray
|
|
| 8 |
|
| 9 |
try:
|
| 10 |
from supervision import Detections
|
| 11 |
-
from ultralytics import YOLO # type: ignore
|
| 12 |
except ImportError as e:
|
| 13 |
raise ImportError(
|
| 14 |
"To use YOLO head tracker, please install the extra dependencies: pip install '.[yolo_vision]'",
|
|
|
|
| 8 |
|
| 9 |
try:
|
| 10 |
from supervision import Detections
|
| 11 |
+
from ultralytics import YOLO # type: ignore
|
| 12 |
except ImportError as e:
|
| 13 |
raise ImportError(
|
| 14 |
"To use YOLO head tracker, please install the extra dependencies: pip install '.[yolo_vision]'",
|
style.css
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
:root {
|
| 2 |
+
--bg: #060c1d;
|
| 3 |
+
--panel: #0c172b;
|
| 4 |
+
--glass: rgba(17, 27, 48, 0.7);
|
| 5 |
+
--card: rgba(255, 255, 255, 0.04);
|
| 6 |
+
--accent: #7af5c4;
|
| 7 |
+
--accent-2: #f6c452;
|
| 8 |
+
--text: #e8edf7;
|
| 9 |
+
--muted: #9fb3ce;
|
| 10 |
+
--border: rgba(255, 255, 255, 0.08);
|
| 11 |
+
--shadow: 0 25px 70px rgba(0, 0, 0, 0.45);
|
| 12 |
+
font-family: "Space Grotesk", "Manrope", system-ui, -apple-system, sans-serif;
|
| 13 |
+
}
|
| 14 |
+
|
| 15 |
+
* {
|
| 16 |
+
margin: 0;
|
| 17 |
+
padding: 0;
|
| 18 |
+
box-sizing: border-box;
|
| 19 |
+
}
|
| 20 |
+
|
| 21 |
+
body {
|
| 22 |
+
background: radial-gradient(circle at 20% 20%, rgba(122, 245, 196, 0.12), transparent 30%),
|
| 23 |
+
radial-gradient(circle at 80% 0%, rgba(246, 196, 82, 0.14), transparent 32%),
|
| 24 |
+
radial-gradient(circle at 50% 70%, rgba(124, 142, 255, 0.1), transparent 30%),
|
| 25 |
+
var(--bg);
|
| 26 |
+
color: var(--text);
|
| 27 |
+
min-height: 100vh;
|
| 28 |
+
line-height: 1.6;
|
| 29 |
+
padding-bottom: 3rem;
|
| 30 |
+
}
|
| 31 |
+
|
| 32 |
+
a {
|
| 33 |
+
color: inherit;
|
| 34 |
+
text-decoration: none;
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
.hero {
|
| 38 |
+
padding: 3.5rem clamp(1.5rem, 3vw, 3rem) 2.5rem;
|
| 39 |
+
position: relative;
|
| 40 |
+
overflow: hidden;
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
.hero::after {
|
| 44 |
+
content: "";
|
| 45 |
+
position: absolute;
|
| 46 |
+
inset: 0;
|
| 47 |
+
background: linear-gradient(120deg, rgba(122, 245, 196, 0.12), rgba(246, 196, 82, 0.08), transparent);
|
| 48 |
+
pointer-events: none;
|
| 49 |
+
}
|
| 50 |
+
|
| 51 |
+
.topline {
|
| 52 |
+
display: flex;
|
| 53 |
+
align-items: center;
|
| 54 |
+
justify-content: space-between;
|
| 55 |
+
max-width: 1200px;
|
| 56 |
+
margin: 0 auto 2rem;
|
| 57 |
+
position: relative;
|
| 58 |
+
z-index: 2;
|
| 59 |
+
}
|
| 60 |
+
|
| 61 |
+
.brand {
|
| 62 |
+
display: flex;
|
| 63 |
+
align-items: center;
|
| 64 |
+
gap: 0.5rem;
|
| 65 |
+
font-weight: 700;
|
| 66 |
+
letter-spacing: 0.5px;
|
| 67 |
+
color: var(--text);
|
| 68 |
+
}
|
| 69 |
+
|
| 70 |
+
.logo {
|
| 71 |
+
display: inline-flex;
|
| 72 |
+
align-items: center;
|
| 73 |
+
justify-content: center;
|
| 74 |
+
width: 2.2rem;
|
| 75 |
+
height: 2.2rem;
|
| 76 |
+
border-radius: 10px;
|
| 77 |
+
background: linear-gradient(145deg, rgba(122, 245, 196, 0.15), rgba(124, 142, 255, 0.15));
|
| 78 |
+
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.25);
|
| 79 |
+
}
|
| 80 |
+
|
| 81 |
+
.brand-name {
|
| 82 |
+
font-size: 1.1rem;
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
.pill {
|
| 86 |
+
background: rgba(255, 255, 255, 0.06);
|
| 87 |
+
border: 1px solid var(--border);
|
| 88 |
+
padding: 0.6rem 1rem;
|
| 89 |
+
border-radius: 999px;
|
| 90 |
+
color: var(--muted);
|
| 91 |
+
font-size: 0.9rem;
|
| 92 |
+
box-shadow: 0 12px 30px rgba(0, 0, 0, 0.2);
|
| 93 |
+
}
|
| 94 |
+
|
| 95 |
+
.hero-grid {
|
| 96 |
+
display: grid;
|
| 97 |
+
grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
|
| 98 |
+
gap: clamp(1.5rem, 2.5vw, 2.5rem);
|
| 99 |
+
max-width: 1200px;
|
| 100 |
+
margin: 0 auto;
|
| 101 |
+
position: relative;
|
| 102 |
+
z-index: 2;
|
| 103 |
+
align-items: center;
|
| 104 |
+
}
|
| 105 |
+
|
| 106 |
+
.hero-copy h1 {
|
| 107 |
+
font-size: clamp(2.6rem, 4vw, 3.6rem);
|
| 108 |
+
margin-bottom: 1rem;
|
| 109 |
+
line-height: 1.1;
|
| 110 |
+
letter-spacing: -0.5px;
|
| 111 |
+
}
|
| 112 |
+
|
| 113 |
+
.eyebrow {
|
| 114 |
+
display: inline-flex;
|
| 115 |
+
align-items: center;
|
| 116 |
+
gap: 0.5rem;
|
| 117 |
+
text-transform: uppercase;
|
| 118 |
+
letter-spacing: 1px;
|
| 119 |
+
font-size: 0.8rem;
|
| 120 |
+
color: var(--muted);
|
| 121 |
+
margin-bottom: 0.75rem;
|
| 122 |
+
}
|
| 123 |
+
|
| 124 |
+
.eyebrow::before {
|
| 125 |
+
content: "";
|
| 126 |
+
display: inline-block;
|
| 127 |
+
width: 24px;
|
| 128 |
+
height: 2px;
|
| 129 |
+
background: linear-gradient(90deg, var(--accent), var(--accent-2));
|
| 130 |
+
border-radius: 999px;
|
| 131 |
+
}
|
| 132 |
+
|
| 133 |
+
.lede {
|
| 134 |
+
font-size: 1.1rem;
|
| 135 |
+
color: var(--muted);
|
| 136 |
+
max-width: 620px;
|
| 137 |
+
}
|
| 138 |
+
|
| 139 |
+
.hero-actions {
|
| 140 |
+
display: flex;
|
| 141 |
+
gap: 1rem;
|
| 142 |
+
align-items: center;
|
| 143 |
+
margin: 1.6rem 0 1.2rem;
|
| 144 |
+
flex-wrap: wrap;
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
.btn {
|
| 148 |
+
display: inline-flex;
|
| 149 |
+
align-items: center;
|
| 150 |
+
justify-content: center;
|
| 151 |
+
gap: 0.6rem;
|
| 152 |
+
padding: 0.85rem 1.4rem;
|
| 153 |
+
border-radius: 12px;
|
| 154 |
+
font-weight: 700;
|
| 155 |
+
border: 1px solid transparent;
|
| 156 |
+
cursor: pointer;
|
| 157 |
+
transition: transform 0.2s ease, box-shadow 0.2s ease, background 0.2s ease, border-color 0.2s ease;
|
| 158 |
+
}
|
| 159 |
+
|
| 160 |
+
.btn.primary {
|
| 161 |
+
background: linear-gradient(135deg, #7af5c4, #7c8eff);
|
| 162 |
+
color: #0a0f1f;
|
| 163 |
+
box-shadow: 0 15px 30px rgba(122, 245, 196, 0.25);
|
| 164 |
+
}
|
| 165 |
+
|
| 166 |
+
.btn.primary:hover {
|
| 167 |
+
transform: translateY(-2px);
|
| 168 |
+
box-shadow: 0 25px 45px rgba(122, 245, 196, 0.35);
|
| 169 |
+
}
|
| 170 |
+
|
| 171 |
+
.btn.ghost {
|
| 172 |
+
background: rgba(255, 255, 255, 0.05);
|
| 173 |
+
border-color: var(--border);
|
| 174 |
+
color: var(--text);
|
| 175 |
+
}
|
| 176 |
+
|
| 177 |
+
.btn.ghost:hover {
|
| 178 |
+
border-color: rgba(255, 255, 255, 0.3);
|
| 179 |
+
transform: translateY(-2px);
|
| 180 |
+
}
|
| 181 |
+
|
| 182 |
+
.btn.wide {
|
| 183 |
+
width: 100%;
|
| 184 |
+
justify-content: center;
|
| 185 |
+
}
|
| 186 |
+
|
| 187 |
+
.hero-badges {
|
| 188 |
+
display: flex;
|
| 189 |
+
flex-wrap: wrap;
|
| 190 |
+
gap: 0.6rem;
|
| 191 |
+
color: var(--muted);
|
| 192 |
+
font-size: 0.9rem;
|
| 193 |
+
}
|
| 194 |
+
|
| 195 |
+
.hero-badges span {
|
| 196 |
+
padding: 0.5rem 0.8rem;
|
| 197 |
+
border-radius: 10px;
|
| 198 |
+
border: 1px solid var(--border);
|
| 199 |
+
background: rgba(255, 255, 255, 0.04);
|
| 200 |
+
}
|
| 201 |
+
|
| 202 |
+
.hero-visual .glass-card {
|
| 203 |
+
background: rgba(255, 255, 255, 0.03);
|
| 204 |
+
border: 1px solid var(--border);
|
| 205 |
+
border-radius: 18px;
|
| 206 |
+
padding: 1.2rem;
|
| 207 |
+
box-shadow: var(--shadow);
|
| 208 |
+
backdrop-filter: blur(10px);
|
| 209 |
+
}
|
| 210 |
+
|
| 211 |
+
.hero-gif {
|
| 212 |
+
width: 100%;
|
| 213 |
+
display: block;
|
| 214 |
+
border-radius: 14px;
|
| 215 |
+
border: 1px solid var(--border);
|
| 216 |
+
box-shadow: 0 12px 35px rgba(0, 0, 0, 0.35);
|
| 217 |
+
}
|
| 218 |
+
|
| 219 |
+
.caption {
|
| 220 |
+
margin-top: 0.75rem;
|
| 221 |
+
color: var(--muted);
|
| 222 |
+
font-size: 0.95rem;
|
| 223 |
+
}
|
| 224 |
+
|
| 225 |
+
.section {
|
| 226 |
+
max-width: 1200px;
|
| 227 |
+
margin: 0 auto;
|
| 228 |
+
padding: clamp(2rem, 4vw, 3.5rem) clamp(1.5rem, 3vw, 3rem);
|
| 229 |
+
}
|
| 230 |
+
|
| 231 |
+
.section-header {
|
| 232 |
+
text-align: center;
|
| 233 |
+
max-width: 780px;
|
| 234 |
+
margin: 0 auto 2rem;
|
| 235 |
+
}
|
| 236 |
+
|
| 237 |
+
.section-header h2 {
|
| 238 |
+
font-size: clamp(2rem, 3vw, 2.6rem);
|
| 239 |
+
margin-bottom: 0.5rem;
|
| 240 |
+
}
|
| 241 |
+
|
| 242 |
+
.intro {
|
| 243 |
+
color: var(--muted);
|
| 244 |
+
font-size: 1.05rem;
|
| 245 |
+
}
|
| 246 |
+
|
| 247 |
+
.feature-grid {
|
| 248 |
+
display: grid;
|
| 249 |
+
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
|
| 250 |
+
gap: 1rem;
|
| 251 |
+
}
|
| 252 |
+
|
| 253 |
+
.feature-card {
|
| 254 |
+
background: rgba(255, 255, 255, 0.03);
|
| 255 |
+
border: 1px solid var(--border);
|
| 256 |
+
border-radius: 16px;
|
| 257 |
+
padding: 1.25rem;
|
| 258 |
+
box-shadow: 0 10px 30px rgba(0, 0, 0, 0.2);
|
| 259 |
+
transition: transform 0.2s ease, border-color 0.2s ease, box-shadow 0.2s ease;
|
| 260 |
+
}
|
| 261 |
+
|
| 262 |
+
.feature-card:hover {
|
| 263 |
+
transform: translateY(-4px);
|
| 264 |
+
border-color: rgba(122, 245, 196, 0.3);
|
| 265 |
+
box-shadow: 0 18px 40px rgba(0, 0, 0, 0.3);
|
| 266 |
+
}
|
| 267 |
+
|
| 268 |
+
.feature-card .icon {
|
| 269 |
+
width: 48px;
|
| 270 |
+
height: 48px;
|
| 271 |
+
border-radius: 12px;
|
| 272 |
+
display: grid;
|
| 273 |
+
place-items: center;
|
| 274 |
+
background: rgba(122, 245, 196, 0.14);
|
| 275 |
+
margin-bottom: 0.8rem;
|
| 276 |
+
font-size: 1.4rem;
|
| 277 |
+
}
|
| 278 |
+
|
| 279 |
+
.feature-card h3 {
|
| 280 |
+
margin-bottom: 0.35rem;
|
| 281 |
+
}
|
| 282 |
+
|
| 283 |
+
.feature-card p {
|
| 284 |
+
color: var(--muted);
|
| 285 |
+
}
|
| 286 |
+
|
| 287 |
+
.story {
|
| 288 |
+
padding-top: 1rem;
|
| 289 |
+
}
|
| 290 |
+
|
| 291 |
+
.story-grid {
|
| 292 |
+
display: grid;
|
| 293 |
+
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
|
| 294 |
+
gap: 1rem;
|
| 295 |
+
}
|
| 296 |
+
|
| 297 |
+
.story-card {
|
| 298 |
+
background: rgba(255, 255, 255, 0.03);
|
| 299 |
+
border: 1px solid var(--border);
|
| 300 |
+
border-radius: 18px;
|
| 301 |
+
padding: 1.5rem;
|
| 302 |
+
box-shadow: var(--shadow);
|
| 303 |
+
}
|
| 304 |
+
|
| 305 |
+
.story-card.secondary {
|
| 306 |
+
background: linear-gradient(145deg, rgba(124, 142, 255, 0.08), rgba(122, 245, 196, 0.06));
|
| 307 |
+
}
|
| 308 |
+
|
| 309 |
+
.story-card h3 {
|
| 310 |
+
margin-bottom: 0.8rem;
|
| 311 |
+
}
|
| 312 |
+
|
| 313 |
+
.story-list {
|
| 314 |
+
list-style: none;
|
| 315 |
+
display: grid;
|
| 316 |
+
gap: 0.7rem;
|
| 317 |
+
color: var(--muted);
|
| 318 |
+
font-size: 0.98rem;
|
| 319 |
+
}
|
| 320 |
+
|
| 321 |
+
.story-list li {
|
| 322 |
+
display: flex;
|
| 323 |
+
gap: 0.7rem;
|
| 324 |
+
align-items: flex-start;
|
| 325 |
+
}
|
| 326 |
+
|
| 327 |
+
.story-text {
|
| 328 |
+
color: var(--muted);
|
| 329 |
+
line-height: 1.7;
|
| 330 |
+
margin-bottom: 1rem;
|
| 331 |
+
}
|
| 332 |
+
|
| 333 |
+
.chips {
|
| 334 |
+
display: flex;
|
| 335 |
+
flex-wrap: wrap;
|
| 336 |
+
gap: 0.5rem;
|
| 337 |
+
}
|
| 338 |
+
|
| 339 |
+
.chip {
|
| 340 |
+
padding: 0.45rem 0.8rem;
|
| 341 |
+
border-radius: 12px;
|
| 342 |
+
background: rgba(0, 0, 0, 0.2);
|
| 343 |
+
border: 1px solid var(--border);
|
| 344 |
+
color: var(--text);
|
| 345 |
+
font-size: 0.9rem;
|
| 346 |
+
}
|
| 347 |
+
|
| 348 |
+
.footer {
|
| 349 |
+
text-align: center;
|
| 350 |
+
color: var(--muted);
|
| 351 |
+
padding: 2rem 1.5rem 0;
|
| 352 |
+
}
|
| 353 |
+
|
| 354 |
+
.footer a {
|
| 355 |
+
color: var(--text);
|
| 356 |
+
border-bottom: 1px solid transparent;
|
| 357 |
+
}
|
| 358 |
+
|
| 359 |
+
.footer a:hover {
|
| 360 |
+
border-color: rgba(255, 255, 255, 0.5);
|
| 361 |
+
}
|
| 362 |
+
|
| 363 |
+
@media (max-width: 768px) {
|
| 364 |
+
.hero {
|
| 365 |
+
padding-top: 2.5rem;
|
| 366 |
+
}
|
| 367 |
+
|
| 368 |
+
.topline {
|
| 369 |
+
flex-direction: column;
|
| 370 |
+
gap: 0.8rem;
|
| 371 |
+
align-items: flex-start;
|
| 372 |
+
}
|
| 373 |
+
|
| 374 |
+
.hero-actions {
|
| 375 |
+
width: 100%;
|
| 376 |
+
}
|
| 377 |
+
|
| 378 |
+
.btn {
|
| 379 |
+
width: 100%;
|
| 380 |
+
justify-content: center;
|
| 381 |
+
}
|
| 382 |
+
|
| 383 |
+
.hero-badges {
|
| 384 |
+
gap: 0.4rem;
|
| 385 |
+
}
|
| 386 |
+
}
|