Adolfo Reyna
80 lines
3.0 KiB
Markdown
80 lines
3.0 KiB
Markdown
# Desktop Companion Client for AI Agents
|
|
|
|
This is a cross-platform (Linux, macOS, Windows) client that allows your computer to be used directly by AI Agents as a physical-like communication and display companion.
|
|
|
|
It implements the Model Context Protocol (MCP) and exposes the host machine's actual hardware:
|
|
- **Audio Output**: Plays pure tones and WAV files on your actual speakers.
|
|
- **Audio Input**: Records voice messages from your microphone.
|
|
- **Battery Status**: Exposes your laptop's real battery voltage and percentage.
|
|
- **System Telemetry**: Reads host metrics (uptime, CPU loading).
|
|
- **Dedicated Canvas Window**: Renders images and text drawn by the agent.
|
|
- **Video Streaming**: Renders incoming 1-bit monochrome video streams (from sources streaming to TCP port 8081 or UDP port 8082).
|
|
- **UDP Discovery**: Responds to local discovery beacons.
|
|
|
|
## Requirements
|
|
|
|
- Python 3.10+
|
|
- **Pillow** (PIL) library (already installed in your environment)
|
|
- **Audio Playback**: Uses native system players (`afplay` on macOS, `aplay`/`paplay`/`pw-play` on Linux, `winsound` on Windows).
|
|
- **Audio Recording**:
|
|
- **Linux**: Standard `arecord` (pre-installed via `alsa-utils`).
|
|
- **macOS / Windows**: If `sounddevice` or `pyaudio` Python packages are installed, they will be used. Otherwise, it falls back to `sox`/`rec` if available, or generates a clean simulated wave format if no audio recorder is present.
|
|
|
|
## Running the Client
|
|
|
|
Start the client manually with:
|
|
```bash
|
|
python3 desktop_client/client.py
|
|
```
|
|
|
|
Options:
|
|
- `--port`: HTTP JSON-RPC port (default is `8080`)
|
|
- `--width`: Canvas width (default `480`)
|
|
- `--height`: Canvas height (default `320`)
|
|
- `--headless`: Run headlessly without Pygame GUI window (ideal for remote SSH servers)
|
|
|
|
### Keyboard Controls (GUI Mode)
|
|
- **`D`**: Switch the screen canvas back to the local **Host Telemetry Dashboard**.
|
|
- **`S`**: Save a screenshot of the companion window.
|
|
|
|
### Hidden Mode Behavior
|
|
To avoid cluttering your screen, the GUI window starts **hidden** on boot. It runs silently in the background and only pops to the foreground when a message (drawing or video stream) is sent by the AI Agent.
|
|
|
|
---
|
|
|
|
## Running as a macOS Service (LaunchAgent)
|
|
|
|
To run the companion client persistently as a background service that launches automatically on login, use the provided scripts:
|
|
|
|
1. **Install and Start the Service**:
|
|
```bash
|
|
./desktop_client/setup_service.sh
|
|
```
|
|
2. **Uninstall/Stop the Service**:
|
|
```bash
|
|
./desktop_client/uninstall_service.sh
|
|
```
|
|
|
|
Logs are captured dynamically at:
|
|
- Standard Out: `desktop_client/stdout.log`
|
|
- Standard Error: `desktop_client/stderr.log`
|
|
|
|
---
|
|
|
|
## Configuring Claude Desktop / Agent Host
|
|
|
|
To add this desktop client as an MCP tool provider, add the following to your `claude_desktop_config.json`:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"desktop-companion": {
|
|
"command": "/Users/adolforeyna/.pyenv/versions/3.10.12/bin/python3",
|
|
"args": [
|
|
"/Users/adolforeyna/Projects/MicroPython/test1/Screen/desktop_client/client.py"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|