# AirDesk 0.9.2 · Windows prototype

Control the Windows pointer with one hand and an ordinary webcam. Point and pause to click: no pinching or finger contraction. AirDesk runs locally in a native C++ interface, without a browser, Python, .NET, account, or backend. This unsigned development prototype is not a validated assistive device or production workshop product.

This is a free experimental Windows binary. Original application source remains private; an open-source release has not been authorized. Bundled third-party components retain their own licences and notices.

## Start

1. Keep the entire extracted AirDesk folder together and open `AirDesk.exe`.
2. Click **Start control**. It opens the camera and enables control once tracking is ready; no second click is needed. Your saved camera/display choices are reused. To change them, open **Settings** first.
3. Hold up your index finger, with your thumb apart and the other fingers curled. Keep your palm toward the camera. Allow 1.5 seconds after using the physical mouse, which always takes priority.
4. Move your hand to point, then pause comfortably over an item. After a brief settling moment, the **circle fills for one second**, then clicks once. A moving fill and bold outer arc show progress. Completion turns the circle green, pulses it outward and shows a large white check. Natural hand shakes are tolerated and the cursor stays on the item while filling. A brief wobble just outside the hold area pauses progress rather than restarting it. Move clearly away before clicking again. Large movement, sustained departure, lowering your finger or losing tracking cancels.
5. Minimize AirDesk to use other apps. Only the click indicator follows the pointer. **There is no action toolbar.**

**Open your hand to rest**; point again to continue. No menu or mouse is needed for desktop gestures. **Stop control** stops both control and the camera. **Ctrl + Alt + Space** is an emergency pause that leaves the camera on; click **Start control** to resume after that shortcut. Camera and control always start off when you reopen the app.

## Actions

Use one finger to aim first. Then use the same hand for every action. These are the default assignments:

| Action | How to use it |
|---|---|
| Click | One finger: point, stop and let the circle fill |
| Right-click | Aim first, then hold up index + middle fingers and keep still |
| Scroll | Index + middle fingers: move up/down; lower the middle finger to finish |
| Double-click | Aim first, then hold up index + middle + ring fingers and keep still |
| Drag | Aim first, close your whole hand briefly to grab, then move the closed hand |
| Drop / rest | Open the whole hand; point again to continue |

Returning to one finger returns to pointing without immediately clicking into a just-opened menu. Holding still after a click will not repeat it; a check mark stays visible until you move away. For right/double-click, raising extra fingers preserves your prior aim. Two-finger movement starts scrolling and cannot also right-click when you stop; lower the middle finger first. A closed hand cannot start a drag unless you recently aimed with one finger.

### Choose your movements

The home page shows **Your gestures**: read-only pictures of your active movements and their actions. Shared poses appear once; for example, two fingers can show both right-click and scrolling. Pointing remains visible because it always aims the pointer.

For the same reference during desktop control, enable **Settings → General → Show live gesture reference**. It appears as a translucent strip on the selected screen when AirDesk is in the background. It cannot take focus or intercept clicks, and fades when your pointer approaches. It is hidden during pause and practice. This optional setting starts off; the home reference is always visible. Both views use the same images, captions and assignments.

To change a movement, open **Settings → Gestures**, choose an action, then one of its **three pictured alternatives**. **Turn off** disables that action; **Restore defaults** restores the original assignments. Changes save immediately. The selected picture has a teal border and check. Only this Settings tab edits assignments; the home guide is derived from the same saved profile.

Alongside the actions above, you can assign **Back, Forward, Copy, Paste, Play/pause and Mute**. These extra functions start off. Custom assignments use supported poses shown in the images; this does not train arbitrary new movements. Shortcut results depend on the active application and selection. Aim with one finger before changing to an assigned pose.

A held pose has one assignment. Choosing one already in use replaces its previous held function, with a notice explaining the change. Scrolling may share a pose with a held action because movement latches scrolling until the pose changes. Drag and scroll cannot share a pose. Open palm remains reserved for rest/drop and cannot be reassigned.

Opening your whole hand completes a drop. Losing tracking, an ambiguous pose, or pausing also releases a held drag for safety. This may **drop an object at its current location**; AirDesk cannot guarantee application-level undo. Test dragging with disposable items, not valuable files or machinery.

## Practice and guidance

Press **Practice** on the home page. Targets appear in the camera preview alongside hand landmarks. A guided sequence covers enabled click, right-click, double-click, drag/drop and scroll actions, followed by open-hand rest. Each step uses your saved pose and highlights its reference image; disabled actions are skipped. There is no separate selection menu, and optional keyboard shortcuts are not part of this lesson.

Practice shows every relevant enabled gesture plus an illustrated **Let go / rest** sign. During a drag, that release picture is highlighted and the destination circle says **Drop**. The learning targets are twice their previous diameter; drawing and hit detection share the same size. Their centres stay transparent over the camera until progress fills them, with a green check on completion. Only the completed target is shown during the short confirmation pause.

Aim at the target and follow the instruction. A confirmed hit advances to the next step after a short readable pause. Drag requires grabbing the target and dropping it at the destination; pickup alone cannot complete the step. The final step asks you to open your whole hand briefly. Mouse clicks remain separate and cannot complete lesson steps. **Finish practice** or **Esc** ends the session. Settings ends practice and stops the camera. Practice uses the same provider and dwell controller but never sends actions to Windows or starts desktop control afterward.

To let go, open your hand normally over the destination. The safety release remains immediate as the grab pose changes. Practice also accepts a fresh open palm within 0.7 seconds after that release, at the original release position, so a gradual opening does not lose a valid drop. Tracking loss, stale frames and longer gaps cancel this confirmation. If you release outside the target, point at the circle and try again; you are no longer holding it.

Practice now scores fresh actions when they arrive, independently of screen redraws. This fixes accepted clicks being lost during a slow redraw. Targets no longer require the upright hand to reach extreme camera edges, and modest jitter no longer resets every countdown frame.

The main page shows the preview, Start control, Practice and your active gesture pictures. There is no full-screen practice or separate gesture-guide page.

One live hint appears when needed:

- **Move your hand closer:** the detected hand is too small for reliable pointing.
- **Move your hand farther away:** the hand occupies too much of the image.
- **Keep your whole hand in view:** the hand reaches the image edge.
- **Show one hand to the camera:** there is no usable detection; AirDesk does not guess its distance.

These are image-size and framing heuristics, not measurements in centimetres. Selection pauses during a warning. Hints appear in the main preview or practice, or briefly near the bottom of the display when another app is in front. No permanent extra guidance panel is shown. Use even front lighting and an uncluttered background; gloves remain unverified.

Guidance waits for a new message to remain consistent for about 0.6 seconds and keeps the current message readable for at least 1.5 seconds. Actual click confirmation can appear immediately. This delay affects text only: unsafe framing, tracking loss, the emergency pause and physical-mouse priority still stop input immediately. Stale-input rejection remains 250 ms; a short capture hiccup no longer flashes a different status message.

## Settings

**Settings** has two tabs: **General** for camera, display, cursor feel, click delay and distance; **Gestures** for movement assignments. **Done** returns home. Changes save immediately in the same profile. Opening Settings stops the camera and control, so camera selection is available; Done does not restart either.

- **Cursor feel** replaces the numerical slider. **Steady — recommended** now smooths hand shakes more strongly; alternating shakes no longer boost its sensitivity. Deliberate travel still speeds up the pointer. **Balanced** reacts faster with less smoothing. **Direct — sensitive** follows every small movement, including shakes. These choices affect steadiness, not how much screen area you can reach. The app shows a short explanation beneath your choice.
- **Pause before click:** 1 second (recommended), 1.3 seconds (more time), or 0.7 seconds (faster), after a short settling moment. Deliberate motion cancels immediately; a pose-classifier flicker of less than 100 ms freezes progress without allowing a click on the invalid frame.
- **Display:** the camera box maps to the entire selected monitor, including edges and taskbar. **All displays** spans the virtual desktop. Irregular monitor arrangements may contain gaps; one display is more predictable. Restart after changing the Windows display arrangement.
- **Mirror movement:** mirrors preview and horizontal pointer mapping together.
- Scrolling is enabled or turned off through its assignment under Gestures; there is no duplicate scrolling checkbox.
- **Distance — Automatic (default):** adapts the working area to observed hand size while starting or resting. Open your hand briefly after moving noticeably nearer or farther away. Mapping stays fixed during pointing, countdowns and dragging to prevent jumps.
- **Distance — Fixed:** keeps the same working area. Only this mode shows **Set movement area**, an optional five-position setup (centre, left, right, up, down). Setup never controls Windows; cancellation or interrupted tracking preserves the previous saved area. There is no separate Comfortable reach section.

The white preview ring marks the detected fingertip; the teal ring marks its mapped pointer position. The preview is not a touch surface. The filling circle follows the actual pointer over other apps and cannot intercept clicks. Once the countdown starts, both the visible pointer and click location stay at the same anchor until you move deliberately.

New installs default to **Steady**, a **one-second fill** and **Automatic distance**. Version 0.9.1 retains previous preferences and gesture assignments. An older disabled scrolling checkbox becomes a disabled scroll assignment. Your previous movement area is retained for the selected camera and mirror direction; changing either invalidates it. A changed capture aspect ratio uses fallback mapping. Automatic adaptation is relearned, not stored as physical distance, and stays fixed while pointing and dragging.

Both distance modes accept reasonable nearer/farther variation while sufficient detail and the whole hand remain visible. You do not need one exact hand size. Automatic mode adapts cursor reach, not optical camera zoom: it cannot recover fingers outside the image or measure centimetres. The usable range depends on camera quality, lighting and hand visibility.

The camera prefers bounded extra detail only when a native mode offers approximately 30 fps at the original aspect ratio; incompatible modes fall back to the lighter capture format. The palm and hand models still process fixed 192px and 224px inputs. Framing now accepts more of a fully visible close hand and smaller hands with sufficient pixel detail. This is not optical auto-zoom, a distance measurement, or a guarantee that a hand outside the camera view can be recovered. Low confidence, insufficient detail, clipping and stale frames still stop input.

Starting practice pauses desktop control. **Stop camera** appears on the home page while previewing with control paused. During desktop control, **Stop control** releases both camera and input. Physical mouse activity suppresses gestures for 1.5 seconds and resets pending actions. Closing the app, locking/disconnecting Windows or suspending stops capture; unlocking does not restart it. Preferences live in `settings.ini` next to the executable; camera/control-on state is never saved.

## Requirements and limits

- Windows 10 version 2004 or later / Windows 11 **x64**, ordinary interactive desktop, camera available to desktop apps.
- ONNX Runtime needs the Microsoft Visual C++ v14 x64 runtime. If Windows reports missing MSVCP140/VCRUNTIME140 files, use [Microsoft's official runtime download](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist), not individual DLL download sites. Windows N needs Microsoft's Media Feature Pack. Do not disable security protections to run this prototype.
- Models/runtime are bundled. No network access is needed during use, no frames are recorded, and ONNX Runtime telemetry is disabled.
- Elevated apps, UAC/security screens, lock screens, protected games, and remote desktop forwarding are unsupported. AirDesk never elevates itself or bypasses protections.
- No native multitouch, gaze control, or built-in on-screen keyboard. Tiny controls, fast motion, camera buffering, occlusion, and difficult lighting remain limitations. Ordinary webcams do not sense actual touch.
- Tracking is capped at 30 fps and searches more slowly when no usable hand is present. Frames are coalesced, not queued. A 250 ms freshness check rejects stale input; a watchdog releases held buttons during a stalled pipeline. A complete OS freeze or forced termination cannot guarantee cleanup. Keep a physical mouse available.

## Architecture and development

`Camera → Hand provider → shared DwellController → ControlFrame / Action → SafetyPolicy → WindowsSink`

Practice consumes the same actions without the Windows sink. Mapping, gesture definitions, named settings, timing, and feedback have shared definitions. [Architecture](docs/ARCHITECTURE.md) describes these boundaries. A future gaze provider can reuse dwell selection and safety; [eye control is planned, not implemented](docs/EYE-EXTENSION.md).

No ChatGPT API is needed for this control loop. Small local vision models keep camera frames private and avoid network latency and recurring API costs. The hand illustrations were created with OpenAI's built-in image-generation tool; [provenance and prompts](assets/PROVENANCE.md). Assets are decoded and cached once, not regenerated during use. The legacy pinch panel in the source asset is not displayed by the app.

Build C++17 source with the pinned dependencies in `DEPENDENCIES.md`:

```powershell
./setup.ps1
./build.ps1
./build.ps1 -TestsOnly
```

Setup downloads the portable compiler/runtime into local folders; it does not change PATH or install system software. Existing dependencies can be supplied with `-Toolchain` and `-Runtime`. Pure tests never open a camera or send system input.

Diagnostics also never send mouse input:

```powershell
./build/AirDesk.exe --diagnostics report.txt
./build/AirDesk.exe --test-image path/to/hand.png report.txt
./build/AirDesk.exe --smoke-test report.txt
./build/AirDesk.exe --camera-test report.txt
```

Wait for the process to exit before reading the report. Only `--camera-test` opens the live webcam: it checks capture/stop over three cycles without recording frames. The smoke test checks startup, controls, gesture image decoding, and native overlay creation with camera/control off.

ONNX Runtime 1.22.0 (MIT and bundled notices) and OpenCV Zoo's MediaPipe-derived models (Apache 2.0) are bundled. See `licenses` and `DEPENDENCIES.md`. No commercial accuracy or performance guarantees are made.
