# Future eye-control extension

Status: eye tracking is planned only. Version 0.5 provides shared dwell selection and finger-only hand control, without an action toolbar. Do not enable hand and eye inference together by default; only the selected provider should consume compute.

## Product scope

Optional gaze positioning and deliberate wink or dwell selection across the normal Windows desktop. It reuses the current standard-user input boundary; it does not gain access to security screens or elevated applications. First validate ordinary-webcam performance with intended users. Do not promise equivalence to a dedicated infrared tracker, or claim suitability for all accessibility needs.

## Integration boundary

Implement a `GazeProvider : IControlProvider` alongside `HandProvider`. Both accept timestamped `Frame` input and return a `ControlFrame` containing normalized `Action` values, confidence, state, and optional overlay points. Only `WindowsSink` emits system input; a provider cannot bypass `SafetyPolicy`.

The following remain shared: selected-display coordinates, physical-mouse override, explicit enabling, epoch rejection of in-flight results after pause/configuration changes, input freshness checks, button-release watchdog, shutdown, and diagnostic output. Switching providers must pause control, release held input, reset both providers, and require deliberate re-arming.

The eye provider will need a separate profile type for calibration and gaze-specific confidence/pose checks. Reuse `DwellController`, `DwellSettings`, `DwellFeedback`, action types, the filling-circle renderer, and practice state for selection. Do not copy the hand pose classifier or build a second dwell state machine. Gaze-specific tolerance tuning should be a typed setting on the shared controller, validated independently from hand defaults. Hand mode has no toolbar; a future gaze-specific method of choosing actions needs its own usability design and must not reintroduce a bar into hand mode. Extend the camera's requested resolution to benchmark 720p versus 480p and crop eyes after face localization. Preserve latest-frame scheduling.

## Stage 1 — prove usable gaze estimation

1. Benchmark a maintained, commercially usable face/eye + gaze pipeline. Face/iris landmarks alone are not a calibrated screen-gaze estimator. Check model/runtime provenance and license requirements before choosing dependencies.
2. Run one face detector intermittently and track face/eye crops between detections. Account for head pose. Measure CPU, memory, latency, and power on the intended device; do not assume eyes are cheaper to process than hands.
3. Use a 9-point calibration and **separate held-out validation targets**. Save a local calibration tied to camera, display geometry, resolution, and user profile. Never save eye video by default.
4. Report median and 95th-percentile pixel error, recovery after head movement, and failed targets. Detect calibration drift and request recalibration instead of silently steering the pointer incorrectly.
5. Pause on multiple faces or uncertainty rather than switching control to another person. Test glasses, varied lighting, camera positions, eye shapes, and comfortable user movement.

Go/no-go: proceed only if the intended users can consistently select agreed target sizes. An initial proposal is at least 95% success on 64px targets and under 150ms median motion-to-pointer latency on the target device; these are proposed acceptance criteria, not current measured capabilities. Small 16–24px desktop controls need a separate refinement mechanism.

## Stage 2 — intention without accidental clicks

- Start with the existing adjustable **dwell selection**, progress ring, movement cancellation, explicit modes, and repeat block. Add gaze eligibility and calibration checks before passing a sample to the shared controller. Looking around should not select items; validate intention and accidental activation with real users rather than assuming hand-mode defaults transfer unchanged.
- Add optional deliberate one-eye wink after collecting each user's comfortable baseline. Normal blinks must be ignored. Not everyone can wink; never make winking mandatory.
- On wink onset, latch the last stable target. Emit one click only after a valid completion/reopen sequence and a refractory interval. Test false clicks during normal reading, blinking, talking, and looking away.
- Do not initially use both-eye closure as a click: gaze is unavailable during closure. If later added, specify a separate intentional-occlusion state. In that state the shared router may release a button but must not move or click. After reopening, reacquire and validate the target before any click. Never label ordinary tracking loss as a successful gesture.
- Treat target-position age separately from camera-frame age. The current 250ms frame freshness rule must remain in force. Any longer-lived latched target needs an explicit expiry and tests.
- Keep drag and scroll as explicit modes in a small overlay rather than complicated eye gesture combinations. Exiting a mode, looking away, switching source, or losing tracking must release held input.

Suggested pilot gates: fewer than one false activation per 30 minutes of ordinary viewing and at least 95% intended activation success. Measure these with real users over repeated sessions, including users who cannot wink. Do not infer usability from a short developer demonstration.

## Stage 3 — accurate desktop selection

Implement an optional magnified target area or a coarse-to-fine selection overlay. Enter it by deliberate dwell/command, choose the finer location, then confirm. This should help with small desktop controls without claiming raw webcam gaze is pixel-accurate. Keep the real pointer still while refining the target.

Use a target overlay that does not steal application focus unnecessarily. Account for per-monitor scaling, negative coordinates, multiple resolutions, popups, and changed screen geometry. Any display change invalidates the calibration and pauses control.

## Stage 4 — pilot and packaging

Compare against hand mode and a conventional input method on the **same real tasks**, measuring completion, errors, fatigue/comfort as reported by users, setup time, and willingness to pay. If webcam accuracy fails the user's tasks, consider optional supported dedicated tracking hardware rather than weakening safety thresholds. Keep privacy local, provide an obvious camera indicator and pause control, and review commercial dependency terms before distribution.

## Required automated and user tests

- Ordinary blink, prolonged closure, wink, eye occlusion, gaze loss, head turn, face switching, glasses reflection.
- Drift, wrong display/camera, resolution change, bad calibration, validation failure, invalid/NaN predictions.
- Dwell start/cancel/complete/refractory; no repeated activation on the same fixation.
- Wink target stays fixed; a late or stale result cannot click after pause, physical input, or source switch.
- Lost tracking while dragging releases promptly; model crash, camera disconnect, lock/suspend, and exit leave no held buttons.
- Real-user small-target and extended-session tests. Synthetic eye landmarks cannot establish commercial usability.

## Reference products and research

- [Microsoft Windows Eye Control](https://support.microsoft.com/en-us/accessibility/windows/eye-control/eye-control-troubleshooting-guide): desktop eye input and supported tracking devices.
- [WebGazer](https://webgazer.cs.brown.edu/): webcam gaze estimation/calibration research reference. Its site states GPLv3 and no guaranteed ongoing updates; do not assume it is a drop-in dependency for a closed-source product.
- [MediaPipe Face Landmarker](https://developers.google.com/edge/mediapipe/solutions/vision/face_landmarker): face landmarks and expression output, not an out-of-the-box screen gaze model.
- [Beam Eye Tracker](https://beam.eyeware.tech/): existing webcam eye/head tracking product, useful as a comparison when validating differentiation.
