Part 1: Setup Interface

The setup interface is the first component of a visual interface in MyoGestic.

It is responsible for configuring the parameters, initializing the system, and managing the communication between MyoGestic and the visual interface.

Step 1: Create the setup UI

You will need an UI that will allow the user to start and stop the interface. This UI can be as simple as a button or as complex as you need.

Note

To start copy the UI files in myogestic > gui > widgets > visual_interfaces > ui and adapt them with the functionality you need.

You need to modify them with QT-Designer and convert them using UIC to a python file.

Step 2: Understand what is needed in the setup interface

Note

A setup interface is a class that inherits from

SetupInterfaceTemplate(main_window[, name, ui])

Base class for the setup interface of a visual interface.

Please read the documentation of the class and make a mental note of what you have to provide (e.g. signals, methods, attributes) and what you have to implement (e.g. start_interface, stop_interface, toggle_interface).

Step 3: Implement a setup interface (Example Virtual Hand Interface)

This example focuses on implementing and adding the setup interface for the Virtual Hand Interface using the VirtualHandInterface_SetupInterface class from setup_interface.py. We explain how it is constructed and registered into MyoGestic via CONFIG_REGISTRY in config.py.

Steps: 1. Define a custom SetupInterface class. 2. Manage the interface’s state and initialization. 3. Handle communication with MyoGestic’s runtime. 4. Handle custom data signals and data processing. 5. Register this class in MyoGestic’s configuration registry.

Step 3.1: Define the Setup Interface Class

This step implements the VirtualHandInterface_SetupInterface class, which initializes the Virtual Hand Interface, manages its state, and ensures communication with MyoGestic’s runtime.

Imports

import ast
import platform
import sys
import time
from pathlib import Path

import numpy as np
from PySide6.QtCore import QByteArray, QMetaObject, QProcess, QTimer, Qt, Signal
from PySide6.QtGui import QCloseEvent
from PySide6.QtNetwork import QHostAddress, QUdpSocket
from PySide6.QtWidgets import QCheckBox, QPushButton, QWidget

from myogestic.gui.widgets.logger import LoggerLevel
from myogestic.gui.widgets.templates.visual_interface import SetupInterfaceTemplate
from myogestic.gui.widgets.visual_interfaces.virtual_hand_interface.ui import (
    Ui_SetupVirtualHandInterface
)
from myogestic.utils.constants import MYOGESTIC_UDP_PORT

# Stylesheets
NOT_CONNECTED_STYLESHEET = "background-color: red; border-radius: 5px;"
CONNECTED_STYLESHEET = "background-color: green; border-radius: 5px;"

# Constants
STREAMING_FREQUENCY = 32
TIME_BETWEEN_MESSAGES = 1 / STREAMING_FREQUENCY

SOCKET_IP = "127.0.0.1"
STATUS_REQUEST = "status"
STATUS_RESPONSE = "active"


# Ports
# on this port the VHI listens for incoming messages from MyoGestic
VHI__UDP_PORT = 1236

# on this port the VHI sends the currently displayed predicted hand after having applied linear interpolation
VHI_PREDICTION__UDP_PORT = 1234

Class Definition

class VirtualHandInterface_SetupInterface(SetupInterfaceTemplate):
    """
    Setup interface for the Virtual Hand Interface.

    This class is responsible for setting up the Virtual Hand Interface.

    Attributes
    ----------
    predicted_hand__signal : Signal
        Signal that emits the predicted hand data.

    Parameters
    ----------
    main_window : QMainWindow
        The main window of the application.
    name : str
        The name of the interface. Default is "VirtualHandInterface".

        .. important:: The name of the interface must be unique.
    """

    predicted_hand__signal = Signal(np.ndarray)

    def __init__(self, main_window, name="VirtualHandInterface"):
        super().__init__(main_window, name, ui=Ui_SetupVirtualHandInterface())

        self._unity_process = QProcess()
        self._unity_process.setProgram(str(self._get_unity_executable()))
        self._unity_process.started.connect(
            lambda: self._main_window.toggle_selected_visual_interface(self.name)
        )
        self._unity_process.finished.connect(self.interface_was_killed)
        self._unity_process.finished.connect(
            lambda: self._main_window.toggle_selected_visual_interface(self.name)
        )

        self._setup_timers()

        self._last_message_time = time.time()
        self._is_connected: bool = False
        self._streaming__udp_socket: QUdpSocket | None = None

        # Custom Stuff
        self._predicted_hand__udp_socket: QUdpSocket | None = None
        self._predicted_hand_recording__buffer: list[(float, np.ndarray)] = []

        # Initialize Virtual Hand Interface UI
        self.initialize_ui_logic()

Helper Functions

    @staticmethod
    def _get_unity_executable() -> Path:
        """Get the path to the Unity executable based on the platform."""
        base_dir = (
            Path("dist") if not hasattr(sys, "_MEIPASS") else Path(sys._MEIPASS, "dist")
        )
        unity_executable_paths = {
            "Windows": base_dir / "windows" / "Virtual Hand Interface.exe",
            "Darwin": base_dir
            / "macOS"
            / "Virtual Hand Interface.app"
            / "Contents"
            / "MacOS"
            / "Virtual Hand Interface",
            "Linux": base_dir / "linux" / "VirtualHandInterface.x86_64",
        }

        executable = unity_executable_paths.get(platform.system())
        if executable and executable.exists():
            return executable
        raise FileNotFoundError(
            f"Unity executable not found for platform {platform.system()}."
        )

    def _setup_timers(self):
        """Setup the timers for the Virtual Hand Interface."""
        self._status_request__timer = QTimer(self)
        self._status_request__timer.setInterval(2000)
        self._status_request__timer.timeout.connect(self.write_status_message)

        self._status_request_timeout__timer = QTimer(self)
        self._status_request_timeout__timer.setSingleShot(True)
        self._status_request_timeout__timer.setInterval(1000)
        self._status_request_timeout__timer.timeout.connect(self._update_status)

UI Setup

    def initialize_ui_logic(self):
        """Initialize the UI logic for the Virtual Hand Interface."""
        self._main_window.ui.visualInterfacesVerticalLayout.addWidget(self.ui.groupBox)

        self._toggle_virtual_hand_interface__push_button: QPushButton = (
            self.ui.toggleVirtualHandInterfacePushButton
        )
        self._toggle_virtual_hand_interface__push_button.clicked.connect(
            self.toggle_virtual_hand_interface
        )
        self._virtual_hand_interface__status_widget: QWidget = (
            self.ui.virtualHandInterfaceStatusWidget
        )

        self._virtual_hand_interface__status_widget.setStyleSheet(
            NOT_CONNECTED_STYLESHEET
        )

        self._use_external_virtual_hand_interface__check_box: QCheckBox = (
            self.ui.useExternalVirtualHandInterfaceCheckBox
        )

Step 3.2: Manage the Interface’s State and Initialization

This step manages the state of the Virtual Hand Interface and initializes the interface’s parameters. It also ensures that the interface is correctly set up and ready for use.

Start Interface - This method starts the interface.

    def start_interface(self):
        """Start the Virtual Hand Interface."""
        if not self._use_external_virtual_hand_interface__check_box.isChecked():
            self._unity_process.start()
            self._unity_process.waitForStarted()
        self._status_request__timer.start()
        self.toggle_streaming()

Stop Interface - This method stops the interface.

    def stop_interface(self):
        """Stop the Virtual Hand Interface."""
        if not self._use_external_virtual_hand_interface__check_box.isChecked():
            self._unity_process.kill()
            self._unity_process.waitForFinished()
        # In case the stop function would be called from outside the main thread we need to use invokeMethod
        QMetaObject.invokeMethod(self._status_request__timer, "stop", Qt.QueuedConnection)
        self.toggle_streaming()

Interface Was Killed - This method checks if the interface was killed (e.g. by the user closing the window).

    def interface_was_killed(self) -> None:
        """Handle the case when the Virtual Hand Interface was killed."""
        self._toggle_virtual_hand_interface__push_button.setChecked(False)
        self._toggle_virtual_hand_interface__push_button.setText("Open")
        self._use_external_virtual_hand_interface__check_box.setEnabled(True)
        self._virtual_hand_interface__status_widget.setStyleSheet(
            NOT_CONNECTED_STYLESHEET
        )
        self._is_connected = False

Update Status - This method updates the interface’s status. If the interface is not running, it will be set to “Stopped”.

    def _update_status(self) -> None:
        """Update the status of the Virtual Hand Interface."""
        self._is_connected = False
        self._virtual_hand_interface__status_widget.setStyleSheet(
            NOT_CONNECTED_STYLESHEET
        )

Toggle Virtual Hand Interface - This method toggles the Virtual Hand Interface. If the interface is running, it will be stopped. If it is stopped, it will be started.

    def toggle_virtual_hand_interface(self):
        """Toggle the Virtual Hand Interface."""
        if self._toggle_virtual_hand_interface__push_button.isChecked():
            print("Opening Virtual Hand Interface")
            self.start_interface()
            self._use_external_virtual_hand_interface__check_box.setEnabled(False)
            self._toggle_virtual_hand_interface__push_button.setText("Close")
        else:
            print("Closing Virtual Hand Interface")
            self.stop_interface()
            self._use_external_virtual_hand_interface__check_box.setEnabled(True)
            self._toggle_virtual_hand_interface__push_button.setText("Open")

Step 3.3: Handle Communication with MyoGestic

This step manages the communication between the Virtual Hand Interface and MyoGestic’s runtime. It ensures that the interface is correctly set up and ready for use.

Toggle Streaming - This method toggles the streaming of data from MyoGestic to the Virtual Hand Interface.

    def toggle_streaming(self) -> None:
        """Toggle the streaming of the Virtual Hand Interface."""
        if self._toggle_virtual_hand_interface__push_button.isChecked():
            self._streaming__udp_socket = QUdpSocket(self)
            self._streaming__udp_socket.readyRead.connect(self.read_message)
            self.outgoing_message_signal.connect(self.write_message)
            self._streaming__udp_socket.bind(
                QHostAddress(SOCKET_IP), MYOGESTIC_UDP_PORT
            )

            self._predicted_hand__udp_socket = QUdpSocket(self)
            self._predicted_hand__udp_socket.bind(
                QHostAddress(SOCKET_IP), VHI_PREDICTION__UDP_PORT
            )
            self._predicted_hand__udp_socket.readyRead.connect(self.read_predicted_hand)

            self._last_message_time = time.time()
        else:
            try:
                self._streaming__udp_socket.close()
                self._predicted_hand__udp_socket.close()
            except AttributeError:
                pass
            self._streaming__udp_socket = None
            self._predicted_hand__udp_socket = None
            self._is_connected = False
            self._virtual_hand_interface__status_widget.setStyleSheet(
                NOT_CONNECTED_STYLESHEET
            )

Read Predicted Hand - This method reads the predicted hand from MyoGestic.

    def read_predicted_hand(self) -> None:
        """Read the predicted hand data from the Virtual Hand Interface."""
        while self._predicted_hand__udp_socket.hasPendingDatagrams():
            datagram, _, _ = self._predicted_hand__udp_socket.readDatagram(
                self._predicted_hand__udp_socket.pendingDatagramSize()
            )

            data = datagram.data().decode("utf-8")
            if not data:
                return

            self.predicted_hand__signal.emit(np.array(ast.literal_eval(data)))

Read/Write Message - This methods read and write messages from/to MyoGestic.

    def read_message(self) -> None:
        """Read a message from the Virtual Hand Interface."""
        if self._toggle_virtual_hand_interface__push_button.isChecked():
            while self._streaming__udp_socket.hasPendingDatagrams():
                datagram, _, _ = self._streaming__udp_socket.readDatagram(
                    self._streaming__udp_socket.pendingDatagramSize()
                )

                data = datagram.data().decode("utf-8")
                if not data:
                    return

                try:
                    if data == STATUS_RESPONSE:
                        self._is_connected = True
                        self._virtual_hand_interface__status_widget.setStyleSheet(
                            CONNECTED_STYLESHEET
                        )
                        self._status_request_timeout__timer.stop()
                        return

                    self.incoming_message_signal.emit(np.array(ast.literal_eval(data)))
                except (UnicodeDecodeError, SyntaxError):
                    pass

    def write_message(self, message: QByteArray) -> None:
        """Write a message to the Virtual Hand Interface."""
        if self._is_connected and (
            time.time() - self._last_message_time >= TIME_BETWEEN_MESSAGES
        ):
            self._last_message_time = time.time()
            output_bytes = self._streaming__udp_socket.writeDatagram(
                message, QHostAddress(SOCKET_IP), VHI__UDP_PORT
            )

            if output_bytes == -1:
                self._main_window.logger.print(
                    "Error in sending message to Virtual Hand Interface!",
                    level=LoggerLevel.ERROR,
                )

    def write_status_message(self) -> None:
        """Write a status message to the Virtual Hand Interface."""
        if self._toggle_virtual_hand_interface__push_button.isChecked():
            output_bytes = self._streaming__udp_socket.writeDatagram(
                STATUS_REQUEST.encode("utf-8"),
                QHostAddress(SOCKET_IP),
                VHI__UDP_PORT,
            )

            if output_bytes == -1:
                self._main_window.logger.print(
                    "Error in sending status message to Virtual Hand Interface!",
                    level=LoggerLevel.ERROR,
                )
                return

            self._status_request_timeout__timer.start()

Step 3.4: Handle Custom Data Signals

This step manages custom data signals and data processing for the Virtual Hand Interface. It ensures that the interface is correctly set up and ready for use.

Connect/Disconnect Custom Signals - This method connects/disconnects custom signals.

    def connect_custom_signals(self) -> None:
        """Connect custom signals for the Virtual Hand Interface."""
        self.predicted_hand__signal.connect(self.online_predicted_hand_update)

Online Predicted Hand Update - This method updates the predicted hand buffer with the current data.

    def online_predicted_hand_update(self, data: np.ndarray) -> None:
        """Update the predicted hand data for the online protocol."""
        if self._online_protocol.online_record_toggle_push_button.isChecked():
            self._predicted_hand_recording__buffer.append(
                (time.time() - self._online_protocol.recording_start_time, data)
            )

Clear Custom Signal Buffers - This method clears the custom signal buffers.

    def clear_custom_signal_buffers(self) -> None:
        """Clear custom signal buffers for the Virtual Hand Interface."""
        self._predicted_hand_recording__buffer = []

Get Custom Save Data - This method will be called when saving the interface’s data. This should return a dictionary with the data that needs to be saved.

    def get_custom_save_data(self) -> dict:
        """Get custom save data for the Virtual Hand Interface."""
        return {
            "predicted_hand": np.vstack(
                [data for _, data in self._predicted_hand_recording__buffer],
            ).T,
            "predicted_hand_timings": np.array(
                [time for time, _ in self._predicted_hand_recording__buffer],
            ),
        }

Step 3.5: Register the Setup Interface in Config

This step integrates the VirtualHandInterface_SetupInterface class into the MyoGestic configuration. Once registered, the interface becomes available for use in the framework.

from myogestic.gui.widgets.visual_interfaces.virtual_hand_interface import VirtualHandInterface_SetupInterface
from myogestic.utils.config import CONFIG_REGISTRY


CONFIG_REGISTRY.register_visual_interface(
    name="VirtualHandInterface",  # Unique identifier for the visual interface.
    setup_interface_ui=VirtualHandInterface_SetupInterface,  # Associated setup class.
    recording_interface_ui=None,  # Placeholder (can be extended with a recording interface).
)

Estimated memory usage: 528 MB