Skip to main content

Modality Framework

This section is normative.

The Modality Framework defines how all data structures in MIND are identified, versioned, grouped, extended, and validated. It provides the foundation for interoperability across motion capture systems, XR devices, biosensors, computer vision pipelines, robotics systems, and embodied AI agents.

Modality Identifiers​

A Modality Identifier MUST take the form:

<namespace>.<family>/<name>@<version>

Where:

  • <namespace> MUST be one of:
    • MIND for core modalities
    • MIND_EXT for official extensions
    • Vendor namespaces (e.g., ACME, LAB_X, XR_CORP)
  • <family> MUST be a valid Modality Family (Section 5.2)
  • <name> MUST be an alphanumeric identifier
  • <version> MUST follow semantic versioning (major.minor.patch)

Example:

MIND.pose/SkeletalPose@1.0.0
MIND.bio/EEG@1.1.0
MIND.cv/Pose2D@1.0.0
MIND_EXT.pose/HandPose@1.0.0
VENDOR_X.device/IMURaw@2.3.1

Unknown namespaces MUST be preserved and treated as optional unless declared required.

Modality Families​

All modalities MUST belong to one of the following canonical families:

  • pose/ (e.g., SkeletalPose, HandPose, FullBodyPose)
  • contact/ (e.g., ContactState, FingerContact)
  • input/ (e.g., ControlSignal, ControllerButtonState)
  • bio/ (e.g., EEG, EMG, HeartRate)
  • cv/ (e.g., Pose2D, DepthFrame, ObjectDetections)
  • device/ (e.g., IMURaw, MarkerSet)
  • agent/ (e.g., JointCommand, GazeCommand)
  • annotation/ (e.g., VariableUpdate, Annotation)

A Modality Family determines shared metadata expectations and validation behavior.

Core vs Extension Modalities​

Core Modalities​

The following MUST be supported by all conforming implementations:

  • MIND.pose/SkeletalPose@1.x.x
  • MIND.contact/ContactState@1.x.x
  • MIND.input/ControlSignal@1.x.x
  • MIND.annotation/Annotation@1.x.x
  • MIND.bio/* (only if biosensing devices are present)
  • MIND.cv/* (only if CV sources are present)
  • MIND.device/* (for devices contributing raw sensor data)

Extension Modalities​

Any modality not included in the core MUST:

  • Reside in the MIND_EXT or vendor namespace
  • Be registered in MIND-Registry/extensions.json
  • Declare its family correctly
  • Specify any required metadata references

Unknown extension modalities MUST be preserved during parsing.

Versioning Requirements​

Modality versions MUST follow semantic versioning:

  • Major version increments indicate breaking changes
  • Minor version increments indicate backward-compatible additions
  • Patch version increments indicate non-breaking fixes

Example:

MIND.pose/SkeletalPose@1.2.1

A Container MUST declare which versions it supports for each modality.

Required Metadata References​

Each modality MUST define any required metadata references, such as:

  • SkeletonProfile (pose modalities)
  • CameraProfile (CV modalities)
  • BiosensorDeviceProfile (bio modalities)
  • IMUCalibrationProfile (device modalities)
  • RobotModelProfile (agent modalities)

Missing required metadata MUST invalidate the Container.

Unknown and Unsupported Modalities​

Unknown modalities MUST be:

  • Preserved when reading
  • Ignored unless:
    • explicitly referenced by another stream/event, or
    • marked as required: true

Modality Registry​

A machine-readable registry MUST exist at:

MIND-Registry/modality_registry.json

It MUST list:

  • core modalities
  • extension modalities
  • vendor modalities
  • versions
  • status (stable, experimental, deprecated)

Summary​

  • Modalities are identified by namespaced, versioned IDs
  • Modalities belong to canonical families
  • Extensions must be registered
  • Unknown modalities are preserved
  • Modalities define required metadata
  • A central registry governs lifecycle and evolution

This completes Section 5 (formal).