Skip to main content

Modality Framework

This section is informative and explains the Modality Framework in a human-friendly way.

What Is a Modality?​

A Modality is a structured type of data — like:

  • a skeletal pose
  • a contact state
  • an EEG window
  • a depth frame
  • a robot joint command

Think of it like a “class” of data with:

  • a name
  • a family/domain
  • a version
  • some metadata requirements
  • a schema defining its fields

Example:

MIND.pose/SkeletalPose@1.0.0

Why namespaces?​

Namespaces prevent collisions and allow ecosystems to coexist:

MIND.pose/SkeletalPose
MIND_EXT.pose/HandPose
ACME.cv/HandKeypoints
LAB_X.bio/EEGHighDensity

This is exactly how:

  • OpenXR uses XR_EXT_*
  • glTF uses KHR_*
  • ROS uses sensor_msgs/*, geometry_msgs/*
  • USD uses UsdGeom*, UsdSkel*

What are Modality Families?​

Families help organize the ecosystem.

Examples:

  • pose/ → joint rotations, full-body tracking
  • bio/ → EEG, EMG
  • cv/ → 2D/3D landmarks, depth images
  • agent/ → commands to a robot or virtual body

This keeps things clean and scalable.

ASCII diagram:

MIND namespace
 ├── pose/
 │     ├── SkeletalPose
 │     ├── HandPose
 │     └── FullBodyPose
 ├── bio/
 │     ├── EEG
 │     └── EMG
 ├── cv/
 │     ├── Pose2D
 │     ├── DepthFrame
 │     └── ObjectDetections
 └── agent/
       ├── JointCommand
       └── GazeCommand

Versioning Explained​

Every modality evolves:

  • New fields
  • Adjusted constraints
  • Better precision

Semantic versioning makes this predictable:

  • 1.x.x → stable and compatible
  • 2.0.0 → breaking changes

Required Metadata: Why Important?​

A modality cannot stand alone.

Examples:

  • A SkeletalPose without a SkeletonProfile is meaningless
  • EEG without a sensor layout is unusable
  • A robot joint command without a RobotModelProfile can’t be applied

So modality specs define:

  • which metadata profiles are required
  • how they link together

Unknown Modalities — Safe and Future-Proof​

If a new lab invents:

LAB_X.bio/GalvanicSkinResponse

Tools should:

  • keep the data
  • ignore it unless needed
  • not crash

This makes MIND future-proof.

Registry Overview​

The Modality Registry is the authoritative list of:

  • core modalities
  • extensions
  • vendor additions

It tracks:

  • names
  • versions
  • stability
  • family
  • required metadata

It acts like the “package index” for the MIND ecosystem.

Summary​

Modality design in MIND follows the same patterns used in:

  • OpenXR
  • glTF
  • USD
  • ROS
  • BIDS

It is:

  • scalable
  • extensible
  • future-proof
  • versioned
  • namespaced
  • metadata-aware

This makes MIND suitable for:

  • mocap
  • XR
  • biosensing
  • CV pipelines
  • robotics
  • embodied AI