Research work on Optical Music Recognition

Project Description

Optical Music Recognition (OMR) is a field of research that focuses on using machine learning, computer vision, and immersive XR technologies to automatically recognize, decode, and interact with musical scores from scanned or digital document images into machine-readable formats (such as MusicXML or MIDI).

Decoding complex musical notations typically involves multi-stage processing steps:

Low-level Analysis: Detecting and recognizing base musical primitives, including staff lines, individual noteheads, clefs, accidentals, and rests.
Mid-level Analysis: Establishing spatial and structural relationships among musical symbols (e.g., mapping notes to correct staff lines and measuring pitches).
High-level Analysis: Synthesizing structural semantics, semantic formatting, and full digital score generation and immersive XR workspace translation.

Datasets & Heterogeneous Alignment

Training a unified OMR model for immersive environments faces a core domain gap: Western staff notation encodes pitch vertically, while percussion notation encodes timbre. To resolve this, we combine DeepScores V2 with our proposed SWARA-Percussion dataset using a unified harmonization pipeline.

SWARA-Percussion Dataset

Existing datasets exclude percussion notations and fail to capture shape-based notehead variants (such as $\times$ for cymbals). SWARA-Percussion is a dense-object detection benchmark containing 1,000 single-page drum notation scores with 38,000 annotated symbols.

Table 1: SWARA percussion symbol classes using compact symbolic representations.

ID	0	1	2	3	4	5	6	7
Class	clef	beam	barline	snare	cymbal	diamondNotehead	accent	rest

Table 2: Class Distribution Analysis illustrating the "Symbol Gap" addressed by SWARA compared to standard subsets.

Dataset	Noteheads ($\bullet$)	Noteheads ($\times$)	Beams	Total Symbols
DeepScores (Subset)	15,400	< 100	4,200	45,000
SWARA (Ours)	12,100	8,500	3,800	38,000

Link to SWARA Repository

Publications

Multi-Modal Heterogeneous Music Notation Recognition from Documents: A Framework for Real-Time Symbolic-to-Immersive Mappings

Ankit Sinha, Atanu Saha, Chiranjoy Chattopadhyay, Rahul Kumar Ray
International Conference on Document Analysis and Recognition (ICDAR), Vienna, Austria, August-September, 2026.

View Project Page

From Notes to Keys: A VR Learning Environment for Sheet Music Interpretation

Sandeep Khanna, Atanu Saha, Rahul Kumar Ray, Rakesh Patibanda, Chiranjoy Chattopadhyay
International Conference on Document Analysis and Recognition (ICDAR), September, 2025.

View Project Page