Deep Learning-Based Optical Music Recognition for Semantic Representation of Non-overlap and Overlap Music Notes

Rana L. Abdulazeez; Fattah  Alizadeh

doi:10.14500/aro.11402

Rana L. Abdulazeez Department of Software and Informatics Engineering, College of Engineering, Salahaddin University-Erbil, Kurdistan Region – F.R. Iraq https://orcid.org/0009-0004-5204-2218
Fattah Alizadeh Department of Computer Engineering, School of Science and Engineering, University of Kurdistan Hewler, Kurdistan Region – F.R. Iraq https://orcid.org/0000-0002-0690-1147

Keywords: Sequence-To-Sequence, Long Short-term Memory Network, Convolutional Neural Network, Segmentation, Semantic representation, Overlapping

Abstract

In the technology era, the process of teaching a computer to interpret musical notation is termed optical music recognition (OMR). It aims to convert musical note sheets presented in an image into a computer-readable format. Recently, the sequence-to-sequence model along with the attention mechanism (which is used in text and handwritten recognition) has been used in music notes recognition. However, due to the gradual disappearance of excessively long sequences of musical sheets, the mentioned OMR models which consist of long short-term memory are facing difficulties in learning the relationships among the musical notations. Consequently, a new framework has been proposed, leveraging the image segmentation technique to break up the procedure into several steps. In addition, an overlap problem in OMR has been addressed in this study. Overlapping can result in misinterpretation of music notations, producing inaccurate findings. Thus, a novel algorithm is being suggested to detect and segment the notations that are extremely close to each other. Our experiments are based on the usage of the Convolutional Neural Network block as a feature extractor from the image of the musical sheet and the sequence-to-sequence model to retrieve the corresponding semantic representation. The proposed approach is evaluated on The Printed Images of Music Staves dataset. The achieved results confirm that our suggested framework successfully solves the problem of long sequence music sheets, obtaining SER 0% for the non-overlap symbols in the best scenario. Furthermore, our approach has shown promising results in addressing the overlapping problem: 23.12 % SER for overlapping symbols.

Downloads

Download data is not yet available.

Author Biographies

Rana L. Abdulazeez, Department of Software and Informatics Engineering, College of Engineering, Salahaddin University-Erbil, Kurdistan Region – F.R. Iraq

Rana L. Abdulazeez is an Assistant Lecturer at the department of Software and Informatics Engineering, Collage of Engineering, Salahaddin University-Erbil. She got the B.Sc. degree in Computer Science and the M.Sc. degree in Computer Science. Her research interest is in deep learning, computer vision, machine learning and image processing.

Fattah Alizadeh, Department of Computer Engineering, School of Science and Engineering, University of Kurdistan Hewler, Kurdistan Region – F.R. Iraq

Fattah Alizadeh is a Professor at the department of Computer science, Toronto Metropolitan University. He got the B.Sc. degree in Computer Science, the M.Sc. degree in Computer Science and the Ph.D. degree in 3D Image Processing. His research interests are in Image Processing, Computer Vision, Machine Learning and Deep learning.

References

Baró, A., Badal, C., and Fornés, A., 2020. Handwritten Historical Music Recognition by Sequence-to-Sequence with Attention Mechanism. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE, United States, pp.205-210. DOI: https://doi.org/10.1109/ICFHR2020.2020.00046

Brownlee, J., 2019. Deep Learning for Computer Vision: Image Classification, Object Detection, and Face Recognition in Python. Machine Learning Mastery, Vermont.

Calvo-Zaragoza, J., and Rizo, D., 2018. End-to-end neural optical music recognition of monophonic scores. Applied Sciences, 8(4), p.606. DOI: https://doi.org/10.3390/app8040606

Calvo-Zaragoza, J., Valero-Mas, J.J., and Pertusa, A., 2017. End-to-End Optical Music Recognition using Neural Networks. In: Proceedings of the 18th International Society for Music Information Retrieval Conference. ISMIR, Canada, pp.23-27.

Castellanos, F.J., Calvo-Zaragoza, J., and Inesta, J.M., 2020. A Neural Approach for Full-Page Optical Music Recognition of Mensural Documents. ISMIR, Canada, pp.558-565.

Jang, M., Seo, S., and Kang, P., 2019. Recurrent neural network-based semantic variational autoencoder for sequence-to-sequence learning. Information Sciences, 490, pp.59-73. DOI: https://doi.org/10.1016/j.ins.2019.03.066

Matrenin, P.V., Manusov, V.Z., Khalyasmaa, A.I., Antonenkov, D.V., Eroshenko, S.A., and Butusov, D.N., 2020. Improving accuracy and generalization performance of small-size recurrent neural networks applied to short-term load forecasting. Mathematics, 8(12), p.2169. DOI: https://doi.org/10.3390/math8122169

Michael, J., Labahn, R., Grüning, T., and Zöllner, J., 2019. Evaluating Sequenceto-Sequence Models for Handwritten Text Recognition. In: 2019 International Conference on Document Analysis and Recognition. ICDAR. IEEE, United States, pp.1286-1293. DOI: https://doi.org/10.1109/ICDAR.2019.00208

Mondal, R., Malakar, S., Barney Smith, E.H., and Sarkar, R., 2022. Handwritten English word recognition using a deep learning based object detection architecture. Multimedia Tools and Applications, 81, pp.1-26. DOI: https://doi.org/10.1007/s11042-021-11425-7

Neubig, G., 2017. Neural Machine Translation and Sequence-to-Sequence Models: A Tutorial. [arXiv Preprint] arXiv:1703.01619.

Pugin, L., 2006. Optical Music Recognitoin of Early Typographic Prints using Hidden Markov Models. ISMIR, Canada, pp.53-56.

Pugin, L., Burgoyne, J.A., and Fujinaga, I., 2007. MAP Adaptation to Improve Optical Music Recognition of Early Music Documents Using Hidden Markov Models. ISMIR, Canada, pp.513-516.

Rosen, K.H., 2007. Discrete Mathematics and Its Applications. The McGraw Hill Companies, United States.

Shatri, E., and Fazekas, G., 2020. Optical Music Recognition: State of the Art and Major Challenges. Computer Science, Engineering. [arXiv preprint] arXiv:2006.07885.

Sutskever, I., Vinyals, O., and Le, Q.V., 2014. Sequence to Sequence Learning with Neural Networks. In: Advances in Neural Information Processing Systems. Vol. 27. The MIT Press, United States.

Torras, P., Baró, A., Fornés, A., and Kang, L., 2022. Improving Handwritten Music Recognition through Language Model Integration. In: 4th International Workshop on Reading Music Systems, p.42.

Van Der Wel, E., and Ullrich, K., 2017. Optical Music Recognition with Convolutional Sequence-to-Sequence Models. [arXiv preprint] arXiv:1707.04877.

Wen, C., and Zhu, L., 2022. A sequence-to-sequence framework based on transformer with masked language model for optical music recognition. IEEE Access, 10, pp.118243-118252 DOI: https://doi.org/10.1109/ACCESS.2022.3220878