Scene Analysis for Music Understanding
Music source separation has drawn plenty of attention for its ability to facilitate tasks in music information retrieval. However, source separation is difficult because there does not exist a unique solution; given a musical mixture, there are several valid ways to decompose the mixture into its individual sources.
Researchers have devoted years of effort toward solving the fully automatic blind source separation problem. Although significant progress has been made along the theoretical and algorithmic aspects of source separation, the separation performance achievable by the state of the art is still not adequate for widespread commercial use. Because of a total lack of prior information, blind source separation remains hard to solve.
In our work, we have proposed a novel method for performing source separation of a musical signal given side information in MIDI format of a similar performance. Our method uses nonnegative matrix factorization (NMF) — a popular, convenient, and effective method for decomposing spectrograms. By exploiting the information that MIDI provides, our method is able to separate sources within highly polyphonic musical mixtures. We have also proposed a novel method that imposes additional harmonic constraints upon the musical atoms learned by NMF. When there is significant spectral-temporal overlap among the musical sources, our learning method has better recall and precision than other popular existing matrix factorization methods.
Image Forensics via Intrinsic Fingerprints
Multimedia forensic methods allow us to maintain the integrity of the multimedia data around us. For example, we can embed a watermark into a digital image to bind the identity of its owner to the image itself. However, traditional forensic approaches such as watermarking are not applicable in many real-world scenarios, for example, when we do not have access to the original data.
Instead, the research we perform at the University of Maryland focuses on intrinsic fingerprints — subsets of data which are, or have become, an intrinsic part of the data in question. By examining these intrinsic fingerprints, we can assess the authenticity of data without embedding any watermarks, thus increasing the applicability of these forensic methods.
We recently formulated a forensic methodology to identify the compression history of a digital image. By examining the intrinsic fingerprints in an image, we can tell what compression method (e.g., JPEG, etc.) was used in order to determine the origin of the image and thereby assess its authenticity. Current efforts include the examination and detection of intrinsic fingerprints in other domains, particularly digital audio. Please see the publications below for more information.
Publications
See publications.