@INPROCEEDINGS{tjoa2011ismir,
  title = "Factorization of Overlapping Harmonic Sounds Using Approximate Matching Pursuit",
  author = "Steven K. Tjoa and K. J. Ray Liu",
  booktitle = "Proc. Int. Soc. Music Information Retrieval Conf.",
  address = "Miami, FL",
  year = "2011",
  month = oct,
  pages = "257--262"
};

Sparse coding and nonnegative matrix factorization (NMF) have revolutionized the area of music transcription. These methods usually decompose spectra from input signals as linear combinations of dictionary spectra. Often, the dictionary is either learned (as in NMF) or synthetic (e.g., harmonic templates, Gabor atoms, etc.).

Instead, what if you already have a massive overcomplete dictionary containing millions of well-labeled atoms from real-world sounds? Assuming that the dictionary is large and representative enough, there is no longer a need to learn. However, coding (i.e., finding the coefficients of the input with respect to the dictionary) remains a problem. With such a large dictionary, complexity becomes an issue.

We propose a method called Approximate Matching Pursuit (AMP) which allows you to efficiently obtain a sparse decomposition of an input vector using a massive overcomplete dictionary. AMP is a simple variant of matching pursuit or orthogonal matching pursuit. However, instead of finding an exact match between the input (or residue) and the dictionary at each iteration, we only find an approximate match. In doing so, we sacrifice a bit of accuracy to obtain significant savings in computation.

Using AMP, we decomposed input spectra of containing overlapping harmonic sounds, where the amount of polyphony varies. Our results showed that, while retrieval performance is similar to orthogonal matching pursuit based upon F-measure, execution time and computation is reduced significantly.

@INPROCEEDINGS{keegan2011waspaa,
  title = "Super-resolution of Musical Signals Using Approximate Matching Pursuit",
  author = "Brennan P. Keegan and Steven K. Tjoa and K. J. Ray Liu",
  booktitle = "Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics",
  address = "New Paltz, NY",
  year = "2011",
  month = oct,
  pages = "81--84"
};

Super-resolution is the act of increasing the resolution (or sampling rate) of a signal in an intelligent manner that preserves the signal’s ground truth information, i.e., not through some parametric method like linear or sinusoidal interpolation. For example, when you use Adobe Photoshop or GIMP to make an image larger, the new image will likely be blurry because the program does not know how to restore sharp edges.

Super-resolution of images and video has been studied extensively, but not as much for audio. When an audio signal is upsampled from 4 kHz to 44.1 kHz using traditional sinc interpolation, the upsampled signal will not contain any new information that is not already present in the 4 kHz version. As a result, high-frequency components will be absent, and the signal will sound dull and muffled.

There are a few methods for audio super-resolution. For example, Smaragdis et al. have proposed a method whose basic idea is to take the low-resolution input signal and project it upon a low-resolution basis. Using that projection (or the coefficients with respect to the low-resolution basis), a high-resolution signal is constructed by multiplying the coefficients with components from a high-resolution basis.

What basis do you use? In the literature, that basis (or overcomplete dictionary — perhaps not orthogonal) is usually obtained in one of two ways: use a synthetic basis (Gabor, etc.), or learn it from real data (NMF, K-SVD, etc.).

We ask this question: What if you already have a massive overcomplete dictionary containing millions of well-labeled atoms from real-world sounds? Can you make use of it? How? With such a large dictionary, complexity becomes an issue. Using ordinary matching pursuit, each iteration has complexity that is linear in the size of the dictionary. This is not scalable.

We propose a method called Approximate Matching Pursuit which allows you to efficiently obtain a sparse decomposition of an input vector using a massive overcomplete dictionary. Using it, we perform super-resolution of signals containing piano music. Using a dictionary composed of sounds from the University of Iowa data set, we were able to accurately estimate missing high-frequency information using low-resolution inputs sampled as low as 2-4 kHz. For more details, please see the paper above.

For more about Approximate Matching Pursuit, please see our upcoming paper in ISMIR 2011.

It is interesting to see how thesis topics evolve from advisor to advisee, yet my research area — music information retrieval — is not so removed from Helmholtz’s work on acoustics; in fact, MIR scholars still cite and recognize his contributions!

Here are the people that make up my lineage since Helmholtz, each person’s dissertation title, place of graduate study, and graduation year.

1. Hermann von Helmholtz, De fabrica systematis nervosi evertebratorum, Universität Berlin, 1842.
2. Edward Leamington Nichols, Ueber das von glühendem Platin ausgestrahlte Licht: Ein Beitrag zur allgemeinen Ausstrahlungslehr, Georg-August-Universität Göttingen, 1879.
3. Ernest Fox Nichols, Radiometric Researches in the Remote Infra-Red Spectrum, Cornell, 1897.
4. Frederic Columbus Blake, The Reflection and Transmission of Electric Waves by Screens of Resonators and by Grids, Columbia, 1906.
5. William Littell Everitt, The Calculation and Design of Alternating Current Networks Employing Triodes Operating During a Portion of a Cycle, Ohio State, 1933.
6. Karl Ralph Spangenberg, The Effect of Grid Current Flow Upon the Dynamic Characteristics of Vacuum Tube Power, Ohio State, 1937.
7. Willis Harman, Tunable Waveguide Cavity Resonators for Broadband Operation of Reflex Klystrons, Stanford, 1948.
8. John Bowman Thomas, On the Statistical Design of Demodulation Systems for Signals in Additive Noise, Stanford, 1955.
9. Kung Yao, On Some Representations and Sampling Expansions for Band Limited Signals, Princeton, 1965.
10. K. J. Ray Liu, Efficient and Reliable Parallel Processing Algorithms and Architectures for Modern Signal Processing, UCLA, 1990.
11. Steve Tjoa, Sparse and Nonnegative Factorizations for Music Understanding, Maryland, 2011.

• Steven K. Tjoa and K. J. Ray Liu
• Int. Soc. Music Information Retrieval Conf., August 2010
• Download: Paper, Poster, BibTeX

  @INPROCEEDINGS{tjoa2010ismir,
    title = "Musical Instrument Recognition Using Biologically Inspired Filtering of Temporal Dictionary Atoms",
    author = "Steven K. Tjoa and K. J. Ray Liu",
    booktitle = "Proc. Int. Soc. Music Information Retrieval Conf.",
    address = "Utrecht, Netherlands",
    year = "2010",
    month = aug,
    pages = "435--440"
  };

Most musical instrument recognition systems rely upon spectral information to classify sounds. Can temporal information improve classification accuracy even further?

Evidence from the psychoacoustic literature suggests that both spectral and temporal content carry information about acoustic timbre through the human auditory system. In other words, because humans recognize musical instruments so effortlessly, then perhaps machines would also benefit from a combination of spectral and temporal information. Researchers use a model known as the cortical representation to emulate the information output from the middle stage of the human auditory system. Although some engineers have tried to build music information retrieval (MIR) systems that use the cortical representation, its high dimensionality makes it difficult to employ in practical systems.

But what if we can embody the traits of the cortical representation into another representation that also includes spectral and temporal content? One such candidate is nonnegative matrix factorization (NMF), a tool that can extract spectral and temporal information from spectrograms.

In this paper, we test the usefulness of temporal information extracted using NMF in instrument recognition. We mimic the multiresolution aspect of the cortical representation by using a multiresolution gamma filterbank that parameterizes the shape of temporal envelopes. Our results show that this method of temporal processing can classify among isolated sounds among 24 instrument classes with 92.3% accuracy.
]]> http://stevetjoa.com/334/feed 0 http://stevetjoa.com/269 http://stevetjoa.com/269#comments Mon, 22 Mar 2010 07:50:51 +0000 Steve http://stevetjoa.com/?p=269 Multiplicative Update Rules for Nonnegative Matrix Factorization with Co-occurrence Constraints

• Steven K. Tjoa and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, March 2010
• Download: Paper, Poster, BibTeX

  @INPROCEEDINGS{tjoa2010icassp_cooccurrence,
    title = "Multiplicative Update Rules for Nonnegative Matrix Factorization with Co-occurrence Constraints",
    author = "Steven K. Tjoa and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    address = "Dallas, TX",
    year = "2010",
    month = mar,
    pages = "449--452"
  };

Nonnegative matrix factorization (NMF) has become a popular tool for discovering structure in a variety of signals. When applied to a musical audio signal, NMF builds a set of dictionary atoms that represent the individual musical sources in the signal. To perform music transcription, we map the learned dictionary atoms to musical notes and beats.

The basic formulation of NMF has the notable disadvantage that sources may require more than a single dictionary atom in order to be approximated accurately. For example, one note played by a violin may require multiple dictionary atoms to be accurately represented due to the vibrato induced by the performer. In the presence of many other musical sources, the correspondence between atoms becomes unclear.

In this paper, we introduce three new update rules to enforce dependence among dictionary atoms by incorporating co-occurrence constraints into NMF. These co-occurrence constraints have shown to be useful for describing sources with multiple, co-occurring dictionary atoms by grouping similar atoms into sets. The proposed rules are conceptually simple, easy to implement, and effective for describing sources using multiple dictionary atoms.

• Steven K. Tjoa, Matthew C. Stamm, W. Sabrina Lin, and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, March 2010
• Download: Paper, Poster, BibTeX

  @INPROCEEDINGS{tjoa2010icassp_harmonic,
    title = "Harmonic Variable-Size Dictionary Learning for Music Source Separation",
    author = "Steven K. Tjoa and Matthew C. Stamm and W. Sabrina Lin and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    address = "Dallas, TX",
    year = "2010",
    month = mar,
    pages = "413--416"
  };

Methods that employ dictionary learning and sparse coding have become popular for discovering structure in acoustic signals. Unfortunately, these methods also share a common limitation. When there is significant spectral-temporal overlap among the dictionary atoms present in a signal, it becomes difficult for these methods to learn atoms properly. Often, information from multiple atoms is represented as a single atom by the learning procedure. If an atom in the output dictionary contains musical information from multiple sources, transcription and source separation cannot be accurately performed.

In this paper, we propose a novel dictionary learning method that performs well despite the presence of spectral-temporal overlap among dictionary atoms. Our method imposes a harmonic constraint that restricts each atom to represent at most one pitch. Furthermore, our method is flexible by allowing the size of the dictionary to grow based upon the complexity of the input signal. Our method consistently achieves higher recall and precision than other well-known dictionary learning algorithms.

• Matthew C. Stamm, Steven K. Tjoa, W. Sabrina Lin, and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, March 2010
• Download: Paper, BibTeX

  @INPROCEEDINGS{stamm2010icassp_antiforensics,
    title = "Anti-Forensics of {JPEG} Compression",
    author = "Matthew C. Stamm and Steven K. Tjoa and W. Sabrina Lin and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    address = "Dallas, TX",
    year = "2010",
    month = mar,
    pages = "1694--1697"
  };

Digital image forensics is a relatively new field that has only become popular within the last seven years. A newer body of research involves anti-forensics — methods of processing images that disguise evidence of earlier image processing or tampering. By altering the forensically significant properties in an image, existing image forensic systems may be rendered useless. Like many areas of security, we benefit from the study of anti-forensics by understanding the attacks that are most likely to occur before they actually do. Then, we can refine our image forensic systems further.

Earlier, our group proposed the first forensic system that identifies which compression method was used upon a digital image. In this paper, we propose an anti-forensic method for defeating the identification of JPEG compression in an image. Our method alters the intrinsic fingerprint of JPEG-compressed images in the transform domain such that the attacked image is perceptually similar to the image before attack. To our knowledge, this work is the first to propose a method specifically designed to defeat compression identification systems, and we anticipate that more work will follow.

• W. Sabrina Lin, Steven K. Tjoa, H. Vicky Zhao, and K. J. Ray Liu
• IEEE Trans. Information Forensics and Security, September 2009
• Download: Paper, BibTeX

  @ARTICLE{lin2009tifs,
    title = "Digital Image Source Coder Forensics via Intrinsic Fingerprints",
    author = "W. Sabrina Lin and Steven K. Tjoa and H. Vicky Zhao and K. J. Ray Liu",
    journal = "IEEE Trans. Information Forensics and Security",
    year = "2009",
    month = sep,
    volume = "4",
    number = "3",
    pages = "460--475"
  };

Multimedia forensic methods allow us to verify and maintain the integrity of our multimedia data. For example, we can embed a watermark into a digital image to bind the identity of its owner to the image itself. However, traditional forensic approaches such as watermarking are not applicable in many real-world scenarios, for example, when we do not have access to the original data.

In this paper, we investigate the use of intrinsic fingerprints — subsets of data which are, or have become, an intrinsic part of the data in question — for formulating a forensic methodology that can identify the compression history of a digital image. By examining the intrinsic fingerprints in an image, we can tell what compression method (e.g., JPEG, etc.) was used in order to determine the origin of the image and thereby assess its authenticity.

Although there exist methods that estimate parameters of individual compression schemes, to our knowledge, our work is the first to address the problem of image compression identification and offer a unified forensic framework incorporating multiple encoders.

• Steven Tjoa, W. Sabrina Lin, and K. J. Ray Liu
• IEEE Int. Conf. Image Processing, September 2007
• Download: Paper, Presentation, BibTeX

  @INPROCEEDINGS{tjoa2007icip,
    title = "Transform Coder Classification for Digital Image Forensics",
    author = "Steven Tjoa and W. Sabrina Lin and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. on Image Processing",
    address = "San Antonio, TX",
    year = "2007",
    month = sep,
    volume = "6",
    pages = {VI-105--VI-108},
  };

Our TIFS paper was the first work to offer a unified forensic framework that can identify the compression method used upon an image. Among these compression methods are transform coders — compression methods that apply some form of entropy coding in the transform domain (e.g., via discrete cosine transform, wavelet transform, etc.). Transform coding is undoubtedly the most popular category of image coders in use today. Therefore, the forensic problem of transform coder classification deserves attention.

In this paper, we propose a method that can identify which transform was used upon an image during the compression process. By analyzing the transform coefficient histogram obtained after applying a candidate transform, we can determine if the image was compressed using the transform tested. Results show that forensic classification accuracy is high when testing from among six well-known transforms.

To our knowledge, this work is the first to address the problem of transform coder classification and propose a working solution.

• W. Sabrina Lin, Steven Tjoa, H. Vicky Zhao, and K. J. Ray Liu
• IEEE Int. Conf. Multimedia and Expo, July 2007
• Download: Paper, BibTeX

  @INPROCEEDINGS{lin2007icme,
    title = "Image Source Coding Forensics via Intrinsic Fingerprints",
    author = "W. Sabrina Lin and Steven Tjoa and H. Vicky Zhao and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Multimedia and Expo",
    address = "Beijing, China",
    year = "2007",
    month = jul,
    pages = "1127--1130",
  };

The most popular image compression schemes in use today are lossy, i.e., compression imposes some irreversible distortion in the image in order to achieve a smaller file size. Because each compression method imposes different kinds of distortion, the distortion can act as a fingerprint of the compression method. Existing image compression methods can be grouped into a few categories — e.g., transform coding, vector quantization, subband coding, linear predictive coding, embedded coding — where all methods in the same category leave behind the same type of fingerprint in an image during compression.

In this paper, we analyze the intrinsic fingerprints of different types of image compression methods and propose a forensic system that identifies the type of the compression and provides a confidence measure in the system’s decision. Results show that the system can achieve a probability of detection of 0.82 for an image PSNR of 40 dB and even higher accuracy for lower PSNRs.

• Steven Tjoa, W. Sabrina Lin, H. Vicky Zhao, and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, April 2007
• Download: Paper, Presentation, BibTeX

  @INPROCEEDINGS{tjoa2007icassp,
    title = "Block Size Forensic Analysis in Digital Images",
    author = "Steven Tjoa and W. Sabrina Lin and H. Vicky Zhao and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    year = "2007",
    address = "Honolulu, HI",
    month = apr,
    volume = "1",
    pages = {I-633--I-636}
  };

Our work on image compression forensics attempts to identify the compression method used upon an image by analyzing its intrinsic fingerprint. Many popular image compression methods such as JPEG employ block processing. Therefore, to even begin forensic analysis for digital images, we must first address the presence of block processing on our image data. For forensic analysis of block-based coding schemes, estimating the block size is an obvious and crucial first step, because inaccurate block size estimation can possibly invalidate subsequent forensic tests.

Block artifact measurement is a well-established research area, with purposes primarily related to image restoration and distortion measurement. However, artifact measurement for the purpose of forensic analysis has not been explored. Here, we pose the following question: given a compressed image, can we detect the presence of block processing? If so, can we estimate the block size? Existing work in block artifact measurement is not tailored to answer this question due to strong assumptions placed upon the input data.

In this paper, we propose a novel scheme to detect the presence of block processing along with estimation of the block size. We first obtain a block artifact signature and then estimate the block size through maximum-likelihood estimation. Finally, we propose a binary hypothesis test that identifies the presence or absence of block processing with high detection rates and low false alarm rates.