In particular, Table 4.1 on page 14, pictured above, shows the relationships between the continuous-time Fourier series (CTFS), discrete-time Fourier series (DTFS), continuous-time Fourier transform (CTFT), and discrete-time Fourier transform (DTFT). Note the similarities and differences among the four operations:

• “Series”: periodic in time, discrete in frequency
• “Transform”: aperiodic in time, continuous in frequency
• “Continuous Time”: continuous in time, aperiodic in frequency
• “Discrete Time”: discrete in time, periodic in frequency

My motive behind writing these notes was to present the most important concepts from Oppenheim and Willsky as concisely as possible. Often, textbooks sacrifice clarity for depth. Here, I present the main concepts from separate chapters as close together as possible so that students can more easily discover the relationships between concepts rather than studying each chapter in isolation.

For anyone taking or teaching signal processing, please feel free to use these notes as you wish, and let me know what you think!

(This post was inspired by this answer that I posted to dsp.stackexchange.com.)

@INPROCEEDINGS{tjoa2011ismir,
  title = "Factorization of Overlapping Harmonic Sounds Using Approximate Matching Pursuit",
  author = "Steven K. Tjoa and K. J. Ray Liu",
  booktitle = "Proc. Int. Soc. Music Information Retrieval Conf.",
  address = "Miami, FL",
  year = "2011",
  month = oct,
  pages = "257--262"
};

Sparse coding and nonnegative matrix factorization (NMF) have revolutionized the area of music transcription. These methods usually decompose spectra from input signals as linear combinations of dictionary spectra. Often, the dictionary is either learned (as in NMF) or synthetic (e.g., harmonic templates, Gabor atoms, etc.).

Instead, what if you already have a massive overcomplete dictionary containing millions of well-labeled atoms from real-world sounds? Assuming that the dictionary is large and representative enough, there is no longer a need to learn. However, coding (i.e., finding the coefficients of the input with respect to the dictionary) remains a problem. With such a large dictionary, complexity becomes an issue.

We propose a method called Approximate Matching Pursuit (AMP) which allows you to efficiently obtain a sparse decomposition of an input vector using a massive overcomplete dictionary. AMP is a simple variant of matching pursuit or orthogonal matching pursuit. However, instead of finding an exact match between the input (or residue) and the dictionary at each iteration, we only find an approximate match. In doing so, we sacrifice a bit of accuracy to obtain significant savings in computation.

Using AMP, we decomposed input spectra of containing overlapping harmonic sounds, where the amount of polyphony varies. Our results showed that, while retrieval performance is similar to orthogonal matching pursuit based upon F-measure, execution time and computation is reduced significantly.

@INPROCEEDINGS{keegan2011waspaa,
  title = "Super-resolution of Musical Signals Using Approximate Matching Pursuit",
  author = "Brennan P. Keegan and Steven K. Tjoa and K. J. Ray Liu",
  booktitle = "Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics",
  address = "New Paltz, NY",
  year = "2011",
  month = oct,
  pages = "81--84"
};

Super-resolution is the act of increasing the resolution (or sampling rate) of a signal in an intelligent manner that preserves the signal’s ground truth information, i.e., not through some parametric method like linear or sinusoidal interpolation. For example, when you use Adobe Photoshop or GIMP to make an image larger, the new image will likely be blurry because the program does not know how to restore sharp edges.

Super-resolution of images and video has been studied extensively, but not as much for audio. When an audio signal is upsampled from 4 kHz to 44.1 kHz using traditional sinc interpolation, the upsampled signal will not contain any new information that is not already present in the 4 kHz version. As a result, high-frequency components will be absent, and the signal will sound dull and muffled.

There are a few methods for audio super-resolution. For example, Smaragdis et al. have proposed a method whose basic idea is to take the low-resolution input signal and project it upon a low-resolution basis. Using that projection (or the coefficients with respect to the low-resolution basis), a high-resolution signal is constructed by multiplying the coefficients with components from a high-resolution basis.

What basis do you use? In the literature, that basis (or overcomplete dictionary — perhaps not orthogonal) is usually obtained in one of two ways: use a synthetic basis (Gabor, etc.), or learn it from real data (NMF, K-SVD, etc.).

We ask this question: What if you already have a massive overcomplete dictionary containing millions of well-labeled atoms from real-world sounds? Can you make use of it? How? With such a large dictionary, complexity becomes an issue. Using ordinary matching pursuit, each iteration has complexity that is linear in the size of the dictionary. This is not scalable.

We propose a method called Approximate Matching Pursuit which allows you to efficiently obtain a sparse decomposition of an input vector using a massive overcomplete dictionary. Using it, we perform super-resolution of signals containing piano music. Using a dictionary composed of sounds from the University of Iowa data set, we were able to accurately estimate missing high-frequency information using low-resolution inputs sampled as low as 2-4 kHz. For more details, please see the paper above.

For more about Approximate Matching Pursuit, please see our upcoming paper in ISMIR 2011.

It is interesting to see how thesis topics evolve from advisor to advisee, yet my research area — music information retrieval — is not so removed from Helmholtz’s work on acoustics; in fact, MIR scholars still cite and recognize his contributions!

Here are the people that make up my lineage since Helmholtz, each person’s dissertation title, place of graduate study, and graduation year.

1. Hermann von Helmholtz, De fabrica systematis nervosi evertebratorum, Universität Berlin, 1842.
2. Edward Leamington Nichols, Ueber das von glühendem Platin ausgestrahlte Licht: Ein Beitrag zur allgemeinen Ausstrahlungslehr, Georg-August-Universität Göttingen, 1879.
3. Ernest Fox Nichols, Radiometric Researches in the Remote Infra-Red Spectrum, Cornell, 1897.
4. Frederic Columbus Blake, The Reflection and Transmission of Electric Waves by Screens of Resonators and by Grids, Columbia, 1906.
5. William Littell Everitt, The Calculation and Design of Alternating Current Networks Employing Triodes Operating During a Portion of a Cycle, Ohio State, 1933.
6. Karl Ralph Spangenberg, The Effect of Grid Current Flow Upon the Dynamic Characteristics of Vacuum Tube Power, Ohio State, 1937.
7. Willis Harman, Tunable Waveguide Cavity Resonators for Broadband Operation of Reflex Klystrons, Stanford, 1948.
8. John Bowman Thomas, On the Statistical Design of Demodulation Systems for Signals in Additive Noise, Stanford, 1955.
9. Kung Yao, On Some Representations and Sampling Expansions for Band Limited Signals, Princeton, 1965.
10. K. J. Ray Liu, Efficient and Reliable Parallel Processing Algorithms and Architectures for Modern Signal Processing, UCLA, 1990.
11. Steve Tjoa, Sparse and Nonnegative Factorizations for Music Understanding, Maryland, 2011.

Led by Imagine Research CEO Jay LeBoeuf, this workshop included other notable scholars and instructors from the MIR community such as Rebecca Fiebrink, Stephen Pope, Leigh Smith, George Tzanetakis, and Doug Eck.

We discussed MIR tools and applications such as machine learning, feature extraction, real-time processing, rhythm/tempo estimation, pitch detection, psychoacoustics, cover song retrieval, recommendation, collaborative filtering, and more. Participants included both students and industry practitioners. I presented a few things related to sparse coding and nonnegative matrix factorization for music transcription and source separation. This year, labs included basic programming tasks in Matlab, C++, and Python. For more details about the workshop, please see the workshop wiki.

We hope to see you at the workshop next year! Feel free to email me if you have any questions.

Here is a photo taken on the last day of the workshop right before we departed. (Click to see larger photo.)

From left: Chris Colatos, Jeff Albert, Kamlesh Lakshminarayanan, Sean Zhang, Doug Eck, Eli Stine, David Bird, Gina Collecchia, Stephen Pope, Steve Tjoa. Not pictured: Jay LeBoeuf, Rebecca Fiebrink, George Tzanetakis, Leigh Smith, Dekun Zou, Bill Paseman, John Amuedo.

On May 12, 2011, I defended my dissertation entitled “Sparse and Nonnegative Factorizations for Music Understanding“. Matrix factorizations have become popular for performing music transcription and source separation. However, the basic factorization algorithms have trouble decomposing highly polyphonic music. In this dissertation, I proposed factorization methods that are specifically suited for analyzing musical signals, including a method that can efficiently use a musical spectral dictionary if one already exists.

On May 20, I graduated.

On May 28, I boarded a one-way flight to San Francisco. I left my 1999 Saturn SC2 in New Jersey, shipped nineteen boxes to San Francisco, and sold everything else.

On June 6, I began work at Imagine Research, Inc., in San Francisco. We build software that understands sound. My position is sponsored by a National Science Foundation (NSF) Postdoctoral Research Fellowship and administered by the American Society for Engineering Education (ASEE).

On June 14, I signed a lease for my new apartment. After two weeks of apartment hunting, I found a wonderful place in the Sunset district of San Francisco for only $800 per month with views of Golden Gate Park right across the street.

From June 27 to July 1, I attended and helped instruct a workshop on music information retrieval at the Center for Computer Research in Music and Acoustics (CCRMA) at Stanford. We all had a great time talking about music, signal processing, and machine learning.

Writing Strategies

1. Writing is not revising. When you are writing, just write! Don’t stop, don’t backspace, don’t correct. Keep moving forward. Forward, forward, forward. Revise later. Perfectionism is your worst enemy, particularly at early stages.
2. Write in layers. First, crude main ideas. Then, fill in the gaps. Finally, citations.
3. Start with what you already know. “First drafts come from the heart.” In verbal conversation, you effortlessly tell the other person what you already know. When writing, pretend to tell someone the story. Then write the story. You may actually find it useful to record your own voice as you tell yourself the story so you can play it back as you write.
4. Write at least one page of your dissertation per day. Often more, but at least one page. And it doesn’t have to be in order. Any one page is fine.
5. Allow rough drafts. Launch quickly and iterate rapidly. The rough draft should be very rough. Don’t worry about errors. Just write, baby. But also turn around revisions quickly.
6. Use an outline. It helps keep a coherent flow throughout your dissertation. Try constructing a mindmap — a “tree” of ideas with the core (root) idea in the center and branch ideas around the root, and so on.
7. “Powerpoint” your ideas. Then turn your slides into prose.
8. Seek feedback regularly. Feedback helps turn revisions around quickly, too.
9. At first, ignore your audience. Write for yourself, first. Then, once you have written a fair amount, consider your audience, and revise. Write outward, not inward.
10. Writing out of order is fine, perhaps even preferable. I made the mistake of writing my early research papers in order from introduction to conclusion. However, research is always so unpredictable and amorphous that the important middle sections would always change. Subsequently, so would the introduction and conclusion. Now, I begin the middle sections first, then write the introduction, conclusion, and abstract last.

Goals and Planning

11. Set many small goals. Break writing tasks into small sections. Incorporate transitions later.
12. Use organic goal setting. Only set goals for each week. Believe it or not, do not set rigid goals for a long period, e.g., semester. Shifting the finish line is deadly. Shift it once, and you get comfortable with it.
13. Set specific, measurable goals. “Finish dissertation” is too broad. “Finish chapter 1″ is still too broad. “Write five pages of subsection 2.1 from 1 pm to 1:45 pm” is better. Think in terms of pages per chapter or pages per section.
14. Plan your progress. Keep track, e.g., with a calendar.
15. Time yourself! You can use website timers. I am timing myself right now. I started this article at 8:53 pm and completely ended it at 9:39 pm. (Small typo corrections came after.)
16. The optimal chunk of time for contiguous writing is 45 minutes.
17. Aim for 5-7 hours of writing on a writing day. Do not exceed 7 hours. And not every day should be a dedicated writing day.
18. Take fifteen-minute breaks. Get up and walk around. Observe your surroundings. “To be a good writer, one needs to be a good observer.”

Environment

19. Work in a sparse, uncluttered space. Visual clutter tends to clutter the mind.
20. Write as early in the day as possible. That is when you are freshest. If you must procrastinate, do not, under any circumstances, do it in the morning.
21. Prepare the night before. Set your goals for the next day. That will make the following morning more productive.
22. Change physical environments. I have heard this advice often. Sometimes, after working in the same place for so long, you grow too comfortable and complacent around it. Try the library or the cafe. Be around others who are also working. Of course, avoid areas that are too disruptive. Classical or soft instrumental music is fine. Music with words is naturally distracting to humans, whether we consciously realize it or not.
23. Write with others. Being around others united in the same general goal is energizing. That type of work ethic becomes contagious.
24. But don’t compare with others. Focus on yourself.

Internal Attitudes and Self-Improvement

25. Write something every day, even if it is not on the dissertation. Every skill takes 10,000 hours to master. That includes writing. Writing is like exercising a muscle. You only get good at it through practice. That can include a personal journal, emails, a blog post like this one, etc.
26. Feel good first. Then write. A burdened mind is an unproductive mind. Negative thoughts are blocking. Get everything else in your life in order to the greatest extent possible. Think positive. Then write.
27. If you cannot get started, write about what troubles you. Notice what you keep thinking about.
28. If you still cannot get started, ask yourself, “Why did I care?” What made you start this project in the first place? Writer’s block comes from a lack of emotional engagement with one’s work. As time passes, it is natural to become more emotionally detached from the work. Step back, and remind yourself why you care and what makes your project so awesome.
29. Reward yourself. Reward positive behavior. Reward your accomplishments, but only when you actually reach a milestone. Otherwise, don’t. (I asked, “But when I have momentum, I hate to disrupt my momentum with something frivolous like a reward. Is that okay to skip rewards?” Her answer: “No.” Because if you rely on momentum, then something must be wrong with the system. Therefore, improve the system. Ask yourself why you are so reliant on momentum in the first place. Otherwise, writer’s block will inevitably occur again.)

… and a final reminder

30. Activity is not necessarily productivity. Output is what counts.

At AdMIRe 2009, I remember hearing renowned researcher Malcolm Slaney say the following, paraphrased: “Computers should solve computer problems, and humans should solve human problems.” I agree 100 percent. Yet, in the exciting and sometimes amusing world of machine learning, this principle is violated all the time. The ECCV paper is just one example.

I can understand why this paper was accepted. To be fair, the paper is well written. (Having reviewed many papers, I find that proper English can go a long way.) It uses standard machine learning approaches that have been applied to similar problems such as face recognition and gender recognition, e.g., PCA, neural networks, and gradient descent. Papers that ask provocative questions and address undersaturated problems tend to be accepted, as they should. Then again, as a reviewer, I would claim that there exists no ground truth for beauty, and so the proposed system can spit out whatever nonsense it wants, and there would be no way to validate or invalidate it.

More importantly, this research is just plain mean. People are sensitive about their appearance. To propose a computerized system that can quantify and rank the beauty of human faces is provocative at best and unprofessional at worst. Oh, and all of the test subjects are female. Draw your own conclusions.

Here’s one clever piece of marketing: the authors derive two mathematical operators, beautify and beastify, where one is simply the negative of the other. (Beastify? Seriously??) Applying the beautify operator to an image makes the person more beautiful, and the beastify operator does the opposite. How cute.

The practitioner in me can never imagine such a system being used for any practical purpose. I can find utility — at least an iota of utility — in related problems such as face recognition, speaker recognition, gender recognition, genre recognition (for music), ethnicity recognition, language recognition, and gesture recognition. But where would beauty measurement be used? The Miss Universe Pageant? American Idol? Hollywood auditions? Facebook? (Actually, I wouldn’t put that last one outside the realm of possibility.)

Computers should solve computer problems. “Multiply 9832473 by 29834″ is a computer problem. “Compress this 12-megapixel image” is a computer problem. “Encode this bitstream so that it is cryptographically secure and robust to channel errors” is a computer problem. Let’s leave the human problems for humans.

I wrote the template myself. A lot of it is self-explanatory. The template relies mainly on section*, subparagraph, and itemize. Comments:

1. The fancyhdr package allows custom headers and footers. That’s how I made the three-part footer.
2. I used renewcommand to remove item labels.
3. The Publications section uses the multibib package to embed a bibliography (or multiple bibliographies) in the middle of a document. With multibib, here are the commands for an entire build:
   
pdflatex cv.tex
bibtex c.aux
bibtex j.aux
pdflatex cv.tex
pdflatex cv.tex


Feel free to modify. I would appreciate at least some modification so your resumé doesn’t look exactly the same as mine. Let me know how it goes. Good luck.

But what about academia? For example, how well does my research group perform on the Joel Test? After all, we write software, too. Should academic researchers be held to the same standard as a full-time software engineer in industry? I asked Stack Overflow and got one response that says no, and I agree. Academia and industry have different goals and different resources. Most research groups I know don’t have full-time testers. (Duh.) They don’t make daily builds, and they certainly don’t have the best tools money can buy.

Nevertheless, item 2 of this test reads, “Can you make a build in one step?” Spolsky elaborates:

… how many steps does it take to make a shipping build from the latest source snapshot? … If the process takes any more than one step, it is prone to errors. And when you get closer to shipping, you want to have a very fast cycle of fixing the “last” bug, making the final EXEs, etc. If it takes 20 steps to compile the code, run the installation builder, etc., you’re going to go crazy and you’re going to make silly mistakes.

I definitely did not have a one-step build for my research projects. And as predicted by Spolsky, without a one-step build, I went crazy. I could not remember the entire process to produce my results and incorporate them into my paper, especially after a few days of programming inactivity. When my numerical results changed, I had to repopulate the tables in the paper. All of these things could have been avoided by using a one-step build.

So I wrote one in Python. It is a single file named build.py. As shown in Figure 1, this build accepts three inputs: source code, data, and a paper template. The primary output is a paper — the ultimate goal of any research project.

The build has three internal components:

1. Compile the source code to generate an executable. This step is most important for languages such as C or C++. In languages such as Python or Matlab, you don’t need to worry about this step.
2. Feed data into the executable to generate numerical results. I save these results to a file either as text or a serialized format (e.g., using pickle in Python).
3. Feed results into a paper template to generate the paper. I have a paper_template.tex file that contains placeholders to be filled in. Then, I use the string.Template module in Python to fill in the results. (Perl is a good choice, too.) Finally, I use pdflatex to compile the tex file into a PDF document.

To run the build, I simply type python build.py at the command line. That is the beauty of the one-step build: at the single push of a button, you generate a new program, new results, and a new paper. If you are really lazy, you could even tell Linux to automatically run this script when you turn on the computer.

If you are having trouble managing the software for your research projects, write a one-step build. Good luck, and let me know how it goes.

I noticed that the use of python to do signal processing tasks is slowly increasing (saw it mentioned in some papers). I am curious to know what are the advantages/disadvantages of python vis-a-vis C or Matlab. What do you think are the main pros and cons that have led you to use python?

Because this precise question — why Python over Matlab? — has become so popular, I thought I would offer my reasons for switching.

1. Python is totally free. Matlab is not. Undergrads living on campus are able to get Matlab for free through… ahem… alternative means. As a grad student disconnected from on-campus life, it is hard for me to use such… ahem… alternative means to get a copy of Matlab, especially for Linux.
2. Python is a general-purpose language, i.e., it is used for scientific computing, enterprise software, web design, back end, front end, and everything in between. Matlab is purely for scientific computing, although it does a great job at it.
3. Python syntax is beautiful. Once you get over the use of meaningful whitespace, you realize how much it makes sense. Famous entrepreneur and investor Paul Graham mentions a friend who uses Python because “he likes the way source code looks.” Graham explains further:
   

   That may seem a frivolous reason to choose one language over another. But it is not so frivolous as it sounds: when you program, you spend more time reading code than writing it. You push blobs of source code around the way a sculptor does blobs of clay. So a language that makes source code ugly is maddening to an exacting programmer, as clay full of lumps would be to a sculptor.

4. Python is inherently object oriented. Almost everything is an object: strings, lists, dictionaries, tuples, functions, classes, and more. The implied usefulness is that these things each have their own members and methods that encapsulate its functionality and information.
5. Python is high level, easy to learn, and fast to develop. (So is Matlab.) Program scripts are easily profiled and debugged. Although C++ may save you one minute of computation, Python may save you one month of development.
6. Python has so many cool features that makes tasks that are difficult in other languages easy. For example, see these features.
7. Python is fast enough. Bottlenecks can be addressed by writing a bit of C (something I have not yet needed to do), or you can parallelize your code easily in Python. For what it’s worth, Python was written in C.
8. Python is popular and has a great community among programmers, scientists, mathematicians, and engineers, making it easy to find help on the Internet. At the moment, “Python” is the tenth most popular tag on StackOverflow (which, by the way, may be my favorite site on the Internet), and Python is the second most popular language among Project Euler members.
9. Python has great libraries. It also makes any library you want already available. Want something that does hierarchical clustering? There are a bunch of Python packages that do exactly that.
10. Python packages can do nearly everything Matlab can do for signal processing. NumPy is like the Matlab core, SciPy is like Matlab toolboxes, Matplotlib lets you print pretty graphs, and IPython emulates the Matlab desktop environment. Over the years, additions and revisions to NumPy/SciPy were written to make Matlab users comfortable, e.g., PyLab. Many sites offer help regarding the transition from Matlab to NumPy/SciPy, such as this.
11. The demand for Python programmers is increasing. Many large companies (e.g., Google, Facebook) and small startups are hiring people with Python experience. Even the finance industry is interested. The U.S. Securities and Exchange Commission recently proposed a mandate that would require securities issuers to submit a computer program that maps the logic flow of funds, and that “this computer program be filed… in Python”.

Disadvantages? Python is probably easier to learn from scratch than C++, but it still takes time. If you are already comfortable with Matlab, and you have Matlab, then you should keep using Matlab. But if you want a language that is free, flexible, popular, and may make you more marketable, then give Python a try. Believe me, I like Matlab. I just like Python better.

• Steven K. Tjoa and K. J. Ray Liu
• Int. Soc. Music Information Retrieval Conf., August 2010
• Download: Paper, Poster, BibTeX

  @INPROCEEDINGS{tjoa2010ismir,
    title = "Musical Instrument Recognition Using Biologically Inspired Filtering of Temporal Dictionary Atoms",
    author = "Steven K. Tjoa and K. J. Ray Liu",
    booktitle = "Proc. Int. Soc. Music Information Retrieval Conf.",
    address = "Utrecht, Netherlands",
    year = "2010",
    month = aug,
    pages = "435--440"
  };

Most musical instrument recognition systems rely upon spectral information to classify sounds. Can temporal information improve classification accuracy even further?

Evidence from the psychoacoustic literature suggests that both spectral and temporal content carry information about acoustic timbre through the human auditory system. In other words, because humans recognize musical instruments so effortlessly, then perhaps machines would also benefit from a combination of spectral and temporal information. Researchers use a model known as the cortical representation to emulate the information output from the middle stage of the human auditory system. Although some engineers have tried to build music information retrieval (MIR) systems that use the cortical representation, its high dimensionality makes it difficult to employ in practical systems.

But what if we can embody the traits of the cortical representation into another representation that also includes spectral and temporal content? One such candidate is nonnegative matrix factorization (NMF), a tool that can extract spectral and temporal information from spectrograms.

In this paper, we test the usefulness of temporal information extracted using NMF in instrument recognition. We mimic the multiresolution aspect of the cortical representation by using a multiresolution gamma filterbank that parameterizes the shape of temporal envelopes. Our results show that this method of temporal processing can classify among isolated sounds among 24 instrument classes with 92.3% accuracy.
]]> http://stevetjoa.com/334/feed 0 http://stevetjoa.com/269 http://stevetjoa.com/269#comments Mon, 22 Mar 2010 07:50:51 +0000 Steve http://stevetjoa.com/?p=269 Multiplicative Update Rules for Nonnegative Matrix Factorization with Co-occurrence Constraints

• Steven K. Tjoa and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, March 2010
• Download: Paper, Poster, BibTeX

  @INPROCEEDINGS{tjoa2010icassp_cooccurrence,
    title = "Multiplicative Update Rules for Nonnegative Matrix Factorization with Co-occurrence Constraints",
    author = "Steven K. Tjoa and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    address = "Dallas, TX",
    year = "2010",
    month = mar,
    pages = "449--452"
  };

Nonnegative matrix factorization (NMF) has become a popular tool for discovering structure in a variety of signals. When applied to a musical audio signal, NMF builds a set of dictionary atoms that represent the individual musical sources in the signal. To perform music transcription, we map the learned dictionary atoms to musical notes and beats.

The basic formulation of NMF has the notable disadvantage that sources may require more than a single dictionary atom in order to be approximated accurately. For example, one note played by a violin may require multiple dictionary atoms to be accurately represented due to the vibrato induced by the performer. In the presence of many other musical sources, the correspondence between atoms becomes unclear.

In this paper, we introduce three new update rules to enforce dependence among dictionary atoms by incorporating co-occurrence constraints into NMF. These co-occurrence constraints have shown to be useful for describing sources with multiple, co-occurring dictionary atoms by grouping similar atoms into sets. The proposed rules are conceptually simple, easy to implement, and effective for describing sources using multiple dictionary atoms.

• Steven K. Tjoa, Matthew C. Stamm, W. Sabrina Lin, and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, March 2010
• Download: Paper, Poster, BibTeX

  @INPROCEEDINGS{tjoa2010icassp_harmonic,
    title = "Harmonic Variable-Size Dictionary Learning for Music Source Separation",
    author = "Steven K. Tjoa and Matthew C. Stamm and W. Sabrina Lin and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    address = "Dallas, TX",
    year = "2010",
    month = mar,
    pages = "413--416"
  };

Methods that employ dictionary learning and sparse coding have become popular for discovering structure in acoustic signals. Unfortunately, these methods also share a common limitation. When there is significant spectral-temporal overlap among the dictionary atoms present in a signal, it becomes difficult for these methods to learn atoms properly. Often, information from multiple atoms is represented as a single atom by the learning procedure. If an atom in the output dictionary contains musical information from multiple sources, transcription and source separation cannot be accurately performed.

In this paper, we propose a novel dictionary learning method that performs well despite the presence of spectral-temporal overlap among dictionary atoms. Our method imposes a harmonic constraint that restricts each atom to represent at most one pitch. Furthermore, our method is flexible by allowing the size of the dictionary to grow based upon the complexity of the input signal. Our method consistently achieves higher recall and precision than other well-known dictionary learning algorithms.

• Matthew C. Stamm, Steven K. Tjoa, W. Sabrina Lin, and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, March 2010
• Download: Paper, BibTeX

  @INPROCEEDINGS{stamm2010icassp_antiforensics,
    title = "Anti-Forensics of {JPEG} Compression",
    author = "Matthew C. Stamm and Steven K. Tjoa and W. Sabrina Lin and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    address = "Dallas, TX",
    year = "2010",
    month = mar,
    pages = "1694--1697"
  };

Digital image forensics is a relatively new field that has only become popular within the last seven years. A newer body of research involves anti-forensics — methods of processing images that disguise evidence of earlier image processing or tampering. By altering the forensically significant properties in an image, existing image forensic systems may be rendered useless. Like many areas of security, we benefit from the study of anti-forensics by understanding the attacks that are most likely to occur before they actually do. Then, we can refine our image forensic systems further.

Earlier, our group proposed the first forensic system that identifies which compression method was used upon a digital image. In this paper, we propose an anti-forensic method for defeating the identification of JPEG compression in an image. Our method alters the intrinsic fingerprint of JPEG-compressed images in the transform domain such that the attacked image is perceptually similar to the image before attack. To our knowledge, this work is the first to propose a method specifically designed to defeat compression identification systems, and we anticipate that more work will follow.

• W. Sabrina Lin, Steven K. Tjoa, H. Vicky Zhao, and K. J. Ray Liu
• IEEE Trans. Information Forensics and Security, September 2009
• Download: Paper, BibTeX

  @ARTICLE{lin2009tifs,
    title = "Digital Image Source Coder Forensics via Intrinsic Fingerprints",
    author = "W. Sabrina Lin and Steven K. Tjoa and H. Vicky Zhao and K. J. Ray Liu",
    journal = "IEEE Trans. Information Forensics and Security",
    year = "2009",
    month = sep,
    volume = "4",
    number = "3",
    pages = "460--475"
  };

Multimedia forensic methods allow us to verify and maintain the integrity of our multimedia data. For example, we can embed a watermark into a digital image to bind the identity of its owner to the image itself. However, traditional forensic approaches such as watermarking are not applicable in many real-world scenarios, for example, when we do not have access to the original data.

In this paper, we investigate the use of intrinsic fingerprints — subsets of data which are, or have become, an intrinsic part of the data in question — for formulating a forensic methodology that can identify the compression history of a digital image. By examining the intrinsic fingerprints in an image, we can tell what compression method (e.g., JPEG, etc.) was used in order to determine the origin of the image and thereby assess its authenticity.

Although there exist methods that estimate parameters of individual compression schemes, to our knowledge, our work is the first to address the problem of image compression identification and offer a unified forensic framework incorporating multiple encoders.

• Steven Tjoa, W. Sabrina Lin, and K. J. Ray Liu
• IEEE Int. Conf. Image Processing, September 2007
• Download: Paper, Presentation, BibTeX

  @INPROCEEDINGS{tjoa2007icip,
    title = "Transform Coder Classification for Digital Image Forensics",
    author = "Steven Tjoa and W. Sabrina Lin and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. on Image Processing",
    address = "San Antonio, TX",
    year = "2007",
    month = sep,
    volume = "6",
    pages = {VI-105--VI-108},
  };

Our TIFS paper was the first work to offer a unified forensic framework that can identify the compression method used upon an image. Among these compression methods are transform coders — compression methods that apply some form of entropy coding in the transform domain (e.g., via discrete cosine transform, wavelet transform, etc.). Transform coding is undoubtedly the most popular category of image coders in use today. Therefore, the forensic problem of transform coder classification deserves attention.

In this paper, we propose a method that can identify which transform was used upon an image during the compression process. By analyzing the transform coefficient histogram obtained after applying a candidate transform, we can determine if the image was compressed using the transform tested. Results show that forensic classification accuracy is high when testing from among six well-known transforms.

To our knowledge, this work is the first to address the problem of transform coder classification and propose a working solution.

• W. Sabrina Lin, Steven Tjoa, H. Vicky Zhao, and K. J. Ray Liu
• IEEE Int. Conf. Multimedia and Expo, July 2007
• Download: Paper, BibTeX

  @INPROCEEDINGS{lin2007icme,
    title = "Image Source Coding Forensics via Intrinsic Fingerprints",
    author = "W. Sabrina Lin and Steven Tjoa and H. Vicky Zhao and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Multimedia and Expo",
    address = "Beijing, China",
    year = "2007",
    month = jul,
    pages = "1127--1130",
  };

The most popular image compression schemes in use today are lossy, i.e., compression imposes some irreversible distortion in the image in order to achieve a smaller file size. Because each compression method imposes different kinds of distortion, the distortion can act as a fingerprint of the compression method. Existing image compression methods can be grouped into a few categories — e.g., transform coding, vector quantization, subband coding, linear predictive coding, embedded coding — where all methods in the same category leave behind the same type of fingerprint in an image during compression.

In this paper, we analyze the intrinsic fingerprints of different types of image compression methods and propose a forensic system that identifies the type of the compression and provides a confidence measure in the system’s decision. Results show that the system can achieve a probability of detection of 0.82 for an image PSNR of 40 dB and even higher accuracy for lower PSNRs.

• Steven Tjoa, W. Sabrina Lin, H. Vicky Zhao, and K. J. Ray Liu
• IEEE Int. Conf. Acoustics, Speech, and Signal Processing, April 2007
• Download: Paper, Presentation, BibTeX

  @INPROCEEDINGS{tjoa2007icassp,
    title = "Block Size Forensic Analysis in Digital Images",
    author = "Steven Tjoa and W. Sabrina Lin and H. Vicky Zhao and K. J. Ray Liu",
    booktitle = "Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing",
    year = "2007",
    address = "Honolulu, HI",
    month = apr,
    volume = "1",
    pages = {I-633--I-636}
  };

Our work on image compression forensics attempts to identify the compression method used upon an image by analyzing its intrinsic fingerprint. Many popular image compression methods such as JPEG employ block processing. Therefore, to even begin forensic analysis for digital images, we must first address the presence of block processing on our image data. For forensic analysis of block-based coding schemes, estimating the block size is an obvious and crucial first step, because inaccurate block size estimation can possibly invalidate subsequent forensic tests.

Block artifact measurement is a well-established research area, with purposes primarily related to image restoration and distortion measurement. However, artifact measurement for the purpose of forensic analysis has not been explored. Here, we pose the following question: given a compressed image, can we detect the presence of block processing? If so, can we estimate the block size? Existing work in block artifact measurement is not tailored to answer this question due to strong assumptions placed upon the input data.

In this paper, we propose a novel scheme to detect the presence of block processing along with estimation of the block size. We first obtain a block artifact signature and then estimate the block size through maximum-likelihood estimation. Finally, we propose a binary hypothesis test that identifies the presence or absence of block processing with high detection rates and low false alarm rates.