Real Time Implementation for Laugh Track Removal

Overview In order to make best use of the Laugh Track Assassinator's algorithm, we need to be able to run it in real time with as wide a range of source materials as possible. To accomplish this lofty goal, we have implemented a DirectShow filter. DirectShow is Microsoft's technology for manipulating media on the Windows platform. Nearly all media players, such as Windows Media Player, Media Player Classic, and various DVD program, use DirectShow to render video and audio. By writing a DirectShow filter, our algorithm can be used to manipulate nearly any type of media, be it a DVD, an encoded movie, or a live TV video stream.

Direct Show All DirectShow operations are based on filters. Filters describe the translation of data from one source or type to another. DirectShow automatically finds what filters are needed to play a particular media file. The generated graph can be visualized in Microsoft's GraphEdit program. Here is what the generated graph looks like for a source video file with the Laugh Track Assassinator filter inserted:

Filter Graph This is the filter graph generated by Microsoft DirectShow with the Laugh Track Assassinator filter already inserted.DirectShow has generated an AVI splitter to transform the file data into an audio and video stream. The video is then sent to the ffdshow Video Decoder filter, which is then sent to the Video Renderer. The audio stream is sent from the file, through the MP3 Decoder, an AC3Filter, the Laugh Track Assassinator, and finally rendered to the speakers through the DirectSound filter. To create the DirectShow-compatible filter we used Microsoft's Windows SDK, and rewrote the audio transform filter example. (The Windows SDK can be downloaded from Microsoft here). We then coded the two main steps in our algorithm: a low pass filter and a threshold detection scheme.

Low Pass Filter In order to find a balance between frequency resolution and speed, we chose a 1000-point finite impulse response low pass filter. We had Matlab generate the one thousand filter weights, and then we converted them into a C++ format suitable for DirectShow. Since the filter requires 1000 previous samples to calculate one low pass filtered sample, we created a 1000 point circular buffer to hold the last 1000 samples of the input at any given time.

Finite State Machine The final step in our removal algorithm requires a threshold detection in both amplitude (vertical) and time (horizontal). The requirement for a time-based threshold meant we had to delay the input signal by at least the width of the horizontal threshold. In the end we decided on a 1 second delay to allow for the width threshold of 0.8 seconds, as well as making it easier to resynchronize the video signal with the audio afterward. The actual threshold test are performed by means of a finite state machine. Here is an overview of the FSM:

State Diagram This is the state diagram for the real time Laugh Track Assassinator filter.As soon as the amplitude threshold for the low-passed signal is met, the filter enters the Possible Laugh state. From here, if the signal falls below the falling amplitude threshold, the machine returns to the Initial State. If the width threshold is reached, then the machine enters the Laugh Detected state, and continually suppress the output audio. During this transition, the last second of audio is also eliminated from the output buffer. Since the filter is delayed by at least 1 second, as long as the width threshold is less than this value, the output will reflect the proper changes. Finally, as soon as the falling amplitude threshold is passed, the machine again returns to its Initial State.

Optimization The scheme described above generates a working laugh track removal filter. One big problem, however, is speed. Though the above system works on a high-end computer for a real time video signal, any moderate computer will not be able to run it. The chief problem is in the low pass filtering phase. The low pass filter takes 1000 samples to calculate 1 sample of the low passed signal. This means there are roughly 2000 operations (1000 additions and 1000 multiplications) per sample. With a standard sampling rate of 44.1 kHz, that means the filter uses 44.1 million operations per second. This is generally unacceptable when accounting for the overhead in the filtering process. To speed the filter up, we must first realize that we do not need an accurate low pass signal value for every sample. In fact, if we took every 1000 samples of the low pass signal, we would only need to perform 2 operations per sample to get the same results. Using this method gives us a speed increase of 1000x by effectively sampling the low pass filter output. Generally, strictly sampling a signal like this produces rather severe aliasing. But, since the signal is already low-pass-filtered, the signal has already gone anti-aliasing processing, and the optimization works out.

Download and Installation The Laugh Track Assassinator filter can be downloaded here. Since this is implemented as a DirectShow filter, this will only run on Windows-based computers. To install, follow these steps: Copy the LaughTrackAssassinator.dll file into your C:\Windows\System32 folder. Open a command prompt window (Start->Run->“cmd”). Type “regsvr32 LaughTrackAssassinator.dll” and press enter in the command box. The Laugh Track Assassinator is now registered with DirectShow. Now that the filter is registered, most any DirectShow based media player should be able to use the filter on any media. We tested the filter with Media Player Classic, a free media player that can be downloaded here. Here are the steps to get it to work: Open Media Player Classic. Go to View->Options->External Filters. Select “Add filter...”. Select the Laugh Track Assassinator from the list of available filters. Select the newly added filter, and select the “Prefer” radio button. You can now view any media that has audio and it will automatically run the Laugh Track Assassinator. In order to get video back in sync with the audio, you can set the audio delay to 500ms in Media Player Classic by using the + and – keys on the numpad of your keyboard.