Skip to content Skip to navigation Skip to collection information
Red cup detection and replacement is an application of image recognition technology that could be viable today. Our demonstration shows that the technology has promise, but there are definitely still large technical hurdles to be overcome. Our template matching based cup recognition program ignores important issues like partially obscured cups, rotated cups, or deformed (crushed) cups. Template matching is ill suited to deal with these cases, but a “professional” version of the algorithm could use a combination of computer vision techniques (edge detection/ shape detection, color blob detection, size transformation, multiple templates, etc.) to improve both accuracy and runtime.
On the replacement end of things, the program could be vastly improved by a more structured and complete replacement image bank. Consistent sizing of the bank images (so the algorithm knows the whole size of the image is the whole size of the replacement region) would improve accuracy and speed up runtime by an entire polynomial order. While we avoided the question of runtime in our demonstration, any practical implementations would need to run much more efficiently. Red cup detection seems particularly suited to a smartphone application, allowing users to sanitize pictures as they are taken. Yet in its current form the algorithm would be prohibitively costly in terms of both time and battery life in a mobile environment. Offloading the image bank search to the cloud could greatly improve the load on the phone since the cup detection and meshing parts of the algorithm are relatively efficient. The phone could find suspect regions in the image, offload the detected regions to the cloud where a server could quickly search a large and organized image bank, and then locally re-integrate the replaced regions to alleviate privacy concerns and keep the whole image in the user’s possession.
Our process leaves lots of room to improve the final image mesh step too. A better and more natural blend (bicubic instead of linear blending, plus blended corners) would make the final results look much more realistic. The blending algorithm could use edge detection to keep features continuous, avoiding blur around sharp edges that would be obvious to the viewer in the final image. Matching lighting and highlight intensity with the original image would also greatly improve the quality of the final product.
Additionally, we could implement different replacement algorithms depending on characteristics of the area surrounding the cups. A technique known as quilting works well for patterns, but not for unique objects such as hands. Quilting avoids many of the challenges posed by using a replacement image bank, because it uses other areas of the original image to “quilt” over the hole left by the cup. This means that quilting could be applied to a hole left by any image, eliminating the need for a huge bank of replacement images. Ideally, our program could choose the better replacement method based upon characteristics of the surrounding image.
On an even grander scale, the algorithm could be generalized to allow users to select and replace arbitrary offending items in an image (houses on a scenic hillside, aggie caps at a longhorns game, etc.). This presents new challenges in detection and in the construction of the image bank, but it could foreseeably work using the resources of large image databases like Flickr or the Google Images cache.
Ultimately, our algorithm is only a proof of concept. Future researchers could focus on improving many things from the accuracy of detection to the seamlessness of reintegration to the accuracy and computational complexity of an image bank search. The distinctive shape and color of red cups lend them well to computer detection and replacement, but the algorithms principles could be generalized to other objects. There is a long way to go to make a seamless professional implementation of the technology, but all the pieces exist with today.
More about this module: Metadata | Downloads | Version History
This work is licensed by Blake Miller, John Slack, and Thomas Ladd under a Creative Commons Attribution License (CC-BY 3.0), and is an Open Educational Resource.
Last edited by John Slack on Dec 19, 2011 4:03 pm -0600.
