Posts Tagged ‘Testing’

Video Testing for Encoder Manufacturers

This paper explores the challenges Video Encoder (MPEG, JPEG, WAVLET Compression) Equipment Manufacturers face when assessing video quality. Assessing video quality ultimately depends on the customer’s reaction on their new display (PC, POD, TV, etc.).


For visual and audio data, some loss of quality can be tolerated without losing the essential nature of the data. By taking advantage of the limitations of the human sensory system, a great deal of space can be saved while producing an output which is nearly indistinguishable from the original. These lossy data compression methods typically offer a three-way tradeoff between compression speed, compressed data size and quality loss.


The question must be asked. How much data can be removed before the customer has a violent reaction. So testing is increasing important. The test setup simply stated is

* Start with a known video sequence.

* Compress the video sequence.

* Decode the processed video sequence.

* Capture the processed video sequence.

* Display the original and processed video sequences.

* Bring in experts to subjectively vote.


Complexity arises as

* New Video Processing systems may need new equipment to playback the video sequences.

* The original and processed video sequences should be displayed in random orders.

* Expert viewers are expensive and do not produce repeatable results.


So a need arises to perform quantitative, repeatable objective scoring of the video.


Previous Options

Each vendor builds unique test equipment to verify their new algorithms. So the first job is to debug the test equipment before it can be used to verify a new design. Debugging the test equipment can take as long if not longer than debugging the display equipment.

Easier Solution

To streamline the process, equipment for video quality testing needs to be defined, which can capture, play, and analyze any two video sequences. Further, as new input/output modules are continuously under development, the test equipment should use an open-architecture approach to ease upgradeability.


Video Clarity defined the ClearView product line with these objectives in mind.

* Capture video sequences in as many formats as possible.

* Convert all video sequences to user-selectable resolution.

* Translate all video sequences to uncompressed Y’CbCr 4:2:2 or RGB 4:4:4.

* Support 8 and 10-bit data paths with upgradeability to future 12-bit modes.

* Store the video sequences as frames (fields) so that they can be played at any rate.

* Display the video sequences in real time in multiple viewing modes.

* VTR controls – play, shuttle, jog, pause

* Integer-based zoom & pan.

* Apply objective metrics to the video sequences.

* Export pieces of video sequences to further analyze off-line.

* Use a standard operating system so that the operator can run 3rd party analysis applications.


To further simplify the work flow, any video sequence can be played; while capturing another video sequence, thus, combining the video server and capture device into one unit. By doing this, the original source is already inside the test equipment so captured content alignment is easily obtained.


The original and processed video sequences can be displayed – side-by-side, mirrored, or seamless split – on a single display. This eliminates the need to calibrate two separate displays.


ClearView applies various objective metrics to the video sequences, generates graphs, and calculates an objective score. ClearView includes PSNR, Spatial Information and Temporal Information (as proposed in ITU-T P.9210). These metrics are the basis for more sophisticate metrics like VQM, JND, which are being analyzed by organizations like VQEG, VPQM, and SMPTE.


ClearView displays multiple video sequences, even if they are played at different rates (i.e. mobile phone video compared to TV) for the expert viewers; while recording the objective metric scores. While the MOS cannot be repeated, the objective metric can, easily and readily.


Since the system is based on an open, Windows-based architecture, any objective measurement algorithm can be modeled off-line using the stored video sequences.


Benefits

* Repeatable tests, quantitative results, and a streamlined setup.

* Analyze 2 video sequences in real-time up to 1080P.

* Input virtually any file type or capture from any digital or analog source.

* Multiple viewing modes are presented on a single display – no need to calibrate 2 separate Television displays to compare video sequences.

* Integrated uncompressed, high definition Video Server and Capture Device.

* Ability to Play 2 fully uncompressed, HD Streams in Real-Time.

* Hybrid Solution with Integrated Objective Metrics and Subjective Viewing Modes.


Case Examples

In all of the following examples, the tests can be performed using software mockups of the actual hardware unit. ClearView exports video sequences as files and accepts video files as inputs.


An Encoder manufacturer needs to

* Source for their video processing or encoders in SDI, Component, or DVI.

* Capture the output of their unit.

* Capture the output of their unit after it has been decoded.

* Visually inspect the new algorithm compared to the original and/or previous algorithms.

* Generate a Score for repeatability.


Summary

ClearView takes advantage of the high-reliability of today’s off-the-shelf computer platforms. This ensures that products are made with state-of-the-art hardware, while at the same time avoiding the high cost of custom designs.


ClearView provides broadcasters, video researchers, compression developers with the unique ability to capture, play-out, and analyze video sequences. Objective measurements are generated and logged for repeatable tests.

Bill Reckwerdt, Vice President Marketing & Business Development
Bill’s extensive career spans 20 years in the digital video and medical imaging industries. He brings to Video Clarity expertise in compression, digital transmission, and video servers.
To contact Bill directly send an email to bill@videoclarity.com or visit our http://www.videoclarity.com

Be the first to comment - What do you think?  Posted by admin - August 12, 2009 at 1:55 pm

Categories: Video   Tags: , , ,

Repeatable Video Testing Tools for Equipment Manufacturers

This paper explores the challenges Video Equipment Manufacturers face when assessing video quality. Assessing video quality ultimately depends on the customer’s reaction on their new display (PC, POD, TV, etc.).


Over the next decade, it is expected that digital television penetration into the general television market will increase from less than 5% in 2003 to more than 50% by 2010.


Currently, the FCC requires that 50% of televisions 36 inches or larger have integrated digital capability by July 2005. By July 2007, the FCC will require that all new televisions 13 inches or larger must incorporate a digital tuner. Obviously the FCC’s ruling provides additional fuel for the growth of the DTV market.


So testing is increasing important. The test setup simply stated is

* Start with a known video sequence.

* New Video Processing system alters the video sequence.

* Decode the processed video sequence.

* Capture the processed video sequence.

* Display the original and processed video sequences.

* Bring in experts to subjectively vote.


Complexity arises as

* New Video Processing systems may need new equipment to playback the video sequences.

* The original and processed video sequences should be displayed in random orders.

* Expert viewers are expensive and do not produce repeatable results.

* Digital displays scale incoming video sequences based on their native resolution.

* Display manufactures want to check resolutions at 2x the video rate.


Previous Options

Each vendor builds unique test equipment to verify their new algorithms. So the first job is to debug the test equipment before it can be used to verify a new design. Debugging the test equipment can take as long if not longer than debugging the display equipment.


Easier Solution

To streamline the process, equipment for video quality testing needs to be defined, which can capture, play, and analyze any two video sequences. Further, as new input/output modules are continuously under development, the test equipment should use an open-architecture approach to ease upgradeability.


Video Clarity defined the ClearView product line with these objectives in mind.

* Capture video sequences in as many formats as possible.

* Convert all video sequences to user-selectable resolution.

* Translate all video sequences to uncompressed Y’CbCr 4:2:2 or RGB 4:4:4.

* Support 8 and 10-bit data paths with upgradeability to future 12-bit modes.

* Store the video sequences as frames (fields) so that they can be played at any rate.

* Display the video sequences in real time in multiple viewing modes.

* VTR controls – play, shuttle, jog, pause

* Integer-based zoom & pan.

* Apply objective metrics to the video sequences.

* Export pieces of video sequences to further analyze off-line.

* Use a standard operating system so that the operator can run 3rd party analysis applications.


By working in the uncompressed domain, any two video processing algorithms can be compared independent of compression or other processing.


To further simplify the work flow, any video sequence can be played; while capturing another video sequence, thus, combining the video server and capture device into one unit. By doing this, the original source is already inside the test equipment so captured content alignment is easily obtained.


The operator chooses the output video resolution and output rate independent of the input. Video sequences are cropped or centered with black borders to meet the desired resolution, and then played-out at rates up to 120Hz. The video sequence is sent to the display adapter or panel to test desired resolutions and frame rates.


Further the operator can play any stored video sequence, at any speed, for any duration either manually or using automated play lists.


The original and processed video sequences can be displayed – side-by-side, mirrored, or seamless split – on a single display. This eliminates the need to calibrate two separate displays.


ClearView applies various objective metrics to the video sequences, generates graphs, and calculates an objective score.


While development of additional objective algorithms is ongoing, we have built a hybrid system that takes into consideration subjective testing with objective measurements. ClearView can easily be programmed to display video sequences for the expert viewers; while recording the objective metric scores along with the MOS. While the MOS cannot be repeated, the objective metric can, easily and readily.


Since the system is based on an open, Windows-based architecture, any objective measurement algorithm can be modeled off-line using the stored video sequences.


Benefits

* Repeatable tests, quantitative results, and a streamlined setup.

* Analyze 2 video sequences in real-time up to 1080P.

* Input virtually any file type or capture from any digital or analog source.

* Multiple viewing modes are presented on a single display – no need to calibrate 2 separate Television displays to compare video sequences.

* Integrated uncompressed, high definition Video Server and Capture Device.

* Ability to Play 2 fully uncompressed, HD Streams in Real-Time.

* Hybrid Solution with Integrated Objective Metrics and Subjective Viewing Modes.


Case Examples

In all of the following examples, the tests can be performed using software mockups of the actual hardware unit. ClearView exports video sequences as files and accepts video files as inputs.


A Video Processing manufacturer needs to

* Source for their video processing or encoders in SDI, Component, or DVI.

* Capture the output of their unit.

* Capture the output of their unit after it has been decoded.

* Visually inspect the new algorithm compared to the original and/or previous algorithms.

* Generate a Score for repeatability.


A VOD Server manufacture needs to

* A way to bring in multiple file formats into their system through SDI or Component.

* Visually inspect the output of their VOD server compared to the original Source.

* Generate a Score for repeatability.


A Set-top Box (STB) or Decoder manufacture needs to

* Capture a “Golden” video stream or the results of one of their competitors.

* Capture the output of their unit.

* Visually inspect the “Golden” stream compared to their decoded stream.

* Generate a Score for pass/fail.


A Graphic or Display chip manufacture needs to

* Play out “Source” video sequences at various resolutions and rates to their graphic or display chip through DVI, HDMI, or VGA.

* Capture the output of their unit.

* Visually inspect the output of their algorithm compared to the original Source.

* Generate a Score for repeatability.


A Display manufacture needs to

* Play out “Source” video sequences at various resolutions and rates to test their display through DVI, HDMI, or VGA.

* Visually inspect the performance of their Display.

* Generate a count of dropped frames if any occurred.


Summary

ClearView takes advantage of the high-reliability of today’s off-the-shelf computer platforms. This ensures that products are made with state-of-the-art hardware, while at the same time avoiding the high cost of custom designs.


ClearView provides broadcasters, video researchers, compression developers with the unique ability to capture, play-out, and analyze video sequences. Objective measurements are generated and logged for repeatable tests.

Bill Reckwerdt, Vice President Marketing & Business Development
Bill’s extensive career spans 20 years in the digital video industries. He brings to Video Clarity expertise in compression, digital transmission, and video servers.
To contact Bill directly send an email to bill@videoclarity.com or visit http://www.videoclarity.com/ManufacturerSolutions.html#top

Be the first to comment - What do you think?  Posted by admin - March 25, 2009 at 2:40 am

Categories: Video   Tags: , , , , ,

Video Testing for Broadcasters

This paper explores the challenges Broadcasters face when assessing video quality. Many factors affect the video before it gets to the TV: compression, image processing, scaling, decoding, transmission, etc.


Video processing and compression algorithms change the characteristics of the original program in the quest of reducing the bandwidth needed to send the programming information to the home. The art is to do this without allowing the audience to perceive a change in video quality. Successful video processing and compression algorithms perform the desired modifications while presenting a result to the viewer that, subjectively, looks natural and realistic. This sounds difficult, but it is necessary when transmitting many channels of high-quality programming.


Each broadcaster – traditional or web caster – must deal with rapidly changing varieties of programming, new video processing algorithms, and new compression algorithms. Video processing and compression companies continuously invent sophisticated ways to reduce the huge bandwidth requirements to manageable levels. How can broadcasters know if a new algorithm is better than their current choice?


Broadcasters invite the various video processing and compression companies into their R&D facilities, and perform side-by-side tests also known as a “bake-off”. Each vendor starts with the same source material, and does their best to reduce the bandwidth while keeping the video quality high.


The broadcaster shows the results to a group of experts and asks them, which one is the best. This is termed subjective video analysis, and it measures the overall perceived video quality. The most commonly used video quality evaluation method is the Mean Opinion Score (MOS), recommended by the ITU. It consists in having several experts viewing a known distorted video sequences in order to rate its quality, according to a predefined quality scale. By doing this the expert viewers are trained to build a mapping between the quality scale and a set of processed video sequences. After the “training” is complete, the subjects are then asked to rate the new video processing algorithms.


Simply stated, the test setup is

* Start with a known video sequence.

* New Video Processing system alters the video sequence.

* Display the original and processed video sequences.

* Bring in experts to subjectively vote.


Complexity arises as

* New Video Processing systems may need new equipment to playback the video sequences.

* The original and processed video sequences should be displayed in random orders.

* Expert viewers are expensive and do not produce repeatable results.


Easier Solution

To streamline the process, equipment for video quality testing needs to be defined, which can capture, play, and analyze any two video sequences. Further, as new input/output modules are continuously under development, the test equipment should use an open-architecture approach to ease upgradeability.


Video Clarity defined the ClearView product line with these objectives in mind.

* Capture video sequences in as many formats as possible.

* Convert all video sequences to user-selectable resolution.

* Translate all video sequences to uncompressed Y’CbCr 4:2:2 or RGB 4:4:4.

* Support 8 and 10-bit data paths with upgradeability to future 12-bit modes.

* Store the video sequences as frames (fields) so that they can be played at any rate.

* Display the video sequences in real time in multiple viewing modes.

* Apply objective metrics to the video sequences.

* Export pieces of video sequences to further analyze off-line.

* Use a standard operating system so that the operator can run 3rd party analysis applications.


By working in the uncompressed domain, any two video processing algorithms can be compared independent of compression or other processing.


To further simplify the work flow, any video sequence can be played; while capturing another video sequence, thus, combining the video server and capture device into one unit. By doing this, the original source is already inside the test equipment so captured content alignment is easily obtained.


The original and processed video sequences can be displayed – side-by-side, mirrored, or seamless split – on a single display. This eliminates the need to calibrate two separate displays.


ClearView applies various objective metrics to the video sequences, generates graphs, and calculates an objective score. ClearView includes PSNR, Spatial Information and Temporal Information (as proposed in ITU-T P.9210). These metrics are the basis for more sophisticate metrics like VQM, JND, which are being analyzed by organizations like VQEG, VPQM, and SMPTE.


ClearView displays multiple video sequences, even if they are played at different rates (i.e. mobile phone video compared to TV) for the expert viewers; while recording the objective metric scores. While the MOS cannot be repeated, the objective metric can, easily and readily.


Since the system is based on an open, Windows-based architecture, any objective measurement algorithm can be modeled off-line using the stored video sequences.

Benefits

* Repeatable tests, quantitative results, and a streamlined setup.

* Analyze 2 video sequences in real-time up to 1080P.

* Input virtually any file type or capture from any digital or analog source.

* Multiple viewing modes are presented on a single display – no need to calibrate 2 separate Television displays to compare video sequences.

* Integrated uncompressed, high definition Video Server and Capture Device.

* Ability to Play 2 fully uncompressed, HD Streams in Real-Time.

* Hybrid Solution with Integrated Objective Metrics and Subjective Viewing Modes.


Case Examples

A Broadcaster or Service Provider needs to

* Play out “Source” video sequences at various resolutions and rates to their encoder and/or video processing equipment.

* Capture the output of the encoder and/or video processing equipment.

* Capture the output from the STB.

* Test the quality of the encoder, video processing, and set-top box equipments. To do this, they need to

o Bring in Subjective Experts to choose the appropriate vendor (“bake-off”) or to choose the correct settings.

o Generate a Score for repeatability.


A Solution/System Integrator creates a broadcast solution using encoder, network, video processing, VOD, conditional access, and set-top box providers. They market and sell their solution to Broadcasters or Service Providers. So they perform all of the above tests. When they cannot bring the Broadcaster or Service Provider into their viewing room to see the solution, they would like to take the test results to the Broadcaster. Thus, they have the additional requirement for a

* Portable solution to show side-by-side comparisons to the Broadcaster or Service Provider.


Summary

ClearView takes advantage of the high-reliability of today’s off-the-shelf computer platforms. This ensures that products are made with state-of-the-art hardware, while at the same time avoiding the high cost of custom designs.


ClearView provides broadcasters, video researchers, compression developers with the unique ability to capture, play-out, and analyze video sequences. Objective measurements are generated and logged for repeatable tests.

Bill Reckwerdt, Vice President Marketing & Business Development
Bill’s extensive career spans 20 years in the digital video and medical imaging industries. He brings to Video Clarity expertise in compression, digital transmission, and video servers.
To contact Bill directly send an email to bill@videoclarity.com or visit our http://www.videoclarity.com

Be the first to comment - What do you think?  Posted by admin - March 19, 2009 at 3:33 pm

Categories: Video   Tags: , ,

How to Do Objective Video Testing

Over recent decades, the role of video images has grown steadily. Advances in technologies underlying the capture, transfer, storage, and display of images have created situations where communicating using images has become economically feasible. More importantly, video images are in many situations an extremely efficient way of communicating as witnessed by the proverb “a picture is worth a 1000 words.”


Notwithstanding these technological advances, the current state of the art requires many compromises. Examples of these compromises are temporal resolution versus noise, spatial resolution versus image size, and luminance/color range versus gamut. These choices affect the video quality of the reproduced images. To make optimal choices, it is necessary to have knowledge about how particular choices affect the impression of the viewer. This is the central question of all video quality research.


Current video quality research can be divided into 2 approaches: experimental evaluation and modeling.


Experimental Evaluation

A group of human subjects is invited to judge the quality of video sequences under defined conditions. Several recommendations are found in ITU-R BT.500.10 “Methodology for Subjective Assessment of the quality of Television Pictures” and ITU-T P.9210 “Subjective Video Quality Assessment methods for Multimedia Applications.


The main subjective quality methods are Degradation Category Rating (DCR), Pair Comparison (PC) and Absolute Category Rating (ACR). The human subjects are shown 2 sequences (original and processed) and are asked to assess the overall quality of the processed sequence with respect to the original (reference) sequence. The test is divided into multiple sessions and each session should not last more than 30 minutes. For every session, several dummy sequences are added, which are used to train the human subjects and are not included in the final score. The subjects score the processed video sequence on a scale (usually 5 or 9) corresponding to their mental measure of the quality – this is termed Mean Observer Score (MOS).


Two serious drawbacks of this approach are:

* It is extremely time consuming, and tiresome for the participants.

* The obtained knowledge cannot be generalized because relationships between design choices and video quality are descriptive rather than based on understanding.


As a result, in a single series of experiments only a small fraction of the possible design decisions can be investigated. This makes the process even longer and more tedious.


Modeling

The second approach tries to address these drawbacks by means of developing models that describe the influences of several physical image characteristics on video quality, usually through a set of video attributes thought to determine video quality. When the influence of a set of design choices on physical video characteristics is known, then models can predict video quality. The models express video quality in terms of visible distortions, or artifacts introduced during the design process. Examples of typical distortions include flickering, blockiness, noisiness, or color shifts.


Two types of models exist, where the fundamental difference between them is how the impairment is calculated.


In the first type, physiologically or psychophysically models of early visual processing are used to calculate impairment from a difference between the video sequences. Many well known metrics exist, which compare the “original” to the “processed” output:

* PSNR – Peak Signal to Noise Ratio

* JND – Just noticeable differences

* SSIM – Structural SIMilarity

* VQM – Video Quality Metric

* MPQM – Moving Picture Quality Metric

* NVFM – Normalize Video Fidelity Metric


The two most important drawbacks of this approach are

* It is unclear what exactly the “original” version of a video is.

* These algorithms are measuring visible differences not video quality.


The second type of model tries to estimate visible distortions directly from the “processed” video; instead of comparing it to the “original”. In this type of model, visible distortions of a video, such as unsharpness or noisiness are predicted by estimating physical attributes of the video. The advantage of this approach is that the “original” video sequence is not needed. The uncertain translation from visible distortions to video quality is an important drawback to this approach.


Video Clarity Solution

Regardless of whether experimentation or modeling is chosen, certain features are common:

* Video must be presented in an “unaltered” state.

* Results must be tabulated and preserved.


Since processing video may take many forms: compression, video enhancements, and color space conversions to name a few, a system must be put in place to normalize the video information so that a comparison can be done. It must be remembered that the end consumer does not care if the original video was compressed with Windows Media, DviX, or MPEG-4. All that matters is whether the video left them with the desired impression.


To streamline the process, equipment for video quality testing needs to be defined, which can capture, play, and analyze multiple video sequences. Further, as new input/output interfaces are continuously under development, the test equipment should use an open-architecture approach to ease upgradeability.


Video Clarity defined the ClearView product line with these objectives in mind.

* Capture video sequences in as many formats as possible.

* Matt/Crop all video sequences to user-selectable resolution.

* Translate all video sequences to uncompressed Y’CbCr 4:2:2 or RGB 4:4:4.

* Support 8 and 10-bit data paths with upgradeability to future 12-bit modes.

* Store the video sequences as frames (fields) so that they can be played at any rate.

* Display the video sequences in real time in multiple viewing modes.

* Playback controls include play, shuttle, jog, pause, pan, and zoom.

* Apply objective metrics to the video sequences.

* Export pieces of video sequences to further analyze off-line.

* Use a standard operating system so that the operator can run 3rd party analysis applications.


By working in the uncompressed domain, any two video processing algorithms can be compared independent of compression or other processing.


To further simplify the work flow, any video sequence can be played; while capturing another video sequence, thus, combining the video server and capture device into one unit. By doing this, the original source is already inside the test equipment for easy comparison.


The original and processed video sequences can be displayed – side-by-side, mirrored, or seamless split – on a single display. This eliminates the need to calibrate two separate displays.


ClearView applies various objective metrics to the video sequences, generates graphs, and calculates an objective score. ClearView includes PSNR, Spatial Information and Temporal Information (as proposed in ITU-T P.9210). These metrics are the basis for more sophisticate metrics like VQM, JND, which are being analyzed by organizations like VQEG, VPQM, and SMPTE.


ClearView displays multiple video sequences, even if they are played at different rates (i.e. mobile phone video compared to TV) for the expert viewers; while recording the objective metric scores. While the MOS cannot be repeated, the objective metric can, easily and readily.


Benefits

* Repeatable tests, quantitative results, and a streamlined setup.

* Analyze 2 video sequences in real-time up to 1080P.

* Input virtually any file type or capture from any digital or analog source.

* Multiple viewing modes are presented on a single display – no need to calibrate 2 separate Television displays to compare video sequences.

* Integrated uncompressed, high definition Video Server and Capture Device.

* Ability to Play 2 fully uncompressed, HD Streams in Real-Time.

* Hybrid Solution with Integrated Objective Metrics and Subjective Viewing Modes.


Implementation

ClearView takes advantage of the high-reliability of today’s off-the-shelf, high-performance computer platforms. This ensures that products are made with state-of-the-art hardware, while at the same time avoiding the high cost of custom designs.


ClearView provides broadcasters, researchers, and compression developers with the unique ability to capture, play-out, and analyze video. Objective measurements are generated, graphed, and logged for repeatable tests.

Bill Reckwerdt, Vice President Marketing & Business Development
Bill’s extensive career spans 20 years in the digital video and medical imaging industries. He brings to Video Clarity expertise in compression, digital transmission, and video servers.
To contact Bill directly send an email to bill@videoclarity.com or visit our http://www.videoclarity.com

Be the first to comment - What do you think?  Posted by admin - January 30, 2009 at 1:25 pm

Categories: Video   Tags: , ,

Powered by Yahoo! Answers