点评:Subjectify.us，一个主观视频比较网站
If Chrome supports playback of the uploaded content, as it does VP9 and H.264, Subjectify downloads the actual test files to the viewer’s computer before the tests start. With unsupported content like HEVC, the service converts these files to H.264使用恒定速率因子(CRF), which should deliver sufficient quality to avoid prejudicing the results.
Participants download the files, view the instructions, and start watching the videos. The videos are presented without playback controls, so there’s no way to stop the video to inspect. 也没有计时器或倒计时钟. After each video pair, participants are presented with the screen in 图4.
取决于测试, 你应该会在几个小时内看到效果, and most tests should be finished within three days.
请告诉我结果
在项目选择屏幕中, 你可以看到有多少参与者给你的视频打分, 有多少失败了, 以及他们做出了多少选择. 在CBR vs. VBR test, there were 149 total participants, 3 of whom failed, with 1,752 rated comparisons.
The presentation of results varies according to the test parameters. 在CBR vs. VBR业务比较, 哪个是最简单的测试, 我们可以按测试片段或整体对结果进行排序, display the results using the Bradley-Terry or Crowd Bradley-Terry models, and display a confidence interval from 85 percent to 99 percent relative to either parameter or ground truth.
The concept of “ground truth” relates to studies that compare the various alternatives to a super-high-quality option, 我们没有用过的. The Bradley-Terry model is a probability model that attempts to predict the outcome of a comparison. In this particular test case, the result is clear (图5); the results predict that most viewers will prefer the 200 percent constrained VBR files (the top line extending beyond 5) over CBR, 但差别并不大.
Overall, 200 percent constrained VBR (the top line) won out over CBR.
Digging into the details, the overall comparison is greatly impacted by the results from 大巴克兔， 受限VBR得分为200%，约为6分.2点，4点.70. The results for the real-world video files were neck and neck, with the participants rating CBR higher quality than VBR for 超级名模. Note that you can download all reports as JPEG or PNG images, SVG vector images, and PDF files.
编解码器比较
We had to run our codec comparisons twice because during our initial run, we noticed that the poor quality of the initial frames of the HEVC-encoded videos might be influencing the score. We reprocessed all the files to eliminate these frames and ultimately collected data from 923 participants who rated 11,076视频对.
结果是 图6 are grouped by data rate to show what looks like the standard rate-distortion curve for both test videos. The results show VP9 slightly ahead of HEVC through 4Mbps, 5Mbps的反转, 然后在6Mbps接近等值, 这三种技术是如何紧密结合在一起的, with H.有一点优势.
VP9 edged HEVC at most data rates in this comparison.
In addition to presenting overall results, you could also present results for each clip. 如图6右侧所示, beyond the graphic export options available for all tests, when the data set includes variables like data rate you can also export as CSV or XLS files, 查看数据表, 或在Highcharts中打开图表, 高级图表和图形的云服务.
主观评价的价值
We all have preconceived notions regarding video quality that dictate many of our encoding decisions—some grounded in objective metrics, 有些是我们自己的主观评价. 多年来, I’ve argued that the quality difference between CBR and VBR was much less than most compressionists believe, 和Subjectify. 我们的研究结果证实了这一点. 如上所述, 对于三个真实世界的视频文件, the quality difference was minimal; only the animated file showed a significant advantage for VBR. Given that CBR-encoded files are more reliable to deliver over limited-bandwidth conditions, 这是一个有用的数据点.
Conversely, my VMAF-fueled notions regarding switching points were proven wrong by the Subjectify.我们的结果. 直观地, 以相同的数据速率, a 1080p video will deliver more detail but potentially more compression artifacts, while a 720p video will deliver less detail and fewer artifacts. My credo has been that higher resolutions almost always win at the relevant switch points.
While likely true for animated files (which we didn’t test), 这被主体化彻底驳倒了.我们的结果 for American football and a concert video. 有了这些结果, I can experiment with different metrics to identify which does the best job predicting subjective results. 特别是, SSIM seems to favor low-resolution videos over high-resolution videos; it will be interesting to replot the switch points with SSIM and see if the results match the Subjectify.我们的预测.
就像我在开头说的, 虽然客观指标总是有用的, 主观评价是黄金标准. Subjectify.us provides an affordable and useful way to access them.
[本文发表于2018年10月号。 流媒体杂志 如“复习:主体化”.us."]
相关文章
Moscow State University's 视频质量 Measurement Tool was already good. 新版本中的增强功能, including new metrics and the ability to run multiple analyses simultaneously, 让它变得更好.
2018.03.23
Latest codec quality comparison study finds AV1 tops on quality, 但在速度上远远落后, VP9打败了HEVC. 莫斯科政府还发起了“主体化”运动.us, a new service for subjectively comparing video quality.
1月30日2018
How can video compressionists assess the quality of different files? Only by combining objective mathematical comparisons with their own professional judgments.
7月13日