Nvidia Broadcast: AI in video streaming and telecommunications services
We are introducing a new testing methodology for graphics cards. One of the reviewed cards will be the GeForce RTX 3070 and we have an interesting one: Aorus GeForce RTX 3070 Master 8G with six video outputs. Before the review comes out, we’ve use it for a visual comparison to see the impact of ray tracing and how the different DLSS settings in the latest Cyberpunk 2077 patch will affect the quality of the game graphics.
Tensor cores present in Nvidia GPUs to accelerate AI computations, are most importantly employed in DLSS (including DLSS 2.x), where they provide performance for the neural network-based interpolation algorithm and most likely contributing the lion’s share to the result that DLSS is capable of achieving. But one more use of tensor cores has come with GeForce RTX 3000 graphics: Nvidia uses them in the Broadcast App application, which is designed to improve the video quality when you stream your game, or even for use in video production, but also in ordinary video chatting.
Our plan to test this technology was foiled by a missing webcam. But since we didn’t previously cover the Broadcast application (as it is still more of a software feature, albeit hardware-accelerated, we ended up not including a writeup on it in our Ampere architecture analysis), it doesn’t hurt to take a look at what Broadcast does and why it may be good for you, now.
Nvidia Broadcast App features
Nvidia Broadcast is likely to be developed further in the future, but currently it offers several enhancements that you can use with your video. The first functionality, called Virtual Background, automatically removes distracting backgrounds behind you. This can be used to insert an overlay of your own face or figure from a webcam into your game stream, with the video from the game put behind you/around you. Or you can hide the environment in which you are shooting and replace it with a photo or some virtual scene, like in a TV studio. You normally need a “green screen” for this, but thanks to the artificial intelligence algortim, the Broadcast App can erase any background even without such special equpiment. So with this, you should be able to broadcast with such green-screen stlye virtual background from just about anywhere (don’t do it from bathroom, please). And a bonus for less orderly people: they don’t have to clean their room, with this. There is also a simple option to just blur the background while still keeping the person in the foreground fully sharp and defined.
Broadcast App also seems to build on the previously announced RTX Audio technology and offers similar sound “cleaning” functionality as well – but this time it is about suppressing ambient noise in a audio track/input. The Noise Removal filter uses a trained neural network that removes unwanted sounds and noise, so it again allows you to broadcast (or communicate) well, even from an environment where there would otherwise be too much interference. Or other sounds that the other party should ideally not hear.
Another intelligent feature is Auto Frame. This, in turn, solves the problem that your webcam is not a living cameraman and does not capture you perfectly, especially when you move in front of it. Auto Frame solves this – it automatically detects the person in the shot and tries to zoom in and out on them and move the cropped area of the picture such that the focus on the person stays correct and good looking even as they move. In particular, Auto Frame can dynamically adjust the relevant cut-out from the camera’s unprocessed footage when you move, tilt, and so on. By adjusting crop and zoom, you will always stay in the shot, as if the camera was moving and zooming to automatically capture your position. You can see it demonstrated here on video (not ours) from the official presentation of the application:
Nvidia Broadcast can be used from a variety of existing applications, so you can already use it from your favorite apps. Broadcast App works as an intermediate layer between the input from your real camera (and microphone), which it processes. The application adds a Nvidia Broadcast virtual device to the system that acts as a camera and microphone: you can then set it as input in a communication program, streaming application or anywhere, and just like that, the application now uses input modified by Nvidia Broadcast App filters – without the need for adding any explicit support or needing updates.
Broadcast App is already available for download. It may be worth noting that it is not directly bound to GeForce RTX 3000 cards. It only requires the presence of tensor cores, so it also works on GeForce RTX 2000 graphics (from RTX 2060 and higher).
Translated, original text by:
- Contents
- A sneak peek ahead of review: RTX 3070 with six video outs from Gigabyte
- Ray-tracing reflections in Cyberpunk and DLSS 2.0
- Nvidia Broadcast: AI in video streaming and telecommunications services
