Third generation FSR image interpolation partially unveiled at GDC 2023
Few months ago, AMD has announced ongoing work on the third generation of FidelityFX Super Resolution (FSR) upscaling, which should bring a similar feature to DLSS 3 – interpolating of intermediate frames inserted between legitimate game frames, much like TVs do with video and movies. AMD has now formally confirmed this and presented FSR 3.0, which will offer frame interpolation and will again be open technology (including source code).
At GDC 2023, AMD held a presentation called Temporal upscaling: past, present, and future, in which the first two generations of this upscaling – the purely spatial FSR 1.0 and the second gen FSR 2.x with temporal filtering (stabilization and detail reconstruction) – were presented in a retrospective. This presentation confirms that the third generation, which is being worked on now, will compute extra interpolated frames to insert between real frames rendered by the game, i.e. as Nvidia’s DLSS 3.
The presentation doesn’t say much about FSR 3.0 yet. AMD states in it that the goal is to increase the frame rate (as expected) by again increasing the number of pixels presented relative to the pixels that the GPU actually renders. The frame rate interpolation is apparently based on prior Fluid Motion Video technology (used for frame rate conversions in video), among other things, and will work with FSR 2.x by also using vectors passed by the game engine.
As with Nvidia, the input for interpolation will be whole, completely finished rendered frames including everything – so in terms of image data, interpolation will again work like a video filter processing a bitmap, not scene objects. AMD showed a schematic of the whole process and confirmed that the input for interpolation is the final frames including post-processing and also with various overlays, like OSDs and GUI of the game. So these will be interpolated along with the actual 3D scene.
The presentation states that interpolation has to overcome various problems, it is not possible to use output pixel color clamping as in previous versions of FSR, and it is complicated to deal with nonlinear motion (rotation, perspective, zoom). But according to AMD, the interpolator will be separated from the rest of the rendering, so any artifacts and errors in the image will only “flash” for a single frame and will not affect subsequent frames.
According to the schematic, the interpolated frames will be displayed roughly in the middle od the time the GPU is taking to compute the next regular frame. It is expected that only one interpolated frame will be inserted between the two GPU-rendered frames, which means that the technology will roughly double the frame rate compared to the input. The resulting frequency may be a bit lower than twice the native frequency achieved with FSR 2.x, because interpolating the intermediate frames itself will consume some extra GPU time, as you can see even in the schematic.
What seems to be confirmed: the interpolated frames will be inserted in such a way that the artificial frame will be generated bidirectionally as an intermediate state between two frames, i.e. between the previous frame already shown and its future successor. This is the same principle as in DLSS 3. Doing it the other way (forward-only projection from multiple past frames) it is probably not very useful, because there would be frequent errors in the estimated motion of objects and the scene, which would be very disruptive to the gaming experience.
However, this solution will worsen latency and reactivity, which AMD admits. This is because the algorithm will have to work one frame ahead – in order to display the interpolated frame, the frame after it must already be ready (and stored for later), too. This also occurs with DLSS 3, although Nvidia doesn’t talk admit to this impact much, and its marketing is actually trying to claim that latency is being improved (which is nonsense). In reality, Nvidia has added the otherwise separately available Reflex technology to DLSS 3 to reduce input latency, and the improvements are solely part of that (if the game already has Reflex enabled and then you add DLSS 3, you’ll make the latency worse). AMD states in the presentation that it will probably address the latency degradation introduced by interpolation in the same way, i.e. by combining FSR 3.0 with their own latency reduction technology (AMD AntiLag).
Recommended for games already running at least 60 Hz
According to AMD, FSR 3.0 is recommended to be used when the input frame rate is at least around 60 FPS. This will reduce the impact of the increased latency (one extra frame takes less milliseconds), but it is also because the interpolated frames are not perfect and may have artifacts. The higher the final frame rate, the shorter the time the interpolated frame is shown, which hides its imperfections (this doesn’t work perfectly though) and the positive impression of smoothness of motion prevails.
What has not yet been revealed is the interpolation algorithm itself and its nature. Which means we don’t know whether some kind of neural network (AI) will be used or whether it will be a conventional, human-designed algorithm. However, like FSR 2.x, AMD will be limited by the fact that this code must run on a larger range of different GPUs, and thus probably won’t be able to depend on any specialized hardware. It will therefore have to run with less computing power than DLSS 3. Apart from this difference though, the principle seems to be very similar to DLSS 3, so similar benefits and negatives can be expected.
AMD has not yet revealed when FSR 3.0 will be finally completed and introduced in games. The current presentation is more of a report on ongoing development, so it will probably still be a few months before the finished version is released and the integrations in actual games start to arrive. Implementing it in games that already have FSR 2.x should probably be easier than adding FSR 2.0 on top of FSR 1.0, as hopefully there will be less need to add as much tooling and interfacing from the game’s side, this time.
English translation and edit by Jozef Dudáš
⠀
