ASICs provide a tremendous energy efficiency, and yet suffer from being fixed function with limited programmability. This was a core engineering challenge that we addressed in the development of the Codensity ASIC family with upgradeable firmware that can be used for a variety of purposes, including adding new features and improving coding performance, and functionality.
To explore these capabilities, we spoke with two members of the NETINT development team, Neil Gunn, who is NETINT’s Video Firmware Tech Lead, and Savio Lam, a firmware engineer. In this short discussion, they describe how firmware allows Codensity video transcoders and VPU’s to evolve and improve long after leaving the foundry.
This conversation focuses mainly on our Codensity G4 ASIC, however the capability to upgrade firmware applies to all of our ASIC platforms including the Codensity G5.
What do you do with NETINT?
Neil Gunn
I am a firmware architect and also develop the firmware and to a lesser extent, the host side software (libxcoder and FFmpeg) for NETINT transcoding ASICs. I started at NETINT in 2018 working on T408 (Codensity G4 based) firmware development. Then, I moved to Quadra (Codensity G5 based) as a software architect and firmware/software developer. I continue to support T408 in the background.
Savio Lam
I am a firmware engineer working on our video transcoding products.
What did you do on the T408?
Neil Gunn
I implemented a number of video features in the firmware such as 10-bit transcoding, close captions, HDR10, HDR10+, HLG10, Dolby Vision, HRD, Region of Interest, encoder parameter change, etc. I also worked on bug fixes and customer issues.
Savio Lam
I worked on the system design and integration. I mainly developed code that controls how video data comes in and out of our transcoder in the most efficient and reliable way.
What is firmware in an ASIC?
Neil Gunn
The firmware is software that runs on embedded CPUs within the ASIC. The firmware provides a high-level interface to the low-level encoding and decoding hardware. The firmware does a lot of the high-level bitstream processing, such as creating VPS, SPS, and PPS headers, and SEI processing, leaving the ASIC hardware to do the low-level number crunching. Functions that consume a lot of processing and are likely not to change are implemented in hardware.
Savio Lam
To add to what Neil has already described, the firmware in our T408 ASIC manages several significant functions. For example, it comprises code responsible for the NVMe protocol, which allows us to efficiently receive and return up to 8GB/s of video input and output data. To properly consume and process the video data, the firmware sets up and schedules tasks to the appropriate hardware blocks.
Our firmware is also the brain that oversees the bigger picture part of the rate control. In this role, it’s part of a feedback loop that inputs subpicture data from low-level hardware blocks and uses that data to make better decisions that improve picture quality.
To sum up, the firmware is the brain that controls all the hardware blocks in the ASIC and gives instructions to each of them to perform their tasks as efficiently as possible.
How is firmware different from the gates burned into the chip?
Neil Gunn
Firmware, like all software, can be changed, unlike actual gates in a chip. It’s called firmware because it’s a little harder to change than software. Firmware is stored in Flash memory which can be reprogrammed through an upgrade process. A T408 firmware release typically consists of new host-side software and firmware that must be version-matched for proper operation. Software provided to our customers with the release simplify the upgrade for one or more T408s in a system.
Savio Lam
There is logic in our T408 ASIC, which could have been designed as part of the hardware for better performance. However, that would significantly limit us from adding and improving the certain product features to suit different customer needs. We believe we have found the right balance on deciding what should be implemented in the firmware or hardware.
What functions can you adjust and/or improve within firmware?
Neil Gunn
Things like the codec headers, seis, and rate control, to a certain extent, can be adjusted and/or improved within the firmware. Some lower-level rate control features are fixed in the hardware. Lower-level parts of the encoding standard are fixed in the hardware as these require a lot of processing and are unlikely to change.
Savio Lam
As Neil said, we are quite flexible when it comes to adding or improving support for different video metadata. And as we both explained earlier, since the firmware is also part of the brain that operates the picture rate control for encoding, we can continue to improve quality to a certain degree post-ASIC development.
Do you have any examples of significant improvements with the T408?
Neil Gunn
We significantly reduced codec delay on both the encoder and decoder. Our low delay mode removes all frame buffering and encodes and decodes a single frame at a time. Our encoder uses a low delay GOP and sets flags in the bitstream appropriately so that another decoder knows that it doesn’t need to add any delay while decoding.
Savio Lam
Based on different customers’ feedback, we have made several improvements (or fixes) in the past to our rate control through firmware fixes which improved or resolved some of the video quality-related problems they have encountered.
When you hear people say ASICs are obsolete the day they come out of the foundry, what’s your response?
Neil Gunn
It’s not true. It is true that the hardware is fixed in an ASIC. Still, the functions implemented in the hardware are typically the lower-level parts of a video codec standard that do not change over time and so the hardware does not need to be updated. The higher levels parts of the video codecs are in firmware and driver software and can still be changed. For example, the T408 encoder hardware is designed for H.264 and H.265. We cannot add new codecs to the T408, but we can add new features to the existing codecs.
Savio Lam
There is a fine balance between what needs to be implemented in hardware for performance and what needs to be implemented in the firmware for flexibility (programmability). We think we struck the perfect balance with the Codensity G4 which is what makes it a great ASIC.
This conversation focuses mainly on our Codensity G4 ASIC, however the capability to upgrade firmware applies to all of our ASIC platforms including the Codensity G5.
