Sorry for answer late.
I think the key point is the data format conversion. Now the underlying data format provided by the camera is yuv. In my demo, every frame is converted to rgb format via cpu.
If you can use yuv format for your project, do not convert it, and then test it?