Using V4L2 to realize VideoDevice

[ZhangGaoxing]

I tested it on Raspberry Pi 3B+ and Orange PI Zero using CSI and USB cameras. This draft version can capture the static image correctly. It can't shoot videos yet because I don't know much about how video streams are handled. But the method to get the raw frame data is done. V4L2 struct and consts are not complete, they are too many, and I just transplanted what the methods needed. Now I have some questions, such as whether it should be placed in System.Device.xxx or Iot.Device.xxx namespace, and whether there are some extensions to help transform the image format(my USB camera only supports YUV format😄). I hope to receive your reply and help😄. @joperezr @krwq

A simple example

static void Main(string[] args)
{
    VideoConnectionSettings settings = new VideoConnectionSettings(0)
    {
        CaptureSize = (2560, 1920),
        PixelFormat = PixelFormat.JPEG,
        ExposureType = ExposureType.Auto,
    };
    using VideoDevice device = VideoDevice.Create(settings);

    // Get the supported formats of the device
    foreach (var item in device.GetSupportedPixelFormats())
    {
        Console.Write($"{item} ");
    }
    Console.WriteLine();

    // Get the resolutions of the format
    foreach (var item in device.GetPixelFormatResolutions(PixelFormat.JPEG))
    {
        Console.Write($"{item.Width}x{item.Height} ");
    }
    Console.WriteLine();

    // Query v4l2 controls default and current value
    var value = device.GetVideoDeviceValue(VideoDeviceValueType.ExposureTime);
    Console.WriteLine($"{value.Name} Min: {value.Minimum} Max: {value.Maximum} Step: {value.Step} Default: {value.DefaultValue} Current: {value.CurrentValue}");

    // Capture static image
    device.Capture("/home/pi/test1.jpg");

    // Change capture setting
    device.Settings.HorizontalFlip = true;

    // Capture static image
    device.Capture("/home/pi/test2.jpg");
}

[ZhangGaoxing]

My Raspberry Pi CSI camera(unofficial) couldn't get maximum resolution, but the USB camera could get it. I guess it's a driver problem.

[krwq]

Did you try changing other parameters while increasing the size? (i.e. slow down capture rate or something similar - might be worth to check with strace what other software is doing)

[krwq]

That explains why you were not so active recently 😄 Nice work @ZhangGaoxing!

[krwq]

Please add some sample code, haven't seen anything blocking!

Per your question, you should put it in Iot.Device.Bindings. I'm not much familiar with how videos work on linux but here is some info I got last time I looked at it:
#287

[ZhangGaoxing]

In the new commit, I added some new v4l2 settings and converters for common pixel formats. For video shooting, there are still many problems to consider. Because the functions is not only for Raspberry Pi CSI cameras, but also for all USB cameras that meet the Linux standard. For example, Raspberry Pi CSI camera can directly output H264 format video. We don't need to do much work, just read and save data from the driver. Some video players can play H264 format video directly. However, many webcams only support YUV format, which requires some transformations to save videos. And the file format saved is avi or MP4 or flv... Implementing this requires a lot of work, and I haven't found the right NuGet package, no one has used C# to implement these functions, and. NET itself lacks some methods of media processing. I also have my own job, and I'm unlikely to concentrate on it... I have the following suggestions:

1. Not adding video method for the time being.
2. I looked at some similar libraries, some of which use the command to perform the function of shooting videos. In C#, we can use Process class to run ffmpeg to realize video shooting.

[krwq]

@ZhangGaoxing thanks, just pictures are fine for now. For H264 there is very likely some C library you can wrap to do it instead of implementing it from the scratch

[krwq]

LGTM as first pass on video devices.

[krwq]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ZhangGaoxing]

@krwq

[krwq]

Please move all of those to Interop.Libc.cs since this seems a bit too granular. Also for the common stuff please also move out stuff from https://github.com/dotnet/iot/blob/master/src/devices/SocketCan/Interop.cs with whatever is common (close at minimum; got mixed feelings about ioctl; please review what makes sense to move to common)

[krwq]

Why can't you use FileStream and just get a handle out of it?

[ZhangGaoxing]

I referred to the implementation of I2C, and Interop was copied from System.Device.Gpio project.

[krwq]

ok, we should clean duplicate code in the future but fine in this PR

[krwq]

can you just always initialize in the constructor and remove the lock at all?

[ZhangGaoxing]

If I change the shooting parameters after the first capture, then the second capture will throw an error. So the process must be initialization -> shooting -> close.

[krwq]

should this be happening only in Dispose?

[ZhangGaoxing]

It happens at the end of each capture.

[krwq]

@ZhangGaoxing only treat comment about the leak and returning Stream as blocking. Anything else I'd prefer to be fixed but I'm fine with this being left as is.

[krwq]

LGTM, thanks!