Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jetson Orin NX and IMX296C (R35.2.1) - Gstreamer Internal data stream error. #138

Open
mertturanli opened this issue Feb 15, 2025 · 1 comment

Comments

@mertturanli
Copy link

Hello,

The configuration is Jetson Orin NX 8GB with Auvidea JNX42 and Jetpack R35.2.1.
The camera we are using is VC IMX296C.

The problem is that gstreamer is not working.

The device tree is attached:

/*
 * Copyright (c) 2024, Vision Components GmbH.  All rights reserved.
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
 * more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program.  If not, see <http://www.gnu.org/licenses/>.
 */

#include <dt-bindings/media/camera.h>
#include <dt-bindings/pinctrl/pinctrl-tegra.h>

// ------------------------------------------------------------------------------------------------
//  Driver Configuration for NVIDIA Jetson Orin NX on Auvidea JNX42 LM Developer Kit
// ------------------------------------------------------------------------------------------------
#define VC_MIPI_CAM_0       1   // 1: Enabled, 0: Disabled  (serial_a => CSI Port 0 => VI Stream 0)
#define VC_MIPI_CAM_1       1   // 1: Enabled, 0: Disabled  (serial_c => CSI Port 2 => VI Stream 2)
// ------------------------------------------------------------------------------------------------
//  Supported number of lanes
// -----+------------------------------------------------------------------------------------------
//  1   | OV7251, IMX296, IMX297
//  2   | OV9281, IMX264, IMX265
//  2,4 | IMX178, IMX183, IMX226, IMX250, IMX252, IMX273, IMX290, IMX327, IMX335, IMX392, 
//      | IMX412, IMX415, IMX462, IMX565, IMX566, IMX567, IMX568, IMX900
// -----+------------------------------------------------------------------------------------------
#define VC_MIPI_LANES       1   // 1, 2, 4 Lanes
// ------------------------------------------------------------------------------------------------
//  Embedded Metadata Height
// -----+------------------------------------------------------------------------------------------
//  0   | IMX178, IMX183, IMX226, IMX250, IMX252, IMX264, IMX265, IMX273, IMX392, OV7251, OV9281
//  1   | IMX290, IMX327, IMX335, IMX415, IMX462, IMX900
//  2   | IMX296, IMX297, IMX412
//  4   | IMX565, IMX566, IMX567, IMX568
// -----+------------------------------------------------------------------------------------------
#define VC_MIPI_METADATA_H  "2" // "0", "1", "2", "4" Lines of meta data
// ------------------------------------------------------------------------------------------------
//   Sensor Manufacturer
// -----+------------------------------------------------------------------------------------------
//   1  | Sony Sensor (IMX)
//   2  | Omni Vision Sensor (OV)
// -----+------------------------------------------------------------------------------------------
#define VC_MIPI_MANUFACTURER 1
// ------------------------------------------------------------------------------------------------
//  GStreamer Support
// ------------------------------------------------------------------------------------------------
//  If you want to use GStreamer with nvarguscamerasrc you have to adjust this settings in the 
//  device tree below. The sections which have to be modified are marked by a comment. 
//  To find the correct parameter values please follow the instruction in the main README.md of 
//  this repository https://github.com/VC-MIPI-modules/vc_mipi_nvidia#gstreamer-support
// ------------------------------------------------------------------------------------------------

//#########################################################################
//            _                 ___
//           [U]               [USB]     ___               JNX42 LM
//           [S]      ____     [===]    [ETH]    _         Rear view
//__<usb>____[B]_____[HDMI]____[USB]____[''']___[o]__
//===================================================      Revision 1
//                    [*]        [*]     [*]               
//                   Reset      F-Rec    Pwr               Buttons with
//                                                         Force Recovery
//#########################################################################
//            _                 ___
//           [U]               [USB]     ___               JNX42 LM
//           [S]      ____     [===]    [ETH]    _         Rear view
//__<usb>____[B]_____[HDMI]____[USB]____[''']___[o]__
//===================================================      Revision >= 2
//                    [*]      [...]     [*]               
//                   Reset               Pwr               Buttons without
//                                                         Force Recovery
//#########################################################################

// Cam0 and Cam1 Trigger pins are swapped in revision 1.0
// Auvidea board revision 2 and upwards will be compatible with NVIDIA device tree.
#define REVISION 1
#if REVISION == 1
        #define CAM0_PWDN       TEGRA234_MAIN_GPIO(AC, 0)
        #define CAM1_PWDN       TEGRA234_MAIN_GPIO(H, 6)
#else
        #define CAM0_PWDN       TEGRA234_MAIN_GPIO(H, 6)
        #define CAM1_PWDN       TEGRA234_MAIN_GPIO(AC, 0)
#endif

#if VC_MIPI_LANES == 1
    #define BUS_WIDTH 1
    #define NUM_LANES "1" 
    #define NUM_CSI_LANES 2
    #define PIX_CLK_HZ "150000000"
#endif
#if VC_MIPI_LANES == 2
    #define BUS_WIDTH 2
    #define NUM_LANES "2" 
    #define NUM_CSI_LANES 4
    #define PIX_CLK_HZ "300000000"
#endif
#if VC_MIPI_LANES == 4
    #define BUS_WIDTH 4
    #define NUM_LANES "4" 
    #define NUM_CSI_LANES 8
    #define PIX_CLK_HZ "600000000"
#endif

#if VC_MIPI_CAM_0 == 1 && VC_MIPI_CAM_1 == 1
    #define VC_MIPI_CAMERAS 2
#else
    #define VC_MIPI_CAMERAS 1
#endif

/ {
        tegra-capture-vi  {
                num-channels = <VC_MIPI_CAMERAS>;
                ports {
                        #address-cells = <1>;
                        #size-cells = <0>;
#if VC_MIPI_CAM_0 == 1
                        vc_vi_port0: port@0 {
                                reg = <0>;
                                vc_vi_in0: endpoint {
                                        port-index = <0>;
                                        bus-width = <BUS_WIDTH>;
                                        remote-endpoint = <&vc_csi_out0>;
                                };
                        };
#endif
#if VC_MIPI_CAM_1 == 1
                        vc_vi_port1: port@1 {
                                reg = <1>;
                                vc_vi_in1: endpoint {
                                        port-index = <2>;
                                        bus-width = <BUS_WIDTH>;
                                        remote-endpoint = <&vc_csi_out1>;
                                };
                        };
#endif
                }; // ports
        };

        host1x@13e00000 {
                nvcsi@15a00000 {
                        num-channels = <VC_MIPI_CAMERAS>;
                        #address-cells = <1>;
                        #size-cells = <0>;
#if VC_MIPI_CAM_0 == 1
                        vc_csi_chan0: channel@0 {
                                reg = <0>;
                                ports {
                                        #address-cells = <1>;
                                        #size-cells = <0>;
                                                vc_csi_chan0_port0: port@0 {
                                                reg = <0>;
                                                vc_csi_in0: endpoint@0 {
                                                        port-index = <0>;
                                                        bus-width = <BUS_WIDTH>;
                                                        remote-endpoint = <&vc_mipi_out0>;
                                                };
                                        };
                                        vc_csi_chan0_port1: port@1 {
                                                reg = <1>;
                                                vc_csi_out0: endpoint@1 {
                                                        remote-endpoint = <&vc_vi_in0>;
                                                };
                                        };
                                };
                        };
#endif

#if VC_MIPI_CAM_1 == 1
                        vc_csi_chan1: channel@1 {
                                reg = <1>;
                                ports {
                                        #address-cells = <1>;
                                        #size-cells = <0>;
                                        vc_csi_chan1_port0: port@0 {
                                                reg = <0>;
                                                vc_csi_in1: endpoint@2 {
                                                        port-index = <2>;
                                                        bus-width = <BUS_WIDTH>;
                                                        remote-endpoint = <&vc_mipi_out1>;
                                                };
                                        };
                                        vc_csi_chan1_port1: port@1 {
                                                reg = <1>;
                                                vc_csi_out1: endpoint@3 {
                                                        remote-endpoint = <&vc_vi_in1>;
                                                };
                                        };
                                };
                        };
#endif
                };
        };

        cam_i2cmux {
#if VC_MIPI_CAM_0 == 1
                i2c@0 {
#if VC_MIPI_MANUFACTURER == 1
                        vc_mipi_cam0: vc_mipi@1a {
                                reg                     = <0x1a>;
#else
                        vc_mipi_cam0: vc_mipi@60 {
                                reg                     = <0x60>;
#endif
                                compatible              = "nvidia,vc_mipi";
                                devnode                 = "video0";
                                use_sensor_mode_id      = "false";
                                sensor_model            = "vc_mipi";

                                num_lanes               = NUM_LANES;
                                trigger_mode            = "0";
                                io_mode                 = "0";

                                // ----------------------------------------------------
                                // If you want to use GStreamer with nvarguscamerasrc
                                // you have to adjust this settings
                                physical_w              = "4.968";
                                physical_h              = "3.726";
                                // ----------------------------------------------------

                                // This node is needed by the Tegra framework.
                                // You don't have to change any settings if just want 
                                // to use the V4L API.
                                mode0 {
                                        num_lanes                = NUM_LANES;
                                        tegra_sinterface         = "serial_a";
                                        embedded_metadata_height = VC_MIPI_METADATA_H;
                                        readout_orientation      = "0";
                                        lane_polarity            = "6";

                                        // ----------------------------------------------------
                                        // If you want to use GStreamer with nvarguscamerasrc
                                        // you have to adjust this settings. 
                                        active_l                 = "0";
                                        active_t                 = "0";
                                        active_w                 = "1440";
                                        active_h                 = "1080";
                                        mode_type                = "bayer";
                                        pixel_phase              = "rggb";
                                        csi_pixel_bit_depth      = "10";

                                        min_gain_val             = "0";         // mdB
                                        max_gain_val             = "48000";     // mdB
                                        step_gain_val            = "100";       // mdB
                                        default_gain             = "0";         // mdB

                                        min_exp_time             = "1";         // us
                                        max_exp_time             = "1000000";   // us
                                        step_exp_time            = "1";         // us
                                        default_exp_time         = "10000";     // us

                                        // For applications like argus_camera, there should 
                                        // be a minimal frame rate greater than zero!
                                        min_framerate            = "100";       // mHz
                                        max_framerate            = "60000";     // mHz
                                        step_framerate           = "100";       // mHz
                                        default_framerate        = "60000";     // mHz
                                        // ----------------------------------------------------

                                        gain_factor              = "1000";
                                        exposure_factor          = "1000000";
                                        framerate_factor         = "1000";
                                        inherent_gain            = "1";
                                        min_hdr_ratio            = "1";
                                        max_hdr_ratio            = "1";

                                        line_length              = "0";
                                        phy_mode                 = "DPHY";
                                        discontinuous_clk        = "no";
                                        mclk_khz                 = "24000";
                                        pix_clk_hz               = PIX_CLK_HZ;
                                        mclk_multiplier          = "0.0";       // deprecated
                                        cil_settletime           = "0";
                                        dpcm_enable              = "false";
                                };

                                ports {
                                        #address-cells = <1>;
                                        #size-cells = <0>;
                                        port@0 {
                                                reg = <0>;
                                                vc_mipi_out0: endpoint {
                                                        port-index = <0>;
                                                        bus-width = <BUS_WIDTH>;
                                                        remote-endpoint = <&vc_csi_in0>;
                                                };
                                        };
                                };
                        };
                };
#endif // VC_MIPI_CAM_0

#if VC_MIPI_CAM_1 == 1
                i2c@1 {
#if VC_MIPI_MANUFACTURER == 1
                        vc_mipi_cam1: vc_mipi@1a {
                                reg                     = <0x1a>;
#else
                        vc_mipi_cam1: vc_mipi@60 {
                                reg                     = <0x60>;
#endif
                                compatible              = "nvidia,vc_mipi";
                                devnode                 = "video1";
                                use_sensor_mode_id      = "false";
                                sensor_model            = "vc_mipi";

                                num_lanes               = NUM_LANES;
                                trigger_mode            = "0";
                                io_mode                 = "0";

                                // ----------------------------------------------------
                                // If you want to use GStreamer with nvarguscamerasrc
                                // you have to adjust this settings
                                physical_w              = "4.968";
                                physical_h              = "3.726";
                                // ----------------------------------------------------

                                // This node is needed by the Tegra framework.
                                // You don't have to change any settings if just want 
                                // to use the V4L API.
                                mode0 {
                                        num_lanes                = NUM_LANES;
                                        tegra_sinterface         = "serial_c";
                                        embedded_metadata_height = VC_MIPI_METADATA_H;
                                        readout_orientation      = "0";
                                        lane_polarity            = "0";

                                        // ----------------------------------------------------
                                        // If you want to use GStreamer with nvarguscamerasrc
                                        // you have to adjust this settings. 
                                        active_l                 = "0";
                                        active_t                 = "0";
                                        active_w                 = "1440";
                                        active_h                 = "1080";
                                        mode_type                = "bayer";
                                        pixel_phase              = "rggb";
                                        csi_pixel_bit_depth      = "10";

                                        min_gain_val             = "0";         // mdB
                                        max_gain_val             = "48000";     // mdB
                                        step_gain_val            = "100";       // mdB
                                        default_gain             = "0";         // mdB

                                        min_exp_time             = "1";         // us
                                        max_exp_time             = "1000000";   // us
                                        step_exp_time            = "1";         // us
                                        default_exp_time         = "10000";     // us

                                        // For applications like argus_camera, there should 
                                        // be a minimal frame rate greater than zero!
                                        min_framerate            = "100";       // mHz
                                        max_framerate            = "60000";     // mHz
                                        step_framerate           = "100";       // mHz
                                        default_framerate        = "60000";     // mHz
                                        // ----------------------------------------------------

                                        gain_factor              = "1000";
                                        exposure_factor          = "1000000";
                                        framerate_factor         = "1000";
                                        inherent_gain            = "1";
                                        min_hdr_ratio            = "1";
                                        max_hdr_ratio            = "1";

                                        line_length              = "0";
                                        phy_mode                 = "DPHY";
                                        discontinuous_clk        = "no";
                                        mclk_khz                 = "24000";

                                        pix_clk_hz               = PIX_CLK_HZ;
                                        mclk_multiplier          = "0.0";       // deprecated
                                        cil_settletime           = "0";
                                        dpcm_enable              = "false";
                                };

                                ports {
                                        #address-cells = <1>;
                                        #size-cells = <0>;
                                        port@0 {
                                                reg = <0>;
                                                vc_mipi_out1: endpoint {
                                                        port-index = <2>;
                                                        bus-width = <BUS_WIDTH>;
                                                        remote-endpoint = <&vc_csi_in1>;
                                                };
                                        };
                                };
                        };
                };
#endif // VC_MIPI_CAM_1
        };

        lens@vc_mipi {
                min_focus_distance  = "0.0";
                hyper_focal         = "0.0";
                focal_length        = "6.0";
                f_number            = "2.0";
                aperture            = "0.0";
        };
};

/ {
        tcp: tegra-camera-platform {
                compatible = "nvidia, tegra-camera-platform";

                num_csi_lanes = <NUM_CSI_LANES>;
                max_lane_speed = <1500000>;
                min_bits_per_pixel = <10>;
                vi_peak_byte_per_pixel = <2>;
                vi_bw_margin_pct = <25>;
                max_pixel_rate = <240000>;
                isp_peak_byte_per_pixel = <5>;
                isp_bw_margin_pct = <25>;

                modules {
#if VC_MIPI_CAM_0 == 1
                        cam_module0: module0 {
                                badge = "jakku_front_vc_mipi";
                                position = "front";
                                status = "okay";
                                orientation = "1";
                                cam_module0_drivernode0: drivernode0 {
                                        pcl_id = "v4l2_sensor";
#if VC_MIPI_MANUFACTURER == 1
                                        devname = "vc_mipi 9-001a";
                                        proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/vc_mipi@1a";
#else
                                        devname = "vc_mipi 9-0060";
                                        proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/vc_mipi@60";
#endif
                                };
                                cam_module0_drivernode1: drivernode1 {
                                        pcl_id = "v4l2_lens";
                                        proc-device-tree = "/proc/device-tree/lens@vc_mipi/";
                                };
                        };
#endif // VC_MIPI_CAM_0

#if VC_MIPI_CAM_1 == 1
                        cam_module1: module1 {
                                badge = "jakku_rear_vc_mipi";
                                position = "rear";
                                status = "okay";
                                orientation = "1";
                                cam_module1_drivernode0: drivernode0 {
                                pcl_id = "v4l2_sensor";
#if VC_MIPI_MANUFACTURER == 1
                                devname = "vc_mipi 10-001a";
                                proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@1/vc_mipi@1a";
#else
                                devname = "vc_mipi 10-0060";
                                proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@1/vc_mipi@60";
#endif	
                                };
                                cam_module1_drivernode1: drivernode1 {
                                        pcl_id = "v4l2_lens";
                                        proc-device-tree = "/proc/device-tree/lens@vc_mipi/";
                                };
                        };
#endif // VC_MIPI_CAM_1
                };
        };
};

#define CAM_I2C_MUX	TEGRA234_AON_GPIO(CC, 3)

/ {
        cam_i2cmux {
                status = "okay";
                compatible = "i2c-mux-gpio";
                #address-cells = <1>;
                #size-cells = <0>;
                mux-gpios = <&tegra_aon_gpio CAM_I2C_MUX GPIO_ACTIVE_HIGH>;
                i2c-parent = <&cam_i2c>;
#if VC_MIPI_CAM_0 == 1
                i2c@0 {
                        status = "okay";
                        reg = <0>;
                        #address-cells = <1>;
                        #size-cells = <0>;
#if VC_MIPI_MANUFACTURER == 1
                        vc_mipi@1a {
#else
                        vc_mipi@60 {
#endif
                                reset-gpios = <&tegra_main_gpio CAM0_PWDN GPIO_ACTIVE_HIGH>;
                        };
                };
#endif
#if VC_MIPI_CAM_1 == 1
                i2c@1 {
                        status = "okay";
                        reg = <1>;
                        #address-cells = <1>;
                        #size-cells = <0>;
#if VC_MIPI_MANUFACTURER == 1
                        vc_mipi@1a {
#else
                        vc_mipi@60 {
#endif
                                reset-gpios = <&tegra_main_gpio CAM1_PWDN GPIO_ACTIVE_HIGH>;
                        };
                };
#endif
	};

        gpio@2200000 {
                camera-control-output-low {
                        gpio-hog;
                        output-low;
                        gpios = <CAM0_PWDN 0 CAM1_PWDN 0>;
                        label = "cam0-pwdn","cam1-pwdn";
                };
        };
};

The kernel sees the driver but the gstreamer gives streaming error:

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1440,height=1080,framerate=20/1' ! autovideosink
Setting pipeline to PAUSED ...

Using winsys: x11 

Using winsys: x11 
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0-actual-sink-nveglgles': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1440 x 1080 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 0.000000, max 48.000000; Exposure Range min 1000, max 1000000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 1440 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.593696096
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success

(gst-launch-1.0:2210): GStreamer-CRITICAL **: 08:59:48.352: gst_object_unref: assertion '((GObject *) object)->ref_count > 0' failed
Freeing pipeline ...

Any suggestions?

Thanks in advance.

Best regards,

Mert T.

@bazo80
Copy link
Collaborator

bazo80 commented Feb 19, 2025

Hello @mertturanli ,

have you tried it directly on the Jetson? Because ssh connection with X11 forwarding won't work in most cases. Alternatively you could use a vnc connection to the Jetson and run the same command there.

best regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants