Nvidia Nano平台deepstream-app视频颠倒调试记录

1.前言

试图运行deepstream-app与示例

“source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt”

在Jetson Nano上与“deepstream-4.0_4.0-1_arm64.deb” 使用Raspberry Pi V2相机NoIr，

但图像显示面朝上。

修改配置

[source0]
enable = 1
#Type - 1 = CameraV4L2 2 = URI 3 = MultiURI 4 = RTSP 5 = Camera (CSI) (Jetson only)
type = 5
intra-decode-enable = 1
gpu-id = 0
camera-id = 0
camera-width = 1920
camera-height = 1280
camera-fps-n = 30
camera-fps-d = 1
camera-csi-sensor-id = 0
drop-frame-interval = 0

当运行时，图像出现颠倒。

另一方面，当在同一台计算机上执行jetson推理示例时 ~/git/jetson-inference/build/aarch64/bin ./camera-viewer 图像看起来没问题，并在管道中显示 “nvvidconv flip-method = 2”。

那么问题是: 如何在配置文件中指示相机的方向来运行deepstream-app?

JetPack版本:4.2.2

2. 修改create_camera_source_bin

nvvideoconvert中不支持' flip method '属性。

修改create_camera_source_bin()运行如下命令：

nvarguscamerasrc bufapi-version=0 ! nvvidconv flip-method=2 ! video/x-raw,forma=NV12 ! nvvideoconvert ! video/x-raw(memory:NVMM),format=NV12

修改源代码后执行 log：

./deepstream-app -c ../../../../samples/configs/deepstream-app/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano-csiCam-test01.txt --gst-debug=3 2>&1 | tee debug.txt 
0:00:00.242025044 10454   0x557ee9c290 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/mherrera/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)

(deepstream-app:10454): GLib-GObject-WARNING **: 02:31:43.653: cannot register existing type 'GstInterpolationMethod'

(deepstream-app:10454): GLib-GObject-CRITICAL **: 02:31:43.654: g_param_spec_enum: assertion 'G_TYPE_IS_ENUM (enum_type)' failed

(deepstream-app:10454): GLib-GObject-CRITICAL **: 02:31:43.654: validate_pspec_to_install: assertion 'G_IS_PARAM_SPEC (pspec)' failed
0:00:11.051427092 10454   0x557ef64850 FIXME                default gstutils.c:3981:gst_pad_create_stream_id_internal: Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Creating LL OSD context new

Runtime commands:
    h: Print this help
    q: Quit

    p: Pause
    r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg) 
**PERF: 0.00 (0.00) 
** INFO: : Pipeline ready

** INFO: : Pipeline running

Creating LL OSD context new
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: PowerService: requested_clock_Hz=627200000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: nvbuffer Payload Type not supported
NvBufferGetParams failed for src_dmabuf_fd
nvbuffer_transform Failed
gst_nvvconv_transform: NvBufferTransform Failed 
0:00:11.977186035 10454   0x557ef64850 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
0:00:11.977212963 10454   0x557ef64850 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason error (-5)
ERROR from src_elem: Internal data stream error.
Debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstNvArgusCameraSrc:src_elem:
streaming stopped, reason error (-5)
Quitting
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
App run failed
GST_ARGUS: 
PowerServiceHwVic::cleanupResources
mherrera@JN01EXPL:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-app$ 

它应该是' bufapi-version=FALSE '

g_object_set (G_OBJECT (bin->src_elem), "bufapi-version", FALSE, NULL);

更新代码

g_object_set (G_OBJECT (bin->src_elem), "bufapi-version", FALSE, NULL);

log：

ERROR from src_bin_muxer: Input buffer number of surfaces (0) must be equal to mux->num_surfaces_per_frame (1)

作为参考，附加了输出文件和deepstream_source_bin.c。

尝试了不同的代码更新deepstream_source_bin.c没有成功。

回到原来的代码，只使用之前的建议进行更新，错误信息是一样的。

3.使用NvBufSurface api

下面是一个补丁供参考: sources\apps\apps-common\src\deepstream_source_bin.c:

include 

+#include "nvbufsurface.h"
+#include "nvbufsurftransform.h"
 #include "gstnvdsmeta.h"
 #include "deepstream_common.h"
 #include "deepstream_sources.h"
@@ -62,6 +64,64 @@ set_camera_v4l2_params (NvDsSourceConfig * config, NvDsSrcBin * bin)
   return TRUE;
 }

+static GstPadProbeReturn
+nvargus_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
+    gpointer u_data)
+{
+    GstBuffer *buf = (GstBuffer *) info->data;
+    GstMapInfo outmap = GST_MAP_INFO_INIT;
+    gst_buffer_map (buf, &outmap, GST_MAP_WRITE);
+    NvBufSurface*  surface = (NvBufSurface *)outmap.data;
+
+    NvBufSurfTransformRect src_rect, dst_rect;
+    src_rect.top   = 0;
+    src_rect.left  = 0;
+    src_rect.width = (guint) surface->surfaceList[0].width;
+    src_rect.height= (guint) surface->surfaceList[0].height;
+
+    dst_rect.top   = 0;
+    dst_rect.left  = 0;
+    dst_rect.width = (guint) surface->surfaceList[0].width;
+    dst_rect.height= (guint) surface->surfaceList[0].height;
+
+    NvBufSurface *dst_surface = NULL;
+    NvBufSurfaceCreateParams nvbufsurface_create_params;
+
+    nvbufsurface_create_params.gpuId  = surface->gpuId;
+    nvbufsurface_create_params.width  = (gint) surface->surfaceList[0].width;
+    nvbufsurface_create_params.height = (gint) surface->surfaceList[0].height;
+    nvbufsurface_create_params.size = 0;
+    nvbufsurface_create_params.colorFormat = surface->surfaceList[0].colorFormat;
+    nvbufsurface_create_params.layout = surface->surfaceList[0].layout;
+    nvbufsurface_create_params.memType = surface->memType;
+
+    NvBufSurfaceCreate(&dst_surface,1,&nvbufsurface_create_params);
+
+    NvBufSurfTransformParams nvbufsurface_params;
+    nvbufsurface_params.src_rect = &src_rect;
+    nvbufsurface_params.dst_rect = &dst_rect;
+    nvbufsurface_params.transform_flag =  0;
+    nvbufsurface_params.transform_filter = NvBufSurfTransformInter_Default;
+
+    NvBufSurfTransformConfigParams transform_config_params;
+    NvBufSurfTransform_Error err;
+
+    transform_config_params.compute_mode = NvBufSurfTransformCompute_Default;
+    transform_config_params.gpu_id = surface->gpuId;
+    transform_config_params.cuda_stream = NULL;
+    err = NvBufSurfTransformSetSessionParams (&transform_config_params);
+    // copy to dst_surface
+    err = NvBufSurfTransform (surface, dst_surface, &nvbufsurface_params);
+    // rototate 180 degree to original surface
+    nvbufsurface_params.transform_flag =  NVBUFSURF_TRANSFORM_FLIP;
+    nvbufsurface_params.transform_flip = NvBufSurfTransform_Rotate180;
+    err = NvBufSurfTransform (dst_surface, surface, &nvbufsurface_params);
+    NvBufSurfaceDestroy(dst_surface);
+
+    gst_buffer_unmap (buf, &outmap);
+    return GST_PAD_PROBE_OK;
+}
+
 static gboolean
 create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)
 {
@@ -144,12 +204,16 @@ create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)
     NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter, "src");

   } else {
+    GstPad *src_pad;

     g_object_set (G_OBJECT (bin->cap_filter), "caps", caps, NULL);

     gst_bin_add_many (GST_BIN (bin->bin), bin->src_elem, bin->cap_filter, NULL);

     NVGSTDS_LINK_ELEMENT (bin->src_elem, bin->cap_filter);
+    src_pad = gst_element_get_static_pad (bin->src_elem, "src");
+    gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER,
+        nvargus_src_pad_buffer_probe, NULL, NULL);

     NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter, "src");
   }

sources\apps\sample_apps\deepstream-app\Makefile

-24,6 +24,7 @@ APP:= deepstream-app

 TARGET_DEVICE = $(shell gcc -dumpmachine | cut -f1 -d -)

+CUDA_VER:=10.0
 NVDS_VERSION:=4.0

 LIB_INSTALL_DIR?=/opt/nvidia/deepstream/deepstream-$(NVDS_VERSION)/lib/
@@ -41,10 +42,10 @@ PKGS:= gstreamer-1.0 gstreamer-video-1.0 x11

 OBJS:= $(SRCS:.c=.o)

-CFLAGS+= -I../../apps-common/includes -I../../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4
+CFLAGS+= -I../../apps-common/includes -I../../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 -I /usr/local/cuda-$(CUDA_VER)/include

 LIBS+= -L$(LIB_INSTALL_DIR) -lnvdsgst_meta -lnvds_meta -lnvdsgst_helper -lnvds_utils -lm \
-       -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,$(LIB_INSTALL_DIR)
+       -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,$(LIB_INSTALL_DIR) -lnvbufsurface -lnvbufsurftransform

 CFLAGS+= `pkg-config --cflags $(PKGS)`

按照上述修改后

它起作用了，但只是部分起作用。

RTSP的一些帧是旋转的，一些不是，它看起来是随机的

已经将建议的补丁应用到源代码\apps\apps-common\src\deepstream_source_bin.c:

} else{
NVGSTDS_LINK_ELEMENT (bin->cap_filter, bin->nvvidconv);
NVGSTDS_LINK_ELEMENT (bin->nvvidconv, bin->cap_filter1);

/// added here, line 888
GstPad *src_pad;
src_pad = gst_element_get_static_pad (bin->nvvidconv, "src");
gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER,
    nvargus_src_pad_buffer_probe, NULL, NULL);
///

NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter1, "src");

以下是deepstream配置文件:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=4
columns=2
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://192.168.2.216:8554/test
#num-sources=8
#drop-frame-interval=2
latency=200
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=testf.mp4
source-id=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=8
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
## attach-sys-ts-as-ntp=1

看起来添加了prob函数nvvideoconvert的源垫，而不是nvarguscamerasrc。

确保申请了这个

+    src_pad = gst_element_get_static_pad (bin->src_elem, "src");
+    gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER,
+        nvargus_src_pad_buffer_probe, NULL, NULL);

4. 调试gst_element_get_static_pad

怀疑之前的猜测，有些帧是旋转的，有些没有，是有缺陷的。

所以进一步测试它

vbufsurface_params.transform_flip = NvBufSurfTransform_Rotate90;

事实上，我的怀疑被证明是正确的。 旋转确实起作用，但似乎同一个坐标系被旋转了几次并被推到下游。

所以不是有一个90度的旋转流，有一些帧旋转了90度，180度，270度，360度， 按照这个特定的顺序。

请注意，上面的视频发生时，试图修补源代码\apps\apps-common\src\deepstream_source_bin.c:

src_pad = gst_element_get_static_pad (bin->cap_filter, "src");
or:
src_pad = gst_element_get_static_pad (bin->nvvidconv, "src");
or:
src_pad = gst_element_get_static_pad (bin->cap_filter1, "src");

尝试探测src_elem

src_pad = gst_element_get_static_pad (bin->src_elem, "src");

但是视频完全没有旋转。 请注意，这是用于RTSP输入，如果有帮助的话。 补丁在这个函数中:

static gboolean
create_rtsp_src_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)
{
   ....
}