本文主要是介绍完整指南：CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (二) 源码架构流程梳理、代码编写，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
目录
1 视频解码代码编写----利用jetson-ffmpeg
1.1 nvstream中视频解码的代码流程框架
1.1.1 类的层次关系
1.1.2 各个类的初始化函数调用层次关系
1.1.3 各个类的process函数调用层次关系
1.2 编写视频解码代码
1.2.1 修改VideoInfo结构体定义
1.2.2 修改解封装代码
1.2.3 decode_impl_nv.hpp
1.2.4 decode_impl_nv.cpp
2 硬件相关的图像格式、内存申请接口、内存释放、内存释放等代码修改
2.1 infer_server/include/common/utils.hpp文件内容修改
2.2 cuda的四种内存
2.3 infer_server/src/core/device.cpp修改
3 图像缩放、裁剪、色域转换等代码编写----利用CV-CUDA
3.1 nvstream中图像处理代码流程框架
3.1.1 类的层次关系
3.1.2 各个类的初始化函数调用层次关系
3.1.3 各个类的transform函数调用层次关系
3.2 编写图像缩放、裁剪、色域转换等代码
3.2.1 infer_server/src/nv/transform_impl_nv.hpp
3.2.2 infer_server/src/nv/transform_impl_nv.cpp
4 算法推理相关代码修改
4.1 ./infer_server/src/model/model.h
4.2 ./infer_server/src/model/model.cpp
5 其他代码修改
参考文献：
记录下将CNStream流处理多路并发Pipeline框架适配到NVIDIA Jetson AGX Orin的过程，以及过程中遇到的问题，我的jetson盒子是用jetpack5.1.3重新刷机之后的，这是系列博客的第二篇。
1 视频解码代码编写----利用jetson-ffmpeg
用jetson-ffmpeg来做视频解码，本质上还是用ffmpeg解码，是不是jetson-ffmpeg底层调用了英伟达的nvmpi做硬件加速。
1.1 nvstream中视频解码的代码流程框架
视频解码的代码流程框架在博客：aclStream流处理多路并发Pipeline框架中 视频解码 代码调用流程整理、类的层次关系整理、回调函数赋值和调用流程整理-CSDN博客
上面博客里面的是详细代码阅读，从中提炼出来简单点的框架
1.1.1 类的层次关系
class FileHandler类里面有个FileHandlerImpl *impl_ = nullptr成员-->class FileHandlerImpl-->class FFParser 解封装的类-->class DeviceDecoder-->class DecodeService-->class IDecoder 只是个虚基类，被用来继承的 -->IDecoder *decoder_ = CreateDecoder();这里面就是new DecoderAcl()了。 DecoderAcl继承IDecoder -->再往下就是各种的硬件相关的类了， 上面是类的关系，这次英伟达平台上就是从IDecoder *decoder_ = CreateDecoder();开始写一个新的类然后用来做具体的解码，上层的那些类还是保持不变的。
1.1.2 各个类的初始化函数调用层次关系
FileHandlerImpl::Open()std::thread(&FileHandlerImpl::LoopPrepareResources()parser_.Openimpl_->Open(url, result, only_key_frame);result_->OnParserInfo(info); decoder_->Create(info, &extra)这个decoder_就是DeviceDecoder类VdecCreate(void **vdec, VdecCreateParams *params) 这是个单纯的函数，不在任何类里面，函数内容在下一行infer_server::DecodeService::Instance().Create(vdec, params);DecodeService::Create里面有下面两行IDecoder *decoder_ = CreateDecoder()先创建DecoderAcl decoder_->Create(params)然后调用DecoderAcl 的create函数， 再往下就是硬件相关的各种类的init和open函数了，这次英伟达的也是要创建个新的类，然后在这里新类的open或者init函数被调用 1.1.3 各个类的process函数调用层次关系
process分两个方向，一个是从上到下送解码数据的，另一个是解码完之后获取解码数据，然后从下往上回传的。先看从上到下送frame的流程
bool FileHandlerImpl::Process() {
parser_.Parse();impl_->Parse();result_->OnParserFrame(&frame);DeviceDecoder::Process(VideoEsPacket *pkt) { int VdecSendStream(void *vdec, const VdecStream *stream, int timeout_ms){return infer_server::DecodeService::Instance().SendStream(vdec, stream, timeout_ms);}decoder_->SendStream(stream, timeout_ms);这就已经是DecoderAcl的sendstream了，已经到硬件了。vdec_->Decode(data_ptr, data_size, frame_id, this);vdec_是AclLiteVideoProc再往下就不看了，下面几层类都是硬件解码的。然后看一下从下往上回传的，就是解码之后的数据一层层传到上层的类。
VideoDecoder::DvppVdecCallbackV2(hi_video_frame_info *frame, void *userdata) {CallBackVdec(const std::shared_ptr<acllite::ImageData> decoded_image, uint32_t channel_id, uint32_t frame_id, vdecoder->OnFrame(decoded_image, channel_id, frame_id);这个decoder就是DecoderAcl类，create_params_.OnFrame(surf, create_params_.userdata);OnFrame_(BufSurface *surf, void *userdata)class DeviceDecoder类result_->OnDecodeFrame(wrapper);FileHandlerImpl::OnDecodeFrame1.2 编写视频解码代码
从上面分析可以知道，需要写一个类，替换掉之前的底层硬件解码类，在jetson上，用jetson-ffmpeg做解码，jetson-ffmpeg会调用硬件加速。
1.2.1 修改VideoInfo结构体定义
首先修改一个结构体的定义，因为之前的cnstream中解码不是用ffmpeg解码的，他只是用ffmpeg解封装，所以这个结构体定义是这样的modules/source/src/video_parser.hpp，他由于不用ffmpeg解码所以不需要 AVCodecParameters* codecpar = nullptr成员。
namespace cnstream {struct VideoInfo {AVCodecID codec_id;
#ifdef HAVE_FFMPEG_AVDEVICE // for usb cameraint format;int width;int height;
#endifint progressive;std::vector<unsigned char> extra_data;
};...其他代码...然后在解码的demo那里，easydk/samples/simple_demo/common/video_parser.h，还有一个结构体的定义。
struct VideoInfo {AVCodecID codec_id = AV_CODEC_ID_NONE;
#if LIBAVFORMAT_VERSION_INT >= FFMPEG_VERSION_3_1AVCodecParameters* codecpar = nullptr;
#endifAVCodecContext* codec_ctx = nullptr;std::vector<uint8_t> extra_data{};int width = 0;int height = 0;int progressive = 0;
};所以我这里要修改一下这个结构体的定义，把第一个的结构体定义改成下面的格式。
namespace cnstream {struct VideoInfo {AVCodecID codec_id;
#if LIBAVFORMAT_VERSION_INT >= FFMPEG_VERSION_3_1AVCodecParameters* codecpar = nullptr;
#endifAVCodecContext* codec_ctx = nullptr;std::vector<unsigned char> extra_data;int width;int height;int progressive;
#ifdef HAVE_FFMPEG_AVDEVICE // for usb cameraint format;
#endif
};1.2.2 修改解封装代码
然后解封装那里的代码，需要修改一下给AVCodecParameters* codecpar成员赋值。参考CNStream中easydk/samples/simple_demo/common/video_parser.cpp的代码去修改NVStream中modules/source/src/video_parser.cpp的代码，
int Open(const std::string& url, IParserResult* result, bool only_key_frame = false) {std::unique_lock<std::mutex> guard(mutex_);if (!result) return -1;result_ = result;// format contextfmt_ctx_ = avformat_alloc_context();if (!fmt_ctx_) {return -1;}url_name_ = url;AVInputFormat* ifmt = NULL;// for usb camera
#ifdef HAVE_FFMPEG_AVDEVICEconst char* usb_prefix = "/dev/video";if (0 == strncasecmp(url_name_.c_str(), usb_prefix, strlen(usb_prefix))) {// open v4l2 input
#if defined(__linux) || defined(__unix)ifmt = av_find_input_format("video4linux2");if (!ifmt) {LOGE(SOURCE) << "[" << stream_id_ << "]: Could not find v4l2 format.";return false;}
#elif defined(_WIN32) || defined(_WIN64)ifmt = av_find_input_format("dshow");if (!ifmt) {LOGE(SOURCE) << "[" << stream_id_ << "]: Could not find dshow.";return false;}
#elseLOGE(SOURCE) << "[" << stream_id_ << "]: Unsupported Platform";return false;
#endif}
#endifint ret_code;const char* p_rtsp_start_str = "rtsp://";if (0 == strncasecmp(url_name_.c_str(), p_rtsp_start_str, strlen(p_rtsp_start_str))) {AVIOInterruptCB intrpt_callback = { InterruptCallBack, this };fmt_ctx_->interrupt_callback = intrpt_callback;last_receive_frame_time_ = GetTickCount();// optionsav_dict_set(&options_, "buffer_size", "1024000", 0);av_dict_set(&options_, "max_delay", "500000", 0);av_dict_set(&options_, "stimeout", "20000000", 0);av_dict_set(&options_, "rtsp_flags", "prefer_tcp", 0);rtsp_source_ = true;}else {// optionsav_dict_set(&options_, "buffer_size", "1024000", 0);av_dict_set(&options_, "max_delay", "500000", 0);}// open inputret_code = avformat_open_input(&fmt_ctx_, url_name_.c_str(), ifmt, &options_);if (0 != ret_code) {LOGI(SOURCE) << "[" << stream_id_ << "]: Couldn't open input stream -- " << url_name_;return -1;}// find video stream informationret_code = avformat_find_stream_info(fmt_ctx_, NULL);if (ret_code < 0) {LOGI(SOURCE) << "[" << stream_id_ << "]: Couldn't find stream information -- " << url_name_;return -1;}// fill infoVideoInfo video_info;VideoInfo* info = &video_info;int video_index = -1;AVStream* st = nullptr;for (uint32_t loop_i = 0; loop_i < fmt_ctx_->nb_streams; loop_i++) {st = fmt_ctx_->streams[loop_i];
#if LIBAVFORMAT_VERSION_INT >= FFMPEG_VERSION_3_1if (st->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
#elseif (st->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
#endifvideo_index = loop_i;break;}}if (video_index == -1) {LOGI(SOURCE) << "[" << stream_id_ << "]: Couldn't find a video stream -- " << url_name_;return -1;}video_index_ = video_index;info->width = st->codecpar->width;info->height = st->codecpar->height;#if LIBAVFORMAT_VERSION_INT >= FFMPEG_VERSION_3_1info->codec_id = st->codecpar->codec_id;int field_order = st->codecpar->field_order;
#ifdef HAVE_FFMPEG_AVDEVICE // for usb camerainfo->format = st->codecpar->format;
#endif
#elseinfo->codec_id = st->codec->codec_id;int field_order = st->codec->field_order;
#ifdef HAVE_FFMPEG_AVDEVICE // for usb camerainfo->format = st->codec->format;
#endif#endif#if LIBAVFORMAT_VERSION_INT >= FFMPEG_VERSION_3_1info->codecpar = fmt_ctx_->streams[video_index_]->codecpar;
#endifinfo->codec_ctx = fmt_ctx_->streams[video_index_]->codec;/*At this moment, if the demuxer does not set this value (avctx->field_order == UNKNOWN),* the input stream will be assumed as progressive one.*/switch (field_order) {case AV_FIELD_TT:case AV_FIELD_BB:case AV_FIELD_TB:case AV_FIELD_BT:info->progressive = 0;break;case AV_FIELD_PROGRESSIVE: // fall throughdefault:info->progressive = 1;break;}#if LIBAVFORMAT_VERSION_INT >= FFMPEG_VERSION_3_1unsigned char* extradata = st->codecpar->extradata;int extradata_size = st->codecpar->extradata_size;
#elseunsigned char* extradata = st->codec->extradata;int extradata_size = st->codec->extradata_size;
#endifif (extradata && extradata_size) {info->extra_data.resize(extradata_size);memcpy(info->extra_data.data(), extradata, extradata_size);}// bitstream filterbsf_ctx_ = nullptr;const AVBitStreamFilter *pfilter{};if (strstr(fmt_ctx_->iformat->name, "mp4") || strstr(fmt_ctx_->iformat->name, "flv") ||strstr(fmt_ctx_->iformat->name, "matroska")) {if (AV_CODEC_ID_H264 == info->codec_id) {pfilter = av_bsf_get_by_name("h264_mp4toannexb");}else if (AV_CODEC_ID_HEVC == info->codec_id) {pfilter = av_bsf_get_by_name("hevc_mp4toannexb");}else {pfilter = nullptr;}}if (pfilter == nullptr) {bsf_ctx_ = nullptr;}else {av_bsf_alloc(pfilter, &bsf_ctx_);}if (result_) {result_->OnParserInfo(info);}av_init_packet(&packet_);first_frame_ = true;eos_reached_ = false;open_success_ = true;only_key_frame_ = only_key_frame;return 0;}1.2.3 decode_impl_nv.hpp
创建个新的类，用来做视频解码，初步代码如下，后面编译以及运行的时候有错误再修改。
#ifndef _DECODE_IMPL_NV_HPP_
#define _DECODE_IMPL_NV_HPP_#include <atomic>
#include <string>#include <libavformat/avformat.h>
#include <libavcodec/avcodec.h>
#include "../decode_impl.hpp"namespace infer_server {class DecodeFFmpeg : public IDecoder {public:DecodeFFmpeg() = default;~DecodeFFmpeg() = default;int Create(VdecCreateParams *params) override;int Destroy() override;int SendStream(const VdecStream *stream, int timeout_ms) override;void OnFrame(AVFrame *av_frame_, uint32_t frame_id) override;void OnEos() override;void OnError(int errcode) override;private:void ResetFlags();private:std::atomic<bool> eos_sent_{false}; // flag for acl eos has been sent to decoderstd::atomic<bool> created_{false};VdecCreateParams create_params_;AVCodec *av_codec_;AVCodecContext* codec_context_{nullptr};AVFrame *av_frame_ = nullptr;void *transformer_{};
};} // namespace
#endif // DECODE_IMPL_NV_HPP
1.2.4 decode_impl_nv.cpp
#include <iostream>#include "glog/logging.h"
#include "decode_impl_nv.hpp"
#include "defer.hpp"namespace infer_server {bool DecodeFFmpeg::Create(VdecCreateParams *params) {create_params_ = *params;switch (params->type) {case VDEC_TYPE_H264:av_codec_ = avcodec_find_decoder(AV_CODEC_ID_H264);break;case VDEC_TYPE_H265:av_codec_ = avcodec_find_decoder(AV_CODEC_ID_H265);break;case VDEC_TYPE_JPEG:default:LOG(ERROR) << "[InferServer] [DecodeFFmpeg] Create(): Unsupported codec type: " << create_params_.type;return -1;}if (!av_codec_) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] avcodec_find_decoder failed";return false;}codec_context_ = avcodec_alloc_context3(av_codec_);if (!codec_context_) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] Failed to do avcodec_alloc_context3";return false;}AVDictionary *decoder_opts = nullptr;defer([&decoder_opts] {if (decoder_opts) av_dict_free(&decoder_opts);});av_dict_set_int(&decoder_opts, "device_id", 0, 0);if (avcodec_open2(codec_context_, av_codec_, &decoder_opts) < 0) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] Failed to open codec";return false;}av_frame_ = av_frame_alloc();if (!av_frame_) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] Could not alloc frame";return false;} ResetFlags();//待修改//这里还需要增加创建transform用来做图像缩放的代码。created_ = true;return true;}int DecodeFFmpeg::SendStream(const VdecStream *stream, int timeout_ms) {if (!created_) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] SendStream(): Decoder is not created";return -1;}if (nullptr == stream || nullptr == stream->bits) {if (eos_sent_) {LOG(WARNING) << "[InferServer] [DecodeFFmpeg] SendStream(): EOS packet has been send";return 0;}AVPacket framePacket = {};av_init_packet(&framePacket);framePacket.data = nullptr;framePacket.size = 0;avcodec_send_packet(decode_, &packet);// flush all frames ...int ret = 0;do {ret = avcodec_receive_frame(decode_, av_frame_);if(ret >= 0){OnFrame(av_frame_, stream->pts);}} while (ret >= 0);eos_sent_ = true;OnEos();}else{if (eos_sent_) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] SendStream(): EOS has been sent, process packet failed, pts:"<< stream->pts;return -1;}AVPacket framePacket = {};av_init_packet(&framePacket);framePacket.data = stream.bits;framePacket.size = stream.len;//开始解码int ret = avcodec_send_packet(codec_context_, framePacket);if (ret < 0) {LOG(ERROR) << "[InferServer] [DecodeFFmpeg] avcodec_send_packet failed, data ptr, size:"<< framePacket->data << ", " << framePacket->size;return false;}ret = avcodec_receive_frame(codec_context_, av_frame_);OnFrame(av_frame_, stream->pts);}}void DecodeFFmpeg::OnFrame(AVFrame *av_frame_, uint32_t frame_id) {BufSurface *surf = nullptr;if (create_params_.GetBufSurf(&surf, av_frame_->width, av_frame_->height, CastColorFmt(av_frame_->format),create_params_.surf_timeout_ms, create_params_.userdata) < 0) {LOG(ERROR) << "[InferServer] [DecoderAcl] OnFrame(): Get BufSurface failed";OnError(-1);return;}if (surf->mem_type != BUF_MEMORY_DVPP) {LOG(ERROR) << "[InferServer] [DecoderAcl] OnFrame(): BufSurface memory type must be BUF_MEMORY_DVPP";return;}switch (av_frame_->format) {case acllite::ImageFormat::YUV_SP_420:case acllite::ImageFormat::YVU_SP_420:if (surf->surface_list[0].width != av_frame_->width || surf->surface_list[0].height != av_frame_->height) {BufSurface transform_src;BufSurfaceParams src_param;memset(&transform_src, 0, sizeof(BufSurface));memset(&src_param, 0, sizeof(BufSurfaceParams));src_param.color_format = CastColorFmt(av_frame_->format);src_param.data_size = codec_image->size;//待修改。src_param.data_ptr = reinterpret_cast<void *>(codec_image->data.get());//待修改。VLOG(5) << "[InferServer] [DecoderAcl] OnFrame(): codec_frame: "<< " width = " << av_frame_->width<< ", height = " << av_frame_->height<< ", width stride = " << av_frame_->alignWidth<< ", height stride = " << av_frame_->alignHeight;VLOG(5) << "[InferServer] [DecoderAcl] OnFrame(): surf->surface_list[0]: "<< " width = " << surf->surface_list[0].width<< ", height = " << surf->surface_list[0].height<< ", width stride = " << surf->surface_list[0].width_stride<< ", height stride = " << surf->surface_list[0].height_stride;src_param.width = av_frame_->width;src_param.height = av_frame_->height;src_param.width_stride = codec_image->alignWidth;//待修改。src_param.height_stride = codec_image->alignHeight;//待修改。transform_src.batch_size = 1;transform_src.num_filled = 1;transform_src.device_id = create_params_.device_id;transform_src.mem_type = BUF_MEMORY_DVPP;transform_src.surface_list = &src_param;TransformParams trans_params;memset(&trans_params, 0, sizeof(trans_params));trans_params.transform_flag = TRANSFORM_RESIZE_SRC;if (Transform(transformer_, &transform_src, surf, &trans_params) < 0) {LOG(ERROR) << "[InferServer] [DecoderAcl] OnFrame(): Transfrom failed";break;}}else {std::chrono::high_resolution_clock::time_point tnow = std::chrono::high_resolution_clock::now();CALL_ACL_FUNC(acllite::CopyDataToHostEx(surf->surface_list[0].data_ptr, codec_image->size, codec_image->data.get(), codec_image->size, codec_image->deviceId), "[DecoderAcl] OnFrame(): copy codec buffer data to surf failed");std::chrono::high_resolution_clock::time_point tpost = std::chrono::high_resolution_clock::now();//std::cout << "<<<<<<================================ CopyDataToHostEx time = " << std::chrono::duration_cast<std::chrono::duration<double>>(tpost - tnow).count() * 1000 << " ms" << std::endl;}break;default:break;}surf->pts = frame_id;//std::chrono::high_resolution_clock::time_point tnow = std::chrono::high_resolution_clock::now();create_params_.OnFrame(surf, create_params_.userdata);//std::chrono::high_resolution_clock::time_point tpost = std::chrono::high_resolution_clock::now();//std::cout << "<<<<<<================================ create_params_.OnFrame time = " << std::chrono::duration_cast<std::chrono::duration<double>>(tpost - tnow).count() * 1000 << " ms" << std::endl;}void DecodeFFmpeg::Destroy() {if (!created_) {LOG(WARNING) << "[InferServer] [DecoderAcl] Destroy(): Decoder is not created";return 0;}// if error happened, destroy directly, eos maybe not be transmitted from the decoderif (!eos_sent_) {SendStream(nullptr, 10000);}ResetFlags();if (av_frame_) {av_frame_free(&av_frame_);av_frame_ = nullptr;}if (codec_context_) {avcodec_close(codec_context_);avcodec_free_context(&codec_context_);codec_context_ = nullptr;}//待修改，还需要增加销毁transform的相关代码。}DecodeFFmpeg::~DecodeFFmpeg() {DecodeFFmpeg::Destroy();}void DecodeFFmpeg::ResetFlags() {eos_sent_ = false;created_ = false;}void DecodeFFmpeg::OnEos() {create_params_.OnEos(create_params_.userdata);}void DecodeFFmpeg::OnError(int errcode) {//convert the error codecreate_params_.OnError(static_cast<int>(errcode), create_params_.userdata);}} // namespace 2 硬件相关的图像格式、内存申请接口、内存释放、内存释放等代码修改
infer_server/include/common/utils.hpp文件内容如下
/************************************************************************** Copyright (C) [2022] by Cambricon, Inc. All rights reserved** Licensed under the Apache License, Version 2.0 (the "License");* you may not use this file except in compliance with the License.* You may obtain a copy of the License at** http://www.apache.org/licenses/LICENSE-2.0** The above copyright notice and this permission notice shall be included in* all copies or substantial portions of the Software.* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS* OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN* THE SOFTWARE.*************************************************************************/#ifndef COMMON_UTILS_HPP_
#define COMMON_UTILS_HPP_#include <string>#include <glog/logging.h>#include "buf_surface.h"#include "AclLite/AclLite.h"#define _SAFECALL(func, expected, msg, ret_val) \do { \int _ret = (func); \if ((expected) != _ret) { \LOG(ERROR) << "[InferServer] Call [" << #func << "] failed, ret = " << _ret << ". " << msg; \return (ret_val); \} \} while (0)#define ACL_SAFECALL(func, msg, ret_val) _SAFECALL(func, acllite::ACLLITE_OK, msg, ret_val)#define _CALLFUNC(func, expected, msg) \do { \int _ret = (func); \if ((expected) != _ret) { \LOG(ERROR) << "[InferServer] Call [" << #func << "] failed, ret = " << _ret << ". " << msg; \} \} while (0)#define CALL_ACL_FUNC(func, msg) _CALLFUNC(func, acllite::ACLLITE_OK, msg)inline BufSurfaceMemType CastMemoryType(acllite::MemoryType type) noexcept{switch (type) {
#define RETURN_MEMORY_TYPE(type) \case acllite::MemoryType::type: \return BUF_##type;RETURN_MEMORY_TYPE(MEMORY_HOST)RETURN_MEMORY_TYPE(MEMORY_DEVICE)RETURN_MEMORY_TYPE(MEMORY_DVPP)RETURN_MEMORY_TYPE(MEMORY_NORMAL)
#undef RETURN_MEMORY_TYPEdefault:LOG(ERROR) << "[InferServer] CastMemoryType(): Unsupported memory type";return BUF_MEMORY_HOST;}
}inline acllite::MemoryType CastMemoryType(BufSurfaceMemType type) noexcept{switch (type) {
#define RETURN_MEMORY_TYPE(type) \case BUF_##type: \return acllite::MemoryType::type;RETURN_MEMORY_TYPE(MEMORY_HOST)RETURN_MEMORY_TYPE(MEMORY_DEVICE)RETURN_MEMORY_TYPE(MEMORY_DVPP)RETURN_MEMORY_TYPE(MEMORY_NORMAL)
#undef RETURN_MEMORY_TYPEdefault:LOG(ERROR) << "[InferServer] CastMemoryType(): Unsupported memory type";return acllite::MemoryType::MEMORY_HOST;}
}inline BufSurfaceColorFormat CastColorFmt(acllite::ImageFormat format) {static std::map<acllite::ImageFormat, BufSurfaceColorFormat> color_map{{ acllite::ImageFormat::YUV_SP_420, BUF_COLOR_FORMAT_NV12 },{ acllite::ImageFormat::YVU_SP_420, BUF_COLOR_FORMAT_NV21 },{ acllite::ImageFormat::RGB_888, BUF_COLOR_FORMAT_RGB },{ acllite::ImageFormat::BGR_888, BUF_COLOR_FORMAT_BGR },};return color_map[format];
}inline acllite::ImageFormat CastColorFmt(BufSurfaceColorFormat format) {static std::map<BufSurfaceColorFormat, acllite::ImageFormat> color_map{{ BUF_COLOR_FORMAT_NV12, acllite::ImageFormat::YUV_SP_420 },{ BUF_COLOR_FORMAT_NV21, acllite::ImageFormat::YVU_SP_420 },{ BUF_COLOR_FORMAT_RGB, acllite::ImageFormat::RGB_888 },{ BUF_COLOR_FORMAT_BGR, acllite::ImageFormat::BGR_888 },};return color_map[format];
}#endif // COMMON_UTILS_HPP_
上面这个文件内容目前是改成了华为acl相关的，现在把整个成功移植到英伟达的Jetson，那么这个文件的内容也要修改，另外，整个工程中所有用到了ACL_SAFECALL和CALL_ACL_FUNC的地方，不仅要把ACL_SAFECALL和CALL_ACL_FUNC这两个名字改掉，还要把用到这两个名字的硬件相关接口全都修改掉，比如
比如这张截图中，所有的这些acllite::AclLiteMalloc acllite::AclLiteFree acllite::AclLiteMemcpy这些接口都要相应的改成英伟达Jetson平台的接口。
2.1 infer_server/include/common/utils.hpp文件内容修改
/************************************************************************** Copyright (C) [2022] by Cambricon, Inc. All rights reserved** Licensed under the Apache License, Version 2.0 (the "License");* you may not use this file except in compliance with the License.* You may obtain a copy of the License at** http://www.apache.org/licenses/LICENSE-2.0** The above copyright notice and this permission notice shall be included in* all copies or substantial portions of the Software.* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS* OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN* THE SOFTWARE.*************************************************************************/#ifndef COMMON_UTILS_HPP_
#define COMMON_UTILS_HPP_#include <string>#include <glog/logging.h>#include "buf_surface.h"#include "cuda_runtime.h"
#include "nvcv/ImageFormat.h"#define _SAFECALL(func, expected, msg, ret_val) \do { \int _ret = (func); \if ((expected) != _ret) { \LOG(ERROR) << "[InferServer] Call [" << #func << "] failed, ret = " << _ret << ". " << msg; \return (ret_val); \} \} while (0)#define CUDA_SAFECALL(func, msg, ret_val) _SAFECALL(func, cudaSuccess, msg, ret_val)#define _CALLFUNC(func, expected, msg) \do { \int _ret = (func); \if ((expected) != _ret) { \LOG(ERROR) << "[InferServer] Call [" << #func << "] failed, ret = " << _ret << ". " << msg; \} \} while (0)#define CALL_CUDA_FUNC(func, msg) _CALLFUNC(func, cudaSuccess, msg)inline BufSurfaceMemType CastMemoryType(cudaMemoryType type) noexcept{switch (type) {case cudaMemoryTypeUnregistered:return BUF_MEMORY_UNREGISTERED;case cudaMemoryTypeHost:return BUF_MEMORY_HOST;case cudaMemoryTypeDevice:return BUF_MEMORY_DEVICE;case cudaMemoryTypeManaged:return BUF_MEMORY_MANAGED;default:LOG(ERROR) << "[InferServer] CastMemoryType(): Unsupported memory type";return BUF_MEMORY_HOST;}
}inline cudaMemoryType CastMemoryType(BufSurfaceMemType type) noexcept{switch (type) {case BUF_MEMORY_UNREGISTERED:return cudaMemoryTypeUnregistered;case BUF_MEMORY_HOST:return cudaMemoryTypeHost;case BUF_MEMORY_DEVICE:return cudaMemoryTypeDevice;case BUF_MEMORY_MANAGED:return cudaMemoryTypeManaged;default:LOG(ERROR) << "[InferServer] CastMemoryType(): Unsupported memory type";return cudaMemoryTypeHost;}
}inline BufSurfaceColorFormat CastColorFmt(NVCVImageFormat format) {static std::map<NVCVImageFormat, BufSurfaceColorFormat> color_map{{ NVCV_IMAGE_FORMAT_NV12, BUF_COLOR_FORMAT_NV12 },{ NVCV_IMAGE_FORMAT_NV21, BUF_COLOR_FORMAT_NV21 },{ NVCV_IMAGE_FORMAT_RGB8, BUF_COLOR_FORMAT_RGB },{ NVCV_IMAGE_FORMAT_BGR8, BUF_COLOR_FORMAT_BGR },};return color_map[format];
}inline NVCVImageFormat CastColorFmt(BufSurfaceColorFormat format) {static std::map<BufSurfaceColorFormat, NVCVImageFormat> color_map{{ BUF_COLOR_FORMAT_NV12, NVCV_IMAGE_FORMAT_NV12 },{ BUF_COLOR_FORMAT_NV21, NVCV_IMAGE_FORMAT_NV21 },{ BUF_COLOR_FORMAT_RGB, NVCV_IMAGE_FORMAT_RGB8 },{ BUF_COLOR_FORMAT_BGR, NVCV_IMAGE_FORMAT_BGR8 },};return color_map[format];
}#endif // COMMON_UTILS_HPP_
2.2 cuda的四种内存
在CUDA编程中，内存类型是指定数据应该存储在哪种类型的内存中的关键概念。CUDA支持多种内存类型，每种类型都有其特定的用途和性能特点。以下是你提到的几种CUDA内存类型的解释和区别：
cudaMemoryTypeUnregistered：
- 这个枚举值表示内存类型未注册。在CUDA中，通常不会直接使用这个值，因为它表示内存没有被明确指定为主机或设备内存。
cudaMemoryTypeHost：
- 这表示内存是分配在主机（CPU）上的。主机内存可以被CPU直接访问，但GPU访问它通常较慢，因为需要通过PCIe总线进行数据传输。这种内存类型适用于需要CPU频繁访问的数据，或者在CPU和GPU之间需要频繁传输数据的场景。
cudaMemoryTypeDevice：
- 这表示内存是分配在设备（GPU）上的。设备内存只能被GPU直接访问，CPU访问它需要通过CUDA的内存复制操作。这种内存类型适用于GPU计算密集型任务，因为数据已经在GPU上，可以减少数据传输的开销。
cudaMemoryTypeManaged：
- 这表示内存是“统一内存”，即由CUDA统一管理的内存。在这种内存类型下，数据可以同时被CPU和GPU访问，而不需要显式的数据复制操作。CUDA运行时会自动处理数据在主机和设备之间的迁移。这种内存类型简化了内存管理，但可能会引入额外的性能开销，因为运行时需要决定何时以及如何迁移数据。
区别总结：
cudaMemoryTypeHost和cudaMemoryTypeDevice提供了明确的内存位置，分别对应CPU和GPU，适用于对性能有明确要求的场景。cudaMemoryTypeManaged提供了更简单的编程模型，但可能会牺牲一些性能，因为数据迁移是自动和隐式的。cudaMemoryTypeUnregistered通常不用于实际编程，它更多是一个占位符，表示内存类型尚未确定。
2.3 infer_server/src/core/device.cpp修改
/*************************************************************************
* Copyright (C) [2020] by Cambricon, Inc. All rights reserved
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
* OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*************************************************************************/#include <glog/logging.h>#include "nvis/infer_server.h"namespace infer_server {cudaMemcpyKind GetMemcpyKind(BufSurfaceMemType src_mem_type, BufSurfaceMemType dst_mem_type) {// 确保未注册内存不会被使用assert(src_mem_type != BUF_MEMORY_UNREGISTERED);assert(dst_mem_type != BUF_MEMORY_UNREGISTERED);// 根据源和目标内存类型确定 cudaMemcpyKindif (src_mem_type == BUF_MEMORY_HOST && dst_mem_type == BUF_MEMORY_HOST) {return cudaMemcpyHostToHost;}else if (src_mem_type == BUF_MEMORY_HOST && dst_mem_type == BUF_MEMORY_DEVICE) {return cudaMemcpyHostToDevice;}else if (src_mem_type == BUF_MEMORY_DEVICE && dst_mem_type == BUF_MEMORY_HOST) {return cudaMemcpyDeviceToHost;}else if (src_mem_type == BUF_MEMORY_DEVICE && dst_mem_type == BUF_MEMORY_DEVICE) {return cudaMemcpyDeviceToDevice;}else if (src_mem_type == BUF_MEMORY_MANAGED || dst_mem_type == BUF_MEMORY_MANAGED) {// 管理内存可以视为主机或设备内存return cudaMemcpyDefault;}// 默认情况下返回 cudaMemcpyDefaultreturn cudaMemcpyDefault;}bool SetCurrentDevice(int device_id) noexcept{CUDA_SAFECALL(cudaSetDevice(device_id), "Set device failed", false);VLOG(3) << "[InferServer] SetCurrentDevice(): Set device [" << device_id << "] for this thread";return true;}uint32_t TotalDeviceCount() noexcept{uint32_t dev_cnt;CUDA_SAFECALL(cudaGetDeviceCount(dev_cnt), "Set device failed", 0);return dev_cnt;}bool CheckDevice(int device_id) noexcept{uint32_t dev_cnt;CUDA_SAFECALL(cudaGetDeviceCountdev_cnt), "Check device failed", false);return device_id < static_cast<int>(dev_cnt) && device_id >= 0;}void* MallocDeviceMem(size_t size) noexcept{void *device_ptr{};CUDA_SAFECALL(cudaMallocManaged(&device_ptr, size), "Malloc device memory failed", nullptr);return device_ptr;}int FreeDeviceMem(void *p) noexcept{CUDA_SAFECALL(cudaFree(p), "Free device memory failed", -1);return 0;}void* AllocHostMem(size_t size) noexcept{void *host_ptr{};CUDA_SAFECALL(cudaMallocHost(&host_ptr, size), "Malloc host memory failed", nullptr);return host_ptr;}int FreeHostMem(void *p) noexcept{CUDA_SAFECALL(cudaFreeHost(p), "Free host memory failed", -1);}int MemcpyHD(void* dst, BufSurfaceMemType dst_mem_type, void* src, BufSurfaceMemType src_mem_type, size_t size) noexcept{cudaMemcpyKind cpy_type = GetMemcpyKind(src_mem_type, dst_mem_type);CUDA_SAFECALL(cudaMemcpy(dst, src, size, cpy_type), "Memcpy HD failed", -1);return 0;}bool IsItegratedGPU(int device_id) {static int s_integrated = [device_id]() {cudaDeviceProp prop;CUDA_SAFECALL(cudaGetDeviceProperties(&prop, device_id));};return s_integrated == 1;}int GetCurrentDevice(int& device_id) noexcept{CUDA_SAFECALL(cudaGetDevice(&device_id));return 0;}} // namespace infer_server
3 图像缩放、裁剪、色域转换等代码编写----利用CV-CUDA
3.1 nvstream中图像处理代码流程框架
和前面一样，先大体看一下nvstream中图像处理的代码流程框架。
3.1.1 类的层次关系
class TransformServiceclass ITransformer 只是个基类，被用来继承的，class TransformerAcl : public ITransformer 然后就是硬件处理的了AclLiteImageProc现在应该是直接把TransformerAcl 和 AclLiteImageProc合成一个英伟达的类。3.1.2 各个类的初始化函数调用层次关系
在视频解码类的create函数里面，或者在算法的Preproc都会有这样一行
if (TransformCreate(&transformer_, &config) != 0)这个transformer_ 是视频解码类或者算法预处理类的一个成员。这个create函数是这样的，int TransformCreate(void **transformer, TransformConfigParams *params) {return infer_server::TransformService::Instance().Create(transformer, params);}然后TransformService类的create函数里面有 ITransformer *transformer_ = CreateTransformer(); CreateTransformer里面就是new TransformerAcl也就是具体的硬件处理类了 transformer_->Create(params)*transformer = transformer_;回传给解码的那个类或者算法预处理类，3.1.3 各个类的transform函数调用层次关系
具体硬件解码处理类的OnFrame函数里面会有一个
Transform(transformer_, &transform_src, surf, &trans_params)应该是调用了这个int Transform(void *transformer, BufSurface *src, BufSurface *dst, TransformParams *transform_params) {return infer_server::TransformService::Instance().Transform(transformer, src, dst, transform_params);}然后到了这里int Transform(void *transformer, BufSurface *src, BufSurface *dst, TransformParams *transform_params) {if (!dst || !src || !transform_params) {LOG(ERROR) << "[InferServer] [TransformService] Transform(): src, dst BufSurface or parameters pointer is invalid";return -1;}ITransformer *transformer_ = static_cast<ITransformer *>(transformer); 那个具体的硬件处理类就是继承的ITransformer return transformer_->Transform(src, dst, transform_params);这就到了具体硬件transform类的函数了，}3.2 编写图像缩放、裁剪、色域转换等代码
3.2.1 infer_server/src/nv/transform_impl_nv.hpp
#ifndef TRANSFORM_IMPL_NV_HPP_
#define TRANSFORM_IMPL_NV_HPP_#include <algorithm>
#include <cstring> // for memset
#include <atomic>#include "../transform_impl.hpp"
#include "transform.h"namespace infer_server {class TransformerNV : public ITransformer {public:TransformerNV() {}~TransformerNV() = default;int Create(TransformConfigParams *params) override;int Destroy() override;int Transform(BufSurface *src, BufSurface *dst, TransformParams *transform_params) override;private:int DoNVTransform(BufSurface *src, BufSurface *dst, TransformParams *transform_params);int NVResize(BufSurface *src, BufSurface *dst, TransformParams *transform_params);int NVCrop(BufSurface *src, BufSurface *dst, TransformParams *transform_params);int NVCropResize(BufSurface *src, BufSurface *dst, TransformParams *transform_params);int NVCropResizePaste(BufSurface *src, BufSurface *dst, TransformParams *transform_params);int NVConvertFormat(BufSurface *src, BufSurface *dst, TransformParams *transform_params);private:TransformConfigParams create_params_;cudaStream_t* cu_stream_{nullptr};std::shared_ptr<nvcv::ITensor> crop_tensor_;std::shared_ptr<nvcv::ITensor> resized_tensor_;std::shared_ptr<nvcv::ITensor> cvtcolor_tensor_;std::shared_ptr<nvcv::ITensor> copymakeborder_tensor_;std::shared_ptr<cvcuda::CustomCrop> crop_op_; std::shared_ptr<cvcuda::Resize> resize_op_;std::shared_ptr<cvcuda::CvtColor> cvtcolor_op_;std::shared_ptr<cvcuda::CopyMakeBorder> copymakeborder_op_;std::atomic<bool> created_{};};} // namespace #endif // TRANSFORM_IMPL_MLU370_HPP_
3.2.2 infer_server/src/nv/transform_impl_nv.cpp
#include "transform_impl_nv.hpp"#include <algorithm>
#include <atomic>
#include <cstring> // for memset
#include <map>
#include <memory>
#include <string>
#include <vector>#include "glog/logging.h"namespace infer_server {int TransformerNV::Create(TransformConfigParams *params) {create_params_ = *params;if(nullptr == cu_stream_){cuCreateStream(&cu_stream_, create_params_.device_id);}if (!crop_op_){crop_op_ = std::make_shared<cvcuda::CustomCrop>();}if (!cvtcolor_op_){cvtcolor_op_ = std::make_shared<cvcuda::CvtColor>();}if (!resize_op_){resize_op_ = std::make_shared<cvcuda::Resize>();}if (!copymakeborder_op_){copymakeborder_op_ = std::make_shared<cvcuda::CopyMakeBorder>();}created_ = true;return 0;}int TransformerNV::Destroy() {if (!created_) {LOG(WARNING) << "[InferServer] [TransformerNV] Destroy(): Transformer is not created";return 0;}if (cu_stream_ != nullptr) {cuDestroyStream(cu_stream_);cu_stream_ = nullptr;}cvtcolor_op_.reset();resize_op_.reset();crop_op_.reset();copymakeborder_op_.reset();return 0;}int TransformerNV::Transform(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {if (!created_) {LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): transformer is not created";return -1;}if (src->num_filled > dst->batch_size) {LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): The number of inputs exceeds batch size: "<< src->num_filled << " v.s. " << dst->batch_size;return -1;}if (src->device_id != dst->device_id || src->device_id != create_params_.device_id) {LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): The device id of src, dst and transformer is not the same: src device: " << src->device_id<< " , dst device: " << dst->device_id<< " , transformer device: " << create_params_.device_id;return -1;}if (src->mem_type != BUF_MEMORY_DVPP) {LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): The src and dst mem_type must be BUF_MEMORY_DVPP";return -1;}if (src->surface_list[0].data_size == 0) {LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): Input data size is 0";return -1;}return DoNVTransform(src, dst, transform_params);}int TransformerNV::DoNVTransform(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {switch (transform_params->transform_flag) {case TRANSFORM_RESIZE_SRC:return NVResize(src, dst, transform_params);break;case TRANSFORM_CROP_SRC:return NVCrop(src, dst, transform_params);break;case TRANSFORM_CROP_RESIZE_SRC:return NVCropResize(src, dst, transform_params);break;case TRANSFORM_CROP_RESIZE_PASTE_SRC:return NVCropResizePaste(src, dst, transform_params);break;default:LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): Transform flag not supported currently, flag = " << transform_params->transform_flag;return -1;}}int TransformerNV::NVResize(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {if (src->batch_size != 1 || dst->batch_size != 1) {LOG(ERROR) << "[InferServer] [TransformerNV] Transform(): NVResize now only support src/dst one image, src batch size = " << src->batch_size <<" , dst batch size = " << dst->batch_size;return -1;}auto& src_surf = src->surface_list[0];nvcv::Tensor::Requirements in_reqs = nvcv::Tensor::CalcRequirements(1, {src_surf.width_stride, src_surf.height}, nvcv::FMT_BGR8);nvcv::TensorDataStridedCuda::Buffer in_buf;std::copy(in_reqs.strides, in_reqs.strides + NVCV_TENSOR_MAX_RANK, in_buf.strides);in_buf.basePtr = reinterpret_cast<NVCVByte *>(src->data_ptr);nvcv::TensorDataStridedCuda in_data(nvcv::TensorShape{in_reqs.shape, in_reqs.rank, in_reqs.layout}, nvcv::DataType{in_reqs.dtype}, in_buf);nvcv::TensorWrapData in_tensor(in_data);if (!resized_tensor_){resized_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, {dst.width_stride, src_surf.height}, nvcv::FMT_BGR8));}(*resize_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), in_tensor, *resized_tensor_, NVCV_INTERP_LINEAR);auto& dst_surf = dst->surface_list[0];auto out_data = resized_tensor_->exportData<nvcv::TensorDataStridedCuda>();cudaMemcpyAsync(dst_surf.data_ptr, (const unsigned char *)out_data->basePtr(), dst_surf.data_size, cudaMemcpyDeviceToHost);cuStreamSynchronize(cu_stream_);resized_tensor_.reset();return 0;}int TransformerNV::NVCrop(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {auto& src_surf = src->surface_list[0];auto& dst_surf = dst->surface_list[0];nvcv::Tensor::Requirements in_reqs = nvcv::Tensor::CalcRequirements(1, {src_surf.width_stride, src_surf.height}, nvcv::FMT_BGR8);nvcv::TensorDataStridedCuda::Buffer in_buf;std::copy(in_reqs.strides, in_reqs.strides + NVCV_TENSOR_MAX_RANK, in_buf.strides);in_buf.basePtr = reinterpret_cast<NVCVByte *>(src->data_ptr);nvcv::TensorDataStridedCuda in_data(nvcv::TensorShape{in_reqs.shape, in_reqs.rank, in_reqs.layout}, nvcv::DataType{in_reqs.dtype}, in_buf);nvcv::TensorWrapData in_tensor(in_data);if (!crop_tensor_){crop_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, {dst_surf.width, dst_surf.height}, nvcv::FMT_BGR8));}TransformRect rect = transform_params->src_rect[0];rect.left = rect.left >= src_surf.width ? 0 : rect.left;rect.top = rect.top >= src_surf.height ? 0 : rect.top;rect.width = rect.width <= 0 ? (src_surf.width - rect.left) : rect.width;rect.height = rect.height <= 0 ? (src_surf.height - rect.top) : rect.height;NVCVRectI crpRect = { rect.left, rect.top, rect.left + rect.width - 1, rect.top + rect.height - 1 };(*crop_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), in_tensor, *crop_tensor_, crpRect);auto out_data = crop_tensor_->exportData<nvcv::TensorDataStridedCuda>();cudaMemcpyAsync(dst_surf.data_ptr, (const unsigned char *)out_data->basePtr(), dst_surf.data_size, cudaMemcpyDeviceToHost);cuStreamSynchronize(cu_stream_);crop_tensor_.reset();return 0;}int TransformerNV::NVCropResize(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {auto& src_surf = src->surface_list[0];auto& dst_surf = dst->surface_list[0];nvcv::Tensor::Requirements in_reqs = nvcv::Tensor::CalcRequirements(1, {src_surf.width_stride, src_surf.height}, nvcv::FMT_BGR8);nvcv::TensorDataStridedCuda::Buffer in_buf;std::copy(in_reqs.strides, in_reqs.strides + NVCV_TENSOR_MAX_RANK, in_buf.strides);in_buf.basePtr = reinterpret_cast<NVCVByte *>(src->data_ptr);nvcv::TensorDataStridedCuda in_data(nvcv::TensorShape{in_reqs.shape, in_reqs.rank, in_reqs.layout}, nvcv::DataType{in_reqs.dtype}, in_buf);nvcv::TensorWrapData in_tensor(in_data);TransformRect rect = transform_params->src_rect[0];rect.left = rect.left >= src_surf.width ? 0 : rect.left;rect.top = rect.top >= src_surf.height ? 0 : rect.top;rect.width = rect.width <= 0 ? (src_surf.width - rect.left) : rect.width;rect.height = rect.height <= 0 ? (src_surf.height - rect.top) : rect.height;NVCVRectI crpRect = { rect.left, rect.top, rect.left + rect.width - 1, rect.top + rect.height - 1 };if (!crop_tensor_){crop_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, {rect.width, rect.height}, nvcv::FMT_BGR8));}(*crop_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), in_tensor, *crop_tensor_, crpRect);if (!resized_tensor_){resized_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, { dst_surf.width, dst_surf.height }, nvcv::FMT_BGR8));}(*resize_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), *crop_tensor_, *resized_tensor_, NVCV_INTERP_LINEAR);auto out_data = resized_tensor_->exportData<nvcv::TensorDataStridedCuda>();cudaMemcpyAsync(dst_surf.data_ptr, (const unsigned char *)out_data->basePtr(), dst_surf.data_size, cudaMemcpyDeviceToHost);cuStreamSynchronize(cu_stream_);crop_tensor_.reset();resized_tensor_.reset();return 0;}int TransformerNV::NVCropResizePaste(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {auto& src_surf = src->surface_list[0];auto& dst_surf = dst->surface_list[0];nvcv::Tensor::Requirements in_reqs = nvcv::Tensor::CalcRequirements(1, {src_surf.width_stride, src_surf.height}, nvcv::FMT_BGR8);nvcv::TensorDataStridedCuda::Buffer in_buf;std::copy(in_reqs.strides, in_reqs.strides + NVCV_TENSOR_MAX_RANK, in_buf.strides);in_buf.basePtr = reinterpret_cast<NVCVByte *>(src->data_ptr);nvcv::TensorDataStridedCuda in_data(nvcv::TensorShape{in_reqs.shape, in_reqs.rank, in_reqs.layout}, nvcv::DataType{in_reqs.dtype}, in_buf);nvcv::TensorWrapData in_tensor(in_data);TransformRect rect = transform_params->src_rect[0];rect.left = rect.left >= src_surf.width ? 0 : rect.left;rect.top = rect.top >= src_surf.height ? 0 : rect.top;rect.width = rect.width <= 0 ? (src_surf.width - rect.left) : rect.width;rect.height = rect.height <= 0 ? (src_surf.height - rect.top) : rect.height;NVCVRectI crop_src_rect = { rect.left, rect.top, rect.left + rect.width - 1, rect.top + rect.height - 1 };TransformRect dst_rect = transform_params->dst_rect[0];dst_rect.left = dst_rect.left >= src_surf.width ? 0 : dst_rect.left;dst_rect.top = dst_rect.top >= src_surf.height ? 0 : dst_rect.top;dst_rect.width = dst_rect.width <= 0 ? (src_surf.width - dst_rect.left) : dst_rect.width;dst_rect.height = dst_rect.height <= 0 ? (src_surf.height - dst_rect.top) : dst_rect.height;NVCVRectI paste_dst_rect(dst_rect.left, dst_rect.top, dst_rect.left + dst_rect.width - 1, dst_rect.top + dst_rect.height - 1);if (!crop_tensor_){crop_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, {rect.width, rect.height}, nvcv::FMT_BGR8));}(*crop_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), in_tensor, *crop_tensor_, crop_src_rect);if (!resized_tensor_){resized_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, { dst_rect.width, dst_rect.height }, nvcv::FMT_BGR8));}(*resize_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), *crop_tensor_, *resized_tensor_, NVCV_INTERP_LINEAR);if (!copymakeborder_tensor_){copymakeborder_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, { dst_surf.width, dst_surf.height }, nvcv::FMT_BGR8));}(*copymakeborder_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), *resized_tensor_, *copymakeborder_tensor_, dst_rect.top, dst_rect.left, NVCV_BORDER_CONSTANT, 0);auto out_data = copymakeborder_tensor_->exportData<nvcv::TensorDataStridedCuda>();cudaMemcpyAsync(dst_surf.data_ptr, (const unsigned char *)out_data->basePtr(), dst_surf.data_size, cudaMemcpyDeviceToHost);cuStreamSynchronize(cu_stream_);crop_tensor_.reset();resized_tensor_.reset();copymakeborder_tensor_.reset();return 0;}int TransformerNV::NVConvertFormat(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {auto& src_surf = src->surface_list[0];nvcv::Tensor::Requirements in_reqs = nvcv::Tensor::CalcRequirements(1, {src_surf.width_stride, src_surf.height}, nvcv::FMT_NV12);nvcv::TensorDataStridedCuda::Buffer in_buf;std::copy(in_reqs.strides, in_reqs.strides + NVCV_TENSOR_MAX_RANK, in_buf.strides);in_buf.basePtr = reinterpret_cast<NVCVByte *>(src->data_ptr);nvcv::TensorDataStridedCuda in_data(nvcv::TensorShape{in_reqs.shape, in_reqs.rank, in_reqs.layout}, nvcv::DataType{in_reqs.dtype}, in_buf);nvcv::TensorWrapData in_tensor(in_data);if (!cvtcolor_tensor_){cvtcolor_tensor_ = std::shared_ptr<nvcv::Tensor>(new nvcv::Tensor(1, {src_surf.width_stride, src_surf.height}, nvcv::FMT_BGR8));}(*cvtcolor_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), in_tensor, *cvtcolor_tensor_, NVCV_COLOR_YUV2BGR_NV12);auto& dst_surf = dst->surface_list[0];auto out_data = resized_tensor_->exportData<nvcv::TensorDataStridedCuda>();cudaMemcpyAsync(dst_surf.data_ptr, (const unsigned char *)out_data->basePtr(), dst_surf.data_size, cudaMemcpyDeviceToHost);cuStreamSynchronize(cu_stream_);cvtcolor_tensor_.reset();return 0;}} // namespace
4 算法推理相关代码修改
4.1 ./infer_server/src/model/model.h
/************************************************************************** Copyright (C) [2020] by Cambricon, Inc. All rights reserved** Licensed under the Apache License, Version 2.0 (the "License");* you may not use this file except in compliance with the License.* You may obtain a copy of the License at** http://www.apache.org/licenses/LICENSE-2.0** The above copyright notice and this permission notice shall be included in* all copies or substantial portions of the Software.* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS* OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN* THE SOFTWARE.*************************************************************************/#ifndef INFER_SERVER_MODEL_H_
#define INFER_SERVER_MODEL_H_#include <glog/logging.h>
#include <algorithm>
#include <map>
#include <memory>
#include <mutex>
#include <string>
#include <unordered_map>
#include <vector>#include "nvis/infer_server.h"
#include "nvis/processor.h"
#include "nvis/shape.h"namespace trt {
#include "trteng_exp/export_funtions.h"
}namespace infer_server {using TEngine = std::unique_ptr<trt::trtnet_t>;class Model;class ModelRunner {public:explicit ModelRunner(int device_id) : device_id_(device_id) {}ModelRunner(const ModelRunner& other) = delete;ModelRunner& operator=(const ModelRunner& other) = delete;ModelRunner(ModelRunner&& other) = default;ModelRunner& operator=(ModelRunner&& other) = default;~ModelRunner() = default;bool Init(TEngine engine) noexcept;Status Run(ModelIO* input, ModelIO* output) noexcept; // NOLINTvoid SetInputNum(uint32_t i_num) noexcept{ input_num_ = i_num; }void SetOutputNum(uint32_t o_num) noexcept{ output_num_ = o_num; }void SetInputShapes(std::vector<Shape> const& i_shapes) noexcept{ i_shapes_ = i_shapes; }void SetOutputShapes(std::vector<Shape> const& o_shapes) noexcept{ o_shapes_ = o_shapes; }void SetInputLayouts(std::vector<DataLayout> const& i_layouts) noexcept{ i_layouts_ = i_layouts; }void SetOutputLayouts(std::vector<DataLayout> const& o_layouts) noexcept{ o_layouts_ = o_layouts; }private:TEngine engine_{};uint32_t input_num_{};uint32_t output_num_{};std::vector<Shape> i_shapes_;std::vector<Shape> o_shapes_;std::vector<DataLayout> i_layouts_;std::vector<DataLayout> o_layouts_;int device_id_{};}; // class RuntimeContextclass Model : public ModelInfo {public:Model() = default;bool Init(const std::string& model_path) noexcept;~Model();bool HasInit() const noexcept{ return has_init_; }const Shape& InputShape(int index) const noexcept override{CHECK(index < i_num_ || index >= 0) << "[InferServer] [Model] Input shape index overflow";return input_shapes_[index];}const Shape& OutputShape(int index) const noexcept override{CHECK(index < o_num_ || index >= 0) << "[InferServer] [Model] Output shape index overflow";return output_shapes_[index];}const DataLayout& InputLayout(int index) const noexcept override{CHECK(index < i_num_ || index >= 0) << "[InferServer] [Model] Input shape index overflow";return i_layouts_[index];}const DataLayout& OutputLayout(int index) const noexcept override{CHECK(index < o_num_ || index >= 0) << "[InferServer] [Model] Input shape index overflow";return o_layouts_[index];}uint32_t InputNum() const noexcept override{ return i_num_; }uint32_t OutputNum() const noexcept override{ return o_num_; }uint32_t BatchSize() const noexcept override{ return model_batch_size_; }bool FixedOutputShape() noexcept override{ return FixedShape(output_shapes_); }std::shared_ptr<ModelRunner> GetRunner(int device_id) noexcept{trt::ErrInfo ei{};trt::trtnet_t* net = trt::load_net_from_file(model_file_.data(), &ei);CHECK(!net) << "trt::load_net_from_file failed: model file: " << te2fullpath << ", error: " << ei.errmsg;TEngine engine = TEngine(net, [](trt::trtnet_t* n) { trt::release_net(n); });auto runner = std::make_shared<ModelRunner>(device_id);runner->SetInputNum(i_num_);runner->SetOutputNum(o_num_);runner->SetInputShapes(input_shapes_);runner->SetOutputShapes(output_shapes_);runner->SetInputLayouts(i_layouts_);runner->SetOutputLayouts(o_layouts_); if (!runner->Init(std::move(engine))) return nullptr;return runner;}std::string GetKey() const noexcept override{ return model_file_; }private:bool GetModelInfo(trt::trtnet_t* net) noexcept;bool FixedShape(const std::vector<Shape>& shapes) noexcept{for (auto &shape : shapes) {auto vectorized_shape = shape.Vectorize();if (!std::all_of(vectorized_shape.begin(), vectorized_shape.end(), [](int64_t v) { return v > 0; })) {return false;}}return !shapes.empty();}Model(const Model&) = delete;Model& operator=(const Model&) = delete;private:std::string model_file_;std::vector<DataLayout> i_layouts_, o_layouts_;std::vector<Shape> input_shapes_, output_shapes_;int i_num_{}, o_num_{};uint32_t model_batch_size_{ 1 };bool has_init_{ false };}; // class Model// use environment CNIS_MODEL_CACHE_LIMIT to control cache limitclass ModelManager {public:static ModelManager* Instance() noexcept;void SetModelDir(const std::string& model_dir) noexcept{ model_dir_ = model_dir; }ModelPtr Load(const std::string& model_file) noexcept;ModelPtr Load(void* mem_ptr, size_t size) noexcept;bool Unload(ModelPtr model) noexcept;void ClearCache() noexcept;int CacheSize() noexcept;std::shared_ptr<Model> GetModel(const std::string& name) noexcept;private:std::string DownloadModel(const std::string& url) noexcept;void CheckAndCleanCache() noexcept;std::string model_dir_{ "." };static std::unordered_map<std::string, std::shared_ptr<Model>> model_cache_;static std::mutex model_cache_mutex_;}; // class ModelManager} // namespace infer_server#endif // INFER_SERVER_MODEL_H_
4.2 ./infer_server/src/model/model.cpp
/************************************************************************** Copyright (C) [2020] by Cambricon, Inc. All rights reserved** Licensed under the Apache License, Version 2.0 (the "License");* you may not use this file except in compliance with the License.* You may obtain a copy of the License at** http://www.apache.org/licenses/LICENSE-2.0** The above copyright notice and this permission notice shall be included in* all copies or substantial portions of the Software.* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS* OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN* THE SOFTWARE.*************************************************************************/#include "model.h"#include <glog/logging.h>
#include <algorithm>
#include <memory>
#include <string>
#include <utility>
#include <vector>#include "core/data_type.h"
#include "common/utils.hpp"using std::string;
using std::vector;namespace infer_server {bool ModelRunner::Init(TEngine engine) noexcept{if (engine == nullptr) return false;engine_ = std::move(engine);return true;}Status ModelRunner::Run(ModelIO* in, ModelIO* out) noexcept{ // NOLINTauto& input = in->surfs;auto& output = out->surfs;CHECK_EQ(input_num_, input.size()) << "[InferServer] [ModelRunner] Input number is mismatched";VLOG(5) << "[InferServer] [ModelRunner] Process inference once, input num: " << input_num_ << " output num: "<< output_num_;std::vector<trt::NetInoutLayerData> inputData(input_num_);std::vector<trt::NetInoutLayerData> outputData(output_num_);for (uint32_t i_idx = 0; i_idx < input_num_; ++i_idx) {CHECK_EQ(input_num_, input.size()) << "[InferServer] [ModelRunner] Input number is mismatched";inputData[i].data = input[i_idx]->GetData(0);inputData[i].size = input[i_idx]->GetSize(0);inputData[i].layer_idx = i_idx;}for (uint32_t o_idx = 0; o_idx < output_num_; ++o_idx) {CHECK_EQ(output_num_, output.size()) << "[InferServer] [ModelRunner] Output number is mismatched";outputData[i].data = input[o_idx]->GetData(0);outputData[i].size = input[o_idx]->GetSize(0);outputData[i].layer_idx = o_idx;}uint32_t batchsize = input[0]->GetNumFilled();trt::ErrInfo ei{};_SAFECALL(trt::net_do_inference(engine_.get(), batchsize, inputData.data(), inputData.size(), outputData.data(), outputData.size(), &ei), "[InferServer] [ModelRunner] Infer failed.", Status::ERROR_BACKEND);return Status::SUCCESS;}bool Model::Init(const string& model_file) noexcept{model_file_ = model_file;trt::ErrInfo ei{};trt::trtnet_t* net = trt::load_net_from_file(model_file_.data(), &ei);CHECK(!net) << "trt::load_net_from_file failed: model file: " << te2fullpath << ", error: " << ei.errmsg;has_init_ = GetModelInfo(model);VLOG(1) << "[InferServer] [Model] (success) Load model from file: " << model_file_;trt::release_net(net);return has_init_;}bool Model::GetModelInfo(trt::trtnet_t* net) noexcept{VLOG(1) << "[InferServer] [Model] (success) Load model from graph file: " << model_file_;int model_batch_size_ = trt::net_max_batch_size(net);// get IO messages// get io number and data sizei_num_ = trt::net_num_inputs(net);o_num_ = trt::net_num_outputs(net);// get input infofor (int i = 0; i < ninp; i++) {trt::LayerDims ldim{};trt::net_input_layer_dims(net, i, &ldim);input_shapes_.emplace_back(std::move(Shape({ std::max(ldim.n, model_batch_size_), ldim.c, ldim.h, ldim.w })));DataLayout layout;layout.dtype = DataType::FLOAT;layout.order = DimOrder::NCHW;i_layouts_.emplace_back(std::move(layout));}// get output infoint noup = net_num_outputs(net);runner->SetOutputNum(noup);for (int i = 0; i < noup; i++) {trt::LayerDims ldim{};trt::net_output_layer_dims(net, i, &ldim);output_shapes_.emplace_back(std::move(Shape({ std::max(ldim.n, model_batch_size_), ldim.c, ldim.h, ldim.w })));DataLayout layout;layout.dtype = DataType::FLOAT;layout.order = DimOrder::NCHW;o_layouts_.emplace_back(std::move(layout));}VLOG(1) << "[InferServer] [Model] Model Info: input number = " << i_num_ << ";\toutput number = " << o_num_;VLOG(1) << "[InferServer] [Model] batch size = " << model_batch_size_;for (int i = 0; i < i_num_; ++i) {VLOG(1) << "[InferServer] [Model] ----- input index [" << i;VLOG(1) << "[InferServer] [Model] data type " << detail::DataTypeStr(i_layouts_[i].dtype);VLOG(1) << "[InferServer] [Model] dim order " << detail::DimOrderStr(i_layouts_[i].order);VLOG(1) << "[InferServer] [Model] shape " << input_shapes_[i];}for (int i = 0; i < o_num_; ++i) {VLOG(1) << "[InferServer] [Model] ----- output index [" << i;VLOG(1) << "[InferServer] [Model] data type " << detail::DataTypeStr(o_layouts_[i].dtype);VLOG(1) << "[InferServer] [Model] dim order " << detail::DimOrderStr(o_layouts_[i].order);VLOG(1) << "[InferServer] [Model] shape " << output_shapes_[i];}return true;}Model::~Model() {VLOG(1) << "[InferServer] [Model] Unload model: " << model_file_;}
} // namespace infer_server
5 其他代码修改
其他的还有很多零碎代码，比如一些命名空间的名字，还有一些其他名称，还有很多文件里面调用的一些函数的名字，太乱了，不写到博客里面了。
参考文献：
在NVIDIA Jetson AGX Orin中使用jetson-ffmpeg调用硬件编解码加速处理-CSDN博客
NVIDIA Jetson AGX Orin源码编译安装CV-CUDA-CSDN博客
GitHub - Cambricon/CNStream: CNStream is a streaming framework for building Cambricon machine learning pipelines http://forum.cambricon.com https://gitee.com/SolutionSDK/CNStream
easydk/samples/simple_demo/common/video_decoder.cpp at master · Cambricon/easydk · GitHub
aclStream流处理多路并发Pipeline框架中 视频解码 代码调用流程整理、类的层次关系整理、回调函数赋值和调用流程整理-CSDN博客
aclStream流处理多路并发Pipeline框架中VEncode Module代码调用流程整理、类的层次关系整理、回调函数赋值和调用流程整理-CSDN博客
FFmpeg/doc/examples at master · FFmpeg/FFmpeg · GitHub
GitHub - CVCUDA/CV-CUDA: CV-CUDA™ is an open-source, GPU accelerated library for cloud-scale image processing and computer vision.
如何使用FFmpeg的解码器—FFmpeg API教程 · FFmpeg原理
C++ API — CV-CUDA Beta documentation (cvcuda.github.io)
CV-CUDA/tests/cvcuda/system at main · CVCUDA/CV-CUDA · GitHub
Resize — CV-CUDA Beta documentation
CUDA Runtime API :: CUDA Toolkit Documentation
CUDA Toolkit Documentation 12.6 Update 1
这篇关于完整指南：CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (二) 源码架构流程梳理、代码编写的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
