linux内核下，(rapidio)TSI721芯片的block DMA分析笔记

目录

1 简介

2 大致流程

3 A机器(发送端)中，内存~TSI721的流程

2.2 流程

2.3 dmesg信息

4 A机器的 TSI721 ~ B机器的 TSI721 (S-RIO NWRITE_R)；

5 B机器中的 TSI721~内存

5.1 inbound windows 和 S-RIO Address Translation

5.2 接收的程序流程

5.3 dmesg信息

6 block DMA相关的其他问题

6.1 地址对齐问题

简介

机理分析

6.2 DMA申请大内存

机理分析

解决方法

---

1 简介

        当前使用的驱动源码来自Releases · RapidIO/kernel-rapidio · GitHub，驱动源码中包含block DMA的测试应用层工具——rio_test_dma。

        TSI721在处理block DMA Descriptors时，可以将Descriptors转换成以下4中rapidio操作(参考：《Tsi721 ™ User Manual》P189)

• S-RIO NREAD 
• S-RIO maintenance read
• S-RIO NWRITE/SWRITE/NWRITE_R
• S-RIO maintenance write

        rio_test_dma工具是使用的NWRITE_R。

        本文是在使用rio_test_dma工具进行block DMA测试的驱动流程分析笔记。

2 大致流程

在执行rio_test_dma进行数据传输时，主要涉及3个过程：

• A机器(发送端)中的 内存~TSI721；
• A机器的 TSI721 ~ B机器的 TSI721 (S-RIO NWRITE_R)；
• B机器中的 TSI721~内存。

下面将对这三个过程进行分析。

3 A机器(发送端)中，内存~TSI721的流程

2.2 流程

2.3 dmesg信息

命令：# ./rio_test_dma -M 0 -D 0x0 -A 0x2000000 -S 0x200000 -T 2 -d -v

发送端 dmesg 信息(安装 tsi721_mport.ko 需要指定模块参数: “dbg_level=0xffff”)

[ 338.944178] dma dma0chan0: tsi721_alloc_chan_resources: DMAC0
[ 338.944184] dma dma0chan0: tsi721_bdma_ch_init: DMAC0
[ 338.944200] dma dma0chan0: tsi721_bdma_ch_init: DMAC0 descriptors @ 00000000d8a79ecb (phys = 0x000000037a880000)
[ 338.944233] dma dma0chan0: tsi721_bdma_ch_init: DMAC0 desc status FIFO @ 000000004ee70bd4 (phys = 0x000000037a900000) size=0x2000
[ 338.944685] dma dma0chan0: tsi721_prep_rio_sg: DMAC0 WRITE
[ 338.944689] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.944694] dma dma0chan0: tsi721_submit_sg: DMAC0 BD ring status: rdi=0 wri=0
[ 338.944697] dma dma0chan0: tsi721_submit_sg: DMAC0 sg0/128 addr: 0x410308010 len: 16368
[ 338.944700] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000d8a79ecb did=0 raddr=0x2000000
[ 338.944702] dma dma0chan0: tsi721_submit_sg: DMAC0 sg1/128 addr: 0x3ef070000 len: 16384
[ 338.944704] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16368
[ 338.944706] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 000000004a131af7 did=0 raddr=0x2003ff0
[ 338.944707] dma dma0chan0: tsi721_submit_sg: DMAC0 sg2/128 addr: 0x3a8c90000 len: 16384
[ 338.944709] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.944711] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000ceafa227 did=0 raddr=0x2007ff0
[ 338.944713] dma dma0chan0: tsi721_submit_sg: DMAC0 sg3/128 addr: 0x3b277c000 len: 16384
[ 338.944714] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
……
[ 338.945324] dma dma0chan0: tsi721_submit_sg: DMAC0 sg126/128 addr: 0x37d128000 len: 16384
[ 338.945325] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.945327] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 000000001a9b98b4 did=0 raddr=0x21fbff0
[ 338.945329] dma dma0chan0: tsi721_submit_sg: DMAC0 sg127/128 addr: 0x37da14000 len: 16
[ 338.945330] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.945332] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000f9580096 did=0 raddr=0x21ffff0
[ 338.945333] dma dma0chan0: tsi721_submit_sg: DMAC0 last desc final len: 16
[ 338.945336] dma dma0chan0: tsi721_start_dma: DMAC0 (wrc=128) 2588
[ 338.945339] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.945341] dma dma0chan0: tsi721_issue_pending: DMAC0
[ 338.946634] dma dma0chan0: tsi721_dma_tasklet: DMAC0_INT = 0x8
[ 338.946637] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.946640] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.946946] dma dma0chan0: tsi721_prep_rio_sg: DMAC0 READ
[ 338.946948] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.946953] dma dma0chan0: tsi721_submit_sg: DMAC0 BD ring status: rdi=128 wri=128
[ 338.946955] dma dma0chan0: tsi721_submit_sg: DMAC0 sg0/99 addr: 0x3f03fc010 len: 16368
[ 338.946957] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000732f8730 did=0 raddr=0x2000000
[ 338.946958] dma dma0chan0: tsi721_submit_sg: DMAC0 sg1/99 addr: 0x37d53c000 len: 16384
[ 338.946960] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16368
[ 338.946961] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000e8ad552d did=0 raddr=0x2003ff0
[ 338.946965] dma dma0chan0: tsi721_submit_sg: DMAC0 sg2/99 addr: 0x37cea0000 len: 16384
[ 338.946966] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.946968] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000c19d1d22 did=0 raddr=0x2007ff0
[ 338.946969] dma dma0chan0: tsi721_submit_sg: DMAC0 sg3/99 addr: 0x37d8d8000 len: 16384
[ 338.946970] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.946972] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 0000000053540148 did=0 raddr=0x200bff0
……
[ 338.947456] dma dma0chan0: tsi721_submit_sg: DMAC0 sg97/99 addr: 0x3808e8000 len: 65536
[ 338.947458] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 32768
[ 338.947461] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 000000006420b5c7 did=0 raddr=0x21efff0
[ 338.947465] dma dma0chan0: tsi721_submit_sg: DMAC0 sg98/99 addr: 0x3f75f8000 len: 16
[ 338.947468] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 65536
[ 338.947471] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 000000004239733d did=0 raddr=0x21ffff0
[ 338.947473] dma dma0chan0: tsi721_submit_sg: DMAC0 last desc final len: 16
[ 338.947476] dma dma0chan0: tsi721_start_dma: DMAC0 (wrc=227) 2588
[ 338.947481] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.947487] dma dma0chan0: tsi721_issue_pending: DMAC0
[ 338.948773] dma dma0chan0: tsi721_dma_tasklet: DMAC0_INT = 0x8
[ 338.948776] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.948780] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.948870] dma dma0chan0: tsi721_prep_rio_sg: DMAC0 WRITE
[ 338.948873] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.948878] dma dma0chan0: tsi721_submit_sg: DMAC0 BD ring status: rdi=227 wri=227
[ 338.948880] dma dma0chan0: tsi721_submit_sg: DMAC0 sg0/128 addr: 0x410308010 len: 16368
[ 338.948882] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 000000008672dc4c did=0 raddr=0x2000000
[ 338.948884] dma dma0chan0: tsi721_submit_sg: DMAC0 sg1/128 addr: 0x3ef070000 len: 16384
[ 338.948885] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16368
[ 338.948887] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000e915eee4 did=0 raddr=0x2003ff0
[ 338.948889] dma dma0chan0: tsi721_submit_sg: DMAC0 sg2/128 addr: 0x3a8c90000 len: 16384
[ 338.948890] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.948892] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 0000000091833282 did=0 raddr=0x2007ff0
[ 338.948893] dma dma0chan0: tsi721_submit_sg: DMAC0 sg3/128 addr: 0x3b277c000 len: 16384
[ 338.948895] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.948896] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000209615cd did=0 raddr=0x200bff0
接收端的 hdid 接收端如果加上 -v ，发送端也要加上 -v
……
[ 338.949633] dma dma0chan0: tsi721_submit_sg: DMAC0 sg126/128 addr: 0x37d128000 len: 16384
[ 338.949635] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.949638] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000b76696ad did=0 raddr=0x21fbff0
[ 338.949642] dma dma0chan0: tsi721_submit_sg: DMAC0 sg127/128 addr: 0x37da14000 len: 16
[ 338.949644] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.949648] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000cfd81cc2 did=0 raddr=0x21ffff0
[ 338.949650] dma dma0chan0: tsi721_submit_sg: DMAC0 last desc final len: 16
[ 338.949655] dma dma0chan0: tsi721_start_dma: DMAC0 (wrc=355) 2588
[ 338.949659] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.949661] dma dma0chan0: tsi721_issue_pending: DMAC0
[ 338.950953] dma dma0chan0: tsi721_dma_tasklet: DMAC0_INT = 0x8
[ 338.950956] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.950958] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.950995] dma dma0chan0: tsi721_prep_rio_sg: DMAC0 READ
[ 338.950997] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.951002] dma dma0chan0: tsi721_submit_sg: DMAC0 BD ring status: rdi=355 wri=355
[ 338.951003] dma dma0chan0: tsi721_submit_sg: DMAC0 sg0/99 addr: 0x3f03fc010 len: 16368
[ 338.951005] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000ae8cb52a did=0 raddr=0x2000000
[ 338.951007] dma dma0chan0: tsi721_submit_sg: DMAC0 sg1/99 addr: 0x37d53c000 len: 16384
[ 338.951009] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16368
[ 338.951011] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000f083cb61 did=0 raddr=0x2003ff0
[ 338.951012] dma dma0chan0: tsi721_submit_sg: DMAC0 sg2/99 addr: 0x37cea0000 len: 16384
[ 338.951013] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 16384
[ 338.951015] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 0000000093629186 did=0 raddr=0x2007ff0
[ 338.951017] dma dma0chan0: tsi721_submit_sg: DMAC0 sg3/99 addr: 0x37d8d8000 len: 16384
……
[ 338.951478] dma dma0chan0: tsi721_submit_sg: DMAC0 sg97/99 addr: 0x3808e8000 len: 65536
[ 338.951480] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 32768
[ 338.951482] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 000000004dabbcda did=0 raddr=0x21efff0
[ 338.951483] dma dma0chan0: tsi721_submit_sg: DMAC0 sg98/99 addr: 0x3f75f8000 len: 16
[ 338.951485] dma dma0chan0: tsi721_submit_sg: DMAC0 prev desc final len: 65536
[ 338.951486] dma dma0chan0: tsi721_submit_sg: DMAC0 bd_ptr = 00000000a0e9b510 did=0 raddr=0x21ffff0
[ 338.951488] dma dma0chan0: tsi721_submit_sg: DMAC0 last desc final len: 16
[ 338.951490] dma dma0chan0: tsi721_start_dma: DMAC0 (wrc=454) 2588
[ 338.951493] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.951498] dma dma0chan0: tsi721_issue_pending: DMAC0
[ 338.952786] dma dma0chan0: tsi721_dma_tasklet: DMAC0_INT = 0x8
[ 338.952789] dma dma0chan0: tsi721_advance_work: DMAC0
[ 338.952791] dma dma0chan0: tsi721_advance_work: DMAC0 Exit
[ 338.952936] dma dma0chan0: tsi721_free_chan_resources: DMAC0

4 A机器的 TSI721 ~ B机器的 TSI721 (S-RIO NWRITE_R)；

        机器A 将数据传递给 机器B时，是将block DMA的数据转换成了rapidio协议里的NWRITE_R操作。这部分的工作由TSI721芯片自动完成。

        NWRITE_R的具体信息请看《RapidIO ™ Interconnect Specification Part 1: Input/Output Logical Specification》(Rev. 2.1, 08/2009), PAGE 26

5 B机器中的 TSI721~内存

5.1 inbound windows 和 S-RIO Address Translation

The 34/50/66-bit address of a S-RIO request packet (NREAD, NWRITE/SWRITE/NWRITE_R) is translated using one of the
eight 66-bit inbound windows implemented in the SR2PC block.       

Each window is defined by Inbound Window Translated Lower Address Register {0..7} and Inbound Window Translated
Upper Address {0..7}, and Inbound Window Size Register{0..7}

                                                                                        《Tsi721 ™ User Manual》P158

A机器通过NWRITE_R向B机器的SRIO Address写数据，经过S-RIO Address Translation后，A机器写过来的数据实际上是写到了Inbound Window Translated Lower Address Register {0..7} and Inbound Window Translated Upper Address {0..7}指定的内存中了。

5.2 接收的程序流程

5.3 dmesg信息

命令：# ./rio_test_dma -M 0 -I 0x1000000 -R 0x2000000 -v

dmesg 信息

[ 1012.293052] tsi721 0000:03:00.0: tsi721_rio_map_inb_mem: Translated (RIO_0x2000000 -> PCIe_0x000000043e000000), size=0x1000000
[ 1012.293063] tsi721 0000:03:00.0: tsi721_rio_map_inb_mem: Configured IBWIN0 (RIO_0x2000000 -> PCIe_0x000000043e000000), size=0x1000000
[ 1018.033461] tsi721 0000:03:00.0: tsi721_rio_unmap_inb_mem: Unmap IBW mapped to PCIe_0x000000043e000000
[ 1018.033468] tsi721 0000:03:00.0: tsi721_rio_unmap_inb_mem: Disable IBWIN_0

6 block DMA相关的其他问题

6.1 地址对齐问题

简介

        在使用rapidio的DMA功能时，需要linux内核申请DMA物理内存空间。由于TSI721芯片内部特有的地址转换功能，要求linux内核申请的DMA空间的物理内存起始地址要按照申请空间的大小对齐。

例如，当接收端指定Inbound Window大小为64M时，执行以下命令：

# ./rio_test_dma -M 0 -I 0x4000000 -R 0x000000 -v+++ SRIO mport configuration +++
mport: hdid=0, id=0, idx=0, flags=0x3, sys_size=small
link: speed=5.0Gb width=4x
DMA: max_sge=0 max_size=67108864 alignment=0 (HW SG)+++ RapidIO Inbound Window Mode +++mport0 ib_size=0x4000000 PID:1832Allocated/mapped IB buffer (rio_base=0x0_00000000).... press Enter key to exit ....

对应的dmesg信息

[  950.503834] tsi721 0000:03:00.0: tsi721_rio_map_inb_mem: Translated (RIO_0x0 -> PCIe_0x0000000438000000), size=0x4000000
[  950.503843] tsi721 0000:03:00.0: tsi721_rio_map_inb_mem: Configured IBWIN0 (RIO_0x0 -> PCIe_0x0000000438000000), size=0x4000000
[  953.920837] tsi721 0000:03:00.0: tsi721_rio_unmap_inb_mem: Unmap IBW mapped to PCIe_0x0000000438000000
[  953.920844] tsi721 0000:03:00.0: tsi721_rio_unmap_inb_mem: Disable IBWIN_0

        接收端需要的Inbound Window大小为64M(0x4000000)，linux内核DMA申请到的物理内存起始地址是0x438000000，是0x4000000的倍数(或者说是以0x4000000对齐)。

机理分析

        TSI721手册第8.3节“S-RIO Address Translation”介绍了TSI721支持的窗口大小必须是2的N次方，并且在4K～16GB之间。

TSI721手册的信息

        上面图片中的SRIO Address可以通过rio_test_dma命令的“-R”参数指定，而PCIE Address实际上是linux内核DMA申请到的物理内存起始地址。

对应TSI721驱动里的tsi721_rio_map_inb_mem()函数代码

if (!is_power_of_2(size) || size < 0x1000 ||((u64)lstart & (size - 1)) || (rstart & (size - 1)))   //lstart: DMA addr; rstart:SRIO addrreturn -EINVAL;

        根据上面的图片，和tsi721_rio_map_inb_mem()函数代码，要求linux内核申请的DMA空间的物理内存起始地址要按照申请空间的大小对齐。

6.2 DMA申请大内存

机理分析

        根据TSI721的手册，TSI721单次DMA传输最大数据量是64MB，对应的linux内核需要申请的最大DMA内存空间也是64MB。

        DMA申请的物理内存是连续内存，而linux内核下单次申请的最大连续物理内存受内核配置CONFIG_FORCE_MAX_ZONEORDER控制。

例如：

当内核使用4K页时，CONFIG_FORCE_MAX_ZONEORDER的默认值是11，对应的单次申请的最大连续物理内存是(2的10次方) * 4K = 4MB。

当内核使用16K页时，CONFIG_FORCE_MAX_ZONEORDER的默认值是12，对应的单次申请的最大连续物理内存是(2的11次方) * 16K = 32MB。

当内核使用64K页时，CONFIG_FORCE_MAX_ZONEORDER的默认值是14，对应的单次申请的最大连续物理内存是(2的13次方) * 64K = 512MB。

所以，当linux内核使用的页大小是4K 或者 16K时，无法单次申请64MB的连续物理内存。

解决方法

方案1、修改CONFIG_FORCE_MAX_ZONEORDER

方案2、使用CMA

方案3、指定DMA使用"mem="预留的内存

        第一步、预留内存

                        在启动参数里添加: mem=8G，然后重启。查看/proc/iomem的最后一行，例如最后一行是“2183fe7000-2183ffffff : reserved”，表示预留的内存起始地址为0x2184000000。

        第二部、指定rio_mport_cdev模块参数

                        insmod rio_mport_cdev.ko dbg_level=0xffff rio_res_mem=0x2184000000 rio_res_size=0x200000

        第三步、运行测试程序(rio_test_dma)时，指定使用预留内存(接收端)

                        ./rio_test_dma -M 0 -I 0x200000 -R 0x200000 -L 0x2184000000