DeepSpeed Profiling

2024-06-11 04:12
文章标签 deepspeed profiling

本文主要是介绍DeepSpeed Profiling,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

DeepSpeed自带的Profiler

-------------------------- DeepSpeed Flops Profiler --------------------------
Profile Summary at step 10:
Notations:
data parallel size (dp_size), model parallel size(mp_size),
number of parameters (params), number of multiply-accumulate operations(MACs),
number of floating-point operations (flops), floating-point operations per second (FLOPS),
fwd latency (forward propagation latency), bwd latency (backward propagation latency),
step (weights update latency), iter latency (sum of fwd, bwd and step latency)world size:                                                   1
data parallel size:                                           1
model parallel size:                                          1
batch size per GPU:                                           80
params per gpu:                                               336.23 M
params of model = params per GPU * mp_size:                   336.23 M
fwd MACs per GPU:                                             3139.93 G
fwd flops per GPU:                                            6279.86 G
fwd flops of model = fwd flops per GPU * mp_size:             6279.86 G
fwd latency:                                                  76.67 ms
bwd latency:                                                  108.02 ms
fwd FLOPS per GPU = fwd flops per GPU / fwd latency:          81.9 TFLOPS
bwd FLOPS per GPU = 2 * fwd flops per GPU / bwd latency:      116.27 TFLOPS
fwd+bwd FLOPS per GPU = 3 * fwd flops per GPU / (fwd+bwd latency):   102.0 TFLOPS
step latency:                                                 34.09 us
iter latency:                                                 184.73 ms
samples/second:                                               433.07----------------------------- Aggregated Profile per GPU -----------------------------
Top modules in terms of params, MACs or fwd latency at different model depths:
depth 0:params      - {'BertForPreTrainingPreLN': '336.23 M'}MACs        - {'BertForPreTrainingPreLN': '3139.93 GMACs'}fwd latency - {'BertForPreTrainingPreLN': '76.39 ms'}
depth 1:params      - {'BertModel': '335.15 M', 'BertPreTrainingHeads': '32.34 M'}MACs        - {'BertModel': '3092.96 GMACs', 'BertPreTrainingHeads': '46.97 GMACs'}fwd latency - {'BertModel': '34.29 ms', 'BertPreTrainingHeads': '3.23 ms'}
depth 2:params      - {'BertEncoder': '302.31 M', 'BertLMPredictionHead': '32.34 M'}MACs        - {'BertEncoder': '3092.88 GMACs', 'BertLMPredictionHead': '46.97 GMACs'}fwd latency - {'BertEncoder': '33.45 ms', 'BertLMPredictionHead': '2.61 ms'}
depth 3:params      - {'ModuleList': '302.31 M', 'Embedding': '31.79 M', 'Linear': '31.26 M'}MACs        - {'ModuleList': '3092.88 GMACs', 'Linear': '36.23 GMACs'}fwd latency - {'ModuleList': '33.11 ms', 'BertPredictionHeadTransform': '1.83 ms''}
depth 4:params      - {'BertLayer': '302.31 M', 'LinearActivation': '1.05 M''}MACs        - {'BertLayer': '3092.88 GMACs', 'LinearActivation': '10.74 GMACs'}fwd latency - {'BertLayer': '33.11 ms', 'LinearActivation': '1.43 ms'}
depth 5:params      - {'BertAttention': '100.76 M', 'BertIntermediate': '100.76 M'}MACs        - {'BertAttention': '1031.3 GMACs', 'BertIntermediate': '1030.79 GMACs'}fwd latency - {'BertAttention': '19.83 ms', 'BertOutput': '4.38 ms'}
depth 6:params      - {'LinearActivation': '100.76 M', 'Linear': '100.69 M'}MACs        - {'LinearActivation': '1030.79 GMACs', 'Linear': '1030.79 GMACs'}fwd latency - {'BertSelfAttention': '16.29 ms', 'LinearActivation': '3.48 ms'}------------------------------ Detailed Profile per GPU ------------------------------
Each module profile is listed after its name in the following order:
params, percentage of total params, MACs, percentage of total MACs, fwd latency, percentage of total fwd latency, fwd FLOPSBertForPreTrainingPreLN(336.23 M, 100.00% Params, 3139.93 GMACs, 100.00% MACs, 76.39 ms, 100.00% latency, 82.21 TFLOPS,(bert): BertModel(335.15 M, 99.68% Params, 3092.96 GMACs, 98.50% MACs, 34.29 ms, 44.89% latency, 180.4 TFLOPS,(embeddings): BertEmbeddings(...)(encoder): BertEncoder(302.31 M, 89.91% Params, 3092.88 GMACs, 98.50% MACs, 33.45 ms, 43.79% latency, 184.93 TFLOPS,(FinalLayerNorm): FusedLayerNorm(...)(layer): ModuleList(302.31 M, 89.91% Params, 3092.88 GMACs, 98.50% MACs, 33.11 ms, 43.35% latency, 186.8 TFLOPS,(0): BertLayer(12.6 M, 3.75% Params, 128.87 GMACs, 4.10% MACs, 1.29 ms, 1.69% latency, 199.49 TFLOPS,(attention): BertAttention(4.2 M, 1.25% Params, 42.97 GMACs, 1.37% MACs, 833.75 us, 1.09% latency, 103.08 TFLOPS,(self): BertSelfAttention(3.15 M, 0.94% Params, 32.23 GMACs, 1.03% MACs, 699.04 us, 0.92% latency, 92.22 TFLOPS,(query): Linear(1.05 M, 0.31% Params, 10.74 GMACs, 0.34% MACs, 182.39 us, 0.24% latency, 117.74 TFLOPS,...)(key): Linear(1.05 M, 0.31% Params, 10.74 GMACs, 0.34% MACs, 57.22 us, 0.07% latency, 375.3 TFLOPS,...)(value): Linear(1.05 M, 0.31% Params, 10.74 GMACs, 0.34% MACs, 53.17 us, 0.07% latency, 403.91 TFLOPS,...)(dropout): Dropout(...)(softmax): Softmax(...))(output): BertSelfOutput(1.05 M, 0.31% Params, 10.74 GMACs, 0.34% MACs, 114.68 us, 0.15% latency, 187.26 TFLOPS,(dense): Linear(1.05 M, 0.31% Params, 10.74 GMACs, 0.34% MACs, 64.13 us, 0.08% latency, 334.84 TFLOPS, ...)(dropout): Dropout(...)))(PreAttentionLayerNorm): FusedLayerNorm(...)(PostAttentionLayerNorm): FusedLayerNorm(...)(intermediate): BertIntermediate(4.2 M, 1.25% Params, 42.95 GMACs, 1.37% MACs, 186.68 us, 0.24% latency, 460.14 TFLOPS,(dense_act): LinearActivation(4.2 M, 1.25% Params, 42.95 GMACs, 1.37% MACs, 175.0 us, 0.23% latency, 490.86 TFLOPS,...))(output): BertOutput(4.2 M, 1.25% Params, 42.95 GMACs, 1.37% MACs, 116.83 us, 0.15% latency, 735.28 TFLOPS,(dense): Linear(4.2 M, 1.25% Params, 42.95 GMACs, 1.37% MACs, 65.57 us, 0.09% latency, 1310.14 TFLOPS,...)(dropout): Dropout(...)))...(23): BertLayer(...)))(pooler): BertPooler(...))(cls): BertPreTrainingHeads(...)
)
------------------------------------------------------------------------------

注意:

小写flops指计算量;大写FLOPS指flops每秒;

backward计算量,是forward的2倍;

乘加运算量(MAC) * 2 = flops

FLOPS,可以和GPU理论上界比较,看看差多少;

通过观察上层module的参数量、计算量、延迟、吞吐量,找到包含很多”小“计算的module,从而看看有没有可能进一步优化为一个”大“计算;(kernel fusion)

和PyTorch profiler的区别:DeepSpeed是从Module角度来看;PyTorch是从operator角度来看;

推理阶段,用:

get_model_profile

训练阶段,用:

FlopsProfiler

PyTorch Profiler:

torch.profiler.schedule(wait=5, warmup=2, active=6, repeat=2)

(跳过5个step,然后运行profiler 2个step(结果丢弃),运行profiler且保留结果6个step),重复()步骤2轮;目的:跳过一上来warmup阶段的慢;

选项:

with_stack=True: 记录call stack;增加一些overhead;

activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]:CPU是记录PyTorch operators,CUDA是记录CUDA kernel;

profile_memory=True:对CPU和GPU的memory allocate/release事件进行记录;区分Allocated和Reserved这2种;

自定义scope: (record_function)

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:with record_function("model_forward"):model_engine(inputs)print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

这篇关于DeepSpeed Profiling的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/1050161

相关文章

通过Python库ydata-profiling生成数据分析报告

一:ydata-profiling库的介绍 ydata-profiling是一个强大的 Python 库,它为 Pandas DataFrame 提供了快速的探索性数据分析(EDA)。它能够自动生成包含详细统计信息的交互式 HTML 报告,使得数据分析变得更加直观和便捷。  安装方法: 可以通过 pip 进行安装: pip install ydata-profiling 主要特点: 自

Qt WebEngine Debugging and Profiling

控制台记录 在Qt WebEngine中执行的JavaScript可以使用Chrome控制台API将信息记录到控制台。日志消息将转发到日志js 记录类别中的 Qt日志记录工具。但是,默认情况下仅打印警告和致命消息。要更改此设置,您必须为js类别设置自定义规则,或者通过重新实现QWebEnginePage :: javaScriptConsoleMessage()或连接到WebEngineView

k8s volcano + deepspeed多机训练 + RDMA ROCE+ 用户权限安全方案【建议收藏】

前提:nvidia、cuda、nvidia-fabricmanager等相关的组件已经在宿主机正确安装,如果没有安装可以参考我之前发的文章GPU A800 A100系列NVIDIA环境和PyTorch2.0基础环境配置【建议收藏】_a800多卡运行环境配置-CSDN博客文章浏览阅读1.1k次,点赞8次,收藏16次。Ant系列GPU支持 NvLink & NvSwitch,若您使用多GPU卡的机型,

deepspeed win11 安装

目录 git地址: aio报错: 编译 报错 ops已存在: 修改拷贝代码: git地址: Bug Report: Issues Building DeepSpeed on Windows · Issue #5679 · microsoft/DeepSpeed · GitHub aio报错: setup.py 配置变量 os.environ['DISTUTILS_U

MixtralForCausalLM DeepSpeed Inference节约HOST内存【最新的方案】

MixtralForCausalLM DeepSpeed Inference节约HOST内存【最新的方案】 一.效果二.特别说明三.测试步骤1.创建Mixtral-8x7B配置文件(简化了)2.生成随机模型,运行cpu float32推理,输出结果3.加载模型,cuda 单卡推理4.DS 4 TP cuda 推理5.分别保存DS 4TP每个rank上engine.module的权值6.DS

DeepSpeed Mixture-of-Quantization (MoQ)

属于QAT (Quantization-Aware Training)的一种,训练阶段用量化。 特点是: 1. 从16-bit INT开始训练,逐渐减1bit,训练一些steps就减1bit,直至减至8bit INT; 2. (可选,不一定非用)多久减1bit,这个策略,使用模型参数的二阶特征来决定,每层独立的(同一时刻,每层的特征值们大小不一致,也就造成bit减少速度不一致,造成bit数目

DeepSpeed MoE

MoE概念 模型参数增加很多;计算量没有增加(gating+小FNN,比以前的大FNN计算量要小);收敛速度变快; 效果:PR-MoE > 普通MoE > DenseTransformer MoE模型,可视为Sparse Model,因为每次参与计算的是一部分参数; Expert并行,可以和其他并行方式,同时使用:  ep_size指定了MoE进程组大小,一个模型replica的所

DeepSpeed Learning Rate Scheduler

Learning Rate Range Test (LRRT) 训练试跑,该lr scheduler从小到大增长lr,同时记录下validatin loss;人来观察在训练多少step之后,loss崩掉(diverge)了,进而为真正跑训练,挑选合适的lr区间; "scheduler": {"type": "LRRangeTest","params": {"lr_range_test_min_l

DeepSpeed Huggingface模型的自动Tensor并行

推理阶段。在后台,1. DeepSpeed会把运行高性能kernel(kernel injection),加快推理速度,这些对用户是透明的; 2. DeepSpeed会根据mp_size来将模型放置在多个GPU卡上,自动模型并行; import osimport torchimport transformersimport deepspeedlocal_rank = int(os.get

DeepSpeed Autotuning

AutoTuning 用不同的系统参数试跑用户的模型训练,尝试不同的参数组合,给出每种参数组合的速度,供用户去选择较块的来进行真正的训练。 ZeRO optimization stages;micro-batch sizes;optimizer, scheduler, fp16等; 在DeepSpeed配置文件里,设定: "autotuning": { "enabled": true }