最近幾個月，隨著ChatGPT的現象級表現，大模型如雨后春筍般涌現。而模型推理是抽象的算法模型觸達具體的實際業務的最后一公里。

但是在這個環節中，仍然還有很多已經是大家共識的痛點和訴求，比如：

任何線上產品的用戶體驗都與服務的響應時長成反比，復雜的模型如何極致地壓縮請求時延？

模型推理通常是資源常駐型服務，如何通過提升服務單機性能從而增加QPS，同時大幅降低資源成本？

端-邊-云是現在模型服務發展的必然趨勢，如何讓離線訓練的模型“瘦身塑形”從而在更多設備上快速部署使用？

因此，模型推理的加速優化成為了AI界的重要研究領域。

本文給大家分享大模型推理加速引擎FasterTransformer的基本使用。

image.png

FasterTransformer簡介

NVIDIA FasterTransformer (FT)是一個用于實現基于Transformer的神經網絡推理的加速引擎。它包含Transformer塊的高度優化版本的實現，其中包含編碼器和解碼器部分。使用此模塊，您可以運行編碼器-解碼器架構模型（如：T5）、僅編碼器架構模型（如：BERT）和僅解碼器架構模型（如：GPT）的推理。

FT框架是用C++/CUDA編寫的，依賴于高度優化的 cuBLAS、cuBLASLt 和 cuSPARSELt 庫，這使您可以在 GPU 上進行快速的 Transformer 推理。

與NVIDIA TensorRT等其他編譯器相比，FT 的最大特點是它支持以分布式方式進行 Transformer 大模型推理。

下圖顯示了如何使用張量并行 (TP) 和流水線并行 (PP) 技術將基于Transformer架構的神經網絡拆分到多個 GPU 和節點上。

當每個張量被分成多個塊時，就會發生張量并行，并且張量的每個塊都可以放置在單獨的 GPU 上。在計算過程中，每個塊在不同的 GPU 上單獨并行處理；最后，可以通過組合來自多個 GPU 的結果來計算最終張量。

當模型被深度拆分，并將不同的完整層放置到不同的 GPU/節點上時，就會發生流水線并行。

在底層，節點間或節點內通信依賴于 MPI 、 NVIDIA NCCL、Gloo等。因此，使用FasterTransformer，您可以在多個 GPU 上以張量并行運行大型Transformer，以減少計算延遲。同時，TP 和 PP 可以結合在一起，在多 GPU 節點環境中運行具有數十億、數萬億個參數的大型 Transformer 模型。

除了使用 C ++ 作為后端部署，FasterTransformer 還集成了 TensorFlow（使用 TensorFlow op）、PyTorch （使用 Pytorch op）和 Triton作為后端框架進行部署。當前，TensorFlow op 僅支持單 GPU，而 PyTorch op 和 Triton 后端都支持多 GPU 和多節點。

目前，FT 支持了 Megatron-LM GPT-3、GPT-J、BERT、ViT、Swin Transformer、Longformer、T5 和 XLNet 等模型。您可以在 GitHub 上的 FasterTransformer庫中查看最新的支持矩陣。

FT 適用于計算能力 >= 7.0 的 GPU，例如 V100、A10、A100 等。

下圖展示了 GPT-J 6B 參數的模型推斷加速比較：

image.png

FasterTransformer 中的優化技術

與深度學習訓練的通用框架相比，FT 使您能夠獲得更快的推理流水線以及基于 Transformer 的神經網絡具有更低的延遲和更高的吞吐量。FT 對 GPT-3 和其他大型Transformer模型進行的一些優化技術包括：

層融合（Layer fusion）

這是預處理階段的一組技術，將多層神經網絡組合成一個單一的神經網絡，將使用一個單一的核（kernel）進行計算。這種技術減少了數據傳輸并增加了數學密度，從而加速了推理階段的計算。例如， multi-head attention 塊中的所有操作都可以合并到一個核（kernel）中。

自回歸模型的推理優化(激活緩存)

為了防止通過Transformer重新計算每個新 token 生成器的先前鍵和值，FT 分配了一個緩沖區來在每一步存儲它們。

雖然需要一些額外的內存使用，但 FT 可以節省重新計算的成本。該過程如下圖所示。相同的緩存機制用于 NN 的多個部分。

image.png

內存優化

與 BERT 等傳統模型不同，大型 Transformer 模型具有多達數萬億個參數，占用數百 GB 存儲空間。即使我們以半精度存儲模型，GPT-3 175b 也需要 350 GB。因此有必要減少其他部分的內存使用。

例如，在 FasterTransformer 中，我們在不同的解碼器層重用了激活/輸出的內存緩沖（buffer）。由于 GPT-3 中的層數為 96，因此我們只需要 1/96 的內存量用于激活。

使用 MPI 和 NCCL 實現節點間/節點內通信并支持模型并行

FasterTransormer 同時提供張量并行和流水線并行。對于張量并行，FasterTransformer 遵循了 Megatron 的思想。對于自注意力塊和前饋網絡塊，FT 按行拆分第一個矩陣的權重，并按列拆分第二個矩陣的權重。通過優化，FT 可以將每個 Transformer 塊的歸約（reduction）操作減少到兩次。

對于流水線并行，FasterTransformer 將整批請求拆分為多個微批，隱藏了通信的空泡（bubble）。FasterTransformer 會針對不同情況自動調整微批量大小。

MatMul 核自動調整（GEMM 自動調整）

矩陣乘法是基于Transformer的神經網絡中最主要和繁重的操作。FT 使用來自 CuBLAS 和 CuTLASS 庫的功能來執行這些類型的操作。重要的是要知道 MatMul 操作可以在“硬件”級別使用不同的底層（low-level）算法以數十種不同的方式執行。

GemmBatchedEx函數實現了 MatMul 操作，并以cublasGemmAlgo_t作為輸入參數。使用此參數，您可以選擇不同的底層算法進行操作。

FasterTransformer 庫使用此參數對所有底層算法進行實時基準測試，并為模型的參數和您的輸入數據（注意層的大小、注意頭的數量、隱藏層的大小）選擇最佳的一個。此外，FT 對網絡的某些部分使用硬件加速的底層函數，例如：__expf、__shfl_xor_sync。

低精度推理

FT 的核（kernels）支持使用 fp16 和 int8 等低精度輸入數據進行推理。由于較少的數據傳輸量和所需的內存，這兩種機制都會加速。同時，int8 和 fp16 計算可以在特殊硬件上執行，例如：Tensor Core（適用于從 Volta 開始的所有 GPU 架構）和即將推出的 Hopper GPU 中的Transformer引擎。

除此之外還有快速的 C++ BeamSearch 實現、當模型的權重部分分配到八個 GPU 之間時，針對 TensorParallelism 8 模式優化的 all-reduce。

上面簡述了FasterTransformer，下面將演示針對 Bloom 模型以 PyTorch 作為后端使用FasterTransformer。

FasterTransformer GPT 簡介

下文將會使用BLOOM模型進行演示，而 BLOOM 是一個利用 ALiBi(用于添加位置嵌入) 的 GPT 模型的變體，因此，本文先簡要介紹一下 GPT 的相關工作。GPT是僅解碼器架構模型的一種變體，沒有編碼器模塊，使用GeLU作為激活。

FasterTransformer GPT 工作流程

下圖展示了 FasterTransformer GPT 的工作流程。與 BERT（僅編碼器結構） 和編碼器-解碼器結構不同，GPT 接收一些輸入 id 作為上下文，并生成相應的輸出 id 作為響應。在此工作流程中，主要瓶頸是 GptDecoderLayer（transformer塊），因為當我們增加層數時，時間會線性增加。在 GPT-3 中，GptDecoderLayer 占用了大約 95% 的總時間。

image.png

FasterTransformer 將整個工作流程分成兩部分。

第一部分是“計算上下文（輸入 ids）的 k/v 緩存”。

第二部分是“自回歸生成輸出 ids”。

這兩部分的操作類似，只是SelfAttention中張量的形狀不同。因此，我們使用 2 種不同的實現來處理兩種不同的情況，如下圖所示。

image.png

在 DecoderSelfAttention 中，查詢的序列長度始終為 1，因此我們使用自定義的 fused masked multi-head attention kernel 來處理。另一方面，ContextSelfAttention 中查詢的序列長度是最大輸入長度，因此我們使用 cuBLAS 來利用tensor core。

以下的示例演示了如何運行多 GPU 和多節點的 GPT 模型。

examples/cpp/multi_gpu_gpt_example.cc：它使用MPI來組織所有的GPU。

examples/cpp/multi_gpu_gpt_triton_example.cc：它在節點內使用線程，在節點間使用 MPI。此示例還演示了如何使用基于 FasterTransformer 的 Triton 后端 API 來運行 GPT 模型。

examples/pytorch/gpt/multi_gpu_gpt_example.py：這個例子和examples/cpp/multi_gpu_gpt_example.cc類似，但是通過PyTorch OP封裝了FasterTransformer的實例。

總之，運行 GPT 模型的工作流程是：

通過 MPI 或線程初始化 NCCL 通信并設置張量并行和流水線并行的ranks

按張量并行、流水線并行和其他模型超參數的ranks加載權重。

通過張量并行、流水線并行和其他模型超參數的ranks創建ParalelGpt實例。

接收來自客戶端的請求并將請求轉換為 ParallelGpt 的輸入張量格式。

運行forward

將 ParallelGpt 的輸出張量轉換為客戶端的響應并返回響應。

在C++示例代碼中，我們跳過第4步和第6步，通過examples/cpp/multi_gpu_gpt/start_ids.csv加載該請求。在 PyTorch 示例代碼中，該請求來自 PyTorch 端。在 Triton 示例代碼中，我們有從步驟 1 到步驟 6 的完整示例。

源代碼放在 src/fastertransformer/models/multi_gpu_gpt/ParallelGpt.cc 中。其中，GPT的構造函數參數包括head_num、num_layer、tensor_para、pipeline_para等，GPT的輸入參數包括input_ids、input_lengths、output_seq_len等；GPT的輸出參數包括output_ids（包含 input_ids 和生成的 id）、sequence_length、output_log_probs、cum_log_probs、context_embeddings。

FasterTransformer GPT 優化

核優化：很多核都是基于已經高度優化的解碼器和解碼碼模塊的核。為了防止重新計算以前的鍵和值，我們將在每一步分配一個緩沖區來存儲它們。雖然它需要一些額外的內存使用，但我們可以節省重新計算的成本。

內存優化：與 BERT 等傳統模型不同，GPT-3 有 1750 億個參數，即使我們以半精度存儲模型也需要 350 GB。因此，我們必須減少其他部分的內存使用。在 FasterTransformer 中，我們將重用不同解碼器層的內存緩沖。由于 GPT-3 的層數是 96，我們只需要 1/96 的內存。

模型并行：在GPT模型中，FasterTransormer同時提供張量并行和流水線并行。對于張量并行，FasterTransformer 遵循了 Megatron 的思想。對于自注意力塊和前饋網絡塊，我們按行拆分第一個矩陣乘法的權重，按列拆分第二個矩陣乘法的權重。通過優化，我們可以將每個transformer block的歸約操作減少到 2 次，工作流程如下圖所示。

對于流水線并行，FasterTransformer 將整批請求拆分為多個微批并隱藏通信空泡。FasterTransformer 會針對不同情況自動調整微批量大小。用戶可以通過修改 gpt_config.ini 文件來調整模型并行度。我們建議在節點內使用張量并行，在節點間使用流水線并行，因為，張量并行需要更多的 NCCL 通信。

多框架：FasterTransformer除了c上的源代碼，還提供了TensorFlow op、PyTorch op和Triton backend。目前TensorFlow op只支持單GPU，而PyTorch op和Triton backend支持多GPU和多節點。FasterTransformer 還提供了一個工具，可以將 Megatron 的模型拆分并轉換為FasterTransformer二進制文件，以便 FasterTransformer 可以直接加載二進制文件，從而避免為模型并行而進行的額外拆分模型工作。

FasterTransformer GPT 推理選項

FasterTransformer GPT 還提供環境變量以針對特定用途進行調整。

環境搭建

基礎環境配置

首先確保您具有以下組件：

NVIDIA Docker 和 NGC 容器

NVIDIA Pascal/Volta/Turing/Ampere 系列的 GPU

基礎組件版本要求：

CMake: 3.13及以上版本

CUDA: 11.0及以上版本

NCCL: 2.10及以上版本

Python: 3.8.13

PyTorch: 1.13.0

這些組件在 Nvidia 官方提供的 TensorFlow/PyTorch Docker 鏡像中很容易獲得。

構建FasterTransformer

推薦使用Nvidia官方提供的鏡像，如：nvcr.io/nvidia/tensorflow:22.09-tf1-py3 、 nvcr.io/nvidia/pytorch:22.09-py3等，當然也可以使用Pytorch官方提供的鏡像。

首先，拉取相應版本的PyTorch鏡像。

docker pull nvcr.io/nvidia/pytorch:22.09-py3

鏡像下載完成之后，創建容器，以便后續進行編譯和構建FasterTransformer。

nvidia-docker run -dti --name bloom_faster_transformer 
--restart=always --gpus all --network=host 
--shm-size 5g 
-v /home/gdong/workspace/code:/workspace/code 
-v /home/gdong/workspace/data:/workspace/data 
-v /home/gdong/workspace/model:/workspace/model 
-v /home/gdong/workspace/output:/workspace/output 
-w /workspace 
nvcr.io/nvidia/pytorch:22.09-py3 
bash

進入容器。

docker exec -it bloom_faster_transformer bash

下載FasterTransformer代碼。

cd code
git clone https://github.com/NVIDIA/FasterTransformer.git
cd FasterTransformer/
git submodule init && git submodule update

進入build構建FasterTransformer。

mkdir -p build
cd build

然后，執行cmake PATH命令生成 Makefile 文件。

cmake -DSM=80 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON -DBUILD_MULTI_GPU=ON ..

注意：

第一點：腳本中-DMS=xx的xx表示GPU的計算能力。下表顯示了常見GPU的計算能力。

默認情況下，-DSM 設置為 70、75、80 和 86。當用戶設置更多類型的 -DSM 時，需要更長的編譯時間。因此，我們建議只為您使用的設備設置 -DSM。

第二點：本文使用Pytorch作為后端，因此，腳本中添加了-DBUILD_PYT=ON配置項。這將構建 TorchScript 自定義類。因此，請確保 PyTorch 版本大于 1.5.0。

運行過程：

-- The CXX compiler identification is GNU 9.4.0
-- The CUDA compiler identification is NVIDIA 11.8.89
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /usr/local/cuda (found suitable version "11.8", minimum required is "10.2") 
CUDA_VERSION 11.8 is greater or equal than 11.0, enable -DENABLE_BF16 flag
-- Found CUDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so  
-- Add DBUILD_CUTLASS_MOE, requires CUTLASS. Increases compilation time
-- Add DBUILD_CUTLASS_MIXED_GEMM, requires CUTLASS. Increases compilation time
-- Running submodule update to fetch cutlass
-- Add DBUILD_MULTI_GPU, requires MPI and NCCL
-- Found MPI_CXX: /opt/hpcx/ompi/lib/libmpi.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Found NCCL: /usr/include  
-- Determining NCCL version from /usr/include/nccl.h...
-- Looking for NCCL_VERSION_CODE
-- Looking for NCCL_VERSION_CODE - not found
-- Found NCCL (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnccl.so.2.15.1)
-- NVTX is enabled.
-- Assign GPU architecture (sm=80)
-- Use WMMA
CMAKE_CUDA_FLAGS_RELEASE: -O3 -DNDEBUG -Xcompiler -O3 -DCUDA_PTX_FP8_F2FP_ENABLED --use_fast_math
-- COMMON_HEADER_DIRS: /workspace/code/FasterTransformer;/usr/local/cuda/include;/workspace/code/FasterTransformer/3rdparty/cutlass/include;/workspace/code/FasterTransformer/src/fastertransformer/cutlass_extensions/include;/workspace/code/FasterTransformer/3rdparty/trt_fp8_fmha/src;/workspace/code/FasterTransformer/3rdparty/trt_fp8_fmha/generated
-- Found CUDA: /usr/local/cuda (found version "11.8") 
-- Caffe2: CUDA detected: 11.8
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.8
-- Found cuDNN: v8.6.0  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 672ee683
-- Added CUDA NVCC flags for: -gencode;arch=compute_80,code=sm_80
-- Found Torch: /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch.so  
-- USE_CXX11_ABI=True
-- The C compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /opt/conda/bin/python3.8 (found version "3.8.13") found components: Interpreter 
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/code/FasterTransformer/build

之后，通過make使用12個線程去執行編譯加快編譯速度：

make -j12

運行過程：

[  0%] Building CXX object src/fastertransformer/kernels/cutlass_kernels/CMakeFiles/cutlass_preprocessors.dir/cutlass_preprocessors.cc.o
[  0%] Building CXX object src/fastertransformer/utils/CMakeFiles/nvtx_utils.dir/nvtx_utils.cc.o
[  0%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/layernorm_kernels.dir/layernorm_kernels.cu.o
[  0%] Building CXX object src/fastertransformer/utils/CMakeFiles/cuda_utils.dir/cuda_utils.cc.o
[  0%] Building CXX object src/fastertransformer/utils/CMakeFiles/logger.dir/logger.cc.o
[  1%] Building CXX object 3rdparty/common/CMakeFiles/cuda_driver_wrapper.dir/cudaDriverWrapper.cpp.o
[  1%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/custom_ar_kernels.dir/custom_ar_kernels.cu.o
[  1%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/add_residual_kernels.dir/add_residual_kernels.cu.o
[  1%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/activation_kernels.dir/activation_kernels.cu.o
[  1%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/transpose_int8_kernels.dir/transpose_int8_kernels.cu.o
[  2%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/unfused_attention_kernels.cu.o
[  2%] Building CUDA object src/fastertransformer/kernels/CMakeFiles/bert_preprocess_kernels.dir/bert_preprocess_kernels.cu.o
[  2%] Linking CUDA device code CMakeFiles/cuda_driver_wrapper.dir/cmake_device_link.o
[  2%] Linking CXX static library ../../lib/libcuda_driver_wrapper.a
[  2%] Built target cuda_driver_wrapper
...
[100%] Linking CXX executable ../../../bin/gptneox_example
[100%] Built target gptj_triton_example
[100%] Building CXX object examples/cpp/multi_gpu_gpt/CMakeFiles/multi_gpu_gpt_triton_example.dir/multi_gpu_gpt_triton_example.cc.o
[100%] Built target gptj_example
[100%] Building CXX object examples/cpp/multi_gpu_gpt/CMakeFiles/multi_gpu_gpt_interactive_example.dir/multi_gpu_gpt_interactive_example.cc.o
[100%] Built target gptneox_example
[100%] Linking CXX executable ../../../bin/multi_gpu_gpt_example
[100%] Linking CXX executable ../../../bin/gptneox_triton_example
[100%] Built target multi_gpu_gpt_example
[100%] Built target gptneox_triton_example
[100%] Linking CXX executable ../../../bin/multi_gpu_gpt_triton_example
[100%] Linking CXX static library ../../../../lib/libth_t5.a
[100%] Built target th_t5
[100%] Built target multi_gpu_gpt_triton_example
[100%] Linking CXX executable ../../../bin/multi_gpu_gpt_async_example
[100%] Linking CXX executable ../../../bin/multi_gpu_gpt_interactive_example
[100%] Built target multi_gpu_gpt_async_example
[100%] Linking CXX static library ../../../../lib/libth_parallel_gpt.a
[100%] Built target th_parallel_gpt
[100%] Linking CXX shared library ../../../lib/libth_transformer.so
[100%] Built target multi_gpu_gpt_interactive_example
[100%] Built target th_transformer

至此，構建FasterTransformer完成。

安裝依賴包

安裝進行模型推理所需要的依賴包。

cd /workspace/code/FasterTransformer
pip install -r examples/pytorch/gpt/requirement.txt -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn

數據與模型準備

模型

本文使用BLOOM模型進行演示，它不需要學習位置編碼，并允許模型生成比訓練中使用的序列長度更長的序列。BLOOM 也具有與 OpenAI GPT 相似的結構。因此，像 OPT 一樣，FT 通過 GPT 類提供了 BLOOM 模型作為變體。用戶可以使用 examples/pytorch/gpt/utils/huggingface_bloom_convert.py 將預訓練的 Huggingface BLOOM 模型轉換為 fastertransformer 文件格式。

我們使用bloomz-560m作為基礎模型。該模型是基于bloom-560m在xP3數據集上對多任務進行了微調而得到的。

下載模型：

cd /workspace/model
git lfs clone https://huggingface.co/bigscience/bloomz-560m

模型文件：

> ls -al bloomz-560m
total 2198796
drwxr-xr-x 4 root root       4096 Apr 25 16:50 .
drwxr-xr-x 4 root root       4096 Apr 26 07:06 ..
drwxr-xr-x 9 root root       4096 Apr 25 16:53 .git
-rw-r--r-- 1 root root       1489 Apr 25 16:50 .gitattributes
-rw-r--r-- 1 root root      24778 Apr 25 16:50 README.md
-rw-r--r-- 1 root root        715 Apr 25 16:50 config.json
drwxr-xr-x 4 root root       4096 Apr 25 16:50 logs
-rw-r--r-- 1 root root 1118459450 Apr 25 16:53 model.safetensors
-rw-r--r-- 1 root root 1118530423 Apr 25 16:53 pytorch_model.bin
-rw-r--r-- 1 root root         85 Apr 25 16:50 special_tokens_map.json
-rw-r--r-- 1 root root   14500438 Apr 25 16:50 tokenizer.json
-rw-r--r-- 1 root root        222 Apr 25 16:50 tokenizer_config.json

數據集

本文使用Lambada數據集，它是一個NLP（自然語言處理）任務中使用的數據集。它包含大量的英文句子，并要求模型去預測下一個單詞，這種任務稱為語言建模。Lambada數據集的特點是它的句子長度較長，并且包含更豐富的語義信息。因此，對于語言模型的評估來說是一個很好的測試數據集。

下載LAMBADA測試數據集。

cd /workspace/data
wget -c https://github.com/cybertronai/bflm/raw/master/lambada_test.jsonl

數據格式如下：

{"text": "In my palm is a clear stone, and inside it is a small ivory statuette. A guardian angel.

"Figured if you're going to be out at night getting hit by cars, you might as well have some backup."

I look at him, feeling stunned. Like this is some sort of sign. But as I stare at Harlin, his mouth curved in a confident grin, I don't care about signs"}
{"text": "Give me a minute to change and I'll meet you at the docks." She'd forced those words through her teeth.

"No need to change. We won't be that long."

Shane gripped her arm and started leading her to the dock.

"I can make it there on my own, Shane"}
...
{"text": ""Only one source I know of that would be likely to cough up enough money to finance a phony sleep research facility and pay people big bucks to solve crimes in their dreams," Farrell concluded dryly.

"What can I say?" Ellis unfolded his arms and widened his hands. "Your tax dollars at work."

Before Farrell could respond, Leila's voice rose from inside the house.

"No insurance?" she wailed. "What do you mean you don't have any insurance"}
{"text": "Helen's heart broke a little in the face of Miss Mabel's selfless courage. She thought that because she was old, her life was of less value than the others'. For all Helen knew, Miss Mabel had a lot more years to live than she did. "Not going to happen," replied Helen"}
{"text": "Preston had been the last person to wear those chains, and I knew what I'd see and feel if they were slipped onto my skin-the Reaper's unending hatred of me. I'd felt enough of that emotion already in the amphitheater. I didn't want to feel anymore.

"Don't put those on me," I whispered. "Please."

Sergei looked at me, surprised by my low, raspy please, but he put down the chains"}

模型格式轉換

為了避免在模型并行時，拆分模型的額外工作，FasterTransformer 提供了一個工具，用于將模型從不同格式拆分和轉換為 FasterTransformer 二進制文件格式；然后， FasterTransformer 可以直接以二進制格式加載模型。

將Huggingface Transformer模型權重文件格式轉換成FasterTransformer格式。

cd /workspace/code/FasterTransformer

python examples/pytorch/gpt/utils/huggingface_bloom_convert.py 
    --input-dir /workspace/model/bloomz-560m 
    --output-dir /workspace/model/bloomz-560m-convert 
    --data-type fp16 
    -tp 1 -v

轉換過程：

python examples/pytorch/gpt/utils/huggingface_bloom_convert.py 
>     --input-dir /workspace/model/bloomz-560m 
>     --output-dir /workspace/model/bloomz-560m-convert 
>     --data-type fp16 
>     -tp 1 -v

======================= Arguments =======================
 - input_dir...........: /workspace/model/bloomz-560m
 - output_dir..........: /workspace/model/bloomz-560m-convert
 - tensor_para_size....: 1
 - data_type...........: fp16
 - processes...........: 1
 - verbose.............: True
 - by_shard............: False
=========================================================
loading from pytorch bin format
model file num: 1
 - model.wte.......................................: shape (250880, 1024)     | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.wte.bin
 - model.pre_decoder_layernorm.weight..............: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.pre_decoder_layernorm.weight.bin
 - model.pre_decoder_layernorm.bias................: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.pre_decoder_layernorm.bias.bin
 - model.layers.0.input_layernorm.weight...........: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.input_layernorm.weight.bin
 - model.layers.0.input_layernorm.bias.............: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.input_layernorm.bias.bin
 - model.layers.0.attention.query_key_value.weight.: shape (1024, 3, 1024)  s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.attention.query_key_value.weight.0.bin (0/1)
 - model.layers.0.attention.query_key_value.bias...: shape (3, 1024)        s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.attention.query_key_value.bias.0.bin (0/1)
 - model.layers.0.attention.dense.weight...........: shape (1024, 1024)     s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.attention.dense.weight.0.bin (0/1)
 - model.layers.0.attention.dense.bias.............: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.attention.dense.bias.bin
 - model.layers.0.post_attention_layernorm.weight..: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.post_attention_layernorm.weight.bin
 - model.layers.0.post_attention_layernorm.bias....: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.post_attention_layernorm.bias.bin
 - model.layers.0.mlp.dense_h_to_4h.weight.........: shape (1024, 4096)     s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.mlp.dense_h_to_4h.weight.0.bin (0/1)
 - model.layers.0.mlp.dense_h_to_4h.bias...........: shape (4096,)          s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.0.mlp.dense_h_to_4h.bias.0.bin (0/1)
...
rs.22.mlp.dense_4h_to_h.bias.bin
 - model.layers.23.input_layernorm.weight..........: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.input_layernorm.weight.bin
 - model.layers.23.input_layernorm.bias............: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.input_layernorm.bias.bin
 - model.layers.23.attention.query_key_value.weight: shape (1024, 3, 1024)  s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.attention.query_key_value.weight.0.bin (0/1)
 - model.layers.23.attention.query_key_value.bias..: shape (3, 1024)        s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.attention.query_key_value.bias.0.bin (0/1)
 - model.layers.23.attention.dense.weight..........: shape (1024, 1024)     s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.attention.dense.weight.0.bin (0/1)
 - model.layers.23.attention.dense.bias............: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.attention.dense.bias.bin
 - model.layers.23.post_attention_layernorm.weight.: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.post_attention_layernorm.weight.bin
 - model.layers.23.post_attention_layernorm.bias...: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.post_attention_layernorm.bias.bin
 - model.layers.23.mlp.dense_h_to_4h.weight........: shape (1024, 4096)     s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.mlp.dense_h_to_4h.weight.0.bin (0/1)
 - model.layers.23.mlp.dense_h_to_4h.bias..........: shape (4096,)          s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.mlp.dense_h_to_4h.bias.0.bin (0/1)
 - model.layers.23.mlp.dense_4h_to_h.weight........: shape (4096, 1024)     s | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.mlp.dense_4h_to_h.weight.0.bin (0/1)
 - model.layers.23.mlp.dense_4h_to_h.bias..........: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.layers.23.mlp.dense_4h_to_h.bias.bin
 - model.final_layernorm.weight....................: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.final_layernorm.weight.bin
 - model.final_layernorm.bias......................: shape (1024,)            | saved at /workspace/model/bloomz-560m-convert/1-gpu/model.final_layernorm.bias.bin
Checkpoint conversion (HF >> FT) has done (elapsed time: 17.07 sec)

轉換成FasterTransformer格式后的文件如下所示：

> tree bloomz-560m-convert/
bloomz-560m-convert/
└── 1-gpu
    ├── config.ini
    ├── model.final_layernorm.bias.bin
    ├── model.final_layernorm.weight.bin
    ├── model.layers.0.attention.dense.bias.bin
    ├── model.layers.0.attention.dense.weight.0.bin
    ├── model.layers.0.attention.query_key_value.bias.0.bin
    ├── model.layers.0.attention.query_key_value.weight.0.bin
    ├── model.layers.0.input_layernorm.bias.bin
    ├── model.layers.0.input_layernorm.weight.bin
    ├── model.layers.0.mlp.dense_4h_to_h.bias.bin
    ├── model.layers.0.mlp.dense_4h_to_h.weight.0.bin
    ├── model.layers.0.mlp.dense_h_to_4h.bias.0.bin
    ├── model.layers.0.mlp.dense_h_to_4h.weight.0.bin
    ├── model.layers.0.post_attention_layernorm.bias.bin
    ├── model.layers.0.post_attention_layernorm.weight.bin
    ├── model.layers.1.attention.dense.bias.bin
    ...
    ├── model.layers.8.post_attention_layernorm.weight.bin
    ├── model.layers.9.attention.dense.bias.bin
    ├── model.layers.9.attention.dense.weight.0.bin
    ├── model.layers.9.attention.query_key_value.bias.0.bin
    ├── model.layers.9.attention.query_key_value.weight.0.bin
    ├── model.layers.9.input_layernorm.bias.bin
    ├── model.layers.9.input_layernorm.weight.bin
    ├── model.layers.9.mlp.dense_4h_to_h.bias.bin
    ├── model.layers.9.mlp.dense_4h_to_h.weight.0.bin
    ├── model.layers.9.mlp.dense_h_to_4h.bias.0.bin
    ├── model.layers.9.mlp.dense_h_to_4h.weight.0.bin
    ├── model.layers.9.post_attention_layernorm.bias.bin
    ├── model.layers.9.post_attention_layernorm.weight.bin
    ├── model.pre_decoder_layernorm.bias.bin
    ├── model.pre_decoder_layernorm.weight.bin
    └── model.wte.bin

模型基準測試

下面使用官方提供的樣例進行基準測試對比下Huggingface Transformers和FasterTransformer的響應時長。

Huggingface Transformers基準測試

運行命令：

# Run HF benchmark
CUDA_VISIBLE_DEVICES=1 python examples/pytorch/gpt/bloom_lambada.py 
    --tokenizer-path /workspace/model/bloomz-560m 
    --dataset-path /workspace/data/lambada_test.jsonl 
    --lib-path bulid/lib/libth_transformer.so 
    --test-hf 
    --show-progress

運行過程：

python examples/pytorch/gpt/bloom_lambada.py 
>     --tokenizer-path /workspace/model/bloomz-560m 
>     --dataset-path /workspace/data/lambada_test.jsonl 
>     --lib-path bulid/lib/libth_transformer.so 
>     --test-hf 
>     --show-progress

=================== Arguments ===================
 - num_heads................: None
 - size_per_head............: None
 - inter_size...............: None
 - num_layers...............: None
 - vocab_size...............: None
 - tensor_para_size.........: 1
 - pipeline_para_size.......: 1
 - remove_padding...........: True
 - shared_contexts_ratio....: 1.0
 - batch_size...............: 8
 - output_length............: 32
 - beam_width...............: 1
 - top_k....................: 1
 - top_p....................: 1.0
 - temperature..............: 1.0
 - len_penalty..............: 0.0
 - beam_search_diversity_rate: 0.0
 - start_id.................: 0
 - end_id...................: 2
 - repetition_penalty.......: 1.0
 - random_seed..............: None
 - return_cum_log_probs.....: 0
 - checkpoint_path..........: None
 - dataset_path.............: /workspace/data/lambada_test.jsonl
 - output_path..............: None
 - tokenizer_path...........: /workspace/model/bloomz-560m
 - lib_path.................: bulid/lib/libth_transformer.so
 - test_hf..................: True
 - acc_threshold............: None
 - show_progress............: True
 - inference_data_type......: None
 - weights_data_type........: None
 - int8_mode................: 0
=================================================
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 645/645 [02:33<00:00,  4.21it/s]
Accuracy: 39.4722% (2034/5153) (elapsed time: 146.7230 sec)

FasterTransformer基準測試

運行命令：

# Run FT benchmark
python examples/pytorch/gpt/bloom_lambada.py 
    --checkpoint-path /workspace/model/bloomz-560m-convert/1-gpu 
    --tokenizer-path /workspace/model/bloomz-560m 
    --dataset-path /workspace/data/lambada_test.jsonl 
    --lib-path build/lib/libth_transformer.so 
    --show-progress

注：還可添加--data-type fp16以半精度方式加載模型，以減少模型對于顯存的消耗。

運行過程：

python examples/pytorch/gpt/bloom_lambada.py 
>     --checkpoint-path /workspace/model/bloomz-560m-convert/1-gpu 
>     --tokenizer-path /workspace/model/bloomz-560m 
>     --dataset-path /workspace/data/lambada_test.jsonl 
>     --lib-path build/lib/libth_transformer.so 
>     --show-progress

=================== Arguments ===================
 - num_heads................: None
 - size_per_head............: None
 - inter_size...............: None
 - num_layers...............: None
 - vocab_size...............: None
 - tensor_para_size.........: 1
 - pipeline_para_size.......: 1
 - remove_padding...........: True
 - shared_contexts_ratio....: 1.0
 - batch_size...............: 8
 - output_length............: 32
 - beam_width...............: 1
 - top_k....................: 1
 - top_p....................: 1.0
 - temperature..............: 1.0
 - len_penalty..............: 0.0
 - beam_search_diversity_rate: 0.0
 - start_id.................: 0
 - end_id...................: 2
 - repetition_penalty.......: 1.0
 - random_seed..............: None
 - return_cum_log_probs.....: 0
 - checkpoint_path..........: /workspace/model/bloomz-560m-convert/1-gpu
 - dataset_path.............: /workspace/data/lambada_test.jsonl
 - output_path..............: None
 - tokenizer_path...........: /workspace/model/bloomz-560m
 - lib_path.................: build/lib/libth_transformer.so
 - test_hf..................: False
 - acc_threshold............: None
 - show_progress............: True
 - inference_data_type......: None
 - weights_data_type........: None
 - int8_mode................: 0
=================================================
[FT][INFO] Load BLOOM model
 - head_num.................: 16
 - size_per_head............: 64
 - layer_num................: 24
 - tensor_para_size.........: 1
 - vocab_size...............: 250880
 - start_id.................: 1
 - end_id...................: 2
 - weights_data_type........: fp16
 - layernorm_eps............: 1e-05
 - inference_data_type......: fp16
 - lib_path.................: build/lib/libth_transformer.so
 - pipeline_para_size.......: 1
 - shared_contexts_ratio....: 1.0
 - int8_mode................: 0
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA A800 80GB PCIe
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 645/645 [00:18<00:00, 34.58it/s]
Accuracy: 39.4722% (2034/5153) (elapsed time: 13.0032 sec)

對比Huggingface Transformers和FasterTransformer

HF:    Accuracy: 39.4722% (2034/5153) (elapsed time: 146.7230 sec)
FT:    Accuracy: 39.4722% (2034/5153) (elapsed time: 13.0032 sec)

可以看到它們的準確率一致，但是FasterTransformer比Huggingface Transformers的推理速度更加快速。

模型并行推理（多卡）

對于像GPT3（175B）、OPT-175B這樣的大模型，單卡無法加載整個模型，因此，我們需要以分布式（模型并行）方式進行大模型推理。模型并行推理有兩種方式：張量并行和流水線并行，前面已經進行過相應的說明，這里不再贅述。

張量并行

模型轉換

如果想使用張量并行 (TP) 技術將模型拆分多個GPU進行推理，可參考如下命令將模型轉換到2個GPU上進行推理。

python examples/pytorch/gpt/utils/huggingface_bloom_convert.py 
--input-dir /workspace/model/bloomz-560m 
--output-dir /workspace/model/bloomz-560m-convert 
--data-type fp16 
-tp 2 -v

轉換成張量并行度為2的FasterTransformer格式后的文件如下所示：

tree /workspace/model/bloomz-560m-convert/2-gpu
/workspace/model/bloomz-560m-convert/2-gpu
├── config.ini
├── model.final_layernorm.bias.bin
├── model.final_layernorm.weight.bin
├── model.layers.0.attention.dense.bias.bin
├── model.layers.0.attention.dense.weight.0.bin
├── model.layers.0.attention.dense.weight.1.bin
├── model.layers.0.attention.query_key_value.bias.0.bin
├── model.layers.0.attention.query_key_value.bias.1.bin
├── model.layers.0.attention.query_key_value.weight.0.bin
├── model.layers.0.attention.query_key_value.weight.1.bin
├── model.layers.0.input_layernorm.bias.bin
├── model.layers.0.input_layernorm.weight.bin
├── model.layers.0.mlp.dense_4h_to_h.bias.bin
├── model.layers.0.mlp.dense_4h_to_h.weight.0.bin
├── model.layers.0.mlp.dense_4h_to_h.weight.1.bin
├── model.layers.0.mlp.dense_h_to_4h.bias.0.bin
├── model.layers.0.mlp.dense_h_to_4h.bias.1.bin
├── model.layers.0.mlp.dense_h_to_4h.weight.0.bin
├── model.layers.0.mlp.dense_h_to_4h.weight.1.bin
├── model.layers.0.post_attention_layernorm.bias.bin
├── model.layers.0.post_attention_layernorm.weight.bin
...
├── model.layers.9.attention.dense.bias.bin
├── model.layers.9.attention.dense.weight.0.bin
├── model.layers.9.attention.dense.weight.1.bin
├── model.layers.9.attention.query_key_value.bias.0.bin
├── model.layers.9.attention.query_key_value.bias.1.bin
├── model.layers.9.attention.query_key_value.weight.0.bin
├── model.layers.9.attention.query_key_value.weight.1.bin
├── model.layers.9.input_layernorm.bias.bin
├── model.layers.9.input_layernorm.weight.bin
├── model.layers.9.mlp.dense_4h_to_h.bias.bin
├── model.layers.9.mlp.dense_4h_to_h.weight.0.bin
├── model.layers.9.mlp.dense_4h_to_h.weight.1.bin
├── model.layers.9.mlp.dense_h_to_4h.bias.0.bin
├── model.layers.9.mlp.dense_h_to_4h.bias.1.bin
├── model.layers.9.mlp.dense_h_to_4h.weight.0.bin
├── model.layers.9.mlp.dense_h_to_4h.weight.1.bin
├── model.layers.9.post_attention_layernorm.bias.bin
├── model.layers.9.post_attention_layernorm.weight.bin
├── model.pre_decoder_layernorm.bias.bin
├── model.pre_decoder_layernorm.weight.bin
└── model.wte.bin

0 directories, 438 files

張量并行模型推理

運行命令：

mpirun -n 2 --allow-run-as-root python examples/pytorch/gpt/bloom_lambada.py 
    --checkpoint-path /workspace/model/bloomz-560m-convert/2-gpu 
    --tokenizer-path /workspace/model/bloomz-560m 
    --dataset-path /workspace/data/lambada_test.jsonl 
    --lib-path build/lib/libth_transformer.so 
    --tensor-para-size 2 
    --pipeline-para-size 1 
    --show-progress

運行過程：

mpirun -n 2 --allow-run-as-root python examples/pytorch/gpt/bloom_lambada.py 
>     --checkpoint-path /workspace/model/bloomz-560m-convert/2-gpu 
>     --tokenizer-path /workspace/model/bloomz-560m 
>     --dataset-path /workspace/data/lambada_test.jsonl 
>     --lib-path build/lib/libth_transformer.so 
>     --tensor-para-size 2 
>     --pipeline-para-size 1 
>     --show-progress

=================== Arguments ===================
 - num_heads................: None
 - size_per_head............: None
 - inter_size...............: None
 - num_layers...............: None
 - vocab_size...............: None
 - tensor_para_size.........: 2
 - pipeline_para_size.......: 1
 - remove_padding...........: True
 - shared_contexts_ratio....: 1.0
 - batch_size...............: 8
 - output_length............: 32
 - beam_width...............: 1
 - top_k....................: 1
 - top_p....................: 1.0
 - temperature..............: 1.0
 - len_penalty..............: 0.0
 - beam_search_diversity_rate: 0.0
 - start_id.................: 0
 - end_id...................: 2
 - repetition_penalty.......: 1.0
 - random_seed..............: None
 - return_cum_log_probs.....: 0
 - checkpoint_path..........: /workspace/model/bloomz-560m-convert/2-gpu
 - dataset_path.............: /workspace/data/lambada_test.jsonl
 - output_path..............: None
 - tokenizer_path...........: /workspace/model/bloomz-560m
 - lib_path.................: build/lib/libth_transformer.so
 - test_hf..................: False
 - acc_threshold............: None
 - show_progress............: True
 - inference_data_type......: None
 - weights_data_type........: None
 - int8_mode................: 0
=================================================

=================== Arguments ===================
 - num_heads................: None
 - size_per_head............: None
 - inter_size...............: None
 - num_layers...............: None
 - vocab_size...............: None
 - tensor_para_size.........: 2
 - pipeline_para_size.......: 1
 - remove_padding...........: True
 - shared_contexts_ratio....: 1.0
 - batch_size...............: 8
 - output_length............: 32
 - beam_width...............: 1
 - top_k....................: 1
 - top_p....................: 1.0
 - temperature..............: 1.0
 - len_penalty..............: 0.0
 - beam_search_diversity_rate: 0.0
 - start_id.................: 0
 - end_id...................: 2
 - repetition_penalty.......: 1.0
 - random_seed..............: None
 - return_cum_log_probs.....: 0
 - checkpoint_path..........: /workspace/model/bloomz-560m-convert/2-gpu
 - dataset_path.............: /workspace/data/lambada_test.jsonl
 - output_path..............: None
 - tokenizer_path...........: /workspace/model/bloomz-560m
 - lib_path.................: build/lib/libth_transformer.so
 - test_hf..................: False
 - acc_threshold............: None
 - show_progress............: True
 - inference_data_type......: None
 - weights_data_type........: None
 - int8_mode................: 0
=================================================
[FT][INFO] Load BLOOM model
 - head_num.................: 16
 - size_per_head............: 64
 - layer_num................: 24
 - tensor_para_size.........: 2
 - vocab_size...............: 250880
 - start_id.................: 1
 - end_id...................: 2
 - weights_data_type........: fp16
 - layernorm_eps............: 1e-05
 - inference_data_type......: fp16
 - lib_path.................: build/lib/libth_transformer.so
 - pipeline_para_size.......: 1
 - shared_contexts_ratio....: 1.0
 - int8_mode................: 0
[FT][INFO] Load BLOOM model
 - head_num.................: 16
 - size_per_head............: 64
 - layer_num................: 24
 - tensor_para_size.........: 2
 - vocab_size...............: 250880
 - start_id.................: 1
 - end_id...................: 2
 - weights_data_type........: fp16
 - layernorm_eps............: 1e-05
 - inference_data_type......: fp16
 - lib_path.................: build/lib/libth_transformer.so
 - pipeline_para_size.......: 1
 - shared_contexts_ratio....: 1.0
 - int8_mode................: 0
world_size: 2
world_size: 2
[WARNING] gemm_config.in is not found; using default GEMM algo
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][INFO] NCCL initialized rank=0 world_size=2 tensor_para=NcclParam[rank=0, world_size=2, nccl_comm=0x5556305627d0] pipeline_para=NcclParam[rank=0, world_size=1, nccl_comm=0x5556305d5d20]
[FT][INFO] Device NVIDIA A800 80GB PCIe
[FT][INFO] NCCL initialized rank=1 world_size=2 tensor_para=NcclParam[rank=1, world_size=2, nccl_comm=0x55b9600a9ca0] pipeline_para=NcclParam[rank=0, world_size=1, nccl_comm=0x55b96011cff0]
[FT][INFO] Device NVIDIA A800 80GB PCIe
/workspace/code/FasterTransformer/examples/pytorch/gpt/utils/gpt.py SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(self.pre_embed_idx < self.post_embed_idx, "Pre decoder embedding index should be lower than post decoder embedding index.")
  0%|          | 0/645 [00:00

	

	流水線并行

	模型轉換

	如果僅使用流水線并行，不使用張量并行，則tp設置為1即可，如果需要同時進行張量并行和流水線并行，則需要將tp設置成張量并行度大小。具體命令參考前面的模型轉換部分。

	流水線并行模型推理

	運行命令：

	
CUDA_VISIBLE_DEVICES=1,2 mpirun -n 2 --allow-run-as-root python examples/pytorch/gpt/bloom_lambada.py 
    --checkpoint-path /workspace/model/bloomz-560m-convert/1-gpu 
    --tokenizer-path /workspace/model/bloomz-560m 
    --dataset-path /workspace/data/lambada_test.jsonl 
    --lib-path build/lib/libth_transformer.so 
    --tensor-para-size 1 
    --pipeline-para-size 2 
    --batch-size 1 
    --show-progress


	

	運行過程：

	
CUDA_VISIBLE_DEVICES=1,2 mpirun -n 2 --allow-run-as-root python examples/pytorch/gpt/bloom_lambada.py       
>     --checkpoint-path /workspace/model/bloomz-560m-convert/1-gpu                                       
>     --tokenizer-path /workspace/model/bloomz-560m 
>     --dataset-path /workspace/data/lambada_test.jsonl 
>     --lib-path build/lib/libth_transformer.so 
>     --tensor-para-size 1 
>     --pipeline-para-size 2 
>     --batch-size 1 
>     --show-progress

=================== Arguments ===================
 - num_heads................: None
 - size_per_head............: None
 - inter_size...............: None
 - num_layers...............: None
 - vocab_size...............: None
 - tensor_para_size.........: 1
 - pipeline_para_size.......: 2
 - remove_padding...........: True
 - shared_contexts_ratio....: 1.0
 - batch_size...............: 1
 - output_length............: 32
 - beam_width...............: 1
 - top_k....................: 1
 - top_p....................: 1.0
 - temperature..............: 1.0
 - len_penalty..............: 0.0
 - beam_search_diversity_rate: 0.0
 - start_id.................: 0
 - end_id...................: 2
 - repetition_penalty.......: 1.0
 - random_seed..............: None
 - return_cum_log_probs.....: 0
 - checkpoint_path..........: /workspace/model/bloomz-560m-convert/1-gpu
 - dataset_path.............: /workspace/data/lambada_test.jsonl
 - output_path..............: None
 - tokenizer_path...........: /workspace/model/bloomz-560m
 - lib_path.................: build/lib/libth_transformer.so
 - test_hf..................: False
 - acc_threshold............: None
 - show_progress............: True
 - inference_data_type......: None
 - weights_data_type........: None
 - int8_mode................: 0
=================================================

=================== Arguments ===================
 - num_heads................: None
 - size_per_head............: None
 - inter_size...............: None
 - num_layers...............: None
 - vocab_size...............: None
 - tensor_para_size.........: 1
 - pipeline_para_size.......: 2
 - remove_padding...........: True
 - shared_contexts_ratio....: 1.0
 - batch_size...............: 1
 - output_length............: 32
 - beam_width...............: 1
 - top_k....................: 1
 - top_p....................: 1.0
 - temperature..............: 1.0
 - len_penalty..............: 0.0
 - beam_search_diversity_rate: 0.0
 - start_id.................: 0
 - end_id...................: 2
 - repetition_penalty.......: 1.0
 - random_seed..............: None
 - return_cum_log_probs.....: 0
 - checkpoint_path..........: /workspace/model/bloomz-560m-convert/1-gpu
 - dataset_path.............: /workspace/data/lambada_test.jsonl
 - output_path..............: None
 - tokenizer_path...........: /workspace/model/bloomz-560m
 - lib_path.................: build/lib/libth_transformer.so
 - test_hf..................: False
 - acc_threshold............: None
 - show_progress............: True
 - inference_data_type......: None
 - weights_data_type........: None
 - int8_mode................: 0
=================================================
[FT][INFO] Load BLOOM model
 - head_num.................: 16
 - size_per_head............: 64
 - layer_num................: 24
 - tensor_para_size.........: 1
 - vocab_size...............: 250880
 - start_id.................: 1
 - end_id...................: 2
 - weights_data_type........: fp16
 - layernorm_eps............: 1e-05
 - inference_data_type......: fp16
 - lib_path.................: build/lib/libth_transformer.so
 - pipeline_para_size.......: 2
 - shared_contexts_ratio....: 1.0
 - int8_mode................: 0
[FT][INFO] Load BLOOM model
 - head_num.................: 16
 - size_per_head............: 64
 - layer_num................: 24
 - tensor_para_size.........: 1
 - vocab_size...............: 250880
 - start_id.................: 1
 - end_id...................: 2
 - weights_data_type........: fp16
 - layernorm_eps............: 1e-05
 - inference_data_type......: fp16
 - lib_path.................: build/lib/libth_transformer.so
 - pipeline_para_size.......: 2
 - shared_contexts_ratio....: 1.0
 - int8_mode................: 0
world_size: 2
world_size: 2
[WARNING] gemm_config.in is not found; using default GEMM algo
[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][INFO] NCCL initialized rank=0 world_size=2 tensor_para=NcclParam[rank=0, world_size=1, nccl_comm=0x5557a53dc1b0] pipeline_para=NcclParam[rank=0, world_size=2, nccl_comm=0x5557a5444df0]
[FT][INFO] NCCL initialized rank=1 world_size=2 tensor_para=NcclParam[rank=0, world_size=1, nccl_comm=0x560cf3452820] pipeline_para=NcclParam[rank=1, world_size=2, nccl_comm=0x560cf34bb190]
[FT][INFO] Device NVIDIA A800 80GB PCIe
[FT][INFO] Device NVIDIA A800 80GB PCIe
100%|██████████| 5153/5153 [01:51<00:00, 46.12it/s] current process id: 47861   Accuracy: 39.4527% (2033/5153) (elapsed time: 102.1145 sec)
current process id: 47862   Accuracy: 39.4527% (2033/5153) (elapsed time: 102.3391 sec)


	

	單卡、流水線并行、張量并行對比

	下面在BatchSize為1的情況下，對單卡、張量并行、流水線并行進行了簡單的測試，僅供參考（由于測試時，有其他訓練任務也在運行，可能對結果會產生干擾）。

	TP=1、PP=1、BZ=1：

	
累積響應時長：
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5153/5153 [02:21<00:00, 36.31it/s]
current process id: 47645   Accuracy: 39.4527% (2033/5153) (elapsed time: 132.2274 sec)

顯存占用：
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      8356      C   python                           1740MiB |
+-----------------------------------------------------------------------------+


	

	TP=2、PP=1、BZ=1：

	
累積響應時長：
100%|██████████| 5153/5153 [00:35<00:00, 144.80it/s]current process id: 49111   Accuracy: 39.4916% (2035/5153) (elapsed time: 26.1384 sec)
current process id: 49112   Accuracy: 39.4916% (2035/5153) (elapsed time: 26.5110 sec)


顯存占用：
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    1   N/A  N/A     41339      C   python                           1692MiB |
|    2   N/A  N/A     41340      C   python                           1692MiB |
+-----------------------------------------------------------------------------+


	

	TP=1、PP=2、BZ=1：

	
累積響應時長：
100%|██████████| 5153/5153 [00:33<00:00, 153.92it/s]current process id: 48755   Accuracy: 39.4527% (2033/5153) (elapsed time: 24.1695 sec)
current process id: 48754   Accuracy: 39.4527% (2033/5153) (elapsed time: 24.4391 sec)


顯存占用：
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    1   N/A  N/A      4001      C   python                           1952MiB |
|    2   N/A  N/A      4002      C   python                           1952MiB |
+-----------------------------------------------------------------------------+


	

	TP=1、PP=3、BZ=1：

	
累積響應時長：
100%|██████████| 5153/5153 [00:33<00:00, 152.46it/s]current process id: 48220   Accuracy: 0.0000% (0/5153) (elapsed time: 24.9212 sec)
100%|██████████| 5153/5153 [00:33<00:00, 153.63it/s]current process id: 48219   Accuracy: 39.4527% (2033/5153) (elapsed time: 24.9767 sec)
current process id: 48221   Accuracy: 39.4527% (2033/5153) (elapsed time: 24.3489 sec)

顯存占用：
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     57588      C   python                           1420MiB |
|    1   N/A  N/A     57589      C   python                           1468MiB |
|    2   N/A  N/A     57590      C   python                           1468MiB |
+-----------------------------------------------------------------------------+


	

	結語

	本文給大家簡要介紹了FasterTransformer的基本概念以及如何使用FasterTransformer進行單機及分布式模型推理，希望能夠幫助大家快速了解FasterTransformer。

	




	審核編輯：劉清