Methods and tools for efficient training on a single GPU
<aside>
Mixed precision training is a technique that aims to optimize the computational efficiency of training models by utilizing lower-precision numerical formats for certain variables.
→ 혼합 정밀도 학습은 일부 변수에 대해 더 낮은 정밀도의 숫자 형식을 사용함으로써 모델 학습의 계산 효율성을 최적화하는 것을 목표로 하는 기술
→ 다시 말해, 일부 숫자의 표현 방식을 단순화해서 모델을 더 빠르고 효율적으로 학습시키는 것
Traditionally, most models use 32-bit floating point precision (fp32 or float32) to represent and process variables.
→ 대부분 딥러닝 모델들은 32비트 사용
However, not all variables require this high precision level to achieve accurate results.
→ 모든 변수들이 이 정도를 필요로 하지 않음
→ 다시 말해, 일부 변수들은 더 낮은 정밀도로도 충분한 결과 얻을 수 있음
By reducing the precision of certain variables to lower numerical formats like 16-bit floating point (fp16 or float16), we can speed up the computations.
→ 일부 변수들은 정밀도를 낮추면 계산량이 줄어들어 연산 속도 빨라짐 (같은 말)
Because in this approach some computations are performed in half-precision, while some are still in full precision, the approach is called mixed precision training.
→ 일부 연산은 절반 정밀도 + 일부 연산은 전체 정밀도, 따라서 혼합 정밀도
Most commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures (such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check out the NVIDIA Blog to learn more about the differences between these data types.
→ 일반적으로 혼합 정밀도 학습에는 fp16을 사용하지만, Ampere GPU에서는 bf16과 tf32도 지원함
fp16 16비트 크기의 부동소수점(floating point) 데이터 형식bf16tf32
</aside><aside>
Accelerating AI Training with NVIDIA TF32 Tensor Cores | NVIDIA Technical Blog
NVIDIA Ampere GPU architecture introduced the third generation of Tensor Cores, with the new TensorFloat32 (TF32) mode for accelerating FP32 convolutions and matrix multiplications.
→ Ampere GPU에서 FP32 연산을 더 빠르게 하기 위해 TF32 모드를 추가함
TF32 mode is the default option for AI training with 32-bit variables on Ampere GPU architecture.
→ Ampere GPU에서는 AI 훈련 시 TF32가 기본적으로 활성화
It brings Tensor Core acceleration to single-precision DL workloads, without needing any changes to model scripts.
→ TF32를 쓰면 코드 수정 없이 FP32 속도를 높일 수 있음
Mixed-precision training with a native 16-bit format (FP16/BF16) is still the fastest option, requiring just a few lines of code in model scripts.
→ TF32는 편하지만, 최고의 속도를 원하면 FP16/BF16을 사용해야 함
Table 1 shows the math throughput of A100 Tensor Cores, compared to FP32 CUDA cores.
It’s also worth pointing out that for single-precision training, the A100 delivers 10x higher math throughput than the previous generation training GPU, V100. → A100의 Tensor Core는 기존 V100보다 10배 빠름
Internals
TF32 is a new compute mode added to Tensor Cores in the Ampere generation of GPU architecture.
→ TF32는 Ampere부터 Tensor Core에서 지원되는 새로운 연산 방식
Dot product computation, which forms the building block for both matrix multiplies and convolutions, rounds FP32 inputs to TF32, computes the products without loss of precision, then accumulates those products into an FP32 output (Figure 1).
→ FP32 입력을 TF32로 변환한 후 곱셈을 수행하고, 정밀도 손실 없이 FP32 출력으로 결과를 누적한다
TF32 is only exposed as a Tensor Core operation mode, not a type.
→ TF32는 연산 방식, 별도의 데이터 타입이 아님
All storage in memory and other operations remain completely in FP32, only convolutions and matrix-multiplications convert their inputs to TF32 right before multiplication.
→ 메모리에 저장되는 값과 다른 연산들은 여전히 FP32로 유지되며, 합성곱과 행렬 곱셈 연산에서만 TF32로 변환되어 연산
In contrast, 16-bit types provide storage, various math operators, and so on.
Numerics
Figure 2 shows the various precision options.
TF32 mode in the Ampere generation of GPUs adopts 8 exponent bits, 10 bits of mantissa, and one sign bit.
As a result, it covers the same range of values as FP32.
TF32 also maintains more precision than BF16 and the same amount as FP16.
→ TF32는 FP16과 같은 정밀도를 가지지만, BF16보다는 정밀도가 더 높음
The precision for TF32 remains the only difference from FP32 and has been shown to have more than sufficient margin for AI workloads with extensive studies.
→ TF32는 FP32보다 정밀도가 낮지만, AI 훈련에는 충분히 정확
We validated single-precision training in TF32 mode on a wide breadth of AI networks across a variety of applications from computer vision to natural language processing to recommender systems.
→ TF32가 다양한 AI 작업에서도 충분한 성능을 낸다는 걸 검증했음
All the dozens of considered DL workloads match FP32 accuracy, loss values, and training behavior, with no changes to hyperparameters or training scripts.
Figure 3 shows a sampling of networks trained.
All workloads use identical hyperparameters for training in FP32 and TF32 modes, all differences in accuracy are within respective bounds of run-to-run variation (different random seeds, and so on) for each network.
→ TF32를 사용해도 FP32와 동일한 결과를 얻을 수 있었음
Figure 4 shows the training curves for a few select models on a sampling of networks trained.
Training speedups
As shown earlier, TF32 math mode, the default for single-precision DL training on the Ampere generation of GPUs, achieves the same accuracy as FP32 training, requires no changes to hyperparameters for training scripts, and provides an out-of-the-box 10X faster “tensor math” (convolutions and matrix multiplies) than single-precision math on Volta GPUs.
However, speedups observed for networks in practice vary, since all memory accesses remain FP32 and TF32 mode doesn’t affect layers that are not convolutions or matrix multiplies.
→ TF32가 빠르긴 하지만, 모델의 모든 연산을 최적화하는 건 아님
→ TF32는 행렬 연산(Matrix Multiply)과 합성곱(Convolution)에서만 적용됨
→ 모델에 따라 속도 향상 폭이 달라질 수 있음!
Figure 5 shows that speedups of 2-6x are observed in practice for single-precision training of various workloads when moving from V100 to A100.
Furthermore, switching to mixed precision with FP16 gives a further speedup of up to ~2x, as 16-bit Tensor Cores are 2x faster than TF32 mode and memory traffic is reduced by accessing half the bytes.
→ TF32도 빠르지만, FP16을 사용하면 메모리 사용량까지 줄어들어서 더 빠르게 학습할 수 있음
Thus, TF32 is a great starting point for models trained in FP32 on Volta or other processors, while mixed-precision training is the option to maximize training speed on A100.
→ TF32는 기존 FP32 코드 그대로 사용할 수 있는 장점이 있고, 더 빠른 성능을 원하면 혼합 정밀도를 사용하는 것을 추천
Conclusion
<aside>
The main advantage of mixed precision training comes from saving the activations in half precision (fp16).
→ 혼합 정밀도 학습의 가장 큰 장점은 활성값을 16비트(fp16)로 저장할 수 있다는 것
Although the gradients are also computed in half precision they are converted back to full precision for the optimization step so no memory is saved here.
→ 그래디언트도 절반 정밀도(fp16)로 계산되지만, 최적화 단계에서는 다시 전체 정밀도(fp32)로 변환되므로 여기서는 메모리가 절약되지 않음
While mixed precision training results in faster computations, it can also lead to more GPU memory being utilized, especially for small batch sizes.
→ 혼합 정밀도 학습은 연산 속도를 빠르게 해주지만, 배치 크기가 작은 경우 GPU 메모리 사용량이 증가할 수도 있다
This is because the model is now present on the GPU in both 16-bit and 32-bit precision (1.5x the original model on the GPU).
→ 모델이 GPU에서 16비트와 32비트 두 가지로 저장되기 때문에, 원래 모델보다 1.5배 더 많은 GPU 메모리를 사용하기 때문
To enable mixed precision training, set the fp16 flag to True:
training_args = TrainingArguments(
per_device_train_batch_size=4,
fp16=True,
**default_args)
If you prefer to use 🤗 Accelerate, find the 🤗 Accelerate example further in this guide.
</aside>
<aside>
</aside>
<aside>
</aside>