Accelerate documentation

๐Ÿค— accelerate์˜ Context Parallel

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v1.16.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

๐Ÿค— accelerate์˜ Context Parallel

์ด ๊ฐ€์ด๋“œ์—์„œ๋Š” ๐Ÿค—accelerate์—์„œ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ์‚ฌ์šฉํ•˜๋Š” ๊ธฐ๋ณธ ์‚ฌํ•ญ์„ ๋‹ค๋ฃน๋‹ˆ๋‹ค. ๋” ์ž์„ธํ•œ ๋‚ด์šฉ์„ ์•Œ๊ณ  ์‹ถ์€ ๋…์ž๋ฅผ ์œ„ํ•ด ํ›„๋ฐ˜๋ถ€ ์„น์…˜์—์„œ๋Š” ์ผ๋ถ€ ๊ธฐ์ˆ ์ ์ธ ์‚ฌํ•ญ๋„ ๋‹ค๋ฃน๋‹ˆ๋‹ค.

๋งค์šฐ ๊ด€๋ จ์ด ์žˆ๋Š” Sequence Parallelism ๊ฐ€์ด๋“œ๋„ ์ฐธ๊ณ ํ•˜์„ธ์š”.

์™œ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ์‚ฌ์šฉํ•ด์•ผ ํ•˜๋‚˜์š”?

๋Œ€ํ˜• ์–ธ์–ด ๋ชจ๋ธ, ๊ทธ๋ฆฌ๊ณ  ์ตœ๊ทผ์—๋Š” ์ถ”๋ก  ๋ชจ๋ธ์˜ ๋“ฑ์žฅ์œผ๋กœ ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ ๋น ๋ฅด๊ฒŒ ์ฆ๊ฐ€ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์— attention์˜ 2์ฐจ ๋ฉ”๋ชจ๋ฆฌ ๋ณต์žก๋„๊ฐ€ ๋”ํ•ด์ง€๋ฉด์„œ, ๊ธด ์‹œํ€€์Šค๋กœ ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋” ํšจ์œจ์ ์ธ ๋ฐฉ์‹์— ๋Œ€ํ•œ ํ•„์š”์„ฑ์ด ์ปค์กŒ์Šต๋‹ˆ๋‹ค. ์‹œํ€€์Šค ๊ธธ์ด๊ฐ€ 128k์ผ ๋•Œ, vanilla attention ๊ตฌํ˜„์„ ๊ฐ€์ •ํ•˜๋ฉด bf16 ์ •๋ฐ€๋„์—์„œ attention matrix์˜ ๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰์€ 128k * 128k * 2 bytes * num_heads = ~32 GB * num_heads์ž…๋‹ˆ๋‹ค. ๋ฌผ๋ก , ์ด๋Ÿฌํ•œ attention ๊ฐ€์ค‘์น˜๋ฅผ ๋ฌผ๋ฆฌ์ ์œผ๋กœ ์ƒ์„ฑํ•˜์ง€ ์•Š๋Š” flash attention๋‚˜ SDPA๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์ด ํฌ๊ฒŒ ์ค„์ง€๋งŒ, ๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰์˜ ์ฆ๊ฐ€ ํญ์€ ์—ฌ์ „ํžˆ ์ƒ๋‹นํ•ฉ๋‹ˆ๋‹ค.

Context parallelism์€ attention ๊ณ„์‚ฐ์˜ ์ž…๋ ฅ์„ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉํ•˜์—ฌ ์—ฌ๋Ÿฌ GPU์—์„œ attention์„ ๋ณ‘๋ ฌ๋กœ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๊ธด ์‹œํ€€์Šค๋ฅผ ๊ฐ€์ง„ ๋ชจ๋ธ์„ ํ•™์Šตํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์ž ์žฌ์ ์œผ๋กœ 1M ์ด์ƒ์˜ ์‹œํ€€์Šค ๊ธธ์ด๊นŒ์ง€ ํ™•์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ(context parallelism)์„ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฉ๋ฒ•

from accelerate.utils import ParallelismConfig, TorchContextParallelConfig

+ cp_config = TorchContextParallelConfig(
+       cp_comm_strategy="alltoall", # no need to use cp_config at all, if you want to use the default "allgather"
+ )

+ parallelism_config = ParallelismConfig(
+     cp_size=8,
+     cp_handler=cp_config,  # or just cp_size=8, if you want to use the default "allgather"
+ )

accelerator = Accelerator(
    ...,
    parallelism_config=parallelism_config,
)

๐Ÿค—accelerate์˜ ๋‹ค๋ฅธ ๋ชจ๋“  ๊ธฐ๋Šฅ๊ณผ ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ, accelerate launch์— ํ•ด๋‹น ํ”Œ๋ž˜๊ทธ๋ฅผ ์ „๋‹ฌํ•˜์—ฌ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ๋„ ํ™œ์„ฑํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฒฝ์šฐ์—๋„ ๋‹ค๋ฅด์ง€ ์•Š์Šต๋‹ˆ๋‹ค:

accelerate launch --parallelism-config-cp-size 8 --parallelism-config-cp-comm-strategy [allgather|alltoall] ...

accelerate config ๋ช…๋ น์—์„œ cp_size์™€ cp_comm_strategy๋ฅผ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ฐ’๋“ค์€ accelerate ๊ตฌ์„ฑ ํŒŒ์ผ์— ์ €์žฅ๋˜๋ฏ€๋กœ, ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์‹คํ–‰ํ•  ๋•Œ๋งˆ๋‹ค ๋งค๋ฒˆ ์ „๋‹ฌํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค.

Context parallelism์€ data parallelism, tensor parallelism, FSDP2์™€ ๊ฐ™์€ ๋‹ค๋ฅธ ๋ณ‘๋ ฌํ™” ์ „๋žต๊ณผ ํ˜ธํ™˜๋ฉ๋‹ˆ๋‹ค. --parallelism-config-dp-size 8 --parallelism-config-tp-size 2 --parallelism-config-cp-size 8์ฒ˜๋Ÿผ ๋ณ‘๋ ฌํ™” ํฌ๊ธฐ๋ฅผ ์›ํ•˜๋Š” ๊ฐ’์œผ๋กœ ์„ค์ •ํ•˜์—ฌ ๊ฐ„๋‹จํžˆ ์กฐํ•ฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜๋Š” ParallelismConfig ํด๋ž˜์Šค๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ”„๋กœ๊ทธ๋ž˜๋ฐ ๋ฐฉ์‹์œผ๋กœ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

Context parallelism์€ FSDP2์™€ ๊ธด๋ฐ€ํ•˜๊ฒŒ ๊ฒฐํ•ฉ๋˜์–ด ์žˆ์œผ๋ฉฐ, FSDP2 ์†Œ๊ฐœ์—์„œ ์ž์„ธํžˆ ์•Œ์•„๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ฆ‰, context parallelism์€ ํ”„๋กœ๊ทธ๋žจ์— FullyShardedDataParallelPlugin ๋˜๋Š” ๋ฒ„์ „์ด 2๋กœ ์„ค์ •๋œ --use-fsdp๋ฅผ ์‚ฌ์šฉํ•  ๋•Œ๋งŒ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค. FSDP2๋ฅผ ์‚ฌ์šฉํ•˜์ง€ ์•Š์œผ๋ฉด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

Context parallelism์€ SDPA์—์„œ๋งŒ ์ž‘๋™ํ•˜๋ฉฐ, mask๋‚˜ causal mask๊ฐ€ ์—†๋Š” ๊ฒฝ์šฐ์—๋งŒ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ฅผ ์ž๋™์œผ๋กœ ์ ์ ˆํ•˜๊ฒŒ ๊ฐ์ง€ํ•  ์ˆ˜ ์—†์œผ๋ฏ€๋กœ, mask๋‚˜ causal mask ์—†์ด SDPA๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธํ•˜๋Š” ๊ฒƒ์€ ์‚ฌ์šฉ์ž์˜ ์ฑ…์ž„์ž…๋‹ˆ๋‹ค. ๋‹ค๋ฅธ attention ๊ตฌํ˜„์„ ์‚ฌ์šฉํ•˜๋ฉด ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค.

์œ„์—์„œ ์–ธ๊ธ‰ํ•œ ๋ฐฉ๋ฒ•์œผ๋กœ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ํ™œ์„ฑํ™”ํ•œ ํ›„, ์ด๋ฅผ ํ•™์Šต ๋ฃจํ”„์— ์ ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ํ•™์Šต ๋ฃจํ”„์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” torch.distributed.tensor.experimental.context_parallel์˜ ์–‡์€ ๋ž˜ํผ๋ฅผ ์ œ๊ณตํ•˜๋ฉฐ, ์ด๋Š” ํ•ด๋‹น ๊ธฐ๋Šฅ ์‚ฌ์šฉ์˜ ์ผ๋ถ€ ๋ณต์žก์„ฑ์„ ์ถ”์ƒํ™”ํ•ฉ๋‹ˆ๋‹ค(์ดํ›„ ์ž์„ธํžˆ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค). ํ•™์Šต ๋ฃจํ”„์—์„œ ์ˆ˜ํ–‰ํ•ด์•ผ ํ•˜๋Š” ๋ณ€๊ฒฝ ์‚ฌํ•ญ์„ ์ตœ์†Œํ™”ํ•˜๊ธฐ ์œ„ํ•ด, ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์ด ๋น„ํ™œ์„ฑํ™”๋˜์–ด ์žˆ์œผ๋ฉด noop์œผ๋กœ ๋™์ž‘ํ•˜๊ณ  ํ™œ์„ฑํ™”๋˜์–ด ์žˆ์œผ๋ฉด ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ์ ์šฉํ•˜๋Š” ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋ฅผ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๋ณ‘๋ ฌ ๊ตฌ์„ฑ์— ๋”ฐ๋ผ ์ฝ”๋“œ๋ฅผ ์ˆ˜์ •ํ•˜์ง€ ์•Š๊ณ ๋„ ํ•™์Šต ๋ฃจํ”„์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

for batch in dataloader:
    with accelerator.maybe_context_parallel(
        buffers=[batch["input_ids"], batch["attention_mask"]],
        buffer_seq_dims=[1, 1],
        no_restore_buffers={batch["input_ids"], batch["labels"]},
    ):
        outputs = model(**batch)
        ...

์œ„ ์˜ˆ์‹œ์™€ ๊ฐ™์ด ์ด ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋Š” ๊ฐ ํ•™์Šต ๋‹จ๊ณ„๋งˆ๋‹ค ๋‹ค์‹œ ์ƒ์„ฑ๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ์ˆ˜ํ–‰ํ•˜๋Š” ๊ฒƒ์ด ๋งค์šฐ ์ค‘์š”ํ•ฉ๋‹ˆ๋‹ค.

์ด๋ฅผ ํ†ตํ•ด ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ๋ฅผ 1M+ ์‹œํ€€์Šค ๊ธธ์ด๊นŒ์ง€ ํ™•์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์•„๋ž˜์—์„œ๋Š” ์ตœ๋Œ€ 256k ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ์— ๋Œ€ํ•œ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์˜ ์†๋„์™€ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค. ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ์™€ GPU ์ˆ˜๋ฅผ 2๋ฐฐ๋กœ ๋Š˜๋ฆฌ๋ฉด ์ผ๊ด€๋œ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰์„ ์œ ์ง€ํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, ์ด๋ฅผ ํ†ตํ•ด ๋ฌดํ•œํ•œ ์ปจํ…์ŠคํŠธ ๊ธธ์ด ํ™•์žฅ์ด ๊ฐ€๋Šฅํ•ด์งˆ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

context parallelism memory usage
๊ทธ๋ฆผ 1: ์ตœ๋Œ€ 256k ์ปจํ…์ŠคํŠธ ํฌ๊ธฐ์—์„œ์˜ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ ๋ฉ”๋ชจ๋ฆฌ ์‚ฌ์šฉ๋Ÿ‰ ๋ฐ ์†๋„.

์ด ์˜ˆ์ œ๋“ค์€ examples ํด๋”์—์„œ ์ฐพ์„ ์ˆ˜ ์žˆ๋Š” ์Šคํฌ๋ฆฝํŠธ๋กœ ์ž‘์„ฑ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. 8๊ฐœ์˜ H100 GPU(128k ์‹œํ€€์Šค ๊ธธ์ด)์—์„œ ์˜ˆ์ œ๋ฅผ ์‹คํ–‰ํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

accelerate launch --use-fsdp --fsdp-activation-checkpointing=TRUE examples/torch_native_parallelism/nd_parallel.py --cp-size=8 --sequence-length=128000

Accelerate์˜ ์ธํ„ฐํŽ˜์ด์Šค

์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋Š” ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ๊ตฌ์„ฑํ•˜๋Š” ๋ฐ ์‚ฌ์šฉ๋˜๋Š” ๋ช‡ ๊ฐ€์ง€ ์ธ์ˆ˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.

  • buffers: ์‹œํ€€์Šค ์ฐจ์›์— ๊ฑธ์ณ ์ƒค๋”ฉ๋  ํ…์„œ ๋ชฉ๋ก์ž…๋‹ˆ๋‹ค. ์ด๋Ÿฌํ•œ ํ…์„œ๋Š” ์ผ๋ฐ˜์ ์œผ๋กœ input ids, labels, attention mask์ž…๋‹ˆ๋‹ค.
  • buffer_seq_dims: buffers ๋ฆฌ์ŠคํŠธ์˜ ์ˆœ์„œ๋Œ€๋กœ ๊ฐ ๋ฒ„ํผ์˜ ์‹œํ€€์Šค ์ฐจ์›์„ ์ง€์ •ํ•˜๋Š” ์ •์ˆ˜ ๋ฆฌ์ŠคํŠธ์ž…๋‹ˆ๋‹ค. buffers=[input_ids, shift_labels]๋ฅผ ์ „๋‹ฌํ•˜๋˜, ๋‘ ํ…์„œ์˜ shape๊ฐ€ ๋ชจ๋‘ [batch_size, sequence_length]์ธ ๊ฒฝ์šฐ, ์‹œํ€€์Šค ์ฐจ์›์ด ํ…์„œ์˜ ๋‘ ๋ฒˆ์งธ ์ฐจ์›์ด๋ฏ€๋กœ buffer_seq_dims=[1, 1]์„ ์ „๋‹ฌํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด๋Š” ๋ชจ๋ธ ์ถœ๋ ฅ์„ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ๊ณ„์‚ฐํ•˜๋Š” ๋ฐ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • no_restore_buffers: ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ ๊ตฌํ˜„์€ ๋ฒ„ํผ๋ฅผ ์ธํ”Œ๋ ˆ์ด์Šค๋กœ ์ˆ˜์ •ํ•˜์—ฌ torch.distributed.tensor.Dtensor๋กœ ๋ณ€ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋ฅผ ๋ฒ—์–ด๋‚˜๋ฉด ๋ฒ„ํผ๋ฅผ ์›๋ž˜ ์ƒํƒœ๋กœ ๋ณต์›ํ•˜๊ธฐ ์œ„ํ•ด ํ†ต์‹  ์ปค๋„(๋ณดํ†ต all-gather)์„ ์‹คํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ด ๊ณผ์ •์— ์‹œ๊ฐ„์ด ์†Œ์š”๋˜๋ฏ€๋กœ, ์ปจํ…์ŠคํŠธ ๋งค๋‹ˆ์ €๋ฅผ ๋ฒ—์–ด๋‚œ ํ›„ ๋ฒ„ํผ๋ฅผ ์‚ฌ์šฉํ•ด์•ผ ํ•œ๋‹ค๋Š” ํ™•์‹ ์ด ์—†๋Š” ํ•œ ๋ถˆํ•„์š”ํ•œ ํ†ต์‹ ์„ ํ”ผํ•˜๊ธฐ ์œ„ํ•ด buffers ์ธ์ž์™€ ๋™์ผํ•œ ํ…์„œ๋ฅผ ์ „๋‹ฌํ•˜๋Š” ๊ฒƒ์ด ๊ถŒ์žฅ๋ฉ๋‹ˆ๋‹ค.

Context parallelism์€ ๐Ÿค— transformers์˜ ๋ชจ๋ธ์ด ์ž์ฒด์ ์œผ๋กœ ์‹œํ”„ํŠธํ•˜์—ฌ causal language modeling์„ ํ™œ์„ฑํ™”ํ•  ์ˆ˜ ์žˆ๋Š” input_ids์˜ ๋ณต์‚ฌ๋ณธ์ธ labels์™€ ํ˜ธํ™˜๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๊ฒฝ์šฐ๋ฅผ ์ƒ๊ฐํ•ด ๋ณด์„ธ์š”: labels = [l1, l2, l3, l4, โ€ฆ li] context parallelism์„ ์ ์šฉํ•˜๋ฉด ๊ฐ rank๊ฐ€ labels์˜ ์ผ๋ถ€, ์˜ˆ๋ฅผ ๋“ค์–ด ๋‹ค์Œ๊ณผ ๊ฐ™์€ ํ˜•ํƒœ๋ฅผ ๊ฐ–๊ฒŒ ๋ฉ๋‹ˆ๋‹ค: labels_rank_0 = [l1, l2], labels_rank_1 = [l3, l4], โ€ฆ transformers ๋ชจ๋ธ๋ง ์ฝ”๋“œ๊ฐ€ labels๋ฅผ ์‹œํ”„ํŠธํ•œ ํ›„์—๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๊ฒฐ๊ณผ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค: labels_rank_0 = [l2, PAD], labels_rank_1 = [l3, PAD], โ€ฆ ์—ฌ๊ธฐ์„œ PAD๋Š” padding token์ž…๋‹ˆ๋‹ค. ์ด๋กœ ์ธํ•ด labels๊ฐ€ ๋” ์ด์ƒ ์ž…๋ ฅ๊ณผ ์ •๋ ฌ๋˜์ง€ ์•Š์•„ ์ž˜๋ชป๋œ loss ๊ณ„์‚ฐ์ด ๋ฐœ์ƒํ•ฉ๋‹ˆ๋‹ค. ์ด ๋•Œ๋ฌธ์— ๋ชจ๋ธ์— ์ „๋‹ฌํ•˜๊ธฐ ์ „์— labels๋ฅผ ์ˆ˜๋™์œผ๋กœ ์‹œํ”„ํŠธํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

๊ตฌ์„ฑ ๊ฐ€๋Šฅํ•œ ์˜ต์…˜

Accelerate๋Š” cp_size๋ฅผ ์ œ์™ธํ•˜๊ณ  ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ์„ ๊ตฌ์„ฑํ•˜๊ธฐ ์œ„ํ•œ ๋‹จ์ผ ์˜ต์…˜๋งŒ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค

  • cp_comm_strategy: ์ƒค๋“œ์—์„œ ์‚ฌ์šฉํ•  ๋กœํ…Œ์ด์…˜ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. ๋Œ€๋ถ€๋ถ„์˜ ๊ฒฝ์šฐ "alltoall"๋ณด๋‹ค "allgather"๊ฐ€ ๋” ๋‚˜์€ ์„ฑ๋Šฅ์„ ๋‚ผ ๊ฐ€๋Šฅ์„ฑ์ด ๋งค์šฐ ๋†’์œผ๋ฏ€๋กœ, ์ด ๊ฐ’์„ "allgather"๋กœ ์œ ์ง€ํ•˜๋Š” ๊ฒƒ์„ ๊ฐ•๋ ฅํžˆ ๊ถŒ์žฅํ•ฉ๋‹ˆ๋‹ค.

Context parallel size์€ ์ด๋ฆ„ ๊ทธ๋Œ€๋กœ, ์ž…๋ ฅ์ด ์ƒค๋”ฉ๋  rank์˜ ์ˆ˜๋ฅผ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. Context parallel shard rotation์€ ์ž…๋ ฅ์˜ ์ƒค๋“œ๊ฐ€ rank ๊ฐ„์— ์–ด๋–ป๊ฒŒ ํšŒ์ „๋˜๋Š”์ง€๋ฅผ ์ •์˜ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ์„น์…˜์—์„œ ๋‘ ๊ฐ€์ง€ ์˜ต์…˜์„ ์ž์„ธํžˆ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

ND parallel example ํŒŒ์ผ์—์„œ 8xH100 ๋…ธ๋“œ 1๋Œ€์—์„œ ์ตœ๋Œ€ 128k ์ปจํ…์ŠคํŠธ ๊ธธ์ด๋ฅผ ๊ฐ€์ง„ 8B ๋ชจ๋ธ์„ ํ•™์Šตํ•˜๋Š” ๋์—์„œ ๋(end-to-end) ์˜ˆ์‹œ๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฉ€ํ‹ฐ ๋…ธ๋“œ ํ•™์Šต์„ ์‚ฌ์šฉํ•˜๋ฉด ์—ฌ๋Ÿฌ GPU์—์„œ ์‹œํ€€์Šค ๊ธธ์ด๋ฅผ 1M ์ด์ƒ์œผ๋กœ ํ™•์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ ํ•„์š”์— ๋”ฐ๋ผ ๋‹ค๋ฅธ ๋ณ‘๋ ฌํ™” ์ „๋žต๊ณผ๋„ ์›ํ™œํ•˜๊ฒŒ ๊ฒฐํ•ฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๊ธฐ์ˆ ์  ์„ธ๋ถ€ ์‚ฌํ•ญ

์ด ์„น์…˜์€ ์ƒ๋‹นํžˆ ๊ธฐ์ˆ ์ ์ธ ๋‚ด์šฉ์ž…๋‹ˆ๋‹ค. ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์˜ ๋‚ด๋ถ€ ๊ตฌ์กฐ๋ฅผ ํ•™์Šตํ•  ํ•„์š”๊ฐ€ ์—†๋‹ค๋ฉด ๊ฑด๋„ˆ๋›ฐ๊ณ  ๋ฐ”๋กœ ๊ตฌ์ถ•์„ ์‹œ์ž‘ํ•˜์„ธ์š” ๐Ÿš€

๋‹ค์Œ ์„น์…˜์—์„œ shard๋ผ๋Š” ์šฉ์–ด๋ฅผ ๊ด‘๋ฒ”์œ„ํ•˜๊ฒŒ ์‚ฌ์šฉํ•  ์˜ˆ์ •์ด๋ฏ€๋กœ, ๋จผ์ € ์ •์˜ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ํ…์„œ๋ฅผ Dth ์ฐจ์›์„ ๋”ฐ๋ผ N๊ฐœ์˜ rank์— ๊ฑธ์ณ sharded๋ผ๊ณ  ๋ถ€๋ฅผ ๋•Œ, ์ด๋Š” ํ•ด๋‹น ํ…์„œ๊ฐ€ N๊ฐœ์˜ ๋ถ€๋ถ„์œผ๋กœ ๋‚˜๋‰˜์–ด ์žˆ์œผ๋ฉฐ ๊ฐ ๋ถ€๋ถ„์˜ shape์ด [..., D//N, ...]์ž„์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค.

๊ทธ๋ ‡๋‹ค๋ฉด ์–ด๋–ป๊ฒŒ ์ž‘๋™ํ•˜๋‚˜์š”?

Context parallelism์€ Q, K and V ํ–‰๋ ฌ์„ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉํ•˜๋Š” ๋ฐฉ์‹์œผ๋กœ ๋™์ž‘ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ ๋žญํฌ์—๋Š” ํ• ๋‹น๋œ Q์˜ ์ƒค๋“œ๊ฐ€ ์žˆ์œผ๋ฉฐ, ์ด๋ฅผ Q_i๋ผ๊ณ  ๋ถ€๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ด ํ–‰๋ ฌ์€ ์ „์ฒด ๊ณ„์‚ฐ ๊ณผ์ • ๋™์•ˆ ํ•ด๋‹น ๋žญํฌ์—๋งŒ ์œ ์ง€๋ฉ๋‹ˆ๋‹ค. ๋งˆ์ฐฌ๊ฐ€์ง€๋กœ, ๊ฐ ๋žญํฌ์—๋Š” K์™€ V์˜ ์ž์ฒด ์ƒค๋“œ๊ฐ€ ์žˆ์œผ๋ฉฐ, ์ด๋ฅผ ๊ฐ๊ฐ K_i์™€ V_i๋ผ๊ณ  ๋ถ€๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ ๋‹ค์Œ, ๊ฐ ๋žญํฌ๋Š” ์ž์‹ ์˜ Q_i, K_i, V_i ์ƒค๋“œ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ attention์„ ๊ณ„์‚ฐํ•˜๋ฉฐ, ์ด๋ฅผ attn_i๋ผ๊ณ  ๋ถ€๋ฅด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ด ๊ณ„์‚ฐ ์ค‘์— ๋‹ค๋ฅธ ๋ชจ๋“  ๋žญํฌ์—์„œ Ks์™€ Vs๋ฅผ ์ˆ˜์ง‘ํ•˜๊ธฐ ์œ„ํ•œ ํ†ต์‹  ์ปค๋„์ด ์‹คํ–‰๋ฉ๋‹ˆ๋‹ค. ์‚ฌ์šฉ๋˜๋Š” ํ†ต์‹  ์›์‹œ(primitive)๋Š” context_parallel_shard_rotation ์˜ต์…˜์— ๋”ฐ๋ผ ๋‹ฌ๋ผ์ง‘๋‹ˆ๋‹ค. ์ด ๋ฐฉ์‹์„ ํ†ตํ•ด ๊ฐ ๋žญํฌ๋Š” ๋จผ์ € Q_i, K_i, V_i๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ, ๊ทธ ๋‹ค์Œ์—๋Š” ๋‹ค๋ฅธ ๋ชจ๋“  ๋žญํฌ์˜ K_j์™€ V_j๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋กœ์ปฌ attention์„ ๊ณ„์‚ฐํ•˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ๊ฐ ๋žญํฌ๊ฐ€ ์‹œํ€€์Šค ์ฐจ์›๋ณ„๋กœ ์ƒค๋”ฉ๋œ Q, K and V ํ–‰๋ ฌ์„ ๋ณด์œ ํ•˜๋ฏ€๋กœ, ๊ฒฐ๊ณผ ํ–‰๋ ฌ์˜ ํฌ๊ธฐ๊ฐ€ ์ž‘์•„์ ธ ๋‹จ์ผ GPU์— ์ ์žฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ ์˜์‚ฌ ์ฝ”๋“œ๋กœ ์ด๋ฅผ ๊ณต์‹ํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

comm_kernel = {"allgather": allgather, "alltoall": alltoall}[context_parallel_shard_rotation]
Qi, Ki, Vi = shard(Q, K, V, seq_dim)
attn[i] = attn(Qi, Ki, Vi)
for j in range(context_parallel_size):
    Kj, Vj = comm_kernel()
    attn[j] = attn(Qi, Kj, Vj) # [batch, num_heads, seq_len // context_parallel_size, head_dim]

final_attn = combine(attn)

all-to-all๊ณผ all-gather

all-gather

all-to-all๊ณผ all-gather์˜ ์ฐจ์ด๋Š” ๋ฌด์—‡์ผ๊นŒ์š”? all-gather์˜ ๊ฒฝ์šฐ ํ†ต์‹ ์ด ๋งค์šฐ ๋‹จ์ˆœํ•ฉ๋‹ˆ๋‹ค. (๋ณดํ†ต ๋” ์˜ค๋ž˜ ๊ฑธ๋ฆฌ๋ฏ€๋กœ) ์‹ค์ œ๋กœ๋Š” ๊ณ„์‚ฐ ์ „์—, ๋กœ์ปฌ ์–ดํ…์…˜ attn_i์„ ๊ณ„์‚ฐํ•œ ํ›„ all-gather๋ฅผ ์‹œ์ž‘ํ•˜์—ฌ ๋‹ค๋ฅธ ๋ชจ๋“  rank์˜ Ks์™€ Vs๋ฅผ ์ˆ˜์ง‘ํ•ฉ๋‹ˆ๋‹ค. ์ด ํ†ต์‹ ์ด ์™„๋ฃŒ๋˜๋ฉด ๊ฐ rank๋Š” ๋‹ค๋ฅธ ๋ชจ๋“  rank์˜ Ks์™€ Vs๋ฅผ ๋ชจ๋‘ ๋ณด์œ ํ•˜๊ฒŒ ๋˜๋ฉฐ, ์ด๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์ˆœ์ฐจ์ ์œผ๋กœ ์–ดํ…์…˜์„ ๊ณ„์‚ฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด์ƒ์ ์ธ ์‹œ๋‚˜๋ฆฌ์˜ค์—์„œ๋Š” attn_i์˜ ๊ณ„์‚ฐ์ด ์™„๋ฃŒ๋˜๋Š” ์ •ํ™•ํ•œ ์ˆœ๊ฐ„์— all-gather๊ฐ€ ๋๋‚ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ์‹ค์ œ๋กœ๋Š” ์ด๋Ÿฐ ์ผ์ด ์ผ์–ด๋‚˜์ง€ ์•Š์œผ๋ฏ€๋กœ, attn_i ์ „์ฒด๊ฐ€ ํ†ต์‹ ์˜ ์ผ๋ถ€์™€ ์˜ค๋ฒ„๋žฉ๋˜๊ณ , K_j์™€ V_j๋ฅผ ์‚ฌ์šฉํ•œ ๊ณ„์‚ฐ์„ ์‹œ์ž‘ํ•˜๊ธฐ ์œ„ํ•ด all-gather๊ฐ€ ์™„๋ฃŒ๋  ๋•Œ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ๋Š” ๊ฒƒ์ด ์ด์ƒ์ ์ธ ์‹ค์ œ ์˜ค๋ฒ„๋žฉ์ž…๋‹ˆ๋‹ค.

all-to-all

All-to-all, or sometimes called ring-rotation utilizes a ring-like communication pattern. After concluding attn_i computation, an all-to-all is launched to send K_i and V_i to the neighbouring ranks. We then repeat this context_parallel_size-1 times, so that each rank sees all the shards of K and V from all other ranks once. In ideal scenario, we prefetch shards K_i+1 and V_i+1 from the neighbouring rank and this communication is exactly overlapped with computation of our current attn_i. Again, realistically, this perfect overlap doesnโ€™t ever happen. Given the nature of this approach, if we donโ€™t achieve perfect overlap, the penalty is way larger than with all-gather.

์–ด๋–ค ํšŒ์ „ ๋ฐฉ๋ฒ•์„ ์„ ํƒํ•ด์•ผ ํ• ๊นŒ์š”?

In theory, all-to-all should be the better choice. Though in practice, it rarely is. Therefore, we default to all-gather, as itโ€™s more likely to achieve better performance. Extensive benchmarks from the torchtitan team also show that all-to-all rarely outperforms all-gather. Though, we still provide both options, as you might find one to be better for your use case.

์•„๋ž˜ ์ด๋ฏธ์ง€์˜ ํ”„๋กœํŒŒ์ผ๋Ÿฌ ์ถœ๋ ฅ์—์„œ ์ด ๋ฌธ์ œ๋ฅผ ์ง์ ‘ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

all-to-all profiler output
Figure 1: In red you can see the idle time, while we wait for the all-to-all kernel to finish. Highlighted in the first blue bar, you can see that it takes ~250us to finish, which is repeated N-1 times for each attention call, where N is the context parallel size.

์™œ FSDP2๋งŒ ์‚ฌ์šฉํ• ๊นŒ์š”?

FSDP2์™€ ํ•จ๊ป˜ ์ปจํ…์ŠคํŠธ ๋ณ‘๋ ฌ์„ฑ๋งŒ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. context_parallel_size์™€ dp_shard_size์˜ ๊ฒฐํ•ฉ๋œ ๋ฉ”์‹œ๋ฅผ ์ƒ์„ฑํ•˜์—ฌ ๊ทธ ์ž ์žฌ๋ ฅ์„ ์ตœ๋Œ€ํ•œ ํ™œ์šฉํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋™์ž‘ ๋ฐฉ์‹์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. cp_size*dp_shard_size ํฌ๊ธฐ์˜ ๊ฒฐํ•ฉ๋œ ๋ฉ”์‹œ์— ๊ฑธ์ณ ๋ชจ๋ธ์„ ์ƒค๋”ฉํ•˜์—ฌ ๋ฉ”๋ชจ๋ฆฌ ์ ˆ๊ฐ ํšจ๊ณผ๋ฅผ ๊ทน๋Œ€ํ™”ํ•ฉ๋‹ˆ๋‹ค. ์•„๋ž˜ ์ด๋ฏธ์ง€์— ํ‘œ์‹œ๋œ ๋ฐ”์™€ ๊ฐ™์ด FSDP ํ†ต์‹ ์ด ์–ดํ…์…˜ ๊ณ„์‚ฐ๊ณผ ์™„์ „ํžˆ ์˜ค๋ฒ„๋žฉ๋˜๋ฏ€๋กœ, ์ผ์ข…์˜ '๋ฌด์ƒ ํ˜œํƒ'์ด๋ผ๊ณ  ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

why FSDP2+CP
๊ทธ๋ฆผ 2: ํŒŒ๋ž€์ƒ‰ ์‚ฌ๊ฐํ˜•(Stream 23)์—์„œ `FSDP ์ƒค๋“œ์˜ ํ”„๋ฆฌํŽ˜์น˜๊ฐ€ ์–ดํ…์…˜ ๊ณ„์‚ฐ(Stream 7)๊ณผ ์™„์ „ํžˆ ๊ฒน์ณ ์žˆ๋Š” ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์œผ๋ฉฐ, ๋นจ๊ฐ„์ƒ‰ ์‚ฌ๊ฐํ˜•(Stream 24)์—์„œ๋Š” ์˜ฌ๊ฐ€๋” ์ปค๋„๋กœ ์ธํ•ด ์œ ํœด ์‹œ๊ฐ„ ๋ฒ„๋ธ”์ด ๋ฐœ์ƒํ•˜์—ฌ ๊ณ„์‚ฐ ์ŠคํŠธ๋ฆผ(7)์ด ์œ ํœด ์ƒํƒœ๊ฐ€ ๋˜๋Š” ๊ฒƒ์„ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

In the figure above, you can also note the difference between all-to-all and all-gather. While in all-to-all (Figure 1), we launch a communication kernel N-1 times for each attention call, in all-gather (Figure 2), we launch a communication kernel only once. This results in a bigger bubble, but it only happens once per attention call, while in all-to-all, it happens N-1 times.

์กฐ์ธํŠธ ๋ฉ”์‹œ์—์„œ์˜ ๋ฐ์ดํ„ฐ ๋””์ŠคํŒจ์นญ

cp ์„œ๋ธŒ๊ทธ๋ฃน ์ „์ฒด์— ๋™์ผํ•œ ๋ฐ์ดํ„ฐ ๋ฐฐ์น˜๋กœ ๋””์ŠคํŒจ์น˜ํ•˜์—ฌ ๊ฒฐ๊ณผ๊ฐ€ ์ •ํ™•ํ•˜๋„๋ก ๋ณด์žฅํ•ฉ๋‹ˆ๋‹ค. (์ฆ‰, cp ์„œ๋ธŒ๊ทธ๋ฃน์˜ ๊ฐ ๋žญํฌ๋Š” ๋™์ผํ•œ ๋ฐ์ดํ„ฐ ๋ฐฐ์น˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.) ํ•˜์ง€๋งŒ dp_shard ๊ทธ๋ฃน์˜ ๊ฐ ๋žญํฌ์—๋Š” ์„œ๋กœ ๋‹ค๋ฅธ ๋ฐฐ์น˜๋ฅผ ๋””์ŠคํŒจ์น˜ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ƒ๊ฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค:

# 8 GPUS, --dp_shard_size 4, --cp_size 2
# mesh = [[0, 1], [2, 3], [4, 5], [6, 7]]
# model is sharded across the whole mesh (each GPU holds 1/8 of the model)
# GPUs 0,1 = batch 0
# GPUs 2,3 = batch 1
... and so on.
Update on GitHub