NeoAnd/Docling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: enable cuda_use_flash_attention2 for PictureDescriptionVlmModel (#1496)

Browse Source 
fix: enable use_cuda_flash_attention2 for PictureDescriptionVlmModel

Signed-off-by: Zach Cox <zach.s.cox@gmail.com>
This commit is contained in:
Zach Cox 2025-04-30 02:02:52 -04:00 committed by GitHub
parent 976e92e289
commit cc453961a9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docling/models/picture_description_vlm_model.py
View File

@ -57,7 +57,10 @@ class PictureDescriptionVlmModel(PictureDescriptionBaseModel):
artifacts_path, artifacts_path,
torch_dtype=torch.bfloat16, torch_dtype=torch.bfloat16,
_attn_implementation=( _attn_implementation=(
"flash_attention_2" if self.device.startswith("cuda") else "eager" "flash_attention_2"
if self.device.startswith("cuda")
and accelerator_options.cuda_use_flash_attention2
else "eager"
), ),
).to(self.device) ).to(self.device)