
* Upgraded Layout Postprocessing, sending old code back to ERZ Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Implement hierachical cluster layout processing Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Pass nested cluster processing through full pipeline Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Pass nested clusters through GLM as payload Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Move to_docling_document from ds-glm to this repo Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Clean up imports again Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * feat(Accelerator): Introduce options to control the num_threads and device from API, envvars, CLI. - Introduce the AcceleratorOptions, AcceleratorDevice and use them to set the device where the models run. - Introduce the accelerator_utils with function to decide the device and resolve the AUTO setting. - Refactor the way how the docling-ibm-models are called to match the new init signature of models. - Translate the accelerator options to the specific inputs for third-party models. - Extend the docling CLI with parameters to set the num_threads and device. - Add new unit tests. - Write new example how to use the accelerator options. * fix: Improve the pydantic objects in the pipeline_options and imports. Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * fix: TableStructureModel: Refactor the artifacts path to use the new structure for fast/accurate model Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * Updated test ground-truth Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Updated test ground-truth (again), bugfix for empty layout Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * fix: Do proper check to set the device in EasyOCR, RapidOCR. Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * fix: Correct the way to set GPU for EasyOCR, RapidOCR Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * fix: Ocr AccleratorDevice Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> * Merge pull request #556 from DS4SD/cau/layout-processing-improvement feat: layout processing improvements and bugfixes * Update lockfile Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Update tests Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Update HF model ref, reset test generate Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Repin to release package versions Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Many layout processing improvements, add document index type Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Update pinnings to docling-core Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Update test GT Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fix table box snapping Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fixes for cluster pre-ordering Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Introduce OCR confidence, propagate to orphan in post-processing Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Fix form and key value area groups Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Adjust confidence in EasyOcr Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Roll back CLI changes from main Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Update test GT Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Update docling-core pinning Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Annoying fixes for historical python versions Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Updated test GT for legacy Signed-off-by: Christoph Auer <cau@zurich.ibm.com> * Comment cleanup Signed-off-by: Christoph Auer <cau@zurich.ibm.com> --------- Signed-off-by: Christoph Auer <cau@zurich.ibm.com> Signed-off-by: Nikos Livathinos <nli@zurich.ibm.com> Co-authored-by: Nikos Livathinos <nli@zurich.ibm.com>
51 lines
1.3 KiB
Python
51 lines
1.3 KiB
Python
import sys
|
|
from pathlib import Path
|
|
|
|
from pydantic import BaseModel
|
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
|
|
|
|
|
class DocumentLimits(BaseModel):
|
|
max_num_pages: int = sys.maxsize
|
|
max_file_size: int = sys.maxsize
|
|
|
|
|
|
class BatchConcurrencySettings(BaseModel):
|
|
doc_batch_size: int = 2
|
|
doc_batch_concurrency: int = 2
|
|
page_batch_size: int = 4
|
|
page_batch_concurrency: int = 2
|
|
elements_batch_size: int = 16
|
|
|
|
# doc_batch_size: int = 1
|
|
# doc_batch_concurrency: int = 1
|
|
# page_batch_size: int = 1
|
|
# page_batch_concurrency: int = 1
|
|
|
|
# model_concurrency: int = 2
|
|
|
|
# To force models into single core: export OMP_NUM_THREADS=1
|
|
|
|
|
|
class DebugSettings(BaseModel):
|
|
visualize_cells: bool = False
|
|
visualize_ocr: bool = False
|
|
visualize_layout: bool = False
|
|
visualize_raw_layout: bool = False
|
|
visualize_tables: bool = False
|
|
|
|
profile_pipeline_timings: bool = False
|
|
|
|
# Path used to output debug information.
|
|
debug_output_path: str = str(Path.cwd() / "debug")
|
|
|
|
|
|
class AppSettings(BaseSettings):
|
|
model_config = SettingsConfigDict(env_prefix="DOCLING_", env_nested_delimiter="_")
|
|
|
|
perf: BatchConcurrencySettings
|
|
debug: DebugSettings
|
|
|
|
|
|
settings = AppSettings(perf=BatchConcurrencySettings(), debug=DebugSettings())
|