NeoAnd/Docling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Support AsciiDoc and Markdown input format (#168)

Browse Source 
* updated the base-model and added the asciidoc_backend

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* updated the asciidoc backend

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Ensure all models work only on valid pages (#158)

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* ci: run ci also on forks (#160)


---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>

* fix: fix legacy doc ref (#162)

Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>

* docs: typo fix (#155)

* Docs: Typo fix

- Corrected spelling of invidual to automatic

Signed-off-by: ABHISHEK FADAKE <31249309+fadkeabhi@users.noreply.github.com>

* add synchronize event for forks

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: ABHISHEK FADAKE <31249309+fadkeabhi@users.noreply.github.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>

* feat: add coverage_threshold to skip OCR for small images (#161)

* feat: add coverage_threshold to skip OCR for small images

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* filter individual boxes

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* rename option

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

---------

Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>

* chore: bump version to 2.1.0 [skip ci]

* adding tests for asciidocs

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* first working asciidoc parser

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* reformatted the code

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the mypy

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* adding test_02.asciidoc

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Drafting Markdown backend via Marko library

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* work in progress on MD backend

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* md_backend produces docling document with headers, paragraphs, lists

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Improvements in md parsing

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Detecting and assembling tables in markdown in temporary buffers

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Added initial docling table support to md_backend

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Cleaned code, improved logging for MD

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixes MyPy requirements, and rest of pre-commit

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixed example run_md, added origin info to md_backend

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* working on asciidocs, struggling with ImageRef

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* able to parse the captions and image uri's

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* fixed the mypy

Signed-off-by: Peter Staar <taa@zurich.ibm.com>

* Update all backends with proper filename in DocumentOrigin

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Update to docling-core v2.1.0

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Fixes for MD Backend, to avoid duplicated text inserts into docling doc

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fix styling

Signed-off-by: Christoph Auer <cau@zurich.ibm.com>

* Added support for code blocks and fenced code in MD

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* cleaned prints

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Added proper processing of in-line textual elements for MD backend

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixed issues with duplicated paragraphs and incorrect lists in pptx

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

* Fixed issue with group ordeering in pptx backend, added gebug log into run with formats

Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>

---------

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Signed-off-by: Christoph Auer <cau@zurich.ibm.com>
Signed-off-by: Michele Dolfi <dol@zurich.ibm.com>
Signed-off-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Signed-off-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Signed-off-by: ABHISHEK FADAKE <31249309+fadkeabhi@users.noreply.github.com>
Signed-off-by: Maksym Lysak <mly@zurich.ibm.com>
Co-authored-by: Peter Staar <taa@zurich.ibm.com>
Co-authored-by: Michele Dolfi <97102151+dolfim-ibm@users.noreply.github.com>
Co-authored-by: Panos Vagenas <35837085+vagenas@users.noreply.github.com>
Co-authored-by: ABHISHEK FADAKE <31249309+fadkeabhi@users.noreply.github.com>
Co-authored-by: Michele Dolfi <dol@zurich.ibm.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Maksym Lysak <mly@zurich.ibm.com>
This commit is contained in:
Christoph Auer
2024-10-23 16:14:26 +02:00
committed by GitHub
parent 3496b4838f
commit 3023f18ba0
52 changed files with 3731 additions and 3517 deletions
Download Patch File Download Diff File Expand all files Collapse all files

202
tests/data/groundtruth/docling_v1/redp5695.md
View File

@@ -40,17 +40,17 @@ Ever wonder how many transactions a bank processes per day? What about the pace
The most recent platform for IBM Z is IBM z16™. The IBM z16 supports the following features:
GLYPH<SM590000> On-chip AI acceleration
- GLYPH<SM590000> On-chip AI acceleration
GLYPH<SM590000> Quantum-safe crypto discovery
- GLYPH<SM590000> Quantum-safe crypto discovery
GLYPH<SM590000> Simplified compliance
- GLYPH<SM590000> Simplified compliance
GLYPH<SM590000> Flexible capacity
- GLYPH<SM590000> Flexible capacity
GLYPH<SM590000> Modernization of applications
- GLYPH<SM590000> Modernization of applications
GLYPH<SM590000> Sustainability
- GLYPH<SM590000> Sustainability
With these features, enterprises can upgrade applications while preserving secure and resilient data.
@@ -106,13 +106,13 @@ Figure 6 Solution overview of Cloud Pak for Data
We highlight the four main pillars that make IBM Z the correct infrastructure for CP4D:
GLYPH<SM590000> Performance and Scale
- GLYPH<SM590000> Performance and Scale
GLYPH<SM590000> Embedded Accelerators
- GLYPH<SM590000> Embedded Accelerators
GLYPH<SM590000> Reliability and Availability
- GLYPH<SM590000> Reliability and Availability
GLYPH<SM590000> Security and Governance.
- GLYPH<SM590000> Security and Governance.
From a performance perspective, CP4D on IBM Z provides your data and AI with high transaction processing and a powerful infrastructure. From the embedded accelerators perspective, CP4D on IBM Z can investigate each transaction thanks to a cutting-edge DL inference technology even in the most demanding, sensitive, and latency-prone real-time workloads. From a reliability perspective, CP4D on IBM Z provides high availability and resiliency. Lastly from the security perspective, CP4D on IBM Z is suitable for protecting sensitive data and AI models for enterprises in highly regulated industries or those industries that are worried about security.
@@ -120,17 +120,17 @@ From a performance perspective, CP4D on IBM Z provides your data and AI with hig
With CP4D on IBM Z and IBM LinuxONE, users can develop, train, and deploy AI and ML models. Users can accomplish this task by using the CP4D IBM Watsonfi Studio and IBM Watson Machine Learning (WLM) services. By using these two fundamental services, users can accomplish the following tasks:
GLYPH<SM590000> Provision various containerized databases.
- GLYPH<SM590000> Provision various containerized databases.
GLYPH<SM590000> Explore, clean, shape, and alter data by using Data Refinery.
- GLYPH<SM590000> Explore, clean, shape, and alter data by using Data Refinery.
GLYPH<SM590000> Use project-specific data that is uploaded, or connect to distant data.
- GLYPH<SM590000> Use project-specific data that is uploaded, or connect to distant data.
GLYPH<SM590000> Create Spark run times and applications.
- GLYPH<SM590000> Create Spark run times and applications.
GLYPH<SM590000> Create, build, evaluate, and deploy analytics and ML models with trust and transparency.
- GLYPH<SM590000> Create, build, evaluate, and deploy analytics and ML models with trust and transparency.
GLYPH<SM590000> Leverage the AI Integrated Accelerator for TensorFlow 2.7.2 and Snap ML 1.9.
- GLYPH<SM590000> Leverage the AI Integrated Accelerator for TensorFlow 2.7.2 and Snap ML 1.9.
For more information about the specifics of these capabilities, see Capabilities on Linux on IBM Z and IBM LinuxONE.
@@ -167,17 +167,17 @@ Figure 7 Developing, training, and deploying an AI model on Cloud Pak for Data o
In summary, here are some of the reasons why you should choose AI on IBM Z:
GLYPH<SM590000> World-class AI inference platform for enterprise workloads:
- GLYPH<SM590000> World-class AI inference platform for enterprise workloads:
-Embedded accelerators: A centralized on-chip AI accelerator that is shared by all cores.
- -Embedded accelerators: A centralized on-chip AI accelerator that is shared by all cores.
-Industry standard AI ecosystem: Many industry open-source data science frameworks are available on the platform.
- -Industry standard AI ecosystem: Many industry open-source data science frameworks are available on the platform.
-Seamlessly integrate AI into existing enterprise workload stacks: Train anywhere, and then deploy on IBM Z.
- -Seamlessly integrate AI into existing enterprise workload stacks: Train anywhere, and then deploy on IBM Z.
GLYPH<SM590000> Security: Encrypted memory, and improved trusted execution environments.
- GLYPH<SM590000> Security: Encrypted memory, and improved trusted execution environments.
GLYPH<SM590000> Sustainability: Reduce your energy consumption with real-time monitoring tools about the energy consumption of the system.
- GLYPH<SM590000> Sustainability: Reduce your energy consumption with real-time monitoring tools about the energy consumption of the system.
## AI use cases
@@ -193,23 +193,23 @@ For the airline industry, processes such as air traffic management, flight manag
In the following sections, we describe the following use cases:
GLYPH<SM590000> "Use case 1: Responsible AI augmented with risk and regulatory compliance" on page 12 AI model lifecycle governance, risk management, and regulatory compliance are key to the success of the enterprises. It is imperative to adopt a typical AI model lifecycle to protect new end-to-end risks.
- GLYPH<SM590000> "Use case 1: Responsible AI augmented with risk and regulatory compliance" on page 12 AI model lifecycle governance, risk management, and regulatory compliance are key to the success of the enterprises. It is imperative to adopt a typical AI model lifecycle to protect new end-to-end risks.
GLYPH<SM590000> "Use case 2: Credit default risk assessment" on page 22
- GLYPH<SM590000> "Use case 2: Credit default risk assessment" on page 22
Core banking solutions running on IBM Z that are involved in processing inbound transactions need real-time fraud detection to prevent fraud. Other types of possible use cases might be credit risk analysis, anti-money laundering, loan approval, fraud detection in payments, and instant payments.
- Core banking solutions running on IBM Z that are involved in processing inbound transactions need real-time fraud detection to prevent fraud. Other types of possible use cases might be credit risk analysis, anti-money laundering, loan approval, fraud detection in payments, and instant payments.
GLYPH<SM590000> "Use case 3: Clearing and settlement" on page 25
- GLYPH<SM590000> "Use case 3: Clearing and settlement" on page 25
The use of AI can help to predict which trades or transactions have high risk exposures, and propose solutions for a more efficient settlement process.
- The use of AI can help to predict which trades or transactions have high risk exposures, and propose solutions for a more efficient settlement process.
GLYPH<SM590000> "Use case 4: Remaining Useful Life of an aircraft engine" on page 27
- GLYPH<SM590000> "Use case 4: Remaining Useful Life of an aircraft engine" on page 27
We describe how AI can help to avoid unplanned aircraft downtime by determining the remaining time or cycles that an aircraft engine is likely to operate before failure.
- We describe how AI can help to avoid unplanned aircraft downtime by determining the remaining time or cycles that an aircraft engine is likely to operate before failure.
GLYPH<SM590000> "Use case 5: AI-powered video analytics on an infant's motions for health prediction" on page 30
- GLYPH<SM590000> "Use case 5: AI-powered video analytics on an infant's motions for health prediction" on page 30
In this section, we describe how AI can predict an infant's health conditions by monitoring real-time body movements.
- In this section, we describe how AI can predict an infant's health conditions by monitoring real-time body movements.
## Use case 1: Responsible AI augmented with risk and regulatory compliance
@@ -221,11 +221,11 @@ How mature is your AI governance? In this section, we provide a use case demonst
Here are the three main reasons why organizations struggle with the adoption of AI:
GLYPH<SM590000> Scaling with growing regulations
- GLYPH<SM590000> Scaling with growing regulations
GLYPH<SM590000> Lack of confidence in operationalized AI (making responsible AI)
- GLYPH<SM590000> Lack of confidence in operationalized AI (making responsible AI)
GLYPH<SM590000> Challenges around managing the risk throughout the entire AI workflow
- GLYPH<SM590000> Challenges around managing the risk throughout the entire AI workflow
## Scaling with growing regulations
@@ -239,17 +239,17 @@ Responsible AI protects against loss of data privacy, and reduced customer loyal
Organizations need to mitigate risk of the following items:
GLYPH<SM590000> Deciding not to use certain technologies or practices
- GLYPH<SM590000> Deciding not to use certain technologies or practices
GLYPH<SM590000> Using personal information when needed and with a user's consent
- GLYPH<SM590000> Using personal information when needed and with a user's consent
GLYPH<SM590000> Ensuring automated decisions are free from bias
- GLYPH<SM590000> Ensuring automated decisions are free from bias
GLYPH<SM590000> Customer confidence by providing explanations for business decisions
- GLYPH<SM590000> Customer confidence by providing explanations for business decisions
GLYPH<SM590000> Fraud to the organization and to customer's accounts
- GLYPH<SM590000> Fraud to the organization and to customer's accounts
GLYPH<SM590000> Delays in putting models into production
- GLYPH<SM590000> Delays in putting models into production
In fact, in a recent survey, these concerns were echoed by real AI adopters when asked what aspects of trust are most important to them. Although explaining how AI decides is the primary concern, all of these concerns are important.
@@ -274,39 +274,39 @@ In a world where trust, transparency and explainable AI matters, every organizat
Lifecycle governance helps you manage your business information throughout its lifecycle, that is, from creation to deletion. IBM AI governance addresses the problems that challenge records managements:
GLYPH<SM590000> Monitor, catalog, and govern AI models from anywhere throughout the AI lifecycle.
- GLYPH<SM590000> Monitor, catalog, and govern AI models from anywhere throughout the AI lifecycle.
GLYPH<SM590000> Automate the capture of model metadata for report generation.
- GLYPH<SM590000> Automate the capture of model metadata for report generation.
GLYPH<SM590000> Drive transparent and explainable AI at scale.
- GLYPH<SM590000> Drive transparent and explainable AI at scale.
GLYPH<SM590000> Increase accuracy of predictions by identifying how AI is used and where it is lagging.
- GLYPH<SM590000> Increase accuracy of predictions by identifying how AI is used and where it is lagging.
## Risk management
Risk management is used in IBM AI governance to identify, manage, monitor, and report on risk and compliance initiatives at scale:
GLYPH<SM590000> Automate facts and workflow management to comply with business standards.
- GLYPH<SM590000> Automate facts and workflow management to comply with business standards.
GLYPH<SM590000> Use dynamic dashboards for clear and concise customizable results.
- GLYPH<SM590000> Use dynamic dashboards for clear and concise customizable results.
GLYPH<SM590000> Enhanced collaboration across multiple regions and geographies.
- GLYPH<SM590000> Enhanced collaboration across multiple regions and geographies.
## Regulatory compliance
Regulatory compliance is a set of rules that organizations must follow to protect sensitive information and ensure human safety. Any business that works with digital assets, consumer data, health regulations, employee safety, and private communications is subject to regulatory compliance.$^{3}$ The IBM AI governance solution for IBM Z includes the following tasks:
GLYPH<SM590000> Help adhere to external AI regulations for audit and compliance.
- GLYPH<SM590000> Help adhere to external AI regulations for audit and compliance.
GLYPH<SM590000> Convert external AI regulations into policies for automatic enforcement.
- GLYPH<SM590000> Convert external AI regulations into policies for automatic enforcement.
GLYPH<SM590000> Use dynamic dashboards for compliance status across policies and regulations.
- GLYPH<SM590000> Use dynamic dashboards for compliance status across policies and regulations.
Enterprises can develop AI models and deploy them by using IBM Watson Studio or WML on CP4D on Red Hat OpenShift on a virtual machine that is based on IBM z/VM or Red Hat Enterprise Linux KVM on IBM Z. AI governance on IBM LinuxONE is supported in the following two ways:
GLYPH<SM590000> Monitor the AI models with Watson OpenScale on CP4D on Red Hat OpenShift on a virtual machine on IBM Z.
- GLYPH<SM590000> Monitor the AI models with Watson OpenScale on CP4D on Red Hat OpenShift on a virtual machine on IBM Z.
GLYPH<SM590000> Enterprises can develop AI models by creating and training models by using Watson Studio and development tools such as Jupyter Notebook or JupyterLab, and then deploying the model onto WML on CP4D on Red Hat OpenShift on a virtual machine on IBM Z. Then, these enterprises can achieve end-end AI governance by running AI Factsheets, IBM Watson OpenScale, and IBM Watson OpenPagesfi on CP4D on x86.
- GLYPH<SM590000> Enterprises can develop AI models by creating and training models by using Watson Studio and development tools such as Jupyter Notebook or JupyterLab, and then deploying the model onto WML on CP4D on Red Hat OpenShift on a virtual machine on IBM Z. Then, these enterprises can achieve end-end AI governance by running AI Factsheets, IBM Watson OpenScale, and IBM Watson OpenPagesfi on CP4D on x86.
Figure 9 on page 16 shows the end-to-end flow for a remote AI governance solution.
@@ -315,22 +315,22 @@ Figure 9 Remote AI governance solution end-to-end flow
To achieve end-to-end AI governance, complete the following steps:
1. Create a model entry in IBM OpenPages by using CP4D on a x86 platform, as shown in Figure 10.
- 1. Create a model entry in IBM OpenPages by using CP4D on a x86 platform, as shown in Figure 10.
Figure 10 Creating a model entry in IBM OpenPages
<!-- image -->
2. Train a model by using Watson Studio and by using development tools such as Jupyter Notebook or JupyterLab on CP4D on Red Hat OpenShift on a virtual machine on IBM Z, as shown in Figure 11.
- 2. Train a model by using Watson Studio and by using development tools such as Jupyter Notebook or JupyterLab on CP4D on Red Hat OpenShift on a virtual machine on IBM Z, as shown in Figure 11.
Figure 11 Training an AI model by using Watson Studio
<!-- image -->
3. Deploy the model by using WML on CP4D on Red Hat OpenShift on a virtual machine on IBM Z, as shown in Figure 12.
- 3. Deploy the model by using WML on CP4D on Red Hat OpenShift on a virtual machine on IBM Z, as shown in Figure 12.
Figure 12 Deploying an AI model by using WML on Cloud Pak for Data
<!-- image -->
4. Track the external model lifecycle by browsing through the Catalogs/Platform assets catalog by using AI Factsheets and OpenPages while using CP4D on an x86 platform, as shown in Figure 13. The external model (deployed on CP4D on Red Hat OpenShift on a virtual machine on IBM Z) is saved as a platform asset catalog on the x86 platform.
- 4. Track the external model lifecycle by browsing through the Catalogs/Platform assets catalog by using AI Factsheets and OpenPages while using CP4D on an x86 platform, as shown in Figure 13. The external model (deployed on CP4D on Red Hat OpenShift on a virtual machine on IBM Z) is saved as a platform asset catalog on the x86 platform.
Figure 13 External model
<!-- image -->
@@ -345,7 +345,7 @@ You can see that the model facts are tracked and synchronized to IBM OpenPages f
Figure 15 Model facts that are tracked and synchronized to IBM OpenPages on an x86 platform
<!-- image -->
5. Create an external model by using IBM OpenScale on the x86 platform, as shown in Figure 16.
- 5. Create an external model by using IBM OpenScale on the x86 platform, as shown in Figure 16.
Figure 16 Creating an external model on an x86 platform
<!-- image -->
@@ -398,15 +398,15 @@ Data scientists can start creating and training a DL AI model by using a Jupyter
In summary, here are some considerations for developing real-time AI models, such as credit risk assessment:
GLYPH<SM590000> A preference for in-platform run times of the model, such as faster execution results.
- GLYPH<SM590000> A preference for in-platform run times of the model, such as faster execution results.
GLYPH<SM590000> Less overhead in the end-to-end flows might improve scoring time.
- GLYPH<SM590000> Less overhead in the end-to-end flows might improve scoring time.
GLYPH<SM590000> If you are using models that are not deployable, CP4D offers a custom Python run time to build your own stack if they are not available on the platform.
- GLYPH<SM590000> If you are using models that are not deployable, CP4D offers a custom Python run time to build your own stack if they are not available on the platform.
GLYPH<SM590000> AI inferencing based on ML or DL models can increase the accuracy of better credit risk assessment.
- GLYPH<SM590000> AI inferencing based on ML or DL models can increase the accuracy of better credit risk assessment.
GLYPH<SM590000> Using IBM z16 and on-chip AI acceleration with the Telum chip that is embedded with regular Integrated Facility for Linux (IFLs) provides an execution speed for your transactions that cannot be achieved by other means.
- GLYPH<SM590000> Using IBM z16 and on-chip AI acceleration with the Telum chip that is embedded with regular Integrated Facility for Linux (IFLs) provides an execution speed for your transactions that cannot be achieved by other means.
## Use case 3: Clearing and settlement
@@ -433,49 +433,49 @@ Figure 21 Clearing and settlement use case for financial transactions by using C
Here are the steps of the high-level process flow:
1. Create a connection to a database (for example, an IBM Db2fi database) where the historical data will be used for ML model building.
- 1. Create a connection to a database (for example, an IBM Db2fi database) where the historical data will be used for ML model building.
2. Read the data from the database and prepare the data for AI by using the Data Refinery tool in CP4D.
- 2. Read the data from the database and prepare the data for AI by using the Data Refinery tool in CP4D.
3. A Jupyter Notebook or JupyterLab IDE that is provided by the Watson Studio component in CP4D helps us build and train the AI model. The trained model can be saved into a WML repository.
- 3. A Jupyter Notebook or JupyterLab IDE that is provided by the Watson Studio component in CP4D helps us build and train the AI model. The trained model can be saved into a WML repository.
4. Deploy the saved model into a deployment space for batch deployment.
- 4. Deploy the saved model into a deployment space for batch deployment.
5. Create a batch deployment by using any of these interfaces:
- 5. Create a batch deployment by using any of these interfaces:
a. Watson Studio user interface from an Analytics deployment space.
- a. Watson Studio user interface from an Analytics deployment space.
b. WML Python client.
- b. WML Python client.
c. WML REST APIs.
- c. WML REST APIs.
6. A hardware configuration can be chosen for the deployment.
- 6. A hardware configuration can be chosen for the deployment.
7. A batch deployment processes input data from a file, data connection, or connected data in a storage bucket, and writes the output to a selected destination.
- 7. A batch deployment processes input data from a file, data connection, or connected data in a storage bucket, and writes the output to a selected destination.
8. One way to run batch deployment to predict or score is to create and run a batch deployment job.
- 8. One way to run batch deployment to predict or score is to create and run a batch deployment job.
9. Provide an input data type:
- 9. Provide an input data type:
a. Inline data for entering a JSON format payload.
- a. Inline data for entering a JSON format payload.
b. Select Data asset , click Select data source , and then specify your asset.
- b. Select Data asset , click Select data source , and then specify your asset.
10.The output data type can be a new output file or a connected data asset.
- 10.The output data type can be a new output file or a connected data asset.
11.A Kubernetes admin can change the maximum number of concurrent batch jobs that can be run.
- 11.A Kubernetes admin can change the maximum number of concurrent batch jobs that can be run.
12.Get the deployment endpoint URL. For more information, see Getting the deployment endpoint URL.
- 12.Get the deployment endpoint URL. For more information, see Getting the deployment endpoint URL.
## Summary
With this use case, we attempted to demonstrate how to predict, in real time, whether the transaction that is being processed might be a fraudulent transaction or not. By using the method, you have the following advantages:
GLYPH<SM590000> No Impact to SLAs and the batch process window.
- GLYPH<SM590000> No Impact to SLAs and the batch process window.
GLYPH<SM590000> Proactively stop losses, and lower operational, regulatory, and compliance costs.
- GLYPH<SM590000> Proactively stop losses, and lower operational, regulatory, and compliance costs.
GLYPH<SM590000> The solution is using a DL framework like TensorFlow for high-performing, low latency scoring.
- GLYPH<SM590000> The solution is using a DL framework like TensorFlow for high-performing, low latency scoring.
## Use case 4: Remaining Useful Life of an aircraft engine
@@ -511,15 +511,15 @@ Figure 23 In-depth architectural view
In summary, consider the following points while developing an AI-based predictive maintenance application:
GLYPH<SM590000> CP4D offers a Python run time to build a custom solution stack, but also supports different components like Watson Studio, WML, Db2, Data Refinery, OpenScale, AI Factsheets, and OpenPages.
- GLYPH<SM590000> CP4D offers a Python run time to build a custom solution stack, but also supports different components like Watson Studio, WML, Db2, Data Refinery, OpenScale, AI Factsheets, and OpenPages.
GLYPH<SM590000> The trustworthiness of the predicted output is important for critical use cases.
- GLYPH<SM590000> The trustworthiness of the predicted output is important for critical use cases.
GLYPH<SM590000> IBM Z provides high data security and low latency requirements at scale for the critical applications.
- GLYPH<SM590000> IBM Z provides high data security and low latency requirements at scale for the critical applications.
GLYPH<SM590000> A data scientist can choose to train the model and deploy it on CP4D seamlessly with the latest tech stack that is available.
- GLYPH<SM590000> A data scientist can choose to train the model and deploy it on CP4D seamlessly with the latest tech stack that is available.
GLYPH<SM590000> The AIOps and MLOps supported by CP4D to track AI model and data lifecycle throughout the application lifecycle.
- GLYPH<SM590000> The AIOps and MLOps supported by CP4D to track AI model and data lifecycle throughout the application lifecycle.
## Use case 5: AI-powered video analytics on an infant's motions for health prediction
@@ -558,29 +558,29 @@ When changes in body posture or movement happen, analytics can be performed, and
We can leverage the following AI technology stack for this use case:
GLYPH<SM590000> Convolutional neural network: Build an artificial neural network model on video streaming and images.
- GLYPH<SM590000> Convolutional neural network: Build an artificial neural network model on video streaming and images.
GLYPH<SM590000> TensorFlow: A DL back-end framework that is based on TensorFlow.
- GLYPH<SM590000> TensorFlow: A DL back-end framework that is based on TensorFlow.
GLYPH<SM590000> Mediapipe: A library that helps with video streaming processing and prediction of human pose estimation.
- GLYPH<SM590000> Mediapipe: A library that helps with video streaming processing and prediction of human pose estimation.
GLYPH<SM590000> OpenCV: A real-time computer vision library that helps perform image processing.
- GLYPH<SM590000> OpenCV: A real-time computer vision library that helps perform image processing.
WML was used for deployment of the pose detection model and generated notifications to users with web and mobile applications, and it integrates with Fitbit for push notifications so that hospitals and parents can take preventive actions.
## Additional resources
GLYPH<SM590000> The Cloud Pak for Data 4.5 on IBM Z Overview Demo video provides an overview of some of the more important features of CP4D on IBM Z.
- GLYPH<SM590000> The Cloud Pak for Data 4.5 on IBM Z Overview Demo video provides an overview of some of the more important features of CP4D on IBM Z.
GLYPH<SM590000> IBM Cloud Pak for Data Tutorials.
- GLYPH<SM590000> IBM Cloud Pak for Data Tutorials.
GLYPH<SM590000> Here are some additional use cases that use the data science frameworks that are available as part of CP4D on IBM Z and IBM LinuxONE:
- GLYPH<SM590000> Here are some additional use cases that use the data science frameworks that are available as part of CP4D on IBM Z and IBM LinuxONE:
-Payment Card Fraud Detection by using TensorFlow on CP4D on IBM Z and IBM LinuxONE is a payment card fraud detection use case.
- -Payment Card Fraud Detection by using TensorFlow on CP4D on IBM Z and IBM LinuxONE is a payment card fraud detection use case.
-Fashion-MNIST clothing classification with PyTorch on Cloud Pak for Data on IBM Z and IBM LinuxONE is a Fashion-MNIST clothing classification use case.
- -Fashion-MNIST clothing classification with PyTorch on Cloud Pak for Data on IBM Z and IBM LinuxONE is a Fashion-MNIST clothing classification use case.
-Payment Card Fraud Prevention by using Snap ML on IBM Cloud Pak for Data on Red Hat OpenShift on a virtual machine on IBM Z and IBM LinuxONE, which leverage the z16 integrated AI accelerator describes a use case that uses Snap Machine Learning in Cloud Pak for Data on IBM Z and IBM LinuxONE. It is a Snap ML use case.
- -Payment Card Fraud Prevention by using Snap ML on IBM Cloud Pak for Data on Red Hat OpenShift on a virtual machine on IBM Z and IBM LinuxONE, which leverage the z16 integrated AI accelerator describes a use case that uses Snap Machine Learning in Cloud Pak for Data on IBM Z and IBM LinuxONE. It is a Snap ML use case.
A companion video can be found at Credit Card Fraud Detection by using Snap ML on IBM Cloud Pak for Data on IBM Z and IBM LinuxONE.
@@ -618,15 +618,15 @@ ibm.com /redbooks/residencies.html
## Stay connected to IBM Redbooks
GLYPH<SM590000> Find us on LinkedIn:
- GLYPH<SM590000> Find us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
GLYPH<SM590000> Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks weekly newsletter:
- GLYPH<SM590000> Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks weekly newsletter:
https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
- https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
GLYPH<SM590000> Stay current on recent Redbooks publications with RSS Feeds:
- GLYPH<SM590000> Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html