- Update README.md and README_zh-CN.md to include new model download instructions
- Provide detailed steps on how to download models after PDF-Extract-Kit 1.0 repository change
- Emphasize the need to re-download models due to repository change
- Update README.md and README_zh-CN.md to include new model download instructions
- Provide detailed steps on how to download models after PDF-Extract-Kit 1.0 repository change
- Emphasize the need to re-download models due to repository change
- Remove import and usage of StructTableModel- Add support for TableMaster model- Update table model initialization logic to support TableMaster
- Log error and exit if StructEqTable is selected, as it's under upgrade
- Update README files to reflect changes in table parsing capabilities
- Change the logo path from 'docs/images/MinerU-logo.png' to 'old_docs/images/MinerU-logo.png' in both README.md and README_zh-CN.md- This update ensures that the correct logo is displayed in the project's README files
- Add changelog for v0.9.0 release with major refactoring and improvements
- Update key features list to include new functionalities
- Modify system requirements and hardware support information
- Add section for deploying derived projects
- Update known issues and TODO list
- Improve reading order with model-based sorting- Add list recognition within text
- Implement table of contents recognition
- Support table recognition
- Enhance code block and geometric shape recognition
- Address known issues in both English and Chinese READMEs
- Update GPU hardware support information in README.md and README_zh-CN.md
- Enhance CUDA acceleration guides for Ubuntu and Windows
- Modify README_zh-CN.md to reflect changes in GPU requirements and configurations
- Update TODO list to mark semantic reading order as completed
Added a link to the layoutreader repository in the Related Projects sections of both the README.md and README_zh-CN.md files. This addition helps to provide users with more resources and tools related to document layout analysis and processing.
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
* fix(ocr_mkcontent): improve language detection and content formatting (#458)
Optimize the language detection logic to enhance content formatting. This
change addresses issues with long word segmentation. Language detection now uses a
threshold to determine the language of a text based on the proportion of English characters.
Formatting rules for content have been updated to consider a list of languages (initially
including Chinese, Japanese, and Korean) where no space is added between content segments
for inline equations and text spans, improving the handling of Asian languages.
The impact of these changes includes improved accuracy in language detection, better
segmentation of long words, and more appropriate spacing in content formatting for multiple
languages.
* fix(self_modify): merge detection boxes for optimized text region detection (#448)
Merge adjacent and overlapping detection boxes to optimize text region detection in
the document. Post processing of text boxes is enhanced by consolidating them into
larger text lines, taking into account their vertical and horizontal alignment. This
improvement reduces fragmentation and improves the readability of detected text blocks.
* fix(pdf-extract): adjust box threshold for OCR detection (#447)
Tuned the detection box threshold parameter in the OCR model initialization to improve the
accuracy of text extraction from images. The threshold was modified from 0.6 to
0.3 to filter out smaller detection boxes, which is expected to enhance the quality of the extracted
text by reducing noise and false positives in the OCR process.
* feat: rename the file generated by command line tools (#401)
* feat: rename the file generated by command line tools
* feat: add pdf filename as prefix to {span,layout,model}.pdf
---------
Co-authored-by: icecraft <tmortred@gmail.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* fix(ocr_mkcontent): revise table caption output (#397)
* fix(ocr_mkcontent): revise table caption output
- Ensuring that
table captions are properly included in the output.
- Remove the redundant `table_caption` variable。
* Update cla.yml
* Update bug_report.yml
* feat(cli): add debug option for detailed error handling
Enable users to invoke the CLI command with a new debug flag to get detailed debugging information.
* fix(pdf-extract-kit): adjust crop_paste parameters for better accuracyThe crop_paste_x and crop_paste_y values in the pdf_extract_kit.py have been modified
to improve the accuracy and consistency of OCR processing. The new values are set to 25
to ensure more precise image cropping and pasting which leads to better OCR recognition
results.
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* fix(pdf-extract-kit): increase crop_paste margin for OCR processingDouble the crop_paste margin from25 to 50 to ensure better OCR accuracy and
handling of border cases. This change will help in improving the overall quality of
OCR'ed text by providing more context around the detected text areas.
* fix(common): deep copy model list before drawing model bbox
Use a deep copy of the original model list in `drow_model_bbox` to avoid potential
modifications to the source data. This ensures the integrity of the original models
is maintained while generating the model bounding boxes visualization.
---------
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* build(docker): update docker build step (#471)
* build(docker): update base image to Ubuntu 22.04 and install PaddlePaddleUpgrade the Docker base image from ubuntu:latest to ubuntu:22.04 for improved
performance and stability.
Additionally, integrate PaddlePaddle GPU version 3.0.0b1
into the Docker build for enhanced AI capabilities. The MinIO configuration file has
also been updated to the latest version.
* build(dockerfile): Updated the Dockerfile
* build(Dockerfile): update Dockerfile
* docs(docker): add instructions for quick deployment with Docker
Include Docker-based deployment instructions in the README for both English and
Chinese locales. This update provides users a quick-start guide to using Docker for
deployment, with notes on GPU VRAM requirements and default acceleration features.
* build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself.
* build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself.
* upload an introduction about chemical formula and update readme.md (#489)
* upload an introduction about chemical formula
* rename 2 files
* update readme.md at TODO in chemstery
* rename 2 files and update readme.md at TODO in chemstery
* update README_zh-CN.md at TODO in chemstery
* upload an introduction about chemical formula and update readme.md (#489)
* upload an introduction about chemical formula
* rename 2 files
* update readme.md at TODO in chemstery
* rename 2 files and update readme.md at TODO in chemstery
* update README_zh-CN.md at TODO in chemstery
* fix: remove the default value of output option in tools/cli.py and tools/cli_dev.py (#494)
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* feat: add test case (#499)
Co-authored-by: quyuan <quyuan@pjlab.org>
* Update cla.yml
* Update gpu-ci.yml
* Update cli.yml
* Delete .github/workflows/gpu-ci.yml
* fix(pdf-parse-union-core): #492 decrease span threshold for block filling (#500)
Reduce the span threshold used in fill_spans_in_blocks from 0.6 to 0.3 to
improve the accuracy of block filling based on layout analysis.
* fix(detect_all_bboxes): remove small overlapping blocks by merging (#501)
Previously, small blocks that overlapped with larger ones were merely removed. This fix
changes the approach to merge smaller blocks into the larger block instead, ensuring that
no information is lost and the larger block encompasses all the text content fully.
* feat(cli&analyze&pipeline): add start_page and end_page args for pagination (#507)
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* feat(cli&analyze&pipeline): add start_page and end_page args for paginationAdd start_page_id and end_page_id arguments to various components of the PDF parsing
pipeline to support pagination functionality. This feature allows users to specify the
range of pages to be processed, enhancing the efficiency and flexibility of the system.
* Feat/support rag (#510)
* Create requirements-docker.txt
* feat: update deps to support rag
* feat: add support to rag, add rag_data_reader api for rag integration
* feat: let user retrieve the filename of the processed file
* feat: add projects demo for rag integrations
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* Update Dockerfile
* feat(gradio): add app by gradio (#512)
* fix: replace \u0002, \u0003 in common text (#521)
* fix replace \u0002, \u0003 in common text
* fix(para): When an English line ends with a hyphen, do not add a space at the end.
* fix(end_page_id):Fix the issue where end_page_id is corrected to len-1 when its input is 0. (#518)
* fix(para): When an English line ends with a hyphen, do not add a space at the end. (#523)
* fix replace \u0002, \u0003 in common text
* fix(para): When an English line ends with a hyphen, do not add a space at the end.
* fix: delete hyphen at end of line
* Release: Release 0.7.1 verison, update dev (#527)
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#508)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
---------
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* Hotfix readme 0.7.1 (#529)
* release: release 0.7.1 version (#526)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#508)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511)
* Update cla.yml
* Update bug_report.yml
* Update README_zh-CN.md (#404)
correct FAQ url
* Update README_zh-CN.md (#404) (#409) (#410)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* Update FAQ_zh_cn.md
add new issue
* Update FAQ_en_us.md
* Update README_Windows_CUDA_Acceleration_zh_CN.md
* Update README_zh-CN.md
* @Thepathakarpit has signed the CLA in opendatalab/MinerU#418
* Update cla.yml
* feat: add tablemaster_paddle (#463)
* Update README_zh-CN.md (#404) (#409)
correct FAQ url
Co-authored-by: sfk <18810651050@163.com>
* add dockerfile (#189)
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
* Update cla.yml
* Update cla.yml
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
* <fix>(para_split_v2): index out of range issue of span_text first char (#396)
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
* @Matthijz98 has signed the CLA in opendatalab/MinerU#467
* Create download_models.py
* Create requirements-docker.txt
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* @strongerfly has signed the CLA in opendatalab/MinerU#487
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* Update cla.yml
* Delete .github/workflows/gpu-ci.yml
* Update Huggingface and ModelScope links to organization account
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
* feat<table model>: add tablemaster with paddleocr to detect and recognize table
---------
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: wangbinDL <wangbin_research@163.com>
---------
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* Update README.md
* Update README_zh-CN.md
* Update README_zh-CN.md
---------
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* Update README_zh-CN.md
delete Known issue about table recognition
* Update Dockerfile
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 (#542)
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: typo error in markdown (#536)
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* fix(gradio): remove unused imports and simplify pdf display (#534)
Removed the previously used gradio and gradio-pdf imports which were not leveraged in the code. Also,
replaced the custom `show_pdf` function with direct use of the `PDF` component from gradio for a simpler
and more integrated PDF upload and display solution, improving code maintainability and readability.
* Feat/support footnote in figure (#532)
* feat: support figure footnote
* feat: using the relative position to combine footnote, table, image
* feat: add the readme of projects
* fix: code spell in unittest
---------
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
* refactor(pdf_extract_kit): implement singleton pattern for atomic models (#533)
Refactor the pdf_extract_kit module to utilize a singleton pattern when initializing
atomic models. This change ensures that atomic models are instantiated at most once,
optimizing memory usage and reducing redundant initialization steps. The AtomModelSingleton
class now manages the instantiation and retrieval of atomic models, improving the
overall structure and efficiency of the codebase.
* Update README.md
* Update README_zh-CN.md
* Update README_zh-CN.md
add HF、modelscope、colab url
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update README_zh-CN.md
* Rename README.md to README_zh-CN.md
* Create readme.md
* Rename readme.md to README.md
* Rename README.md to README_zh-CN.md
* Update README_zh-CN.md
* Create README.md
* Update README.md
* Update README.md
* Update README.md
* Update README_zh-CN.md
* Update README.md
* Update README_zh-CN.md
* Update README_zh-CN.md
* Update README.md
* Update README_zh-CN.md
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384 (#573)
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* fix: resolve inaccuracy of drawing layout box caused by paragraphs combination #384
* Update README_zh-CN.md
* Update README.md
* Update README.md
* Update README.md
* Update README_zh-CN.md
* add rag data api
* Update README_zh-CN.md
update rag api image
* Update README.md
docs: remove RAG related release notes
* Update README_zh-CN.md
docs: remove RAG related release notes
* Update README_zh-CN.md
update 更新记录
---------
Co-authored-by: sfk <18810651050@163.com>
Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn>
Co-authored-by: Xiaomeng Zhao <moe@myhloli.com>
Co-authored-by: icecraft <tmortred@163.com>
Co-authored-by: icecraft <tmortred@gmail.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Siyu Hao <131659128+GDDGCZ518@users.noreply.github.com>
Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com>
Co-authored-by: quyuan <quyuan@pjlab.org>
Co-authored-by: Kaiwen Liu <lkw_buaa@163.com>
Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn>
Co-authored-by: wangbinDL <wangbin_research@163.com>
* upload an introduction about chemical formula
* rename 2 files
* update readme.md at TODO in chemstery
* rename 2 files and update readme.md at TODO in chemstery
* update README_zh-CN.md at TODO in chemstery
* build(docker): update base image to Ubuntu 22.04 and install PaddlePaddleUpgrade the Docker base image from ubuntu:latest to ubuntu:22.04 for improved
performance and stability.
Additionally, integrate PaddlePaddle GPU version 3.0.0b1
into the Docker build for enhanced AI capabilities. The MinIO configuration file has
also been updated to the latest version.
* build(dockerfile): Updated the Dockerfile
* build(Dockerfile): update Dockerfile
* docs(docker): add instructions for quick deployment with Docker
Include Docker-based deployment instructions in the README for both English and
Chinese locales. This update provides users a quick-start guide to using Docker for
deployment, with notes on GPU VRAM requirements and default acceleration features.
* build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself.
* build(docker): Layer the installation of dependencies, downloading the model, and the setup of the program itself.
* feat: rename the file generated by command line tools
* feat: add pdf filename as prefix to {span,layout,model}.pdf
---------
Co-authored-by: icecraft <tmortred@gmail.com>
Co-authored-by: icecraft <xurui1@pjlab.org.cn>
Reorganize the installation instructions for Magic-PDF to clarify the dependency on
detectron2 and provide a more straightforward installation process. The update includes
separating the dependency installation from the package installation and adding a note
about precompiled wheels for Python 3.10.
BREAKING CHANGE: The installation guide now assumes basic familiarity with detectron2
installation requirements. Users who need to compile detectron2 from source should refer
to the official detectron2 documentation.
Update the PyTorch installation command in the README files for both English and Chinese
versions to reflect the required version compatibility with CUDA 11.8. Include explicit
instructions to specify the PyTorch version to avoid automatic installation of higher,
unsupported versions. Additionally, clarify the importance of modifying the "device-mode"
parameter in the magic-pdf.json configuration file for proper CUDA device selection.
Update the PyTorch installation guide in both English and Chinese READMEs to explicitly
recommend using torch==2.3.1 and torchvision==0.18.1 for CUDA 11.8. Emphasize the
importance of specifying these versions to avoid compatibility issues with higher,
unsupported versions.
We have pre-released the 0.6.2 beta version, which addresses numerous issues
reported in our logs. This commit updates the installation guide in the README to
include information on how to install this beta version. Users are advised that
this build has not undergone full QA testing and may contain issues. A
revert instruction to version 0.6.1 is also provided for users who encounter
problems.
BREAKING CHANGE: Installation instructions now include beta version
information. Users should be aware of potential issues with the0.6.2 beta
version and consider reverting to 0.6.1 if necessary.
