* feat<table model>: add tablemaster with paddleocr to detect and recognize table (#493) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by: sfk <18810651050@163.com> * Update FAQ_zh_cn.md add new issue * Update FAQ_en_us.md * Update README_Windows_CUDA_Acceleration_zh_CN.md * Update README_zh-CN.md * @Thepathakarpit has signed the CLA in opendatalab/MinerU#418 * Update cla.yml * feat: add tablemaster_paddle (#463) * Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by: sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml --------- Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> * <fix>(para_split_v2): index out of range issue of span_text first char (#396) Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#508) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by: sfk <18810651050@163.com> * Update FAQ_zh_cn.md add new issue * Update FAQ_en_us.md * Update README_Windows_CUDA_Acceleration_zh_CN.md * Update README_zh-CN.md * @Thepathakarpit has signed the CLA in opendatalab/MinerU#418 * Update cla.yml * feat: add tablemaster_paddle (#463) * Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by: sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml --------- Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> * <fix>(para_split_v2): index out of range issue of span_text first char (#396) Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * Update cla.yml * Delete .github/workflows/gpu-ci.yml * Update Huggingface and ModelScope links to organization account * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com> Co-authored-by: wangbinDL <wangbin_research@163.com> * feat<table model>: add tablemaster with paddleocr to detect and recognize table (#511) * Update cla.yml * Update bug_report.yml * Update README_zh-CN.md (#404) correct FAQ url * Update README_zh-CN.md (#404) (#409) (#410) correct FAQ url Co-authored-by: sfk <18810651050@163.com> * Update FAQ_zh_cn.md add new issue * Update FAQ_en_us.md * Update README_Windows_CUDA_Acceleration_zh_CN.md * Update README_zh-CN.md * @Thepathakarpit has signed the CLA in opendatalab/MinerU#418 * Update cla.yml * feat: add tablemaster_paddle (#463) * Update README_zh-CN.md (#404) (#409) correct FAQ url Co-authored-by: sfk <18810651050@163.com> * add dockerfile (#189) Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> * Update cla.yml * Update cla.yml --------- Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> * <fix>(para_split_v2): index out of range issue of span_text first char (#396) Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> * @Matthijz98 has signed the CLA in opendatalab/MinerU#467 * Create download_models.py * Create requirements-docker.txt * feat<table model>: add tablemaster with paddleocr to detect and recognize table * @strongerfly has signed the CLA in opendatalab/MinerU#487 * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * Update cla.yml * Delete .github/workflows/gpu-ci.yml * Update Huggingface and ModelScope links to organization account * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table * feat<table model>: add tablemaster with paddleocr to detect and recognize table --------- Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> Co-authored-by: yyy <102640628+dt-yy@users.noreply.github.com> Co-authored-by: wangbinDL <wangbin_research@163.com> --------- Co-authored-by: Kaiwen Liu <lkw_buaa@163.com> Co-authored-by: Xiaomeng Zhao <moe@myhloli.com> Co-authored-by: sfk <18810651050@163.com> Co-authored-by: drunkpig <60862764+drunkpig@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Aoyang Fang <222010547@link.cuhk.edu.cn> Co-authored-by: liukaiwen <liukaiwen@pjlab.org.cn> Co-authored-by: wangbinDL <wangbin_research@163.com>
4.1 KiB
Windows 10/11
1. Install CUDA and cuDNN
Required versions: CUDA 11.8 + cuDNN 8.7.0
- CUDA 11.8: https://developer.nvidia.com/cuda-11-8-0-download-archive
- cuDNN v8.7.0 (November 28th, 2022), for CUDA 11.x: https://developer.nvidia.com/rdp/cudnn-archive
2. Install Anaconda
If Anaconda is already installed, you can skip this step.
Download link: https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Windows-x86_64.exe
3. Create an Environment Using Conda
Python version must be 3.10.
conda create -n MinerU python=3.10
conda activate MinerU
4. Install Applications
pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com
❗️After installation, verify the version of
magic-pdf:magic-pdf --versionIf the version number is less than 0.7.0, please report it in the issues section.
5. Download Models
Refer to detailed instructions on how to download model files.
After downloading, move the models directory to an SSD with more space.
❗ After downloading the models, ensure they are complete:
- Check that the file sizes match the description on the website.
- If possible, verify the integrity using SHA256.
6. Configuration Before the First Run
Obtain the configuration template file magic-pdf.template.json from the repository root directory.
❗️Execute the following command to copy the configuration file to your user directory, or the program will not run.
In Windows, user directory is "C:\Users\username"
(New-Object System.Net.WebClient).DownloadFile('https://github.com/opendatalab/MinerU/raw/master/magic-pdf.template.json', 'magic-pdf.template.json')
cp magic-pdf.template.json ~/magic-pdf.json
Find the magic-pdf.json file in your user directory and configure "models-dir" to point to the directory where the model weights from step 5 were downloaded.
❗️Ensure the absolute path of the model weights directory is correctly configured, or the program will fail to run due to not finding the model files.
In Windows, this path should include the drive letter and replace all
"\"to"/".Example: If the models are placed in the root directory of drive D, the value for
model-dirshould be"D:/models".
{
"models-dir": "/tmp/models"
}
7. First Run
Download a sample file from the repository and test it.
(New-Object System.Net.WebClient).DownloadFile('https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf', 'small_ocr.pdf')
magic-pdf -p small_ocr.pdf
8. Test CUDA Acceleration
If your graphics card has at least 8GB of VRAM, follow these steps to test CUDA-accelerated parsing performance.
-
Overwrite the installation of torch and torchvision supporting CUDA.
pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118❗️Ensure the following versions are specified in the command:
torch==2.3.1 torchvision==0.18.1These are the highest versions we support. Installing higher versions without specifying them will cause the program to fail.
-
Modify the value of
"device-mode"in themagic-pdf.jsonconfiguration file located in your user directory.{ "device-mode": "cuda" } -
Run the following command to test CUDA acceleration:
magic-pdf -p small_ocr.pdf
9. Enable CUDA Acceleration for OCR
❗️This operation requires at least 16GB of VRAM on your graphics card, otherwise it will cause the program to crash or slow down.
- Download paddlepaddle-gpu, which will automatically enable OCR acceleration upon installation.
pip install paddlepaddle-gpu==2.6.1 - Run the following command to test OCR acceleration:
magic-pdf -p small_ocr.pdf