---
title: Sync Data from Website
version: 'English'
---

Dify knowledge base supports crawling content from public web pages using third-party tools such as [Jina Reader](https://jina.ai/reader/) and [Firecrawl](https://www.firecrawl.dev/), parsing it into Markdown content, and importing it into the knowledge base.

<Note>
​[Firecrawl](https://www.firecrawl.dev/) and [Jina Reader](https://jina.ai/reader/) are both open-source web parsing tools that can convert web pages into clean Markdown format text that is easy for LLMs to recognize, while providing easy-to-use API services.Comment
</Note>

The following sections will introduce the usage methods for Firecrawl and Jina Reader respectively.

### Firecrawl <a href="#how-to-configure" id="how-to-configure"></a>

#### **1. Configure Firecrawl API credentials**

Click on the avatar in the upper right corner, then go to the **DataSource** page, and click the **Configure** button next to Firecrawl.

<Frame caption="Firecrawlの設定">
  <img src="/ja-jp/img/image (6) (2).png" alt="" />
</Frame>

Log in to the [Firecrawl website](https://www.firecrawl.dev/) to complete registration, get your API Key, and then enter and save it in Dify.

<Frame caption="">
  ![](/images/sync-from-firecrawl.png)
</Frame>

#### 2. Scrape target webpage

On the knowledge base creation page, select **Sync from website**, choose Firecrawl as the provider, and enter the target URL to be crawled.

<Frame caption="Webページの取得設定">
  <img src="/ja-jp/img/image (102).png" alt="" />
</Frame>

The configuration options include: Whether to crawl sub-pages, Page crawling limit, Page scraping max depth, Excluded paths, Include only paths, and Content extraction scope. After completing the configuration, click **Run** to preview the parsed pages.

<Frame caption="取得実行">
  <img src="/ja-jp/img/image (103).png" alt="" />
</Frame>

#### 3. Review import results

After importing the parsed text from the webpage, it is stored in the knowledge base documents. View the import results and click **Add URL** to continue importing new web pages.

<Frame caption="Webページ解析テキストのインポート">
  <img src="/ja-jp/img/image (104).png" alt="" />
</Frame>

***

### Jina Reader

#### 1. Configuring Jina Reader Credentials

Click on the avatar in the upper right corner, then go to the **DataSource** page, and click the **Configure** button next to Jina Reader.

<Frame>
  <img src="/images/assets/image (105).png" alt="Configuring Jina Reader" />
</Frame>

Log in to the [Jina Reader website](https://jina.ai/reader/), complete registration, obtain the API Key, then fill it in and save.

<Frame>
  <img src="/images/assets/image (106).png" alt="Fill in Jina configuration" />
</Frame>

#### 2. Using Jina Reader to Crawl Web Content

On the knowledge base creation page, select Sync from website, choose Jina Reader as the provider, and enter the target URL to be crawled.

<Frame>
  <img src="/images/assets/image (107).png" alt="Web Crawling Configuration" />
</Frame>

Configuration options include: whether to crawl subpages, maximum number of pages to crawl, and whether to use sitemap for crawling. After completing the configuration, click the **Run** button to preview the page links to be crawled.

<Frame>
  <img src="/images/assets/image (109).png" alt="Executing the Crawl" />
</Frame>

Import the parsed text from web pages and store it in the knowledge base documents, then view the import results. To continue adding web pages, click the Add URL button on the right to import new web pages.

<Frame>
  <img src="/images/assets/image (110).png" alt="Importing parsed web text into the knowledge base" />
</Frame>

After crawling is complete, the content from the web pages will be incorporated into the knowledge base.

{/*
Contributing Section
DO NOT edit this section!
It will be automatically generated by the script.
*/}

---

[Edit this page](https://github.com/langgenius/dify-docs/edit/main/en/guides/knowledge-base/knowledge-base-creation/sync-from-website.mdx) | [Report an issue](https://github.com/langgenius/dify-docs/issues/new?template=docs.yml)

