Help: When I save the question and answer pairs into embeding database, how can I save the original chunk and label as meta data together at the same time?

[simonjhy]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

When I save the question and answer pairs into embeding database, how can I save the original chunk and label as meta data together at the same time?

2. Additional context or comments

No response

[dosubot[bot]]

You can save the original chunk and label as metadata with your question-answer pairs in the embedding database by adding them to the metadata dictionary of each Document object before the embedding is created. Dify’s pipeline (for example, in the QAIndexProcessor and DatasetDocumentStore) passes this metadata all the way through to the vector database, where it is stored as a JSON object alongside the embedding and the text chunk. This works with backends like PGVector, which has a dedicated meta column for arbitrary metadata.

Here’s a simplified example:

doc = Document(
    page_content=question_text,
    metadata={
        "original_chunk": original_chunk_text,
        "label": your_label,
        "answer": answer_text,
        # ...any other fields you want
    }
)
# Add doc to your document store and proceed with embedding as usual

When you call the vector creation method, this metadata will be persisted and retrievable with the embedding. You can later filter or query based on these metadata fields as well. For more details, see the code references for QAIndexProcessor, DatasetDocumentStore, and PGVector.

If you’re using the API, note that as of v1.1.0, you may need to update the metadata after document creation, since direct metadata saving during creation isn’t always supported yet. There are also feature requests to make segment-level metadata easier to use in future versions.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[simonjhy]

Do you mean I can not directly use Knowledge base node to save meta data and I can only manually write some codes to save meta data information instead of using Knowledge base node in pipeline workflow? I am using the dify-api and web with latest main version docker image, the other parts with v1.10.1 version.

Below is the part of pipeline I used to create knowledge workflow.

[dosubot[bot]]

You’re correct: the Knowledge Base node in Dify’s pipeline workflow does not support saving custom metadata (like original chunk or label) directly through its UI or configuration. Metadata is managed at the document and dataset level, so you need to update it after document creation using the metadata API or service—manual code or API calls are required for this step. There’s no built-in way to inject custom metadata during ingestion via the Knowledge Base node itself in either the latest main version or v1.10.1.

The recommended approach is to define the metadata fields you need and use the metadata service (or API) to attach your metadata to the documents after they’re created by the Knowledge Base node. Segment-level metadata (per chunk or Q&A pair) is a frequently requested feature and may be available in future releases, but it’s not supported yet in the open-source version. For more details, see the metadata service implementation and feature request discussion.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[simonjhy]

Can I use like this api "https://api.dify.ai/v1/datasets/{dataset_id}/documents/create-by-qa" to manually save such information together?
"question": item['question'],
"answer": item['content'],
"metadata": item['metadata']

Can you give me a guide line about how to call and use this type api to manually insert question and answer pairs data with meta information?

The onter question is if I can insert these meta data into knowledge embeding database, Can you tell me what steps I need to use these meta information in a chat pipeline application?

[simonjhy]

what api I can use to implement the add new question and answer pair with meta information at the same time?

[dosubot[bot]]

There isn't a single API call in Dify that lets you add a new question and answer pair with custom metadata (like original chunk or label) in one step. Instead, you need to use a two-step process:

1. Create the Q&A document using the document creation API (such as /v1/datasets/{dataset_id}/documents/create-by-text or similar). This will add your question and answer to the knowledge base, but without your custom metadata yet.
2. Update the document's metadata using the metadata update API (/v1/datasets/{dataset_id}/documents/metadata). Here, you specify the document ID and the metadata fields you want to attach (like original_chunk, label, etc.) — this metadata will then be stored and available for filtering and retrieval in downstream workflows.

Example for step 2 (updating metadata):

POST /v1/datasets/{dataset_id}/documents/metadata
{
  "document_id": "your_document_id",
  "metadata": [
    {"name": "original_chunk", "value": "the original chunk text"},
    {"name": "label", "value": "your label"}
  ]
}

Limitations: only document-level metadata is supported (not per Q&A pair/chunk), and you can't set metadata during the initial creation call — it must be updated after the document is created. Once attached, you can use these metadata fields for filtering or retrieval in your chat or RAG pipelines see code reference.

If you need segment-level metadata (per Q&A pair), that's not supported yet in the open-source version.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[simonjhy]

From your reply, I felt that every the question and answer pair does not support to have their private meta data and only the whole document(a pdf or a docx) can has metadata?

But I want to let every question and answer pair can map to a chunk. Does dify support this target?