当前位置：首页 > news >正文

LangChain：向量存储和检索器（入门篇三）

news 2025/7/6 15:19:18

前言

向量存储和检索器支持从（向量）数据库和其他来源检索数据，以便与大型语言模型工作流集成。它们对于获取数据以进行推理的应用程序非常重要，例如在检索增强生成（RAG）的情况下。

前言

一、环境篇

二、文档

三、向量存储

四、检索器

五、一个简单的RAG案例

一、环境篇

pip install langchain langchain-chroma langchain-openai

准备大模型apikey（本文基于硅基流动提供的apikey）

硅基流动用户系统，统一登录 SSO

model.py

from langchain_openai import ChatOpenAI
from langchain_openai import OpenAIEmbeddings# 1、创建模型
model = ChatOpenAI(model='Pro/deepseek-ai/DeepSeek-V3',base_url="https://api.siliconflow.cn/v1/",api_key="sk-xxx"
)emb =OpenAIEmbeddings(model="BAAI/bge-m3",base_url="https://api.siliconflow.cn/v1/",api_key="sk-xxx"
)

二、文档

angChain 实现了一个文档抽象，旨在表示一个文本单元及其相关元数据。它有两个属性：

page_content：一个表示内容的字符串；
metadata：一个包含任意元数据的字典。

metadata 属性可以捕获有关文档来源、与其他文档的关系以及其他信息。请注意，单个 Document 对象通常表示一个较大文档的一部分。

以下是一个文档的示例

from langchain_core.documents import Documentdocuments = [Document(page_content="Dogs are great companions, known for their loyalty and friendliness.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Cats are independent pets that often enjoy their own space.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Goldfish are popular pets for beginners, requiring relatively simple care.",metadata={"source": "fish-pets-doc"},),Document(page_content="Parrots are intelligent birds capable of mimicking human speech.",metadata={"source": "bird-pets-doc"},),Document(page_content="Rabbits are social animals that need plenty of space to hop around.",metadata={"source": "mammal-pets-doc"},),
]

三、向量存储

量搜索是一种常见的存储和搜索非结构化数据（例如非结构化文本）的方法。其思想是存储与文本相关联的数值向量。给定一个查询，我们可以将其嵌入为相同维度的向量，并使用向量相似性度量来识别存储中的相关数据。

LangChain 向量存储对象包含用于将文本和 文档 对象添加到存储中以及使用各种相似性度量进行查询的方法。它们通常使用嵌入模型进行初始化，这决定了文本数据如何转换为数值向量。

LangChain 包含与不同向量存储技术的集成套件。一些向量存储由提供商（例如各种云提供商）托管，并需要特定的凭据才能使用；一些（如 Postgres）在可以本地运行或通过第三方运行的独立基础设施中运行；其他可以在内存中运行以处理轻量级工作负载。这里我们将演示使用 Chroma 的 LangChain 向量存储，它包括一个内存实现。

要实例化一个向量存储，我们通常需要提供一个嵌入模型，以指定文本应如何转换为数值向量。这里我们将使用 OpenAI 嵌入。

from langchain_core.documents import Documentfrom langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from model import model, embdocuments = [Document(page_content="Dogs are great companions, known for their loyalty and friendliness.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Cats are independent pets that often enjoy their own space.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Goldfish are popular pets for beginners, requiring relatively simple care.",metadata={"source": "fish-pets-doc"},),Document(page_content="Parrots are intelligent birds capable of mimicking human speech.",metadata={"source": "bird-pets-doc"},),Document(page_content="Rabbits are social animals that need plenty of space to hop around.",metadata={"source": "mammal-pets-doc"},),
]vectorstore = Chroma.from_documents(documents,embedding=emb,
)
#根据与字符串查询的相似性返回文档：
similarity = vectorstore.similarity_search("cat")
print("相似性查询:", similarity)
#返回分数：
score = vectorstore.similarity_search_with_score("cat")
print("相似性分数", score)
#根据与嵌入查询的相似性返回文档：
embedding = emb.embed_query("cat")
search_by_vector = vectorstore.similarity_search_by_vector(embedding)
print("根据与嵌入查询的相似性返回文档", search_by_vector)

相似性查询: [Document(id='a9a53586-732f-4cdf-859c-1dccda1a823a', metadata={'source': 'mammal-pets-doc'}, page_content='Rabbits are social animals that need plenty of space to hop around.'), Document(id='ff0ad8cc-3bd1-4ca9-835a-629d90019f25', metadata={'source': 'bird-pets-doc'}, page_content='Parrots are intelligent birds capable of mimicking human speech.'), Document(id='c6b774fc-3dea-4d02-9981-d8fee828084d', metadata={'source': 'mammal-pets-doc'}, page_content='Dogs are great companions, known for their loyalty and friendliness.'), Document(id='022afbe9-3f7f-4b63-9370-db16f11ca2aa', metadata={'source': 'mammal-pets-doc'}, page_content='Cats are independent pets that often enjoy their own space.')]
相似性分数 [(Document(id='a9a53586-732f-4cdf-859c-1dccda1a823a', metadata={'source': 'mammal-pets-doc'}, page_content='Rabbits are social animals that need plenty of space to hop around.'), 0.785333514213562), (Document(id='ff0ad8cc-3bd1-4ca9-835a-629d90019f25', metadata={'source': 'bird-pets-doc'}, page_content='Parrots are intelligent birds capable of mimicking human speech.'), 0.8099995851516724), (Document(id='c6b774fc-3dea-4d02-9981-d8fee828084d', metadata={'source': 'mammal-pets-doc'}, page_content='Dogs are great companions, known for their loyalty and friendliness.'), 0.8387392163276672), (Document(id='022afbe9-3f7f-4b63-9370-db16f11ca2aa', metadata={'source': 'mammal-pets-doc'}, page_content='Cats are independent pets that often enjoy their own space.'), 0.858168363571167)]
根据与嵌入查询的相似性返回文档 [Document(id='a9a53586-732f-4cdf-859c-1dccda1a823a', metadata={'source': 'mammal-pets-doc'}, page_content='Rabbits are social animals that need plenty of space to hop around.'), Document(id='ff0ad8cc-3bd1-4ca9-835a-629d90019f25', metadata={'source': 'bird-pets-doc'}, page_content='Parrots are intelligent birds capable of mimicking human speech.'), Document(id='c6b774fc-3dea-4d02-9981-d8fee828084d', metadata={'source': 'mammal-pets-doc'}, page_content='Dogs are great companions, known for their loyalty and friendliness.'), Document(id='022afbe9-3f7f-4b63-9370-db16f11ca2aa', metadata={'source': 'mammal-pets-doc'}, page_content='Cats are independent pets that often enjoy their own space.')]

四、检索器

LangChain VectorStore 对象不继承 Runnable，因此无法立即集成到 LangChain 表达式 chains 中。

LangChain 检索器是 Runnables，因此它们实现了一组标准方法（例如，同步和异步的 invoke 和 batch 操作），并设计为可以纳入 LCEL 链中。

向量存储实现了一个 as_retriever 方法，该方法将生成一个检索器，特别是一个 VectorStoreRetriever。这些检索器包括特定的 search_type 和 search_kwargs 属性，用于识别调用底层向量存储的方法，以及如何对其进行参数化。

from langchain_core.documents import Document
from langchain_core.runnables import RunnableLambdafrom langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from model import model, embdocuments = [Document(page_content="Dogs are great companions, known for their loyalty and friendliness.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Cats are independent pets that often enjoy their own space.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Goldfish are popular pets for beginners, requiring relatively simple care.",metadata={"source": "fish-pets-doc"},),Document(page_content="Parrots are intelligent birds capable of mimicking human speech.",metadata={"source": "bird-pets-doc"},),Document(page_content="Rabbits are social animals that need plenty of space to hop around.",metadata={"source": "mammal-pets-doc"},),
]vectorstore = Chroma.from_documents(documents,embedding=emb,
)
# #根据与字符串查询的相似性返回文档：
# similarity = vectorstore.similarity_search("cat")
# print("相似性查询:", similarity)
# #返回分数：
# score = vectorstore.similarity_search_with_score("cat")
# print("相似性分数", score)
# #根据与嵌入查询的相似性返回文档：
# embedding = emb.embed_query("cat")
# search_by_vector = vectorstore.similarity_search_by_vector(embedding)
# print("根据与嵌入查询的相似性返回文档", search_by_vector)retriever = vectorstore.as_retriever(search_type="similarity",search_kwargs={"k": 1},
)
result = retriever.batch(["cat", "shark"])
print(result)

[[Document(id='a2b6ecfa-873b-4110-9e81-5c74b456e999', metadata={'source': 'mammal-pets-doc'}, page_content='Rabbits are social animals that need plenty of space to hop around.')], [Document(id='56fab89d-8313-490a-b2a8-54ea593b955c', metadata={'source': 'mammal-pets-doc'}, page_content='Dogs are great companions, known for their loyalty and friendliness.')]]

五、一个简单的RAG案例

检索器可以轻松地纳入更复杂的应用程序，例如检索增强生成（RAG）应用程序，这些应用程序将给定问题与检索到的上下文结合成 LLM 的提示。

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.documents import Document
from langchain_chroma import Chroma
from langchain_core.output_parsers import StrOutputParserfrom model import model, embdocuments = [Document(page_content="Dogs are great companions, known for their loyalty and friendliness.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Cats are independent pets that often enjoy their own space.",metadata={"source": "mammal-pets-doc"},),Document(page_content="Goldfish are popular pets for beginners, requiring relatively simple care.",metadata={"source": "fish-pets-doc"},),Document(page_content="Parrots are intelligent birds capable of mimicking human speech.",metadata={"source": "bird-pets-doc"},),Document(page_content="Rabbits are social animals that need plenty of space to hop around.",metadata={"source": "mammal-pets-doc"},),
]vectorstore = Chroma.from_documents(documents,embedding=emb,
)retriever = vectorstore.as_retriever(search_type="similarity",search_kwargs={"k": 1},
)message = """
Answer this question using the provided context only.{question}Context:
{context}
"""prompt = ChatPromptTemplate.from_messages([("human", message)])rag_chain = {"context": retriever, "question": RunnablePassthrough()} | prompt | modelresponse = rag_chain.invoke("tell me about cats")print(response.content)

Based on the provided context, there is no information about cats. The context only mentions dogs and their characteristics as loyal and friendly companions.

Would you like me to search for information about cats from another source?