Semantic search over your own text with Pinecone

What you get

A Python script that creates a Pinecone document index, embeds your text with Pinecone Inference, and returns the closest chunks for a natural-language question.

Prerequisites

Python 3.10+ and Pinecone Python SDK v10 or later
A Pinecone API key from https://app.pinecone.io

Steps

Install and set your key:

Shell


pip install --upgrade pinecone

export PINECONE_API_KEY="YOUR_API_KEY"

Create the index (dense vector field sized for llama-text-embed-v2):

Python


import os, time

from pinecone import Pinecone, SchemaBuilder



pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])



schema = (

    SchemaBuilder()

      .add_dense_vector_field(name="embedding", dimension=1024, metric="cosine")

      .build()

)

if not pc.indexes.exists(name="my-files"):

    pc.indexes.create(name="my-files", schema=schema)



while not pc.indexes.describe(name="my-files").status.ready:

    time.sleep(2)



index = pc.Index(name="my-files")

Chunk your documents (replace docs with your own text):

Python


docs = [

    {"id": "refunds", "text": "Refunds are available within 30 days of purchase for unused items."},

    {"id": "shipping", "text": "Standard shipping takes 3-5 business days."},

]



def chunk(text, size=500):

    return [text[i:i + size] for i in range(0, len(text), size)]



chunks = [

    {"id": f"{d['id']}#{i}", "text": part}

    for d in docs

    for i, part in enumerate(chunk(d["text"]))

]

Embed and upsert:

Python


embeddings = []

for i in range(0, len(chunks), 96):

    resp = pc.inference.embed(

        model="llama-text-embed-v2",

        inputs=[c["text"] for c in chunks[i:i + 96]],

        parameters={"input_type": "passage"},

    )

    embeddings.extend(resp)



index.documents.upsert(

    namespace="__default__",

    documents=[

        {"_id": c["id"], "embedding": e['values'], "text": c["text"]}

        for c, e in zip(chunks, embeddings)

    ],

)

Search by meaning:

Python


q = pc.inference.embed(

    model="llama-text-embed-v2",

    inputs=["what is the refund policy?"],

    parameters={"input_type": "query"},

)



resp = index.documents.search(

    namespace="__default__",

    top_k=3,

    score_by=[{"type": "dense_vector", "fields": ["embedding"], "values": q[0]['values']}],

    include_fields=["*"],

)



for m in resp.matches:

    print(m._id, m._score, getattr(m, "text", ""))

Test it

Run the script (wait a few seconds after upsert before searching). The refunds#0 chunk should come back first with the highest score.