I moved our search stack across a hard path: S3 vectors → Pinecone → PGVector, across two services, without breaking the core user experience.

From the outside, this sounds like a backend detail. From the inside, this is the difference between “search kind of works” and “search is reliable enough to trust.”Why I Made This Move

As we scaled, I had to ask a simple founder question: Are we adding complexity because it helps users, or because it’s the default architecture everyone copies?

We already run Postgres at the center of our system. User data, media metadata, jobs, auth , all there.

Keeping vectors far away in another runtime sounded flexible, but in reality it created friction:

• More moving parts

• Harder debugging

• Identity mismatches between metadata and vectors

• Too many places where failures could hide

PGVector wasn’t just a cost or tooling choice. It was a coherence choice.

When your vectors live near your source-of-truth data, the system gets easier to reason about.What Broke (and What It Taught Me)

The hard part wasn’t ANN math. The hard part was identity discipline.

We had multiple ID forms floating around:

• S3 object keys

• Generated vector IDs

• Canonical media IDs in Postgres

Those are not interchangeable.

In old flows, some embeddings were being written with IDs that looked valid but didn’t match relational truth. Once we enforced foreign keys in PGVector tables, those bad writes started failing loudly.

It hurt for a day. It saved us months of silent corruption.

That’s a founder lesson I keep relearning:Systems don’t fail because of one big wrong decision. They fail because of small naming mismatches no one thought were important.Why PGVector Won for Us (Right Now)

Not because it’s trendy. Because it matches our product reality.

• We need strong ownership boundaries per user.

• We need deterministic joins between vectors and metadata.

• We need one operational surface we can monitor deeply.

• We need to debug incidents quickly, not guess across multiple platforms.

PGVector gave us that.

Does this mean external vector systems are bad? No. It means for our current stage, operational simplicity beats architectural novelty.The Migration Principle I Care About Most

I didn’t want a one-way rewrite.

So I kept a provider abstraction in place:

• postgres for current primary

• s3 as switchable path

Why? Because founders don’t optimize for being “right forever.” We optimize for being adaptable without rewrites.

The goal is not “never change infra again.” The goal is: changing infra should not require rebuilding your company.What Changed Culturally in Our Engineering Process

This migration also reinforced a process shift for me:

• No more “it passed locally, ship it.”

• Every critical path gets deterministic smoke checks.

• Every bug class gets explicit logs.

• Every production-like failure gets turned into a codified test scenario.

And maybe most importantly: we now treat “debuggability” as a feature, not an afterthought.What I’m Proud Of

Not that we switched a vector backend. I’m proud that we did it while preserving product flow, learning from failures quickly, and making the system more understandable for the team after me.

Startups often celebrate speed. I still value speed. But I’m increasingly convinced that clarity is compounding.

Clear systems. Clear IDs. Clear ownership. Clear logs.

That’s what scales.

Vector Search: When Meaning Is Close

Vector search is where most GenAI journeys start, and for good reason.

It works by turning text into numbers and placing similar meanings close together in a mathematical space. When you search, the system retrieves things that feel similar in intent or language.

This is great for semantic matching.

Ask for “electric toothbrush” and you’ll get… electric toothbrushes.

But there’s a quiet limitation here.

Vector search struggles when two things are related but not similar.

A login event and a financial transaction don’t live near each other in vector space, even if both are part of the same fraud story.

Similarity isn’t the same as relevance.

Knowledge Graphs: When Meaning Is Connected

This is where graphs enter the picture.

A knowledge graph doesn’t care if two things sound alike. It cares about how they’re connected. Entities are nodes. Relationships are edges. Meaning emerges from how many “hops” it takes to get from one thing to another.

Graphs shine when:

• relationships matter more than wording

• context lives across systems

• the question is “how is this connected?” rather than “what is this similar to?”

Think social networks, supply chains, audits, investigations, anywhere cause, sequence, or dependency matters.

Scoring Isn’t a Single Step Anymore

The most important insight for me wasn’t vector vs graph.

It was how scoring actually happens in real AI systems.

Modern search is almost always hybrid.

Pattern 1: Vector first, graph later

You start with similarity. Grab the top results. Then expand through relationships.

This is how you recommend a lubrication gel after someone adds fissure meds to their cart.

Not because they’re similar, but because they belong to the same world.

Pattern 2: Graph first, vector later

You start wide. Pull a large set of related entities. Then rank them by similarity.

This is how “friends of friends” becomes “friends of friends you’ll actually get along with.”

In both cases, scoring isn’t just distance in vector space.

It’s distance plus structure.

Why GraphRAG Matters

The most convincing example was GraphRAG.

A system asked:

“What are the sales prospects for Company X in India?”

Vector-only retrieval found positive articles. The model answered optimistically and confidently wrong.

GraphRAG did something subtle:

• It found the company via vectors

• Then followed relationships the text never made explicit

• A logistics partner

• A canal

• A blockage

Suddenly, the answer changed. Not because of new text, but because of missing context now made visible.

This is the difference between sounding smart and being correct.

Where This Leaves Me

I’m starting to see AI search less as “better retrieval” and more as progressive understanding.

• Vectors help you enter the problem space

• Graphs help you navigate it

• The model synthesises only after both have done their job

This feels like the natural evolution of search:

from matching words

to matching meaning

to understanding systems.

This is just the beginning. The real work lies ahead in how we design these layers, how we control scoring, and how we prevent confidence without grounding.

That’s a journey that interests me next.

A small note on where I’m learning this from

A lot of my current thinking around AI search especially the
“retrieve → expand context → re-rank → answer”
shape is being influenced by the RAG-Anything work from HKUDS. Their framework diagram helped me see the pipeline as a system, not a single retrieval trick.

Framework diagram

Repository: https://github.com/HKUDS/RAG-Anything.git

Thanks:

Thank you to the HKUDS team and contributors, open diagrams and open repos like this make it easier for builders like me to learn in public, iterate fast, and stay honest about what we’re actually standing on.

The Problem

On paper, it sounds simple:

• Company GitHub account → already set up with an SSH key (~/.ssh/id_rsa), global Git config uses tushar@work.com.

• Personal GitHub account → I want a new SSH key, repos under a dedicated folder (~/tushar/), and commits showing my personal email.

In reality, Git doesn’t make it that easy:

• Identity bleed: Git’s user.name and user.email are global by default. If you don’t override them, your work email leaks into personal repos.

• SSH collisions: By default, GitHub only has one hostname: github.com. Which key it uses depends on your SSH config. Without care, everything routes through your company identity.

• Muscle memory: You don’t want to remember special clone commands. You want git clone to just work.

Implementation

I didn’t want a brittle workflow where I manually swap configs or run custom scripts. The goal was event-driven identity switching — Git should know which account to use simply based on where I’m working.

Here’s how I wired it up:

1. Two SSH Keys, Two Hosts

I generated a second SSH key for my personal account:

ssh-keygen -t ed25519 -C "me@gmail.com" -f ~/.ssh/id_rsa_personal

Then I taught SSH to treat GitHub as two different hosts:

vi ~/.ssh/config

# Default: Company
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa
  IdentitiesOnly yes

# Personal
Host github-personal
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_personal
  IdentitiesOnly yes

Now I can talk to GitHub through github.com (work) or github-personal (personal).

2. Directory-Based Git Config

Next problem: user.email. I wanted anything in ~/tushar/ to use my personal identity. Luckily, Git has an underrated feature called includeIf.

In ~/.gitconfig:

[user]
    name = Work User
    email = work@company.com

[includeIf "gitdir:/Users/tusharkhanka/tushar/**"]
    path = /Users/tusharkhanka/.gitconfig-personal

And then in ~/.gitconfig-personal:

[user]
    name = Tushar Khanka
    email = me@gmail.com

[url "git@github-personal:"]
    insteadOf = git@github.com:

This does two things at once:

• Inside ~/tushar/*, Git rewrites my identity to personal.

• Any git@github.com:... URL in that path is transparently rewritten to git@github-personal:..., so SSH uses the right key.

3. Testing Reality

Here’s the catch I stumbled on: includeIf only applies inside a real repo. Running git config user.email in ~/tushar/ will still show your global identity. That confused me for a while.

But once I cd’d into a real repo under ~/tushar/* and checked:

git config user.email
git config --show-origin user.email

I finally saw:

me@gmail.com
file:/Users/tusharkhanka/.gitconfig-personal

That was the “aha” moment.

Results

With this setup:

• Work repos under ~/workspace/* → automatically use my company email + company SSH key.

• Personal repos under ~/tushar/* → automatically use my personal email + personal SSH key.

• Cloning is seamless:

  • Work: git clone git@github.com:CompanyOrg/repo.git

  • Personal: git clone git@github.com:MyUser/repo.git (rewritten under the hood).

No more manual switching, no more accidents where a side project commit shows up with my company identity.

Lessons Learned

1. Git’s config system is powerful but picky.

   • gitdir:~/tushar/ doesn’t work, because ~ won’t expand.

   • You need the absolute path, and often /** to catch subdirectories.

2. SSH aliases are your friend.

   GitHub only has github.com, but nothing stops you from inventing github-personal or github-sideprojects to force key separation.

3. Don’t test in empty directories.

   includeIf only evaluates inside real repos. Test from within a .git directory, otherwise you’ll chase ghosts.

4. Pragmatism beats clever scripts.

   I first wrote a shell script to rewrite remotes, but the config approach was cleaner, automatic, and future-proof.

Conclusion

This wasn’t about elegance; it was about pragmatism. I wanted to prevent identity bleed between work and personal projects without adding cognitive overhead. The end result is boring in the best way: I don’t think about it anymore.

Scaling infrastructure often works the same way. You don’t need a silver bullet; you need guardrails that let you move fast without tripping yourself up. Git’s includeIf and SSH’s Host blocks gave me exactly that.

Now when I hack on a side project over the weekend, I know the commits belong to me, not my employer. That’s the kind of separation that keeps both sides happy.

TL;DR — Copy-Paste Setup

If you just want the working config, here it is:

~/.ssh/config

# Work (default)
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa
  IdentitiesOnly yes

# Personal
Host github-personal
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_personal
  IdentitiesOnly yes

~/.gitconfig

[user]
    name = Work User
    email = work@company.com

[includeIf "gitdir:/Users/yourusername/tushar/**"]
    path = /Users/yourusername/.gitconfig-personal

~/.gitconfig-personal

[user]
    name = Your Name
    email = your@email.com

[url "git@github-personal:"]
    insteadOf = git@github.com:

Usage

• Clone work repos normally:

git clone git@github.com:CompanyOrg/repo.git

• Clone personal repos the same way (Git rewrites it for you):

git clone git@github.com:MyUser/repo.git

Done. Now Git and SSH auto-switch identities based on directory — no manual swapping, no leaks.