This is part 2 of a series on taking your MVP and turning it into a real, scalable product. In part 1, we shipped something that worked. Now let's make sure it keeps working

You built the MVP. It works. A few real users are using it. You're proud of it, and you should be. But It's time to take this to next level using Source Control practices

Source Control for AI?

In traditional software engineering, version control is non-negotiable. No one ships code without Git. AI systems operate differently, and simply tracking code may not be enough.

Classical ML systems - model weights change, training datasets evolve, hyperparameters get tuned. But most teams only version the training code, not the artifacts that actually matter.
Generative AI platforms - prompts are edited directly in production, few-shot examples are swapped informally, and RAG knowledge bases are updated silently. These changes can completely alter system behavior, yet they leave no audit trail.

Why This Matters in Production

When things go wrong -and they will -you need to answer three questions:

1. What changed? A model swap, a prompt edit, a new example in your dataset?

2. When did it change? Can you correlate a degradation in quality metrics with a specific change event?

3. Can you roll it back? Do you even have the previous version?

Without version control, the answer to all three is usually: we don’t know.

This creates real operational risk. A production LLM application with no prompt versioning is essentially running with no change management. Every edit is a direct push to prod.

Why Git Isn’t Enough

Git solves code versioning brilliantly. It does not solve AI system versioning.

Here’s what Git misses:

∙ Model weights - Binary artifacts, often gigabytes. Git doesn’t handle these well.
∙ Training data - Datasets are large, structured, and need lineage tracking, not just diffs.
∙ Hyperparameters and training configs -Often stored informally or not at all.
∙ Prompts and few-shot examples - Plain text, but semantically significant. A one-word change can have massive behavioral impact.
∙ Evaluation results - You need to tie a version of your system to its benchmark performance.

sure, you can use git to track them but that would only take you this far.

An AI system isn’t just code. It’s code ,data ,weights, configuration and prompts. You need a versioning strategy that covers all of it.

What the solution actually looks like

for a full change management system you would need to cover a few layers with different systems in place

use Git for the software layer
use AI-native tooling for artifacts Git does not handle well
tie both back to the same release and evaluation process

Git is still necessary. It should hold:

application code
prompts and prompt templates
model and inference config
infra definitions
eval runners
smaller eval datasets

But Git is not the right system for everything.

Where Git stops being enough

Once you are dealing with large datasets, trained models, fine-tuned checkpoints, experiment history, or prompt traces in production, plain Git starts breaking down.

That is where the rest of the stack comes in, and here are some few examples for these tools:

Experiment Tracking & Model Registry

MLflow for experiment tracking, model lineage, and registry workflows, or cloud-native solutions like Azure ML / Vertex AI

Data Versioning

DVC for dataset versioning and reproducible data pipelines

Model Registry (Production)

Central registry for promoted models, versioning, and deployment history

Observability (LLM & AI Systems)

Langfuse, LangSmith, or similar tools for tracing prompts, monitoring behavior, and linking outputs to model/prompt versions

Large File Management

Git LFS for handling large artifacts that don’t yet require full data pipeline tooling

The practical model is simple:

Git tells you what changed in the software layer
MLflow or your registry tells you which model artifact was trained, tested, and promoted
DVC tells you which dataset or pipeline state produced that artifact
Langfuse or similar tooling tells you what prompt, trace, and model combination actually ran in production

That combination is much closer to real AI versioning than a clean folder structure on its own.

A good rule of thumb

If the artifact is text and reviewed like code, Git is usually enough.

If the artifact is large, binary, experimental, or generated by a training pipeline, it probably belongs in a registry or artifact system.

Why this works

Prompts are files. Not inline strings buried inside application code. That makes diffs, reviews, and rollback straightforward.
Model configuration is explicit and pinned. gpt-4o-2024-08-06 is a version. gpt-4o is a moving target. If the provider changes what the alias resolves to, your system changes without a commit.
Artifacts have lineage. You can answer which dataset, which training run, which prompt version, and which model version produced the behavior now running in production.
Runtime is observable. When something regresses, you can inspect traces and connect them back to the exact prompt/model/config combination that ran.

here is an example of how prompt changes look like in prompt management system like MLFlow

Compare Prompt
Versions — source: MLFlow official documentations

A concrete example

Before

A small team is running an AI support workflow.

prompts are hardcoded in Python
the model is set to a floating alias
retrieval chunk size is controlled by an environment variable
the eval set lives in a spreadsheet

It all works well enough for the first launch.

What changed

The team grows from one engineer to three.

One engineer edits the classification prompt to catch a new ticket type. Another changes chunk size during performance testing and forgets to revert it. A few days later, the model alias resolves to a newer checkpoint. Output quality drops, but only on a subset of tickets.

Why debugging is painful

Now try debugging it.

The code diff shows one prompt edit. It does not show the old model checkpoint. It does not show the config drift between staging and production. It does not show that the eval set being used in reviews is already outdated.

This is how teams lose days on incidents that should take minutes.

What fixes it

The fix is not glamorous:

move prompts into versioned files
pin model versions
commit inference and retrieval config
version eval datasets
require prompt/config changes to go through PR review

That is enough to turn "we think something changed" into "this exact artifact changed in this exact release."

Recommended split

Use Git for code, prompts, configs, infra, lockfiles, and smaller eval assets.
Use Git LFS when datasets get too large for normal repo history.
Use MLflow, Weights & Biases, DVC, or a model registry for large model artifacts, experiment tracking, and lineage.
Use tools such as Langfuse, LangSmith, Braintrust, or Promptfoo to connect versions to traces and evaluation results over time.

The key idea is not tool choice. The key idea is that every production result should be traceable back to a specific set of artifacts.

TL;DR

The short version

Git is necessary, but it only covers part of an AI system
Prompts, model versions, inference config, eval sets, and infra also change behavior
If changing an artifact can change output, version it
Pin model versions instead of using floating aliases
Keep prompts and evals in the repo; use registries or artifact tools for large model assets
The goal is simple: when production changes, you should know exactly what changed and be able to roll it back