DSPy shifts the paradigm for interacting with models from prompt hacking to high-level programming, making LLM applications far easier to maintain and optimise.

At UnlikelyAI, almost every product we build has a language model somewhere inside it. The canonical interface between code and language models is completion-based. Developers essentially write chat messages that are passed to a language model. Nowadays, there are some standard API contracts that most providers support (Ollama, vLLM etc. typically offer an OpenAI-compatible interface for any backing model). But these contracts maintain a “chat” paradigm. They require that developers “prompt” a language model. This means that interfacing with a language model is different to interfacing with other components of our software. Is there a way we can treat language models as just another contract-based component in our system?

Over time, we’ve found a better way to interface with those models, one that trades brittle prompt strings for clean, testable, optimizable code. This post explains why we adopted DSPy, how it fits our neurosymbolic approach, and what it looks like in practice.

The pain DSPy solves

You’ve probably been there: you spend two days getting a prompt exactly right, ship it, and then the model provider pushes an update and suddenly nothing works. Your carefully tuned instructions produce garbage, and you have no idea why. The problem isn’t just the model update; it’s that prompt strings are static and fragile. DSPy treats this as an engineering problem, and solves it like one.

DSPy has seen remarkable adoption across the developer community. The framework has accumulated nearly 23,000 GitHub stars and nearly 300 contributors, with at least 500 projects already depending on it in production. What’s striking is the trajectory: as recently as August 2024, it sat at around 16,000 stars meaning it gained roughly 7,000 stars in under a year. This is a library that is moving fast because it is solving a real problem.

The DSPy mental model: Programming, not prompting

The Stanford NLP group have dreamed up a now widely used library that reframes how developers think about language models. DSPy (read “dee-ess-pie”) is the work of Omar Khattab, et. al., and it solves a lot of challenges we see in trying to build high-scale services with language model components.

To understand how we rely on DSPy, you have to understand its core philosophy. DSPy is an open-source framework for programming – rather than prompting – language models.

In this programming mental model, individual interactions with the LLM are abstracted away. Instead of writing paragraphs of instructions, you define Input/Output contracts and some basic structure, but leave the actual 'chatting' to the library.

To make this concrete, here’s what a typical RAG interaction looks like, both with and without DSPy.

In code, this paradigm shift has big implications for the simplicity and maintainability of your system.

Without DSPy vs. With DSPy: a side-by-side comparison

DSPy? More like DS-Why?

This paradigm shift gives us a lot of nice things out of the box.

• Structured Input/Output: Because DSPy handles the structuring of inputs and outputs via defined contracts, there is no need to write code to parse outputs. In short: go home regex!

• Code Cleanliness: We no longer need complicated code to enforce outputs, with retries and backoffs. That’s all handled by the library.

• Rapid Model Comparison: The framework's architecture makes it easy to swap between language models. Model choice is a simple config-layer option rather than a code-layer re-write.

• Advanced Prompting Built-In: DSPy includes built-in support for prompting paradigms like Chain of Thought and ReAct. Again, the switch is just a config-layer option. No need to re-write the text of your prompt.

DSPy goes further than just being an interface abstraction layer, however. It takes the mindset of “just let the library do all the interfacing work” to its logical extreme: you don’t even need to write prompts!

DSPy optimisers

The true power of DSPy is its optimisation features. Optimisation as a first-class citizen in DSPy, for both prompts and in-context examples. Given a suite of end-to-end data, with a loss function (ex. SemanticF1), you can have DSPy generate the prompt and/or in-context examples that achieve optimal performance on your task.

In the code block below, we can see that the entire prompt definition is a single line of code:

self.respond = dspy.ChainOfThought("context, question -> response")

self.respond = dspy.ChainOfThought("context, question -> response")

self.respond = dspy.ChainOfThought("context, question -> response")

self.respond = dspy.ChainOfThought("context, question -> response")

self.respond = dspy.ChainOfThought("context, question -> response")

This says, essentially: give me a Chain of Thought prompt that accepts a context and a question, and gives a response. It’s the abstract definition of a RAG prompt, in its simplest form. No need to write “Let’s think step by step” or to write out what it means to have a “context” or “question”, or what “retrieval means”. It’s easy, it’s powerful and it’s something we love.

class RAG(dspy.Module):
    def __init__(self, num_docs=5):
        self.num_docs = num_docs
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        context = search(question, k=self.num_docs)   # defined in tutorial linked below
        return self.respond(context=context, question=question)

tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

class RAG(dspy.Module):
    def __init__(self, num_docs=5):
        self.num_docs = num_docs
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        context = search(question, k=self.num_docs)   # defined in tutorial linked below
        return self.respond(context=context, question=question)

tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

class RAG(dspy.Module):
    def __init__(self, num_docs=5):
        self.num_docs = num_docs
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        context = search(question, k=self.num_docs)   # defined in tutorial linked below
        return self.respond(context=context, question=question)

tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

class RAG(dspy.Module):
    def __init__(self, num_docs=5):
        self.num_docs = num_docs
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        context = search(question, k=self.num_docs)   # defined in tutorial linked below
        return self.respond(context=context, question=question)

tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

class RAG(dspy.Module):
    def __init__(self, num_docs=5):
        self.num_docs = num_docs
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        context = search(question, k=self.num_docs)   # defined in tutorial linked below
        return self.respond(context=context, question=question)

tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24)
optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)

(source: https://dspy.ai/)

The downsides

To be fair, DSPy isn’t without its trade-offs. The abstraction that makes it powerful can also make debugging harder: when the generated prompt produces unexpected results, you have to look inside the framework to understand what it actually sent to the model. The optimisation step also requires a labelled dataset, which isn’t always available early in a project. And for very simple, one-shot tasks, the overhead of defining modules and signatures can feel like overkill.

It’s also the case that, like any rapidly-maturing open source project, using DSPy comes with a certain amount of maintenance chaos. Things change quickly, and your favourite pinned version might be obsolete pretty quickly.

We still reach for DSPy because the benefits far outweigh these costs at scale, but it’s worth going in with eyes open.

UnlikelyAI is neurosymbolic. DSPy kinda is too

At UnlikelyAI, we find value in pairing probabilistic systems (LLMs, etc.) with deterministic definitions (the “symbolic” of “neurosymbolic”). The DSPy paradigm of “programming, not prompting” fits this neurosymbolic framing quite nicely. That’s not why we love it. But it’s also not not why we love it.

Here’s a concrete example. Imagine a system that classifies whether a user-submitted claim is supported by a given set of documents. The deterministic (symbolic) layer defines some pre-computed reasoning: basically, at a high level, how we should go about categorising this claim. That symbolic reasoning may include defined evaluations we want a language model to perform. Those evaluations are expressed as DSPy Signatures. The probabilistic (neuro) layer is handled by the LLM, with DSPy managing the prompting strategy and in-context examples. We can swap the underlying model, change from ChainOfThought to ReAct reasoning, or run the optimiser against new labelled data, all without touching the evaluation contract. The symbolic layer stays stable; the neural layer improves.

By defining, in code, a contract for a given language model task, the “neuro” components of our system become more well-defined, more like a block of code. And the implementation of that contract can be left up to us (i.e. we can write our own prompts), or it can be derived from our end-to-end data (DSPy can write the prompt for us).

We get a lot of value out of DSPy. If you’re on the fence, give it a try. The deeper implication is this: as language models become load-bearing infrastructure, the way we engineer around them has to grow up. Prompts that live in strings, that no-one owns, that break silently, these are technical debt accumulating fast. DSPy isn’t a silver bullet, but it is a serious attempt to bring software engineering discipline to a domain that has lacked it. We think that matters. We think it will matter more as models become more capable, not less.