DSPy.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ako1983/simple_cv_evaluation_for_O1/blob/implementing-RAG/DSPy.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2L4S47Bj3xlE"
      },
      "source": [
        "<img src=\"https://github.com/stanfordnlp/dspy/blob/main/docs/docs/static/img/dspy_logo.png?raw=1\" alt=\"DSPy7 Image\" height=\"150\"/>\n",
        "\n",
        "## **DSPy**: Programming with Foundation Models\n",
        "\n",
        "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/intro.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aktT9ESL3xlJ"
      },
      "source": [
        "This notebook introduces the **DSPy** framework for **Programming with Foundation Models**, i.e., language models (LMs) and retrieval models (RMs).\n",
        "\n",
        "**DSPy** emphasizes programming over prompting. It unifies techniques for **prompting** and **fine-tuning** LMs as well as improving them with **reasoning** and **tool/retrieval augmentation**, all expressed through a _minimalistic set of Pythonic operations that compose and learn_.\n",
        "\n",
        "**DSPy** provides **composable and declarative modules** for instructing LMs in a familiar Pythonic syntax. On top of that, **DSPy** introduces an **automatic compiler that teaches LMs** how to conduct the declarative steps in your program. The **DSPy compiler** will internally _trace_ your program and then **craft high-quality prompts for large LMs (or train automatic finetunes for small LMs)** to teach them the steps of your task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MdGxg11z3xlK"
      },
      "source": [
        "### 0] Setting Up\n",
        "\n",
        "As we'll start to see below, **DSPy** can routinely teach powerful models like `GPT-3.5` and local models like `T5-base` or `Llama2-13b` to be much more reliable at complex tasks. **DSPy** will compile the _same program_ into different few-shot prompts and/or finetunes for each LM.\n",
        "\n",
        "Let's begin by setting things up. The snippet below will also install **DSPy** if it's not there already."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k4naO8gp3xlK"
      },
      "outputs": [],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "import sys\n",
        "import os\n",
        "\n",
        "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
        "    import google.colab  # noqa: F401\n",
        "    repo_path = 'dspy'\n",
        "    !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",
        "except:\n",
        "    repo_path = '.'\n",
        "\n",
        "if repo_path not in sys.path:\n",
        "    sys.path.append(repo_path)\n",
        "\n",
        "# Set up the cache for this notebook\n",
        "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(repo_path, 'cache')\n",
        "\n",
        "import pkg_resources # Install the package if it's not installed\n",
        "if \"dspy-ai\" not in {pkg.key for pkg in pkg_resources.working_set}:\n",
        "    !pip install -U pip\n",
        "    !pip install dspy-ai==2.4.17\n",
        "    !pip install openai~=0.28.1\n",
        "    # !pip install -e $repo_path\n",
        "\n",
        "import dspy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNFxUZV03xlM"
      },
      "source": [
        "### 1] Getting Started\n",
        "\n",
        "We'll start by setting up the language model (LM) and retrieval model (RM). **DSPy** supports multiple API and local models. In this notebook, we'll work with GPT-3.5 (`gpt-3.5-turbo`) and the retriever `ColBERTv2`.\n",
        "\n",
        "To make things easy, we've set up a ColBERTv2 server hosting a Wikipedia 2017 \"abstracts\" search index (i.e., containing first paragraph of each article from this [2017 dump](https://hotpotqa.github.io/wiki-readme.html)), so you don't need to worry about setting one up! It's free.\n",
        "\n",
        "**Note:** _If you want to run this notebook without changing the examples, you don't need an API key. All examples are already cached internally so you can inspect them!_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TgcatRfw3xlM"
      },
      "outputs": [],
      "source": [
        "turbo = dspy.OpenAI(model='gpt-3.5-turbo')\n",
        "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",
        "\n",
        "dspy.settings.configure(lm=turbo, rm=colbertv2_wiki17_abstracts)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ecfn7x6O3xlN"
      },
      "source": [
        "In the last line above, we configure **DSPy** to use the turbo LM and the ColBERTv2 retriever (over Wikipedia 2017 abstracts) by default. This will be easy to overwrite for local parts of our programs if needed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "30kPsuOZ3xlN"
      },
      "source": [
        "##### A word on the workflow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0jB_OVb3xlN"
      },
      "source": [
        "You can build your own **DSPy programs** for various tasks, e.g., question answering, information extraction, or text-to-SQL.\n",
        "\n",
        "Whatever the task, the general workflow is:\n",
        "\n",
        "1. **Collect a little bit of data.** Define examples of the inputs and outputs of your program (e.g., questions and their answers). This could just be a handful of quick examples you wrote down. If large datasets exist, the more the merrier!\n",
        "1. **Write your program.** Define the modules (i.e., sub-tasks) of your program and the way they should interact together to solve your task.\n",
        "1. **Define some validation logic.** What makes for a good run of your program? Maybe the answers need to have a certain length or stick to a particular format? Specify the logic that checks that.\n",
        "1. **Compile!** Ask **DSPy** to _compile_ your program using your data. The compiler will use your data and validation logic to optimize your program (e.g., prompts and modules) so it's efficient and effective!\n",
        "1. **Iterate.** Repeat the process by improving your data, program, validation, or by using more advanced features of the **DSPy** compiler.\n",
        "\n",
        "Let's now see this in action."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BqI9WKuu3xlO"
      },
      "source": [
        "### 2] Task Examples\n",
        "\n",
        "**DSPy** accommodates a wide variety of applications and tasks. **In this intro notebook, we will work on the example task of multi-hop question answering (QA).**\n",
        "\n",
        "Other notebooks and tutorials will present different tasks. Now, let us load a tiny sample from the HotPotQA multi-hop dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VQ3DW7453xlO",
        "outputId": "5dfefa4d-8012-4cbe-cbb6-9b080745a3e7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(20, 50)"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from dspy.datasets import HotPotQA\n",
        "\n",
        "# Load the dataset.\n",
        "dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)\n",
        "\n",
        "# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.\n",
        "trainset = [x.with_inputs('question') for x in dataset.train]\n",
        "devset = [x.with_inputs('question') for x in dataset.dev]\n",
        "\n",
        "len(trainset), len(devset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IVjJPheQ3xlP"
      },
      "source": [
        "We just loaded `trainset` (20 examples) and `devset` (50 examples). Each example in our **training set** contains just a **question** and its (human-annotated) **answer**.\n",
        "\n",
        "**DSPy** typically requires very minimal labeling. Whereas your pipeline may involve six or seven complex steps, you only need labels for the initial question and the final answer. **DSPy** will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!\n",
        "\n",
        "Now, let's look at some data examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3DB2uPOi3xlQ",
        "outputId": "ec14f0cf-4c1d-4966-dec6-d33aef52d163"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Question: At My Window was released by which American singer-songwriter?\n",
            "Answer: John Townes Van Zandt\n"
          ]
        }
      ],
      "source": [
        "train_example = trainset[0]\n",
        "print(f\"Question: {train_example.question}\")\n",
        "print(f\"Answer: {train_example.answer}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "phUwjweF3xlQ"
      },
      "source": [
        "Examples in the **dev set** contain a third field, namely, **titles** of relevant Wikipedia articles. This is not essential but, for the sake of this intro, it'll help us get a sense of how well our programs are doing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4C_KyyY-3xlQ",
        "outputId": "8900b1e3-0b94-4db9-cf85-384c09ae57f2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
            "Answer: English\n",
            "Relevant Wikipedia Titles: {'Robert Irvine', 'Restaurant: Impossible'}\n"
          ]
        }
      ],
      "source": [
        "dev_example = devset[18]\n",
        "print(f\"Question: {dev_example.question}\")\n",
        "print(f\"Answer: {dev_example.answer}\")\n",
        "print(f\"Relevant Wikipedia Titles: {dev_example.gold_titles}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fBjMoUTv3xlR"
      },
      "source": [
        "After loading the raw data, we'd applied `x.with_inputs('question')` to each example to tell **DSPy** that our input field in each example will be just `question`. Any other fields are labels or metadata that are not given to the system."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0f9U3h2q3xlR",
        "outputId": "9cc501e7-1529-47f2-84d0-427bc981a461"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "For this dataset, training examples have input keys ['question'] and label keys ['answer']\n",
            "For this dataset, dev examples have input keys ['question'] and label keys ['answer', 'gold_titles']\n"
          ]
        }
      ],
      "source": [
        "print(f\"For this dataset, training examples have input keys {train_example.inputs().keys()} and label keys {train_example.labels().keys()}\")\n",
        "print(f\"For this dataset, dev examples have input keys {dev_example.inputs().keys()} and label keys {dev_example.labels().keys()}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4NKud2T3xlR"
      },
      "source": [
        "Note that there's nothing special about the HotPotQA dataset: it's just a list of examples.\n",
        "\n",
        "You can define your own examples as below. A future notebook will guide you through creating your own data in unusual or data-scarce settings, which is a context where **DSPy** excels.\n",
        "\n",
        "```\n",
        "dspy.Example(field1=value, field2=value2, ...)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DQMJrBLM3xlR"
      },
      "source": [
        "### 3] Building Blocks\n",
        "\n",
        "In **DSPy**, we will maintain a clean separation between **defining your modules in a declarative way** and **calling them in a pipeline to solve the task**.\n",
        "\n",
        "This allows you to focus on the information flow of your pipeline. **DSPy** will then take your program and automatically optimize **how to prompt** (or finetune) LMs **for your particular pipeline** so it works well.\n",
        "\n",
        "If you have experience with PyTorch, you can think of DSPy as the PyTorch of the foundation model space. Before we see this in action, let's first understand some key pieces."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FXo_Or5J3xlS"
      },
      "source": [
        "##### Using the Language Model: **Signatures** & **Predictors**\n",
        "\n",
        "Every call to the LM in a **DSPy** program needs to have a **Signature**.\n",
        "\n",
        "A signature consists of three simple elements:\n",
        "\n",
        "- A minimal description of the sub-task the LM is supposed to solve.\n",
        "- A description of one or more input fields (e.g., input question) that we will give to the LM.\n",
        "- A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.\n",
        "\n",
        "Let's define a simple signature for basic question answering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jK0V103o3xlS"
      },
      "outputs": [],
      "source": [
        "class BasicQA(dspy.Signature):\n",
        "    \"\"\"Answer questions with short factoid answers.\"\"\"\n",
        "\n",
        "    question = dspy.InputField()\n",
        "    answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CW-ov5f3xlS"
      },
      "source": [
        "In `BasicQA`, the docstring describes the sub-task here (i.e., answering questions). Each `InputField` or `OutputField` can optionally contain a description `desc` too. When it's not given, it's inferred from the field's name (e.g., `question`).\n",
        "\n",
        "Notice that there isn't anything special about this signature in **DSPy**. We can just as easily define a signature that takes a long snippet from a PDF and outputs structured information, for instance.\n",
        "\n",
        "Anyway, now that we have a signature, let's define and use a **Predictor**. A predictor is a module that knows how to use the LM to implement a signature. Importantly, predictors can **learn** to fit their behavior to the task!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "55DTdT993xlS",
        "outputId": "60760232-5f20-4510-ab16-00de0be7a26c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
            "Predicted Answer: American\n"
          ]
        }
      ],
      "source": [
        "# Define the predictor.\n",
        "generate_answer = dspy.Predict(BasicQA)\n",
        "\n",
        "# Call the predictor on a particular input.\n",
        "pred = generate_answer(question=dev_example.question)\n",
        "\n",
        "# Print the input and the prediction.\n",
        "print(f\"Question: {dev_example.question}\")\n",
        "print(f\"Predicted Answer: {pred.answer}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hs83gPkX3xlS"
      },
      "source": [
        "In the example above, we asked the predictor about the the chef featured in \"Restaurant: Impossible\". The model outputs an answer (\"American\").\n",
        "\n",
        "For visibility, we can inspect how this extremely basic predictor implemented our signature. Let's inspect the history of our LM (**turbo**)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P0xDnVpH3xlT",
        "outputId": "a5cc9725-6e0b-4c86-bf56-a846647f21e0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Question: ${question}\n",
            "Answer: often between 1 and 5 words\n",
            "\n",
            "---\n",
            "\n",
            "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
            "Answer:\u001b[32m American\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "turbo.inspect_history(n=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3VIvomss3xlT"
      },
      "source": [
        "It happens that this chef is both British and American, but we have no way of knowing if the model just guessed \"American\" because it's a common answer. In general, adding **retrieval** and **learning** will help the LM be more factual, and we'll explore this in a minute!\n",
        "\n",
        "But before we do that, how about we _just_ change the predictor? It would be nice to allow the model to elicit a chain of thought along with the prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6zXhvqv93xlT",
        "outputId": "8f1ff72b-dc9e-47c9-c0d7-63e52b6b4d4c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
            "Thought: We know that the chef and restaurateur featured in Restaurant: Impossible is Robert Irvine.\n",
            "Predicted Answer: British\n"
          ]
        }
      ],
      "source": [
        "# Define the predictor. Notice we're just changing the class. The signature BasicQA is unchanged.\n",
        "generate_answer_with_chain_of_thought = dspy.ChainOfThought(BasicQA)\n",
        "\n",
        "# Call the predictor on the same input.\n",
        "pred = generate_answer_with_chain_of_thought(question=dev_example.question)\n",
        "\n",
        "# Print the input, the chain of thought, and the prediction.\n",
        "print(f\"Question: {dev_example.question}\")\n",
        "print(f\"Thought: {pred.rationale.split('.', 1)[1].strip()}\")\n",
        "print(f\"Predicted Answer: {pred.answer}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XedOCDAA3xlU"
      },
      "source": [
        "This is indeed a better answer: the model figures out that the chef in question is **Robert Irvine** and correctly identifies that he's British.\n",
        "\n",
        "These predictors (`dspy.Predict` and `dspy.ChainOfThought`) can be applied to _any_ signature. As we'll see below, they can also be optimized to learn from your data and validation logic."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PvBEMQGi3xlU"
      },
      "source": [
        "##### Using the Retrieval Model\n",
        "\n",
        "Using the retriever is pretty simple. A module `dspy.Retrieve(k)` will search for the top-`k` passages that match a given query.\n",
        "\n",
        "By default, this will use the retriever we configured at the top of this notebook, namely, ColBERTv2 over a Wikipedia index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vkmn9TyC3xlU",
        "outputId": "620ae670-6258-4e34-c13c-4e455f3cac5d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 3 passages for question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible? \n",
            " ------------------------------ \n",
            "\n",
            "1] Restaurant: Impossible | Restaurant: Impossible is an American reality television series, featuring chef and restaurateur Robert Irvine, that aired on Food Network from 2011 to 2016. \n",
            "\n",
            "2] Jean Joho | Jean Joho is a French-American chef and restaurateur. He is chef/proprietor of Everest in Chicago (founded in 1986), Paris Club Bistro & Bar and Studio Paris in Chicago, The Eiffel Tower Restaurant in Las Vegas, and Brasserie JO in Boston. \n",
            "\n",
            "3] List of Restaurant: Impossible episodes | This is the list of the episodes for the American cooking and reality television series \"Restaurant Impossible\", produced by Food Network. The premise of the series is that within two days and on a budget of $10,000, celebrity chef Robert Irvine renovates a failing American restaurant with the goal of helping to restore it to profitability and prominence. Irvine is assisted by a designer (usually Taniya Nayak, Cheryl Torrenueva, or Lynn Keagan, but sometimes Vanessa De Leon, Krista Watterworth, Yvette Irene, or Nicole Faccuito), along with general contractor Tom Bury, who sometimes does double duty as both general contractor and designer. After assessing the problems with the restaurant, Robert Irvine typically creates a plan for the new decor, oversees the cleaning of the restaurant, reduces the size of the menu and improves the food, develops a promotional activity, educates the restaurant's owners, or trains the staff, as needed by each restaurant. \n",
            "\n"
          ]
        }
      ],
      "source": [
        "retrieve = dspy.Retrieve(k=3)\n",
        "topK_passages = retrieve(dev_example.question).passages\n",
        "\n",
        "print(f\"Top {retrieve.k} passages for question: {dev_example.question} \\n\", '-' * 30, '\\n')\n",
        "\n",
        "for idx, passage in enumerate(topK_passages):\n",
        "    print(f'{idx+1}]', passage, '\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPvqKFMO3xlU"
      },
      "source": [
        "Feel free to any other queries you like."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BD7SvC5n3xlV",
        "outputId": "22836c6b-6aee-4450-f995-3cab7eceb17f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'History of the FIFA World Cup | The FIFA World Cup was first held in 1930, when FIFA president Jules Rimet decided to stage an international football tournament. The inaugural edition, held in 1930, was contested as a final tournament of only thirteen teams invited by the organization. Since then, the World Cup has experienced successive expansions and format remodeling to its current 32-team final tournament preceded by a two-year qualifying process, involving over 200 teams from around the world.'"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "retrieve(\"When was the first FIFA World Cup held?\").passages[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KlsMxy4k3xlV"
      },
      "source": [
        "### 4] Program 1: Basic Retrieval-Augmented Generation (“RAG”)\n",
        "\n",
        "Let's define our first complete program for this task. We'll build a retrieval-augmented pipeline for answer generation.\n",
        "\n",
        "Given a question, we'll search for the top-3 passages in Wikipedia and then feed them as context for answer generation.\n",
        "\n",
        "Let's start by defining this signature: `context, question --> answer`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NAHg5Iny3xlW"
      },
      "outputs": [],
      "source": [
        "class GenerateAnswer(dspy.Signature):\n",
        "    \"\"\"Answer questions with short factoid answers.\"\"\"\n",
        "\n",
        "    context = dspy.InputField(desc=\"may contain relevant facts\")\n",
        "    question = dspy.InputField()\n",
        "    answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CfytFxBB3xlX"
      },
      "source": [
        "Great. Now let's define the actual program. This is a class that inherits from `dspy.Module`.\n",
        "\n",
        "It needs two methods:\n",
        "\n",
        "- The `__init__` method will simply declare the sub-modules it needs: `dspy.Retrieve` and `dspy.ChainOfThought`. The latter is defined to implement our `GenerateAnswer` signature.\n",
        "- The `forward` method will describe the control flow of answering the question using the modules we have."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nvpWclla3xlX"
      },
      "outputs": [],
      "source": [
        "class RAG(dspy.Module):\n",
        "    def __init__(self, num_passages=3):\n",
        "        super().__init__()\n",
        "\n",
        "        self.retrieve = dspy.Retrieve(k=num_passages)\n",
        "        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",
        "\n",
        "    def forward(self, question):\n",
        "        context = self.retrieve(question).passages\n",
        "        prediction = self.generate_answer(context=context, question=question)\n",
        "        return dspy.Prediction(context=context, answer=prediction.answer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "URqnxRNP3xlY"
      },
      "source": [
        "##### Compiling the RAG program\n",
        "\n",
        "Having defined this program, let's now **compile** it. Compiling a program will update the parameters stored in each module. In our setting, this is primarily in the form of collecting and selecting good demonstrations for inclusion in your prompt(s).\n",
        "\n",
        "Compiling depends on three things:\n",
        "\n",
        "1. **A training set.** We'll just use our 20 question–answer examples from `trainset` above.\n",
        "1. **A metric for validation.** We'll define a quick `validate_context_and_answer` that checks that the predicted answer is correct. It'll also check that the retrieved context does actually contain that answer.\n",
        "1. **A specific teleprompter.** The **DSPy** compiler includes a number of **teleprompters** that can optimize your programs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yn_DQ44D3xlY"
      },
      "source": [
        "**Teleprompters:** Teleprompters are powerful optimizers that can take any program and learn to bootstrap and select effective prompts for its modules. Hence the name, which means \"prompting at a distance\".\n",
        "\n",
        "Different teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. We will use a simple default `BootstrapFewShot` in this notebook.\n",
        "\n",
        "\n",
        "_If you're into analogies, you could think of this as your training data, your loss function, and your optimizer in a standard DNN supervised learning setup. Whereas SGD is a basic optimizer, there are more sophisticated (and more expensive!) ones like Adam or RMSProp._"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oLHoJFNa3xlY",
        "outputId": "6636ae4b-7821-4410-e64f-9a0dc7f0bdbc"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            " 50%|█████     | 10/20 [00:00<00:00, 121.99it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bootstrapped 4 full traces after 11 examples in round 0.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "from dspy.teleprompt import BootstrapFewShot\n",
        "\n",
        "# Validation logic: check that the predicted answer is correct.\n",
        "# Also check that the retrieved context does actually contain that answer.\n",
        "def validate_context_and_answer(example, pred, trace=None):\n",
        "    answer_EM = dspy.evaluate.answer_exact_match(example, pred)\n",
        "    answer_PM = dspy.evaluate.answer_passage_match(example, pred)\n",
        "    return answer_EM and answer_PM\n",
        "\n",
        "# Set up a basic teleprompter, which will compile our RAG program.\n",
        "teleprompter = BootstrapFewShot(metric=validate_context_and_answer)\n",
        "\n",
        "# Compile!\n",
        "compiled_rag = teleprompter.compile(RAG(), trainset=trainset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EK3a2Nwf3xlZ"
      },
      "source": [
        "Now that we've compiled our RAG program, let's try it out."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gM9d0ONV3xlZ",
        "outputId": "4c9c347a-d06f-4c20-ae16-9fbfc3ecf32e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Question: What castle did David Gregory inherit?\n",
            "Predicted Answer: Kinnairdy Castle\n",
            "Retrieved Contexts (truncated): ['David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinn...', 'Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 t...', 'David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University ...']\n"
          ]
        }
      ],
      "source": [
        "# Ask any question you like to this simple RAG program.\n",
        "my_question = \"What castle did David Gregory inherit?\"\n",
        "\n",
        "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",
        "pred = compiled_rag(my_question)\n",
        "\n",
        "# Print the contexts and the answer.\n",
        "print(f\"Question: {my_question}\")\n",
        "print(f\"Predicted Answer: {pred.answer}\")\n",
        "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IvlyPqrP3xlZ"
      },
      "source": [
        "Excellent. How about we inspect the last prompt for the LM?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aXGyRkfb3xlZ",
        "outputId": "d2304172-e0fb-4f52-c21e-99ab35aa1733"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: At My Window was released by which American singer-songwriter?\n",
            "Answer: John Townes Van Zandt\n",
            "\n",
            "Question: \"Everything Has Changed\" is a song from an album released under which record label ?\n",
            "Answer: Big Machine Records\n",
            "\n",
            "Question: The Victorians - Their Story In Pictures is a documentary series written by an author born in what year?\n",
            "Answer: 1950\n",
            "\n",
            "Question: Which Pakistani cricket umpire who won 3 consecutive ICC umpire of the year awards in 2009, 2010, and 2011 will be in the ICC World Twenty20?\n",
            "Answer: Aleem Sarwar Dar\n",
            "\n",
            "Question: Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers?\n",
            "Answer: \"Outfield of Dreams\"\n",
            "\n",
            "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
            "Answer: Aleksandr Danilovich Aleksandrov\n",
            "\n",
            "Question: The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?\n",
            "Answer: 2010\n",
            "\n",
            "Question: Tombstone stared an actor born May 17, 1955 known as who?\n",
            "Answer: Bill Paxton\n",
            "\n",
            "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
            "Answer: 1874\n",
            "\n",
            "Question: which American actor was Candace Kita guest starred with\n",
            "Answer: Bill Murray\n",
            "\n",
            "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
            "Answer: The Empire State Building\n",
            "\n",
            "Question: Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?\n",
            "Answer: Buena Vista Distribution\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: often between 1 and 5 words\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. \"Tae Kwon Do Times\" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.»\n",
            "[2] «Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang's for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.»\n",
            "[3] «Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th \"dan\" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including \"Fight to Win\", \"Best of the Best\", \"Bloodsport II\", and \"Bloodsport III\". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both \"Black Belt\" magazine's Hall of Fame and \"Tae Kwon Do Times\" magazine's Hall of Fame.»\n",
            "\n",
            "Question: Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know from the context that \"Tae Kwon Do Times\" is a magazine that covers taekwondo and other Korean martial arts. It has published articles by authors like Scott Shaw. On the other hand, there is no information about Southwest Art magazine in the context.\n",
            "\n",
            "Answer: Tae Kwon Do Times\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Rosario Dawson | Rosario Isabel Dawson (born May 9, 1979) is an American actress, producer, singer, comic book writer, and political activist. She made her film debut in the 1995 teen drama \"Kids\". Her subsequent film roles include \"He Got Game\", \"Men in Black II\", \"25th Hour\", \"Rent\", \"Sin City\", \"Death Proof\", \"Seven Pounds\", \"\", and \"Top Five\". Dawson has also provided voice-over work for Disney and DC.»\n",
            "[2] «Sarai Gonzalez | Sarai Isaura Gonzalez (born 2005) is an American Latina child actress who made her professional debut at the age of 11 on the Spanish-language \"\"Soy Yo\"\" (\"That's Me\") music video by Bomba Estéreo. Cast as a \"nerdy\" tween with a \"sassy\" and \"confident\" attitude, her performance turned her into a \"Latina icon\" for \"female empowerment, identity and self-worth\". She subsequently appeared in two get out the vote videos for Latinos in advance of the 2016 United States elections.»\n",
            "[3] «Gabriela (2001 film) | Gabriela is a 2001 American romance film, starring Seidy Lopez in the title role alongside Jaime Gomez as her admirer Mike. The film has been cited as an inspiration behind the Premiere Weekend Club, which supports Latino film-making.»\n",
            "\n",
            "Question: Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the actress made her film debut in 1995 and co-founded Voto Latino.\n",
            "\n",
            "Answer: Rosario Dawson\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Battle of Kursk | The Battle of Kursk was a Second World War engagement between German and Soviet forces on the Eastern Front near Kursk (450 km south-west of Moscow) in the Soviet Union during July and August 1943. The battle began with the launch of the German offensive, Operation Citadel (German: \"Unternehmen Zitadelle\" ), on 5 July, which had the objective of pinching off the Kursk salient with attacks on the base of the salient from north and south simultaneously. After the German offensive stalled on the northern side of the salient, on 12 July the Soviets commenced their Kursk Strategic Offensive Operation with the launch of Operation Kutuzov (Russian: Кутузов ) against the rear of the German forces in the northern side. On the southern side, the Soviets also launched powerful counterattacks the same day, one of which led to a large armoured clash, the Battle of Prokhorovka. On 3 August, the Soviets began the second phase of the Kursk Strategic Offensive Operation with the launch of Operation Polkovodets Rumyantsev (Russian: Полководец Румянцев ) against the German forces in the southern side of the Kursk salient.»\n",
            "[2] «Operation Mars | Operation Mars, also known as the Second Rzhev-Sychevka Offensive Operation (Russian: Вторая Ржевско-Сычёвская наступательная операция), was the codename for an offensive launched by Soviet forces against German forces during World War II. It took place between 25 November and 20 December 1942 around the Rzhev salient in the vicinity of Moscow.»\n",
            "[3] «Kholm Pocket | The Kholm Pocket (German: \"Kessel von Cholm\" ; Russian: Холмский котёл ) was the name given for the encirclement of German troops by the Red Army around Kholm south of Leningrad, during World War II on the Eastern Front, from 23 January 1942 until 5 May 1942. A much larger pocket was simultaneously surrounded in Demyansk, about 100 km to the northeast. These were the results of German retreat following their defeat during the Battle of Moscow.»\n",
            "\n",
            "Question: What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the German offensive that started the Battle of Kursk was called Operation Citadel.\n",
            "\n",
            "Answer: Operation Citadel\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Kerry Condon | Kerry Condon (born 4 January 1983) is an Irish television and film actress, best known for her role as Octavia of the Julii in the HBO/BBC series \"Rome,\" as Stacey Ehrmantraut in AMC's \"Better Call Saul\" and as the voice of F.R.I.D.A.Y. in various films in the Marvel Cinematic Universe. She is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\"»\n",
            "[2] «Corona Riccardo | Corona Riccardo (c. 1878October 15, 1917) was an Italian born American actress who had a brief Broadway stage career before leaving to become a wife and mother. Born in Naples she came to acting in 1894 playing a Mexican girl in a play at the Empire Theatre. Wilson Barrett engaged her for a role in his play \"The Sign of the Cross\" which he took on tour of the United States. Riccardo played the role of Ancaria and later played Berenice in the same play. Robert B. Mantell in 1898 who struck by her beauty also cast her in two Shakespeare plays, \"Romeo and Juliet\" and \"Othello\". Author Lewis Strang writing in 1899 said Riccardo was the most promising actress in America at the time. Towards the end of 1898 Mantell chose her for another Shakespeare part, Ophelia im Hamlet. Afterwards she was due to join Augustin Daly's Theatre Company but Daly died in 1899. In 1899 she gained her biggest fame by playing Iras in the first stage production of Ben-Hur.»\n",
            "[3] «Judi Dench | Dame Judith Olivia \"Judi\" Dench, {'1': \", '2': \", '3': \", '4': \"} (born 9 December 1934) is an English actress and author. Dench made her professional debut in 1957 with the Old Vic Company. Over the following few years, she performed in several of Shakespeare's plays in such roles as Ophelia in \"Hamlet\", Juliet in \"Romeo and Juliet\", and Lady Macbeth in \"Macbeth\". Although most of her work during this period was in theatre, she also branched into film work and won a BAFTA Award as Most Promising Newcomer. She drew strong reviews for her leading role in the musical \"Cabaret\" in 1968.»\n",
            "\n",
            "Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\" ?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the actress we are looking for is the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\" We also know that she acted in the short film The Shore.\n",
            "\n",
            "Answer: Kerry Condon\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
            "[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n",
            "[3] «David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University of Edinburgh, Savilian Professor of Astronomy at the University of Oxford, and a commentator on Isaac Newton's \"Principia\".»\n",
            "\n",
            "Question: What castle did David Gregory inherit?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m produce the answer. We know that David Gregory inherited a castle. The name of the castle is Kinnairdy Castle.\n",
            "\n",
            "Answer: Kinnairdy Castle\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "turbo.inspect_history(n=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TC350wmp3xla"
      },
      "source": [
        "Even though we haven't written any of this detailed demonstrations, we see that **DSPy** was able to bootstrap this 3,000 token prompt for **3-shot retrieval augmented generation with hard negative passages and chain of thought** from our extremely simple program.\n",
        "\n",
        "This illustrates the power of composition and learning. Of course, this was just generated by a particular teleprompter, which may or may not be perfect in each setting. As you'll see in **DSPy**, there is a large but systematic space of options you have to optimize and validate the quality and cost of your programs.\n",
        "\n",
        "If you're so inclined, you can easily inspect the learned objects themselves."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zXVDLaiC3xla",
        "outputId": "cd2791df-1a35-4203-b33d-3ac3a2107515"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "generate_answer\n",
            "Example({'augmented': True, 'context': ['Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. \"Tae Kwon Do Times\" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.', \"Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang's for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.\", 'Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th \"dan\" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including \"Fight to Win\", \"Best of the Best\", \"Bloodsport II\", and \"Bloodsport III\". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both \"Black Belt\" magazine\\'s Hall of Fame and \"Tae Kwon Do Times\" magazine\\'s Hall of Fame.'], 'question': 'Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?', 'rationale': 'produce the answer. We know from the context that \"Tae Kwon Do Times\" is a magazine that covers taekwondo and other Korean martial arts. It has published articles by authors like Scott Shaw. On the other hand, there is no information about Southwest Art magazine in the context.', 'answer': 'Tae Kwon Do Times'}) (input_keys=None)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for name, parameter in compiled_rag.named_predictors():\n",
        "    print(name)\n",
        "    print(parameter.demos[0])\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Otx73Wd3xla"
      },
      "source": [
        "##### Evaluating the Answers\n",
        "\n",
        "We can now evaluate our `compiled_rag` program on the dev set. Of course, this tiny set is _not_ meant to be a reliable benchmark, but it'll be instructive to use it for illustration.\n",
        "\n",
        "For a start, let's evaluate the accuracy (exact match) of the predicted answer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rdhmtmGm3xla",
        "outputId": "aac41361-6084-4b81-8c88-32c1c2f607b4"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 22 / 50  (44.0): 100%|██████████| 50/50 [00:00<00:00, 116.45it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 22 / 50  (44.0%)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_46dbe th {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_46dbe td {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_46dbe_row0_col0, #T_46dbe_row0_col1, #T_46dbe_row0_col2, #T_46dbe_row0_col3, #T_46dbe_row0_col4, #T_46dbe_row0_col5, #T_46dbe_row1_col0, #T_46dbe_row1_col1, #T_46dbe_row1_col2, #T_46dbe_row1_col3, #T_46dbe_row1_col4, #T_46dbe_row1_col5, #T_46dbe_row2_col0, #T_46dbe_row2_col1, #T_46dbe_row2_col2, #T_46dbe_row2_col3, #T_46dbe_row2_col4, #T_46dbe_row2_col5, #T_46dbe_row3_col0, #T_46dbe_row3_col1, #T_46dbe_row3_col2, #T_46dbe_row3_col3, #T_46dbe_row3_col4, #T_46dbe_row3_col5, #T_46dbe_row4_col0, #T_46dbe_row4_col1, #T_46dbe_row4_col2, #T_46dbe_row4_col3, #T_46dbe_row4_col4, #T_46dbe_row4_col5 {\n",
              "  text-align: left;\n",
              "  white-space: pre-wrap;\n",
              "  word-wrap: break-word;\n",
              "  max-width: 400px;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_46dbe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_46dbe_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
              "      <th id=\"T_46dbe_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
              "      <th id=\"T_46dbe_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
              "      <th id=\"T_46dbe_level0_col3\" class=\"col_heading level0 col3\" >context</th>\n",
              "      <th id=\"T_46dbe_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
              "      <th id=\"T_46dbe_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_46dbe_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_46dbe_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
              "      <td id=\"T_46dbe_row0_col1\" class=\"data row0 col1\" >no</td>\n",
              "      <td id=\"T_46dbe_row0_col2\" class=\"data row0 col2\" >{'Cangzhou', 'Qionghai'}</td>\n",
              "      <td id=\"T_46dbe_row0_col3\" class=\"data row0 col3\" >['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
              "      <td id=\"T_46dbe_row0_col4\" class=\"data row0 col4\" >No</td>\n",
              "      <td id=\"T_46dbe_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_46dbe_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_46dbe_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
              "      <td id=\"T_46dbe_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
              "      <td id=\"T_46dbe_row1_col2\" class=\"data row1 col2\" >{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}</td>\n",
              "      <td id=\"T_46dbe_row1_col3\" class=\"data row1 col3\" >['2017–18 Pittsburgh Penguins season | The 2017–18 Pittsburgh Penguins season will be the 51st season for the National Hockey League ice hockey team that was...</td>\n",
              "      <td id=\"T_46dbe_row1_col4\" class=\"data row1 col4\" >National Hockey League</td>\n",
              "      <td id=\"T_46dbe_row1_col5\" class=\"data row1 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_46dbe_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_46dbe_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
              "      <td id=\"T_46dbe_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
              "      <td id=\"T_46dbe_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
              "      <td id=\"T_46dbe_row2_col3\" class=\"data row2 col3\" >['Steve Yzerman | Stephen Gregory \"Steve\" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...</td>\n",
              "      <td id=\"T_46dbe_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
              "      <td id=\"T_46dbe_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_46dbe_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_46dbe_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
              "      <td id=\"T_46dbe_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
              "      <td id=\"T_46dbe_row3_col2\" class=\"data row3 col2\" >{'Crichton Castle', 'Crichton Collegiate Church'}</td>\n",
              "      <td id=\"T_46dbe_row3_col3\" class=\"data row3 col3\" >[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
              "      <td id=\"T_46dbe_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
              "      <td id=\"T_46dbe_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_46dbe_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_46dbe_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
              "      <td id=\"T_46dbe_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
              "      <td id=\"T_46dbe_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
              "      <td id=\"T_46dbe_row4_col3\" class=\"data row4 col3\" >[\"Æthelweard of East Anglia | Æthelweard (died 854) was a 9th-century king of East Anglia, the long-lived Anglo-Saxon kingdom which today includes the English counties...</td>\n",
              "      <td id=\"T_46dbe_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
              "      <td id=\"T_46dbe_row4_col5\" class=\"data row4 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7fbb387dcee0>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <div style='\n",
              "                    text-align: center; \n",
              "                    font-size: 16px; \n",
              "                    font-weight: bold; \n",
              "                    color: #555; \n",
              "                    margin: 10px 0;'>\n",
              "                    ... 45 more rows not displayed ...\n",
              "                </div>\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "44.0"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from dspy.evaluate.evaluate import Evaluate\n",
        "\n",
        "# Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.\n",
        "evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=1, display_progress=True, display_table=5)\n",
        "\n",
        "# Evaluate the `compiled_rag` program with the `answer_exact_match` metric.\n",
        "metric = dspy.evaluate.answer_exact_match\n",
        "evaluate_on_hotpotqa(compiled_rag, metric=metric)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Z5g-isM3xla"
      },
      "source": [
        "##### Evaluating the Retrieval\n",
        "\n",
        "It may also be instructive to look at the accuracy of retrieval. There are multiple ways to do this. Often, we can just check whether the retrieved passages contain the answer.\n",
        "\n",
        "That said, since our dev set includes the gold titles that should be retrieved, we can just use these here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TeaM254T3xla",
        "outputId": "82e47120-68c1-4bb1-f22f-101a87d3e1d2"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 13 / 50  (26.0): 100%|██████████| 50/50 [00:00<00:00, 671.76it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 13 / 50  (26.0%)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_3ac3d th {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_3ac3d td {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_3ac3d_row0_col0, #T_3ac3d_row0_col1, #T_3ac3d_row0_col2, #T_3ac3d_row0_col3, #T_3ac3d_row0_col4, #T_3ac3d_row0_col5, #T_3ac3d_row1_col0, #T_3ac3d_row1_col1, #T_3ac3d_row1_col2, #T_3ac3d_row1_col3, #T_3ac3d_row1_col4, #T_3ac3d_row1_col5, #T_3ac3d_row2_col0, #T_3ac3d_row2_col1, #T_3ac3d_row2_col2, #T_3ac3d_row2_col3, #T_3ac3d_row2_col4, #T_3ac3d_row2_col5, #T_3ac3d_row3_col0, #T_3ac3d_row3_col1, #T_3ac3d_row3_col2, #T_3ac3d_row3_col3, #T_3ac3d_row3_col4, #T_3ac3d_row3_col5, #T_3ac3d_row4_col0, #T_3ac3d_row4_col1, #T_3ac3d_row4_col2, #T_3ac3d_row4_col3, #T_3ac3d_row4_col4, #T_3ac3d_row4_col5 {\n",
              "  text-align: left;\n",
              "  white-space: pre-wrap;\n",
              "  word-wrap: break-word;\n",
              "  max-width: 400px;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_3ac3d\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_3ac3d_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
              "      <th id=\"T_3ac3d_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
              "      <th id=\"T_3ac3d_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
              "      <th id=\"T_3ac3d_level0_col3\" class=\"col_heading level0 col3\" >context</th>\n",
              "      <th id=\"T_3ac3d_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
              "      <th id=\"T_3ac3d_level0_col5\" class=\"col_heading level0 col5\" >gold_passages_retrieved</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_3ac3d_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_3ac3d_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
              "      <td id=\"T_3ac3d_row0_col1\" class=\"data row0 col1\" >no</td>\n",
              "      <td id=\"T_3ac3d_row0_col2\" class=\"data row0 col2\" >{'Cangzhou', 'Qionghai'}</td>\n",
              "      <td id=\"T_3ac3d_row0_col3\" class=\"data row0 col3\" >['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
              "      <td id=\"T_3ac3d_row0_col4\" class=\"data row0 col4\" >No</td>\n",
              "      <td id=\"T_3ac3d_row0_col5\" class=\"data row0 col5\" >❌ [False]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3ac3d_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_3ac3d_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
              "      <td id=\"T_3ac3d_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
              "      <td id=\"T_3ac3d_row1_col2\" class=\"data row1 col2\" >{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}</td>\n",
              "      <td id=\"T_3ac3d_row1_col3\" class=\"data row1 col3\" >['2017–18 Pittsburgh Penguins season | The 2017–18 Pittsburgh Penguins season will be the 51st season for the National Hockey League ice hockey team that was...</td>\n",
              "      <td id=\"T_3ac3d_row1_col4\" class=\"data row1 col4\" >National Hockey League</td>\n",
              "      <td id=\"T_3ac3d_row1_col5\" class=\"data row1 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3ac3d_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_3ac3d_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
              "      <td id=\"T_3ac3d_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
              "      <td id=\"T_3ac3d_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
              "      <td id=\"T_3ac3d_row2_col3\" class=\"data row2 col3\" >['Steve Yzerman | Stephen Gregory \"Steve\" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...</td>\n",
              "      <td id=\"T_3ac3d_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
              "      <td id=\"T_3ac3d_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3ac3d_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_3ac3d_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
              "      <td id=\"T_3ac3d_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
              "      <td id=\"T_3ac3d_row3_col2\" class=\"data row3 col2\" >{'Crichton Castle', 'Crichton Collegiate Church'}</td>\n",
              "      <td id=\"T_3ac3d_row3_col3\" class=\"data row3 col3\" >[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
              "      <td id=\"T_3ac3d_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
              "      <td id=\"T_3ac3d_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3ac3d_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_3ac3d_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
              "      <td id=\"T_3ac3d_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
              "      <td id=\"T_3ac3d_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
              "      <td id=\"T_3ac3d_row4_col3\" class=\"data row4 col3\" >[\"Æthelweard of East Anglia | Æthelweard (died 854) was a 9th-century king of East Anglia, the long-lived Anglo-Saxon kingdom which today includes the English counties...</td>\n",
              "      <td id=\"T_3ac3d_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
              "      <td id=\"T_3ac3d_row4_col5\" class=\"data row4 col5\" >❌ [False]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7fbb387909d0>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <div style='\n",
              "                    text-align: center; \n",
              "                    font-size: 16px; \n",
              "                    font-weight: bold; \n",
              "                    color: #555; \n",
              "                    margin: 10px 0;'>\n",
              "                    ... 45 more rows not displayed ...\n",
              "                </div>\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def gold_passages_retrieved(example, pred, trace=None):\n",
        "    gold_titles = set(map(dspy.evaluate.normalize_text, example['gold_titles']))\n",
        "    found_titles = set(map(dspy.evaluate.normalize_text, [c.split(' | ')[0] for c in pred.context]))\n",
        "\n",
        "    return gold_titles.issubset(found_titles)\n",
        "\n",
        "compiled_rag_retrieval_score = evaluate_on_hotpotqa(compiled_rag, metric=gold_passages_retrieved)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scRAPmch3xlb"
      },
      "source": [
        "Although this simple `compiled_rag` program is able to answer a decent fraction of the questions correctly (on this tiny set, over 40%), the quality of retrieval is much lower.\n",
        "\n",
        "This potentially suggests that the LM is often relying on the knowledge it memorized during training to answer questions. To address this weak retrieval, let's explore a second program that involves more advanced search behavior."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S0TNWY7E3xlb"
      },
      "source": [
        "### 5] Program 2: Multi-Hop Search (“Baleen”)\n",
        "\n",
        "From exploring the harder questions in the training/dev sets, it becomes clear that a single search query is often not enough for this task. For instance, this can be seen when a question ask about, say, the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it wouldn't figure out when he was born.\n",
        "\n",
        "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary. Using **DSPy**, we can easily simulate such systems in a few lines of code.\n",
        "\n",
        "\n",
        "We'll still use the `GenerateAnswer` signature from the RAG implementation above. All we need now is a **signature** for the \"hop\" behavior: taking some partial context and a question, generate a search query to find missing information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7jwyNQKI3xlb"
      },
      "outputs": [],
      "source": [
        "class GenerateSearchQuery(dspy.Signature):\n",
        "    \"\"\"Write a simple search query that will help answer a complex question.\"\"\"\n",
        "\n",
        "    context = dspy.InputField(desc=\"may contain relevant facts\")\n",
        "    question = dspy.InputField()\n",
        "    query = dspy.OutputField()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K160NMcl3xlb"
      },
      "source": [
        "Note: We could have written `context = GenerateAnswer.signature.context` to avoid duplicating the description of the `context` field.\n",
        "\n",
        "Now, let's define the program itself `SimplifiedBaleen`. There are many possible ways to implement this, but we'll keep this version down to the key elements for simplicity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pxF66Y033xlb"
      },
      "outputs": [],
      "source": [
        "from dsp.utils import deduplicate\n",
        "\n",
        "class SimplifiedBaleen(dspy.Module):\n",
        "    def __init__(self, passages_per_hop=3, max_hops=2):\n",
        "        super().__init__()\n",
        "\n",
        "        self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",
        "        self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",
        "        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",
        "        self.max_hops = max_hops\n",
        "\n",
        "    def forward(self, question):\n",
        "        context = []\n",
        "\n",
        "        for hop in range(self.max_hops):\n",
        "            query = self.generate_query[hop](context=context, question=question).query\n",
        "            passages = self.retrieve(query).passages\n",
        "            context = deduplicate(context + passages)\n",
        "\n",
        "        pred = self.generate_answer(context=context, question=question)\n",
        "        return dspy.Prediction(context=context, answer=pred.answer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E_t_CTFc3xlc"
      },
      "source": [
        "As we can see, the `__init__` method defines a few key sub-modules:\n",
        "\n",
        "- **generate_query**: For each hop, we will have one `dspy.ChainOfThought` predictor with the `GenerateSearchQuery` signature.\n",
        "- **retrieve**: This module will do the actual search, using the generated queries.\n",
        "- **generate_answer**: This `dspy.Predict` module will be used after all the search steps. It has a `GenerateAnswer`, to actually produce an answer.\n",
        "\n",
        "The `forward` method uses these sub-modules in simple control flow.\n",
        "\n",
        "1. First, we'll loop up to `self.max_hops` times.\n",
        "1. In each iteration, we'll generate a search query using the predictor at `self.generate_query[hop]`.\n",
        "1. We'll retrieve the top-k passages using that query.\n",
        "1. We'll add the (deduplicated) passages to our accumulator of `context`.\n",
        "1. After the loop, we'll use `self.generate_answer` to produce an answer.\n",
        "1. We'll return a prediction with the retrieved `context` and predicted `answer`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jtfs1U8m3xlc"
      },
      "source": [
        "##### Inspect the zero-shot version of the Baleen program\n",
        "\n",
        "We will also compile this program shortly. But, before that, we can try it out in a \"zero-shot\" setting (i.e., without any compilation).\n",
        "\n",
        "Using a program in zero-shot (uncompiled) setting doesn't mean that quality will be bad. It just means that we're bottlenecked directly by the reliability of the underlying LM to understand our sub-tasks from minimal instructions.\n",
        "\n",
        "This is often just fine when using the most expensive/powerful models (e.g., GPT-4) on the easiest and most standard tasks (e.g., answering simple questions about popular entities).\n",
        "\n",
        "However, a zero-shot approach quickly falls short for more specialized tasks, for novel domains/settings, and for more efficient (or open) models. **DSPy** can help you in all of these settings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ybMjTsZn3xlc",
        "outputId": "7d38dc38-7988-4e8d-a6ba-bb60d9c840b0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "Predicted Answer: five\n",
            "Retrieved Contexts (truncated): ['David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinn...', 'The Boleyn Inheritance | The Boleyn Inheritance is a novel by British author Philippa Gregory which was first published in 2006. It is a direct sequel to her previous novel \"The Other Boleyn Girl,\" an...', 'Gregory of Gaeta | Gregory was the Duke of Gaeta from 963 until his death. He was the second son of Docibilis II of Gaeta and his wife Orania. He succeeded his brother John II, who had left only daugh...', 'Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy....', 'Kinnaird Head | Kinnaird Head (Scottish Gaelic: \"An Ceann Àrd\" , \"high headland\") is a headland projecting into the North Sea, within the town of Fraserburgh, Aberdeenshire on the east coast of Scotla...', 'Kinnaird Castle, Brechin | Kinnaird Castle is a 15th-century castle in Angus, Scotland. The castle has been home to the Carnegie family, the Earl of Southesk, for more than 600 years....']\n"
          ]
        }
      ],
      "source": [
        "# Ask any question you like to this simple RAG program.\n",
        "my_question = \"How many storeys are in the castle that David Gregory inherited?\"\n",
        "\n",
        "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",
        "uncompiled_baleen = SimplifiedBaleen()  # uncompiled (i.e., zero-shot) program\n",
        "pred = uncompiled_baleen(my_question)\n",
        "\n",
        "# Print the contexts and the answer.\n",
        "print(f\"Question: {my_question}\")\n",
        "print(f\"Predicted Answer: {pred.answer}\")\n",
        "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ew0nKIcM3xld"
      },
      "source": [
        "Let's inspect the last **three** calls to the LM (i.e., generating the first hop's query, generating the second hop's query, and generating the answer)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7NVqog3w3xld",
        "outputId": "f9773656-2259-4ebd-cfd3-b2b52ec65b40"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a simple search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
            "\n",
            "Query: ${query}\n",
            "\n",
            "---\n",
            "\n",
            "Context: N/A\n",
            "\n",
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m find the answer to this question. First, we need to find information about David Gregory and the castle he inherited. Then, we can search for details about the castle's architecture or any historical records that mention the number of storeys.\n",
            "\n",
            "Query: \"David Gregory castle inheritance\"\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a simple search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
            "\n",
            "Query: ${query}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
            "[2] «The Boleyn Inheritance | The Boleyn Inheritance is a novel by British author Philippa Gregory which was first published in 2006. It is a direct sequel to her previous novel \"The Other Boleyn Girl,\" and one of the additions to her six-part series on the Tudor royals. (The other titles in the series are \"The Constant Princess,\" \"The Queen's Fool,\" \"The Virgin's Lover,and The Other Queen.\") * The novel is told through the first-person narratives of – Anne of Cleves, Katherine Howard, and Jane Boleyn, who was mentioned in \"The Other Boleyn Girl.\" It covers a period from 1539 until 1542 and chronicles the fourth and fifth marriages of King Henry VIII of England.»\n",
            "[3] «Gregory of Gaeta | Gregory was the Duke of Gaeta from 963 until his death. He was the second son of Docibilis II of Gaeta and his wife Orania. He succeeded his brother John II, who had left only daughters. Gregory rapidly depleted the \"publicum\" (public land) of the Duchy of Gaeta by doling it out to family members as grants. Gregory disappears from the records in 964 and was succeeded by his younger brother Marinus of Fondi over the heads of his three sons. It is possible that there was an internal power struggle between factions of the Docibilan family and that Gregory was forced out. On the other hand, perhaps he died and his sons fought a losing battle for their inheritance to Gaeta.»\n",
            "\n",
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m produce the query. We know that David Gregory inherited Kinnairdy Castle, so we need to find information about the castle and its characteristics.\n",
            "\n",
            "Query: \"Kinnairdy Castle number of storeys\"\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: often between 1 and 5 words\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
            "[2] «The Boleyn Inheritance | The Boleyn Inheritance is a novel by British author Philippa Gregory which was first published in 2006. It is a direct sequel to her previous novel \"The Other Boleyn Girl,\" and one of the additions to her six-part series on the Tudor royals. (The other titles in the series are \"The Constant Princess,\" \"The Queen's Fool,\" \"The Virgin's Lover,and The Other Queen.\") * The novel is told through the first-person narratives of – Anne of Cleves, Katherine Howard, and Jane Boleyn, who was mentioned in \"The Other Boleyn Girl.\" It covers a period from 1539 until 1542 and chronicles the fourth and fifth marriages of King Henry VIII of England.»\n",
            "[3] «Gregory of Gaeta | Gregory was the Duke of Gaeta from 963 until his death. He was the second son of Docibilis II of Gaeta and his wife Orania. He succeeded his brother John II, who had left only daughters. Gregory rapidly depleted the \"publicum\" (public land) of the Duchy of Gaeta by doling it out to family members as grants. Gregory disappears from the records in 964 and was succeeded by his younger brother Marinus of Fondi over the heads of his three sons. It is possible that there was an internal power struggle between factions of the Docibilan family and that Gregory was forced out. On the other hand, perhaps he died and his sons fought a losing battle for their inheritance to Gaeta.»\n",
            "[4] «Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy.»\n",
            "[5] «Kinnaird Head | Kinnaird Head (Scottish Gaelic: \"An Ceann Àrd\" , \"high headland\") is a headland projecting into the North Sea, within the town of Fraserburgh, Aberdeenshire on the east coast of Scotland. The 16th-century Kinnaird Castle was converted in 1787 for use as the Kinnaird Head Lighthouse, the first lighthouse in Scotland to be lit by the Commissioners of Northern Lights. Kinnaird Castle and the nearby Winetower were described by W. Douglas Simpson as two of the nine castles of the Knuckle, referring to the rocky headland of north-east Aberdeenshire. Both buildings are category A listed buildings.»\n",
            "[6] «Kinnaird Castle, Brechin | Kinnaird Castle is a 15th-century castle in Angus, Scotland. The castle has been home to the Carnegie family, the Earl of Southesk, for more than 600 years.»\n",
            "\n",
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m produce the answer. We know from the context that David Gregory inherited Kinnairdy Castle. \n",
            "\n",
            "Answer: five\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "turbo.inspect_history(n=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJpH8xtt3xld"
      },
      "source": [
        "##### Compiling the Baleen program\n",
        "\n",
        "Now is the time to compile our multi-hop (`SimplifiedBaleen`) program.\n",
        "\n",
        "We will first define our validation logic, which will simply require that:\n",
        "\n",
        "- The predicted answer matches the gold answer.\n",
        "- The retrieved context contains the gold answer.\n",
        "- None of the generated queries is rambling (i.e., none exceeds 100 characters in length).\n",
        "- None of the generated queries is roughly repeated (i.e., none is within 0.8 or higher F1 score of earlier queries)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PQH-E3nQ3xld"
      },
      "outputs": [],
      "source": [
        "def validate_context_and_answer_and_hops(example, pred, trace=None):\n",
        "    if not dspy.evaluate.answer_exact_match(example, pred): return False\n",
        "    if not dspy.evaluate.answer_passage_match(example, pred): return False\n",
        "\n",
        "    hops = [example.question] + [outputs.query for *_, outputs in trace if 'query' in outputs]\n",
        "\n",
        "    if max([len(h) for h in hops]) > 100: return False\n",
        "    if any(dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8) for idx in range(2, len(hops))): return False\n",
        "\n",
        "    return True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GS2gOGQa3xld"
      },
      "source": [
        "Like we did for RAG, we'll use one of the most basic teleprompters in **DSPy**, namely, `BootstrapFewShot`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Zomn20Nf3xle",
        "outputId": "4929c5af-bbad-4dc3-e2c8-a01691ac9b35"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 20/20 [00:00<00:00, 64.11it/s]\n"
          ]
        }
      ],
      "source": [
        "teleprompter = BootstrapFewShot(metric=validate_context_and_answer_and_hops)\n",
        "compiled_baleen = teleprompter.compile(SimplifiedBaleen(), teacher=SimplifiedBaleen(passages_per_hop=2), trainset=trainset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vt9nq9rP3xle"
      },
      "source": [
        "##### Evaluating the Retrieval\n",
        "\n",
        "Earlier, it appeared like our simple RAG program was not very effective at finding all evidence required for answering each question. Is this resolved by the adding some extra steps in the `forward` function of `SimplifiedBaleen`? What about compiling, does it help for that?\n",
        "\n",
        "The answer for these questions is not always going to be obvious. However, **DSPy** makes it extremely easy to try many diverse approaches with minimal effort.\n",
        "\n",
        "Let's evaluate the quality of retrieval of our compiled and uncompiled Baleen pipelines!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ygWLaMz93xle"
      },
      "outputs": [],
      "source": [
        "uncompiled_baleen_retrieval_score = evaluate_on_hotpotqa(uncompiled_baleen, metric=gold_passages_retrieved)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CPq3qbyH3xle",
        "outputId": "79eae46c-0fcd-4cc2-c0b2-59b70126d5a0"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 30 / 50  (60.0): 100%|██████████| 50/50 [00:00<00:00, 54.98it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Average Metric: 30 / 50  (60.0%)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style type=\"text/css\">\n",
              "#T_3bd2b th {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_3bd2b td {\n",
              "  text-align: left;\n",
              "}\n",
              "#T_3bd2b_row0_col0, #T_3bd2b_row0_col1, #T_3bd2b_row0_col2, #T_3bd2b_row0_col3, #T_3bd2b_row0_col4, #T_3bd2b_row0_col5, #T_3bd2b_row1_col0, #T_3bd2b_row1_col1, #T_3bd2b_row1_col2, #T_3bd2b_row1_col3, #T_3bd2b_row1_col4, #T_3bd2b_row1_col5, #T_3bd2b_row2_col0, #T_3bd2b_row2_col1, #T_3bd2b_row2_col2, #T_3bd2b_row2_col3, #T_3bd2b_row2_col4, #T_3bd2b_row2_col5, #T_3bd2b_row3_col0, #T_3bd2b_row3_col1, #T_3bd2b_row3_col2, #T_3bd2b_row3_col3, #T_3bd2b_row3_col4, #T_3bd2b_row3_col5, #T_3bd2b_row4_col0, #T_3bd2b_row4_col1, #T_3bd2b_row4_col2, #T_3bd2b_row4_col3, #T_3bd2b_row4_col4, #T_3bd2b_row4_col5 {\n",
              "  text-align: left;\n",
              "  white-space: pre-wrap;\n",
              "  word-wrap: break-word;\n",
              "  max-width: 400px;\n",
              "}\n",
              "</style>\n",
              "<table id=\"T_3bd2b\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th class=\"blank level0\" >&nbsp;</th>\n",
              "      <th id=\"T_3bd2b_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
              "      <th id=\"T_3bd2b_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
              "      <th id=\"T_3bd2b_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
              "      <th id=\"T_3bd2b_level0_col3\" class=\"col_heading level0 col3\" >context</th>\n",
              "      <th id=\"T_3bd2b_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
              "      <th id=\"T_3bd2b_level0_col5\" class=\"col_heading level0 col5\" >gold_passages_retrieved</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th id=\"T_3bd2b_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
              "      <td id=\"T_3bd2b_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
              "      <td id=\"T_3bd2b_row0_col1\" class=\"data row0 col1\" >no</td>\n",
              "      <td id=\"T_3bd2b_row0_col2\" class=\"data row0 col2\" >{'Cangzhou', 'Qionghai'}</td>\n",
              "      <td id=\"T_3bd2b_row0_col3\" class=\"data row0 col3\" >['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
              "      <td id=\"T_3bd2b_row0_col4\" class=\"data row0 col4\" >No</td>\n",
              "      <td id=\"T_3bd2b_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3bd2b_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
              "      <td id=\"T_3bd2b_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
              "      <td id=\"T_3bd2b_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
              "      <td id=\"T_3bd2b_row1_col2\" class=\"data row1 col2\" >{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}</td>\n",
              "      <td id=\"T_3bd2b_row1_col3\" class=\"data row1 col3\" >[\"2017 NHL Expansion Draft | The 2017 NHL Expansion Draft was an expansion draft conducted by the National Hockey League on June 18–20, 2017 to...</td>\n",
              "      <td id=\"T_3bd2b_row1_col4\" class=\"data row1 col4\" >National Hockey League (NHL)</td>\n",
              "      <td id=\"T_3bd2b_row1_col5\" class=\"data row1 col5\" >❌ [False]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3bd2b_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
              "      <td id=\"T_3bd2b_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
              "      <td id=\"T_3bd2b_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
              "      <td id=\"T_3bd2b_row2_col2\" class=\"data row2 col2\" >{'2006–07 Detroit Red Wings season', 'Steve Yzerman'}</td>\n",
              "      <td id=\"T_3bd2b_row2_col3\" class=\"data row2 col3\" >['List of Tampa Bay Lightning general managers | The Tampa Bay Lightning are an American professional ice hockey team based in Tampa, Florida. They play...</td>\n",
              "      <td id=\"T_3bd2b_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
              "      <td id=\"T_3bd2b_row2_col5\" class=\"data row2 col5\" >❌ [False]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3bd2b_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
              "      <td id=\"T_3bd2b_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
              "      <td id=\"T_3bd2b_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
              "      <td id=\"T_3bd2b_row3_col2\" class=\"data row3 col2\" >{'Crichton Castle', 'Crichton Collegiate Church'}</td>\n",
              "      <td id=\"T_3bd2b_row3_col3\" class=\"data row3 col3\" >[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
              "      <td id=\"T_3bd2b_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
              "      <td id=\"T_3bd2b_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th id=\"T_3bd2b_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
              "      <td id=\"T_3bd2b_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
              "      <td id=\"T_3bd2b_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
              "      <td id=\"T_3bd2b_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
              "      <td id=\"T_3bd2b_row4_col3\" class=\"data row4 col3\" >['Æthelweard (son of Alfred) | Æthelweard (d. 920 or 922) was the younger son of King Alfred the Great and Ealhswith.', 'Æthelred the Unready |...</td>\n",
              "      <td id=\"T_3bd2b_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
              "      <td id=\"T_3bd2b_row4_col5\" class=\"data row4 col5\" >❌ [False]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n"
            ],
            "text/plain": [
              "<pandas.io.formats.style.Styler at 0x7fbaf80fab80>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <div style='\n",
              "                    text-align: center; \n",
              "                    font-size: 16px; \n",
              "                    font-weight: bold; \n",
              "                    color: #555; \n",
              "                    margin: 10px 0;'>\n",
              "                    ... 45 more rows not displayed ...\n",
              "                </div>\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "compiled_baleen_retrieval_score = evaluate_on_hotpotqa(compiled_baleen, metric=gold_passages_retrieved)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0DJMJmOW3xlf",
        "outputId": "f9ad3ff2-2e6b-4e1d-bd8a-7d019fdb28ae"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "## Retrieval Score for RAG: 26.0\n",
            "## Retrieval Score for uncompiled Baleen: 36.0\n",
            "## Retrieval Score for compiled Baleen: 60.0\n"
          ]
        }
      ],
      "source": [
        "print(f\"## Retrieval Score for RAG: {compiled_rag_retrieval_score}\")  # note that for RAG, compilation has no effect on the retrieval step\n",
        "print(f\"## Retrieval Score for uncompiled Baleen: {uncompiled_baleen_retrieval_score}\")\n",
        "print(f\"## Retrieval Score for compiled Baleen: {compiled_baleen_retrieval_score}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AEBOcITF3xlf"
      },
      "source": [
        "Excellent! There might be something to this compiled, multi-hop program then. But this is far from all you can do: **DSPy** gives you a clean space of composable operators to deal with any shortcomings you see.\n",
        "\n",
        "We can inspect a few concrete examples. If we see failure causes, we can:\n",
        "\n",
        "1. Expand our pipeline by using additional sub-modules (e.g., maybe summarize after retrieval?)\n",
        "1. Modify our pipeline by using more complex logic (e.g., maybe we need to break out of the multi-hop loop if we found all information we need?)\n",
        "1. Refine our validation logic (e.g., maybe use a metric that use a second **DSPy** program to do the answer evaluation, instead of relying on strict string matching)\n",
        "1. Use a different teleprompter to optimize your pipeline more aggressively.\n",
        "1. Add more or better training examples!\n",
        "\n",
        "\n",
        "Or, if you really want, we can tweak the descriptions in the Signatures we use in your program to make them more precisely suited for their sub-tasks. This is akin to prompt engineering and should be a final resort, given the other powerful options that **DSPy** gives us!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bKMarVsc3xlf",
        "outputId": "d0ddd550-cad5-46b4-db26-cb1d735042df"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a simple search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
            "\n",
            "Query: ${query}\n",
            "\n",
            "---\n",
            "\n",
            "Context: N/A\n",
            "\n",
            "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the query. We know that the FA Charity Shield is an annual football match played in England between the winners of the previous season's Premier League and FA Cup. In this case, we are looking for the year when Manchester City played against a specific club in the 1972 FA Charity Shield. To find this information, we can search for the history of the FA Charity Shield and the teams that participated in the 1972 edition.\n",
            "\n",
            "Query: \"History of FA Charity Shield 1972\"\n",
            "\n",
            "---\n",
            "\n",
            "Context: N/A\n",
            "\n",
            "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the query. We need to find the heights of both buildings and compare them.\n",
            "\n",
            "Query: \"height of Empire State Building\" OR \"height of Bank of America Tower\"\n",
            "\n",
            "---\n",
            "\n",
            "Context: N/A\n",
            "\n",
            "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the query. We can search for the birth dates of both Aleksandr Danilovich Aleksandrov and Anatoly Fomenko and compare them to determine who is older.\n",
            "\n",
            "Query: \"Birth date Aleksandr Danilovich Aleksandrov\" \"Birth date Anatoly Fomenko\"\n",
            "\n",
            "---\n",
            "\n",
            "Context: N/A\n",
            "\n",
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m produce the query. We need to find information about the castle that David Gregory inherited and determine the number of storeys it has.\n",
            "\n",
            "Query: \"Castle inherited by David Gregory number of storeys\"\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Write a simple search query that will help answer a complex question.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
            "\n",
            "Query: ${query}\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «1972 FA Charity Shield | The 1972 FA Charity Shield was contested between Manchester City and Aston Villa.»\n",
            "[2] «1971 FA Charity Shield | The 1971 FA Charity Shield was a football match between Leicester City and Liverpool at Filbert Street on Saturday 7 August 1971.»\n",
            "\n",
            "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the query. We know that the 1972 FA Charity Shield was contested between Manchester City and Aston Villa. To find out the year the club founded that played against Manchester City, we need to search for the founding year of Aston Villa.\n",
            "\n",
            "Query: \"Aston Villa founding year\"\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Empire State Building | The Empire State Building is a 102-story skyscraper located on Fifth Avenue between West 33rd and 34th Streets in Midtown, Manhattan, New York City. It has a roof height of 1,250 feet (381 m), and with its antenna included, it stands a total of 1454 ft tall. Its name is derived from the nickname for New York, the Empire State.»\n",
            "[2] «Bank of America Plaza (Atlanta) | Bank of America Plaza is a skyscraper located in between Midtown Atlanta and Downtown Atlanta. At 312 m , the tower is the 96th-tallest building in the world. It is the 14th tallest building in the U.S., the tallest building in Georgia and the tallest building in any U.S. state capital, overtaking the 250 m (820 ft), 50 story One Atlantic Center in height, which previously held the record as Georgia's tallest building. It has 55 stories of office space and was completed in 1992, when it was called NationsBank Plaza. Originally intended to be the headquarters for Citizens & Southern National Bank (which merged with Sovran Bank during construction), it became NationsBank's property following its formation in the 1991 hostile takeover of C&S/Sovran by NCNB.»\n",
            "\n",
            "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
            "\n",
            "Reasoning: Let's think step by step in order to answer the question. We know that the Empire State Building has a roof height of 1,250 feet and a total height of 1,454 feet including its antenna. On the other hand, the Bank of America Plaza in Atlanta is 312 meters tall. To compare the heights of the two buildings, we need to convert the height of the Bank of America Plaza from meters to feet.\n",
            "\n",
            "Query: \"Convert 312 meters to feet\"\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Aleksandr Danilovich Aleksandrov | Aleksandr Danilovich Aleksandrov (Russian: Алекса́ндр Дани́лович Алекса́ндров , alternative transliterations: \"Alexandr\" or \"Alexander\" (first name), and \"Alexandrov\" (last name)) (August 4, 1912 – July 27, 1999), was a Soviet/Russian mathematician, physicist, philosopher and mountaineer.»\n",
            "[2] «Aleksandr Pavlovich Aleksandrov | Aleksandr Pavlovich Aleksandrov (Russian: Александр Павлович Александров ; born February 20, 1943) is a former Soviet cosmonaut and twice Hero of the Soviet Union (November 23, 1983 and December 29, 1987).»\n",
            "\n",
            "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the query. We know that Aleksandr Danilovich Aleksandrov was born on August 4, 1912, but we don't have information about Anatoly Fomenko's birthdate. To find out who is older, we need to compare their birthdates.\n",
            "\n",
            "Query: \"Anatoly Fomenko birthdate\"\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
            "[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n",
            "[3] «Gregory Parsloe-Parsloe | Sir Gregory Parsloe-Parsloe, 7th Baronet is a fictional character from the Blandings stories of P. G. Wodehouse. The seventh Baronet, who resides at Matchingham Hall, he is the son of the Very Reverend Dean Parsloe-Parsloe and is the rival and enemy of Lord Emsworth, master of Blandings Castle.»\n",
            "\n",
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m produce the query. We know that David Gregory inherited Kinnairdy Castle and that he had three of his twenty-nine children become mathematics professors. To find out how many storeys are in the castle, we need to search for information about Kinnairdy Castle.\n",
            "\n",
            "Query: \"Kinnairdy Castle number of storeys\"\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Answer questions with short factoid answers.\n",
            "\n",
            "---\n",
            "\n",
            "Question: The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?\n",
            "Answer: 2010\n",
            "\n",
            "Question: On the coast of what ocean is the birthplace of Diogal Sakho?\n",
            "Answer: Atlantic\n",
            "\n",
            "Question: Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?\n",
            "Answer: Buena Vista Distribution\n",
            "\n",
            "Question: Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?\n",
            "Answer: Tae Kwon Do Times\n",
            "\n",
            "Question: Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?\n",
            "Answer: Rosario Dawson\n",
            "\n",
            "Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\" ?\n",
            "Answer: Kerry Condon\n",
            "\n",
            "Question: which American actor was Candace Kita guest starred with\n",
            "Answer: Bill Murray\n",
            "\n",
            "Question: \"Everything Has Changed\" is a song from an album released under which record label ?\n",
            "Answer: Big Machine Records\n",
            "\n",
            "Question: Tombstone stared an actor born May 17, 1955 known as who?\n",
            "Answer: Bill Paxton\n",
            "\n",
            "Question: Which of these publications was most recently published, Who Put the Bomp or Self?\n",
            "Answer: Self\n",
            "\n",
            "Question: What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?\n",
            "Answer: Operation Citadel\n",
            "\n",
            "Question: Samantha Cristoforetti and Mark Shuttleworth are both best known for being first in their field to go where?\n",
            "Answer: space\n",
            "\n",
            "Question: Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers?\n",
            "Answer: \"Outfield of Dreams\"\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: may contain relevant facts\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: often between 1 and 5 words\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «1972 FA Charity Shield | The 1972 FA Charity Shield was contested between Manchester City and Aston Villa.»\n",
            "[2] «1971 FA Charity Shield | The 1971 FA Charity Shield was a football match between Leicester City and Liverpool at Filbert Street on Saturday 7 August 1971.»\n",
            "[3] «1896–97 Aston Villa F.C. season | The 1896-87 season was Aston Villa's ninth season in the Football League since being one of its 12 founding members in 1888.»\n",
            "[4] «List of Aston Villa F.C. seasons | This is a list of seasons played by Aston Villa Football Club in English and European football, from 1879 (the year of the club's first FA Cup entry) to the most recent completed season. Aston Villa football club was founded in March, 1874, by members of the Villa Cross Wesleyan Chapel in Aston. Throughout the 1870s Aston Villa played a small amount of games. At least one game, against Aston Brook St Mary's was played with one half under Rugby rules and the other under football rules. In the 1880s the game became more formalised and in 1888, William McGregor formed the Football League with 11 other clubs.»\n",
            "\n",
            "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the 1972 FA Charity Shield was contested between Manchester City and Aston Villa. According to the context, Aston Villa Football Club was founded in March, 1874.\n",
            "\n",
            "Answer: 1874\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Empire State Building | The Empire State Building is a 102-story skyscraper located on Fifth Avenue between West 33rd and 34th Streets in Midtown, Manhattan, New York City. It has a roof height of 1,250 feet (381 m), and with its antenna included, it stands a total of 1454 ft tall. Its name is derived from the nickname for New York, the Empire State.»\n",
            "[2] «Bank of America Plaza (Atlanta) | Bank of America Plaza is a skyscraper located in between Midtown Atlanta and Downtown Atlanta. At 312 m , the tower is the 96th-tallest building in the world. It is the 14th tallest building in the U.S., the tallest building in Georgia and the tallest building in any U.S. state capital, overtaking the 250 m (820 ft), 50 story One Atlantic Center in height, which previously held the record as Georgia's tallest building. It has 55 stories of office space and was completed in 1992, when it was called NationsBank Plaza. Originally intended to be the headquarters for Citizens & Southern National Bank (which merged with Sovran Bank during construction), it became NationsBank's property following its formation in the 1991 hostile takeover of C&S/Sovran by NCNB.»\n",
            "[3] «Foot (unit) | The foot ( feet; abbreviation: ft; symbol: ′, the prime symbol) is a unit of length in the imperial and US customary systems of measurement. Since 1959, both units have been defined by international agreement as equivalent to 0.3048 meters exactly. In both systems, the foot comprises 12 inches and three feet compose a yard.»\n",
            "[4] «Pentameter | Pentameter (from Greek: πεντάμετρος - 'measuring five (feet)') is a poetic meter. А poem is said to be written in a particular pentameter when the lines of the poem have the length of five feet, where 'foot' is a combination of a particular number (1 or 2) of unstressed (or weak) syllables and a stressed (or strong) syllable. Depending on the pattern of feet, pentameter can be iambic (one of three two-syllable meters alongside trochaic and spondaic) or dactylic (one of two three-syllable meters alongside anapestic) (see links below).»\n",
            "\n",
            "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the Empire State Building has a roof height of 1,250 feet and with its antenna included, it stands a total of 1,454 feet tall. On the other hand, the Bank of America Plaza in Atlanta is 312 meters tall, which is approximately 1,024 feet. Therefore, the Empire State Building is taller than the Bank of America Plaza.\n",
            "\n",
            "Answer: Empire State Building\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «Aleksandr Danilovich Aleksandrov | Aleksandr Danilovich Aleksandrov (Russian: Алекса́ндр Дани́лович Алекса́ндров , alternative transliterations: \"Alexandr\" or \"Alexander\" (first name), and \"Alexandrov\" (last name)) (August 4, 1912 – July 27, 1999), was a Soviet/Russian mathematician, physicist, philosopher and mountaineer.»\n",
            "[2] «Aleksandr Pavlovich Aleksandrov | Aleksandr Pavlovich Aleksandrov (Russian: Александр Павлович Александров ; born February 20, 1943) is a former Soviet cosmonaut and twice Hero of the Soviet Union (November 23, 1983 and December 29, 1987).»\n",
            "[3] «Anatoly Fomenko | Anatoly Timofeevich Fomenko (Russian: Анато́лий Тимофе́евич Фоме́нко ) (born 13 March 1945 in Stalino, USSR) is a Soviet and Russian mathematician, professor at Moscow State University, well known as a topologist, and a member of the Russian Academy of Sciences. He is author of a pseudoscientific theory known as New Chronology. He is also a member of the Russian Academy of Natural Sciences (1991).»\n",
            "[4] «Semyon Fomin | Semyon Anatolyevich Fomin (Russian: Семён Анатольевич Фомин ; born 10 January 1989) is a Russian professional footballer. He plays as a midfielder for FC Luch-Energiya Vladivostok.»\n",
            "\n",
            "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that Aleksandr Danilovich Aleksandrov was born on August 4, 1912, and Anatoly Fomenko was born on March 13, 1945. Therefore, Aleksandr Danilovich Aleksandrov is older.\n",
            "\n",
            "Answer: Aleksandr Danilovich Aleksandrov\n",
            "\n",
            "---\n",
            "\n",
            "Context:\n",
            "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
            "[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n",
            "[3] «Gregory Parsloe-Parsloe | Sir Gregory Parsloe-Parsloe, 7th Baronet is a fictional character from the Blandings stories of P. G. Wodehouse. The seventh Baronet, who resides at Matchingham Hall, he is the son of the Very Reverend Dean Parsloe-Parsloe and is the rival and enemy of Lord Emsworth, master of Blandings Castle.»\n",
            "[4] «Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy.»\n",
            "[5] «Kinnaird Head | Kinnaird Head (Scottish Gaelic: \"An Ceann Àrd\" , \"high headland\") is a headland projecting into the North Sea, within the town of Fraserburgh, Aberdeenshire on the east coast of Scotland. The 16th-century Kinnaird Castle was converted in 1787 for use as the Kinnaird Head Lighthouse, the first lighthouse in Scotland to be lit by the Commissioners of Northern Lights. Kinnaird Castle and the nearby Winetower were described by W. Douglas Simpson as two of the nine castles of the Knuckle, referring to the rocky headland of north-east Aberdeenshire. Both buildings are category A listed buildings.»\n",
            "[6] «Kinnaird Castle, Brechin | Kinnaird Castle is a 15th-century castle in Angus, Scotland. The castle has been home to the Carnegie family, the Earl of Southesk, for more than 600 years.»\n",
            "\n",
            "Question: How many storeys are in the castle that David Gregory inherited?\n",
            "\n",
            "Reasoning: Let's think step by step in order to\u001b[32m produce the answer. We know that David Gregory inherited Kinnairdy Castle. According to the context, Kinnairdy Castle is a tower house with five storeys and a garret.\n",
            "\n",
            "Answer: Five\u001b[0m\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "compiled_baleen(\"How many storeys are in the castle that David Gregory inherited?\")\n",
        "turbo.inspect_history(n=3)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "py39",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.17"
    },
    "orig_nbformat": 4,
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}