ragnar/README.Rmd at main · tidyverse/ragnar · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
---
output: github_document
editor:
  markdown:
    canonical: true
    wrap: 72
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
unlink("r4ds.ragnar.duckdb")
```

# ragnar <a href="https://ragnar.tidyverse.org"><img src="man/figures/logo.png" align="right" height="138" alt="ragnar website" /></a>

<!-- badges: start -->

[![R-CMD-check](https://github.com/tidyverse/ragnar/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/tidyverse/ragnar/actions/workflows/R-CMD-check.yaml)

<!-- badges: end -->

ragnar is an R package that helps implement Retrieval-Augmented
Generation (RAG) workflows. It focuses on providing a complete solution
with sensible defaults, while still giving the knowledgeable user
precise control over each step. We don't believe that you can fully
automate the creation of a good RAG system, so it's important that
ragnar is not a black box. ragnar is designed to be transparent. You
can easily inspect outputs at intermediate steps to understand what's
happening.

## Installation

You can install ragnar from CRAN with:

``` r
install.packages("ragnar")
```

You can install the development version from GitHub with:

``` r
# install.packages("pak")
pak::pak("tidyverse/ragnar")
```

## Key Steps

### 1. Document Processing

ragnar works with a wide variety of document types, using
[MarkItDown](https://github.com/microsoft/markitdown) to convert content
to Markdown.

Key functions:

-   `read_as_markdown()`: Convert a file or URL to markdown
-   `ragnar_find_links()`: Find all links in a webpage

### 2. Text Chunking

Next we divide each document into chunks. ragnar defaults to a strategy
that preserves some of the semantics of the document, but provides
plenty of opportunities to tweak the approach.

Key functions:

-   `markdown_chunk()`: Full-featured chunker that identifies semantic
    boundaries and intelligently chunks text.

### 3. Context Augmentation (Optional)

RAG applications benefit from augmenting text chunks with additional
context, such as document headings and subheadings. ragnar makes it
easy to keep track of headings and subheadings as part of chunking.

`markdown_chunk()` automatically associates each chunk with the headings
that are in scope for that chunk.

### 4. Embedding

ragnar can help compute embeddings for each chunk. The goal is for
ragnar to provide access to embeddings from popular LLM providers.

Key functions:

-   `embed_ollama()`
-   `embed_openai()`
-   `embed_bedrock()`
-   `embed_databricks()`
-   `embed_google_vertex()`

Note that calling the embedding function directly is typically not
necessary. Instead, the embedding function is specified when a store is
first created, and then automatically called when needed by
`ragnar_retrieve()` and `ragnar_store_insert()`.

### 5. Storage

Processed data is stored in a format optimized for efficient searching,
using [DuckDB](https://duckdb.org).
allowing additional packages to implement support for different storage
providers.

Key functions:

-   `ragnar_store_create()`
-   `ragnar_store_connect()`
-   `ragnar_store_insert()`

### 6. Retrieval

Given a prompt, retrieve related chunks based on embedding distance or
bm25 text search.

Key functions:

-   `ragnar_retrieve()`: high-level function that performs both `vss`
    and `bm25` search and de-overlaps retrieved results.
-   `ragnar_retrieve_vss()`: Retrieve using [`vss` DuckDB
    extension](https://duckdb.org/docs/stable/core_extensions/vss)
-   `ragnar_retrieve_bm25()`: Retrieve using
    [`full-text search DuckDB extension`](https://duckdb.org/docs/stable/core_extensions/full_text_search)
-   `chunks_deoverlap()`: Consolidates retrieved chunks that overlap.

### 7. Chat Augmentation

ragnar can equip an `ellmer::Chat` object with a retrieve tool that
enables an LLM to retrieve content from a store on-demand.

-   `ragnar_register_tool_retrieve(chat, store)`.

## Usage

Here's an example of using ragnar to create a knowledge store from the
*R for Data Science (2e)* book:

```{r, code = readLines("examples/example-create-store.R")}
```

Once the store is set up, you can then retrieve the most relevant text
chunks.

```{r, code = readLines("examples/example-retrieve.R")}
```