
<!-- README.md is generated from README.Rmd. Please edit that file -->

# kuzco <img src="man/figures/logo.png" align="right" height="108" alt="" />

<!-- badges: start -->

<!-- badges: end -->

{kuzco} is a simple vision boilerplate built for ollama in R, on top of
{ollamar} & {ellmer}. {kuzco} is designed as a computer vision
assistant, giving local models guidance on classifying images and return
structured data. The goal is to standardize outputs for image
classification and use LLMs as an alternative option to keras or torch.

{kuzco} currently supports: classification, recognition, sentiment, text
extraction, alt-text creation, and **custom** computer vision tasks.

## Installation

You can install the development version of kuzco like so:

``` r
devtools::install_github("frankiethull/kuzco")
```

**kuzco 0.1.0 can be installed from CRAN via
`install.packages("kuzco")!`**

## Example

This is a basic example which shows you how to use kuzco.

``` r
library(kuzco)
library(ollamar)
```

here we have an image and want to learn about it:

``` r
test_img <- file.path(system.file(package = "kuzco"), "img/test_img.jpg") 
```

<figure>
<img
src="https://raw.githubusercontent.com/frankiethull/kuzco/refs/heads/main/inst/img/test_img.jpg"
alt="picture of puppy odin circa 2019." />
<figcaption aria-hidden="true">picture of puppy odin circa
2019.</figcaption>
</figure>

### llm for image classification:

``` r
llm_results <- llm_image_classification(llm_model = "qwen2.5vl", image = test_img)
```

``` r
llm_results |> tibble::as_tibble()
#> # A tibble: 1 × 7
#>   image_classification primary_object secondary_object image_description        
#>   <chr>                <chr>          <chr>            <chr>                    
#> 1 animal portrait      puppy          ""               A close-up portrait of a…
#> # ℹ 3 more variables: image_colors <chr>, image_proba_names <chr>,
#> #   image_proba_values <chr>
```

``` r
llm_results |> str()
#> tibble [1 × 7] (S3: tbl_df/tbl/data.frame)
#>  $ image_classification: chr "animal portrait"
#>  $ primary_object      : chr "puppy"
#>  $ secondary_object    : chr ""
#>  $ image_description   : chr "A close-up portrait of a fluffy, curious-looking puppy with a striking patch on its head. The puppy has a white"| __truncated__
#>  $ image_colors        : chr "The image has a palette with shades of white, black, and hints of gray."
#>  $ image_proba_names   : chr "puppy, fur texture, eye, coat"
#>  $ image_proba_values  : chr "[0.85, 0.10, 0.05, 0.05]"
```

### llm for image sentiment:

``` r
llm_emotion <- llm_image_sentiment(llm_model = "qwen2.5vl", image = test_img)

llm_emotion |> str()
#> tibble [1 × 4] (S3: tbl_df/tbl/data.frame)
#>  $ image_sentiment      : chr "positive"
#>  $ image_score          : num 0.8
#>  $ sentiment_description: chr "The soft, warm lighting and the cute features of the puppy create a feeling of happiness and warmth."
#>  $ image_keywords       : chr "cute, friendly, playful, adorable, lovable"
```

### llm for image recognition:

note that the backend of kuzco is flexible as well. This allows users to
specify between ‘ollamar’, which suggests structured outputs, while
‘ellmer’ enforces structured outputs.

``` r
llm_detection <- llm_image_recognition(llm_model = "qwen2.5vl", 
                                       image = test_img,
                                       recognize_object = "nose")

llm_detection |> str()
#> tibble [1 × 4] (S3: tbl_df/tbl/data.frame)
#>  $ object_recognized : chr "TRUE"
#>  $ object_count      : int 1
#>  $ object_description: chr "A black and white puppy nose, slightly pink inside with dark round nostrils."
#>  $ object_location   : chr "center"
```

### llm for image text extraction:

kuzco is also useful for OCR tasks, extracting text from images is
showcased below:

``` r
text_img <- file.path(system.file(package = "kuzco"), "img/text_img.jpg") 

text_img |> view_image()
```

<img src="man/figures/README-unnamed-chunk-8-1.png" width="100%" />

``` r
llm_extract_txt <- llm_image_extract_text(llm_model = "qwen2.5vl", 
                                          image = text_img,
                                          backend  = "ellmer")

llm_extract_txt |> str()
#> tibble [1 × 2] (S3: tbl_df/tbl/data.frame)
#>  $ text            : chr "Picture of Odin\nas a puppy\ncirca Q4 2019"
#>  $ confidence_score: num 0.99
```

## newer features

### llm image customization:

a new feature in kuzco, is a fully customizable function. This allows
users to further test computer vision techniques without leaving the
kuzco boilerplate.

``` r
llm_customized <- llm_image_custom(llm_model = "qwen2.5vl", 
                                   image = test_img,
                                   system_prompt = "you are a dog breed expert, you know all about dogs. 
                                                    tell me the primary breed, secondary breed, and a brief description about both.",
                                   image_prompt  = "tell me what kind of dog is in the image?",
                                   example_df = data.frame(
                                     dog_breed_primary = "hound",
                                     dog_breed_secondary = "corgi",
                                     dog_breed_information = "information about the primary and secondary breed"
                                   ))

llm_customized |> str()
#> 'data.frame':    1 obs. of  3 variables:
#>  $ dog_breed_primary    : chr "terrier"
#>  $ dog_breed_secondary  : chr "spotted"
#>  $ dog_breed_information: chr "The primary breed is likely a terrier based on the facial features and compact size. The secondary breed is 'sp"| __truncated__
```

### additional enhancements:

#### i/o helpers

kuzco now has `view_image` & `view_llm_results` functions within the
package, making it easy to view images and display llm results. In
addition to this, kuzco now features `kuzco_app` a fully functioning
shiny application within the package. Making it even easier to do
computer vision with LLMs in R.

#### cloud-based LLMs

kuzco now supports all LLM providers that are supported by `ellmer`!
That’s correct, you can now send images to Perplexity, Claude, OpenAI,
Gemini, the list goes on. This defaults to “ollama” to maintain the
original workflows.

*Cloud-hosted LLMs generally offer greater speed and more advanced
capabilities, but require users to obtain an API key since inference is
handled remotely. While some providers offer a free tier with usage
limits, others do not. Keep in mind that using a cloud-hosted LLM comes
with less privacy compared to running a model locally, but it enables
access to powerful, cutting-edge models. To get started, users should
set up their API key in their environment and select a provider-hosted
model that supports image processing.*

A mistral example below using pixtral-12b, which is still a pretty small
model. But leverages mistral’s compute, instead of yours.

``` r
# via base R:
Sys.setenv(MISTRAL_API_KEY = "the_api_key_via_the_provider")
# or usethis:
usethis::edit_r_environ()
```

``` r
kuzco::llm_image_classification(provider = "mistral", llm_model = "pixtral-12b", image = test_img)
```
