In linguistics this is known as genericization (thanks to Riva Quiroga for teaching me the term), and this concept also applies to software. For example, referring to any spreadsheet as an “Excel” . This also happens a lot with R and RStudio:

• R is the programming language and statistical computing environment that has existed for more than a quarter of a century.
• RStudio is an integrated development environment (IDE) that makes it easier to work with R. It has been available since 2011, pretty much when I started learning to code.

In Latin America, and at least in my experience, there is little conceptual distinction between the two. The R instance running inside RStudio is perceived as just another pane in the interface, and the name of the language itself is rarely mentioned explicitly.

In addition to the confusion between the language and the IDE, there is a frequent mixing of visual identity: the RStudio logo is used instead of the official R logo. Interestingly, this tendency is less common in English-language educational offerings, where references to the R language predominate over the IDE.

Although it’s not my area of specialization, I have not noticed a similar confusion with Python. References in courses, theses, or manuals usually focus on the language itself, not on a specific IDE. For example, it’s rare to see courses titled “Introduction to statistics with PyCharm” or “Spyder for biologists”. I assume this is because Python doesn’t have a single dominant IDE.

I brought this up with some colleagues, we concluded that this ambiguity could originate in academic or student communities that perceive RStudio as “another program for doing statistics” that is free and user-friendly in contrast to commercial alternatives like SPSS or SAS. Since RStudio is the executable we interact with daily, its name ends up overshadowing that of the underlying language.

When I teach introductory courses, I always clarify that R is not RStudio:

• R is the programming language: it can be run from the terminal, its native GUI (on Windows), or any IDE.
• RStudio is a graphical interface that makes working with R easier. It’s popular, convenient, and my usual choice for teaching, but it’s not mandatory nor the only option.

In fact, when downloading RStudio from the Posit website, the instructions indicate installing R first and specify that R is not a Posit product. However, I noticed that a clear explanation is missing regarding R is an independent project managed by the R Project for Statistical Computing, with its own foundation (R Foundation) and development model.

At the end of 2024 I presented at Nerdear.la about Positron, Posit’s new IDE for data science. While researching the topic, I learned that IDEs have existed for more than 30 years and that for R there have been multiple options that never achieved massive popularity like:

• Eclipse StatET™
• Nvim-R (for Neovim)
• RKWard
• R extensions for VSCode

To date, most people who program in R continue using RStudio. I have noticed more interest in Positron, especially among those who closely follow social media, events, meetups, etc. In November 2025 I incorporated Positron as an optional tool in my courses, with somewhat positive results and above all to reinforce the conceptual distinction between language and IDE.

This post is mostly to organize my observations about the confusion between R and RStudio, as reading material in my courses, to generate discussion in the community, and as background for in-depth work I want to do on the barriers and opportunities in the adoption of new tools including IDEs. I understand that those who use the terms flexibly are not seeking to undermine the work of the people who maintain the language, nor the R Project for Statistical Computing, nor the R Foundation/R Consortium. I also don’t think Posit PBC (the company that developed RStudio and that was formerly called RStudio) seeks to monopolize the language’s identity. More than anything, it seems to me to be an interesting linguistic and cultural phenomenon.

En lingüística ésto se conoce como antonomasia o antonomasia comercial (gracias a Riva Quiroga por enseñarme el término), y este concepto también aplica a software. Por ejemplo, decirle “un Excel” a cualquier hoja de cálculo. Esto también pasa mucho con R y RStudio:

• R es el lenguaje y entorno de programación estadística que existe desde hace más de un cuarto de siglo.
• RStudio es un entorno de desarrollo integrado (IDE, por sus siglas en inglés) que facilita trabajar con R. Está disponible desde el 2011, justo coincidiendo con mis inicios en la programación.

En Latinoamérica, y al menos en mi experiencia, hay poca distinción conceptual entre ambos. La instancia de R que corre dentro de RStudio se percibe como un panel más de la interfaz, y el nombre del lenguaje como tal rara vez se menciona de forma explícita.

Además de la confusión entre el lenguaje y el IDE, hay una mezcla frecuente de identidad visual: se usa el logo de RStudio en lugar del logo oficial de R. Curiosamente, esta tendencia es menos común en la oferta educativa en inglés, donde predominan referencias al lenguaje R y no al IDE.

Aunque no es mi área de especialización, no he notado una confusión similar con Python. Las referencias en cursos, tesis o manuales suelen centrarse en el lenguaje mismo, no en un IDE específico. Por ejemplo, es raro ver cursos titulados “Introducción a la estadística con PyCharm” o “Spyder para biólogos”. Supongo que esto se debe a que Python no tiene un IDE dominante único.

Hablando con colegas, concluimos que esta ambigüedad podría originarse en comunidades académicas o estudiantiles que perciben a RStudio como “un programa más para hacer estadística” —gratuito y amigable— en contraste con alternativas comerciales como SPSS o SAS. Como RStudio es el ejecutable con el que interactuamos cotidianamente, su nombre termina opacando al del lenguaje subyacente.

Cuando imparto cursos introductorios, siempre aclaro que R no es RStudio:

• R es el lenguaje de programación: puede ejecutarse desde la terminal, su GUI nativa (en Windows) o cualquier IDE.
• RStudio es una interfaz gráfica que facilita el trabajo con R. Es popular, cómodo, y mi elección habitual para enseñar, pero no es obligatorio ni la única opción.

De hecho, al descargar RStudio desde el sitio de Posit, las instrucciones indican instalar primero R y aclaran explícitamente que R no es un producto de Posit. Sin embargo, noté que falta una explicación clara de que R es un proyecto independiente gestionado por el R Project for Statistical Computing, con su propia fundación (R Foundation) y modelo de desarrollo.

A finales del 2024 preparé materiales para presentar en Nerdear.la sobre Positron, el nuevo IDE de Posit para ciencia de datos. Al investigar, aprendí que hay IDEs desde hace más de 30 años y que para R han habido múltiples opciones que nunca alcanzaron popularidad masiva como:

• Eclipse StatET™
• Nvim-R (para Neovim)
• RKWard
• Extensiones de R para VSCode

A la fecha, la mayoría de quienes programan en R siguen usando RStudio. Igual he notado más interés en Positron, especialmente entre quienes siguen de cerca las novedades de la comunidad (redes sociales, eventos, etc.). En noviembre de 2025 incorporé Positron de forma opcional en mis cursos, con resultados medianamente positivos y sobre todo para reforzar la distinción conceptual entre lenguaje e IDE.

Este texto es más que nada para organizar mis observaciones sobre la confusión entre R y RStudio, para que sirva como lectura en mis cursos, para generar discusión en la comunidad y como antecedentes para un trabajo más profundo sobre las barreras y oportunidades en la adopción de nuevas herramientas incluyendo IDEs. Entiendo que quienes usan los términos flexiblemente no buscan invisibilizar el trabajo de la gente que mantiene el lenguaje, ni el R Project for Statistical Computing, ni la R Foundation/R Consortium. Tampoco creo que Posit PBC (la empresa que desarrolló RStudio y que antes se llamba RStudio) busque monopolizar la identidad del lenguaje. Más que nada me parece un fenómeno lingüístico y cultural interesante.

¹ these [1,2,3,4] entries document my descent into insanity journey towards making R play nice with messy spreadsheets.

As a follow-up, this post summarizes patterns and prevalence of cell and text formatting in thousands of real-world spreadsheets from different fields. I did this out of curiosity, but also to help me improve the functionality of my unheadr and forgts packages, both of which are meant to help us work with formatted spreadsheets.

Before getting started, here is some recommended (mandatory) reading:

• Six Tips for Better Spreadsheets. Perkel (2022). Nature (PDF HERE) This brief writeup features insights from lots of experts in the data community

• Data Organization in Spreadsheets. Broman & Woo (2018). The American Statistician (full text here) THE definitive guide

The process

Broadly, this exploration has thee main steps: 1) obtain spreadsheets, 2) extract formatting data, and 3) summarize the patterns.

The code for each step available in this repo, which includes 25 random files for demonstration. The code runs fine but it can get sluggish, and includes some weird stylistic choices that come from trying out different LLM assistants for my ongoing LLMs + R roundup.

1. Get thousands of spreadsheets

Rather than scour the web myself, I downloaded the Fuse dataset. The Fuse corpus includes a collection of 249,376 unique spreadsheets from obtained by Barik et al. (2015) from the Common Crawl index.

This archive is several gigabytes in size. I worked locally using ugly absolute paths and only analyzed a random subset (~3000 of ~250,000) of the files.

Some of the files look like this:

2. Batch process the formatting

The forgts pacakge already has functions (derived from code in unheadr) to extract formatting data from spreadsheets. With slight modifications, these functions can then be used iteratively to process files and ultimately summarize the formatting in each file.

The spreadsheets in Fuse are binary XLS files, so first we need to convert them to xlsx using libreoffice through the command line so that tidyxl can pull the formatting data stored in the underlying XML.

Internally, the functions call this system command whenever an input is not xlsx.

libreoffice --headless --convert-to xlsx myfile

(another option could be to batch convert all binary files first)

I checked some of the conversions and didn’t notice any of the formatting breaking or being lost. I’m not sure but hopefully that’s the case for all the files. This implies having libreoffice installed and running the code with root privileges.

Several of the files were multisheet workbooks but the code only processes one sheet per file, sampled at random.

For a spreadsheet that looks like this:

The functions will output a list of two data frames that look like this:

$combined_format_summary
# A tibble: 3 × 8
  styling_arg cell_fill_count cell_text_count cell_fill_unique_values  
  <chr>                 <int>           <int> <chr>                    
1 color                     7               1 #FF4000, #729FCF, #FFFF00
2 decorate                 NA               3 NA                       
3 weight                   NA               2 NA                       
# ℹ 4 more variables: cell_text_unique_values <chr>, fileid <chr>,
#   sheet <chr>, file_path <chr>

$combined_overall_stats
# A tibble: 6 × 7
  metric                 value color_code occurrences fileid sheet file_path
  <chr>                  <dbl> <chr>            <int> <chr>  <chr> <chr>    
1 Percentage of Cells w…  68.8 NA                  NA binar… 1 of… binary/p…
2 Percentage of Variabl… 100   NA                  NA binar… 1 of… binary/p…
3 Most Common Color 1     NA   #FFFF00              4 binar… 1 of… binary/p…
4 Most Common Color 2     NA   #729FCF              2 binar… 1 of… binary/p…
5 Most Common Color 3     NA   #8D1D75              1 binar… 1 of… binary/p…
6 Most Common Color 4     NA   #FF4000              1 binar… 1 of… binary/p…

3. Overall summary

With two very long data frames that hold overall summaries and more detailed formatting information, some simple data operations can tell us that:

Total files: 3015

Files with formatting: 1877

Percentage of files with formatting: 62%

Mean percentage of cells formatted per sheet: 49%

The most commonly used color (for text, borders, or cell fills) was blue (hex code #0000FF), used in 228 files

In terms of text formatting (italic and bold text) that may or may not be used to encode data (e.g., bold text not just for emphasis but to indicate negative values), we get:

Proportion of files using:

Bold only: 47%;
Italic only: 2%;
Both bold and italic: %12;
Any (bold, italic, or both): 62%

This subset of ~3000 files is small compared with the entire corpus and the result of 62% of files using cell and text formatting is much higher than the 25% reported by Sing et al. (2022) for a massive corpus of 1.8 million files examined.

Even if the real proportion of spreadsheets with formatting is low, and especially formatting used to encode data, I still think it is worth finding ways to deal with it in R.

This is particulary relevent nowadays, as there is a growing need to transform complex documents into simpler representations that can be fed into large language models. We just need to look at how Microsoft recently released markitdown, a powerful python tool for converting various files to Markdown for indexing, text analysis, and other AI-related purposes.

I’ll continue working on forgts to convert formatted spreadhsheets to gts, and hopefully also rewrite some of the functions in unheadr that paste the formatting information as strings when importing these kinds of files.

All feedback welcome

All the contents of this post (and much more) now live in this standalone Quarto Guide

Please note: This post has been updated twice thrice in January 2025 to keep up with new developments in the field

This year has seen significant progress in the genAI/LLM space, and lately a lot of these tools have been integrated with R in various ways. This is nice, because not everything needs to be a standalone chat box in the web browser - which seems prone to misuse (e.g., using things like chatgpt as a search engine for no valid reason).

To help me keep track of what’s happening, I’ve put together this (potentially incomplete) list of relevant LLM+R resources.

First, it’s worth mentioning this perspective in Methods in Ecology and Evolution: Harnessing large language models for coding, teaching and inclusion to empower research in ecology and evolution, led by Natalie Cooper, and part of a special issue on the use of LLMs in ecology and evolution. These papers summarize the pros and cons of genAI/LLMs in the context of research and teaching, and their conclusions go beyond research and biological sciences.

Now, in no particular order, here is a roundup of some notable developments that I’ve come across online.

pal

pal by Simon Couch provides easy to use assistants that can edit, document, or explain code. The package provides an addin that works in both RStudio and Positron. Very cute package logo, and pal seems like a good option for writing boilerplate code and automating some of the more tedious, repetitive tasks. Works with multiple underlying models. Haven’t tried it yet.

• Here’s a tutorial in Spanish for using pal with custom assistants that can have different roles.

kuzco

kuzco by Frank Hull & Johannes Breuer leverages ollamar and ellmer to provide a computer vision assistant right inside R, using LLMs as an alternative to torch for tasks including image classification, recognition, sentiment, and text extraction. The output provided comes in very nice and usable tibbles.

Cool hex logo and the example in the readme features a very cute puppy :3

gander

Simon Couch is on a roll, and released the cool gander package in early 2025.

gander leverages ellmer to go beyond a chat box and actually incorporate a context-aware assisant into our IDE of choice. Works well in both RStudio and Positron, and can actually look for context in files and inside R environment (variable names, objects, function definitions, etc.)

This short screen recording comes from the package readme and shows the tool in action:

This feels similar to using the continue extension in Positron but more oriented to R, and I’ll likely be using this going forward.

ellmer

note that ellmer was previously called elmer, and was renamed in December 2024 to avoid case-insensitive clash with ELMER on bioconductor.

ellmer is a new tidyverse-adjacent package by Hadley Wickham and Joe Cheng for interacting in R with various models, either programmatically or interactively. elmer creates R6 chat objects that remember context, and we can interact with models with a console or browser-based chat box, or programmatically within R functions.

Seems promising, quite flexible, and all the activity in the GitHub repo suggests a very active development process, a solid dev team, and lots of community input.

mall

mall is part of the mlverse ecosystem of open source Data Science and Machine Learning libraries. Rather than a chat-based approach, mall applies LLMs rowwise in the columns of a data frame. Built-in prompts include translating, summarizing, extraction, and sentiment analysis of text strings.

mall uses Ollama and is implemented for both R and Python. I will likely be using it to analyze the comments I collected about loaded packages, which I talked about in my posit::conf(2024) talk.

tidyllm

tidyllm by Eduard Brüll provides pipe-friendly access to multiple LLM APIs in a smooth and readable way. We can pass images, documents, videos, and plots from the plot pane to the package functions, and the package has very thorough documentation. Similar in some ways to ellmer, but looks very useful for both batch processing and interactive work.

gemini.R

gemini.R by Jinhwan Kim connects R with Google’s gemini model via the gemini API. With a valid API key, the gemini() function takes text prompts and the gemini_image() function can work with images and text prompts.

The package algo provides an RStudio addin for creating Roxygen documentation.

GiHub Copilot

GitHub Copilot is a widely-used and well-documented coding assistant for code completions and the possibility of turning natural language prompts into code suggestions. Works on GitHub.com and inside most IDEs.

Subscription-based, although as of a a few weeks-ago there is a free tier available. I have not tried it yet.

This recent talk by Yanina Bellini for RLadies Rome provides a good overview, and Yani’s slides also mention important considerations about the AI skill threat.

continue

A vscode extension that works nicely in Positron. Supports multiple models for chat and code completion. Easy to provide local files and folders for context and nice integration with the source editor.

Tried it out after Julia Silge mentioned it in the Super Data Science podcast. Works nicely, and I used it in Positron with Anthropic’s Claude 3.5 Sonnet model to write and edit the repetitive css, html, and javascript code that powers the hexsession package. Also helped me with some of the nested for loops that play a big role in forgts.

ensure

ensure by Simon Couch helps write code for unit tests using the testthat package. Works through an Rstudio addin, and the documentation mentions that the model has been made aware of testthat syntax and the tidy style guide for code. Will be trying it out for my more recent packages that still have poor test coverage.

codeium

Another cool VScode extension that works well in Positron. Provides autocomplete and chat. Free for individuals. The base model uses Llama 3.1 70B, but paid tiers can choose other models. I’ve tried this out in Positron with good results for code refactoring, at least when playing around with small scripts.

llmR

llmr by Angelo D’Ambrosio provides a unified API to interact with various LLMs and providers through functions with consistent grammar and syntax. Provides easy switching between models, plus logging. Seems similar to elmer. Will try it soon.

• A similar package called LLMR recently appeared on CRAN but I could not find much materials about it.

lang

lang is also part of the mlverse. This package uses LLMs to translate R documentation and display it in the help pane of Rstudio or Positron. Having participated in various translation initiatives, I am very wary of machine-translated function documentation and how it may affect new learners.

The best part of this package (in my opinion) is the infrastructure for package developers to help translate documentation that after editing can be shipped as part of package with multulingual help files.

ollamar

Developed by Hause Lin and Tawab Safi, ollamar Integrates R with Ollama, for running language models locally.

With ollamar we can pull different models, interact with objects that store chat histories, and handle the outputs as data frames, lists, or vectors.

Works nicely with httr2, which is a big advantage.

ragnar

As the name implies, ragnar by Tomasz Kalinowski helps implement Retrieval-Augmented Generation (RAG) workflows in R with transparency and efficiency. Uses duckdb by default for efficient work with big data and currently supports ollama (more models likely to come soon). RAG is a way to link generative models with external resources rich in technical details to ultimately enhance the accuracy and reliability of models with real, citable facts.

I liked the example in the readme, showing how the R for Data Science book was ingested and queried using the package.

Special mention here for Shiny Assistant, a stand-alone AI tool from the Shiny team that can answer questions about Shiny and aldo build apps. Shiny assistant works with both R and python, and the tool takes advantage of Shinylive to optionally forego the need for a server running R or python.

If I missed anything let me know and I’ll add it here!

El paquete hexsession sigue en desarrollo activo, y ahora ya funciona con conjuntos arbitrarios de imágenes. Como ejemplo, aquí dejo una guía de los pasos que podemos seguir para:

• descargar imágenes aleatorias de perros de la API de Dog CEO
• redimensionar y convertir todas las imágenes a blanco y negro en un solo paso
• armar un mosaico hexagonal de las imágenes, cada una con un URL opcional

Siempre se me olvida cómo trabajar con APIs y como procesar varias imágenes en simultáneo, así que espero que esto sea útil para otros y que también me quede de referencia.

El resultado final está hasta arriba,se ve bien y este tipo de mosaicos se pueden usar para paquetes de software y sus logos, o tal vez para personas en una organización con fotos y enlaces al perfil de cada integrante.

Como siempre, el código en estos bloques es reproducible con una conexión a internet y los paquetes relevantes instalados.

Descargar fotos de perros

Podemos descargar fotos de perros de la API de Dog CEO. Esta API tiene un endpoint para obtener fotos de diferentes razas. Las imágenes vienen del Stanford Dogs Dataset más contribuciones de la comunidad. Podemos usar httr2 para enviar una solicitud GET que devuelve una lista de URLs de imágenes. Para llevarla leve con los servidores, hice toda la iteración aquí haciendo pausas (slowly con purrr). Primero obtenemos la lista de razas, seleccionamos algunas al azar, agregamos dos que me gustan y luego descargamos las fotos.

library(httr2)
library(purrr)
library(stringr)
library(magick)

# definir función para obtener una imagen aleatoria de una raza
randomDog <- function(breed) {
  # construir URL
  url <- paste0("https://dog.ceo/api/breed/", breed, "/images/random")
  # solicitud
  req <- request(url)
  resp <- req |>
    req_perform()
  # respuesta JSON
  data <- resp_body_json(resp)
  data$message
}

# todas las razas
allbreeds <- 
  request("https://dog.ceo/api/breeds/list") |> 
  req_perform() |>
  resp_body_json()
breedsvec <- simplify(allbreeds$message)

# lentamente
rate <- rate_delay(0.8)
slow_randomdog <- slowly(\(x) randomDog(x), rate = rate, quiet = FALSE)
# 15 razas aleatorias más dos que me gustan
randomdogsvec <- sample(breedsvec, 15)
randomdogsvec <- c(randomdogsvec, "mastiff/bull","retriever/golden")
# obtener las URLs 
dogphotourls <- map(randomdogsvec, slow_randomdog)

# carpeta y rutas para guardar las imágenes
fs::dir_create("dogimgs")
savepaths <- paste0("dogimgs/",str_replace(randomdogsvec,"/","-"),".jpg") 
slowdl <- slowly(\(x,y) download.file(x,y), rate = rate, quiet = FALSE)
walk2(dogphotourls,savepaths,\(x,y) slowdl(x,y))

Procesar las imágenes

Podemos usar magick para redimensionar todas las imágenes a una altura uniforme de 230px, pasarlas a blanco y negro, y exportar cada una al disco en una nueva carpeta. El truco aquí fue aplicar as.list() al objeto de imágenes apiladas de magick en el paso final.

# directorio y rutas para las imágenes procesadas
fs::dir_create("dogimgs/processedimgs")
processedimgpaths <- paste0("dogimgs/processedimgs/",basename(savepaths))

# leer y procesar
allimgs <- image_read(savepaths)
procimgs <- allimgs |> 
  image_scale("230x") |>  # redimensionar
  image_convert(colorspace = 'gray') |> # desaturar
  image_contrast() # más contraste 

# escribir en disco
walk2(as.list(procimgs),processedimgpaths,\(x,y) image_write(x,y))

Crear el mosaico

Ahora que hexsession acepta conjuntos arbitrarios de rutas de imágenes, podemos trabajar con un vector de rutas. Agregué la misma URL a cada imagen solo para mostrar que la salida HTML funciona.

# reiniciar sesión primero para que no haya paquetes cargado
dogpaths <- fs::dir_ls("dogimgs/processedimgs/")
cturls <- rep("https://betterpet.com/why-are-dogs-so-cute/",length(dogpaths))
hexsession::make_tile(local_images = dogpaths,local_urls = cturls)

Para mostrar la salida HTML en esta publicación, puse el archivo HTML autocontenido en _includes/ y me refiero a él con {% include %} .

Quedo al pendiente para cualquier duda. salu-2

As I continue to work on the hexsession package, which now works for arbitrary sets of images, here’s a walkthrough of the steps needed to:

• download random dog images from the Dog CEO API
• batch resize the images and make them black and white
• make a hexagonal tile of the images, each with an optional URL

I keep forgetting how to do the first two steps, so hopefully others find this helpful and I can also come back to this when I get stuck.

The final output is embedded above, which is pretty nice and this kind of thing can be used for software packages, or maybe for people in an organization with headshots and links to each member’s profile.

As usual, the code in these blocks below should be reproducible as long as you have an internet connection and the relevant packages installed.

Download dog photos

Let’s download dog photos from the Dog CEO API. This API serves images for different breeds, built from the Stanford Dogs Dataset plus community contributions. We can use httr2 to send a GET request to the API and retrieve a list of image URLs. To be nice to the servers I did all the iteration here slowly with purrr. First we get the list of breeds, select some at random, add two that I like, and them download the photos.

library(httr2)
library(purrr)
library(stringr)
library(magick)

# define fn to get a random image for a breed
randomDog <- function(breed) {
  # build URL
  url <- paste0("https://dog.ceo/api/breed/", breed, "/images/random")
  #  request
  req <- request(url)
  resp <- req |>
    req_perform()
  # JSON response
  data <- resp_body_json(resp)
  data$message
}

# get all the breeds
allbreeds <- 
  request("https://dog.ceo/api/breeds/list") |> 
  req_perform() |>
  resp_body_json()
breedsvec <- simplify(allbreeds$message)

# being nice
rate <- rate_delay(0.8)
# wrap the new fn in slowly
slow_randomdog <- slowly(\(x) randomDog(x), rate = rate, quiet = FALSE)
# 15 random breeds plus two that I like
randomdogsvec <- sample(breedsvec, 15)
randomdogsvec <- c(randomdogsvec, "mastiff/bull","retriever/golden")
# get the image urls
dogphotourls <- map(randomdogsvec, slow_randomdog)

# folder and paths to save the images
fs::dir_create("dogimgs")
savepaths <- paste0("dogimgs/",str_replace(randomdogsvec,"/","-"),".jpg") 
slowdl <- slowly(\(x,y) download.file(x,y), rate = rate, quiet = FALSE)
walk2(dogphotourls,savepaths,\(x,y) slowdl(x,y))

Batch process the photos

We can use magick to resize all the images to a uniform height of 230px, desaturate them, and export each one to disk in a new folder. The trick here was to apply as.list() to the magick image stack in the final step.

# process images
# folder and paths for the processed images
fs::dir_create("dogimgs/processedimgs")
processedimgpaths <- paste0("dogimgs/processedimgs/",basename(savepaths))

# read and process
allimgs <- image_read(savepaths)
procimgs <- allimgs |> 
  image_scale("230x") |>  # resize
  image_convert(colorspace = 'gray') |> # desaturate
  image_contrast() # enhance contrast 

# write to disk
walk2(as.list(procimgs),processedimgpaths,\(x,y) image_write(x,y))

Create the tile

Now that hexsession takes arbitrary sets of image paths, we can just build a tile with a vector of paths. I added the same url to each image just to show that the html output works.

# restart session to detach any loaded packages
dogpaths <- fs::dir_ls("dogimgs/processedimgs/")
cturls <- rep("https://betterpet.com/why-are-dogs-so-cute/",length(dogpaths))
hexsession::make_tile(local_images = dogpaths,local_urls = cturls)

To embed the output in this post, I put the self-contained html file in _includes/ and refer to it with {% include %} .

nice!

hexsession

This package creates a nice interactive tile of hex logos for either:

• the libraries attached to the search path in a session (loaded packages) or,
• an arbitrary set of packages as long as they’re installed

The idea came about after witnessing more sticker madness at positconf 2024 and seeing people arrange their hexagonal stickers in nice tiles.

Maëlle Salmon had already written about arranging images of hex logos using magick, and I liked the way hexagonal logos are arranged in html format on the hexbin website with a subtle css hover effect.

After much tinkering, the hexsession package can now use the logo images that come bundled with most packages and arrange them in a responsive, self-contained html tile, which can then be saved as an image. This is done through Quarto and some ugly js code.

Possible uses for these tiles include crediting packages by adding their logos in a tile to presentations/reports/websites, or displaying the logos por the components of a metapackage or package collection.

Note that not all packages bundle their logos in man/figures and the functions are still work in progress. There has been some feedback in the GiHub issues, but I’ve yet to see people making nice logo tiles. Please make some and tag me!

forgts

Spreadsheets will always be around they will usually have cell or text formatting. The forgts package reads the data in a spreadsheet, imports the text and cell formatting (e.g. font face and color, cell borders, cell fills, etc.), and applies it to a gt version of the data.

This idea came from me having to take screenshots of formatted spreadsheets so that I could show things in presentations for teaching. I’m not sure how the functions will behave with large workbooks or really crazy formatting. The package is quite new but please try it out!

Thanks for reading.

The need for this arose because it is fairly common for government agencies to split related data into multiple files for individual download, with each file for example: containing data for a specific year or region. However, this can make it difficult to combine all the separate files into a global dataset when important variables are not recorded explicitly in the data.

For this example, let’s work with the 🚗 gtcars dataset from the gt package. This dataset is like mtcars but for very fancy and premium cars.

We’ll group gtcars by multiple varibles, split the data into groups, and export each group to a separate csv file without the grouping variables but with this information in the file names. Then, we’ll import all the files, splitting the file names and adding the relevant information back into the data rectangle. This will allow us to combine all the content into a single dataset.

📂 Create a folder with multiple csv files

First, load everything we need.

# load gt and various tidyverse pkgs
library(gt)      # CRAN v0.10.1 
library(dplyr)   # CRAN v1.1.4
library(purrr)   # CRAN v1.0.2
library(glue)    # CRAN v1.7.0
library(stringr) # CRAN v1.5.1
library(readr)   # CRAN v2.1.5
library(fs)      # CRAN v1.6.4

data(gtcars) # get the data 

Now, we can group the data by country of origin, year, and drive train and then a) split the data into a list of tibbles (one for each group), and b) use the group keys to build a vector of file names based on the grouping information used to split the data.

# list of tibbles
gtcars_groups <- gtcars |> 
  group_split(ctry_origin, year, drivetrain)

# tibble of the grouping variables
gtgroups <- 
gtcars |> 
  group_by(ctry_origin, year, drivetrain) |> 
  group_keys() 

After minor changes like replacing spaces, the vector of filenames can be built with rowwise glueing of the values in the grouping variables.

grpnames <- 
    gtgroups |> 
      rowwise() |> 
      mutate(gluedvars = glue_collapse(across(everything(), as.character), sep = "_")) |> 
      mutate(gluedvars= str_replace(gluedvars," ","-")) |> 
      pull(gluedvars)

🗋 Let’s use this vector of filenames to name the list of tibbles, and then export each tibble to a separate csv file using walk2 for its side effects.

names(gtcars_groups) <- grpnames
gtcars_groups <- 
  gtcars_groups |> map(select,-c(ctry_origin,year,drivetrain))
# to disk  
  walk2(gtcars_groups,names(gtcars_groups),
        \(x,y) write_csv(x,paste0("carfiles/",y,".csv")))

We should now have in our working directory: a folder (“carfiles” in this example) full of csv files.

Now let’s write a function that will strip the filename, split it into fragments (one for each variable), and add these fragments as values into the data rectangle as part of the import step. Using read.csv here to get around guessing and type conversion.

filenameTovars <- function(filepath, var_names = NULL) {
  # strip the filename
  filename <- basename(filepath)
  filename <- tools::file_path_sans_ext(filename)
  
  # split the filename 
  filename_parts <- str_split(filename, "_", simplify = TRUE)
  
  # import
  data <- read_csv(filepath)
  
  # if no var_names provided
  if (is.null(var_names))  {
    var_names <- paste0("v", seq_along(filename_parts))
  }

  # filename fragments as new variables
  as_tibble(bind_cols(
    data,
    setNames(as.list(filename_parts), var_names) # a df that will be recycled
  ))
}

Nice little function above, it can take an optional vector with the names of the variables being added and if none are provided, it will use “v1”, “v2”, etc. Hard-coded “_” as separator here, but that could be an argument too.

For one of the files chosen at random, let’s compare the output from reading the csv normally vs applying the new function with and without the vector of variable names.

# read the csv normally
read_csv("carfiles/Italy_2015_rwd.csv")

# read with path only
filenameTovars("carfiles/Italy_2015_rwd.csv") 

# read with names for new columns
filenameTovars("carfiles/Italy_2015_rwd.csv",
        var_names = c("ctry_origin","year","drivetrain")) 

The csv file without the data from the filename.

# A tibble: 6 × 12
  mfr   model trim  bdy_style    hp hp_rpm   trq trq_rpm mpg_c mpg_h trsmn
  <chr> <chr> <chr> <chr>     <dbl>  <dbl> <dbl>   <dbl> <dbl> <dbl> <chr>
1 Ferr… 458 … Base… coupe       597   9000   398    6000    13    17 7a   
2 Ferr… 458 … Base  converti…   562   9000   398    6000    13    17 7a   
3 Ferr… Cali… Base… converti…   553   7500   557    4750    16    23 7a   
4 Ferr… F12B… Base… coupe       731   8250   509    6000    11    16 7a   
5 Ferr… LaFe… Base… coupe       949   9000   664    6750    12    16 7a   
6 Lamb… Hura… LP 6… coupe       610   8250   413    6500    16    20 7a   
# ℹ 1 more variable: msrp <dbl>

Making this information explicit, but the new variables are named v1, v2, and v3.

# A tibble: 6 × 15
  mfr   model trim  bdy_style    hp hp_rpm   trq trq_rpm mpg_c mpg_h trsmn
  <chr> <chr> <chr> <chr>     <int>  <int> <int>   <int> <int> <int> <chr>
1 Ferr… 458 … Base… coupe       597   9000   398    6000    13    17 7a   
2 Ferr… 458 … Base  converti…   562   9000   398    6000    13    17 7a   
3 Ferr… Cali… Base… converti…   553   7500   557    4750    16    23 7a   
4 Ferr… F12B… Base… coupe       731   8250   509    6000    11    16 7a   
5 Ferr… LaFe… Base… coupe       949   9000   664    6750    12    16 7a   
6 Lamb… Hura… LP 6… coupe       610   8250   413    6500    16    20 7a   
# ℹ 4 more variables: msrp <int>, v1 <chr>, v2 <chr>, v3 <chr>

Passing the names for the new columns:

# A tibble: 6 × 15
  mfr   model trim  bdy_style    hp hp_rpm   trq trq_rpm mpg_c mpg_h trsmn
  <chr> <chr> <chr> <chr>     <int>  <int> <int>   <int> <int> <int> <chr>
1 Ferr… 458 … Base… coupe       597   9000   398    6000    13    17 7a   
2 Ferr… 458 … Base  converti…   562   9000   398    6000    13    17 7a   
3 Ferr… Cali… Base… converti…   553   7500   557    4750    16    23 7a   
4 Ferr… F12B… Base… coupe       731   8250   509    6000    11    16 7a   
5 Ferr… LaFe… Base… coupe       949   9000   664    6750    12    16 7a   
6 Lamb… Hura… LP 6… coupe       610   8250   413    6500    16    20 7a   
# ℹ 4 more variables: msrp <int>, ctry_origin <chr>, year <chr>,
#   drivetrain <chr>

Finally, we can use purrr to iterate through all the files in a folder and apply the custom function. Note the new recommended approach of using list_rbind rather than map_df.

map(allpaths,\(x) filenameTovars(x,
  var_names = c("ctry_origin","year","drivetrain"))) |> 
  list_rbind() |> type_convert() 

Our output is essentially the same gtcars object we started with.

# A tibble: 47 × 15
   mfr   model    trim    bdy_style    hp hp_rpm   trq trq_rpm mpg_c mpg_h
   <chr> <chr>    <chr>   <chr>     <int>  <int> <int>   <int> <int> <int>
 1 Audi  R8       4.2 (M… coupe       430   7900   317    4500    11    20
 2 BMW   i8       Mega W… coupe       357   5800   420    3700    28    29
 3 Audi  RS 7     Quattr… hatchback   560   5700   516    1750    15    25
 4 Audi  S6       Premiu… sedan       450   5800   406    1400    18    27
 5 Audi  S7       Presti… hatchback   450   5800   406    1400    17    27
 6 Audi  S8       Base S… sedan       520   5800   481    1700    15    25
 7 BMW   6-Series 640 I … coupe       315   5800   330    1400    20    30
 8 BMW   M4       Base C… coupe       425   5500   406    1850    17    24
 9 BMW   M5       Base S… sedan       560   6000   500    1500    15    22
10 BMW   M6       Base C… coupe       560   6000   500    1500    15    22
# ℹ 37 more rows
# ℹ 5 more variables: trsmn <chr>, msrp <int>, ctry_origin <chr>,
#   year <dbl>, drivetrain <chr>

Hopefully others out there find this useful!

I’ve worked on this topic before, so I when I saw the post I had to meddle. I suggested using functions from the unheadr package to address some of these issues. They didn’t work because of assumptions I hard-coded into the package, but these are fixed now. These fixes are part of unheadr v0.4.0. To demonstrate the functions and also to celebrate 20k downloads (🥳), here is my take (with the benefit of hindsight) on tackling this troublesome spreadsheet. ALL CREDIT TO JEREMY FOR COMING UP WITH THE EXAMPLE AND WORKFLOW.

The spreadsheet/workbook in question can be downloaded from Jeremy’s GitHub here.

For context, I’m reproducing the images from the original post about the issues with the data, which are good examples of:

😢 Things we shouldn’t do but which happen anyway in spreadsheets:

1. A date variable with different cell formats, plus whatever strange thing Excel does to dates.

2. Meaningful formatting

Using text color (and nothing else) to indicate units.

3. Numeric variables with some cells formatted as text.

These issues vary in how hard they are to address in downstream analyses, but as they accumulate we are more tempted to give up and work with the file directly in Excel (or Calc).

Here’s my take, which can be a complement to the stricter approach in Jeremy’s post.

First, read the spreasheet with readxl. Out of caution, we can read everything as text initially.

# load some useful packages
library(readxl)    # CRAN v1.4.3 
library(lubridate) # CRAN v1.9.3 
library(janitor)   # CRAN v2.2.0 
library(readr)     # CRAN v2.1.5 
library(dplyr)     # CRAN v1.1.4
library(tidyxl)    # CRAN v1.0.10 
library(stringr)   # CRAN v1.5.1
library(tidyr)     # CRAN v1.3.1 
library(unheadr)   # CRAN v0.4.0 

samp <- read_excel("sample_excel.xlsx", col_types = "text")

Next, type_convert() from readr does a good job of parsing variables in a data frame, and has good heuristics to interpret numeric variables even when these had problematic combinations of cell formatting in the spreadsheet.

samp <- readr::type_convert(samp)

── Column specification ────────────────────────────────────────────────────────────
cols(
  id = col_character(),
  date_issue = col_character(),
  `colour_weight 
Black in pounds
Green in kilograms` = col_double(),
  text_integer_issue = col_double(),
  text_numeric_issue = col_double(),
  numeric_integer_issue = col_double(),
  one_or_zero_issue = col_double()
)

To fix the awkward date, lubridate can handle the dates that were imported more or less properly, and the excel_numeric_to_date() function from janitor was purpose-built to transform the dates stored as weird numbers. Afterwards we put the two together.

samp$newdate <- lubridate::parse_date_time(samp$date_issue,
  orders = c("ymd", "dmy")
)
samp$xldate <- excel_numeric_to_date(as.numeric(as.character(samp$date_issue)),
  date_system = "modern"
)
samp <- samp %>%
  mutate(date_fixed = coalesce(newdate, xldate)) %>%
  select(-newdate, -xldate, -date_issue)

> samp %>% select(date_fixed)
# A tibble: 1,053 × 1
   date_fixed         
   <dttm>             
 1 2017-11-11 00:00:00
 2 2017-10-18 00:00:00
 3 2017-10-09 00:00:00
 4 NA                 
 5 1971-01-08 00:00:00
 6 1985-11-29 00:00:00
 7 1955-02-12 00:00:00
 8 1982-05-01 00:00:00
 9 1969-04-20 00:00:00
10 1962-11-21 00:00:00
# ℹ 1,043 more rows
# ℹ Use `print(n = ...)` to see more rows

Lastly, there’s the issue of units of measurment for a variable encoded as text formatting. In this case the text color indicates the units. This information is lost when reading only the cell values with readxl, but we can embed the color code from each cell as a text annotation using the annotate_mf functions from unheadr.

> annotate_mf_all("sample_excel.xlsx")
Error in annotate_mf_all("sample_excel.xlsx") :                                   
  Check spreadsheet for blank cells in seemingly empty rows

If we run annotate_mf_all() with the path to the spreadsheet, we’ll get an error message. Apparently there are formatted cells outside of the data rectangle, which is what readxl focuses on during the data import.

We can use tidyxl to unravel the problem, looking at the tail end of the output, we see some blank-formatted cells in rows 1054 and 1055, even though our samp data frame only has 1053 rows. These formatted cells create “ghost” rows that trip up the functions from unheadr.

> nrow(samp)
[1] 1053

spsheetcells <- tidyxl::xlsx_cells("sample_excel.xlsx")
spsheetcells %>%
  select(row, is_blank, data_type) %>%
  tail(10)

# A tibble: 10 × 3
     row is_blank data_type
   <int> <lgl>    <chr>    
 1  1053 FALSE    numeric  
 2  1054 FALSE    character
 3  1054 FALSE    character
 4  1054 FALSE    numeric  
 5  1054 FALSE    numeric  
 6  1054 FALSE    numeric  
 7  1054 FALSE    numeric  
 8  1054 FALSE    numeric  
 9  1055 TRUE     blank    
10  1056 TRUE     blank  

After opening the file directly in a spreadsheet program and deleting the problem rows altogether (I saved this as a new file called sample_excel_cln.xlsx), the annotate_mf_all() function is able to translate the colorful text into a text annotation of the hex8 code for each color.

samp_format <- annotate_mf_all("sample_excel_cln.xlsx")

Let’s have a look

samp_format[, 3]

> samp_format[, 3]
# A tibble: 1,053 × 1
   `colour_weight \nBlack in pounds\nGreen in kilograms`
   <chr>                                                
 1 (color-FF00B050) 92                                  
 2 (color-FF000000) 194                                 
 3 (color-FF00B050) 61                                  
 4 (color-FF000000) 165                                 
 5 (color-FF000000) 148                                 
 6 (color-FF00B050) 86                                  
 7 (color-FF00B050) 84                                  
 8 (color-FF000000) 182                                 
 9 (color-FF00B050) 80                                  
10 (color-FF00B050) 78                                  
# ℹ 1,043 more rows
# ℹ Use `print(n = ...)` to see more rows

If we drop the FF from the color codes (it’s shorthand for 100% opacity), recent versions of R Studio will preview a color right in the editor, so now we know which values are green and which are black.

“FF00B050” is green and “FF000000” is black

Let’s separate color code and weight value into their own columns:

samp_format <- samp_format %>%
  separate(`colour_weight
Black in pounds
Green in kilograms`, into = c("hex8code", "weight"), sep = " ") %>%
  mutate(weight = parse_number(weight))

> samp_format %>% select(id, hex8code, weight)
# A tibble: 1,053 × 3
   id                      hex8code         weight
   <chr>                   <chr>             <dbl>
 1 (color-FF000000) ID0001 (color-FF00B050)     92
 2 (color-FF000000) ID0002 (color-FF000000)    194
 3 (color-FF000000) ID0003 (color-FF00B050)     61
 4 (color-FF000000) ID0004 (color-FF000000)    165
 5 (color-FF000000) ID0005 (color-FF000000)    148
 6 (color-FF000000) ID0006 (color-FF00B050)     86
 7 (color-FF000000) ID0007 (color-FF00B050)     84
 8 (color-FF000000) ID0008 (color-FF000000)    182
 9 (color-FF000000) ID0009 (color-FF00B050)     80
10 (color-FF000000) ID0010 (color-FF00B050)     78
# ℹ 1,043 more rows
# ℹ Use `print(n = ...)` to see more rows

Now we can conditionally convert pounds to kilograms, then remove the color annotation from the id variable, which we’ll use to merge this object with the dataframe we were working on initially.

samp_format <-
  samp_format %>%
  mutate(weight = round(if_else(str_detect(hex8code, "00B050"),
    weight, weight * 0.453
  ), 0)) %>%
  mutate(id = str_remove_all(id, "^[^\\s]+ ")) %>% # regex!
  select(id, weight)

> samp_format
# A tibble: 1,053 × 2
   id     weight
   <chr>   <dbl>
 1 ID0001     92
 2 ID0002     88
 3 ID0003     61
 4 ID0004     75
 5 ID0005     67
 6 ID0006     86
 7 ID0007     84
 8 ID0008     82
 9 ID0009     80
10 ID0010     78
# ℹ 1,043 more rows
# ℹ Use `print(n = ...)` to see more rows

After joining the two objects, some minor cleanup of the names leaves us with nice and usable data.

sampjnd <- left_join(samp, samp_format) %>% clean_names()
sampjnd <-
  sampjnd %>% select(id,
    date = date_fixed, weight_kg = weight, text_integer_issue, text_numeric_issue, numeric_integer_issue,
    one_or_zero_issue
  )
names(sampjnd) <- str_remove(names(sampjnd), "_issue$")

> sampjnd
# A tibble: 1,053 × 7
   id     date                weight_kg text_integer text_numeric numeric_integer
   <chr>  <dttm>                  <dbl>        <dbl>        <dbl>           <dbl>
 1 ID0001 2017-11-11 00:00:00        92           74         1.6                1
 2 ID0002 2017-10-18 00:00:00        88           54         0.14              55
 3 ID0003 2017-10-09 00:00:00        61           53         0.96               9
 4 ID0004 NA                         75           64         0.02               2
 5 ID0005 1971-01-08 00:00:00        67           48         0.23               3
 6 ID0006 1985-11-29 00:00:00        86           33         0.01               7
 7 ID0007 1955-02-12 00:00:00        84           35         0.26               1
 8 ID0008 1982-05-01 00:00:00        82           53         0.07               3
 9 ID0009 1969-04-20 00:00:00        80          187         0.06              75
10 ID0010 1962-11-21 00:00:00        78          141         0.01              23
# ℹ 1,043 more rows
# ℹ 1 more variable: one_or_zero <dbl>
# ℹ Use `print(n = ...)` to see more rows

If it’s not too late, following good practices for Data organization in spreadsheets will avoid a lot of pain. Otherwise, tools like the ones shown here can be useful.

All feedback welcome and again thanks to Jeremy Selva for the idea and GitHub issue that sparked all of this.

Further reading:

• Jenny Bryan’s Spreadhseet Resources
• Please don’t do this: Three common bad practices in sharing tables and spreadsheets and how to avoid them

In R, we can get the data from Word files and PDFs into data frames thanks to packages such as docxtractr, PDFtools, tablulizer, and others (or even AI tools apparently), often preserving the original lines and structure which we can then wrangle into something usable.

This post shows a brief walkthrough for cleaning this type of table, based on my experience with real documents and showcasing a new helper function added to the unheadr package for version 0.4.0 (now on CRAN).

This example is based on publicly available data with information on income, property and property obligations for government officials in the Chukotka Autonomous Okrug, which is the easternmost federal subject of Russia.

Various examples can be downloaded here:

The files look like this, some are available as Word documents and some as PDFs.

For this example, a translated and simplified version looks like this:

To replicate this example, the docx file is here, and the code below uses docxtractr to pull the contents of the table into R.

# necessary packages
library(docxtractr) # CRAN v0.6.5
library(dplyr)      # CRAN v1.1.4
library(stringr)    # CRAN v1.5.1
library(tidyr)      # CRAN v1.3.1
library(unheadr)    # CRAN v0.4.0

# path to downloaded Word file
docpath <- "New Sample/squishtest.docx"
sdoc <- read_docx(docpath)
dtable <- docx_extract_tbl(sdoc, header = TRUE, preserve = TRUE)

To skip the download we could also recreate the output from docx_extract_tbl() stage with this code:

dtable <- data.frame(
  stringsAsFactors = FALSE,
  Name = c("Valery V.", "Mavlyanovna I."),
  Asset = c("Apartment\n\nPlot\nHouse\nOther\nApartment",
            "House\nPlot\n\nWarehouse\n\n"),
  Area = c("45.7\n\n69.2\n65.0\n32.2\n36.9\n\n\n",
           "218.3\n669.1\n\n457.0"),
  Status = c("Owned\n\nOwned\nIn use\nUnknown\nOwned\n",
             "In use\nOwned\n\nLeased")
)

The output is ungainly, and everything that appears on separate lines within the borders of the ‘merged’ cells ends up together, separated only by line break sequences (\n).

> dtable
# A tibble: 2 × 4
  Name           Asset                                        Area       Status
  <chr>          <chr>                                        <chr>      <chr> 
1 Valery V.      "Apartment\n\nPlot\nHouse\nOther\nApartment" "45.7\n\n… "Owne…
2 Mavlyanovna I. "House\nPlot\n\nWarehouse\n\n"               "218.3\n6… "In u…

As a first step we can use the squish_newlines() function from unheadr to deduplicate and remove trailing line breaks, and because the rows in the initial table correspond to two different government officials, the operation is grouped using the .by argument.

dtable %>%
  mutate(across(c(Asset:Status),squish_newlines),.by=Name) 

# A tibble: 2 × 4
  Name           Asset                                      Area         Status
  <chr>          <chr>                                      <chr>        <chr> 
1 Valery V.      "Apartment\nPlot\nHouse\nOther\nApartment" "45.7\n69.2… "Owne…
2 Mavlyanovna I. "House\nPlot\nWarehouse"                   "218.3\n669… "In u…

Now we can separate the cell values into their own rows using separate_rows() from tidyr.

dtable %>%
  mutate(across(c(Asset:Status),squish_newlines),.by=Name) %>% 
  tidyr::separate_rows(c(Asset:Status),sep="\n")

# A tibble: 8 × 4
  Name           Asset     Area  Status 
  <chr>          <chr>     <chr> <chr>  
1 Valery V.      Apartment 45.7  Owned  
2 Valery V.      Plot      69.2  Owned  
3 Valery V.      House     65.0  In use 
4 Valery V.      Other     32.2  Unknown
5 Valery V.      Apartment 36.9  Owned  
6 Mavlyanovna I. House     218.3 In use 
7 Mavlyanovna I. Plot      669.1 Owned  
8 Mavlyanovna I. Warehouse 457.0 Leased 

Much nicer and ready for any number of operations. I hope this is helpful and as usual let me know if you run into any issues.