Skip to contents

Step-by-step instructions

Source: vignettes/workflow.Rmd
workflow.Rmd

This vignette walks through each step of the data preparation process, from raw ROXAS / ROXAS AI output files to a submission-ready TRIA dataset.

Step 1: Locate ROXAS files

Set the path to a folder containing the ROXAS or ROXAS AI output files of your QWA dataset. Files may be organised in subdirectories. get_roxas_files() checks that all required files are present for each image and returns a data frame of their full file paths.

For ROXAS, the required files per image are:

  • Cells output ({IMAGEID}_Output_Cells.txt)
  • Rings output ({IMAGEID}_Output_Rings.txt)
  • Settings file ({IMAGEID}_ROXAS_Settings.txt)
  • Image file ({IMAGEID}.jpg or .jpeg)

For ROXAS AI, the required files per image are:

  • Cells table ({IMAGEID}.cells_table.csv)
  • Rings table ({IMAGEID}.rings_table.csv)
  • Metadata file ({IMAGEID}.metadata.json)
  • Image file ({IMAGEID}.jpg or .jpeg)
roxas_version <- "roxas" # or "roxas_ai"
path_in <- "path/to/ROXAS_data"
files <- get_roxas_files(path_in, roxas_version)

Step 2: Extract the data structure

QWA data are organised hierarchically: each analysed image was taken of a microsection/slide, each slide was cut from a woodpiece (e.g. a core or a wedge), and each woodpiece was sampled from a tree of a certain species at a specific site. Typically, some or all of this information is encoded in the image file names. Fonti et al. (2025) recommend the naming convention:

{site}_{species}_{tree/woodpiece}_{slide}_{image}

e.g. S22_LADE_L08b_1_2.jpg is the second image of the first slide from woodpiece b of the Larix decidua tree L08 at site S22.

extract_data_structure() extracts this hierarchy using a regular expression with named groups. Each named group (site, species, tree, woodpiece, slide, image) that appears in the pattern is extracted and used to construct the full hierarchical labels. Not all groups need to be in the pattern: If, for example, all images belong to the same site and/or species, the site_label and/or species_code arguments can be supplied instead of including site/species groups in the pattern. Or if each tree was cored exactly once, there may not be a specific woodpiece identifier, and it can be left out of the pattern.

Adapt the pattern to match your naming convention using named regex groups. Supported group names: site, species, tree, woodpiece, slide, image. Example below assumes the common convention {site}_{species}_{tree}_{slide}_{image} with alphanumeric components and one woodpiece per tree (no separate woodpiece identifier). If your naming includes a woodpiece identifier, you can add it (e.g. by including “(?[[:alnum:]]+)” between the tree and slide groups). See the documentation of extract_data_structure() for more information.

pattern <- "(?<site>[[:alnum:]]+)_(?<species>[[:alnum:]]+)_(?<tree>[[:alnum:]]+)_(?<slide>[[:alnum:]]+)_(?<image>[[:alnum:]]+)"
df_structure <- extract_data_structure(files, pattern)

# or:
# pattern <- "(?<tree>[[:alnum:]]+)_(?<slide>[[:alnum:]]+)_(?<image>[[:alnum:]]+)"
# df_structure <- extract_data_structure(
#   files, pattern, site_label = "S22", species_code = "LADE")

Always inspect the result with View(df_structure) before proceeding to confirm that all labels were extracted correctly and the hierarchy is consistent.

Step 3: Collect metadata

From the ROXAS or ROXAS AI files

collect_settings_data() reads the per-image ROXAS settings (or ROXAS AI metadata) and, for classic ROXAS, also extracts EXIF data from the image files. The result is combined with the data structure from Step 2 into a QWAimages object using build_QWAimages().

Passing the data structure directly, the ROXAS version is auto-detected from the settings file names, and for classic ROXAS the image EXIF data is read from the fname_image column.

df_settings <- collect_settings_data(df_structure)

Alternatively, the settings (and, for classic ROXAS, image) file paths can be supplied individually, and the version can be set explicitly:

df_settings <- collect_settings_data(files_settings = df_structure$fname_settings,
                                     files_images = df_structure$fname_image,
                                     roxas_version = "roxas")

Datetime columns are read as raw character strings to avoid locale- and timezone-dependent conversion errors. You need to convert them explicitly to POSIXct with the format(s) and timezone appropriate for your data. Make sure to address any conversion warnings before overwriting the columns in df_settings.

# for ROXAS only (ROXAS AI has standardised timestamps, so already converted)
if (roxas_version == "roxas") {
  settings_date_orders <- c("%d.%m.%Y %H:%M:%S", "%d/%m/%Y %H:%M") # adjust to your locale
  meas_created_at_converted <- lubridate::parse_date_time(
    df_settings$meas_created_at,
    orders = settings_date_orders,
    tz = Sys.timezone()
  )
  df_settings$meas_created_at <- meas_created_at_converted
}

Combine the extracted data into one object:

rxs_images <- build_QWAimages(df_structure, df_settings)
rm(df_structure, df_settings)

rxs_images  # prints a summary of the image-level metadata

rxs_images is a QWAimages object: a tibble subclass holding the per-image metadata (file paths, image EXIF data, and ROXAS settings), with the roxas_version stored as an attribute. It can be saved to a compressed CSV for later use:

write_QWAimages(rxs_images, "path/to/output_data/example_dataset_QWAimages.csv.gz")
# rxs_images <- read_QWAimages("path/to/output_data/example_dataset_QWAimages.csv.gz")

Additional metadata for TRIA submission (Step 3b)

To provide the site-, tree-, woodpiece-, slide-, and dataset-level metadata required for a TRIA submission, use the interactive metadata Shiny app. It takes a QWAimages object as its starting point and guides you through filling in the required fields. The output is a full QWAmetadata object, which is exported as a .json file. Partially completed submissions can be reloaded and continued at any time.

launch_metadata_app()

# After exporting and closing the app, optionally read the result back:
# QWA_meta <- read_QWAmetadata("path/to/output_data/example_dataset_QWAmetadata.json")

See vignette("metadata-app") for a full walkthrough of the app.

Step 4: Read and clean the measurement data

collect_raw_data() reads the cell and ring measurement tables from all ROXAS output files, combines them, and performs initial cleaning: negative values and error codes are replaced with NA, the dating is checked for gaps or future years, and a warning is issued if CWT estimates appear to be missing (relevant for conifer data).

complete_QWAdata() then adds derived cell and ring measures (lumen area, CWT-based density, earlywood/latewood widths, etc.) and computes ring quality flags:

  • incomplete_ring: ring at the inner or outer image border where the true ring boundary is not visible; MRW is unreliable or unavailable.
  • missing_ring: no cells detected; ring was added manually during crossdating (typically a wedging ring).
  • duplicate_ring: the same year appears in more than one image due to overlapping slides from the same woodpiece.
  • exclude_dupl: TRUE for the duplicate ring(s) not selected as the preferred source (ranked by cell count, penalising incomplete and missing rings).
  • exclude_issues: convenience flag combining incomplete and/or missing rings, controlled by exclude_mode.

The exclude_mode argument determines which ring issues are pre-selected for exclusion: "either" (default) marks both incomplete and missing rings; "incomplete_only" marks only incomplete rings (useful when missing/wedging rings should still contribute to chronologies).

QWA_data <- collect_raw_data(rxs_images)
# or if you already loaded the full metadata from json after completing Step 3:
# QWA_data <- collect_raw_data(QWA_meta$images)

exclude_mode <- "either"
QWA_data <- complete_QWAdata(QWA_data, rxs_images, exclude_mode)

QWA_data  # prints an overview including ring flag counts

Step 5: Save the QWA data

write_QWAdata() saves the QWAdata object to two (optionally compressed) CSV files following the naming convention {dataset_name}_QWAcells.csv(.gz) and {dataset_name}_QWArings.csv(.gz). Explicit file names can also be provided (see ?write_QWAdata).

path_out <- "path/to/output_data"
dataset_name <- "example_dataset"
write_QWAdata(QWA_data, dir = path_out, dataset_name = dataset_name)

# QWA_data <- read_QWAdata(dir = path_out, dataset_name = dataset_name)

At this stage you have the minimal components required for a TRIA submission: the QWAmetadata .json file (from Step 3b) and the two QWAdata .csv files. The following steps are optional but recommended for a more complete submission.

Step 6: Assess the measurements data quality (optional)

Compute radial profiles (Step 6a)

Cell measurements can be aggregated across the ring width into position bins, producing a QWAprofile object. Two binning schemes are available:

  • Sector profiles: divide each ring into n bins of equal relative width (0–100% of ring width). Comparable across rings of different absolute widths.
  • Band profiles: sliding window of fixed absolute width and step size (in µm). Meaningful when absolute radial position matters (e.g. density gradients in µm from the cambium).

Both support computing means and optional quantiles per bin:

prf_sector <- calculate_sector_profiles(QWA_data,
                                        n_sectors = 5,
                                        sel_cell_params = c("la", "cwttan"),
                                        quant_probs = c(0.1, 0.5, 0.9))

# prf_band <- calculate_band_profiles(QWA_data,
#                                     bandwidth = 50, stepsize = 25,
#                                     sel_cell_params = c("la", "cwttan"),
#                                     quant_probs = c(0.1, 0.5, 0.9))

# Save and reload:
# write_QWAprofile(prf_sector, "path/to/output_data/example_dataset_QWAprofile_sector5.csv.gz")
# prf_sector <- read_QWAprofile("path/to/output_data/example_dataset_QWAprofile_sector5.csv.gz")

Visualise measurement series and flag rings (Step 6b)

The flags Shiny app provides an interactive overview of the ring measurements and allows you to annotate individual rings with data quality flags (e.g. compression wood, cracks) and discrete feature flags (e.g. blue rings, frost rings).

The app takes as input (loaded from file or from the current R environment):

  • The $rings component of a QWAdata object (required).
  • A sector QWAprofile (optional) — plotted alongside the ring time series to aid visual inspection.
  • A QWAimages object (optional) — used to display image-level comments and to allow opening image files directly from the app.

The output is the QWAdata$rings data frame with additional logical columns for each flag. It can be saved to file and read back to update the QWAdata object:

launch_flags_app()

# To update QWA_data$rings with the edited output:
# QWA_data$rings <- read_QWAdata(
#   file_rings = "path/to/output_data/example_dataset_edited_QWArings.csv")$rings
# QWA_data$rings <- df_rings_edited  # or directly from a data frame in the env
# write_QWAdata(QWA_data, dir = path_out, dataset_name = dataset_name)

See vignette("flags-app") for a full walkthrough of the app.

Update re-analysed images (Step 6c)

If data processing reveals issues in the raw data and you re-analyse one or more images with ROXAS or ROXAS AI, update_QWAimages() and update_QWAdata() allow you to splice in the new data without reprocessing the entire dataset.

  1. Replace the output files in path_in with the newly generated files.
  2. Update the settings/metadata for those images in the QWAimages object:
rxs_images <- update_QWAimages(rxs_images,
  imgs_to_update = c("IMAGELABEL1", "IMAGELABEL2"),
  settings_date_orders = settings_date_orders) # same as defined in Step 3
# don't forget to update the component in the QWAmetadata object if necessary:
QWA_meta$images <- rxs_images
  1. Update the QWAdata object. For the updated images only, the raw data files are re-read, derived measures are recomputed, and automatic flags are reset. By default, any manual edits made in the flags app are also reset (set reset_manual_flags = FALSE to preserve them):
QWA_data <- update_QWAdata(QWA_data,
  imgs_to_update = c("IMAGELABEL1", "IMAGELABEL2"),
  meta = rxs_images,
  exclude_mode = exclude_mode) # same as defined in Step 4

Step 7: Compile supplementary resources (optional)

A TRIA submission must include at minimum the QWAmetadata .json and the QWAdata .csv(.gz) files. If you want to provide additional (supplementary) files (original images, annotated images, raw ROXAS / ROXAS AI output, reference series .rwl files, etc.) these should be submitted in a single zip alongside a standalone resources manifest listing all the files in the zip.

Not all files may be eligible as supporting material for a TRIA dataset, and it is important to track which file pertains to which image or slide, etc. For this purpose, compile_resources() scans a directory (including nested subfolders) and returns a table with one row per file it finds. For each file it infers the resource_type from the file name and sets the hierarchy level it belongs to (linked_level). Files are also matched to their linked_label (e.g. the image_label) using a QWAimages object, where possible.

QWA_meta <- read_QWAmetadata("path/to/output_data/example_dataset_QWAmetadata.json")
suppl_res <- compile_resources("path/to/submission_files", rxs_images = QWA_meta$images)

Every file gets a status ("ok", "review", or "ignore") and the console reports whether the supplementary path is ready for submission, or which files still need attention. Review the table, add a description for files where the resource_type could not be inferred, and complete any linked_label values that were not filled automatically. Then write it to a CSV to submit alongside the resources zip:

suppl_res
vroom::vroom_write(suppl_res, "path/to/output_data/example_dataset_supplementary_files.csv")

See vignette("resources") for more details on this step, including how to recompile_resources() after making edits, and how to run a final check_supplementary() once you are done.

Step 8: Submit your data

Once you are ready to share your prepared data files with us, email tria@wsl.ch—either with a link to the files on a file transfer service of your choice, or asking for an invite link to WSL’s own one. See vignette("submission") for what to upload and what happens to your files once we receive them.