This notebook corrects the errors detected by 1_quality_check.Rmd, based on the feed back from the people who collected the data. Corrections are done within this script to ensure reproducibility.
The output is a clean kobo file that can be used for analyses.
Load required libraries:
library(tidyr)
library(dplyr)
library(utile.tools)
library(stringr)
library(ggplot2)
library(readr)
library(readxl)Useful custom functions
source("process_attached_files.R")
source("download_kobo_attachment.R")
source("recover_nocoincidence_files.R")
# not in
"%!in%" <- function(x, y)!('%in%'(x,y))Get Kobo raw output data:
kobo_output<-read.csv(file="International_Genetic_Indicator_testing_V_4.0_-_latest_version_-_False_-_2023-11-02-08-23-26.csv", sep=";", header=TRUE) %>%
## add taxon column
mutate(taxon=(utile.tools::paste(genus, species, subspecies_variety, na.rm=TRUE))) %>%
# remove white space at the end of the name
mutate(taxon=str_trim(taxon, "right"))Filter out records which were marked as “not_approved” in the manual Kobo validation interface (this means country assessors determined the is something wrong with that particular record).
# check if any species is flagged as "validation_status_not_approved"
kobo_output %>%
filter(X_validation_status=="validation_status_not_approved")# omit those records from data:
kobo_clean<- kobo_output %>%
filter(X_validation_status!="validation_status_not_approved")Filter out any sort of tests
# select likely columns to say "test"
cols= c("name_assessor", "email_assessor", "genus", "species", "subspecies_variety",
"scientific_authority", "common_name", "GBIF_taxonID", "NCBI_taxonID", "time_populations")
# check for "test" on any of them
kobo_clean %>%
filter(if_any(all_of(cols), ~ grepl("test", .))) %>%
select(country_assessment, name_assessor, genus, species, X_uuid)# filter them out of dataset
kobo_clean<- kobo_clean %>%
filter(if_any(all_of(cols), ~ !grepl("test", .))) %>%
filter(species != "test")In the form, -999 was used to mark taxa with unknown number of extant populations. This was used because answering the question was mandatory, so leaving it blank wasn’t possible. We have to change -999 to NA.
For n extant populations:
kobo_clean<- kobo_clean %>%
mutate(n_extant_populations= na_if(n_extant_populations, -999))For n extinct populations:
kobo_clean<- kobo_clean %>%
mutate(n_extint_populations= na_if(n_extint_populations, -999))Once -999 was replaced by NA there should be no negative number of populations (if they are, they are typos that need to be corrected).
Check for extant populations:
kobo_clean %>%
filter(n_extant_populations<0) %>%
select(country_assessment, taxon, name_assessor, n_extant_populations, n_extint_populations)Check for extinct populations:
kobo_clean %>%
filter(n_extint_populations<0) %>%
select(country_assessment, taxon, name_assessor, n_extant_populations, n_extint_populations)Show which species (if any) have 999 EXTINCT populations. Should this be -999?
Check for extinct populations:
kobo_clean %>%
filter(n_extint_populations==999) %>%
select(country_assessment, taxon, name_assessor, n_extant_populations, n_extint_populations, end)Check for extant populations:
kobo_clean %>%
filter(n_extant_populations==999) %>%
select(country_assessment, taxon, name_assessor, n_extant_populations, n_extint_populations, end)Show which taxa (if any) have 0 (zero) extant populations. Is this correct? needs to be manually checked
kobo_clean %>%
filter(n_extant_populations==0) %>%
select(country_assessment, taxon, name_assessor, n_extant_populations, n_extint_populations, end)According to the assessors feedback the data of the following taxa is correct. For example, there are 0 extant populations because it is regionally extinct or because it was imposible to estimate how many populations it has:
ok_0_extant<-c("Hieracium sandozianum",
"Charadrius alexandrinus",
"Planorbella magnifica",
"Heterelmis stephani")Filtering out those taxa that are correct we should have an empty object, is not, the remaining taxa still need to be verified:
kobo_clean %>%
filter(n_extant_populations==0) %>%
filter(taxon %!in% ok_0_extant) %>%
select(country_assessment, taxon, name_assessor, n_extant_populations, n_extint_populations, end)If there are any remaining taxa with weird number of populations put them together in a single object.
If everything is correct now, the object should be empity.
check_n_pops <- kobo_clean %>%
# variables of interest
select(country_assessment, name_assessor, taxon, n_extant_populations, n_extint_populations, end) %>%
# same filters that discussed above
filter(n_extant_populations<0 |
n_extant_populations==0 |
n_extant_populations==999 |
n_extint_populations==999) %>%
filter(taxon %!in% ok_0_extant) %>%
# add a column stating what needs to be checked:
mutate(need_to_check="check number of extant or extint populations. Are 0 correct? should 999 be -999? are extant/extint confused?")
check_n_popsThe quality check script flagged any records where the GBIF Id is =/= 7, because that is the most common length. Changes after the assessors checked their data:
Corrections requested:
# Gymnobelideus leadbeateri
kobo_clean[kobo_clean$taxon=="Gymnobelideus leadbeateri", "GBIF_taxonID"] ## [1] "220657"kobo_clean[kobo_clean$taxon=="Gymnobelideus leadbeateri", "GBIF_taxonID"] <-2440054
kobo_clean[kobo_clean$taxon=="Gymnobelideus leadbeateri", "GBIF_taxonID"] ## [1] "2440054"# Zingel asper
kobo_clean[kobo_clean$taxon=="Zingel asper", "GBIF_taxonID"] ## [1] "172830100"kobo_clean[kobo_clean$taxon=="Zingel asper", "GBIF_taxonID"] <- 2382117
kobo_clean[kobo_clean$taxon=="Zingel asper", "GBIF_taxonID"] ## [1] "2382117"# Miniopterus schreibersii
kobo_clean[kobo_clean$taxon=="Miniopterus schreibersii", "GBIF_taxonID"] ## [1] "176678740"kobo_clean[kobo_clean$taxon=="Miniopterus schreibersii", "GBIF_taxonID"] <- 9796816
kobo_clean[kobo_clean$taxon=="Miniopterus schreibersii", "GBIF_taxonID"] ## [1] "9796816"# Toxolasma lividum
kobo_clean[kobo_clean$taxon=="Toxolasma lividum", "GBIF_taxonID"] ## [1] "2288024" "2288024"kobo_clean[kobo_clean$taxon=="Toxolasma lividum", "GBIF_taxonID"] <-157572593
kobo_clean[kobo_clean$taxon=="Toxolasma lividum", "GBIF_taxonID"] ## [1] "157572593" "157572593"# Caladenia woolcockiorum
kobo_clean[kobo_clean$taxon=="Caladenia woolcockiorum", "GBIF_taxonID"]## [1] "185788620"kobo_clean[kobo_clean$taxon=="Caladenia woolcockiorum", "GBIF_taxonID"] <- 2841407
kobo_clean[kobo_clean$taxon=="Caladenia woolcockiorum", "GBIF_taxonID"]## [1] "2841407"# Phyllurus kabikabi
kobo_clean[kobo_clean$taxon=="Phyllurus kabikabi", "GBIF_taxonID"]## [1] "185106797"kobo_clean[kobo_clean$taxon=="Phyllurus kabikabi", "GBIF_taxonID"] <- 5843345
kobo_clean[kobo_clean$taxon=="Phyllurus kabikabi", "GBIF_taxonID"]## [1] "5843345"# Ambuchanania leuchbryoides
kobo_clean[kobo_clean$taxon=="Ambuchanania leuchbryoides", "GBIF_taxonID"]## [1] "5792195; 8052578"kobo_clean[kobo_clean$taxon=="Ambuchanania leuchbryoides", "GBIF_taxonID"] <- 5792195
kobo_clean[kobo_clean$taxon=="Ambuchanania leuchbryoides", "GBIF_taxonID"]## [1] "5792195"# Phonotimpus talquian
kobo_clean[kobo_clean$taxon=="Phonotimpus talquian", "GBIF_taxonID"]## [1] "9720487" "157318142"kobo_clean[kobo_clean$taxon=="Phonotimpus talquian", "GBIF_taxonID"] <- 9720487
kobo_clean[kobo_clean$taxon=="Phonotimpus talquian", "GBIF_taxonID"]## [1] "9720487" "9720487"# Ecklonia radiata brevipes
kobo_clean[kobo_clean$taxon=="Ecklonia radiata brevipes", "GBIF_taxonID"]## [1] "177596818"kobo_clean[kobo_clean$taxon=="Ecklonia radiata brevipes", "GBIF_taxonID"] <- 3196095
kobo_clean[kobo_clean$taxon=="Ecklonia radiata brevipes", "GBIF_taxonID"]## [1] "3196095"Correct GBIF ids even if they look weird:
ok_GBIF<-c("Caladenia woolcockiorum",
"Leucopatus anophthalmus",
"Phyllurus kabikabi",
"Phonotimpus talquian, Hypotaenidia sylvestris",
"Chersobius signatus",
"Hypotaenidia sylvestris",
"Megascops gilesi",
"Dubusia carrikeri")
## add the manually corrected too, in case they are != 7
ok_GBIF<-c(ok_GBIF,
"Gymnobelideus leadbeateri",
"Zingel asper",
"Miniopterus schreibersii",
"Toxolasma lividum",
"Caladenia woolcockiorum",
"Phyllurus kabikabi",
"Ambuchanania leuchbryoides",
"Phonotimpus talquian",
"Anaxyrus williamsi")If there are any remaining taxa with weird GBIF ids put them together in a single object.
check_GBIF <- kobo_clean %>%
filter(nchar(GBIF_taxonID)>0, nchar(GBIF_taxonID)!=7) %>%
filter(taxon %!in% ok_GBIF) %>%
# show only relevant columns
select(country_assessment, name_assessor, taxon, GBIF_taxonID, end) %>%
# add a column stating what needs to be checked:
mutate(need_to_check="check the GBIF taxonID. Either it looks plain different, or has more or less than 7 digits (most ids are 7 digits long, and this isn't, it could be an exception, or a mistake).")
check_GBIFGenus, species and subspecies should be a single word, check if there are cases where it isn’t. Only exception would be “var.” or “subsp.” in the subspecies_variety field:
kobo_clean %>%
filter(grepl(" ", genus) |
grepl(" ", species) |
grepl(" ", subspecies_variety)) %>%
filter(!grepl("var.", subspecies_variety)) %>%
filter(!grepl("subsp.", subspecies_variety)) %>%
# show only relevant columns
# show only relevant columns
select(country_assessment, name_assessor, taxon, genus, species, subspecies_variety, end)Corrections requested:
# Gypaetus Gypaetus barbatus
kobo_clean[kobo_clean$taxon=="Gypaetus Gypaetus barbatus", "species"]## character(0)kobo_clean[kobo_clean$taxon=="Gypaetus Gypaetus barbatus", "species"] <-"barbatus"
kobo_clean[kobo_clean$taxon=="Gypaetus Gypaetus barbatus", "taxon"] <-"Gypaetus barbatus"
kobo_clean[kobo_clean$taxon=="Gypaetus barbatus", c("species", "taxon")] # Gyps Gyps coprotheres
kobo_clean[kobo_clean$taxon=="Gyps Gyps coprotheres", "species"]## character(0)kobo_clean[kobo_clean$taxon=="Gyps Gyps coprotheres", "species"] <-"coprotheres"
kobo_clean[kobo_clean$taxon=="Gypaetus Gypaetus barbatus", "taxon"] <-"Gypaetus barbatus"
kobo_clean[kobo_clean$taxon=="Gypaetus barbatus", c("species", "taxon")] # Hippocampus Hippocampus capensis
kobo_clean[kobo_clean$taxon=="Hippocampus Hippocampus capensis", "species"]## character(0)kobo_clean[kobo_clean$taxon=="Hippocampus Hippocampus capensis", "species"] <-"capensis"
kobo_clean[kobo_clean$taxon=="Hippocampus Hippocampus capensis", "taxon"] <-"Hippocampus capensis"
kobo_clean[kobo_clean$taxon=="Hippocampus capensis", c("species", "taxon")] # Poicephalus Poicephalus robustus
kobo_clean[kobo_clean$taxon=="Poicephalus Poicephalus robustus", "species"]## character(0)kobo_clean[kobo_clean$taxon=="Poicephalus Poicephalus robustus", "species"] <-"robustus"
kobo_clean[kobo_clean$taxon=="Poicephalus Poicephalus robustus", "taxon"] <-"Poicephalus robustus"
kobo_clean[kobo_clean$taxon=="Poicephalus robustus", c("species", "taxon")] # Campylorhynchus rufinucha humilis, rufinucha, nigricaudatus, xerophilus, nicaraguae, castaneus, capistratus
kobo_clean[kobo_clean$taxon=="Campylorhynchus rufinucha humilis, rufinucha, nigricaudatus, xerophilus, nicaraguae, castaneus, capistratus", "subspecies_variety"]## character(0)kobo_clean[kobo_clean$taxon=="Campylorhynchus rufinucha humilis, rufinucha, nigricaudatus, xerophilus, nicaraguae, castaneus, capistratus", "subspecies_variety"] <-""
kobo_clean[kobo_clean$taxon=="Campylorhynchus rufinucha humilis, rufinucha, nigricaudatus, xerophilus, nicaraguae, castaneus, capistratus", "taxon"] <- "Campylorhynchus rufinucha"
kobo_clean[kobo_clean$taxon=="Campylorhynchus rufinucha", c("subspecies_variety", "species", "taxon")] # Icterus Icterus gularis gularis, tamaulipensis, yucatanensis, flavescens, troglodytes, gigas
kobo_clean[kobo_clean$taxon=="Icterus Icterus gularis gularis, tamaulipensis, yucatanensis, flavescens, troglodytes, gigas", "subspecies_variety"]## character(0)kobo_clean[kobo_clean$taxon=="Icterus Icterus gularis gularis, tamaulipensis, yucatanensis, flavescens, troglodytes, gigas", "subspecies_variety"] <-""
kobo_clean[kobo_clean$taxon=="Icterus Icterus gularis gularis, tamaulipensis, yucatanensis, flavescens, troglodytes, gigas", "species"]## character(0)kobo_clean[kobo_clean$taxon=="Icterus Icterus gularis gularis, tamaulipensis, yucatanensis, flavescens, troglodytes, gigas", "species"] <-"gularis"
kobo_clean[kobo_clean$taxon=="Icterus Icterus gularis gularis, tamaulipensis, yucatanensis, flavescens, troglodytes, gigas", "taxon"] <- "Icterus gularis"
kobo_clean[kobo_clean$taxon=="Icterus gularis", c("subspecies_variety", "species", "taxon")] # Melanerpes santacruzi grateloupensis, dubius, santacruzi, hughlandi, leei, turneffensis, pauper, insulanus, canescens
kobo_clean[kobo_clean$taxon=="Melanerpes santacruzi grateloupensis, dubius, santacruzi, hughlandi, leei, turneffensis, pauper, insulanus, canescens", "subspecies_variety"]## character(0)kobo_clean[kobo_clean$taxon=="Melanerpes santacruzi grateloupensis, dubius, santacruzi, hughlandi, leei, turneffensis, pauper, insulanus, canescens", "subspecies_variety"] <-""
kobo_clean[kobo_clean$taxon=="Melanerpes santacruzi grateloupensis, dubius, santacruzi, hughlandi, leei, turneffensis, pauper, insulanus, canescens", "taxon"] <- "Melanerpes santacruzi"
kobo_clean[kobo_clean$taxon=="Melanerpes santacruzi", c("subspecies_variety", "species", "taxon")] # Saltator atriceps atriceps, suffuscus, flavicrissus, peeti, raptor, lacertosus
kobo_clean[kobo_clean$taxon=="Saltator atriceps atriceps, suffuscus, flavicrissus, peeti, raptor, lacertosus", "subspecies_variety"]## character(0)kobo_clean[kobo_clean$taxon=="Saltator atriceps atriceps, suffuscus, flavicrissus, peeti, raptor, lacertosus", "subspecies_variety"] <-""
kobo_clean[kobo_clean$taxon=="Saltator atriceps atriceps, suffuscus, flavicrissus, peeti, raptor, lacertosus", "taxon"] <- "Saltator atriceps"
kobo_clean[kobo_clean$taxon=="Saltator atriceps", c("subspecies_variety", "species", "taxon")] # Ambuchanania leucobryoides / Ambuchanania leuchbryoides
# Should be Ambuchanania leucobryoides ('co' not 'ch' in species name)
kobo_clean[kobo_clean$taxon=="Ambuchanania leuchbryoides", "species"]## [1] "leuchbryoides"kobo_clean[kobo_clean$taxon=="Ambuchanania leuchbryoides", "species"] <-"leucobryoides"
kobo_clean[kobo_clean$taxon=="Ambuchanania leuchbryoides", "taxon"] <-"Ambuchanania leucobryoides"
kobo_clean[kobo_clean$taxon=="Ambuchanania leucobryoides", c("species", "taxon")] Correct:
ok_taxon_name<-c("Coregonus albula morphotype trybomi",
"Glyptemys muhlenbergii Northern Population")If there are any remaining taxa with weird taxon names put them together in a single object.
check_taxon_names <- kobo_clean %>%
filter(grepl(" ", genus) |
grepl(" ", species) |
grepl(" ", subspecies_variety)) %>%
filter(!grepl("var.", subspecies_variety)) %>%
filter(!grepl("subsp.", subspecies_variety)) %>%
filter(taxon %!in% ok_taxon_name) %>%
# show only relevant columns
select(country_assessment, name_assessor, taxon, genus, species, subspecies_variety, end) %>%
mutate(need_to_check="check genus, species or subspecies_variety, we are targeting to have single words in each field, except in the ifraspecific names, where 'var.' and 'subsp.' (only) would be accepted. Other details or taxonomic notes should be added in the comments.")
check_taxon_namesIf there are still taxa that need to be checked for whatever reason, write a file enlisting them
to_check<-full_join(check_n_pops, check_GBIF) %>% full_join(check_taxon_names) %>%
# show columns in desired order:
select(country_assessment, name_assessor, taxon, need_to_check, n_extant_populations,
n_extint_populations, GBIF_taxonID, genus, species, subspecies_variety, end)## Joining, by = c("country_assessment", "name_assessor", "taxon", "end",
## "need_to_check")
## Joining, by = c("country_assessment", "name_assessor", "taxon", "end",
## "need_to_check")to_check# save file with a list of taxa if there are remaining issues, or with a message saying that all is done if not):
if(nrow(to_check)>0) {
write.csv(to_check, "kobo_output_tocheck_remaining_issues.csv", row.names = FALSE, fileEncoding = "UTF-8")
} else {
happy_message<-"congratulations! there are no more taxa with issues to be corrected. :)"
write.csv(happy_message, "kobo_output_tocheck_remaining_issues.csv", row.names = FALSE, fileEncoding = "UTF-8")
}The method “management units” to define populations was interpreted differently by countries. For example, Belgium called “management units” a method in which they created a buffer polygons based on reported dispersal distances for each species around observation points, which are used as occurrence points. Polygons that overlap are fused together, and the resulting polygon, along with the points falling within its area, are considered a single population. However, for other countries “management units” refers to managed populations in the wild, sometimes enclosed, for instance as part of hunting schemes. Similarly, Japan used occurrence data in about 10-km grids and known dispersal distances to define populations as one or more grids, but called this a combination of management_units and geographic_boundaries.
Since we would be evaluating the effect of method on the indicators, we decided to leave management units into real management units and to create a new category, called “dispersal buffers”, which would better reflect what countries did.
This is also reflected in the new versions of the Kobo form, where “dispersal buffer” has been added as an option.
Re-label data for Belgium:
# select management_units that need to be changed to "dispersal_buffer"
# look for "managegment_units" in belgium and change that text to "dispersal_buffer" keeping any_text before or after
kobo_clean$defined_populations <- ifelse(kobo_clean$country_assessment == "belgium",
gsub("management_units", "dispersal_buffer", kobo_clean$defined_populations),
kobo_clean$defined_populations)
# Check resutls look as expected:
grep("dispersal_buffer", levels(as.factor(kobo_clean$defined_populations)), value=TRUE)## [1] "dispersal_buffer"
## [2] "dispersal_buffer other"
## [3] "eco_biogeo_proxies dispersal_buffer"
## [4] "genetic_clusters dispersal_buffer"
## [5] "geographic_boundaries dispersal_buffer"Re-label data for Japan. After manually reviweing the data and discussing it with Fumiko, we believe that all taxa fit better in the new category “dispersal_buffer”, except for the following taxa, which we thin would be better labeled as geographic_boundaries:
# geographic_boundaries species:
x<-c("Cryptocarya chinensis", "Polystichum piceopaleaceum", "Oxytropis campestris subsp. rishiriensi", "Sporobolus hancei")
# change selected species to geographic_boundaries
kobo_clean<-kobo_clean %>% mutate(defined_populations = ifelse(country_assessment == "japan" &
kobo_clean$taxon %in% x, # in the list then change to geo
"geographic_boundaries", defined_populations)) %>%
# change the rest of japan species to "dispersal_buffer"
mutate(defined_populations = ifelse(country_assessment == "japan" &
kobo_clean$taxon %!in% x, # if NOT in the list then change to dispersal_buff
"dispersal_buffer", defined_populations))
# check data looks as expected
unique(kobo_clean[kobo_clean$country_assessment == "japan", "defined_populations"])## [1] "dispersal_buffer" "geographic_boundaries"After checking the data with Viktoria she found that “geographic_boundaries” should be changed to “dispersal_buffer” for the species listed below:
# species to change geo boundaries to dispersal
x<-c("Bufotes viridis", "Pelophylax esculentus", "Pelobates fuscus", "Triturus cristatus", "Bombina bombina", "Bufo bufo", "Epidalea calamita", "Hyla arborea", "Lissotriton vulgaris", "Rana temporaria", "Rana arvalis", "Rana dalmatina", "Lepus timidus", "Erinaceus europaeus", "Muscardinus avellanarius", "Lacerta agilis", "Coronella austriaca", "Natrix natrix natrix", "Zootoca vivipara")
# change selected species to dispersal_buffr
kobo_clean$defined_populations <- ifelse(kobo_clean$country_assessment == "sweden" & kobo_clean$taxon %in% x,
gsub("geographic_boundaries", "dispersal_buffer", kobo_clean$defined_populations),
kobo_clean$defined_populations)
# check data looks as expected
filter(kobo_clean, country_assessment == "sweden" & taxon %in% x) %>% select(country_assessment, defined_populations, taxon)Remove from the clean version any remaining taxa with issues
kobo_clean<-kobo_clean %>%
filter(taxon %!in% to_check$taxon)Export clean version
write.csv(kobo_clean, "kobo_output_clean.csv", row.names = FALSE, fileEncoding = "UTF-8")If information of more than 25 populations was be used to collect data for Ne >500 indicator (Section 5 of the Kobo form), it is possible to use a template to upload data instead of using the kobo form. This section of the cleaning script processes those files to create a single clean object as expected to estimate indicator 1 (Ne >500).
Create directories to save files
## original files
original_files_dir="processed_files/original_files"
## Create target directory overwriting previous content
# Remove existing target directory if it exists
if (dir.exists(original_files_dir)) {
# Remove all files and subdirectories within the target directory
files_to_remove <- list.files(original_files_dir, full.names = TRUE, recursive = TRUE)
file.remove(files_to_remove)
# Remove the target directory itself
unlink(original_files_dir, recursive = TRUE)
}
# Recreate the target directory
dir.create(original_files_dir, recursive = TRUE, showWarnings = FALSE)
## processed (and likely corrected) files:
processed_files_dir="processed_files/processed_files"
## Create target directory overwriting previous content
# Remove existing target directory if it exists
if (dir.exists(processed_files_dir)) {
# Remove all files and subdirectories within the target directory
files_to_remove <- list.files(processed_files_dir, full.names = TRUE, recursive = TRUE)
file.remove(files_to_remove)
# Remove the target directory itself
unlink(processed_files_dir, recursive = TRUE)
}
# Recreate the target directory
dir.create(processed_files_dir, recursive = TRUE, showWarnings = FALSE)The url to the attachment file is avaialbe in the kobo_output under
the variable pop_tabular_file_URL. This url could be used
to download each of the attachments. For this, it is necessary to
provide a kobo username with reading permissions and its password. To
avoid showing passwords and usernames within this script, you have to
create a credentials.txt file and save it in the working
directory of this script. The credentials.txt should have
ONLY two rows and one column (no headers). The first row should be the
kobo user name and the second your password. For instance in the
following example myusername is the username and
12334 is the password:
myusername
12334The following code uses the credentials and the url stored in the kobo_output, for each of the records were population data was recorded using the tabular template and for which there is a url avaiable:
# subset data
tabular_records<-kobo_clean %>%
filter(kobo_tabular=="tabular" & pop_tabular_file_URL!="") %>%
select(country_assessment, taxon, kobo_tabular, X_uuid, pop_tabular_file, pop_tabular_file_URL, X__version__,
end, n_extant_populations)
# credentials
credentials<-read.delim("credentials.txt", header = FALSE)
for(i in 1:nrow(tabular_records)){
# log species
print(paste("Downloading data from record number", i, tabular_records$country_assessment[i], tabular_records$taxon[i], "file",
tabular_records$pop_tabular_file[i]))
# build file name with, country , record xuuid and original file name separated by __ (to parse easyly)
new_file_name <- paste(tabular_records$country_assessment[i],
tabular_records$X_uuid[i],
tabular_records$pop_tabular_file[i], sep= "__")
# download data
download_kobo_attachment(kobo_url = tabular_records$pop_tabular_file_URL[i],
local_file_path = file.path(original_files_dir, new_file_name),
username = credentials[1,1],
password = credentials[2,1])
}## [1] "Downloading data from record number 1 australia Notechis scutatus file AustraliaNotechisscutatusGlennShea-14_6_53-10_6_36.txt"
## [1] "Downloading data from record number 2 japan Illigera luzonensis file Illigera_luzonensis_Japan_Fumiko-Ishihama-15_42_40.txt"
## [1] "Downloading data from record number 3 united_states Picoides borealis file Dryobates_borealis_Funk-13_51_34.txt"
## [1] "Downloading data from record number 4 united_states Chrysopsis floridana file united_states__1b1a22d3_corrected_ChrysopsisfloridanaSheelaTurbek-11_17_13.txt"
## [1] "Downloading data from record number 5 united_states Toxolasma lividum file populations_data_Toxolasma lividum_AK_20230622-10_53_41.xlsx"
## [1] "Downloading data from record number 6 united_states Glyptemys muhlenbergii file population_data_bog-turtle-16_19_3.txt"
## [1] "Downloading data from record number 7 united_states Graptopetalum bartramii file bartramstonecrop-CMupdated-15_37_2.txt"
## [1] "Downloading data from record number 8 united_states Oncorhynchus apache file populations_data_apache_trout-14_21_31-11_4_39.csv"
## [1] "Downloading data from record number 9 united_states Sideroxylon thornei file united_states__51d1eb2e_corrected_USGeorgiaBully-12_33_4.txt"
## [1] "Downloading data from record number 10 united_states Pseudemys rubriventris file populations_data_template_43AU-12_53_26-22_33_23.txt"
## [1] "Downloading data from record number 11 japan Swertia pseudochinensis file Swertia_pseudochinensis_Japan_Fumiko-Ishihama-15_39_8.txt"
## [1] "Downloading data from record number 12 japan Primula sieboldii file Primula_sieboldii_Japan_Fumiko-Ishihama-15_41_22-9_42_24.txt"
## [1] "Downloading data from record number 13 australia Litoria booroolongensis file AustraliaLitoriabooroolongensisCatherineGrueber-21_33_0.txt"
## [1] "Downloading data from record number 14 united_states Rhynchospora crinipes file united_states__ba88021c_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_30_57.txt"
## [1] "Downloading data from record number 15 sweden Triturus cristatus file swedentrituruscristatusviktoriaköppä-9_43_19.txt"
## [1] "Downloading data from record number 16 sweden Epidalea calamita file swedenepidaleacalamitaviktoriaköppä-9_45_0.txt"
## [1] "Downloading data from record number 17 france Calotriton asper file Calotriton-asper_France_Iris-Lang-18_9_18.txt"
## [1] "Downloading data from record number 18 united_states Rhynchospora crinipes file united_states_ac33a1e9_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_17_6.txt"
## [1] "Downloading data from record number 19 united_states Alasmidonta varicosa file ALVA_AU_NcTable-18_59_11.txt"
## [1] "Downloading data from record number 20 united_states Vireo atricapilla file Vireo atricapilla-13_17_54.txt"
## [1] "Downloading data from record number 21 united_states Oncorhynchus clarkii virginalis file united_states__c1dbabb5_corrected_USRioGrandeCutthroatTrout-12_20_14.txt"
## [1] "Downloading data from record number 22 united_states Castilleja levisecta file population_data_golden-paintbrush-2018-CMupdated-15_1_30.txt"
## [1] "Downloading data from record number 23 sweden Salmo salar file swedensalmosalarviktoriakoppa-14_33_9.txt"
## [1] "Downloading data from record number 24 belgium Dactylorhiza sphagnicola file BelgiumDactylorhizasphagnicolaLuisCastillo-22_56_45.txt"
## [1] "Downloading data from record number 25 belgium Mentha pulegium file Belgium_Mentha_pulegium_Luis_Castillo_populations_data-18_49_49.txt"
## [1] "Downloading data from record number 26 belgium Scorzonera humilis file Belgium_Scorzonera_humilis_Luis_Castillo_populations_data-16_5_41.txt"
## [1] "Downloading data from record number 27 belgium Wahlenbergia hederacea file Belgium_Wahlenbergia_hederacea_Luis_Castillo_populations_data-17_22_8.txt"
## [1] "Downloading data from record number 28 belgium Lucanus cervus file Belgium_Lucanus_cervus_Luis_Castillo_populations_data-11_33_19.txt"
## [1] "Downloading data from record number 29 united_states Toxolasma lividum file MUs_data_Toxolasma lividum_AK_20230622-11_28_13.xlsx"
## [1] "Downloading data from record number 30 belgium Pelophylax lessonae file Belgium_Pelophylax_lessonae_populations_data-10_7_51.txt"
## [1] "Downloading data from record number 31 belgium Triturus cristatus file Belgium_Triturus_cristatus_Luis_Castillo-18_13_29.txt"
## [1] "Downloading data from record number 32 belgium Salamandra salamandra file Belgium_Salamandra_salamandra_Luis_Castillo-18_14_45.txt"
## [1] "Downloading data from record number 33 belgium Rana arvalis file Belgium_Rana_arvalis_Luis_Castillo-11_36_39.txt"
## [1] "Downloading data from record number 34 belgium Coronella austriaca file Belgium_Coronella_austriaca_Luis_Castillo-21_47_41.txt"
## [1] "Downloading data from record number 35 france Capra ibex file francecapraibexlang-1_2_20.txt"
## [1] "Downloading data from record number 36 belgium Natrix helvetica file Belgium_Natrix_helvetica_Luis_Castillo-7_36_54.txt"
## [1] "Downloading data from record number 37 belgium Rhodeus sericeus file Belgium_Rhodeus_sericeus_Luis_Castillo-16_47_18.txt"
## [1] "Downloading data from record number 38 belgium Alytes obstetricans file Belgium_Alytes_obstetricans_Luis_Castillo-23_33_8.txt"Check which files were downloaded:
result_files<-list.files(original_files_dir)
result_files## [1] "australia__3201ddb0-7017-4420-b560-4c34d1759767__AustraliaLitoriabooroolongensisCatherineGrueber-21_33_0.txt"
## [2] "australia__a09fe312-d061-4e78-a53e-4f5f99d3533a__AustraliaNotechisscutatusGlennShea-14_6_53-10_6_36.txt"
## [3] "belgium__126bc473-9f32-4038-85d6-4172db492c94__Belgium_Scorzonera_humilis_Luis_Castillo_populations_data-16_5_41.txt"
## [4] "belgium__3c9892f2-b001-4c73-9974-a9bbe1b1b6e6__Belgium_Mentha_pulegium_Luis_Castillo_populations_data-18_49_49.txt"
## [5] "belgium__521e2abd-3ab3-45f8-a85e-c6f192ec47fa__Belgium_Lucanus_cervus_Luis_Castillo_populations_data-11_33_19.txt"
## [6] "belgium__543af6c3-03d8-4389-9516-cedd021080c0__Belgium_Coronella_austriaca_Luis_Castillo-21_47_41.txt"
## [7] "belgium__74a0fcc3-ae70-454f-b4bc-79db1ab01350__BelgiumDactylorhizasphagnicolaLuisCastillo-22_56_45.txt"
## [8] "belgium__7e6dae7a-2c2c-4cec-a42a-346f2237a841__Belgium_Salamandra_salamandra_Luis_Castillo-18_14_45.txt"
## [9] "belgium__867bdbf5-8e36-4acd-a968-221882d2f544__Belgium_Natrix_helvetica_Luis_Castillo-7_36_54.txt"
## [10] "belgium__afb12d15-1b70-472a-8ab4-5781425474d6__Belgium_Pelophylax_lessonae_populations_data-10_7_51.txt"
## [11] "belgium__b317bac1-03ab-4ba9-9b81-897f3e454d46__Belgium_Rana_arvalis_Luis_Castillo-11_36_39.txt"
## [12] "belgium__b9147bcf-bb30-42d9-b4bd-8dc6182537e9__Belgium_Triturus_cristatus_Luis_Castillo-18_13_29.txt"
## [13] "belgium__cf6d6047-5fd5-4028-808c-0749d3ab0512__Belgium_Rhodeus_sericeus_Luis_Castillo-16_47_18.txt"
## [14] "belgium__fd17c864-c7e2-42ad-8a35-ab389ba152e2__Belgium_Wahlenbergia_hederacea_Luis_Castillo_populations_data-17_22_8.txt"
## [15] "belgium__ff9fc0ed-87d0-4f70-8083-d92dade3fe62__Belgium_Alytes_obstetricans_Luis_Castillo-23_33_8.txt"
## [16] "france__27c6fc64-0ff3-4cc1-a120-e03933157b2b__Calotriton-asper_France_Iris-Lang-18_9_18.txt"
## [17] "france__674a87f8-9f29-43d6-b951-a1fecb5e590d__francecapraibexlang-1_2_20.txt"
## [18] "japan__1483ad83-a129-4254-981c-8ccb1bebab65__Swertia_pseudochinensis_Japan_Fumiko-Ishihama-15_39_8.txt"
## [19] "japan__1f209615-ad4a-4daf-bac1-61478136712d__Primula_sieboldii_Japan_Fumiko-Ishihama-15_41_22-9_42_24.txt"
## [20] "japan__ea550e1b-0a63-4f73-9785-8c019353b693__Illigera_luzonensis_Japan_Fumiko-Ishihama-15_42_40.txt"
## [21] "sweden__19f96c82-3cd0-4830-9875-de996b24145e__swedenepidaleacalamitaviktoriaköppä-9_45_0.txt"
## [22] "sweden__50f06905-b89d-4976-a73d-23cd0bf93d46__swedentrituruscristatusviktoriaköppä-9_43_19.txt"
## [23] "sweden__7f82d9c4-814e-40b0-a01e-4744fa370458__swedensalmosalarviktoriakoppa-14_33_9.txt"
## [24] "united_states__018d6a54-b069-4bdb-845c-85249bdf2bbb__bartramstonecrop-CMupdated-15_37_2.txt"
## [25] "united_states__2836fb5e-21c9-456e-9df8-7ae0d2656fc3__ALVA_AU_NcTable-18_59_11.txt"
## [26] "united_states__3d40e09e-4829-45cf-8023-7ed2c30eec6b__population_data_golden-paintbrush-2018-CMupdated-15_1_30.txt"
## [27] "united_states__3fcb2789-0a46-4952-aa37-efc89879494e__united_states__c1dbabb5_corrected_USRioGrandeCutthroatTrout-12_20_14.txt"
## [28] "united_states__49f4a7e1-ca75-4507-bbe5-e4a9657b3089__united_states__1b1a22d3_corrected_ChrysopsisfloridanaSheelaTurbek-11_17_13.txt"
## [29] "united_states__524becc3-ceff-4404-9931-48d60ebee340__populations_data_apache_trout-14_21_31-11_4_39.csv"
## [30] "united_states__5544e7a8-c247-4aa1-b0d2-114aeadb145f__united_states__51d1eb2e_corrected_USGeorgiaBully-12_33_4.txt"
## [31] "united_states__7822d752-697e-4c97-a06d-4ad90b629f72__populations_data_template_43AU-12_53_26-22_33_23.txt"
## [32] "united_states__88bf3eb4-ece3-4114-b8c8-40af37fd26eb__population_data_bog-turtle-16_19_3.txt"
## [33] "united_states__91bf443a-426d-45af-a5f6-8e368063c31e__Vireo atricapilla-13_17_54.txt"
## [34] "united_states__a3b00727-817c-4225-b6c7-f35c973a085e__united_states_ac33a1e9_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_17_6.txt"
## [35] "united_states__ae031359-4550-4c31-bd6f-f51c667579c4__MUs_data_Toxolasma lividum_AK_20230622-11_28_13.xlsx"
## [36] "united_states__d939ac91-8d2d-46f1-a0e2-555523fdadb4__Dryobates_borealis_Funk-13_51_34.txt"
## [37] "united_states__d98333f6-1cb4-47ab-9236-afa8681f91ec__united_states__ba88021c_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_30_57.txt"
## [38] "united_states__e7a67f42-cc28-47cc-be82-be9cbae72e5f__populations_data_Toxolasma lividum_AK_20230622-10_53_41.xlsx"If there were files stored as .xlsx, we need to transform those to txt files before proceeding.
# get excel files, if any
excels<- grep(".xlsx", result_files, value = TRUE)
excels## [1] "united_states__ae031359-4550-4c31-bd6f-f51c667579c4__MUs_data_Toxolasma lividum_AK_20230622-11_28_13.xlsx"
## [2] "united_states__e7a67f42-cc28-47cc-be82-be9cbae72e5f__populations_data_Toxolasma lividum_AK_20230622-10_53_41.xlsx"if(length(excels)>0){
# If excel files exist, read and save each one with same but as txt file
for(i in 1:length(excels)){
# Excel file path
file_path <- file.path(original_files_dir, excels[i])
# Read the first sheet of the Excel file as a text file
data <- read_excel(file_path, sheet = 1, col_types = "text")
# Save with same file name, changing the extension
write.table(data,
file = file.path(original_files_dir, paste0(tools::file_path_sans_ext(excels[i]), ".txt")),
sep = "\t", quote = FALSE, row.names = FALSE)
}
}Result files should only be txt or csv files
result_files<-grep("\\.(txt|csv)$", list.files(original_files_dir), value = TRUE)
result_files## [1] "australia__3201ddb0-7017-4420-b560-4c34d1759767__AustraliaLitoriabooroolongensisCatherineGrueber-21_33_0.txt"
## [2] "australia__a09fe312-d061-4e78-a53e-4f5f99d3533a__AustraliaNotechisscutatusGlennShea-14_6_53-10_6_36.txt"
## [3] "belgium__126bc473-9f32-4038-85d6-4172db492c94__Belgium_Scorzonera_humilis_Luis_Castillo_populations_data-16_5_41.txt"
## [4] "belgium__3c9892f2-b001-4c73-9974-a9bbe1b1b6e6__Belgium_Mentha_pulegium_Luis_Castillo_populations_data-18_49_49.txt"
## [5] "belgium__521e2abd-3ab3-45f8-a85e-c6f192ec47fa__Belgium_Lucanus_cervus_Luis_Castillo_populations_data-11_33_19.txt"
## [6] "belgium__543af6c3-03d8-4389-9516-cedd021080c0__Belgium_Coronella_austriaca_Luis_Castillo-21_47_41.txt"
## [7] "belgium__74a0fcc3-ae70-454f-b4bc-79db1ab01350__BelgiumDactylorhizasphagnicolaLuisCastillo-22_56_45.txt"
## [8] "belgium__7e6dae7a-2c2c-4cec-a42a-346f2237a841__Belgium_Salamandra_salamandra_Luis_Castillo-18_14_45.txt"
## [9] "belgium__867bdbf5-8e36-4acd-a968-221882d2f544__Belgium_Natrix_helvetica_Luis_Castillo-7_36_54.txt"
## [10] "belgium__afb12d15-1b70-472a-8ab4-5781425474d6__Belgium_Pelophylax_lessonae_populations_data-10_7_51.txt"
## [11] "belgium__b317bac1-03ab-4ba9-9b81-897f3e454d46__Belgium_Rana_arvalis_Luis_Castillo-11_36_39.txt"
## [12] "belgium__b9147bcf-bb30-42d9-b4bd-8dc6182537e9__Belgium_Triturus_cristatus_Luis_Castillo-18_13_29.txt"
## [13] "belgium__cf6d6047-5fd5-4028-808c-0749d3ab0512__Belgium_Rhodeus_sericeus_Luis_Castillo-16_47_18.txt"
## [14] "belgium__fd17c864-c7e2-42ad-8a35-ab389ba152e2__Belgium_Wahlenbergia_hederacea_Luis_Castillo_populations_data-17_22_8.txt"
## [15] "belgium__ff9fc0ed-87d0-4f70-8083-d92dade3fe62__Belgium_Alytes_obstetricans_Luis_Castillo-23_33_8.txt"
## [16] "france__27c6fc64-0ff3-4cc1-a120-e03933157b2b__Calotriton-asper_France_Iris-Lang-18_9_18.txt"
## [17] "france__674a87f8-9f29-43d6-b951-a1fecb5e590d__francecapraibexlang-1_2_20.txt"
## [18] "japan__1483ad83-a129-4254-981c-8ccb1bebab65__Swertia_pseudochinensis_Japan_Fumiko-Ishihama-15_39_8.txt"
## [19] "japan__1f209615-ad4a-4daf-bac1-61478136712d__Primula_sieboldii_Japan_Fumiko-Ishihama-15_41_22-9_42_24.txt"
## [20] "japan__ea550e1b-0a63-4f73-9785-8c019353b693__Illigera_luzonensis_Japan_Fumiko-Ishihama-15_42_40.txt"
## [21] "sweden__19f96c82-3cd0-4830-9875-de996b24145e__swedenepidaleacalamitaviktoriaköppä-9_45_0.txt"
## [22] "sweden__50f06905-b89d-4976-a73d-23cd0bf93d46__swedentrituruscristatusviktoriaköppä-9_43_19.txt"
## [23] "sweden__7f82d9c4-814e-40b0-a01e-4744fa370458__swedensalmosalarviktoriakoppa-14_33_9.txt"
## [24] "united_states__018d6a54-b069-4bdb-845c-85249bdf2bbb__bartramstonecrop-CMupdated-15_37_2.txt"
## [25] "united_states__2836fb5e-21c9-456e-9df8-7ae0d2656fc3__ALVA_AU_NcTable-18_59_11.txt"
## [26] "united_states__3d40e09e-4829-45cf-8023-7ed2c30eec6b__population_data_golden-paintbrush-2018-CMupdated-15_1_30.txt"
## [27] "united_states__3fcb2789-0a46-4952-aa37-efc89879494e__united_states__c1dbabb5_corrected_USRioGrandeCutthroatTrout-12_20_14.txt"
## [28] "united_states__49f4a7e1-ca75-4507-bbe5-e4a9657b3089__united_states__1b1a22d3_corrected_ChrysopsisfloridanaSheelaTurbek-11_17_13.txt"
## [29] "united_states__524becc3-ceff-4404-9931-48d60ebee340__populations_data_apache_trout-14_21_31-11_4_39.csv"
## [30] "united_states__5544e7a8-c247-4aa1-b0d2-114aeadb145f__united_states__51d1eb2e_corrected_USGeorgiaBully-12_33_4.txt"
## [31] "united_states__7822d752-697e-4c97-a06d-4ad90b629f72__populations_data_template_43AU-12_53_26-22_33_23.txt"
## [32] "united_states__88bf3eb4-ece3-4114-b8c8-40af37fd26eb__population_data_bog-turtle-16_19_3.txt"
## [33] "united_states__91bf443a-426d-45af-a5f6-8e368063c31e__Vireo atricapilla-13_17_54.txt"
## [34] "united_states__a3b00727-817c-4225-b6c7-f35c973a085e__united_states_ac33a1e9_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_17_6.txt"
## [35] "united_states__ae031359-4550-4c31-bd6f-f51c667579c4__MUs_data_Toxolasma lividum_AK_20230622-11_28_13.txt"
## [36] "united_states__d939ac91-8d2d-46f1-a0e2-555523fdadb4__Dryobates_borealis_Funk-13_51_34.txt"
## [37] "united_states__d98333f6-1cb4-47ab-9236-afa8681f91ec__united_states__ba88021c_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_30_57.txt"
## [38] "united_states__e7a67f42-cc28-47cc-be82-be9cbae72e5f__populations_data_Toxolasma lividum_AK_20230622-10_53_41.txt"How many attached files we have (sucessfully downloaded and transformed frome excel, if needed)?
length(result_files)## [1] 38How many files were expected based on the kobo_output answers?
expected<-kobo_clean %>%
filter(taxon %!in% to_check$taxon) %>%
filter(kobo_tabular=="tabular") %>%
select(country_assessment, name_assessor, taxon, X_uuid, map_populations_URL, end)
nrow(expected)## [1] 38From which country(ies)?
table(expected$country_assessment)##
## australia belgium france japan sweden
## 2 13 2 3 3
## united_states
## 15Which are the missing files?
missing<-expected[expected$X_uuid %!in% str_match(result_files, "(?<=__).*?(?=__)"),] # get the Xuud ie the characters after the first "__" and the second "__".
missingThe function process_attached_files()
processes the original attachment files to make them compatible with how
population data is expected to estimate the Ne >500 indicator. You
can check the details in the function code.
Importantly, it checks the columns that should be numbers, but they may have been stored as characters because “,” “-” “(” or other characters may have been typed. The function will try to correct this, which may change the original data. Please manually check any file flagged as transformed (log below) to ensure everything is correct.
These are the transformations that the function does and you should check:
1) In numeric variables (Nc, Ne and their bounds): check if
there are (), and remove them keeping only the value
outside ().
Example: 86 (95% CI) will be transformed to:
86
2) In numeric variables (Nc, Ne and their bounds): convert
, to '' (nothing) IF more than 3 digits
followed the , (ie we assumed , is separating
thousands), OR convert , to . IF 2 digits
followed the , (ie we assumed , is separating
decimal points)
Example: “3,00” “5000,000” “666” will be transformed to: “3.00” “5000000” “666”
3) In numeric variables (Nc, Ne and their bounds): use as.numeric().
Example: NcPoint expects ONLY numbers, an no other character. Therefore 400-600 stored in NcPoint will be transformed to NA. And there would be a warning saying “NAs introduced by coercion”. In this example, if you want to keep “400-600” then you should have filled the column NcRange with in this case less_5000_by_much.
4) NcType and NcRange should only have values from the specified categories, any other value will be changed to NA
Example: * NcRange values could only be ‘less_5000’, ‘less_5000_bymuch’, ‘more_5000’, ‘more_5000_bymuch’, ‘range_includes_5000’. If the assessor types “600-8000” this will be transformed to NA.
Other less serious edits, but that you should also know about, are the following:
NcType = Nc_point for all popsNcType = Nc_range for all popsAlso, remember that excel files (if any), were transformed to .txt, so below we will only process the .txt and .csv files (see transformation above).
Please check the logs below and manually check the original and the processed files to ensure everything is correct.
# files separated by "\t": australia, france, japan, sweden, most US, some belgium
df<-data.frame()
# numbers reflect indexes of the files with the desired delim in result_files[i]
for(i in c(1:2, # australia
5,11, # belgium
16:17, # France
18:20, # Japan
21:23, # sweden
24:28, 30:38) # US
){
file<-paste0(original_files_dir,"/", result_files[i])
cat(paste("## processing element:", i, "\n",
" file:", file, " \n", " \n"))
temp<-process_attached_files(file_path=file, kobo_output=kobo_clean, delim = '\t')
# save processed file
processed_file_path<-paste0(processed_files_dir,"/", "processed__", result_files[i])
write.csv(temp, processed_file_path, row.names = FALSE)
# attach object to df with all results
df<-rbind(df, temp)
cat(" \n") # white space before next text message
}## ## processing element: 1
## file: processed_files/original_files/australia__3201ddb0-7017-4420-b560-4c34d1759767__AustraliaLitoriabooroolongensisCatherineGrueber-21_33_0.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"## Note: Using an external vector in selections is ambiguous.
## ℹ Use `all_of(desired_order)` instead of `desired_order` to silence this message.
## ℹ See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.
## This message is displayed once per session.##
## ## processing element: 2
## file: processed_files/original_files/australia__a09fe312-d061-4e78-a53e-4f5f99d3533a__AustraliaNotechisscutatusGlennShea-14_6_53-10_6_36.txt
##
## [1] "all requiered population columns are present in the file :)"
## varible NeLower is stored as character and should be numeric, so this function will:
##
## 1) check if there are (), for instance `86 (95% CI)`, and remove them keeping only the value outside ().
##
## 2) convert ',' to '' IF more than 3 digits followed the ',' (ie we assumed ',' is separating thousands), OR /n
## convert ',' to '.' IF 2 digits followed the ',' (ie we assumed ',' is separating decimal points) ; and
##
## 3) use as.numeric().
##
## You should check the original data to make sure the transformation was correctNULL
## varible NeUpper is stored as character and should be numeric, so this function will:
##
## 1) check if there are (), for instance `86 (95% CI)`, and remove them keeping only the value outside ().
##
## 2) convert ',' to '' IF more than 3 digits followed the ',' (ie we assumed ',' is separating thousands), OR /n
## convert ',' to '.' IF 2 digits followed the ',' (ie we assumed ',' is separating decimal points) ; and
##
## 3) use as.numeric().
##
## You should check the original data to make sure the transformation was correctNULL
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 5
## file: processed_files/original_files/belgium__521e2abd-3ab3-45f8-a85e-c6f192ec47fa__Belgium_Lucanus_cervus_Luis_Castillo_populations_data-11_33_19.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 11
## file: processed_files/original_files/belgium__b317bac1-03ab-4ba9-9b81-897f3e454d46__Belgium_Rana_arvalis_Luis_Castillo-11_36_39.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 16
## file: processed_files/original_files/france__27c6fc64-0ff3-4cc1-a120-e03933157b2b__Calotriton-asper_France_Iris-Lang-18_9_18.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 17
## file: processed_files/original_files/france__674a87f8-9f29-43d6-b951-a1fecb5e590d__francecapraibexlang-1_2_20.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 18
## file: processed_files/original_files/japan__1483ad83-a129-4254-981c-8ccb1bebab65__Swertia_pseudochinensis_Japan_Fumiko-Ishihama-15_39_8.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 19
## file: processed_files/original_files/japan__1f209615-ad4a-4daf-bac1-61478136712d__Primula_sieboldii_Japan_Fumiko-Ishihama-15_41_22-9_42_24.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 20
## file: processed_files/original_files/japan__ea550e1b-0a63-4f73-9785-8c019353b693__Illigera_luzonensis_Japan_Fumiko-Ishihama-15_42_40.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 21
## file: processed_files/original_files/sweden__19f96c82-3cd0-4830-9875-de996b24145e__swedenepidaleacalamitaviktoriaköppä-9_45_0.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 22
## file: processed_files/original_files/sweden__50f06905-b89d-4976-a73d-23cd0bf93d46__swedentrituruscristatusviktoriaköppä-9_43_19.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 23
## file: processed_files/original_files/sweden__7f82d9c4-814e-40b0-a01e-4744fa370458__swedensalmosalarviktoriakoppa-14_33_9.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 24
## file: processed_files/original_files/united_states__018d6a54-b069-4bdb-845c-85249bdf2bbb__bartramstonecrop-CMupdated-15_37_2.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "NcPoint data was provided so NcType should exist but was not provided, setting NcType = `Nc_point` for relevant pops"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 25
## file: processed_files/original_files/united_states__2836fb5e-21c9-456e-9df8-7ae0d2656fc3__ALVA_AU_NcTable-18_59_11.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "NcType values could only be 'Nc_point', 'Nc_range'. Other values were found and were changed to NA"
##
## ## processing element: 26
## file: processed_files/original_files/united_states__3d40e09e-4829-45cf-8023-7ed2c30eec6b__population_data_golden-paintbrush-2018-CMupdated-15_1_30.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 27
## file: processed_files/original_files/united_states__3fcb2789-0a46-4952-aa37-efc89879494e__united_states__c1dbabb5_corrected_USRioGrandeCutthroatTrout-12_20_14.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 28
## file: processed_files/original_files/united_states__49f4a7e1-ca75-4507-bbe5-e4a9657b3089__united_states__1b1a22d3_corrected_ChrysopsisfloridanaSheelaTurbek-11_17_13.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 30
## file: processed_files/original_files/united_states__5544e7a8-c247-4aa1-b0d2-114aeadb145f__united_states__51d1eb2e_corrected_USGeorgiaBully-12_33_4.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 31
## file: processed_files/original_files/united_states__7822d752-697e-4c97-a06d-4ad90b629f72__populations_data_template_43AU-12_53_26-22_33_23.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 32
## file: processed_files/original_files/united_states__88bf3eb4-ece3-4114-b8c8-40af37fd26eb__population_data_bog-turtle-16_19_3.txt
##
## [1] "all requiered population columns are present in the file :)"
## varible NcLower is stored as character and should be numeric, so this function will:
##
## 1) check if there are (), for instance `86 (95% CI)`, and remove them keeping only the value outside ().
##
## 2) convert ',' to '' IF more than 3 digits followed the ',' (ie we assumed ',' is separating thousands), OR /n
## convert ',' to '.' IF 2 digits followed the ',' (ie we assumed ',' is separating decimal points) ; and
##
## 3) use as.numeric().
##
## You should check the original data to make sure the transformation was correctNULL## Warning in process_attached_files(file_path = file, kobo_output = kobo_clean, :
## NAs introduced by coercion##
## ## processing element: 33
## file: processed_files/original_files/united_states__91bf443a-426d-45af-a5f6-8e368063c31e__Vireo atricapilla-13_17_54.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 34
## file: processed_files/original_files/united_states__a3b00727-817c-4225-b6c7-f35c973a085e__united_states_ac33a1e9_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_17_6.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "NcPoint data was provided so NcType should exist but was not provided, setting NcType = `Nc_point` for relevant pops"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 35
## file: processed_files/original_files/united_states__ae031359-4550-4c31-bd6f-f51c667579c4__MUs_data_Toxolasma lividum_AK_20230622-11_28_13.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 36
## file: processed_files/original_files/united_states__d939ac91-8d2d-46f1-a0e2-555523fdadb4__Dryobates_borealis_Funk-13_51_34.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "NcPoint data was provided so NcType should exist but was not provided, setting NcType = `Nc_point` for relevant pops"
##
## ## processing element: 37
## file: processed_files/original_files/united_states__d98333f6-1cb4-47ab-9236-afa8681f91ec__united_states__ba88021c_corrected_unitedstatesrhynchosporacrinipestaylorstack-12_30_57.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "NcPoint data was provided so NcType should exist but was not provided, setting NcType = `Nc_point` for relevant pops"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 38
## file: processed_files/original_files/united_states__e7a67f42-cc28-47cc-be82-be9cbae72e5f__populations_data_Toxolasma lividum_AK_20230622-10_53_41.txt
##
## [1] "all requiered population columns are present in the file :)"
## ## files separated by ";" belgium
for(i in c(3,4,6:10,12:15)){
file<-paste0(original_files_dir,"/", result_files[i])
cat(paste("## processing element:", i, "\n",
" file:", file, " \n", " \n"))
temp<-process_attached_files(file_path=file, kobo_output=kobo_clean, delim = ';')
# save processed file
processed_file_path<-paste0(processed_files_dir,"/", "processed__", result_files[i])
write.csv(temp, processed_file_path, row.names = FALSE)
# attach object to df with all results
df<-rbind(df, temp)
cat(" \n") # white space before next text message
}## ## processing element: 3
## file: processed_files/original_files/belgium__126bc473-9f32-4038-85d6-4172db492c94__Belgium_Scorzonera_humilis_Luis_Castillo_populations_data-16_5_41.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 4
## file: processed_files/original_files/belgium__3c9892f2-b001-4c73-9974-a9bbe1b1b6e6__Belgium_Mentha_pulegium_Luis_Castillo_populations_data-18_49_49.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 6
## file: processed_files/original_files/belgium__543af6c3-03d8-4389-9516-cedd021080c0__Belgium_Coronella_austriaca_Luis_Castillo-21_47_41.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 7
## file: processed_files/original_files/belgium__74a0fcc3-ae70-454f-b4bc-79db1ab01350__BelgiumDactylorhizasphagnicolaLuisCastillo-22_56_45.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 8
## file: processed_files/original_files/belgium__7e6dae7a-2c2c-4cec-a42a-346f2237a841__Belgium_Salamandra_salamandra_Luis_Castillo-18_14_45.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 9
## file: processed_files/original_files/belgium__867bdbf5-8e36-4acd-a968-221882d2f544__Belgium_Natrix_helvetica_Luis_Castillo-7_36_54.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 10
## file: processed_files/original_files/belgium__afb12d15-1b70-472a-8ab4-5781425474d6__Belgium_Pelophylax_lessonae_populations_data-10_7_51.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "NcRange data was provided so NcType should exist but was not provided, setting NcType = `Nc_range` for relevant pops"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 12
## file: processed_files/original_files/belgium__b9147bcf-bb30-42d9-b4bd-8dc6182537e9__Belgium_Triturus_cristatus_Luis_Castillo-18_13_29.txt
##
## [1] "all requiered population columns are present in the file :)"
## [1] "If there is no Nc data, NcMethod and NcType should be NA, this was not the case so NA were introduced to replace the value. Check original data to make sure it is correct"
##
## ## processing element: 13
## file: processed_files/original_files/belgium__cf6d6047-5fd5-4028-808c-0749d3ab0512__Belgium_Rhodeus_sericeus_Luis_Castillo-16_47_18.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 14
## file: processed_files/original_files/belgium__fd17c864-c7e2-42ad-8a35-ab389ba152e2__Belgium_Wahlenbergia_hederacea_Luis_Castillo_populations_data-17_22_8.txt
##
## [1] "all requiered population columns are present in the file :)"
##
## ## processing element: 15
## file: processed_files/original_files/belgium__ff9fc0ed-87d0-4f70-8083-d92dade3fe62__Belgium_Alytes_obstetricans_Luis_Castillo-23_33_8.txt
##
## [1] "all requiered population columns are present in the file :)"
## ## files separated by "," some Us
for(i in c(29)){
file<-paste0(original_files_dir,"/", result_files[i])
cat(paste("## processing element:", i, "\n",
" file:", file, " \n", " \n"))
temp<-process_attached_files(file_path=file, kobo_output=kobo_clean, delim = ',')
# save processed file
processed_file_path<-paste0(processed_files_dir,"/", "processed__", result_files[i])
write.csv(temp, processed_file_path, row.names = FALSE)
# attach object to df with all results
df<-rbind(df, temp)
cat(" \n") # white space before next text message
}## ## processing element: 29
## file: processed_files/original_files/united_states__524becc3-ceff-4404-9931-48d60ebee340__populations_data_apache_trout-14_21_31-11_4_39.csv
##
## [1] "all requiered population columns are present in the file :)"
## varible NcLower is stored as character and should be numeric, so this function will:
##
## 1) check if there are (), for instance `86 (95% CI)`, and remove them keeping only the value outside ().
##
## 2) convert ',' to '' IF more than 3 digits followed the ',' (ie we assumed ',' is separating thousands), OR /n
## convert ',' to '.' IF 2 digits followed the ',' (ie we assumed ',' is separating decimal points) ; and
##
## 3) use as.numeric().
##
## You should check the original data to make sure the transformation was correctNULL## Warning in process_attached_files(file_path = file, kobo_output = kobo_clean, :
## NAs introduced by coercion### all together
ind1_data_from_templates<-dfAre columns looking good?:
# columns look good?
head(ind1_data_from_templates)These variables should be numeric:
# should be numeric
class(ind1_data_from_templates$NcPoint)## [1] "numeric"class(ind1_data_from_templates$NcLower)## [1] "numeric"class(ind1_data_from_templates$NcUpper)## [1] "numeric"class(ind1_data_from_templates$Ne)## [1] "numeric"class(ind1_data_from_templates$NeLower)## [1] "numeric"class(ind1_data_from_templates$NeUpper)## [1] "numeric"Are the expected levels found in NcRange? The only categories should be: “less_5000” “less_5000_bymuch” “more_5000” “more_5000_bymuch” “range_includes_5000”.
# check levels
levels(as.factor(ind1_data_from_templates$NcRange))## [1] "less_5000" "less_5000_bymuch" "more_5000"
## [4] "more_5000_bymuch" "range_includes_5000"Are the expected levels found in NcType? The only categories should be: “Nc_point” “Nc_range”.
# check levels
levels(as.factor(ind1_data_from_templates$NcType))## [1] "Nc_point" "Nc_range"write.csv(ind1_data_from_templates, "ind1_data_from_templates.csv", row.names = FALSE)sessionInfo()## R version 4.2.1 (2022-06-23)
## Platform: x86_64-apple-darwin17.0 (64-bit)
## Running under: macOS Big Sur ... 10.16
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] readxl_1.4.0 readr_2.1.2 ggplot2_3.4.1 stringr_1.4.0
## [5] utile.tools_0.2.7 dplyr_1.0.9 tidyr_1.2.0
##
## loaded via a namespace (and not attached):
## [1] cellranger_1.1.0 pillar_1.7.0 bslib_0.3.1 compiler_4.2.1
## [5] jquerylib_0.1.4 base64enc_0.1-3 tools_4.2.1 bit_4.0.4
## [9] digest_0.6.29 gtable_0.3.0 jsonlite_1.8.0 evaluate_0.15
## [13] lifecycle_1.0.3 tibble_3.1.7 pkgconfig_2.0.3 rlang_1.0.6
## [17] cli_3.6.0 DBI_1.1.3 rstudioapi_0.13 parallel_4.2.1
## [21] yaml_2.3.5 xfun_0.31 fastmap_1.1.0 withr_2.5.0
## [25] knitr_1.39 hms_1.1.1 generics_0.1.3 vctrs_0.5.2
## [29] sass_0.4.1 bit64_4.0.5 grid_4.2.1 tidyselect_1.1.2
## [33] glue_1.6.2 R6_2.5.1 fansi_1.0.3 vroom_1.5.7
## [37] rmarkdown_2.14 tzdb_0.3.0 purrr_0.3.4 magrittr_2.0.3
## [41] scales_1.2.0 ellipsis_0.3.2 htmltools_0.5.5 assertthat_0.2.1
## [45] colorspace_2.0-3 utf8_1.2.2 stringi_1.7.6 munsell_0.5.0
## [49] crayon_1.5.1