15 Check Problem

15.1 Problem Description

This chapter validates consistency between Camera_problem, problem date fields (ProblemX_from, ProblemX_to), and camera installation dates (Start_date, End_date). It also checks whether species records (Record_date) occurred inside any declared camera-problem interval.

15.2 Problem Solving

Pipeline:

Read and prepare Camera_trap data (ct).
Run direct checks (p1 to p6) on Camera_problem and boundary dates.
Compute and validate problem intervals with datapaperchecks::check_problem_intervals() (p7).
Check species records inside problem intervals with datapaperchecks::check_records_in_problem_intervals() (p8).

15.2.1 Read and prepare camera-trap data

Code

ct <- datapaperchecks::read_sheet(
  path = "Example/13",
  recurse = FALSE,
  sheet = "Camera_trap",
  na = c("NA", "na")
) |>
  purrr::map(\(x) {
    x |>
      dplyr::select(
        Structure_id,
        Camera_id,
        Start_date,
        End_date,
        Camera_problem,
        dplyr::starts_with("Problem")
      )
  })

15.2.2 p1 to p4: direct consistency checks

Code

p1 <- ct |>
  purrr::map(\(x) {
    x |>
      dplyr::filter(
        is.na(Camera_problem),
        dplyr::if_any(dplyr::starts_with("Problem"), ~ !is.na(.x))
      )
  }) |>
  purrr::discard(~ nrow(.x) == 0)

print(p1)

named list()

Code

p2 <- ct |>
  purrr::map(\(x) {
    x |>
      dplyr::filter(
        datapaperchecks::camera_problem_is_yes(Camera_problem),
        dplyr::if_any(dplyr::ends_with("date"), ~ is.na(.x))
      )
  }) |>
  purrr::discard(~ nrow(.x) == 0) |>
  dplyr::bind_rows(.id = "dataset") |>
  dplyr::mutate(
    error = "Camera_problem has a problem and one of the date columns is missing",
    .before = dplyr::everything()
  )

print(p2)

# A tibble: 1 × 31
  error   dataset Structure_id Camera_id Start_date          End_date
  <chr>   <chr>   <chr>        <chr>     <dttm>              <dttm>  
1 Camera… Exampl… D658         MX2_022   2018-08-16 00:00:00 NA      
# ℹ 25 more variables: Camera_problem <chr>, Problem1_from <dttm>,
#   Problem1_to <dttm>, Problem2_from <dttm>, Problem2_to <dttm>,
#   Problem3_from <dttm>, Problem3_to <dttm>, Problem4_from <dttm>,
#   Problem4_to <dttm>, Problem5_from <dttm>, Problem5_to <dttm>,
#   Problem6_from <dttm>, Problem6_to <dttm>, Problem7_from <dttm>,
#   Problem7_to <dttm>, Problem8_from <dttm>, Problem8_to <dttm>,
#   Problem9_from <dttm>, Problem9_to <dttm>, Problem10_from <dttm>, …

Code

p3 <- ct |>
  purrr::map(\(x) {
    x |>
      dplyr::filter(
        datapaperchecks::camera_problem_is_yes(Camera_problem),
        dplyr::if_all(dplyr::starts_with("Problem"), ~ is.na(.x))
      )
  }) |>
  purrr::discard(~ nrow(.x) == 0) |>
  dplyr::bind_rows(.id = "dataset") |>
  dplyr::mutate(
    error = "Camera_problem has a problem and all of the Problem columns are blank",
    .before = dplyr::everything()
  )

print(p3)

# A tibble: 13 × 31
   error  dataset Structure_id Camera_id Start_date          End_date           
   <chr>  <chr>   <chr>        <chr>     <dttm>              <dttm>             
 1 Camer… Exampl… D050         MX2_010   2018-09-09 00:00:00 2019-03-04 00:00:00
 2 Camer… Exampl… D079         MX2_009   2018-09-09 00:00:00 2019-03-05 00:00:00
 3 Camer… Exampl… D094         MX2_001   2018-09-09 00:00:00 2019-07-28 00:00:00
 4 Camer… Exampl… D109         MX2_006   2018-09-08 00:00:00 2019-03-05 00:00:00
 5 Camer… Exampl… D611         MX2_013   2018-08-15 00:00:00 2019-01-27 00:00:00
 6 Camer… Exampl… D617         MX2_014   2018-08-15 00:00:00 2019-03-14 00:00:00
 7 Camer… Exampl… D640         MX2_019   2018-08-16 00:00:00 2018-12-27 00:00:00
 8 Camer… Exampl… D648         MX2_018   2018-08-16 00:00:00 2018-12-30 00:00:00
 9 Camer… Exampl… D658         MX2_022   2018-08-16 00:00:00 NA                 
10 Camer… Exampl… D661         MX2_021   2018-08-16 00:00:00 2018-08-25 00:00:00
11 Camer… Exampl… 12_BDCC03    CB04_B    2022-02-24 00:00:00 2022-02-28 00:00:00
12 Camer… Exampl… 21_BSTC08    CB02      2022-02-24 00:00:00 2022-02-28 00:00:00
13 Camer… Exampl… 56_BDCC08    CB06      2022-02-24 00:00:00 2022-02-28 00:00:00
# ℹ 25 more variables: Camera_problem <chr>, Problem1_from <dttm>,
#   Problem1_to <dttm>, Problem2_from <dttm>, Problem2_to <dttm>,
#   Problem3_from <dttm>, Problem3_to <dttm>, Problem4_from <dttm>,
#   Problem4_to <dttm>, Problem5_from <dttm>, Problem5_to <dttm>,
#   Problem6_from <dttm>, Problem6_to <dttm>, Problem7_from <dttm>,
#   Problem7_to <dttm>, Problem8_from <dttm>, Problem8_to <dttm>,
#   Problem9_from <dttm>, Problem9_to <dttm>, Problem10_from <dttm>, …

Code

p4 <- ct |>
  purrr::map(\(x) {
    x |>
      dplyr::filter(
        datapaperchecks::camera_problem_is_no(Camera_problem),
        dplyr::if_any(dplyr::starts_with("Problem"), ~ !is.na(.x))
      )
  }) |>
  purrr::discard(~ nrow(.x) == 0) |>
  dplyr::bind_rows(.id = "dataset") |>
  dplyr::mutate(
    error = "Camera_problem doesn't have a problem, but at least one Problem column must be filled",
    .before = dplyr::everything()
  )

print(p4)

# A tibble: 1 × 31
  error   dataset Structure_id Camera_id Start_date          End_date           
  <chr>   <chr>   <chr>        <chr>     <dttm>              <dttm>             
1 Camera… Exampl… PIF_km102_T1 Cam030N   2023-08-16 00:00:00 2023-10-24 00:00:00
# ℹ 25 more variables: Camera_problem <chr>, Problem1_from <dttm>,
#   Problem1_to <dttm>, Problem2_from <dttm>, Problem2_to <dttm>,
#   Problem3_from <dttm>, Problem3_to <dttm>, Problem4_from <dttm>,
#   Problem4_to <dttm>, Problem5_from <dttm>, Problem5_to <dttm>,
#   Problem6_from <dttm>, Problem6_to <dttm>, Problem7_from <dttm>,
#   Problem7_to <dttm>, Problem8_from <dttm>, Problem8_to <dttm>,
#   Problem9_from <dttm>, Problem9_to <dttm>, Problem10_from <dttm>, …

15.2.3 p5 and p6: boundary-date checks with explicit `ProblemX` labels

p5 and p6 return problem_col and problem_date so each flagged row can be audited against the original ProblemX_from / ProblemX_to columns.

Code

p5 <- ct |>
  purrr::map(\(x) {
    x_indexed <- x |>
      dplyr::mutate(row_id = dplyr::row_number(), .before = 1)

    hits <- x_indexed |>
      tidyr::pivot_longer(
        cols = dplyr::ends_with("_from"),
        names_to = "problem_col",
        values_to = "problem_date"
      ) |>
      dplyr::filter(
        datapaperchecks::camera_problem_is_yes(Camera_problem),
        !is.na(problem_date),
        !is.na(Start_date),
        problem_date == Start_date
      ) |>
      dplyr::mutate(
        error = paste0(problem_col, " has the same date as Start_date"),
        .before = dplyr::everything()
      ) |>
      dplyr::select(row_id, error, problem_col, problem_date)

    hits |>
      dplyr::left_join(
        x_indexed,
        by = "row_id",
        relationship = "many-to-one"
      ) |>
      dplyr::select(-row_id)
  }) |>
  purrr::discard(~ nrow(.x) == 0) |>
  dplyr::bind_rows(.id = "dataset")

print(p5)

# A tibble: 4 × 33
  dataset  error          problem_col problem_date        Structure_id Camera_id
  <chr>    <chr>          <chr>       <dttm>              <chr>        <chr>    
1 Example1 Problem1_from… Problem1_f… 2024-04-04 00:00:00 PIF_km094    Cam009N  
2 Example4 Problem1_from… Problem1_f… 2019-05-24 00:00:00 05_BTCC01    CB03_A   
3 Example4 Problem1_from… Problem1_f… 2021-02-22 00:00:00 29_BDTC01    CB04     
4 Example4 Problem1_from… Problem1_f… 2021-02-22 00:00:00 44_BSCC07    CB01_B   
# ℹ 27 more variables: Start_date <dttm>, End_date <dttm>,
#   Camera_problem <chr>, Problem1_from <dttm>, Problem1_to <dttm>,
#   Problem2_from <dttm>, Problem2_to <dttm>, Problem3_from <dttm>,
#   Problem3_to <dttm>, Problem4_from <dttm>, Problem4_to <dttm>,
#   Problem5_from <dttm>, Problem5_to <dttm>, Problem6_from <dttm>,
#   Problem6_to <dttm>, Problem7_from <dttm>, Problem7_to <dttm>,
#   Problem8_from <dttm>, Problem8_to <dttm>, Problem9_from <dttm>, …

Code

p6 <- ct |>
  purrr::map(\(x) {
    x_indexed <- x |>
      dplyr::mutate(row_id = dplyr::row_number(), .before = 1)

    hits <- x_indexed |>
      tidyr::pivot_longer(
        cols = dplyr::ends_with("_to"),
        names_to = "problem_col",
        values_to = "problem_date"
      ) |>
      dplyr::filter(
        datapaperchecks::camera_problem_is_yes(Camera_problem),
        !is.na(problem_date),
        !is.na(End_date),
        problem_date == End_date
      ) |>
      dplyr::mutate(
        error = paste0(problem_col, " has the same date as End_date"),
        .before = dplyr::everything()
      ) |>
      dplyr::select(row_id, error, problem_col, problem_date)

    hits |>
      dplyr::left_join(
        x_indexed,
        by = "row_id",
        relationship = "many-to-one"
      ) |>
      dplyr::select(-row_id)
  }) |>
  purrr::discard(~ nrow(.x) == 0) |>
  dplyr::bind_rows(.id = "dataset")

print(p6)

# A tibble: 7 × 33
  dataset  error          problem_col problem_date        Structure_id Camera_id
  <chr>    <chr>          <chr>       <dttm>              <chr>        <chr>    
1 Example1 Problem1_to h… Problem1_to 2024-07-23 00:00:00 PIF_km094    Cam009N  
2 Example1 Problem1_to h… Problem1_to 2023-01-18 00:00:00 PIF_km101    Cam013S  
3 Example1 Problem1_to h… Problem1_to 2023-08-30 00:00:00 PIF_km102_T2 Cam022S  
4 Example4 Problem1_to h… Problem1_to 2018-02-26 00:00:00 52_BTTC01    CB01     
5 Example4 Problem1_to h… Problem1_to 2019-05-28 00:00:00 05_BTCC01    CB03_A   
6 Example4 Problem1_to h… Problem1_to 2021-02-26 00:00:00 29_BDTC01    CB04     
7 Example4 Problem1_to h… Problem1_to 2021-02-26 00:00:00 44_BSCC07    CB01_B   
# ℹ 27 more variables: Start_date <dttm>, End_date <dttm>,
#   Camera_problem <chr>, Problem1_from <dttm>, Problem1_to <dttm>,
#   Problem2_from <dttm>, Problem2_to <dttm>, Problem3_from <dttm>,
#   Problem3_to <dttm>, Problem4_from <dttm>, Problem4_to <dttm>,
#   Problem5_from <dttm>, Problem5_to <dttm>, Problem6_from <dttm>,
#   Problem6_to <dttm>, Problem7_from <dttm>, Problem7_to <dttm>,
#   Problem8_from <dttm>, Problem8_to <dttm>, Problem9_from <dttm>, …

15.2.4 p7: interval consistency inside `Camera_trap`

Code

ct_problem_intervals <- datapaperchecks::check_problem_intervals(ct)

p7 <- ct_problem_intervals |>
  purrr::map(\(x) {
    x |>
      dplyr::filter(!is.na(check_problem)) |>
      dplyr::select(
        -c(
          dplyr::ends_with("_to"),
          dplyr::ends_with("_from")
        )
      )
  }) |>
  purrr::discard(~ nrow(.x) == 0) |>
  dplyr::bind_rows(.id = "dataset")

print(p7)

# A tibble: 21 × 16
   dataset  Structure_id Camera_id Start_date          End_date           
   <chr>    <chr>        <chr>     <dttm>              <dttm>             
 1 Example1 PIF_km094    Cam006S   2022-12-15 00:00:00 2023-01-07 00:00:00
 2 Example1 PIF_km094    Cam008N   2022-08-03 00:00:00 2022-08-13 00:00:00
 3 Example1 PIF_km101    Cam012N   2022-10-13 00:00:00 2022-10-16 00:00:00
 4 Example1 PIF_km101    Cam013S   2022-10-13 00:00:00 2023-01-18 00:00:00
 5 Example1 PIF_km101    Cam016S   2023-11-07 00:00:00 2023-12-08 00:00:00
 6 Example1 PIF_km101    Cam018N   2024-04-04 00:00:00 2024-05-01 00:00:00
 7 Example1 PIF_km102_T1 Cam028S   2023-06-21 00:00:00 2023-08-15 00:00:00
 8 Example1 PIF_km102_T1 Cam029N   2023-06-27 00:00:00 2023-08-15 00:00:00
 9 Example1 PIF_km102_T1 Cam031S   2023-08-16 00:00:00 2023-10-06 00:00:00
10 Example1 PIF_km102_T1 Cam032S   2023-11-07 00:00:00 2023-12-10 00:00:00
# ℹ 11 more rows
# ℹ 11 more variables: Camera_problem <chr>, install_period <Interval>,
#   check_problem <chr>, Problem1_interval <Interval>,
#   Problem2_interval <Interval>, Problem3_interval <Interval>,
#   Problem4_interval <Interval>, Problem5_interval <Interval>,
#   Problem6_interval <Interval>, Problem7_interval <Interval>,
#   Problem8_interval <Interval>

15.2.5 p8: species records inside problem intervals

Code

sps <- datapaperchecks::read_sheet(
  path = "Example/13",
  recurse = FALSE,
  sheet = "Species_records_camera",
  na = c("", "NA")
)

p8 <- datapaperchecks::check_records_in_problem_intervals(
  sps_list = sps,
  ct_problem_list = ct_problem_intervals
)

print(p8)

# A tibble: 114 × 17
   dataset  record_row Structure_id Camera_id Species Record_date Camera_problem
   <chr>         <int> <chr>        <chr>     <chr>   <date>      <chr>         
 1 Example1         62 PIF_km102_T1 Cam028S   Cunicu… 2023-08-15  Sim           
 2 Example1         63 PIF_km102_T1 Cam029N   Cunicu… 2023-08-15  Sim           
 3 Example1        105 PIF_km102_T1 Cam031S   Eira b… 2023-10-06  Sim           
 4 Example1        129 PIF_km102_T1 Cam032S   Guerli… 2023-12-10  Sim           
 5 Example1        130 PIF_km102_T1 Cam032S   Didelp… 2023-12-10  Sim           
 6 Example1        173 PIF_km102_T1 Cam035N   Eira b… 2024-05-18  Sim           
 7 Example1        473 PIF_km102_T2 Cam022S   Didelp… 2023-08-18  Sim           
 8 Example1        539 PIF_km102_T2 Cam024S   Dasypu… 2024-01-13  Sim           
 9 Example1        633 PIF_km101    Cam012N   Didelp… 2022-10-16  Sim           
10 Example1        634 PIF_km101    Cam012N   Didelp… 2022-10-16  Sim           
# ℹ 104 more rows
# ℹ 10 more variables: install_period <Interval>, problem_match <chr>,
#   Problem1_interval <Interval>, Problem2_interval <Interval>,
#   Problem3_interval <Interval>, Problem4_interval <Interval>,
#   Problem5_interval <Interval>, Problem6_interval <Interval>,
#   Problem7_interval <Interval>, Problem8_interval <Interval>

15.1 Problem Description

15.2 Problem Solving

15.2.1 Read and prepare camera-trap data

15.2.2 p1 to p4: direct consistency checks

15.2.3 p5 and p6: boundary-date checks with explicit ProblemX labels

15.2.4 p7: interval consistency inside Camera_trap

15.2.5 p8: species records inside problem intervals

15.2.3 p5 and p6: boundary-date checks with explicit `ProblemX` labels

15.2.4 p7: interval consistency inside `Camera_trap`