header_tag.html

Skip to contents

General Data Import from Water Quality Portal

Source: R/readWQPdata.R
readWQPdata.Rd

Imports data from Water Quality Portal web service. This function gets the data from here: https://www.waterqualitydata.us.

Usage

readWQPdata(
  ...,
  service = "Result",
  querySummary = FALSE,
  tz = "UTC",
  ignore_attributes = FALSE,
  convertType = TRUE
)

Arguments

...

see https://www.waterqualitydata.us/webservices_documentation for a complete list of options. A list of arguments can also be supplied. For more information see the above description for this help file. One way to figure out how to construct a WQP query is to go to the "Advanced" form in the Water Quality Portal. Use the form to discover what parameters are available. Once the query is set in the form, scroll down to the "Query URL". You will see the parameters after "https://www.waterqualitydata.us/#". For example, if you chose "Nutrient" in the Characteristic Group dropdown, you will see characteristicType=Nutrient in the Query URL. The corresponding argument for dataRetrieval is characteristicType = "Nutrient". dataRetrieval users do not need to include mimeType, and providers is optional (these arguments are picked automatically).

service

character. See Details for more information.

querySummary

logical to only return the number of records and unique sites that will be returned from this query. Choosing TRUE is deprecated, readWQPsummary is recommended instead.

tz

character to set timezone attribute of dateTime. Default is "UTC", and converts the date times to UTC, properly accounting for daylight savings times based on the data's provided tz_cd column. Possible values to provide are "America/New_York","America/Chicago", "America/Denver","America/Los_Angeles", "America/Anchorage", as well as the following which do not use daylight savings time: "America/Honolulu", "America/Jamaica","America/Managua","America/Phoenix", and "America/Metlakatla". See also OlsonNames() for more information on time zones.

ignore_attributes

logical to choose to ignore fetching site and status attributes. Default is FALSE.

convertType

logical, defaults to TRUE. If TRUE, the function will convert the data to dates, datetimes, numerics based on a standard algorithm. If false, everything is returned as a character.

Value

A data frame, the specific columns will depend on the "service" and/or "dataProfile".

There are also several useful attributes attached to the data frame:

NameTypeDescription
urlcharacterThe url used to generate the data
siteInfodata.frameA data frame containing information on the requested sites
headerInfodata.frameA data frame returned from the WQP status service
queryTimePOSIXctThe time the data was returned

Details

This function uses ... as a query input, which can be very flexible, but also has a steeper learning curve. For a quick overview, scroll down to the Examples in this help file to see many query options.

There are currently 10 legacy options for data provided by the Water Quality Portal:

Legacy:

WQP Radio Buttonservice argumentBase URL
Sample ResultsResult/data/Result/search
Site Data OnlyStation/data/Station/search
Sampling ActivityActivity/data/Activity/search
Sampling Activity MetricsActivityMetric/data/ActivityMetric/search
Site Summary (not advertised on WQP)SiteSummary/data/summary/monitoringLocation/search
Project DataProject/data/Project/search
Project Monitoring Location Weighting DataProjectMonitoringLocationWeighting/data/ProjectMonitoringLocationWeighting/search
Result Detection Quantitation Limit DataResultDetectionQuantitationLimit/data/ResultDetectionQuantitationLimit/search
Biological Habitat MetricsBiologicalMetric/data/BiologicalMetric/search
Organization DataOrganization/data/Organization/search

There are 4 WQX3 options. These are still in-development, and should be used with caution.

WQP Radio Buttonservice argumentBase URLdataProfile
Monitoring LocationsStationWQX3/wqx3/Station/search
Full Physical ChemicalResultWQX3/wqx3/Result/searchfullPhysChem
NarrowResultWQX3/wqx3/Result/searchnarrow
Basic Physical ChemicalResultWQX3/wqx3/Result/searchbasicPhysChem
Sampling ActivityActivityWQX3/wqx3/Activity/search

Examples

# \donttest{

# Legacy:
nameToUse <- "pH"
pHData <- readWQPdata(siteid = "USGS-04024315", 
                      characteristicName = nameToUse)
#> GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv&count=no
#> GET: https://www.waterqualitydata.us/data/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.
ncol(pHData)
#> Error: object 'pHData' not found
attr(pHData, "siteInfo")
#> Error: object 'pHData' not found
attr(pHData, "queryTime")
#> Error: object 'pHData' not found
attr(pHData, "url")
#> Error: object 'pHData' not found

# WQX3:
pHData_wqx3 <- readWQPdata(siteid = "USGS-04024315", 
                           characteristicName = nameToUse,
                           service = "ResultWQX3",
                           dataProfile = "basicPhysChem")
#> GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=basicPhysChem&mimeType=csv
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.
attr(pHData_wqx3, "url")
#> Error: object 'pHData_wqx3' not found

# More examples:
# querying by county
DeWitt <- readWQPdata(
  statecode = "Illinois",
  countycode = "DeWitt",
  characteristicName = "Nitrogen"
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Result/search?countycode=US%3A17%3A039&characteristicName=Nitrogen&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

attr(DeWitt, "url")
#> Error: object 'DeWitt' not found

DeWitt_wqx3 <- readWQPdata(
   statecode = "Illinois",
   countycode = "DeWitt",
   characteristicName = "Nitrogen",
   service = "ResultWQX3",
   dataProfile = "basicPhysChem",  
   ignore_attributes = TRUE)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/wqx3/Result/search?countycode=US%3A17%3A039&characteristicName=Nitrogen&dataProfile=basicPhysChem&mimeType=csv
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

attr(DeWitt_wqx3, "url")
#> Error: object 'DeWitt_wqx3' not found

# Data profile: "Sampling Activity"
activity <- readWQPdata(
  siteid = "USGS-04024315",
  service = "Activity"
)
#> GET: https://www.waterqualitydata.us/data/Activity/search?siteid=USGS-04024315&mimeType=csv&count=no
#> GET: https://www.waterqualitydata.us/data/Station/search?siteid=USGS-04024315&mimeType=csv&count=no
#> NEWS: Data does not include USGS data newer than March 11, 2024. More details:
#> https://doi-usgs.github.io/dataRetrieval/articles/Status.html
attr(activity, "url")
#> [1] "https://www.waterqualitydata.us/data/Activity/search?siteid=USGS-04024315&mimeType=csv&count=no"

# activity_wqx3 <- readWQPdata(
#   siteid = "USGS-04024315",
#   service = "ActivityWQX3"
# )
# attr(activity_wqx3, "url")

Dane_activity <- readWQPdata(
  statecode = "Wisconsin",
  countycode = "Dane",
  startDateLo = "2023-01-01",
  startDateHi = "2023-12-31",
  service = "Activity"
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Activity/search?countycode=US%3A55%3A025&startDateLo=01-01-2023&startDateHi=12-31-2023&mimeType=csv&count=no
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Station/search?countycode=US%3A55%3A025&startDateLo=01-01-2023&startDateHi=12-31-2023&mimeType=csv&count=no
#> NEWS: Data does not include USGS data newer than March 11, 2024. More details:
#> https://doi-usgs.github.io/dataRetrieval/articles/Status.html
attr(Dane_activity, "url")
#> [1] "https://www.waterqualitydata.us/data/Activity/search?countycode=US%3A55%3A025&startDateLo=01-01-2023&startDateHi=12-31-2023&mimeType=csv&count=no"

# Dane_activity_wqx3 <- readWQPdata(
#   statecode = "Wisconsin",
#   countycode = "Dane",
#   startDateLo = "2023-01-01",
#   startDateHi = "2023-12-31",
#   service = "ActivityWQX3"
# )
# attr(Dane_activity_wqx3, "url")

########################################################
# Additional examples:

 
# Data profiles: "Organization Data" 
org_data <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "Organization"
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Organization/search?countycode=US%3A55%3A025&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# Data profiles: "Project Data"  
project_data <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "Project"
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Project/search?countycode=US%3A55%3A025&mimeType=csv&count=no
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Station/search?countycode=US%3A55%3A025&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# Data profiles: "Project Monitoring Location Weighting Data"
proj_mlwd <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "ProjectMonitoringLocationWeighting"
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/ProjectMonitoringLocationWeighting/search?countycode=US%3A55%3A025&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# Data profiles: "Sample Results (physical/chemical metadata)" 
samp_data <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "resultPhysChem",
  service = "Result"
)
#> GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-04024315&dataProfile=resultPhysChem&mimeType=csv&count=no
#> GET: https://www.waterqualitydata.us/data/Station/search?siteid=USGS-04024315&mimeType=csv&count=no
#> NEWS: Data does not include USGS data newer than March 11, 2024. More details:
#> https://doi-usgs.github.io/dataRetrieval/articles/Status.html

# Data profiles: "Sample Results (biological metadata)"
samp_bio <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "biological",
  service = "Result"
)
#> GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-04024315&dataProfile=biological&mimeType=csv&count=no
#> GET: https://www.waterqualitydata.us/data/Station/search?siteid=USGS-04024315&mimeType=csv&count=no
#> NEWS: Data does not include USGS data newer than March 11, 2024. More details:
#> https://doi-usgs.github.io/dataRetrieval/articles/Status.html


# Data profiles: "Sample Results (narrow)" 
samp_narrow <- readWQPdata(
  siteid = "USGS-04024315",
  service = "Result",
  dataProfile = "narrowResult"
)
#> GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-04024315&dataProfile=narrowResult&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# samp_narrow_wqx3 <- readWQPdata(
#   siteid = "USGS-04024315",
#   service = "ResultWQX3",
#   dataProfile = "narrow"
# )


# Data profiles: "Sampling Activity"  
samp_activity <- readWQPdata(
  siteid = "USGS-04024315",
  dataProfile = "activityAll",
  service = "Activity"
)
#> GET: https://www.waterqualitydata.us/data/Activity/search?siteid=USGS-04024315&dataProfile=activityAll&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# Data profile: "Sampling Activity Metrics"
act_metrics <- readWQPdata(
  statecode = "WI",
  countycode = "Dane",
  service = "ActivityMetric"
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/ActivityMetric/search?countycode=US%3A55%3A025&mimeType=csv&count=no
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Station/search?countycode=US%3A55%3A025&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# Data profile: "Result Detection Quantitation Limit Data"  
dl_data <- readWQPdata(
  siteid = "USGS-04024315",
  service = "ResultDetectionQuantitationLimit"
)
#> GET: https://www.waterqualitydata.us/data/ResultDetectionQuantitationLimit/search?siteid=USGS-04024315&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

# other options:
Phosphorus <- readWQPdata(
  statecode = "WI", countycode = "Dane", 
  characteristicName = "Phosphorus",
  startDateLo = "2023-01-01",
  ignore_attributes = TRUE,
  convertType = FALSE
)
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/counties?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://api.waterdata.usgs.gov/samples-data/codeservice/states?mimeType=application%2Fjson
#> GET: https://www.waterqualitydata.us/data/Result/search?countycode=US%3A55%3A025&characteristicName=Phosphorus&startDateLo=01-01-2023&mimeType=csv&count=no
#> Error in httr2::req_perform(obs_url): HTTP 500 Internal Server Error.

rawPHsites_legacy <- readWQPdata(siteid = c("USGS-05406450", "USGS-05427949", "WIDNR_WQX-133040"),
                        characteristicName = "pH",
                        service = "Result",
                        dataProfile = "narrowResult" )
#> GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-05406450%3BUSGS-05427949%3BWIDNR_WQX-133040&characteristicName=pH&dataProfile=narrowResult&mimeType=csv&count=no
#> GET: https://www.waterqualitydata.us/data/Station/search?siteid=USGS-05406450%3BUSGS-05427949%3BWIDNR_WQX-133040&characteristicName=pH&mimeType=csv&count=no
#> NEWS: Data does not include USGS data newer than March 11, 2024. More details:
#> https://doi-usgs.github.io/dataRetrieval/articles/Status.html

# rawPHsites <- readWQPdata(siteid = c("USGS-05406450", "USGS-05427949", "WIDNR_WQX-133040"),
#                           characteristicName = "pH",
#                           service = "ResultWQX3",
#                           dataProfile = "narrow" )

# }