This page will be updated frequently with information about the status of dataRetrieval services.
Contact CompTools@usgs.gov with additional questions.
Overview
On March 11, 2024, NWIS discrete water quality services were “frozen”: any public data retrieval using readNWISqw()
did not include any new data. As of dataRetrieval v2.7.17, readNWISqw()
has been retired and replaced by readWQPqw()
. Learn more about the change and where to find the new samples data in our blog.
If you have additional questions about the NWIS qw data service, email CompTools@usgs.gov.
Locating USGS data using the Water Quality Portal
New USGS data (post March 11, 2024) are temporarily not accessible on the main Water Quality Portal (WQP) page (www.waterqualitydata.us). Data are still being collected, but are not available on this webpage. This limited availability is expected to last a few months.
However, new USGS data are accessible in a pre-release (beta) version of the WQP web page and new wqx3 web services. Data are available in the “WQX version 3.0 format” (WQX = Water Quality Exchange) for these new “Data Profiles” (how the data is formatted by the WQP):
- Monitoring Location
- Results - Narrow
- Results - Full Physical Chemical
- Results - Basic Physical Chemical
- Sampling Activity
Guidance on how to use the new web page and web services are available in the User Guide and Web Services Guide. Additional profiles will continue to be added over time.
Disclaimer: During the beta period, users may encounter bugs or identify issues with the implementation of the WQX 3.0 format: we welcome (and encourage!) your feedback to help improve these offerings, just send an email to WQX@epa.gov.
The current WQP data profiles (available on the main Water Quality Portal web pages and from the current web services, https://www.waterqualitydata.us) deliver data in “WQX version 2.0” (what we’re referring to as the “legacy”) format. These will remain available for a period of time after the rollout of version 3.0. Eventually they will be retired, but there is not yet an estimated time line.
What to expect: dataRetrieval specific
Function | Status |
---|---|
readWQPdata | Set to legacy options by default. WQX3 options available. |
readWQPqw | Set to legacy options by default. WQX3 options available. |
whatWQPsites | Set to legacy options by default. WQX3 options available. |
whatWQPmetrics | Currently only available via legacy services. |
whatWQPsamples | Currently only available via legacy services. |
whatWQPdata | Currently only available via legacy services. |
readNWISqw | Retired. |
readWQPsummary | Does not have accurate information for USGS data. |
whatNWISdata | Does not have accurate information for qw data. |
readWQPqw
The readWQPqw()
function is generally advertised as a user-friendly function since it only works with a known list of sites, parameter codes or characteristic names, and start/end dates.
As of dataRetrieval
2.7.17, this function will use the default WQX version 2 dataProfile, specified by the legacy = TRUE
argument. Setting legacy = FALSE
will return the WQX 3.0 “narrow” dataProfile. Keep in mind the 2.0 profiles will eventually be retired. For any more flexibility, users will need to use the readWQPdata()
function.
An example of a WQX 3.0 return:
library(dataRetrieval)
rawPcode <- readWQPqw(siteNumbers = "USGS-01594440",
parameterCd = "01075",
legacy = FALSE)
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-01594440&pCode=01075&mimeType=csv&dataProfile=basicPhysChem
## WQX3 services are in-development, use with caution.
Compared to using the WQX 2.0 legacy results:
rawPcode_legacy <- readWQPqw(siteNumbers = "USGS-01594440",
parameterCd = "01075",
legacy = TRUE)
## GET: https://www.waterqualitydata.us/data/Result/search?siteid=USGS-01594440&pCode=01075&mimeType=csv
## NEWS: Data does not include USGS data newer than March 11, 2024. More details:
## https://doi-usgs.github.io/dataRetrieval/articles/Status.html
readWQPdata
The readWQPdata()
function is the most flexible function to get WQP data. Currently there are 11 legacy options and 5 options that use the new WQX 3.0 profiles. Note that readWQPdata()
does not leverage a legacy
argument to specify which profile version the user would like returned, but instead relies on the user’s specification of service
and dataProfile
arguments.
WQX 3.0
There are currently three WQX 3.0 “services” available: ResultWQX, StationWQX and ActivityWQX. The “ResultWQX” service has multiple available “dataProfiles”.
Service | dataProfile |
---|---|
StationWQX | |
ResultWQX | fullPhysChem |
ResultWQX | basicPhysChem |
ResultWQX | narrow |
ActivityWQX |
Examples:
data_full <- readWQPdata(siteid = "USGS-04024315",
characteristicName = "pH",
dataProfile = "fullPhysChem",
service = "ResultWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=fullPhysChem&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_full)
## [1] 182
data_basic <- readWQPdata(siteid = "USGS-04024315",
characteristicName = "pH",
dataProfile = "basicPhysChem",
service = "ResultWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=basicPhysChem&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_basic)
## [1] 100
data_narrow <- readWQPdata(siteid = "USGS-04024315",
characteristicName = "pH",
dataProfile = "narrow",
service = "ResultWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Result/search?siteid=USGS-04024315&characteristicName=pH&dataProfile=narrow&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_narrow)
## [1] 65
data_sites <- readWQPdata(siteid = "USGS-04024315",
characteristicName = "pH",
service = "StationWQX3")
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## GET: https://www.waterqualitydata.us/wqx3/Station/search?siteid=USGS-04024315&characteristicName=pH&mimeType=csv
## WQX3 services are in-development, use with caution.
ncol(data_sites)
## [1] 56
WQX 2.0 - Legacy
There are 8 services available from the legacy WQP. The Station and Result legacy services can still be accessed, but users should move to StationWQX, ResultWQX, and ActivityWQX. As other former services become available in WQX 3.0, we will update these documents.
Service | dataProfile | WQX 3.0 service “analog” |
---|---|---|
Station | StationWQX | |
Result | resultPhysChem | ResultWQX |
Result | biological | |
Result | narrowResult | ResultWQX |
Activity | activityAll | ActivityWQX |
ActivityMetric | ||
Project | ||
ProjectMonitoringLocationWeighting | ||
ResultDetectionQuantitationLimit | ||
BiologicalMetric |
Examples:
# Data profiles: "Organization Data" (legacy)
org_data <- readWQPdata(
statecode = "WI",
countycode = "Dane",
service = "Organization"
)
# Data profiles: "Project Data" (legacy)
project_data <- readWQPdata(
statecode = "WI",
countycode = "Dane",
service = "Project"
)
# Data profiles: "Project Monitoring Location Weighting Data" (legacy)
proj_mlwd <- readWQPdata(
statecode = "WI",
countycode = "Dane",
service = "ProjectMonitoringLocationWeighting"
)
# Data profiles: "Sample Results (physical/chemical metadata)" (legacy)
samp_data <- readWQPdata(
siteid = "USGS-04024315",
dataProfile = "resultPhysChem",
service = "Result"
)
# Data profiles: "Sample Results (biological metadata)" (legacy)
samp_bio <- readWQPdata(
siteid = "USGS-04024315",
dataProfile = "biological",
service = "Result"
)
# Data profiles: "Sample Results (narrow)" (legacy)
samp_narrow <- readWQPdata(
siteid = "USGS-04024315",
dataProfile = "narrowResult",
service = "Result"
)
# Data profiles: "Sampling Activity" (legacy)
samp_activity <- readWQPdata(
siteid = "USGS-04024315",
dataProfile = "activityAll",
service = "Activity"
)
# Data profile: "Sampling Activity Metrics" (legacy)
act_metrics <- readWQPdata(
statecode = "WI",
countycode = "Dane",
service = "ActivityMetric"
)
# Data profile: "Result Detection Quantitation Limit Data" (legacy)
dl_data <- readWQPdata(
siteid = "USGS-04024315",
service = "ResultDetectionQuantitationLimit"
)
whatNWISdata
NWIS discrete water quality services are “frozen”: the returned data availability will also be frozen ONLY for “qw” data_type_cd results. All other data types should not be affected.
When the NWIS services are decommissioned (likely in 2025): there will no longer be any “qw” information provided in the output of whatNWISdata
. Discrete water-quality availability will be available via WQP services. More information will be provided as we learn more.
Here’s an example of what will change:
whatNWISdata(siteNumber = "05114000")
what_NWIS <-nrow(what_NWIS)
1] 407
[nrow(what_NWIS[what_NWIS$data_type_cd == "qw",])
1] 381 [
So for site “05114000”, there are 381 NWIS qw parameters that have been measured. Since mid-March 2024, the data availability for those 381 parameters are frozen…even if new data are collected. Eventually those 381 rows of data will not be returned, only 26 rows of data will be returned (407-381).
New services/functions are being developed to replace the lost functionality so check back here for updated information.
Known Dependencies
HASP
The data_available
function will need to be updated.
If you know of additional R package dependencies that will be affected by these changes, please email CompTools@usgs.gov.