Overview
Welcome to the dataRetrieval status page. This page will
be updated frequently with information about the status of
dataRetrieval services.
Please contact the Computational Tools team at CompTools@usgs.gov with questions.
New Water Data APIs
What is an application programming interface (API)?
When you run adataRetrieval function like
read_USGS_samples() or readNWISdv(), you are
actually generating a URL that contains within it specifications of the
dataset in which you are interested (e.g. which monitoring locations,
characteristics, pcodes, start and end dates, etc.). The format of that
URL is special: it is parsed by a USGS-specific API web service, which
translates the request into a database call. It then packages the
response object from the database (with the data) into a format that can
be sent to and then unpacked by the user. dataRetrieval
takes a lot of the guesswork out of this process by generating the URL,
sending it to the API, and wrangling the response object into a tabular
dataset.
The USGS is in the process of creating new, publicly available APIs
(a.k.a. “web services”) to replace the existing WaterServices.
dataRetrieval relies upon these web services to provide
monitoring location information and water quantity/quality datasets. As
new web services come online, dataRetrieval will be
furnished with new functions to accommodate these changes.
The table below shows the existing NWIS functions
and, if applicable, their slated replacements. Note that several
functions do not have direct replacements because the new services to
support them do not yet exist. We will update this table as new services
and functions come online. Also note that some new functions may only be
available on the “develop” branch of dataRetrieval
(remotes::install_github("DOI-USGS/dataRetrieval", ref = "develop")).
More information on each function’s specifications (complete with
examples) are available in the Introduction
to New USGS Services vignette, the package
index and/or function documentation
(e.g. ?read_USGS_daily).
| WaterServices (legacy) function | Water Data (new) function | Available on (branch name) |
|---|---|---|
| readNWISqw (deprecated) | read_USGS_samples | main (CRAN) |
| readNWISdv | read_USGS_daily | develop |
| readNWISsite | read_USGS_monitoring_location | develop |
| whatNWISsites | read_USGS_ts_meta | develop |
| read_USGS_data | develop | |
| readNWISuv | ||
| readNWISrating | ||
| readNWISstat | ||
| readNWISmeas | ||
| readNWISpeak | ||
| readNWISgwl | ||
| readNWISuse | ||
| readNWISdata | ||
| whatNWISdata | ||
| readNWISpCode |
If you want to learn more about the new water data APIs, check out the “What’s new with WDFN APIs?” blog post, as well as the documentation available on api.waterdata.usgs.gov.
API Keys
Do you make a lot of dataRetrieval
WaterServices calls (e.g. using functions like readNWISdv,
readNWISuv) per day? …On the order of more than 50 function
calls per hour? As you switch your workflows over to the new Water Data
API functions, consider grabbing yourself an API key, which will bump
your limit up to 1,000 requests per hour. Head to the sign up page to get a
token.
One you have your API key, add it to your .Renviron file
like this:
API_USGS_PAT = "[your api key]"Replace [your api key] with the alphanumeric code provided by the
sign-up page. That’s it! dataRetrieval will look for an
.Renviron file in your directories and use it for making
web service calls.
Discrete Data
In March 2024, NWIS discrete water quality services
were “frozen”: any public data retrieval using readNWISqw()
no longer included any new data. Concurrently, the main Water Quality Portal (WQP)
API stopped serving new and updated USGS data (we will refer to this
set of web services as “legacy”). Now, new and updated data are
available from the USGS
Samples API (for USGS data only) or in the beta version of the WQP
(both USGS and non-USGS data).
What does this mean for water quality data users of
dataRetrieval? Check out the sections below for more
specifics.
Samples Data
There’s a new set of functions that access the USGS “samples-data”
services! If you are only interested in USGS discrete
water quality data, you can use the read_USGS_samples
function.
Read more about it in the vignette, Introducing read_USGS_samples.
WQP
dataRetrieval WQP functions continue to default to the
legacy system (that does not include post-March 2024 USGS discrete
sample data). The new replacement services aren’t currently set as the
default because the WQP team still considers these services “beta”, and
therefore performance is not guaranteed. Users may encounter bugs or
identify issues with the implementation of the new services: we welcome
(and encourage!) your feedback to help improve these offerings, just
send an email to WQX@epa.gov.
The table below provides a summary of the current state of WQP
functions in dataRetrieval.
| Function | Status |
|---|---|
| readWQPdata | Set to legacy options by default. WQX3 options available. |
| readWQPqw | Set to legacy options by default. WQX3 options available. |
| whatWQPsites | Set to legacy options by default. WQX3 options available. |
| whatWQPmetrics | Currently only available via legacy services. |
| whatWQPsamples | Currently only available via legacy services. |
| whatWQPdata | Currently only available via legacy services. |
| readWQPsummary | Does not have accurate information for USGS data. |
| whatNWISdata | Does not have accurate information for qw data. |
Leveraging the beta WQP services
The beta WQP offers data in the “WQX version 3.0 format” (WQX = Water Quality Exchange) using new “data profiles” (how the data are formatted by the WQP). There are currently three WQX 3.0 “services” available on beta: ResultWQX3, StationWQX3 and ActivityWQX3. The “ResultWQX3” service has multiple available data profiles:
| Service | dataProfile |
|---|---|
| StationWQX3 | |
| ResultWQX3 | fullPhysChem |
| ResultWQX3 | basicPhysChem |
| ResultWQX3 | narrow |
| ActivityWQX3 |
Quickstart: To access the most recent USGS data from
the new services in dataRetrieval, use the function
readWQPqw and set the argument legacy=FALSE or
use the function readWQPdata and set the argument
service = "ResultWQX3".
If you wish to leverage a specific “ResultWQX3” data profile using
the beta services, your code might look something like this, using the
very flexible readWQPdata function:
data_full <- readWQPdata(siteid = "USGS-04024315",
characteristicName = "pH",
dataProfile = "fullPhysChem",
service = "ResultWQX3")On the other hand, the “StationWQX3” service requires no
dataProfile argument:
data_station <- readWQPdata(siteid = "USGS-04024315",
characteristicName = "pH",
service = "StationWQX3")Guidance on how to use the new web page and web services are available in the User Guide and Web Services Guide. Additional profiles will continue to be added over time.
NWIS “qw” Data
As of dataRetrieval v2.7.17, readNWISqw() has been
retired and replaced by readWQPqw(). The
readWQPqw() function is generally advertised as a
user-friendly function since it only works with a known list of sites,
parameter codes or characteristic names, and start/end dates. Learn more
about the change and where to find the new samples data in the
dataretrieval Changes
to NWIS QW services vignette and the WDFN blog on Improved
Public Delivery of Water Quality and Field Samples.
A note on whatNWISdata and “qw” data
Currently, any “qw” data summaries provided by
whatNWISdata are very likely out of date and incorrect: do
not use this function to query by data_type_cd = "qw". When
the NWIS services are decommissioned (likely in 2025) there will no
longer be any “qw” information provided in the output of
whatNWISdata. Discrete water-quality data will be available
via WQP services and USGS Samples only. More information will be
provided as we learn more.