Stabilize NWIS Tests and Improve 5xx Error Handling#223
Merged
thodson-usgs merged 9 commits intoDOI-USGS:mainfrom Apr 9, 2026
Merged
Stabilize NWIS Tests and Improve 5xx Error Handling#223thodson-usgs merged 9 commits intoDOI-USGS:mainfrom
thodson-usgs merged 9 commits intoDOI-USGS:mainfrom
Conversation
thodson-usgs
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR continues the NWIS deprecation work by ensuring defunct NWIS entry points fail in a controlled way, updating tests to reflect the deprecations (and reducing reliance on live NWIS responses for some cases), and improving handling of transient NWIS 5xx/HTML error responses.
Changes:
- Updated
tests/nwis_test.pyto use mocked JSON fixtures for IV parsing/empty responses and to assert defunct NWIS functions/services raise errors. - Added JSON fixtures to support mocked NWIS IV responses in tests.
- Improved
utils.query()handling for 5xx responses and added HTML-instead-of-JSON detection inget_dv()/get_iv().
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/nwis_test.py | Switches some IV tests to mocked responses, adds defunct-service assertions, and adds a new live NWIS sanity check. |
| tests/data/nwis_iv_mock.json | Adds a minimal JSON response fixture for IV parsing tests. |
| tests/data/nwis_iv_empty_mock.json | Adds an empty timeSeries JSON fixture for empty-response behavior. |
| dataretrieval/utils.py | Raises a clearer ValueError for HTTP 500/502/503 “service unavailable” responses. |
| dataretrieval/nwis.py | Adds HTML-response detection for get_dv()/get_iv(), updates get_record docs/service validation, and deprecates NWIS_Metadata.variable_info. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Collaborator
Author
|
@copilot apply changes based on the comments in this thread |
…ction, and test refinement
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ndling, and defunct service logic
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Remove stray PR217.md tracking file - Extract duplicated HTML-response detection into _parse_json_or_raise helper - Remove dead defunct_services entry from get_record type check - Point get_record defunct NameError to replacement functions - Remove unreachable empty-dfs branch in _walk_pages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses several recent CI failures by stabilizing the NWIS test suite and improving HTTP error handling. It also makes a few other behind-the-scenes improvements to the
waterdatamodule.Key Changes:
utils.pyto better handle 500, 502, and 503 HTTP status codes from USGS services, which have been causing transient CI failures.NWIS_Metadata.variable_infoandget_recordto ensure consistent and informative behavior during service deprecation._walk_pagesinwaterdata/utils.pyto use list-based aggregation, reducing memory copying overhead from_get_argshelper and refactored all 11 API functions inwaterdata/api.pyto use it.to_strinutils.pywithmap(str, ...)and broader iterable support (sets, tuples, generators).waterdata_utils_test.pyand expandedtests/utils_test.py.