fix(source-close-com): surface actionable error when Close _skip offset pagination cap is hit#76456
Draft
devin-ai-integration[bot] wants to merge 1 commit intomasterfrom
Draft
Conversation
…et pagination cap is hit
Adds a DefaultErrorHandler with an HttpResponseFilter to every stream
requester. The filter matches HTTP 400 responses whose body contains
"less than or equal to" (the substring Close.com returns when the
`_skip` offset pagination cap is exceeded) and converts the generic
CDK fallback ("Bad request. Please check your request parameters.")
into an actionable config_error that names the offending constraint.
Resolves airbytehq/airbyte-internal-issues#16215
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Contributor
|
Contributor
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Large workspaces on
source-close-com(>15000 records in a resource such ascontacts) fail with a generic and confusing error from the CDK fallback:The underlying cause is that Close.com's API rejects
_skipoffsets beyond a per-resource cap (15000 for contacts) with an HTTP 400 whose body saysInput should be less than or equal to 15000. Every stream in this connector paginates viaOffsetIncrementon_skip, so any sufficiently large account hits the cap, and the informative upstream message is lost behind the CDK's generic 400 fallback.This PR improves the user-facing error so it names the actual constraint.
Resolves airbytehq/airbyte-internal-issues#16215
Related to airbytehq/oncall#12012
How
Adds a
DefaultErrorHandlerwith a singleHttpResponseFilterto everyHttpRequesterin the manifest. The filter matches HTTP 400 responses whose error body contains the substringless than or equal toand emits an actionableconfig_error:Non-matching 400s continue to fall through to the default CDK mapping, so existing behavior is preserved.
The repetitive diff across all 45 streams (90 requester nodes, counting both
definitionsand top-levelstreams) is intentional — the manifest is fully resolved (no YAML anchors or$refs between streams), so each requester must carry its ownerror_handler.Scope and follow-ups
The parent triage issue explicitly calls out the error-message fix as an immediate, low-risk improvement that can land independently of the deeper pagination refactor. The real root cause — offset pagination that cannot retrieve more than 15000 records per resource — is not addressed here. A separate architectural change is needed to replace
_skip-based pagination with either a cursor or a date-range partition router. Suggested follow-up: open a dedicated issue for the pagination refactor.FailureType
Following the parent issue's recommendation, the classification is set to
config_errorrather than the CDK defaultsystem_error. This keeps the UI framing aligned with the fact that a user cannot resolve the underlying issue by retrying — the sync needs to be narrowed (or await the connector fix) before it will succeed.Review guide
airbyte-integrations/connectors/source-close-com/manifest.yaml— the 90error_handleradditions follow an identical pattern. Reviewing a single stream is sufficient to assess all of them.airbyte-integrations/connectors/source-close-com/metadata.yaml— patch version bump0.6.15→0.6.16.docs/integrations/sources/close-com.md— changelog entry.User Impact
Users hitting the Close.com
_skipcap will now see:instead of:
No behavioral changes for any other code path. Not a breaking change; patch-level bump.
Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/676b6d4368f945e19569f7b02d44db10