Skip to content

opensearch: generalize extra parameters#65613

Open
DonHaul wants to merge 3 commits intoapache:mainfrom
DonHaul:opensearchhook-extend
Open

opensearch: generalize extra parameters#65613
DonHaul wants to merge 3 commits intoapache:mainfrom
DonHaul:opensearchhook-extend

Conversation

@DonHaul
Copy link
Copy Markdown
Contributor

@DonHaul DonHaul commented Apr 21, 2026

The current OpenSearch hook, does not include the possibility to set arguments for the opensearch client that may be important for some users such as http_compress and url_prefix etc. opensearchpy library supports many more arguments, including kwargs - see here
As such I've generalized the hook so all the parameters in extra are passed to the opensearch client


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
  • Codex GPT-5.4

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@eladkal
Copy link
Copy Markdown
Contributor

eladkal commented Apr 22, 2026

cc @Owen-CH-Leung

@DonHaul DonHaul force-pushed the opensearchhook-extend branch from 1d3ef41 to 2af56cd Compare April 22, 2026 14:25
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 22, 2026
Copy link
Copy Markdown
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is too generic as it might update things we do not want to update via extras. Extras are potentially more sensitive than Hook parameters, because they are configurable by "connection editing" users and not "Dag Authors" - we never pass extras directly to hoo params/args we are always explicit which extras should be passed.

@DonHaul
Copy link
Copy Markdown
Contributor Author

DonHaul commented Apr 29, 2026

okay I will specifically define all the parameters in https://opensearch-project.github.io/opensearch-py/_modules/opensearchpy/connection/base.html#Connection . I see that the kwargs is actually only scheme so I will add it as an arg as well

@DonHaul DonHaul force-pushed the opensearchhook-extend branch 3 times, most recently from b9b6998 to 1ff244d Compare April 30, 2026 14:37
@DonHaul DonHaul force-pushed the opensearchhook-extend branch from 1ff244d to 3ba6c52 Compare April 30, 2026 14:54
@DonHaul
Copy link
Copy Markdown
Contributor Author

DonHaul commented Apr 30, 2026

@potiuk comments addressed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants