security: validate URLs in WebScraperTool to prevent SSRF (fixes #1516)#1528
Open
AshrafHosam wants to merge 133 commits into
Open
security: validate URLs in WebScraperTool to prevent SSRF (fixes #1516)#1528AshrafHosam wants to merge 133 commits into
AshrafHosam wants to merge 133 commits into
Conversation
* List File S3 * Unit Test Added
* fixed toolconfig update when installed from marketplace * toolconfig and toolkit test fix
* schedule agent fix --------- Co-authored-by: Rounak Bhatia <rounak@contlo.com>
* Delay Changed * Updated Feed Polling Time
Merged Dev -> Main Released v11 - Models Marketplace - SuperAGI APIs - Support for Weaviate DB - Tool Memory LTM - Google Analytics as External Tool - JSON support for Tool Configuration - Bug Fixes and Minor Enhancements.
- Models Marketplace - SuperAGI APIs - Support for Weaviate DB - Tool Memory LTM - Google Analytics as an External Tool - JSON support for Tool Configuration - Bug Fixes and Minor Enhancements. - Jira Tool Fix
…lle_api_fix_main added tool config for dalle
…x_get_matching_text add null check
- Added new contributors - Added New Features - Added API documentation link
…adme_update_v11 Updated README with new features, contributors, and API docs
…owledge_model_fix_main Knowledge search fix
* fix * fixing the toolkit config in iteration workflow * List file s3 fix (TransformerOptimus#1076) * List File S3 * Unit Test Added --------- Co-authored-by: Taran <97586318+Tarraann@users.noreply.github.com> * workflow changes * minor seed file fix * frontend fixes (TransformerOptimus#1079) * fixed api_bug (TransformerOptimus#1080) * add one condition (TransformerOptimus#1082) * api fix (TransformerOptimus#1087) * Tools error fix (TransformerOptimus#1093) * webhooks frontend + api calls complete almost * Tool-LTM(Updated) (TransformerOptimus#1039) * Toolkit configuration fix (TransformerOptimus#1102) * webhooks compplete frontend * schedule agent fix (TransformerOptimus#1104) Co-authored-by: Rounak Bhatia <rounak@contlo.com> * Models superagi (TransformerOptimus#936) Models Superagi * Models superagi (TransformerOptimus#1108) Bug Fixes * Changes for no receiver address * made changes to github helper * Models superagi (TransformerOptimus#1112) * Models superagi (TransformerOptimus#1117) * Models fixes (TransformerOptimus#1118) * Models Frontend Changes * Models Frontend Changes * Models Frontend Changes * \n bug resolved (TransformerOptimus#1122) * PDF and DOCX support in Write File - Feature Improvement, close TransformerOptimus#548 (TransformerOptimus#928) * Added functions to write various file types and a file handler too * FileManager updated to handle and save HTMLs. * adding PDF + DOCX support to save images * Added Wkhtmltopdf package installation run commands in docker * Added get_all_responses feature for extractng the response for particular tools * Added Image embedding feature, this will extract and embed the images generated by the agent during the run * renaming functions and refactoring * renaming functions and refactoring * Update Dockerfile * removing unsused classmethods * Finding generated images and attached files in the write tool. Images are fetched inorder to be embedded in the respective file type. * Adding the filename and paths to the Resource Manager S3 storage * Code Cleanup * added logger: Fix for the failing TEST --------- Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> * Revert "PDF and DOCX support in Write File - Feature Improvement, close TransformerOptimus#548 (TransformerOptimus#928)" (TransformerOptimus#1124) This reverts commit 8b01357. * expose port * latest safetensors breaking in macs (TransformerOptimus#1134) * Changes in save template (TransformerOptimus#1120) * fix * changes in save template * changes in save template * bug fixes * changes in save template * changes in save template --------- Co-authored-by: Rounak Bhatia <rounak@contlo.com> Co-authored-by: namansleeps <mandhani12@gmail.com> * Main to dev sync (TransformerOptimus#1139) * Models fixes (TransformerOptimus#1126) * Models Frontend Changes * Models Frontend Changes * Models Frontend Changes * Backend Compatibility for New/Existing users on local * DEV api key requirements * removing print statements * removing print statements * removing print statements * removing print statements * backend compatibility * backend compatibility * backend compatibility * added filters in the webhooks * fix * added filters in the webhooks * Models fixes (TransformerOptimus#1145) Fixes related to Models Feature * Jira Bug Fix * Jira Bug Fix 2.0 * Jira Bug Fix 3.0 * added filters in the webhooks * Models fixes (TransformerOptimus#1147) Model Feature Fixes * Bug fix model redirection (TransformerOptimus#1148) Bug Fix - Model URL Redirection * added tool config for dalle * removed model dependency on dalle tool * Remove hardcoded creds * fixed env error * removed refactoring from main * removed refactoring * removed refactoring * handled error * stop agent from executing if model is not found (TransformerOptimus#1156) * entity details (TransformerOptimus#1158) * Metric frontend (TransformerOptimus#1152) * added filters in webhooks * added filters in webhooks * minor changes * webhooks complete * minor changes for PR * minor changes for PR * Publish agent template to marketplace (TransformerOptimus#1106) * publish agent to marketplace --------- Co-authored-by: Rounak Bhatia <rounak@contlo.com> Co-authored-by: namansleeps <mandhani12@gmail.com> * added filters in webhooks * added filters in webhooks * resolving conflicts * added filters in the webhooks * lint issue fixed * bug fix of prev PR * fix for new run and edit agent * error handling * added filters in webhooks * fix for knowledge search tool * Docker digitalocean deployment * changed branch name * added filters in the webhooks * changes * removed region * added button * change in branch * added filters in the webhooks * Update conftest.py * Added filters in the webhooks (TransformerOptimus#1140) * webhooks frontend + web hooks with filters --------- Co-authored-by: namansleeps <mandhani12@gmail.com> Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> * Models calls logs dev (TransformerOptimus#1174) Call logs organisation level bug fix * models scroll fix, format of log timestamp fix, adding of loader to models, toolkit metrics dropdown bug fixed, publish agent dropdown bug (TransformerOptimus#1171) * Update app.yaml (TransformerOptimus#1179) * fixes related to webhooks * fixes for webhooks * Fixes for webhooks (TransformerOptimus#1181) * fixes for webhooks --------- Co-authored-by: namansleeps <mandhani12@gmail.com> Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> * bugs by qa (TransformerOptimus#1178) * Fix for schedule agent (TransformerOptimus#1184) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * Entity fix (TransformerOptimus#1185) * fixes for webhooks * fixes for webhooks * fix added for index state (TransformerOptimus#1188) * fix added for index state * Update KnowledgeTemplate.js --------- Co-authored-by: Tarraann <jot.taran15522@gmail.com> * API bug fixes for SDK (TransformerOptimus#1189) * fix api's for sdk * removed unused imports --------- Co-authored-by: jagtarcontlo <123375045+jagtarcontlo@users.noreply.github.com> * Main to dev sync v12 (TransformerOptimus#1193) sync back to dev ------ Co-authored-by: Taran <97586318+Tarraann@users.noreply.github.com> Co-authored-by: TransformerOptimus <muknrq@gmail.com> Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> Co-authored-by: Maverick-F35 <138012351+Maverick-F35@users.noreply.github.com> Co-authored-by: BoundlessAsura <122777244+boundless-asura@users.noreply.github.com> Co-authored-by: Akshat Jain <92881074+Akki-jain@users.noreply.github.com> Co-authored-by: sayan1101 <139119661+sayan1101@users.noreply.github.com> Co-authored-by: Rounak Bhatia <f20201807@goa.bits-pilani.ac.in> Co-authored-by: Rounak Bhatia <rounak@contlo.com> Co-authored-by: Kalki <97698934+jedan2506@users.noreply.github.com> Co-authored-by: Tarraann <jot.taran15522@gmail.com> Co-authored-by: rakesh-krishna-a-s <akrishna@contlo.com> Co-authored-by: Captain Levi <123375045+CaptainLevi0007@users.noreply.github.com> Co-authored-by: andrew-kelly-neutralaiz <128111428+andrew-kelly-neutralaiz@users.noreply.github.com> Co-authored-by: James Wakelim <james.wakelim@neutralaiz.com> * added button * GitHub pull request tools (TransformerOptimus#1190) * adding github review tools * cleanup and adding code review prompt * fixing comments * PDF and DOCX support in Write File - Feature Improvement, close TransformerOptimus#548 (TransformerOptimus#1125) Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> Co-authored-by: Abhijeet <129729795+luciferlinx101@users.noreply.github.com> * minor documentation fix * Design bugs (TransformerOptimus#1199) * fetching token limit from db * Revert "PDF and DOCX support in Write File - Feature Improvement, close TransformerOptimus#548 (TransformerOptimus#1125)" (TransformerOptimus#1202) This reverts commit 26f6a1d. * Unit Test Fix (TransformerOptimus#1203) * adding of docs and and discord link correction (TransformerOptimus#1205) * openai error handling * error_handling * api call only when agent is running * Feature : Wait block for agent workflow (TransformerOptimus#1186) Agent Wait Block Step * minor changes (TransformerOptimus#1213) Co-authored-by: Jagtar Saggu <jagtarsaggu@Jagtars-MacBook-Pro.local> * error handling * error handling * error handling * error handling * error handling * fix * fix * fix * error handling * models changes (TransformerOptimus#1207) Model related frontend changes. * error handling * models marketplace changes (TransformerOptimus#1219) Co-authored-by: Abhijeet <129729795+luciferlinx101@users.noreply.github.com> * minor changes * error handling * error handling * removing single qoutes (TransformerOptimus#1224) Co-authored-by: namansleeps <mandhani12@gmail.com> * apm changes (TransformerOptimus#1222) APM Bug Fixes * list tool fix * list tool fix * PR CHANGES * entity fix for dev (TransformerOptimus#1230) Entity Fix for Dev * frontend changes (TransformerOptimus#1231) * read_tool_fix * fix * waiting block frontend (TransformerOptimus#1233) Waiting Block Changes and Frontend Addition * Dev Fixes (TransformerOptimus#1242) * read tool fix * Maintaining dev (TransformerOptimus#1244) Dev Fix * added logs (TransformerOptimus#1246) * error_handling fix (TransformerOptimus#1247) Error handling fix. * Feature first login src (TransformerOptimus#1241) Adding source in user database for analytics. * apollo NoneType bug fix (TransformerOptimus#1238) Bug Fix * Mixpanel integration (TransformerOptimus#1256) Mix Panel * Models Marketplace bug fix for dev (TransformerOptimus#1266) * Fix 1257 dev (TransformerOptimus#1269) Bug Fix * add cache layer (TransformerOptimus#1275) Added package caching for github actions workflow * Fix api dev (TransformerOptimus#1283) * save other config to agent_execution config * add config * mixpanel changes (TransformerOptimus#1285) * rename error_handling.py to error_handler.py (TransformerOptimus#1287) * Analytics login (TransformerOptimus#1258) * calendar issues fixed * append fle tool bug fixed (TransformerOptimus#1294) Co-authored-by: Tarraann <jot.taran15522@gmail.com> * adding cookie in access token (TransformerOptimus#1301) * local_llms * local_llms * local_llms * local_llms * local_llms * fixes * models error fixed (TransformerOptimus#1308) * local_llms * local_llms * local_llms * local_llms * local_llms * frontend_changes * local_llms * local_llms * local_llms * local_llms * local_llms_frontend * fixes * fixes * fixes * fixes * merged main into local_llm_final * merged main into local_llm_final * local llms --------- Co-authored-by: TransformerOptimus <muknrq@gmail.com> Co-authored-by: Abhijeet <129729795+luciferlinx101@users.noreply.github.com> Co-authored-by: Taran <97586318+Tarraann@users.noreply.github.com> Co-authored-by: Fluder-Paradyne <121793617+Fluder-Paradyne@users.noreply.github.com> Co-authored-by: Maverick-F35 <138012351+Maverick-F35@users.noreply.github.com> Co-authored-by: namansleeps <mandhani12@gmail.com> Co-authored-by: Aditya Sharma <138581531+AdityaSharma13064@users.noreply.github.com> Co-authored-by: Sayan Samanta <139119661+sayan1101@users.noreply.github.com> Co-authored-by: Kalki <97698934+jedan2506@users.noreply.github.com> Co-authored-by: Tarraann <jot.taran15522@gmail.com> Co-authored-by: Arkajit Datta <61142632+Arkajit-Datta@users.noreply.github.com> Co-authored-by: rakesh-krishna-a-s <akrishna@contlo.com> Co-authored-by: jagtarcontlo <123375045+jagtarcontlo@users.noreply.github.com> Co-authored-by: Captain Levi <123375045+CaptainLevi0007@users.noreply.github.com> Co-authored-by: namansleeps <122260931+namansleeps@users.noreply.github.com> Co-authored-by: I’m <133493246+TransformerOptimus@users.noreply.github.com> Co-authored-by: Jagtar Saggu <jagtarsaggu@Jagtars-MacBook-Pro.local> Co-authored-by: Ubuntu <ubuntu@ip-10-14-2-4.ec2.internal>
…xes-for-main fixes for main
…xes_for_main_1 fixes
* use get_config * add a check on the key
…ransformerOptimus#1255) (TransformerOptimus#1361) * Adds error handling for openai's rate limit error in llms/openai module and its tests - Adds test for rate limit error handling in the llms/openai module - Adds error handling for rate limit error in the llms/openai module - Refactors code in llms/openai module to be readable and modular * Adds error handling for openai's timeout error in llms/openai module and its test - Adds test for timeout error handling in chat_completion in llms/openai module - Adds error handling for openai's timeout error in chat_completion in llms/openai module * Adds error handling for openai's try again error in llms/openai module and its test - Adds test for openai's try again error handling in chat_completion in llms/openai module - Adds error handling for openai's try again error in chat_completion in llms/openai module * Refactors llms/openai module and its tests to return error after retry attempts are exausted * Increases wait time for retry of chat_completion in llms/openai module * Removes unused import
fixed spelling error in config_teemplate.yaml
…unak610-patch-1 Update docker-compose-gpu.yml
…xes_for_settings_error fixes for settings error
* Update user.py issue 1358 * Update auth.py issue 1358
Fixed Discord Link in "Join our [Discord community] for support and discussions" in README.MD
…e_direct_object_reference_idor_1737105809757082 Title: Fix authorization flaw in `download_file_by_id` function to ensure proper user access validation for resource downloads.
Fix IDOR Security Vulnerability on /api/resources/get/{resource_id}
…sformerOptimus#1516) WebScraperTool passes an LLM-supplied URL directly to requests.get via WebpageExtractor. With no URL validation, prompt injection can coerce the agent into fetching internal services — cloud metadata endpoints (169.254.169.254), loopback, or RFC1918 ranges — exfiltrating credentials (CWE-918). Add superagi/helper/url_validator.validate_public_url which: - restricts schemes to http/https - resolves the hostname and rejects private, loopback, link-local, multicast, reserved, and unspecified IP addresses (ipaddress stdlib) - blocks known cloud-metadata hostnames explicitly Call the validator at the entry point of every extractor method and in WebScraperTool._execute. Add 22 unit tests covering valid URLs, rejected schemes, literal internal IPs (IPv4 + IPv6), cloud metadata hosts, and DNS-rebind style hostnames.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1516 — Server-Side Request Forgery (CWE-918, CVSS 7.5) in
WebScraperTool.WebScraperTool._execute()passed the LLM-suppliedwebsite_urldirectly toWebpageExtractor().extract_with_bs4(), which callsrequests.get(url)with no URL validation. A prompt-injected agent could therefore fetch:http://169.254.169.254/latest/meta-data/iam/security-credentials/) → instance credentialsChange
Introduces
superagi/helper/url_validator.pywithvalidate_public_url(url):http/httpssocket.getaddrinfoand rejects any IP that isis_private,is_loopback,is_link_local,is_multicast,is_reserved, oris_unspecified(uses theipaddressstdlib — no new dependency)metadata.google.internal,metadata.azure.com, etc.)Wired in at every outbound-request entry point:
WebpageExtractor.extract_with_3kWebpageExtractor.extract_with_bs4WebpageExtractor.extract_with_lxmlWebScraperTool._execute(returns a clean refusal message on failure)Tests
Adds
tests/unit_tests/helper/test_url_validator.py(22 cases): valid public URLs, empty/non-string inputs, disallowed schemes (ftp,file,gopher,data), missing hostname, AWS/GCP metadata endpoints, literal IPv4/IPv6 internal addresses (127.0.0.1, 10.0.0.1, 172.16.0.1, 192.168.1.1, 169.254.1.1, 0.0.0.0, ::1, fc00::1), DNS-rebind hostnames that resolve to private IPs, unresolvable hostnames.All 22 pass locally.
Test plan
pytest tests/unit_tests/helper/test_url_validator.py— 22/22 pass