Skip to content

[Backend] Lazy Model Initialization & Fault-Tolerant Startup (#302)#323

Open
j4b3-21 wants to merge 6 commits intojenkinsci:mainfrom
j4b3-21:feature/lazy-model-init-302
Open

[Backend] Lazy Model Initialization & Fault-Tolerant Startup (#302)#323
j4b3-21 wants to merge 6 commits intojenkinsci:mainfrom
j4b3-21:feature/lazy-model-init-302

Conversation

@j4b3-21
Copy link
Copy Markdown

@j4b3-21 j4b3-21 commented Mar 29, 2026

This PR implements lazy, thread-safe initialization for embedding and LLM models to improve API startup reliability and reduce cold-start latency.

Previously, models were loaded during import, causing slow startup and failures when models were unavailable.
Now, models initialize only on first use and fallback gracefully on failure.A minor fix was done to footer.jelly to match the root name with main.tsx

Changes

•	Added runtime_models.py for lazy model loading
•	Implemented thread-safe cached getters
•	Added graceful fallback when models fail
•	Guarded retrieval paths to prevent crashes
•	Removed heavy initialization at import time
•	Updated footer.jelly root ID from chatbot-root → root to match main.tsx

Testing Done

•	API startup without models → successful
•	First request → lazy initialization works
•	Concurrent requests → single initialization, cached reuse
•	Simulated failure → fallback response returned
•	Verified UI loads correctly after root ID change (screenshots attached)

Files Modified

•	chat_service.py
•	tools.py
•	embedding_model.py
•	llama_cpp_provider.py
•	main.py
•	footer.jelly

New file:
• runtime_models.py

##Related Issue
Closes #302

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests that demonstrate
Screenshot 2026-03-29 at 12 34 03 PM Screenshot 2026-03-29 at 12 31 20 PM

@j4b3-21 j4b3-21 requested a review from a team as a code owner March 29, 2026 07:20
@berviantoleo berviantoleo added the enhancement For changelog: Minor enhancement. use `major-rfe` for changes to be highlighted label Mar 29, 2026
@j4b3-21
Copy link
Copy Markdown
Author

j4b3-21 commented Apr 12, 2026

To fix the linting issue ,
i would have to run pylint . on the entire chatbot-core right??

edit-
i ran the pylint . command and fixed all the issues

@berviantoleo
Copy link
Copy Markdown
Contributor

To fix the linting issue ,
i would have to run pylint . on the entire chatbot-core right??

edit-
i ran the pylint . command and fixed all the issues

Yes

@j4b3-21
Copy link
Copy Markdown
Author

j4b3-21 commented Apr 12, 2026

To fix the linting issue ,
i would have to run pylint . on the entire chatbot-core right??
edit-
i ran the pylint . command and fixed all the issues

Yes

yes i did that and updated the pr

"""Backward compatibility wrapper."""
model = get_embedding_model()
if model is None:
raise RuntimeError(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should raise Error

Comment thread chatbot-core/api/main.py
except Exception: # pylint: disable=broad-except
pass

# pylint: disable=import-outside-toplevel
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you ignore this pylint?

Check logs for error details.
"""
return _get_cached_model(
"embedding",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define this on top of the file as a var.

"""

return _get_cached_model(
"llm",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define on top

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement For changelog: Minor enhancement. use `major-rfe` for changes to be highlighted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Backend] Make model initialization lazy and fault-tolerant to improve API startup reliability

2 participants