Connection Error : Database is locked after running make etl-pipeline

[kushal45]

Hi,

I am following the article :- https://decodingml.substack.com/p/data-pipelines-for-ai-assistants .

make local-infrastructure-up    # 1. Spin up the infrastructure
make download-notion-dataset    # 2. Download the Notion dataset

these 2 commands worked fine after running them locally where it ran the infrastructure and notion data was downloaded .

But after running the command , make etl-pipeline # 3. Run the ETL pipeline through ZenML , it connected with ZenML server daemon and read from disk pipeline finished properly . But, after the crawler pipeline started , the linux machine is freezing and after some time getting the error :- Connection Error : Database is locked .

Please Help as to how can this error be resolved .

Thanks,
Kushal Bhattacharya
Email ID : bhattacharya.kushal4@gmail.com

[iusztinpaul]

Hello @kushal45,

You probably need to lower the batch size you crawl in a single iteration (to make the crawling step lighter -> less CPU usage/iteration), or you can skip the crawling step and use the snapshot we provide for the next step, as instructed in the docs