Skip to content

[Bug]: Many connections to Redis #2243

Open
#3887
Open
[Bug]: Many connections to Redis#2243
#3887
Labels
bugSomething isn't working
@venticello

Description

@venticello
venticello
opened on Oct 24, 2023

Version

1.15.2

Platform

Python

What happened?

I tried to get data from completed tasks from 2 queues, and in the end I got 34 additional connections to Redis.
Well, we also lack the ability to reuse an existing connection like in NodeJS.

How to reproduce.

import asyncio
from bullmq import Queue

elo_queue = Queue('elo', "redis://default:redispw@localhost:6379")
dna_queue = Queue('dna', "redis://default:redispw@localhost:6379")

async def get_data():
    jobs = await elo_queue.getCompleted(0, 20)
    jobs2 = await dna_queue.getCompleted(0, 20)
    print(jobs, jobs2)
    await asyncio.sleep(10)

if __name__ == "__main__":
    asyncio.run(get_data())

Relevant log output

127.0.0.1:6379> CLIENT LIST
id=20886 addr=172.24.0.1:47940 laddr=172.24.0.2:6379 fd=56 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20887 addr=172.24.0.1:47942 laddr=172.24.0.2:6379 fd=57 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20850 addr=172.24.0.1:47868 laddr=172.24.0.2:6379 fd=20 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20851 addr=172.24.0.1:47870 laddr=172.24.0.2:6379 fd=21 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20852 addr=172.24.0.1:47872 laddr=172.24.0.2:6379 fd=22 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20853 addr=172.24.0.1:47874 laddr=172.24.0.2:6379 fd=23 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20854 addr=172.24.0.1:47876 laddr=172.24.0.2:6379 fd=24 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20855 addr=172.24.0.1:47878 laddr=172.24.0.2:6379 fd=25 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20856 addr=172.24.0.1:47880 laddr=172.24.0.2:6379 fd=26 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20857 addr=172.24.0.1:47882 laddr=172.24.0.2:6379 fd=27 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20858 addr=172.24.0.1:47884 laddr=172.24.0.2:6379 fd=28 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20859 addr=172.24.0.1:47886 laddr=172.24.0.2:6379 fd=29 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20860 addr=172.24.0.1:47888 laddr=172.24.0.2:6379 fd=30 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20861 addr=172.24.0.1:47890 laddr=172.24.0.2:6379 fd=31 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20862 addr=172.24.0.1:47892 laddr=172.24.0.2:6379 fd=32 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20863 addr=172.24.0.1:47894 laddr=172.24.0.2:6379 fd=33 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20864 addr=172.24.0.1:47896 laddr=172.24.0.2:6379 fd=34 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20865 addr=172.24.0.1:47898 laddr=172.24.0.2:6379 fd=35 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20866 addr=172.24.0.1:47900 laddr=172.24.0.2:6379 fd=36 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=46 addr=127.0.0.1:35958 laddr=127.0.0.1:6379 fd=16 name= age=7372 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=26 qbuf-free=20448 argv-mem=10 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22426 events=r cmd=client|list user=default redir=-1 resp=2 lib-name= lib-ver=
id=20846 addr=172.24.0.1:47860 laddr=172.24.0.2:6379 fd=15 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20847 addr=172.24.0.1:47862 laddr=172.24.0.2:6379 fd=17 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20848 addr=172.24.0.1:47864 laddr=172.24.0.2:6379 fd=18 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20849 addr=172.24.0.1:47866 laddr=172.24.0.2:6379 fd=19 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20867 addr=172.24.0.1:47902 laddr=172.24.0.2:6379 fd=37 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20868 addr=172.24.0.1:47904 laddr=172.24.0.2:6379 fd=38 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20869 addr=172.24.0.1:47906 laddr=172.24.0.2:6379 fd=39 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20870 addr=172.24.0.1:47908 laddr=172.24.0.2:6379 fd=40 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20871 addr=172.24.0.1:47910 laddr=172.24.0.2:6379 fd=41 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20872 addr=172.24.0.1:47912 laddr=172.24.0.2:6379 fd=42 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20873 addr=172.24.0.1:47914 laddr=172.24.0.2:6379 fd=43 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20874 addr=172.24.0.1:47916 laddr=172.24.0.2:6379 fd=44 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20875 addr=172.24.0.1:47918 laddr=172.24.0.2:6379 fd=45 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20876 addr=172.24.0.1:47920 laddr=172.24.0.2:6379 fd=46 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20877 addr=172.24.0.1:47922 laddr=172.24.0.2:6379 fd=47 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20878 addr=172.24.0.1:47924 laddr=172.24.0.2:6379 fd=48 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20879 addr=172.24.0.1:47926 laddr=172.24.0.2:6379 fd=49 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20880 addr=172.24.0.1:47928 laddr=172.24.0.2:6379 fd=50 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20881 addr=172.24.0.1:47930 laddr=172.24.0.2:6379 fd=51 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20882 addr=172.24.0.1:47932 laddr=172.24.0.2:6379 fd=52 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20883 addr=172.24.0.1:47934 laddr=172.24.0.2:6379 fd=53 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20884 addr=172.24.0.1:47936 laddr=172.24.0.2:6379 fd=54 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20885 addr=172.24.0.1:47938 laddr=172.24.0.2:6379 fd=55 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions