Not able to create more than 773 PVCs for sequential creation of pvcs

[rmadaka]

Describe the bug

Written a script like, it will create pvc and it will wait for pvc to get bound. Then it will write into file time taken for pvc claim . Afterwards only script will go for creating next pvc. Like this i have tried for first 500 pvc. Then i tried to create 500 more pvc. Total 773 pvcs got bound. after 773 pvc , 774th pvc not able to bound.

Steps to reproduce

1. Using script created 500 pvcs sequentially.
2. PVCS got bound successfully
3. Then tried to create 500 more pvcs sequentially using script, continued from 501 pvc on same setup
4. Total 773 pvcs got bound.
5. Then script is trying for 774th pvc, 774th pvc is pending state, 774th pvc not getting bound..

Actual results
774th pvc is in pending state.

Expected behavior
774th pvc should get bound.

Additional context
csi provisioner Logs:

I1218 20:21:36.524013       1 controller.go:927] provision "default/gcs-pvc774" class "glusterfs-csi": started
I1218 20:21:36.551970       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc774"
W1218 20:21:42.513459       1 controller.go:686] Retrying syncing claim "default/gcs-pvc774" because failures 13 < threshold 15
E1218 20:21:42.513531       1 controller.go:701] error syncing claim "default/gcs-pvc774": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large
I1218 20:21:42.514556       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large
I1218 20:38:22.514310       1 controller.go:927] provision "default/gcs-pvc774" class "glusterfs-csi": started
I1218 20:38:22.529313       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc774"
W1218 20:38:27.876027       1 controller.go:686] Retrying syncing claim "default/gcs-pvc774" because failures 14 < threshold 15
E1218 20:38:27.876090       1 controller.go:701] error syncing claim "default/gcs-pvc774": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large
I1218 20:38:27.876124       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large
I1218 20:55:07.876406       1 controller.go:927] provision "default/gcs-pvc774" class "glusterfs-csi": started
I1218 20:55:07.889061       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc774"
E1218 20:55:13.178673       1 controller.go:689] Giving up syncing claim "default/gcs-pvc774" because failures 15 >= threshold 15
E1218 20:55:13.178712       1 controller.go:701] error syncing claim "default/gcs-pvc774": failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large
I1218 20:55:13.178738       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large

[Madhu-1]

I1218 20:55:13.178738 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc774", UID:"e9d999fd-02ef-11e9-b39f-5254006ba3b4", APIVersion:"v1", ResourceVersion:"80775", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": rpc error: code = Internal desc = failed to create volume: etcdserver: request is too large

@rmadaka I think this error message is from the glusterd2 side, please attach gd2 logs.

[rmadaka]

Sending logs through mail, log file size is more