I use ldap to auth my users.
But sometimes, xwiki is blocked by ldap, timeout seems no effect.
When it be blocked.
I can see a lot of connections to LDAP server by “netstat -n | fgrep :389”
]# netstat -n | fgrep :389 tcp 0 0 ::ffff:10.134.74.202:57712 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57511 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57764 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57633 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57577 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57579 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57763 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57509 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57597 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57607 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57625 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57714 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57678 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57694 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57609 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57513 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57545 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57781 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57780 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57647 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57601 ::ffff:10.134.100.225:389 ESTABLISHED tcp 0 0 ::ffff:10.134.74.202:57599 ::ffff:10.134.100.225:389 ESTABLISHED
And jstack shows a lot of thread is hold on LDAP releated things:
"Thread-17398" #17509 daemon prio=5 os_prio=0 tid=0x00007f469c001800 nid=0x1514 runnable [0x00007f468a093000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.net.SocketInputStream.read(SocketInputStream.java:223) at com.novell.ldap.asn1.ASN1Identifier.<init>(Unknown Source) at com.novell.ldap.Connection$ReaderThread.run(Unknown Source) at java.lang.Thread.run(Thread.java:745) "Thread-17393" #17504 daemon prio=5 os_prio=0 tid=0x00007f46802bc800 nid=0x14f0 runnable [0x00007f46a0944000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.net.SocketInputStream.read(SocketInputStream.java:223) at com.novell.ldap.asn1.ASN1Identifier.<init>(Unknown Source) at com.novell.ldap.Connection$ReaderThread.run(Unknown Source) at java.lang.Thread.run(Thread.java:745) "http://xwiki.sogou-inc.com/xwiki/bin/loginsubmit/XWiki/XWikiLogin" #17499 daemon prio=5 os_prio=0 tid=0x00007f4658026000 nid=0xf20 waiting for monitor entry [0x00007f46a13fb000] java.lang.Thread.State: BLOCKED (on object monitor) at com.xpn.xwiki.plugin.ldap.XWikiLDAPUtils.getGroupMembers(XWikiLDAPUtils.java:777) - waiting to lock <0x00000006b1428100> (a org.xwiki.cache.infinispan.internal.InfinispanCache) at com.xpn.xwiki.plugin.ldap.XWikiLDAPUtils.isMemberOfGroup(XWikiLDAPUtils.java:815) at com.xpn.xwiki.plugin.ldap.XWikiLDAPUtils.isMemberOfGroups(XWikiLDAPUtils.java:841) at com.xpn.xwiki.plugin.ldap.XWikiLDAPUtils.syncGroupsMembership(XWikiLDAPUtils.java:1048) at com.xpn.xwiki.user.impl.LDAP.XWikiLDAPAuthServiceImpl.syncGroupsMembership(XWikiLDAPAuthServiceImpl.java:499) .... "Thread-17388" #17493 daemon prio=5 os_prio=0 tid=0x00007f4678025000 nid=0xeec runnable [0x00007f46a0df8000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.net.SocketInputStream.read(SocketInputStream.java:223) at com.novell.ldap.asn1.ASN1Identifier.<init>(Unknown Source) at com.novell.ldap.Connection$ReaderThread.run(Unknown Source) at java.lang.Thread.run(Thread.java:745) ...
But ldapsearch command line util can work with the same LDAP server very fast.
]# ldapsearch -D kirbyzhou@sogou-inc.com -W CN=web_pm_all -H ldap://10.134.45.215 CN
Enter LDAP Password: # extended LDIF # # LDAPv3 # base <dc=sogou-inc,dc=com> (default) with scope subtree # filter: CN=web_pm_all # requesting: CN # # web_pm_all, maillist, Sogou, sogou-inc.com dn: CN=web_pm_all,OU=maillist,OU=Sogou,DC=sogou-inc,DC=com cn: web_pm_all # search reference ref: ldap://ForestDnsZones.sogou-inc.com/DC=ForestDnsZones,DC=sogou-inc,DC=com # search reference ref: ldap://DomainDnsZones.sogou-inc.com/DC=DomainDnsZones,DC=sogou-inc,DC=com # search reference ref: ldap://sogou-inc.com/CN=Configuration,DC=sogou-inc,DC=com # search result search: 2 result: 0 Success # numResponses: 5 # numEntries: 1 # numReferences: 3
The only solution I can do now is to restart the xwiki tomcat process.
Anyone can help me to find out the real solution?
XWiki-7.4.3
Centos-6
java-1.8.0-oracle-1.8.0.45