AnsweredAssumed Answered

Openfire service crashing daily

Question asked by ddiggler on Dec 16, 2009
Latest reply on Jan 13, 2010 by Guus der Kinderen

The service will run for a day, maybe two, then it will stop with the error message below.  The error is identical to post http://www.igniterealtime.org/community/message/198731#198731.  I posted a response in that forum 6 days ago with no response.  I figured I try creating my own thread.  I also posted a thread in http://kraken.blathersource.org/node/266.  I have a feeling the problems may be related.

 

We are running Openfire 3.6.4 w/ 1.6.0_17 JVM

Windows Server 2003 SP2 (Virtual Server in VMware ESX)

Dual 2.3Ghz Procs / 3GB of RAM

-Xms1280m -Xmx1280m  (You can see all arguments in the error msg below.  We tried 256/256, 512/512, 1024/1024, 1024/1536 and 512/1024 as recommended in the linked article..same problems)

 

Plugins:

Client Control: Only allow Spark client

Kraken IM Gateway 1.1.2: We control IM Gateway access through groups.  Only 170 users are allowed to used it.

Monitoring Service:  Full archiving is enabled.  User to User and group chats

Red5:  Used for video conferencing test (3 users) and where JWCHAT (~200 JWCHAT users) is setup .  HTTP Binding is enabled for this.

Search: Used for easy searching...duh

User Import Export:  Used for migration of user from old IM solution

 

 

We have 3000 accounts w/ a max of 1500 concurrent users.  We average 1300 to 1400 concurrent users.

 

Running SQL 2005 DB (on a dedicated SQL server)

Microsoft SQL Server Management Studio  9.00.4035.00
Microsoft Analysis Services Client Tools 2005.090.4035.00
Microsoft Data Access Components (MDAC)  2000.086.3959.00 (srv03_sp2_rtm.070216-1710)
Microsoft MSXML    2.6 3.0 6.0
Microsoft Internet Explorer  7.0.5730.13
Microsoft .NET Framework  2.0.50727.3082
Operating System    5.2.3790

 

Error Message...I excluded the process list due to length.  If this will help, I can post it.  If anyone has a solution, I am in desperate need of help. Everything worked great during our testing of 400-500 concurrent users.

 

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 835968 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (414C4C4F434154494F4E0E43505000C7), pid=3188, tid=3168
#
# Java VM: Java HotSpot(TM) Server VM (1.6.0_03-b05 mixed mode)
# If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x4801f800):  JavaThread "CompilerThread1" daemon [_thread_in_native,]

Stack: [0x483b0000,0x48400000)
[error occurred during error reporting, step 110, id 0xc0000005]


Current CompileTask:
C2:2692      org.jivesoftware.openfire.plugin.SearchPlugin.replyDataFormResult(Ljava/util/Co llection;Lorg/xmpp/packet/IQ;)Lorg/xmpp/packet/IQ; (416 bytes)

 

VM Arguments:
jvm_args: -Dexe4j.isInstall4j=true -Dexe4j.isService=true -Dexe4j.moduleName=C:\Program Files\Openfire\bin\openfire-service.exe -Dexe4j.processCommFile=C:\WINDOWS\TEMP\e4j_p3188.tmp -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Dexe4j.consoleCodepage=cp0 -Xrs -Xms1024m -Xmx1024m
java_command: <unknown>
Launcher Type: generic

Environment Variables:
PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\program files\openfire\jre\bin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel

 

---------------  S Y S T E M  ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 15 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2096584k(1255816k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot(TM) Server VM (1.6.0_03-b05) for windows-x86, built on Sep 24 2007 22:20:35 by "java_re" with unknown MS VC++:1310

 

 

 

We get the errors I posted below for a few hours then the OpenFire admin console will stop responding for 30-45 minutes, then the service crashes.

 

------------------------------------------------------------------error--------- ---------------------------------------------------

2009.12.16 11:36:00 [org.jivesoftware.openfire.nio.ConnectionHandler.exceptionCaught(ConnectionHand ler.java:110)
]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at net.kano.joscar.flap.AsynchronousFlapProcessor.<init>(AsynchronousFlapProcessor .java:29)
at net.kano.joscar.flap.ClientFlapConn.init(ClientFlapConn.java:82)
at net.kano.joscar.flap.ClientFlapConn.<init>(ClientFlapConn.java:73)
at net.sf.kraken.protocols.oscar.AbstractFlapConnection.<init>(AbstractFlapConnect ion.java:102)
at net.sf.kraken.protocols.oscar.LoginConnection.<init>(LoginConnection.java:41)
at net.sf.kraken.protocols.oscar.OSCARSession.logIn(OSCARSession.java:120)
at net.sf.kraken.protocols.oscar.OSCARTransport.registrationLoggedIn(OSCARTranspor t.java:95)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:398)
at net.sf.kraken.BaseTransport.processPacket(BaseTransport.java:199)
at org.jivesoftware.openfire.component.InternalComponentManager$RoutableComponents .process(InternalComponentManager.java:619)
at org.jivesoftware.openfire.spi.RoutingTableImpl.routePacket(RoutingTableImpl.jav a:260)
at org.jivesoftware.openfire.PresenceRouter.handle(PresenceRouter.java:164)
at org.jivesoftware.openfire.PresenceRouter.route(PresenceRouter.java:70)
at org.jivesoftware.openfire.spi.PacketRouterImpl.route(PacketRouterImpl.java:76)
at org.jivesoftware.openfire.net.StanzaHandler.processPresence(StanzaHandler.java: 337)
at org.jivesoftware.openfire.net.ClientStanzaHandler.processPresence(ClientStanzaH andler.java:85)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:254)
at org.jivesoftware.openfire.net.StanzaHandler.process(StanzaHandler.java:176)
at org.jivesoftware.openfire.nio.ConnectionHandler.messageReceived(ConnectionHandl er.java:133)
at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived (AbstractIoFilterChain.java:570)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.common.IoFilterAdapter.messageReceived(IoFilterAdapter.java:80)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimplePr otocolDecoderOutput.java:58)
at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecF ilter.java:185)
at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(Ab stractIoFilterChain.java:299)
at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilt erChain.java:53)
at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceive d(AbstractIoFilterChain.java:648)
at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilter.java :239)
at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run(Execut orFilter.java:283)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Unknown Source)

2009.12.16 11:35:51 [org.jivesoftware.util.log.util.CommonsLogFactory$1.error(CommonsLogFactory.jav a:92)
] error sending msg: MSG 59 U 92
====================
     Chunk Debug   
====================
MIME-Version: 1.0
Content-Type: text/x-msmsgscontrol
TypingUser: <user>@hotmail.com

 

====================
Binary Chunk Debug
====================
00000000h: 4D 49 4D 45 2D 56 65 72 73 69 6F 6E 3A 20 31 2E ; MIME-Version: 1.
00000010h: 30 0D 0A 43 6F 6E 74 65 6E 74 2D 54 79 70 65 3A ; 0..Content-Type:
00000020h: 20 74 65 78 74 2F 78 2D 6D 73 6D 73 67 73 63 6F ;  text/x-msmsgsco
00000030h: 6E 74 72 6F 6C 0D 0A 54 79 70 69 6E 67 55 73 65 ; ntrol..TypingUse
00000040h: 72 3A 20 6A 72 61 31 39 37 36 40 68 6F 74 6D 61 ; r: <user>@hotma
00000050h: 69 6C 2E 63 6F 6D 0D 0A 0D 0A 0D 0A             ; il.com......

java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at net.sf.jml.net.Session.sendMessage(Session.java:511)
at net.sf.jml.net.Session.access$1300(Session.java:30)
at net.sf.jml.net.Session$MsgSender.run(Session.java:476)
at java.lang.Thread.run(Unknown Source)

------------------------------------------------------------------error--------- ---------------------------------------------------

Outcomes