Chef - SOLR problem - unexpected character

Hello everybody,
I tried to google this but i was not able to find the solution.
We have chef server 0.9.12 and corresponding clients.
I found this problem when I wanted to search nodes by role. But it did not
work. The problem is just with new nodes. Older nodes can be updated without
problems and search on older roles also works.

Solr-indexer tries to index data but receives error from SOLR (conection reset
by peer)

[Wed, 12 Oct 2011 18:07:37 +0200] INFO: Indexing node
5817051a-7920-43a0-ad6b-e4b927c0eddc from chef status ok} [Wed, 12 Oct 2011
18:07:38 +0200] FATAL: POST to Solr ‘http://localhost:8983’ failed.
Chef::Exceptions::SolrConnectionError exception: Errno::ECONNRESET: Connection
reset by peer attempting to contact http://localhost:8983 [Wed, 12 Oct 2011
18:07:38 +0200] INFO: Indexing node 5817051a-7920-43a0-ad6b-e4b927c0eddc from
chef status error Errno::ECONNRESET: Connection reset by peer} [Wed, 12 Oct
2011 18:07:39 +0200] FATAL: POST to Solr ‘http://localhost:8983’ failed.
Chef::Exceptions::SolrConnectionError exception: Errno::ECONNRESET: Connection
reset by peer attempting to contact http://localhost:8983 [Wed, 12 Oct 2011
18:07:39 +0200] INFO: Indexing node 5817051a-7920-43a0-ad6b-e4b927c0eddc from
chef status error Errno::ECONNRESET: Connection reset by peer} [Wed, 12 Oct
2011 18:08:18 +0200] FATAL: POST to Solr ‘http://localhost:8983’ failed.
Chef::Exceptions::SolrConnectionError exception: Errno::ECONNRESET: Connection
reset by peer attempting to contact http://localhost:8983 [Wed, 12 Oct 2011
18:08:18 +0200] INFO: Indexing node 5817051a-7920-43a0-ad6b-e4b927c0eddc from
chef status error Errno::ECONNRESET: Connection reset by peer}

And in SOLR log we can see this. sometimes it is unexpected character ‘5’

SEVERE: java.io.IOException: Unexpected character ‘2’ (code 50) expected '>'
at [row,col {unknown-source}]: [2,2362]
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:73)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1290)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:341)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:244)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character
’2’ (code 50) expected '>'
at [row,col {unknown-source}]: [2,2362]
at
com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:648)
at
com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:2971)
at
com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:2936)
at
com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2848)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:272)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
… 22 more

Do You have any idea how to solve this?

Thanks for any idea

I've seen issues similar to this due to memory corruption, fixed by moving
to a newer Ruby runtime.

On Thu, Oct 13, 2011 at 8:59 AM, Jan Zahradka jan.zahradka@mall.cz wrote:

Hello everybody,
I tried to google this but i was not able to find the solution.
We have chef server 0.9.12 and corresponding clients.
I found this problem when I wanted to search nodes by role. But it did not
work. The problem is just with new nodes. Older nodes can be updated
without
problems and search on older roles also works.

Solr-indexer tries to index data but receives error from SOLR (conection
reset
by peer)

[Wed, 12 Oct 2011 18:07:37 +0200] INFO: Indexing node
5817051a-7920-43a0-ad6b-e4b927c0eddc from chef status ok} [Wed, 12 Oct 2011
18:07:38 +0200] FATAL: POST to Solr 'http://localhost:8983' failed.
Chef::Exceptions::SolrConnectionError exception: Errno::ECONNRESET:
Connection
reset by peer attempting to contact http://localhost:8983 [Wed, 12 Oct
2011
18:07:38 +0200] INFO: Indexing node 5817051a-7920-43a0-ad6b-e4b927c0eddc
from
chef status error Errno::ECONNRESET: Connection reset by peer} [Wed, 12 Oct
2011 18:07:39 +0200] FATAL: POST to Solr 'http://localhost:8983' failed.
Chef::Exceptions::SolrConnectionError exception: Errno::ECONNRESET:
Connection
reset by peer attempting to contact http://localhost:8983 [Wed, 12 Oct
2011
18:07:39 +0200] INFO: Indexing node 5817051a-7920-43a0-ad6b-e4b927c0eddc
from
chef status error Errno::ECONNRESET: Connection reset by peer} [Wed, 12 Oct
2011 18:08:18 +0200] FATAL: POST to Solr 'http://localhost:8983' failed.
Chef::Exceptions::SolrConnectionError exception: Errno::ECONNRESET:
Connection
reset by peer attempting to contact http://localhost:8983 [Wed, 12 Oct
2011
18:08:18 +0200] INFO: Indexing node 5817051a-7920-43a0-ad6b-e4b927c0eddc
from
chef status error Errno::ECONNRESET: Connection reset by peer}

And in SOLR log we can see this. sometimes it is unexpected character '5'

SEVERE: java.io.IOException: Unexpected character '2' (code 50) expected
'>'
at [row,col {unknown-source}]: [2,2362]
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:73)
at

org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at

org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1290)
at

org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:341)
at

org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:244)
at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at

org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at

org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at

org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at

org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at

org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected
character
'2' (code 50) expected '>'
at [row,col {unknown-source}]: [2,2362]
at
com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:648)
at

com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(BasicStreamReader.java:2971)
at

com.ctc.wstx.sr.BasicStreamReader.handleStartElem(BasicStreamReader.java:2936)
at
com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2848)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019)
at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:272)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
... 22 more

Do You have any idea how to solve this?

Thanks for any idea