Getting stack overflow error for join sample application
Posted May 13th, 2008 by chinmay_cite
I have used the join example given in the datarush_getting_started.pdf docuement. When I increased the file size upto 1 mb the application is failing. Initialy I was getting out of memory error, so i increase the memory size in the bat file to 512MB and now I am getting stack overflow error. Please help.
Trackback URL for this post:
http://www.pervasivedatarush.com/trackback/212
Running out of memory on a single megabyte file seems unlikely, unless you have a recordSeparator or fieldSeparator property misconfigured? Check that these settings match the format of your data.
Or do you mean a 1GB file? Even on such a big file, the join sample application as provided doesn't use memory intensively. Have you altered it, perhaps by adding a Sort to deal with a non-sorted input? Sort is very memory intensive, and trying to sort a 1 GB file in a 32-bit JVM can cause either of the problems you mention. It is possible to tune Sort to use less memory, but it's usually much easier and simpler to go 64 bit.
I have not changed any thing on file separator it is still (,). I have changed the data on the file. All PRODUCT_ID are same. Some sample id:
PRODUCT_ID,CHANNEL_NAME,UNIT_PRICE
P001,b,6.14
P001,j,7.35
P001,h,7.05
P001,c,9.01
P001,d,8.01
P001,h,3.18
P001,b,6.2
P001,i,1.38
....
PRODUCT_ID,CHANNEL_NAME,UNITS_SOLD
P001,f,861
P001,h,44
P001,j,693
P001,j,836
P001,b,965
P001,i,189
P001,h,870
I don't see it is a problem with data, because it work for 1000 records but when the record size changes to 10000, it is failing.
Getting the following error:
com.pervasive.dataflow.dev.DRCompositeException: example.join.JoinTextFiles{exam
ple.join.JoinTextFiles=com.pervasive.dataflow.compiler.CompilerInternalException
: java.lang.StackOverflowError}
example.join.JoinTextFiles
com.pervasive.dataflow.compiler.CompilerInternalException: java.lang.StackOverfl
owError
at com.pervasive.dataflow.compiler.OperatorCompiler.createCompositeExcep
tion(OperatorCompiler.java:81)
at com.pervasive.dataflow.compiler.OperatorCompiler.compile(OperatorComp
iler.java:42)
at com.pervasive.dataflow.engine.DRJobImpl.compile(DRJobImpl.java:104)
at com.pervasive.dataflow.tools.cli.DRECLI.processLeftoverArgs(DRECLI.ja
va:190)
at com.pervasive.dataflow.tools.cli.CLIBase.doMain(CLIBase.java:231)
at com.pervasive.dataflow.tools.cli.DRECLI.main(DRECLI.java:114)
Caused by: java.lang.StackOverflowError
at java.lang.System.checkKey(System.java:775)
at java.lang.System.getProperty(System.java:649)
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:67)
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.PrintWriter.(PrintWriter.java:78)
at java.io.PrintWriter.(PrintWriter.java:62)
at org.apache.commons.lang.StringPrintWriter.(StringPrintWriter.ja
va:81)
at org.apache.commons.lang.StringEscapeUtils.unescapeJava(StringEscapeUt
ils.java:289)
at org.apache.commons.lang.StringEscapeUtils.unescapeJavaScript(StringEs
capeUtils.java:410)
at com.pervasive.dataflow.util.string.StringUtil.unescapeAndQuote(String
Util.java:81)
at com.pervasive.dataflow.model.parsers.list.gen.ListParserImpl.Literal(
ListParserImpl.java:141)
at com.pervasive.dataflow.model.parsers.list.gen.ListParserImpl.Element(
ListParserImpl.java:124)
at com.pervasive.dataflow.model.parsers.list.gen.ListParserImpl.Elements
SansSpace(ListParserImpl.java:109)
at com.pervasive.dataflow.model.parsers.list.gen.ListParserImpl.Elements
WithSpace(ListParserImpl.java:92)
Thanks for the help. The issue is resolved, there was an issue with record separator.