Apache Ignite Cluster Does Not Start with Persistent Storage -
i have 3 node (server) apache ignite cluster 1 client. using disk based persistent storage. created cache worth 10m records. @ point cluster crashed wanted restart. running into:
- when restart server nodes, throws following exception. have copied exception message below.
- the client blocks , not , not see exception message appears blocking following message.
- i have inlcuded default-config.xml here.
any in resolving issue appreciated. thank you.
server side exception
severe: failed initialize cache. try rollback cache start routine. [cachename=geo10] class org.apache.ignite.ignitecheckedexception: failed verify store file (invalid page size) [expectedpagesize=4096, filepagesize=2048] @ org.apache.ignite.internal.processors.cache.persistence.file.filepagestore.checkfile(filepagestore.java:185) @ org.apache.ignite.internal.processors.cache.persistence.file.filepagestore.init(filepagestore.java:392) @ org.apache.ignite.internal.processors.cache.persistence.file.filepagestore.read(filepagestore.java:291) @ org.apache.ignite.internal.processors.cache.persistence.file.filepagestoremanager.read(filepagestoremanager.java:288) @ org.apache.ignite.internal.processors.cache.persistence.file.filepagestoremanager.read(filepagestoremanager.java:273) @ org.apache.ignite.internal.processors.cache.persistence.pagemem.pagememoryimpl.acquirepage(pagememoryimpl.java:569) @ org.apache.ignite.internal.processors.cache.persistence.pagemem.pagememoryimpl.acquirepage(pagememoryimpl.java:487) @ org.apache.ignite.internal.processors.cache.persistence.gridcacheoffheapmanager.getorallocatecachemetas(gridcacheoffheapmanager.java:515) @ org.apache.ignite.internal.processors.cache.persistence.gridcacheoffheapmanager.initdatastructures(gridcacheoffheapmanager.java:86) @ org.apache.ignite.internal.processors.cache.ignitecacheoffheapmanagerimpl.start(ignitecacheoffheapmanagerimpl.java:139) @ org.apache.ignite.internal.processors.cache.cachegroupcontext.start(cachegroupcontext.java:868) @ org.apache.ignite.internal.processors.cache.gridcacheprocessor.startcachegroup(gridcacheprocessor.java:1935) @ org.apache.ignite.internal.processors.cache.gridcacheprocessor.preparecachestart(gridcacheprocessor.java:1860) @ org.apache.ignite.internal.processors.cache.cacheaffinitysharedmanager.oncachechangerequest(cacheaffinitysharedmanager.java:748) @ org.apache.ignite.internal.processors.cache.distributed.dht.preloader.griddhtpartitionsexchangefuture.onclusterstatechangerequest(griddhtpartitionsexchangefuture.java:773) @ org.apache.ignite.internal.processors.cache.distributed.dht.preloader.griddhtpartitionsexchangefuture.init(griddhtpartitionsexchangefuture.java:574) @ org.apache.ignite.internal.processors.cache.gridcachepartitionexchangemanager$exchangeworker.body(gridcachepartitionexchangemanager.java:1901) @ org.apache.ignite.internal.util.worker.gridworker.run(gridworker.java:110) @ java.lang.thread.run(thread.java:745) sep 10, 2017 2:42:46 pm org.apache.ignite.logger.java.javalogger error severe: failed perform final activation steps [nodeid=2077e165-e8a2-4989-934c-c24c5c0bea80, client=false, topver=affinitytopologyversion [topver=1, minortopver=1]] java.lang.nullpointerexception @ org.apache.ignite.internal.processors.service.gridserviceprocessor.onkernalstart0(gridserviceprocessor.java:240) @ org.apache.ignite.internal.processors.service.gridserviceprocessor.onactivate(gridserviceprocessor.java:370) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor$5.run(gridclusterstateprocessor.java:576) @ org.apache.ignite.internal.util.igniteutils.wrapthreadloader(igniteutils.java:6664) @ org.apache.ignite.internal.processors.closure.gridclosureprocessor$1.body(gridclosureprocessor.java:817) @ org.apache.ignite.internal.util.worker.gridworker.run(gridworker.java:110) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745) class org.apache.ignite.igniteexception: null @ org.apache.ignite.internal.util.igniteutils.convertexception(igniteutils.java:957) @ org.apache.ignite.internal.ignitekernal.active(ignitekernal.java:3427) @ com.accure.ignite.ignitestarter.main(ignitestarter.java:24) caused by: class org.apache.ignite.ignitecheckedexception: null @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor$gridchangeglobalstatefuture.onallreceived(gridclusterstateprocessor.java:816) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor$gridchangeglobalstatefuture.onresponse(gridclusterstateprocessor.java:809) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor.processchangeglobalstateresponse(gridclusterstateprocessor.java:673) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor.sendchangeglobalstateresponse(gridclusterstateprocessor.java:639) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor.access$2200(gridclusterstateprocessor.java:72) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor$5.run(gridclusterstateprocessor.java:597) @ org.apache.ignite.internal.util.igniteutils.wrapthreadloader(igniteutils.java:6664) @ org.apache.ignite.internal.processors.closure.gridclosureprocessor$1.body(gridclosureprocessor.java:817) @ org.apache.ignite.internal.util.worker.gridworker.run(gridworker.java:110) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745) suppressed: class org.apache.ignite.ignitecheckedexception: failed perform final activation steps @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor$5.run(gridclusterstateprocessor.java:589) ... 6 more caused by: java.lang.nullpointerexception @ org.apache.ignite.internal.processors.service.gridserviceprocessor.onkernalstart0(gridserviceprocessor.java:240) @ org.apache.ignite.internal.processors.service.gridserviceprocessor.onactivate(gridserviceprocessor.java:370) @ org.apache.ignite.internal.processors.cluster.gridclusterstateprocessor$5.run(gridclusterstateprocessor.java:576) ... 6 more [14:43:18] topology snapshot [ver=2, servers=1, clients=1, cpus=8, heap=18.0gb] sep 10, 2017 2:43:18 pm org.apache.ignite.logger.java.javalogger error severe: error when executing service: null java.lang.nullpointerexception @ org.apache.ignite.internal.processors.service.gridserviceprocessor.serviceentries(gridserviceprocessor.java:1289) @ org.apache.ignite.internal.processors.service.gridserviceprocessor.access$2000(gridserviceprocessor.java:119) @ org.apache.ignite.internal.processors.service.gridserviceprocessor$topologylistener$1.run0(gridserviceprocessor.java:1578) @ org.apache.ignite.internal.processors.service.gridserviceprocessor$deprunnable.run(gridserviceprocessor.java:1806) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745) client side exception
[14:43:15] message queue limit set 0 may lead potential oomes when running cache operations in full_async or primary_sync modes due message queues growth on sender , receiver sides. [14:43:16] security status [authentication=off, tls/ssl=off] [14:43:16] rest protocols not start on client node. start protocols on client node set '-dignite_rest_start_on_client=true' system property. default-config.xml
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"> <bean id="ignite.cfg" class="org.apache.ignite.configuration.igniteconfiguration"> <!-- enabling apache ignite persistent store. --> <property name="persistentstoreconfiguration"> <bean class="org.apache.ignite.configuration.persistentstoreconfiguration"/> </property> <property name="binaryconfiguration"> <bean class="org.apache.ignite.configuration.binaryconfiguration"> <property name="compactfooter" value="false"/> </bean> </property> <property name="memoryconfiguration"> <bean class="org.apache.ignite.configuration.memoryconfiguration"> <!-- setting page size 4 kb --> <property name="pagesize" value="#{4 * 1024}"/> </bean> </property> <!-- explicitly configure tcp discovery spi provide list of initial nodes. --> <property name="discoveryspi"> <bean class="org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi"> <property name="ipfinder"> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.tcpdiscoverymulticastipfinder"> <property name="addresses"> <list> <!-- in distributed environment, replace actual host ip address. --> <value>127.0.0.1:55500..55502</value> </list> </property> </bean> </property> </bean> </property> </bean> after made changes in default-config use pagesize=2kb, server still not start , show following exception message. here stacktrace.
severe: failed reinitialize local partitions (preloading stopped): griddhtpartitionexchangeid [topver=affinitytopologyversion [topver=3, minortopver=0], nodeid=4a2cb984, evt=node_joined] class org.apache.ignite.ignitecheckedexception: wal history short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1d9, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1da, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1db, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1dc, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1dd, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1de, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1df, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1e0, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1e1, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1e2, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1e3, org.apache.ignite.internal.processors.cache.persistence.wal.filewriteaheadlogmanager$filedescriptor@1e4], start=filewalpointer [idx=0, fileoffset=0, len=0, forceflush=false]]
looks first, started node default pagesize , later changed to:
so ignite can not read storage files because expected find pagesize 4kb while actual store files have page size 2kb.
try set 2kb.
Comments
Post a Comment